Rant: Big Data Denial is for Fossils

Big Data is real but why should we care?

This week I have mostly been getting wound up by Big Data denial. It’s become the latest thing at conferences where IT people gather to mock it, pick holes in arguments and say it’s “just Bigger Data” and a scaled-up version of Rubbish In, Rubbish Out. This is the software equivalent of looking a gift horse in the mouth and suggesting his teeth could use a clean. And these are the same complacent old crocks that say cloud is just a re-run of time-sharing.

These people say Big Data is nothing new and this is a familiar tactic among the seen-it-all-before crew. For them, Big Data is just a re-run of business intelligence. That’s like saying you’re not impressed by the new Ferrari because the wheel was invented way back in the day.

Big Data might be a marketing term but the principles behind it are sound and distinctive. The ability to interrogate (and for once, that term is appropriate) enormous data sets via Hadoop utterly changes the scope of BI. That on its own is A Very Big Thing but what’s just as important, or more important, is that we have a host of complementary capabilities which, when taken together, create a perfect storm.

Two of them are the collapsing price of memory and the rise of multicore processors that help Moore’s Law to persist and make immensely powerful compute capabilities available to a far wider set of constituents than ever before. Another is the rise of in-memory software and appliances like SAP’s HANA so that huge calculations can be made in real time or something very close to it at least.

And finally there is the trifecta of cloud, faster broadband and platforms such as AWS and Windows Azure that make HPC-class compute capabilities available on a per-use basis.

So Big Data is real but why should we care? Quite simply because the whole history of IT in many organisations has been a sombre tale of keeping up with the Joneses. Processes are automated, sure, and here at Acme Corp.  we stay competitive with Zeon Corp. because we too have Excel, PowerPoint and a database with our customers’ names on. And yet forever the Board has been asking IT to do something very different and more important: to provide insights that shake up industries in the same way that the chap at Dixons discovered it was possible to undercut every ISP by having free access, or the employee at Swan Vesta who saved millions by suggesting the company remove one of the two strips of sandpaper form matchboxes.

Of course, Big Data can’t make you think but it gives you the ability to ask questions and get answers fast. That might change the products you sell, the way you go to market and where, price points and a million other things. The Big Data sceptics are very often the people working for old, dying brands and they will be horribly disintermediated. Success turns to failure very quickly today. The internet and the sheer pace of technological trends are creating inflection points everywhere. The dweebs criticising Big Data need to decide which side they want to be on: the future or on the team with Blockbuster, Woolworth and the rest.


Martin Veitch, Editorial Director, IDG Connect