Data Mining

Rant: Big Data and its Luddite Critics

“Why does it have to have such a stupid name?”

The question from the audience at a recent conference followed a good pithy session by an expert in the Big Data field and the questioner was typical of the Professional Curmudgeons who attend these things purely to listen to the sound of their own voices. (Note the way they don’t listen to the answers to their ‘questions’ but rudely chat away and chortle while the respondent patiently attempts to explain.)

It’s become fashionable among idiocrats to criticise Big Data, to dismiss it out of hand and pass it off as just another silver bullet dreamed up in Silicon Valley for the further enrichment of Sand Hill Road VCs.

Fair enough you might say, these techie tall poppies are ripe for scything. But really, that’s not the most scientific attitude.

First up the name. Big Data. I actually quite like it: three syllables, two words, adjective + noun. Simple. Does what it says on the tin. It’s not an abbreviation or acronym that, once decoded, leaves you no wiser (see ERP, CRM etc.). It’s saying data, lots of data, enough data that you can make a big silo and go from there to make sense of the data.

But it’s not actually new, is it? It’s just a warmed-up version of data analytics, data warehousing/marts, business intelligence, EIS and so on, isn’t it?

Nope. It’s Big Data in that there’s a whole new model out there and that’s why it needs a new name that’s going to tell people that it’s a distinct phenomenon and not a re-tread.

So what’s new?

First, computation, memory and storage have fallen in price so we all have access to supercomputers and, even if we haven’t, we can rent the MIPS over a (similarly commoditised) broadband cloud connection.

Second, the data sets are big… no, they’re massive, and they come from everywhere. Stuff coming from sensors, embedded systems, sounds, videos, surveillance cameras, still images and so on, all with lots of valuable metadata which is often more valuable than the data being tagged.

Third, we can do something with the data rather than just admire its scale. That’s because we have technologies like Hadoop and Cassandra, emanating from the internet giants who had to develop new ways to deal with data glut, recognise elements and point them in the right directions. And these technologies are open sourced so we can use them in lots of ways in lots of versions and from lots of vendors.

Fourth, we can do this faster than previously: in part because of the new algorithms and in part because we can afford in-memory computation with no time-wasting paging to disk.

Some supporters (‘Big Uppers’?) of Big Data point to the ‘4 Vs’, meaning volume, velocity, variety and veracity. Not a bad way of remembering the basic tenets of Big Data and like the term itself, eminently sensible.

Of course, you can point to GIGO principle of ‘garbage in, garbage out’ and fret that some fool will not bother to do their due diligence and build towers, aeroplanes, missile control systems and whatnot that lead to the collapse of civilisation. And you can probably get a few press columns for calling for a new breed of data scientists too, without which Big Data reaches its Big Dead End, but any new opportunity demands a tweaking of the usual way of doing things.

Cure cancer? Stop ebola? Understand the roots of hatred? Anticipate tsunamis? The ingredients to prevention come in ones and zeroes and our ability to trawl the sea of information.

Big Data gets criticised because we live with a generation of IT people who have become jaundiced and unable or unwilling to keep up with technologies capable of changing the world in the most fundamental ways.

Their loss.


Martin Veitch is Editorial Director at IDG Connect


« News Roundup: Elections, Killers and Tuits


Top 5 Apps To Help Your Business Deliver Projects More Effectively »
Martin Veitch

Martin Veitch is Contributing Editor for IDG Connect

  • twt
  • twt
  • Mail


Do you think your smartphone is making you a workaholic?