Security tool key player in Hadoop plot to rule the world

Open source platform celebrates 10 years but has it come of age?

The dancing balloon-like lights moving in time to the backing track of Michael Jackson’s Thriller was merely the entrée for the entertainment that was to follow. The stage at Dublin’s convention centre, a largely glass structure adjacent to the newish, harp-like Samuel Beckett bridge over the Liffey, was soon swamped with Bodhran players and the impressively fast clicking heels of traditional Irish dancers. All good local stuff. Slightly less impressive were the keynotes at this year’s Hadoop Summit. Too salesy and too egocentric, it was easy to forget this was an open source event celebrating the future of connected data.

Of course, Hortonworks, a main sponsor of the event and key exponent of Hadoop, had every right to fanfare its achievements and showcase its new products and partners. No event is ever without its fair share of chest beating but every event also needs greater perspective, a sense of actuality. While David Walker from WorldPay provided customer perspective – the payments firm crunches 13 billion transaction a year apparently - it was left to Arun C. Murthy, co-founder and architect at Hortonworks, to deliver some sense of reality – not easy for a techie that was brave enough to put up a slide full of source code.

Murthy talked about “actionable intelligence”, how access to increased quality and analysed data is leading to smart manufacturing, autonomous cars, future farming projects and real time cybersecurity. The latter is a key one.

So what is new? Certainly security is central to the announcements with Hortonworks revealing its integration with Apache Metron, an open source analytics tool to identify malicious activity. According to Hortonworks, Metron helps users to process unprecedented volumes of data per second. When an organisation is attacked, Metron users can process and compare data from comprehensive feeds across the platform in real time.

“This not only facilitates enhanced detection of malware campaigns, but also impacts the economics for attackers by requiring them to customise malware for each target,” said a company statement.

Jamie Engesser, vice president and general manager of emerging products at Hortonworks, said “traditional security tools with a rules-based approach do not scale to match the speed and frequency of modern cybersecurity threats, and that is why we are so excited about the Apache Metron community's momentum aimed at tackling this issue for the enterprise.”

If Hortonworks can get this bit right it could certainly be well placed to reap the benefits of increased enterprise data management issues. CTO Scott Gnau produced an interesting slide showing Hadoop expansion which ultimately engulfed the zettabyte category of data platforms covering sensors, infotainment systems, wearable devices, cyber security logs, connected vehicles and machine data. This is the vision, what Gnau refers to as “the data tipping point”.

The issue though, says Gnau is that the incremental approach cannot work. It needs a new approach, he says with “HDP [Hortonworks Data Platform] the centre of gravity for the modern data architecture.” Of course he’d say that, he works for the company but the fact Pivotal has chosen to resell it in favour of its own distribution is telling.

There were some additional announcements here, designed to strengthen the Hortonworks offering. The integration of Apache Atlas (a product of Hortonworks' Data Governance Initiative) and Apache Ranger will bring both data governance features and role-based access control across the stack. There is also a new release of its management console for Hadoop clusters, Apache Ambari and Hortonworks announced it is reselling Syncsoft to help with legacy platform integration.

But security was the key component. The company believes that this will give organisations confidence in adopting connected data platform strategies, removing the silos and bringing together an intelligent system that can manage all types of data, whether in rest or in motion, offering analytics at the edge of the network or deeper analytics centrally.

The future though is reserved for the application developers. Matt Morgan, Hortonworks’ VP for product and alliance marketing, says that by providing a connected solution for managing data at rest and data in motion, the company is providing a platform on which developers can build new data analytics software applications, relevant to their verticals and solving specific industry problems.

It’s through these partnerships and relationships that the company believes will enable Hadoop and Hortonworks in particular to ‘manage the world’s data’, a marketing refrain which may after all turn out to be more than just spin.


Related reading:

Hortonworks plots IPO, slates Cloudera

Hortonworks no longer hears a ‘who?’