hadoop100457390orig

Big data gets runtime specification

The Open Data Platform Initiative (ODPi) released its first ODPi Runtime Specification and test suite Monday as part of its goal to ensure a standard deployment model for enterprise big data applications across Apache Hadoop distributions.

"This is the culmination of this whole year's work," says John Mertic, senior manager of ODPi.

The nonprofit ODPi formed last year in an effort to reduce the amount of complexity surrounding the Hadoop and big data environment. The idea was to provide a big data kernel in the form of a tested reference core of Apache Hadoop, Apache Ambari and related Apache source artifacts.

The kernel, called ODPi Core, would be used to simplify upstream and downstream qualification efforts — a "test once, use everywhere" core platform that could eliminate the growing fragmentation in the space. Applications and tools built on the reference platform should integrate with and run on any compliant system.

In September of last year, ODPi officially became a collaborative project of the Linux Foundation.

Mertic explains ODPi is an effort to bring together constituents from all the various "party lines" with a stake in the big data ecosystem.

"What we really wanted to do was to make sure we could have the community well represented," he says. "The biggest feedback that we got was that each distro does things slightly differently; they name their files differently; their APIs behave differently."

The new runtime specification descends from Apache Hadoop 2.7 and features HDFS, YARN and MapReduce components. Mertic says the test framework and self-certification aligns closely with the Apache Software Foundation by leveraging Apache Bigtop for comprehensive packaging, testing and configuration. More than half the code in the latest Bigtop release originated in ODPi. The ODPi Runtime-Compliance tests are linked directly to lines in the ODPi Runtime Specification. To assist with compliance, ODPi has also provided a reference build.

The organization says the published specification includes rules and guidelines on how to incorporate additional, non-breaking features, which are allowed provided source code is made available through relevant Apache community processes.

"It was a little over a year ago that ODPi was formed, and we have already proved beneficial to upstream ASF projects (Hadoop, Bigtop, Ambari)," says Roman Shaposhnik, director of Open Source at Pivotal, and an Apache Hadoop and Bigtop committer and ASF member. "This is why the first release of the ODPi Runtime Specification and test suite is so exciting. It is a big step toward realizing our goal of accelerating the delivery of business outcomes through big data solutions by driving interoperability on an enterprise-ready core platform."

"Big data is the key to enterprises welcoming the cognitive era and there's a need across the board for advancements in the Hadoop ecosystem to ensure companies can get the most out of their deployments in the most efficient ways possible," Rob Thomas, vice president of product development, IBM Analytics, added in a statement Monday. "With the ODPi Runtime Specification, developers can write their application once and run it across a variety of distributions — ensuring more efficient applications that can generate the insights necessary for business change."

With the Runtime Specification out the door, Mertic says the next focus will be the ODPi Operations Specification to help enterprises improve installation and management of Hadoop and Hadoop-based applications. It covers Apache Ambari, which is used for provisioning, managing and monitoring Hadoop clusters. Mertic expects the Operations Specification will be ready this summer.

The ODPi is also getting ready to decide what it will focus on after that. Mertic explains that each ODPi member, regardless of size or investment, has exactly one vote. Some possibilities include work around Spark, Kafka, HBase and Hive.

IDG Insider

PREVIOUS ARTICLE

« Microsoft's cross-platform gaming push gets real with Killer Instinct for Windows 10

NEXT ARTICLE

The Internet of Things is all fun and games until a racist takes over your printer »
author_image
IDG News Service

The IDG News Service is the world's leading daily source of global IT news, commentary and editorial resources. The News Service distributes content to IDG's more than 300 IT publications in more than 60 countries.

  • Mail

Recommended for You

International Women's Day: We've come a long way, but there's still an awfully long way to go

Charlotte Trueman takes a diverse look at today’s tech landscape.

Trump's trade war and the FANG bubble: Good news for Latin America?

Lewis Page gets down to business across global tech

20 Red-Hot, Pre-IPO companies to watch in 2019 B2B tech - Part 1

Martin Veitch's inside track on today’s tech trends

Poll

Do you think your smartphone is making you a workaholic?