speedupfastquick5100643069orig

Big data, but faster: API speeds links between apps and storage

Alluxio, originally known as Tachyon, is giving big data applications fast, unified access to the storage where their data resides.

Now at version 1.0, Alluxio provides frameworks like Spark, MapReduce, Flink, or Presto with access to multiple types of storage systems. Cloud storage providers Amazon S3, Google Cloud Storage, and OpenStack Swift are supported, alongside storage vendors EMC and NetApp.

From the outside, Alluxio might seem like an in-memory caching system like Memcached or Redis. Instead, it's a layer that sits between distributed computing applications and storage, giving the former access to the latter via a unified API. Applications can use Alluxio's API, which offers the highest possible speed, or they can use legacy APIs (an HDFS implementation, for instance), which are slower but more compatible.

In a blog post published earlier this month, engineers at Intel described how Alluxio helps address a few common problems with big data frameworks, such as sharing data between applications. Rather than write data to HDFS and read it back out again, users can write data to Alluxio's in-memory store and read it back out at far greater speed.

Likewise, the JVM's garbage collection and on-heap cache issues, which are exacerbated by frameworks like Spark, can be alleviated by using Alluxio. IBM has claimed that back in the Tachyon days, Alluxio outperformed in-memory HDFS by 110x for writes and "improves the end-to-end latency of a realistic workflow by 4x."

Alluxio complements other solutions; Apache Arrow, for instance, speeds up data processing by making it available to an application in a format that suits modern CPUs. Data requested by Arrow would be fetched from storage and provided by Alluxio.

In its Tachyon incarnation, Alluxio drew support from several big data projects, Spark chief among them. The company plans to continue building support from other big data projects and storage system vendors.

IDG Insider

PREVIOUS ARTICLE

« Here's what tech leaders have said about the Apple-FBI dispute so far

NEXT ARTICLE

Asus settles charges over insecure routers and cloud services »
author_image
IDG News Service

The IDG News Service is the world's leading daily source of global IT news, commentary and editorial resources. The News Service distributes content to IDG's more than 300 IT publications in more than 60 countries.

  • Mail

Recommended for You

International Women's Day: We've come a long way, but there's still an awfully long way to go

Charlotte Trueman takes a diverse look at today’s tech landscape.

Trump's trade war and the FANG bubble: Good news for Latin America?

Lewis Page gets down to business across global tech

20 Red-Hot, Pre-IPO companies to watch in 2019 B2B tech - Part 1

Martin Veitch's inside track on today’s tech trends

Poll

Do you think your smartphone is making you a workaholic?