Data Mining

Databricks has designs on democratising Deep Learning

This week saw Databricks, the company whose founders created the Apache Spark framework, make a new pitch for simplifying Deep Learning and Artificial Intelligence.

“AI is changing the world,” said Ali Ghodsi, CEO and co-founder of Databricks, during the company’s 2016 Spark Summit being held in Brussels, Belgium. “But one of the things we don’t often talk about is that the companies really doing well with AI are these really big companies with thousands of engineers.”

Ghodsi says the reason smaller companies with limited resources struggle is down to the extreme ‘technical debt’ that Machine Learning comes with.

“Most of the time and effort of building machine learning systems goes into configuring them, collecting these massive amounts of data that these algorithms need, doing feature engineering, extracting the features that you need, tuning that, and then running it through machine learning, then doing the verification, using tools to make sure that you’re managing all these resources that you have.

“The hard part of this is really all the other stuff that goes around it, not necessarily running the algorithm. So how do we democratize this?”

The company first released the Open Source TensorFrames - a software library that enables the Google’s Tensorflow deep learning framework to run on Spark – in March. The new announcement adds out-of-the-box support for using TensorFrames with GPUs.

The company says the new features will allow companies to quickly and easily add machine learning capabilities to their data sets, whether it’s recognizing handwriting, translating speech between languages, or distinguishing between malignant and benign tumors.

In a demonstration, Ghodsi took a picture of the UK’s foreign secretary, and trained clusters to hallucinate images of dogs within the image. While teaching machines to create odd images is nothing new, the entire demo was done almost from start to finish in a ten-minute slot using Databricks’ cloud-based Spark platform.

According to Databricks’ latest Spark survey, utilization of Machine Learning has increased by more than 30% in the last year.

Spark Summit EU 2016 Keynote - Databricks CEO Ali Ghodsi from Databricks on Vimeo.


« Why big data is hype… but data analytics isn't


The real meaning of… Wiki »
Dan Swinhoe

Dan is a journalist at CSO Online. Previously he was Senior Staff Writer at IDG Connect.

  • twt
  • twt
  • twt
  • Mail