Microsoft revamps machine learning tools for Apache Spark

Microsoft revamps machine learning tools for Apache Spark

Microsoft has revamped its MMLSpark open source project, the better to integrate “many deep learning and data science tools to the Spark ecosystem,” according to the notes on the project repository.

MMLSpark, originally released last year, is a collection of projects intended to make Spark more useful in many contexts—mainly machine learning, but also in some general-purpose ways.

Some of MMLSpark’s features integrate Spark with Microsoft machine learning offerings such as the Microsoft Cognitive Toolkit (CNTK) and LightGBM, as well as with third-party projects such as OpenCV. Others are about turning Spark into a service or client—for example, allowing Spark computations (including machine learning predictions) to be easily served via the web, or allowing Spark to interact with other web services via HTTP. One function, LIME on Spark, provides annotated results for the predictions served by a given image classifier, an at-a-glance way to determine if the classifier is working right.

MMLSpark wraps all these functions in a set of APIs available for both Scala and Python. The repository contains some quick-start examples, such as using web services in Spark, using OpenCV on Spark for image manipulation, and training a deep image classifier using Azure VMs with GPUs.

MMLSpark itself can be installed on existing Spark clusters as a package, used in the Databricks cloud (or a Databricks appliance on Azure), installed directly in an instance of Python or Anaconda, or run in a Docker container. Integration is also available for the R language, but right now only via a beta auto-generated wrapper.

IDG Insider

PREVIOUS ARTICLE

« Scoring our iOS 12 wishlist

NEXT ARTICLE

iPhone XR review roundup: Worth the wait »
author_image
IDG News Service

The IDG News Service is the world's leading daily source of global IT news, commentary and editorial resources. The News Service distributes content to IDG's more than 300 IT publications in more than 60 countries.

  • Mail

Recommended for You

International Women's Day: We've come a long way, but there's still an awfully long way to go

Charlotte Trueman takes a diverse look at today’s tech landscape.

Trump's trade war and the FANG bubble: Good news for Latin America?

Lewis Page gets down to business across global tech

20 Red-Hot, Pre-IPO companies to watch in 2019 B2B tech - Part 1

Martin Veitch's inside track on today’s tech trends

Poll

Do you think your smartphone is making you a workaholic?