How Microsoft's Windows ML machine learning could help apps anticipate your needs

Microsoft is using machine learning to make its Windows apps a lot smarter. At a Technology Sneak Peek on Monday at Microsoft's Build developer conference in Seattle, executives used Windows Ink to demonstrate the potential of Windows ML, a machine learning API that debuted in the April 2018 Update to Windows 10.

Because Windows ML is an API, regular users will never encounter it. But the API provides the framework for developers to tap the power of your CPU or GPU to anticipate and act upon your needs, in Windows and Windows apps.

The executives at the demo made it clear that the Ink features they were showing off could drastically change, or might never be released at all. Still, the future they showed moves a few steps forward from the way Windows works today.

Case in point: Today, OneNote’s app for Windows 10 is smart enough to translate an inked math problem into a solution that can be solved. But each time you want to perform that action, you have to “lasso” that equation with your pen, signalling to Windows what you want it to do.

The Windows Ink teams showed off how Windows ML could address that and other issues with the inking experience. For example, ink could be used to:

  • Ink the values you want on a slider bar, rather than making you move the slider back and forth until you find the right value.
  • Allow Windows to recognize when users are inking math, a flowchart, or a table.
  • Change behavior when ink is used in OneNote, as opposed to Windows’ Sticky Notes or WordPad apps.
  • Alter the texture of a paintbrush application by speed.
  • Use the recognizer to give Office Lens OCR capabilities.
Windows Ink Mark Hachman / IDG

Windows ML could adjust the depth and texture of an ink brush on the fly.

The Office Lens OCR example shows the potential of the recognizer platform created with Windows ML. Office Lens is a mobile app that’s superb at, say, aligning a photo of a whiteboard for easy reading. But it can’t yet “read” the text that’s been written on it. The ability to “see” the real world has been a Microsoft priority, and machine learning could move that priority forward for Office Lens and more.

As for whether Office Lens would actually gain this capability, Kyle Beck, a program manager at Microsoft said that his team just designs the inking platform—where it ends up isn’t his decision. But, he added, Office Lens would be the “natural fit” for it.

Windows Ink Mark Hachman / IDG

This is how Ink will (or. at least, could) recognize inked equations. 

IDG Insider


« How technology helped foil the rhino poachers


Pillars of Eternity II: Deadfire review: A strong ship in shallow waters »
IDG News Service

The IDG News Service is the world's leading daily source of global IT news, commentary and editorial resources. The News Service distributes content to IDG's more than 300 IT publications in more than 60 countries.

  • Mail

Recommended for You

Trump hits partial pause on Huawei ban, but 5G concerns persist

Phil Muncaster reports on China and beyond

FinancialForce profits from PSA investment

Martin Veitch's inside track on today’s tech trends

Future-proofing the Middle East

Keri Allan looks at the latest trends and technologies


Do you think your smartphone is making you a workaholic?