What should you know about mixed reality?

What should you know about mixed reality?

Forget augmented reality (AR) and virtual reality (VR), mixed reality (MR) is set to change the way we think about how the real and the virtual interact. It is the merging of the real and virtual worlds to “produce new environments and visualizations where physical and digital objects co-exist and interact in real time” and it has applications across a range of areas from communications to entertainment and logistics. Proponents are saying it can extend the Internet of Things in new directions as well.

Despite the hype, there is controversy about what MR really is and whether it exists at all. In fact, some claim that it is a move by Microsoft to claim augmented reality for its own. There are also varying ideas about what differentiates MR from AR.


What is mixed reality?

Daniel Eckert, PriceWaterhouseCooper’s Managing Director of Emerging Technologies, believes that, unlike AR, MR requires a Head Mounted Display (HMD). Rob Minson, CTO at Fracture Reality, explains that MR combines the immersive capabilities of VR – a head-mounted optical device rather than a small mobile phone window – with AR’s ability to bring data into the physical world. This powerful combination makes it a medium suited to both consumer and enterprise domains and marks a new paradigm in the way people interact with an increasingly data-rich world.

Meet one of the early (pre-JavaScript) VR pioneers: VR pioneer Tony Parisi about the future of the industry

For Microsoft’s Greg Sullivan, Director of Communications for Windows and Devices, mixed reality describes the continuum between the physical and digital worlds, from augmented reality to virtual reality and everything in between. 

“Mixed reality blends these worlds to produce new environments where physical and digital objects co-exist and interact in real time. Bringing together the digital and 3D physical worlds enables us to do impossible things and is the future of computing,” he says.

Gil Shefler, marketing manager and VR evangelist at Kaltura, says that the line between augmented and mixed reality is blurry and that, once, they referred to the same thing. Increasingly, AR refers to an “annotated” reality, Shefler says, drawing on the ideas of futurist Jaron Lanier who defined it as: a medium where things like texts and graphs are projected on a person’s field of vision providing information on their surroundings. “MR is when whole images or objects are projected, mixing virtual and real worlds and the two interact,” Shefler says.

The lack of a killer app or product – the expected release of Magic Leap’s device this year may change this – means that there is no clear leader as yet, although a number of developers have staked their claim, most notably Microsoft.


What might mixed reality be used for?

There are already a number of applications of MR in numerous business sectors. Ford, for example, is using HoloLens to unleash creativity in design, by allowing designers to experiment with prototypes without having to physically build every design in clay. Similarly, businesses like thyssenkrupp are using it to drive sales by enabling reps to provide customers with a visualization of the stair lift they’re looking to install in their homes.

Where are we at with VR training? Check out: How virtual reality is changing workplace training

Sullivan adds: “By transforming their business processes, thyssenkrup are able to streamline internal processes whilst also helping their customers make a significant addition to their home with much more confidence.”

The ability to blend digital information and objects seamlessly into our physical world creates a huge potential step change in the way we use computers. Minson explains that perhaps most disruptively, MR will make physical screens a thing of the past. “Want a 60-inch plasma screen on your wall? Just put it there. Now want that same display pocket sized and mobile? No problem,” he says.

He also believes that MR will change how we collaborate. The natural way for humans to interact is to gather around together in a shared space, to talk and to gesture. Traditional 2D screens and video conferencing are a massive step backwards in terms of communication and understanding, but when it comes to digital information they have been our only option until now. “MR allows us to collaborate both locally and remotely around digital objects and data using the same human tools we use for physical objects. These could be interactive architectural models, medical scan data, or 3D data visualizations,” he says.

For retailers, Shefler says, mixed reality means a display at a store might always be full. Computer-generated images of missing items can be projected in their physical absence. In fact, new items that have not yet been released can be on display using graphics. “Taking it a step further, virtual salespeople might replace real ones – a combination of AI and MR,” Shefler says. “You might be able to look at furniture at a store and project it onto your room to see if it fits.” Ikea is already doing similar things in this area.

Eckert says much of the application of MR lies in the Field Service, manufacturing, and training areas. He says that empowering field service personnel with real-time data (and objects) is the killer app for AR/MR.

He also sees great potential in 3D analytics and visualization. Data visualization has been a challenge as it is limited to a bi-planer experience. “Putting the data in 3D and flying and/or manipulating it in 3D enables data scientists (and the Joe Sixpack for that matter) to look at data models in completely new ways.”

Minson adds, by scanning the world around us, MR devices can superimpose data in position over the top of objects or physical spaces. This capability has huge implications for empowering workers in complex, data-rich environments such as engineering, complex manufacturing or emergency response.

Zappar, for example, recently worked with Kindred Group to explore data visualization in high-profile sporting matches using the 2017 Champions League final and a T20 cricket match between South Africa and England from earlier this year as examples. Users were able to explore the team’s players, with insights into their goals, shots or runs – interacting and unearthing new stats whilst putting the user at the heart of the data.

Sullivan believes that the companies that embrace mixed reality as part of a broader digital transformation journey will be the ones who unlock its full potential. “Understanding that mixed reality can complement a business’ transition to cloud or its integration of AI will empower them to use mixed reality to foster remote collaboration and innovative creativity.”

For Ajay Singh, Director at Samsung NEXT Ventures, one of the investment arms of Samsung, focusing on emerging technology such artificial intelligence and VR/AR, the key to leveraging the potential of MR is to ask what real problem you are trying to solve, and will the use of mixed reality really provide a solution. He cautions that mixed reality can’t be approached as a flashy marketing ploy; it must align with the core values of the company and provide real value.

“We haven’t seen a remarkable use-case for mixed reality on the market, yet, and we are still in the early phases of the technology’s life, but it’s not difficult to see the possibilities,” he says. “The largest hurdle is to define the correct use-case for the technology that impacts a company’s core values. There are examples of early adopters of mixed reality out there now that provide very basic level understanding of the potential of the technology and have seen user engagement grow as a result, and I expect engagement to become more and more of a factor as the technology becomes more accessible.”


What do businesses need to understand about mixed reality?

Eckert emphasizes the need for those businesses wanting to invest in MR to really understand what mixed reality is and what it isn’t and to cut through the confusion in the marketplace. “It’s critical that decision makers understand the difference between AR, VR, and MR – and most importantly when and where to use the right technology to solve the problem they are addressing,” he adds.

In addition, a multi-disciplinary team is vital to the success of a MR project. Eckert says that while you could argue that a design agency, game studio, a creative team, a software development team, and so on is all that as needed to create an MR solution, that might be okay for a simple proof of concept. If you are thinking of deploying an MR solution in the enterprise you need to add many more skills into the equation, he says, listing the need to include:

    1. Security: if a company is going to deploy 1,000 HMDs at a cost of over $2.5 million into the field, there needs to be security, provisioning, asset tracking, central management, and so on.
    2. Infrastructure: Most of the HMDs are tethered – and network connectivity on a factory floor is challenging – even in the best of factories. Companies need to consider network connectivity, latency, data encryption, storage, and battery life.
    3. Ergonomics: Wearing an HMD for more than one to two hours can be exhausting and disorientating if the HMD is too heavy, too hot, too distracting. The lenses might get too dirty or the HMD might not work in low light. “It took 34 years for industrial designers to get the feel of the mobile phone right – my point for MR HMDs – it won’t be accomplished overnight,” Eckert says.
    4. User experience: Companies need to consider how the interaction with MR content will take place, whether by gestures, voice commands, joysticks, gloves, or wands, and then they need to train the users how to interact with their environment as there is no standard user interface currently. Bringing voice into the equation adds a whole host of new problems from noise, wind, external voices, language barriers to context and local vernacular. All this must be designed and managed to get the most out of the system.

Addressing these aspects in addition to the core content design will aid the success of an MR project. Content, Eckert says, is really king when it comes to MR, and creating the content requires a strong understanding of art, programming, 3D design, modeling, simulation, AI, lighting, context/intent, and physics. “If you do one thing – learn as much as you can about how to create, build, and manage content for the MR world – because no matter the hardware you select – the objects you create can be repurposed to run on different platforms,” he says.

Eckert adds that just because you can, doesn’t mean you should invest in MR, adding that for now – less is more. David de la Pena, Head of Technology at digital agency DARE West, also cautions that the tech is still maturing. “It’s easy to get caught up in the hype and novelty of any new technology, however be prepared to face a number of barriers and problems to solve. What I would say is that we’re in a period of exciting experimentation. So for any company looking to invest in MR, they need to understand that they won’t see results right off the bat.”

De la Pena adds that the priority should always be added value to the end user, rather than simply a great demonstration of the technology at work. “Real use will come only where it makes sense - where it brings true value. For the best outcome, up-front user research is an essential part of the process for this,” he says.


«12 technologies that can extend Moore’s Law


Are foreign tech firms really a threat? »
Bianca Wright

Bianca Wright is a UK-based freelance business and technology writer, who has written for publications in the UK, the US, Australia and South Africa. She holds an MPhil in science and technology journalism and a DPhil in Media Studies.

Our Case Studies

IDG Connect delivers full creative solutions to meet all your demand generatlon needs. These cover the full scope of options, from customized content and lead delivery through to fully integrated campaigns.


Our Marketing Research

Our in-house analyst and editorial team create a range of insights for the global marketing community. These look at IT buying preferences, the latest soclal media trends and other zeitgeist topics.



Should the government regulate Artificial Intelligence?