Mixed Reality

with Dr. Marc Pollefeys

Dr. Marc Pollefeys is a Professor of Computer Science at ETH Zurich, one of the world’s leading universities for science and technology. He is a Director of Science at Microsoft. Bringing these roles together, he leads Microsoft’s Mixed Reality and AI lab in Zurich, leading a team of scientists and engineers to develop advanced perception capabilities for HoloLens.

Episode 2: Mixed Reality

Public Sector Future

Episode summary

When virtual reality is integrated into our physical reality, we create mixed reality. We speak with Dr. Marc Pollefeys, Professor of Computer Science at ETH Zurich and Director of Science at Microsoft who leads Microsoft’s Mixed Reality & AI lab in Zurich. Learn about emerging areas of technology from 3D computer vision to AI, and practical applications of HoloLens in the public sector.  

Download the episode transcript

Listen to this episode on any of these podcast platforms:

Spotify

Apple Podcasts

Google Podcasts

Deezer

Libsyn

When virtual reality is integrated into our physical reality, we welcome mixed reality. 

We are joined by Dr. Pollefeys, who leads Microsoft’s Mixed Reality and AI lab in Zurich.

Dr. Pollefeys is best known for his work in 3-D computer vision, having been the first to develop a software pipeline to automatically turn photographs into 3-D models. He also works on robotics, graphics, and machine learning problems. 

We explore how these technologies are increasingly emerging as part of our day-to-day lives, look at where these could be used and what this could mean for the public sector.

What is Mixed Reality?

Mixed reality is about being able to combine the real physical, visual world in front of us with virtual elements and the interaction between virtual and real elements.  

“When we talk about what is known as virtual reality, we’re talking about fully going into a virtual world that’s decoupled from the real physical world. With mixed reality, beyond augmented reality, we’re talking of having meaningful interactions between the physical, visual world in front of us, and also these virtual elements that we add to it,” says Pollefeys. 

HoloLens 

Dr. Pollefeys and his team use HoloLens as their key instrument to enable mixed reality. HoloLens devices are handsfree, worn like glasses. By looking around, the person wearing a HoloLens can access information that is placed wherever it is relevant. HoloLens is responsive to the wearer’s voice and tracks their hands. There are natural user interfaces that allow tasks to be done in the real world while getting help from the device.  

Mixed reality tutorials

Instead of needing an instructor or a digital/paper guide, tutorials can be put on a mixed reality device for handsfree experience. Step-by-step instructions can be provided for fixed scenarios, like maintenance of a vehicle or machine.    

“If it’s something that a lot of people have to learn, it’s very efficient to build this kind of mixed reality tutorial,” says Dr. Pollefeys.    

While wearing the mixed reality device, “you look at the object where you have to do the maintenance, and then overlaid on the object, on the actual object of interest, it can point to you, like first, turn this knob here, and then you can do this, in a way that’s not to be misunderstood,” Dr. Pollefeys continues.

COVID-19 support in hospitals  

The HoloLens has been used in hospitals in the UK during the pandemic. A nurse wore a HoloLens to interact with patients so the doctor could see the patients remotely without having to enter the room. Depending on the situation, a doctor could enter the room if there was a need for it.

This saved doctors time when visits could be handled remotely because they could avoid time spent entering and exiting rooms with the disinfection protocols.  

Doctors ended up not needing to physically enter most rooms, “And so, that was a huge productivity gain, which was critical, of course, when there were not enough doctors to handle all the patients when the hospital was overloaded,” says Dr. Pollefeys. 

How the public sector can begin to implement this technology

Microsoft has two solutions for this that can be implemented today. Dynamics 365 Guide can be used to make mixed reality tutorials. With this computer program and a HoloLens, you can define, assemble, and prepare tutorials. People with a HoloLens can access the tutorials. 

Dynamics 365 Remote Assist can be used for people to connect remotely via a HoloLens. This allows people to easily connect for Teams meetings through a HoloLens device.  

To find out more:

Mixed Reality & AI Lab, Zurich – Microsoft Research

Microsoft Dynamics 365 mixed reality documentation

Learn more about how Microsoft support digital transformation in Government