XR typically refers to the spectrum of spatial technologies that cover the gamut between virtual (VR) reality and augmented reality (AR). The world is largely familiar with the concept of VR, the complete immersion in a digital space. It helps that VR is already available on consumer platforms such as Oculus Quest and PlayStation VR. Less understood is AR, which seeks to enhance the real world by overlaying the digital world on top it. Even within these two categories of XR, there are additional levels of immersion—some smart glasses feature a single monochrome display. Others, such as Hololens or Magic Leap, provide more advanced iterations of AR.
While many are familiar with the concept of VR, the market and the world are still becoming acquainted with AR. While VR is already available on consumer platforms such as Oculus Quest and PlayStation VR, consumer AR that everyone wants is still years away. People want AR glasses to have the graphical capabilities of today’s Oculus Quest, inside the fashionable form factor of today’s Ray Ban Stories. However, that’s simply not possible with today’s optical, wireless, battery and processing capabilities. Qualcomm is at the forefront of enabling headworn AR and VR solutions, including the Oculus Quest, Ray Ban Stories, and many other headsets.
I recently spoke with Qualcomm’s GM of XR, Hugo Swart about the company’s vision and foundational work on Mixed Reality. The company has been building chipsets and platforms for XR since the early days and had one of the earliest AR platforms called Vuforia in 2012 and enabled pioneers in headworn AR like ODG. Qualcomm Technologies were also pivotal in enabling early smartphone VR solutions as well as the first Standalone VR headsets including the wildly popular Oculus Quest 2.
Why we need Mixed Reality
In the meantime, technologies like RGB video pass-through AR or Mixed Reality are bridging the gap on the way to AR. You may have heard Microsoft use the term “Mixed Reality” to describe its XR platform. While this is similar, in the context of this column, Mixed Reality refers to enabling both AR and VR within a single VR device. Microsoft’s definition refers to different devices sharing the same spatial computing platform and potentially space. In addition to possessing VR capabilities, Qualcomm’s vision for a Mixed Reality device must be able to use passthrough color cameras to recreate a feed as close to the real world as possible for the purpose of AR.
Mixed Reality capable VR devices must also be lighter and more compact than current VR devices—they need to feel almost invisible to the user much like a pair of AR glasses would be. We’re already seeing this in devices targeted towards Mixed Reality, such as the Lynx R-1 which runs on Qualcomm Snapdragon XR2 chipset. I expect that we’ll see similar devices from Facebook, Apple and other tech giants pursuing AR very soon
Right now, Qualcomm believes there are four paths within AR that are evolving in parallel, on different parts of the XR spectrum:
- Smart glasses, which range from simple head-mounted viewers with a single display (or none at all), to those that provide a complete dual display 3D AR experience.
- Enterprise AR headsets, which perform all computing on-device with very high fidelity and can leverage remote rendering over Wi-Fi or 5G to improve graphics quality.
- Tethered headsets, which rely on a wired or wireless connection to another computing device for connectivity and compute, like a smartphone or a PC.
- Virtual Reality devices with MR, which combine VR and AR capabilities.
Eventually, these capabilities will merge to the point where it becomes difficult to tell the difference between each category. The lines will become very blurry, like what we’re seeing with VR today.
Mixed Reality features
In speaking with Hugo I learned that there are a common series of challenges that must be overcome in order to fully enable this new form of spatial computing. First, Mixed Reality headsets require the lowest latency possible along the entire image processing pipeline, since they must capture what’s happening in the real world, process it, render it and then display it inside of the headset, in almost real-time. If things are too out of sync, the user will feel like they are drunk or have motion sickness. There cannot be any weak links in the chain, not even in the display. Every millisecond matters in Mixed Reality, even more than in VR or AR. This is a problem that Qualcomm recognizes and is working to mitigate. According to Hugo Swart, the company’s GM of XR, Qualcomm has managed to bring photon-to-photon latency down to under 10 milliseconds from the previously standard 30ms+.
Second, Mixed Reality needs to have a deep understanding of the environment and be able to utilize AI algorithms to recognize objects and map the user’s surroundings. This is what allows the headset to overlay digital information over the real-world environment for AR functions. Additionally, even if the user is using the headset in VR mode, it needs to be able to help with boundaries and object avoidance. Snapdragon XR2 chipset, which is the standard for today’s XR headsets, has these capabilities built-in.
In addition to low-latency and environmental awareness, a Mixed Reality device needs to have enhanced interactivity. In other words, users should be able to interact in AR, VR and MR experiences using different user interfaces and control schemes—for example, hand tracking for base-level interactions and controllers for more intensive applications like gaming and creative work.
A Mixed Reality device must also consider how the user interacts with both virtual and physical objects while inside of the headset and what that might mean for the user’s immersion. For example, it will need to understand the differences between different types of objects, applying the proper level of opacity to them depending on their degree of relevance. When you look at the previous generation of XR chipsets, the Snapdragon XR2 increased AI performance by 11 times, which is crucial in enabling such features.
Last but certainly not least is the fact that Mixed Reality headsets require a fundamentally better field of view (FoV) than most AR headsets, which typically feature between 30 and 50 degrees. This is because bending light is difficult, and these devices’ light engines are limited by what can fit inside their necessarily lightweight and compact form factors. Today’s technology simply cannot produce anything beyond 50 degrees, for the most part. There are a few exceptions, but these solutions tend to be more appropriate for engineering environments versus consumer, due to their size and weight.
One workaround to this bottleneck is RGB pass-through technology, which delivers the FoV that users expect, on a much more accessible timeline. Snapdragon XR2 chipset was specifically designed to support up to 7 different simultaneous camera streams including ultra-low latency processing to enable a lag-free RGB passthrough experience. Qualcomm’s powerful Adreno is designed to enable the higher resolutions of Mixed Reality displays to mitigate the loss of visual fidelity that results from an increase in FoV. People want access to AR and MR today, not in three to five years when lightweight AR headsets finally support an acceptable field of view and levels of performance. For AR to be accessible, Mixed Reality headsets must flourish. This will allow developers to build AR experiences on today’s available hardware for tomorrow’s AR headsets.
Mixed Reality: a consumer or enterprise technology?
With how much Mixed Reality borrows from both VR and AR, it would not be surprising for it to inherit many of the same use cases. In fact, consumer and enterprise use cases for AR and VR translate almost perfectly to Mixed Reality. However, there are some use cases that are even better in Mixed Reality than VR or AR by themselves.
One of those is enterprise training. Imagine being able to take advantage of Mixed Reality’s wide FoV, overlaying graphics onto, but still very much seeing the “real world.” This makes Mixed Reality a powerful proposition for job training.
There are additional Mixed Reality applications already in use in the enterprise, such as remote assistance and guided work instructions. However, these stand to become much more powerful and capable than the simple 2D head-mounted solutions we have today. There are also consumer use cases aplenty, since a user can turn virtually any surface in their home into a gigantic display without losing their real-world spatial and situational awareness.
Many in the industry agree the convergence of AR and VR is inevitable. Mixed Reality is still in its early days, but much of the XR industry appears to be moving in that direction in the near term. Mixed Reality will not only benefit from the technological developments of AR and VR, but also facilitate them at the same time. There will be many users who will want a single, Mixed Reality device to cover the gamut of XR. Qualcomm’s technological innovations inside of the XR2 are at the core of making that possible today. However, not everyone will want this unified device and may prefer to only have AR or only have VR. Users want choice and hopefully the market will give them that with a diversity of XR devices and approaches. Much like the XR headsets and Android smartphones of today, many will likely be powered by Qualcomm technologies.