Passthrough camera access has become quite a buzzword in the XR community, with everyone keenly observing what Meta, Apple, and Pico have to say about it. However, the industry’s eyes are now set on Google and its plans for Android XR. After directly engaging with the company, I can reveal that Google is poised to present an approach reminiscent of what we see on smartphones. Stick around to learn more!
The Camera Access Dilemma
Let’s take a step back for those who might not be fully caught up. Today’s standalone VR headsets are generally MR headsets, showing users an RGB passthrough view of their surroundings through front-facing cameras. This technology powers wonderful mixed reality applications like Cubism, Starship Home, Pencil, and more.
VR headsets rely on these camera feeds to offer passthrough vision. As developers, having direct access to these camera frames is a tantalizing prospect. We could employ AI and computer vision algorithms to transform and enhance user reality like never before. In previous discussions, I’ve expressed my strong belief that to truly unlock the potential of mixed reality, camera access is essential. It’s because applications can only be genuinely context-aware if they understand the environment the user is in. I remember using a clever workaround on Quest to prototype an MR application for interior design, an achievement that hinges entirely on having camera access.
That sounds promising, yet there’s a catch: privacy concerns. Granting camera access opens the door for malicious developers to potentially capture and exploit user surroundings. Consider the risk of sensitive information inadvertently being recorded, such as IDs or bank cards lying around while using an app. Images of faces or bodies could also be snagged for nefarious purposes.
Balancing user privacy with the advancement of mixed reality capabilities presents a significant challenge.
The Behavior of XR Companies
Initially, unfettered camera access was available without any major issues. If you’ve been following me for some time, you might recall some of the intriguing camera texture experiments I conducted on the Vive Focus back in 2019 with my team at NTW. These included projects like diminished reality, Aruco marker tracking, and sound reactivity.
However, as mixed reality started gaining traction, companies became more cautious, restricting camera frame access due to privacy concerns. Soon enough, Meta, Pico, HTC, and Apple all followed suit in curtailing access for developers.
These restrictive measures became the norm until the XR developer community began demanding this capability. Many, including pioneers like Cix Liv, Michael Gschwandtner, and myself, pushed back, advocating for controlled yet effective camera access. Our argument was clear: if smartphones can easily handle camera permissions, why can’t XR devices follow the same path?
The advocacy paid off to some extent, with Meta set to launch a "Passthrough API." But what’s Google’s stance on this matter for Android XR?
Android XR to Treat the Headset Like a Phone
Android powers the majority of smartphones globally. Currently, developing an Android app means you can request camera stream access simply by obtaining user permission. Once granted, you can identify the camera you want to access and utilize its frames as needed.
Google aims to align Android XR with its existing app ecosystem, creating similar functionality within its new OS. Although whispers about this had been circulating, it wasn’t until I communicated with a Google spokesperson that it became clear. Here’s the gist of the response I received about Android XR’s camera access policy:
Essentially, developers can access existing camera frames, provided they have user permission, just as they would on an Android phone. A similar request format applies, whether one wants to access the main world-facing or selfie camera streams via standard Android Camera APIs like Camera2 and CameraX.
For developers, this compatibility means they can leverage familiar classes like CameraX for managing camera streams on Android XR headsets. This development opens up exciting possibilities for media capture, frame utilization, and machine learning analysis.
While both front and rear cameras will be available, the rear camera functions as a reconstructed avatar, essentially mirroring Apple’s approach with the Vision Pro. This decision ensures consistency in how the OS handles camera access, ideally providing a seamless transition for apps between phones and XR headsets.
However, there’s caution in the wind. Google isn’t currently allowing applications to access non-standard camera data. This suggests that raw camera streams won’t be accessible for the time being, a limitation that might change, particularly in enterprise applications.
Developers using Unity might wonder how this adjustment affects them. Since Android Camera2 and CameraX are native classes, Unity developers could potentially use the WebcamTexture class to handle camera frames. If obstacles arise, community-driven solutions like JNI-based wrappers are always a feasible workaround.
A Little Caveat About Android XR
Android XR is still at a beta stage, with no headset officially launched that operates on it. Therefore, all the details above may be subject to changes before the official release. However, I’m optimistic these fundamentals will remain consistent.
The Opening Up of Camera Access
With companies like Google and Meta opening the doors to camera access, it’s likely others will follow suit. As we approach 2025, we’re on the cusp of unlocking new horizons in mixed reality. It’s an exhilarating time, and I’m eager to witness the innovations the developer community will bring to life!
(Header image credit goes to a Samsung source.)
Note: This blog contains advertisements and affiliate links. Clicking on an affiliate link gives me a small commission at no additional cost to you. Full disclosure available here.
