Passthrough Cameras – Immersive Learning News

Meta Releases Quest Camera Access for Developers, Promising Even More Immersive MR Games

Meta has finally released the long-awaited Passthrough Camera API for Quest, which gives developers direct access to the headset’s passthrough RGB cameras for better scene understanding—kickstarting the next generation of more immersive mixed reality experiences on Quest.

Until now, Quest’s passthrough cameras were mostly locked down, limiting what developers could do beyond Meta’s built-in functionalities. The company mentioned back at Connect in September it would eventually release Quest’s Passthrough Camera API, although it wasn’t certain when.

Now, in the Meta XR Core SDK v74, Meta has release Passthrough Camera API as a Public Experimental API, providing access to Quest 3 and Quest 3S’ forward-facing RGB cameras.

Passthrough camera access will essentially help developers improve lighting and effects in their mixed reality apps, but also apply machine learning and computer vision to the camera feed for things like detailed object recognition, making mixed reality content less of a guessing game of what’s in a user’s environment.

When it was announced last year, former Meta VP of VR/AR Mark Rabkin said the release of Quest’s Passthrough API would enable “all kinds of cutting-edge MR experiences,” including things like tracked objects, AI applications, “fancy” overlays, scene understanding.

This marks the first time the API has been publicly available, although Meta has previously released early builds with select partners, including Niantic Labs, Creature, and Resolution Games—which are presenting today at GDC 2025 in a Meta talk entitled ‘Merge Realities, Multiply Wonder: Expert Guidance on Mixed Reality Development’.

Granted, as an experimental feature, developers can’t publish apps built using the Passthrough Camera API just yet, although it’s likely Meta is again taking an iterative approach to the feature’s full release.

The v74 release also includes Microgestures for intuitive thumb-based gestures (e.g., thumb taps, swipes), an Immersive Debugger so developers can view and inspect Scene Hierarchy directly within the headset, and new building blocks, such as friends matchmaking and local matchmaking.

Quelle:

Meta Releases Quest Camera Access for Developers, Promising Even More Immersive MR Games

26. März 2025 0 comment

Examples

A Developer Figured Out How To Access Quest 3’s Passthrough Cameras

A developer figured out a clever way to access Quest 3’s passthrough cameras to run custom computer vision models.

Meta does not allow third-party developers to access the cameras of Quest headsets, but Michael Gschwandtner, the co-founder and lead developer of the hand tracking fitness app XRWorkout, posted a video to social media platforms showing the results of an interesting workaround to this restriction.

The demo video shows the open-source MobileNet V2 object detection model running natively on Quest 3 at 0.75 Hz. This was chosen for practicality, rather than for performance reasons.

Gschwandtner sent UploadVR a test build and we independently verified that this is real and works. He also exclusively revealed to UploadVR how exactly he pulled it off.

How It Works

Gschwandtner’s app build creates a webview, a Chromium web browser instance embedded within the app, which automatically loads the oculus.com/casting URL which Quest headsets can cast their view to.

You then need to go through the full process of logging into your Meta account in this webview. Once you’re done, you use the Quest system menu to initiate web casting. Uniquely, you’re casting from the headset to an app inside the same headset, not to a different device.

Gschwandtner’s app then samples the pixel content, the RenderTexture, of the webview receiving the casted view, and from there can do with it whatever he likes with it. In the demo example, he’s feeding it to Google’s MediaPipe on-device SDK.

A key limitation here is that the view will include virtual objects and interfaces too, not just the passthrough camera view, meaning it will only work when most of the image is passthrough. But Gschwandtner claims he has a „plan“ for a workaround for this.

What Happens Next?

It’s unclear how Meta will respond to Gschwandtner’s discovery. It’s possible the company could make this technique inoperable in a future version of Quest’s Horizon OS. Gschwandtner’s social media posts include a public request to „hop on a call“ with Meta CEO Mark Zuckerberg or CTO Andrew Bosworth to „show them“ the technique.

Last month Bosworth gave his thoughts on the idea of intentionally giving Quest developers access to the passthrough cameras during an interview, saying Meta will „keep looking at“ the issue.

In comparison, for its Vision Pro headset Apple only gives raw access to enterprise companies, and only for non-public internal apps.

Quelle:

https://www.uploadvr.com/quest-3-raw-camera-access-workaround-found/

20. August 2024 0 comment