XR-to-Web-Streaming

XR to Web Streaming captures real-time head tracking data from an XR headset, renders it in Unity, and streams the camera output to a browser-based web client using WebRTC — extending the core pipeline from the XR-Head-Stream project with browser-based delivery.

Demo

XR-Head-Stream-Web.MOV

Architecture & Data Flow

The system architecture follows a continuous pipeline from the standalone XR headset to the web browser:

Headset > Predictor > Unity > Web Server (WebRTC)

1. Headset (Android NDK C++ App): A native Android C++ application running on the headset (e.g., Meta Quest) captures high-frequency head pose data and sends it over the network to the Predictor.

2. Predictor: The Predictor acts as an intermediary, buffering the pose data. It waits for Unity to request the current pose and responds with the latest data.

3. Unity Server:

• Unity receives the head tracking data and dynamically updates the Camera GameObject's transform (position and rotation) in the virtual scene.
• Unity renders the updated scene and streams the camera output directly to a local Node.js web server via WebRTC.

4. Web Server (Node.js + HTML/JS): Receives the camera output stream from Unity via WebRTC. An HTML/JS web client connects to this server to view the real-time stream.

Technologies Used

• Android NDK (C++): Native app for extracting hardware-level head poses.
• Unity3D & C#: Scene rendering and camera pose updating.
• WebRTC: Real-time video streaming from the Unity engine.
• Node.js, Express & Socket.io: Local signaling and web server handling WebRTC connections.
• HTML / JavaScript: Web client for viewing the XR stream.