Meta Quest Teleoperation Stack

Sources: Session notes, 2026-04-06; 2026-04-09; 2026-04-17; 2026-04-18; 2026-06-15 Raw: PAPRLE ROS2 VR Integration; Unity ROS Teleoperation ETH; Unity ROS Windows Quest Link; NeuracoreAI MuJoCo Pipeline; ROS2 Bridge Architecture; MuJoCo Setup Pitfalls; Manus SDK WSL2 Docker Setup Updated: 2026-08-12

Overview

Three distinct software stacks enable Meta Quest VR controllers as teleoperation input for robot manipulation: PAPRLE (UIUC, modular middleware), NeuracoreAI’s meta_quest_teleop (pure-Python IK pipeline), and ETH Zurich’s unity_ros_teleoperation (VR visualization platform). The underlying hardware bridge for all three is oculus_reader (RAIL Berkeley) — a Quest APK + ADB approach that is simpler and more Ubuntu-native than OpenXR/ROS bridging.

Hardware Connectivity: oculus_reader (ADB)

Quest 2/3 controller data reaches the host computer via oculus_reader: a custom APK installed on the Quest that transmits 6-DoF controller pose and button state over ADB (wired or wireless WiFi). No OpenXR runtime or ROS bridge is required. Ubuntu is the primary platform (sudo apt install android-tools-adb).

oculus_reader outputs per-controller data. PAPRLE processes it via an oculus2paprle coordinate transform matrix, uses SVD to ensure rotation matrix validity, and outputs delta_eef_pose for downstream IK.

PAPRLE: Modular Middleware (UIUC Kim Lab)

Architecture: Leader device → PAPRLE task-space mapping → ROS1/ROS2 → robot

PAPRLE supports ROS1 and ROS2, and MuJoCo and IsaacGym simulators. Officially supported leader devices: Keyboard, Puppeteer, Nintendo Joycon, PS5 DualSense, Apple Vision Pro. Quest 2/3 is not natively supported; integration requires adding an oculus_reader-based leader.

Adding a new leader: Inherit from base.py, implement get_status() returning joint_pos or delta_eef_pose. The IK and safety filtering layers do not require modification. Estimated cost: ~200–300 lines.

Extension paths:

• Manus gloves: Not natively supported. Add a leader that reads Manus SDK (Python binding/UDP) for finger joints; pair with a separate 6-DoF wrist tracker (SteamVR tracker).
• MoCap (OptiTrack/Vicon): Modify puppeteer.py to subscribe to a MoCap ROS driver topic, or convert lafan_dataset.py (currently offline CSV replay) to real-time streaming.

NeuracoreAI Pipeline: Pure Python IK

Key architectural insight: The actual teleoperation loop does not use ROS2 topics. Following example_agilex, the pipeline is MetaQuestReader → PinkIKSolver → robot controller, fully in Python. ros2_tf_publisher.py is a visualization-only tool that broadcasts controller pose to the ROS2 TF tree for RViz display; it does not control the robot.

MuJoCo integration (Franka/Panda): PinkIKSolver is URDF-agnostic. Swap in Franka FR3’s URDF and set ee_frame=fr3_hand (or use panda_description.URDF_PATH + panda_hand since Panda and FR3 kinematics are nearly identical). A MujocoSimController of ~30 lines sets data.qpos from IK output and calls mujoco.viewer.launch_passive.

MetaQuestReader API note: get_buttons() does not exist; use last_buttons (property, not method), with or {} guard against None.

ROS2 Bridge Architecture

For projects needing ROS2 integration, a quest_bridge_node encapsulates MetaQuestReader and publishes standard message types:

MetaQuestReader (background thread)
        │
        ▼
quest_bridge_node (50 Hz timer)
        ├── /tf  →  right_controller, left_controller (in world frame)
        └── /quest/joy  →  sensor_msgs/Joy
              axes:    [rightGrip, leftGrip, rightThumbX, rightThumbY, leftThumbX, leftThumbY]
              buttons: [A, B, X, Y, rightThumb, leftThumb]

Controller poses use tf2_ros.TransformBroadcaster rather than PoseStamped: timestamp alignment is handled by tf2, RViz visualization is automatic, and MoveIt integration is direct. Button state uses sensor_msgs/Joy (standard, no custom .msg required).

ETH Zurich Unity Stack: VR Visualization Platform

leggedrobotics/unity_ros_teleoperation (IROS 2024: “Radiance Fields for Robotic Teleoperation”) is a VR visualization platform for legged robot navigation, not a policy learning data collection tool.

Architecture: Unity XR ↔ ROS via TCP socket (ROS-TCP-Connector fork). ROS 2 support on main-ros2 branch.

Differentiating components:

• NeRFViewer: Subscribes to /nerf_render topic, drives NeRF rendering in Continuous/Single/VR modes — integrates Nerfstudio/nerf-teleoperation, rare in commercial XR systems.
• MANO hand tracking: OpenXR 21-joint mapping to MANO hand model, published as ManoLandmarksMsg on /quest/hand_pose. Direct integration path for dexterous hand retargeting (e.g., Psyonic Ability Hand).
• Haptics: Shared /quest/haptics (HapticReadings) topic, normalized 0–1 intensity, supporting both BHaptics TactGlove and Oculus controllers.

Maturity: Version 0.1.1, primarily for demo/lab use. TFManager.cs and ROSManager.cs implementations are thin; many features require manual Unity Editor inspector configuration.

Installation Pitfalls (Docker / ROS2 Humble)

Common package name traps in the robotics Python ecosystem:

Wrong	Correct	Notes
pip install pink	pip install pin-pink	pink installs a code formatter
pip install pinocchio	pip install pin	Both import as pinocchio

colcon build README.md error: colcon runs setup.py from build/<pkg>/; relative open("README.md") fails. Fix: sed to use open(os.path.join(os.path.dirname(__file__), "README.md")).

example_agilex is not a ROS2 package (no package.xml): colcon ignores it. Copy pink_ik_solver.py and vectorised_posture_task.py directly alongside the teleop script.

numpy/scipy ABI conflict: apt scipy binds to an older numpy ABI. Fix: pip3 install --upgrade numpy scipy before installing other packages.

VNC vs X11 (Humble containers): tiryoh/ros2-desktop-vnc:humble provides a virtual display :1 — no xhost or X11 socket mounting needed. Add security_opt: seccomp:unconfined and shm_size: 512m.

Minimal working Dockerfile:

FROM tiryoh/ros2-desktop-vnc:humble
RUN apt-get update && apt-get install -y android-tools-adb \
 && rm -rf /var/lib/apt/lists/*
RUN pip3 install --upgrade numpy scipy && \
    pip3 install mujoco pin-pink pin quadprog robot_descriptions

Manus Glove SDK: WSL2 + Docker Setup

Manus Core — the Windows-side glove management service — runs Windows only. This affects Linux/WSL2 integration architecture.

Architecture selection (first decision):

• Integrated mode: Linux application reads dongle directly via HIDAPI/libusb. Requires passing the USB dongle through to WSL2 (usbipd) and having systemd/udevd running.
• Remote mode: dongle → Manus Core (Windows) → SDK client (WSL2/Docker) via network. Bypasses the entire USB passthrough problem. Recommended for WSL2 setups — avoids usbipd, udev, and privileged container requirements.

WSL2-specific blockers for integrated mode (not documented in the SDK’s native Linux guide):

1. usbipd: WSL2 is a Hyper-V VM — USB is not natively visible. Run usbipd bind/attach --wsl to forward the dongle (VID 3325 or 1915:83fd). Verify with lsusb before continuing.
2. systemd/udevd: WSL2 does not start systemd by default. Without it, udev rules don’t trigger and HIDAPI (libudev) cannot enumerate devices. Fix: add [boot] systemd=true to /etc/wsl.conf, then wsl --shutdown to restart.

udev rule (99-manus.rules):

SUBSYSTEMS=="usb", ATTRS{idVendor}=="3325", MODE:="0666"
SUBSYSTEMS=="usb", ATTRS{idVendor}=="1915", ATTRS{idProduct}=="83fd", MODE:="0666"
KERNEL=="hidraw*", ATTRS{idVendor}=="3325", MODE:="0666"

Docker Compose for integrated mode — three mounts are all required:

services:
  manus-integrated:
    image: manus-linux-integrated
    privileged: true
    volumes:
      - /dev:/dev           # hidraw + usb device nodes
      - /run/udev:/run/udev # libudev device enumeration
    ports:
      - "5000:5000"
    stdin_open: true
    tty: true
    command: /bin/bash

Use docker compose run --rm --service-ports manus-integrated (not up) — up exits immediately when bash has no stdin.

Startup sequence (integrated): usbipd attach → verify lsusb//dev/hidraw* → then docker compose run.

Cleanest path for WSL2: remote mode — dongle plugged into Windows, Core running on Windows, WSL container connects via --net=host. Zero USB passthrough friction.

Windows Development Environment (Quest Link)

For Windows-based development with the ETH Unity stack:

Quest Link resolves the headset problem but ROS remains the bottleneck. Quest Link makes the Quest act as a Windows PCVR device — Unity Editor Play Mode can be tested directly without building an APK. The real challenge is the ROS TCP Endpoint, which requires a Linux environment.

WSL2 approach — three recurring pain points:

1. IP not fixed: run wsl hostname -I after each restart to get the current IP; enter it manually in the App
2. Windows Firewall blocks port 10000 by default — requires a manual inbound rule
3. WSL2 virtual NIC occasionally becomes unstable and needs resetting

Cleanest approach: Windows for Unity Editor + Quest Link; a separate Linux machine on the same LAN for ROS. Fill in the local network IP — no WSL2 friction.

ROS on Windows: ROS 1 is experimental; ROS 2 has partial support but many drivers are incomplete and community resources are scarce. Not recommended.

Quest Link vs. standalone APK behavior differences:

• Passthrough may not be available in PC VR mode
• Hand tracking latency is higher than standalone mode
• Some Meta-specific OpenXR extensions (e.g., XR_FB_hand_tracking_aim) behave inconsistently in Link mode

Recommended development workflow: Unity Editor + Quest Link (fast iteration) → WSL2 or same-LAN Linux (ROS) → final APK sideload (production deployment).

Open Questions

• PAPRLE’s lafan_dataset.py knows how to consume MoCap-format poses (offline CSV). Converting it to a real-time MoCap streaming source is the shortest path for OptiTrack/Vicon integration — what latency is introduced?
• The NeuracoreAI pipeline bypasses ROS2 entirely. When should the ROS2 bridge architecture be preferred over pure-Python (logging, multi-node setups, real robot integration)?
• ETH’s NeRFViewer component is unique among open-source teleoperation stacks. Is there a path to combining NeRF-based scene representation with policy learning data collection?

See Also

• Teleoperation Systems
• Simulation Data Collection
• Visualization Tools