Teleoperation Systems for Dexterous Manipulation
Sources: Research synthesis, 2026-03-26; GI Labs, 2026-03-25 Raw: Robotic Teleoperation for Dexterous Manipulation; Large-Scale Teleoperation Datasets: DROID Comparison; [Tracking Dexterous Hands: Practitioner’s Guide to MoCap](../../raw/clippings/Tracking Dexterous Hands A Practitioner’s Guide to Motion Capture for Robot Learning.md) Updated: 2026-08-12
Overview
Teleoperation is the dominant paradigm for collecting high-quality robot manipulation demonstration data. Four open-source systems — DexCap, AnyTeleop, OPEN TEACH, and Open-TeleVision — each attack the cost and quality problem from a different angle. The field is converging toward cheaper VR-based interfaces (Meta Quest 3, ~$500) that can still support bimanual and dexterous manipulation at useful data rates.
Why Teleoperation Wins
Three data collection paradigms compete for imitation learning: simulation, kinesthetic teaching, and teleoperation.
Simulation provides unlimited data at near-zero cost but sim-to-real transfer for contact-rich tasks remains unsolved. Small physics discrepancies cause policies to fail on real hardware in unpredictable ways.
Kinesthetic teaching (physically guiding the robot arm) produces accurate demonstrations but is slow (one demo at a time), physically demanding, and breaks down for multi-finger dexterous hands — you cannot kinesthetically teach a 20-DOF hand by moving each joint.
Teleoperation hits the sweet spot: human-quality demonstrations at moderate cost and higher throughput. The challenge is designing interfaces intuitive enough to produce natural demonstrations without extensive operator training.
The Four Major Systems
DexCap
Hardware-first: wearable mocap gloves capture finger joint angles directly, producing precise dexterous hand data. High fidelity, but specialized hardware and the wearables constrain natural movement. Best suited for tasks requiring full finger joint trajectory data for behavior cloning.
AnyTeleop
Vision-only hand pose estimation from a standard camera — no wearables required. Its key contribution is a unified framework that works across different robot arms and hands without per-platform engineering. Counterintuitively outperforms some hardware-specific systems; wearable artifacts may degrade data quality. Democratization argument: any lab with a camera can use it.
OPEN TEACH
Targets accessibility most aggressively: $500 Meta Quest 3 consumer VR headset, 90Hz control rate, bimanual support, 38 validated tasks. The price point is the main story — brings dexterous teleoperation within reach of groups that cannot afford professional motion capture infrastructure.
Open-TeleVision
Addresses operator perception rather than cost. Standard teleoperation gives operators a fixed camera feed; Open-TeleVision streams stereoscopic video from cameras on the robot’s head to the operator’s VR headset and lets the operator control where the robot looks. This active visual feedback produces more naturalistic demonstrations because operators direct their gaze the same way they would when physically performing the task — analogous to how humans rely on gaze direction for fine motor tasks. The downstream effect: imitation learning policies trained on this data generalize better because the data distribution more closely matches what the robot sees during autonomous deployment.
Operator Experience as Data Quality Driver
An underappreciated insight from Open-TeleVision: the quality of collected demonstrations depends heavily on the operator’s sensory experience. Static-camera teleoperation inadvertently degrades data quality by forcing operators to work with suboptimal viewpoints. This effect is analogous to asking someone to perform fine motor surgery while looking at a fixed overhead camera rather than directly at their hands.
Interface Hardware Trends (2024–2025)
The broader dataset landscape shows that VR hand-held controllers (Oculus Quest 2, Quest 3) have become the dominant teleoperation interface, displacing SpaceMouse and external skeleton systems. The reasons: lower operator learning curve, 6-DOF continuous control, and support for non-robot-expert collectors. DROID’s use of Quest 2 across 13 institutions is illustrative — it enabled 50 collectors to operate 18 robots across 564 distinct scenes without per-person specialized training.
Meta Quest 3 sensing architecture (GI Labs practitioner analysis, 2026): Quest 3 layers three sensing modalities at different accuracy tiers, which explains why it is the default robotics teleoperation platform:
- VI-SLAM (headset localization): ~0.77 cm RPE — suitable for global pose estimation
- IR LED constellation tracking (controllers): low-mm precision relative to the headset — the same principle as PhaseSpace lab mocap but inside-out. This is the high-fidelity channel. Systems like ARCap tap the controller’s low-mm relative precision for fine-grained manipulation input while using VI-SLAM for global pose.
- Markerless hand tracking (computer vision only): ~1.73 cm — lower precision, sufficient for coarse gestures but not dexterous finger tracking
Apple Vision Pro ships without controllers and relies entirely on markerless hand tracking (~1.73 cm tier). This places it at a fundamentally different accuracy tier from Quest 3 for teleoperation despite its superior visual passthrough. The absence of a high-precision controller channel is the structural limitation for fine-grained manipulation.
Sensor fusion architectures for hand pose capture fall into two patterns: (1) Hierarchical fusion — different sensors assigned to complementary roles (SLAM for global pose, EM gloves for fingers, composed through forward kinematics); (2) Redundant fusion — overlapping measurements combined via EKF or factor graph optimization, handling asynchronous sensor rates. The GI Labs survey of 12 recent systems (from DexCap’s SLAM + EM glove hierarchy to OSMO’s tactile augmentation) maps this design space. Tactile augmentation adds a separate finding: vision-only hand tracking policies achieve 30% success on contact-rich wiping tasks vs. 72% with tactile-equipped sensors.
Bimanual and Mobile Gaps
Most published teleoperation research evaluates single-arm tabletop tasks. Real manipulation (kitchen, warehouse, caregiving) is bimanual. OPEN TEACH’s bimanual support and dual-arm VLA systems like GR-Dexter signal the field is moving this direction, but collecting bimanual demonstrations is significantly harder — operator cognitive load doubles and retargeting two arms simultaneously is a harder mapping problem.
Open Questions
- What is the minimum demonstration count needed per task? Current systems require hundreds to thousands of demos. This is not scalable to the diversity of real-world tasks.
- What makes a “good” demonstration? The field lacks principled criteria. Human operators vary enormously in quality, and filtering mechanisms remain ad-hoc.
- Can video learning replace active teleoperation? If policies can learn from YouTube-scale human video, the data bottleneck largely disappears. Current progress is limited.