Robotics is increasingly a data problem. The scaling laws powering robotics foundation models need enormous quantities of high-fidelity action data, and that data has to come from human demonstrations in real environments.
At @gi_labs , we’ve been working on this problem for a while. We wrote a practitioner’s guide to the motion capture systems behind dexterous robotic manipulation. If you’re building teleoperation rigs, collecting demonstration datasets, or designing learning pipelines for dexterous hands, the sensing stack is an important piece for your success.
The fundamental problem: capturing dexterous motion in the wild (kitchens, workshops, the real world) at sufficient fidelity requires sensors that each sacrifice something different, and no combination cleanly covers portability, global pose, and finger articulation under occlusion at once. Lab grade optical motion capture gives sub-mm accuracy but can’t leave its camera cage. Portable sensors can go anywhere but each breaks in its own way: IMUs accumulate yaw drift, SLAM degrades under the heavy occlusion that dexterous manipulation creates, and electromagnetic (EM) gloves like manus distort when you’re holding metal objects. We listed the trade-offs of different sensing systems below
trade-offs of different sensing systems
This is why most practical systems fuse multiple modalities. The sensor choice determines what physical quantities you can observe; the fusion architecture determines how accurately and robustly those observations combine. Hierarchical fusion assigns sensors to complementary roles: SLAM for global pose, EM gloves for fingers, composed through forward kinematics. Redundant fusion combines overlapping measurements, either through Extended Kalman Filter (EKF) filtering that maintains a running state estimate, or factor graph optimization that jointly solves over the full trajectory while handling asynchronous sensor rates. Our survey covers 12 recent systems and how they navigate these choices, from DexCap’s SLAM + EM glove hierarchy to OSMO’s tactile augmentation.
Trying @ManusMeta Glove + Quest VR for teleop.
Two findings stood out:
- Meta Quest 3 is a case study in how a single consumer device layers multiple sensing modalities at different accuracy tiers: Visual-Inertial SLAM (VI-SLAM) for headset localization (~0.77 cm RPE), active IR LED constellation tracking for controllers (the same principle as PhaseSpace lab mocap but inside-out, achieving low-mm precision relative to the headset), and markerless hand tracking via computer vision (~1.73 cm). This layered architecture makes Quest a natural fit for robotics teleoperation. Systems like ARCap tap the controller’s low-mm relative precision for fine-grained manipulation input while using the headset’s SLAM for global pose. Apple Vision Pro, by contrast, ships without controllers and relies entirely on markerless hand tracking, operating at a fundamentally different accuracy tier.
- Tactile sensing is not optional for contact-rich tasks. OSMO augments vision-based hand tracking with 12 three-axis magnetic tactile sensors on fingertips and palm. On a contact-rich wiping task, tactile-equipped policies achieved 72% success vs. 30% vision-only. Contact forces are not supplementary information that makes policies slightly better. For tasks involving sustained physical contact, they are a primary signal. Omitting them from your capture system means your policy never sees them.
We thank @YuXiang_IRVL from UT Dallas for insightful discussions on IMU sensing and sensor fusion, and Jessica Yin from NVIDIA for discussions on electromagnetic tracking and tactile sensing.
Blog: https://www.gilabs.xyz/blog/dexterous-mocap
Full Report: https://www.gilabs.xyz/papers/dexterous-mocap-full-report.pdf