ARCap: Collecting High-quality Human Demonstrations for Robot Learning with Augmented Reality Feedback
Abstract
ARCap is a portable augmented reality data collection system that retargets and visualizes a robot’s motion in real-time, providing the demonstrator with instant visual and haptic feedback to guide them in collecting robot-executable demonstrations. The key insight is that novice users consistently fail to produce demonstrations that satisfy robot kinematics without feedback — AR solves this by overlaying a virtual robot on the real world.
Hardware Setup
All off-the-shelf components:
- Meta Quest 3 VR/AR headset (MR passthrough for real-world overlay)
- Rokoko data gloves with Quest 3 controllers attached (hand + wrist tracking)
- Intel RealSense D435 depth camera (mounted on headset via 3D-printed bracket)
- Windows laptop (i5-13200H) for real-time IK solving and data storage
- Fully portable — fits in a backpack
Three-Layer Feedback System
1. Visual feedback: AR display shows virtual robot overlaid on real environment. Rectangular frame visualizes robot camera field of view. Frame turns yellow when user exceeds robot speed limits; blue when collision detected.
2. Haptic feedback: Controllers vibrate when virtual robot collides with obstacles. Immediate tactile warning prevents kinematically infeasible trajectories.
3. Kinematic visualization: Virtual robot arm is retargeted to user’s hand motions in real-time, making embodiment gap visible to the user before recording.
Retargeting Implementation
Dexterous hand: Two-stage IK:
- Match wrist orientation from Quest 3 controller pose
- Match fingertip positions from Rokoko glove data → robot fingertip positions
Parallel-jaw gripper: User mimics gripper with index finger and thumb; midpoint = gripper position, finger separation = gripper opening width.
Data Format
Records per frame:
- Colored point clouds (RealSense + Quest 3 depth fusion)
- Solved joint angles from IK
- Headset pose (6-DoF)
- Virtual robot pose
Post-processing: world-frame transformation, background removal, point cloud augmentation (virtual robot mesh merged with scene).
Policy Training
Uses diffusion policy with PointNet encoder for 3D point cloud observations. Built on Robomimic IL framework.
Key Results
User study with 20 participants:
- +40% replay success rate vs DexCap baseline (no feedback)
- +60% scene visibility (AR frame prevents occlusion failures)
- Cluttered scene task: 70% success (ARCap) vs 25% (DexCap)
- 3-stage Lego assembly: 40% success (ARCap) vs 0% (DexCap)
- Same system works for both dexterous hands and parallel-jaw grippers
Key Insight
Without real-time kinematic feedback, even motivated users produce demonstrations that exceed joint limits or cause collisions — problems that only appear when the robot tries to replay the data. AR feedback catches these at collection time, resulting in a much higher fraction of usable demonstrations.
Open Source
Fully open-source; all components off-the-shelf. Stanford AI Lab, 2024.