EgoMimic: Scaling Imitation Learning via Egocentric Video

Abstract

EgoMimic addresses the fundamental data scarcity problem in robot imitation learning by treating human embodiment data as equally valid demonstration data. Instead of requiring robot demonstrations, the system collects egocentric human videos with 3D hand tracking, co-trains on human and robot data jointly, and achieves significantly better performance than robot-data-only approaches.

Core Insight

“Human and robot data are treated equally as embodied demonstration data.” The embodiment gap (human hand ≠ robot gripper) is bridged via cross-domain alignment rather than ignored. The key empirical finding: 1 hour of human hand data is more valuable than 1 hour of additional robot data, fundamentally challenging assumptions about what constitutes useful training data.

System Components

1. Ergonomic Data Collection Hardware

• Project Aria glasses (Meta): lightweight egocentric camera with 3D hand tracking
• Captures: RGB-D video + 3D hand joint positions
• No wearable motion capture suits; glasses are worn naturally during task execution
• Enables in-the-wild data collection at scale

2. Robot Platform Design

• Bimanual manipulator designed to minimize kinematic gap to human anatomy
• Low-cost design for practical deployment
• Arm length and joint placement chosen to match human arm proportions

3. Cross-Domain Alignment

• Spatial alignment: camera extrinsics calibrated between Aria glasses and robot camera
• Kinematic alignment: hand joint positions retargeted to robot end-effector poses
• Temporal alignment: action chunking for consistent policy execution

4. Co-Training Architecture

• Unified policy network processes both human hand demos and robot demos
• Action space: end-effector poses (SE3) for both modalities
• Training: interleaved batches from human and robot datasets

Data Collection Workflow

1. Researcher wears Aria glasses and performs tasks naturally (no robot needed)
2. 3D hand tracking recorded automatically
3. Post-process: retarget hand poses to robot end-effector waypoints
4. Mix with small number of robot demonstrations for fine-grained robot-specific behaviors
5. Co-train unified policy

Key Results

• Outperforms state-of-the-art IL methods on diverse long-horizon tasks
• Single-arm and bimanual manipulation tasks both improved
• Generalizes to entirely new scenes not seen during training
• Scaling law: human data more sample-efficient than robot data

Comparison to Alternative Approaches

Approach	Hardware needed	Collection speed	Embodiment gap
Robot teleoperation	Robot required	Slow	None
EgoMimic	Glasses only	Fast (natural motion)	Bridged via alignment
Human Video (OXE)	Camera	Fast	Large (ignored)
MimicGen	Robot + sim	GPU-fast	None (sim)

Limitations

• Kinematic gap not fully eliminated (especially for fine-grained contact tasks)
• Requires precise calibration between glasses and robot camera
• Currently validated on specific bimanual manipulator design

Publication

CMU, Oct 2024.