Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots

Abstract

Universal Manipulation Interface (UMI) presents a framework enabling direct skill transfer from human demonstrations to deployable robot policies. Using hand-held grippers with careful interface design, UMI enables portable, low-cost data collection for challenging bimanual and dynamic manipulation tasks. The system incorporates inference-time latency matching and relative-trajectory action representations, yielding hardware-agnostic policies deployable across multiple robot platforms. Policies trained on diverse human demonstrations achieve zero-shot generalization to novel environments and objects, with 70% success rates on out-of-distribution tests.

Introduction

Current approaches to robot learning from demonstration face significant limitations. Teleoperation demands expensive hardware and expert operators, while human videos present substantial embodiment gaps to robots. Recently, sensorized hand-held grippers emerged as a middle ground, yet struggle balancing action diversity with transferability—prior systems restricted to simple grasping or quasi-static pick-and-place despite extensive environmental diversity.

The authors identified critical technical barriers preventing effective action transfer:

• Insufficient visual context: Wrist-mounted cameras create heavy occlusions, limiting scene understanding
• Action imprecision: Monocular structure-from-motion struggles with scale ambiguity and motion blur
• Latency discrepancies: Training occurs without latency; inference involves sensor, computation, and execution delays
• Limited policy representation: Simple MLPs fail to capture multimodal action distributions from diverse demonstrators

System Design

Demonstration Interface (Hardware)

UMI employs a 3D-printed parallel-jaw gripper (780g, $73BoM)withaGoProcamera($298) as the sole sensor. Key design innovations include:

Fisheye Lens (155° FoV): Provides necessary visual context while preserving center resolution through natural distortion. Tests show 55% success with conventional 69° cameras versus 100% with fisheye for cup arrangement.

Side Mirrors: Physical mirrors create implicit stereo views, simulating multiple camera perspectives without additional weight. Digital reflection of mirror content improved performance from 85% to 100% success.

IMU-Aware SLAM: Integration with GoPro’s built-in IMU enables robust tracking during fast motion and recovery of absolute metric scale through visual-inertial optimization via ORB-SLAM3.

Continuous Gripper Control: Unlike binary open-close actions, continuous width tracking via fiducial markers enables precise tasks like tossing with varied object sizes.

Kinematic Filtering: Recovered absolute end-effector poses enable filtering demonstrations for embodiment-specific kinematic feasibility.

Policy Interface (Software)

Inference-Time Latency Matching:

The system measures individual latencies (camera ~110ms, proprioception ~20-30ms, execution ~50-150ms) and synchronizes observations to the highest-latency stream. Action commands execute with forward compensation for execution delay.

Relative End-Effector Trajectory:

Actions represent desired poses relative to current end-effector position rather than absolute or delta representations. This enables:

• Robustness against tracking errors during collection
• Calibration-free deployment (movable robot bases don’t affect performance)
• Generalization across robot embodiments

Relative Inter-Gripper Proprioception:

Bimanual tasks receive relative pose between grippers through map-then-localize schemes constructing shared scene coordinates.

Diffusion Policy: The framework employs diffusion-based policy learning to model multimodal action distributions inherent in human demonstration data.

Experiments

Capability Experiments

Cup Arrangement (single-arm, multimodal):

Place espresso cups on saucers with specific orientations. Tests both prehensile and non-prehensile actions. Achieved 20/20 (100%) success in trained environment, 18/20 (90%) on different robot platform (Franka FR2). Ablations revealed:

• Fisheye essential: 55% vs 100% success
• Relative trajectory superior: 80-90% vs 25% for absolute actions
• Digital mirror reflection critical: 85% vs 100%

Dynamic Tossing (rapid motion, precision):

Sort 6 YCB objects by tossing into corresponding bins beyond reach range. Achieved 87.5% success. Latency matching critical: 57.5% without matching vs 87.5% with proper compensation.

Bimanual Cloth Folding (coordination, deformable objects):

Execute seven-step sweater folding requiring synchronized two-arm coordination. Achieved 70% success. Inter-gripper proprioception essential: 30% without versus 70% with coordination information.

Dish Washing (long-horizon, fluid dynamics):

Execute seven sequentially-dependent steps including faucet control, plate washing, and cleaning verification. Achieved 70% success. Demonstrated robustness to sauce variations and object perturbations. CLIP-pretrained vision backbone necessary—ResNet-34 from scratch achieved 0%.

In-the-Wild Generalization

Collected 1400 demonstrations across 30 diverse locations (homes, offices, restaurants, outdoors) using 3 demonstrators over 12 person-hours, involving 15 cups with varying colors, shapes, and materials.

Evaluation on unseen environments showed 71.7% combined success rate (70% on training objects, 75% on novel objects). Crucially, comparable training on narrow-domain data achieved 0% success in novel environments, demonstrating that in-the-wild diversity—not just pretrained backbones—enables generalization.

Data Collection Efficiency

UMI gripper demonstrated superior throughput compared to alternatives:

• 3× faster than space-mouse teleoperation on cup arrangement
• 48% speed of human hand (gripper still superior to teleoperation)
• 64% speed of human hand on dynamic tossing (teleoperation produced zero successful demonstrations)

SLAM Accuracy:

Benchmark evaluation achieved 6.1mm position error and 3.5° rotation error for single grippers; 10.1mm and 0.8° for inter-gripper relative pose.

Key Contributions

1. Hardware design balancing portability, information richness, and cost-effectiveness through fisheye optics and mirror-based implicit stereo
2. Robust action extraction via IMU-aware SLAM enabling metric-scale tracking during dynamic motion
3. Hardware-agnostic policy interface using latency matching and relative trajectory representations enabling cross-embodiment transfer
4. Empirical validation across diverse tasks (dynamic, bimanual, precise, long-horizon) with strong generalization to novel environments
5. Open-source release enabling community reproduction and extension

Limitations and Future Work

• Kinematic filtering relies on prior knowledge of target robot; future work could develop embodiment-aware learning
• Visual SLAM requires textured environments; extension to texture-deficient settings possible via external cameras and fiducial markers
• Gripper collection remains less efficient than human hand demonstration; future work could explore lighter materials and improved ergonomics

Conclusion

UMI demonstrates that careful hardware and software interface design enables effective skill transfer from portable hand-held demonstrations to deployed robot policies. The system achieves capabilities previously limited to teleoperated or in-lab systems—dynamic tossing, bimanual coordination, long-horizon reasoning—while maintaining portability and in-the-wild deployability. The 70% out-of-distribution success rate represents a notable achievement in behavior cloning generalization.