ICRA 2026 Beyond Teleoperation Workshop
Sources: Session notes, 2026-06-09 Raw: ICRA 2026 Beyond Teleop Papers Updated: 2026-08-12
Overview
The ICRA 2026 Beyond Teleoperation Workshop addressed a fundamental question: does robot learning need to move past teleoperation as its primary data collection paradigm? Teleoperation’s core limitations — control interface latency, constrained degrees of freedom, and unnatural motion artifacts — prevent demonstrations from capturing the natural dexterity and strategy diversity humans use in daily manipulation. 29 papers across 4 spotlight and 25 poster presentations cluster into eight research directions.
Central Debate
The workshop framed three contested questions:
- Does robot learning actually need to move beyond teleoperation?
- What is the “hidden cost” of using human/simulation data vs. small amounts of clean robot data?
- Is a small amount of high-quality teleoperation data a necessary “anchor” for large-scale human video/simulation training?
Spotlight Papers
MolmoB0T: Large-Scale Simulation → Zero-Shot Transfer (arXiv:2603.16861)
Allen AI / UW. Challenges the conventional belief that simulation training does not transfer effectively to real robots.
- MolmoBot-Engine: Open-source procedural data generation pipeline across diverse robots, tasks, and environments
- MolmoBot-Data: 1.8M expert trajectories (articulated objects + pick-and-place)
- Real-environment desktop pick-and-place success rate: 79.2% (vs. π₀.₅ at 39.2%)
- Zero-shot deployment — no real robot data, no fine-tuning
UniDex-ViTac: Unified Visuo-Tactile Dexterous Policy (arXiv:2603.22264, 2603.17851)
Multi-institution (CVPR 2026) + HUST. Fuses visual, tactile, and kinematic information for dexterous manipulation.
- FAAS (Function-Actuator-Aligned Space): Unified action representation aligned across 8 dexterous hand types (6–24 DoF)
- UniDex-VLA: 81% average success on tool-use tasks, zero-shot cross-hand generalization
- DexViTac collection system: 248 demonstrations/hour, 2,400+ multimodal demonstrations, >85% success on 4 tasks
DemoDiffusion: One-Shot Human Imitation (arXiv:2506.20668)
CMU (Park, Bharadhwaj, Tulsiani). Single human demonstration video → robot execution without training.
- Pipeline: hand motion kinematic retargeting → pre-trained diffusion policy trajectory correction
- 83.8% average success across 8 tasks
- vs. raw retargeting: 52.5%; vs. base diffusion policy: 13.8%
- No additional training, no reward function required
Humanoid Bimanual Dexterous Manipulation from Egocentric Video
Learns bimanual dexterous control from first-person human video. Related work: EgoDex (arXiv:2505.11709), 829 hours of egocentric video with 3D hand tracking.
Research Direction Clusters
① Large-Scale Simulation → Sim-to-Real MolmoB0T, Dex4D (arXiv:2602.15828), Point Bridge (arXiv:2601.16212) — pure synthetic data, zero-shot transfer. Dex4D uses 3D point trajectories as unified visual conditioning for task-agnostic grasping.
② Human Video → Robot Policy DemoDiffusion, X-Diffusion (arXiv:2511.04671, Cornell), HOMimic, UniDex-ViTac — distill skills from human demonstrations. X-Diffusion insight: as diffusion noise increases, low-level action differences fade while high-level task guidance persists — integrate human actions only at the “minimum indistinguishable step.”
③ Alternative Collection Interfaces
- YUBI (AIRoA): finger-aligned gripper mapping, Quest 6-DoF tracking, wide-angle camera; two modes (tabletop fixed + handheld portable)
- DexViTac: 248 demos/hour portable multimodal collection system
- PHABS: handheld haptic device with simultaneous force annotation for bimanual demonstration
- MobileEgo Anywhere (arXiv:2605.05945): smartphone sensors for 200 hours of diverse long-horizon egocentric data, open-source app + VLA-format processing pipeline
④ VR / Motion Transfer MotionTrans (arXiv:2509.17759, Tsinghua/CMU): human VR demonstrations → motion knowledge co-training. 30 tasks simultaneously → 13 successful transfers, 9 non-trivial zero-shot successes, pretraining+finetuning +40% success rate.
⑤ Video Generation / World Model Augmentation
- CRAFT (arXiv:2604.03552): Canny-edge conditioned video diffusion generates temporally coherent photorealistic manipulation videos from simulation trajectories, preserving action labels
- Dream2Flow (arXiv:2512.24766, Stanford): 3D object flow as intermediate representation bridging video generation → robot control; supports rigid, articulated, deformable, granular objects
⑥ Cross-Embodiment Transfer UniLatent (arXiv:2601.15419): c-VAE latent space aligning semantically similar actions across embodiments. X-Diffusion: noise-based human-robot action integration.
⑦ Controller Gain Design Tune to Learn (arXiv:2604.02523, MIT): “effective stiffness” comes from the interaction of learned response and control dynamics — choose gains for learnability, not just task compliance. BC benefits from compliant/overdamped gains; RL succeeds under any gains; sim-to-real is harmed by stiff/overdamped gains.
⑧ Perception and Representation
- IFG (arXiv:2511.09558, CMU): internet-scale visual priors (part segmentation) + simulation force-closure pipeline → real-time diffusion model for functional grasping from scene point cloud + text prompt
- Reconstructing Hand-Held Objects (arXiv:2404.06507, Berkeley/UT Austin): MCC-HO + GPT-4V retrieval for 3D reconstruction of occluded manipulation objects from monocular RGB video
Cross-Workshop Theme
All 29 papers share the goal of reducing dependence on costly robot teleoperation while maintaining transfer quality. The strategies span: synthetic scale (simulation), semantic shortcut (human video), interface improvement (new collection hardware), and representation alignment (cross-embodiment latent spaces). No single strategy dominates — the community is actively exploring all four simultaneously.
Open Questions
- What is the minimum amount of real robot teleoperation data needed as an “anchor” for large-scale human/simulation pretraining?
- MolmoB0T’s 79.2% success on pick-and-place: does this generalize to contact-rich or dexterous tasks where physics simulation accuracy matters more?
- DemoDiffusion requires a pre-trained diffusion policy as a prior. What is the right procedure when no suitable pre-trained policy exists for a novel task class?
- YUBI and DexViTac both target high-throughput dexterous data collection. How do they compare in data quality metrics (DC-IL action divergence / transition diversity)?