LfD Data Collection: Four Paradigms
Sources: Research synthesis, 2026-05-07; Session notes 2026-04-19, 2026-04-20, 2026-05-07, 2026-04-05; π0.7 analysis, 2026-04-20; Hu et al. (Tsinghua/Shanghai Qi Zhi), arXiv:2410.18647, 2024; Qiu et al. (UCSD/CMU), arXiv:2503.13441, 2025-03; Li et al. (Stanford), arXiv:2503.07017, 2025-03; Stanford AI Lab, arXiv:2410.08464, 2024; Tang (NVIDIA/USC), 2024-07; Mandlekar et al. (NVIDIA), CoRL 2023; arXiv:2510.09229, 2025-09; Hsieh et al. (UC Berkeley), arXiv:2512.02011, 2025-12 Raw: Human Demo Methods IL Taxonomy; LfD Task Taxonomy; LfD Framework; Robot Manipulation Data Collection Methods; Simulation as LfD Taxonomy; [π0.7 Data Analysis](../../raw/clippings/π0.7 and Everything Robot Data Companies Are Getting Wrong.md); Data Scaling Laws arXiv:2410.18647; [PH2D/HAT](../../raw/clippings/Humanoid Policy Human Policy PH2D HAT.md); [Demonstration Modality Comparison](../../raw/clippings/How to Train Your Robots Demonstration Modality.md); ARCap; AutoMate; MimicGen; Glovity; DexScrew Updated: 2026-08-12
Overview
Human demonstration for robot learning divides into four paradigms — kinesthetic teaching, teleoperation, natural demonstration, and passive observation — each making a different tradeoff between data quality, collection speed, and scalability. The five technical support dimensions (pose tracking, action representation, latency compensation, force sensing, data quality filtering) cut across all paradigms and determine whether a demonstration translates into a good policy.
The Four Paradigms
Kinesthetic Teaching
The operator physically grasps and moves the robot arm through the desired trajectory while joint states are recorded passively. The robot runs in Cartesian impedance or gravity-compensation mode. Recorded trajectories are replayed as delta-pose commands.
Key properties:
- Yields the highest action consistency (lowest K-NN action variance) of all modalities
- ICRA 2025 benchmark: 95% on Open Drawer; worst on Push Sanitizer due to replay jerkiness
- Cannot scale to dexterous hands (21-DOF) — a hard physical limit, not an engineering gap
- No remote collection; requires physical access to the robot
Hybrid scheme (Li et al., ICRA 2025, arXiv:2503.07017): a small kinesthetic “seed” dataset combined with a larger VR teleoperation dataset achieves ~20% higher performance than either modality alone. Kinesthetic data provides action consistency (clean, low-noise trajectories); VR data provides state diversity (broader coverage). Combining them captures both properties. Practical recipe: collect a small seed kinesthetically, then scale with VR teleoperation.
Use case: Simple single-arm tasks where data quality matters more than speed and dexterous fingers are not required.
Teleoperation (including Real-Time MoCap)
The operator remotely controls the robot in real time via an interface device. The robot mirrors the operator’s movements. The operator receives real-time feedback and can correct errors — this distinguishes it from natural demonstration.
MoCap teleoperation is a subtype: wearable sensors track the human hand and retarget its pose to the robot DOF space in real time.
Representative hardware:
| Device | Cost | Feature |
|---|---|---|
| Leader arm (SO-100) | ~€225 | Lowest cost, single-arm, intuitive |
| Meta Quest 3 + OPEN TEACH | ~$500 | VR immersion, Quest 3 controller precision |
| DexCap (EM glove) | $3000–5000 | 3× faster than leader arm, 21-DOF |
| DOGlove | <$600 | 21-DOF, cable-driven, haptic feedback |
| Glovity | <$300 | Wearable wrench feedback + haptic glove, 6-DOF, open-source; page-flipping IL: 80% (16/20) with wrench vs. 45% (9/20) without (arXiv:2510.09229) |
Key properties:
- High state diversity; susceptible to covariate shift (camera offset → near-zero success)
- Scalable to dexterous hands
- Data quality depends heavily on operator skill; bad demonstrations hurt more than no demonstrations (ML6 finding)
ARCap (Stanford AI Lab, 2024, arXiv:2410.08464): AR-assisted teleoperation that overlays a virtual robot on the real scene via Meta Quest 3 mixed-reality passthrough. Three-layer feedback: visual AR (frame turns yellow at speed limit, blue on collision), haptic controller vibration, and kinematic visualization of retargeted arm motion before recording. Key insight: novice operators consistently violate robot kinematics without feedback — AR catches this at collection time.
- Hardware: Meta Quest 3 + Rokoko data gloves + Intel RealSense D435 — fully portable backpack setup
- User study (20 participants): +40% replay success rate vs. DexCap baseline (no feedback)
- Cluttered scene task: 70% (ARCap) vs. 25% (DexCap)
- 3-stage Lego assembly: 40% (ARCap) vs. 0% (DexCap)
- Works for both dexterous hands and parallel-jaw grippers; open-source
Natural Demonstration
The human performs the task naturally as if with their own hands, without controlling any robot interface. Sensors capture hand pose offline; retargeting happens after collection. The human receives no real-time feedback from the robot.
Representative systems:
- UMI: Cup-shaped handle + GoPro, captures 6-DOF wrist pose in-the-wild; 10k+ episodes
- DexUMI: Extends UMI to 5-finger dexterous hands with finger-joint sensing
- DexWild: In-the-wild bimanual dexterous interactions; 9.5k episodes, 93 environments
Key properties:
- Fastest collection; most natural motions
- Lacks real-time feedback and proprioception
- Retargeting quality is critical — errors propagate directly to policy
PH2D/HAT (Qiu et al., arXiv:2503.13441, 2025): extends natural demonstration to cross-embodiment humanoid learning using VR headsets.
- Hardware: Apple Vision Pro (built-in cameras + ARKit 3D pose tracking) or Meta Quest 3 + ZED Mini stereo cameras (3D-printed mount, <$700)
- Collection: demonstrators wear VR headsets and perform tasks naturally — no robot required. Embodiment gap mitigations: upright seated position + 4× time stretch to match robot execution speed.
- Scale: 50,000+ processed frames with 3D hand-finger poses and language annotations
- Unified state space: 54-dim vector (6D rotations of head + wrists + 3D coordinates of wrists + fingertips); same representation for humans and robots
- Results: +71% OOD generalization improvement vs. robot-only baseline; in-distribution: 49/60 vs. 45/60 (+9%)
- Speed: human demo (VR headset) = 4.09s per task; robot teleoperation = 19.72s — ~5× faster collection
- Few-shot transfer: 20 robot demos + human data significantly outperforms robot-only baselines
- Key finding: human data improves OOD generalization (novel backgrounds, object appearances, spatial arrangements) far more than in-distribution performance. No separate affordance representations needed — treating humans and robots as different embodiments in a unified framework is sufficient.
Passive Observation
The model learns from purely observational human video — no special sensors, no human-robot interaction. The human simply performs tasks while being filmed.
Key properties:
- Near-unlimited data from internet video
- Embodiment gap (human hand ≠ robot hand) is the fundamental obstacle
- No action labels, no proprioception, no tactile data
- Primary use case: large-scale VLA pretraining
- Representative: Human Policy (egocentric video), Open X-Embodiment
Technical Support Dimensions
Five dimensions determine whether collected demonstrations are useful for policy training, cutting across all paradigms:
1. Pose tracking: AprilTag+EKF (~5 Hz, simple but drift) vs ORB-SLAM3+IMU (6.1mm/3.5° accuracy in UMI). Tracking failure corrupts the entire trajectory.
2. Action representation (quantified in UMI):
- Absolute trajectory: 25% success (cup placement)
- Delta action: 80%
- Relative trajectory: 100% (robust to calibration error)
3. Latency compensation: Removing compensation drops dynamic throw from 87.5% to 57.5% (UMI). Mandatory for fast motions.
4. Force sensing: Required for Lv.4+ tasks (precision assembly, peg-in-hole). Multi-axis F/T sensors per finger for dexterous manipulation. RH20T has force-torque data; most datasets do not.
5. Data quality filtering: DC-IL defines action divergence (consistency) + transition diversity (coverage) as quantitative quality metrics. UMI uses kinematic feasibility filtering; VDI uses force-threshold triggering.
Operation Taxonomy: Five Complexity Levels
| Level | Tasks | Technology | Notes |
|---|---|---|---|
| Lv.1 | Push/slide/knock | RGB + joint state | Not dexterous manipulation |
| Lv.2 | Pick-and-place, stack, sort | RGB + joint state | 90% of DROID (76k trajectories) |
| Lv.3 | Open doors, pour liquid | Depth + object pose | — |
| Lv.4 | Precision assembly, peg-in-hole | Multi-axis F/T sensing | Force-control primary |
| Lv.5 | Contact-rich, bimanual, long-horizon | Full sensor stack | UMI, Embodied Survey |
Dexterous manipulation = Lv.4 + Lv.5. Functional threshold: at least one of (1) in-hand reorientation without regrasping, (2) force control primary over position control, (3) three or more independent contact points simultaneously.
Sensor Modality: The Infrastructure Decision
Modality selection is an infrastructure decision made at collection time, not a data quality optimization. If a sensor is absent during collection, the resulting gap is unrecoverable in downstream training:
| Modality | Role in IL | Key threshold |
|---|---|---|
| RGB camera | Visual observation (1–3 views) | Baseline; policy fails on small camera offset (ML6: near-0% on uncalibrated shift) |
| Depth (RGB-D) | 3D structure for spatial reasoning | Enables point cloud input; LeRobot 16k datasets show consistent gains |
| Proprioception | Joint angles, velocities, torques | Low cost; critical for manipulation precision |
| Tactile sensing | Contact forces, pressure distribution | Required for Lv.4+ tasks; +42pp success on contact-rich tasks (GI Labs survey) |
| Language annotations | Task description conditioning | Required for VLA pretraining; must be added at collection time, not retrofitted |
Scaling bottleneck note (arXiv:2510.10903 §7.1): robot manipulation learning has not yet demonstrated a scaling law. Three causes: (1) teleoperation collection speed is human-limited, (2) sim-to-real gap for contact-rich tasks is unsolved, (3) no unified data format (LeRobotDataset is improving this). Community response: data flywheel — SmolVLA trained on 481 shared datasets improved from 51.7% to 78.3% success, validating the open data sharing model.
Simulation as an Extended Paradigm
§7.1.1 of arXiv:2510.10903 classifies “Synthetic / Automatic Data Generation” as a parallel category to human demonstration:
- Sim teleoperation: Human demonstrates inside Isaac Lab / MuJoCo — fully counts as LfD
- Demo-seeded synthesis (MimicGen, DexMimicGen): Start from few human seed demos, auto-synthesize variants via TAMP — Lv.4 intermediate (LfD extended). MimicGen (CoRL 2023): 50,000+ demonstrations across 18 tasks from <200 human seed demos (250x+ multiplier); BC policies on generated data: 59–96% success; generated vs. human demo (Square task): 79% vs 84%. Extensions: DexMimicGen (bimanual dexterous), SkillMimicGen (compositional), SoftMimicGen (deformable).
- Pure RL (SERL, PPO): No demonstrations; outside LfD taxonomy
DexScrew (Hsieh et al., UC Berkeley, arXiv:2512.02011, Dec 2025): sim-to-real framework for thread-contact tasks where accurate simulation is impossible.
- Three-stage pipeline: (1) RL with simplified objects in 8,192 parallel environments to learn rotational finger gaits; (2) use the learned policy as a teleoperation primitive to collect real-world demos with tactile + proprioceptive sensing; (3) BC policy trained on multisensory real demos
- Key insight: direct sim-to-real fails for thread interaction (cannot simulate downward thread engagement); the hybrid approach bypasses this by using sim only for finger-gait primitives, not contact dynamics
- Nut-bolt fastening results (square nuts): 97.50±7.91% progress ratio (tactile + history) vs. 63.75±33.05% without; triangular nuts: 96.25±8.44% vs. 30.00±39.62%
- Screwdriving results: BC + tactile + history: 95.00±13.24% progress ratio, 187.87s; direct sim-to-real: 41.60±26.21%, never completes
- Temporal history matters more than tactile alone; combining both achieves best performance
AutoMate (NVIDIA + USC, 2024): sim-to-real assembly skills across 100 diverse geometric part pairs. Demonstrates that assembly-by-disassembly (record 100 disassembly demonstrations, reverse replay as assembly demonstrations) bypasses the difficulty of manually demonstrating assembly. Three-stage generalist training: BC distillation from specialist → DAgger → Curriculum RL.
- Specialist policy: 86.5% real-world success rate across 80 assembly geometries; 4.2% sim-to-real gap
- Generalist policy: 84.5% real-world success (vs. simulated 80.4%) — real-world outperforms simulation
- Full pipeline with perception initialization: 86.0%
- Key finding: domain randomization + IL hybrid keeps sim-to-real gap to 4.2% for precision contact tasks
Data Annotation Density: The Scaling Lever (π0.7)
Physical Intelligence’s π0.7 (2026) demonstrates that data annotation density — not raw volume — is the key lever for scaling multi-task robot policies:
Why naive scaling fails: when a dataset contains conflicting strategies for the same task (different humans do it differently), a policy trained without context “averages together different behaviors.” Adding more heterogeneous data makes this worse, not better — the model learns the mean of incompatible strategies, which executes none correctly.
The fix: a data-quality score in the episode metadata. During training, the model can now receive information about which examples to imitate faithfully vs. which to merely observe. Adding this metadata allows large, messy, multi-source datasets to be trained without the averaging problem. This is the key annotation engineering insight — not architectural.
Subgoal conditioning as bootstrapped world model: rather than running an expensive world model to generate full future trajectories, π0.7 predicts a single future frame (the subgoal image) and conditions the policy on it. With a subgoal image in the prompt:
- Action prediction becomes inverse dynamics (“what action gets me from current observation to this future observation?”)
- The hypothesis space collapses — multimodal action distributions become unimodal conditioned on the target
- Instruction following is more reliably driven by the subgoal image than by language (subgoal overrides the training data’s language distribution)
Cross-embodiment transfer without task-specific data: π0.7 demonstrates cross-embodiment transfer for new tasks with zero task-specific demonstrations — but the robot had seen the target embodiment doing other tasks. The transfer relies on compositional generalization from shared motion primitives.
Compositional generalization is the north star: the goal the entire field is now converging on. The disagreement is only about which data source achieves it — π0.7’s answer is much better-annotated data of many kinds, not more data of one kind.
Data Scaling Laws for Imitation Learning
Hu et al. (Tsinghua University / Shanghai Qi Zhi Institute, arXiv:2410.18647) provide the most systematic empirical study of data scaling in robot manipulation IL. They collected over 40,000 demonstrations using UMI (hand-held gripper) across 4 tasks and conducted more than 15,000 real-world robot rollouts.
Power-law relationship: Policy generalization follows a power law with the number of training environments and objects. For Pour Water, the optimality gap scales as y = 1.180 · x^{-0.844} with number of training objects or environments (Pearson r = −0.987 and −0.960 respectively). The power law does not hold for number of demonstrations per environment — performance plateaus quickly once a threshold is reached.
Diversity is all you need: Increasing environment and object diversity is far more effective than increasing the number of demonstrations per environment-object pair. When environment-object pairs reach 32, performance saturates at around 25–50 demonstrations per pair. Adding more demonstrations per pair beyond saturation gives near-zero marginal gain.
Practical recipe: 32 environment-object pairs × 50 demonstrations each → ~90% success rate on unseen environments and objects, achievable in one afternoon with four data collectors. Verified on four tasks: Pour Water (85.0% success), Mouse Arrangement (92.5%), Fold Towels (87.5%), Unplug Charger (90.0%).
Visual encoder scaling matters; diffusion model scaling does not:
- DINOv2 ViT-L/14: 0.90 normalized score
- DINOv2 ViT-B/14: 0.81
- DINOv2 ViT-S/14: 0.66
- Small U-Net (diffusion): 0.88 vs large U-Net: 0.83 — no consistent improvement from scaling the action model
This is consistent with the Compression Gap principle (arXiv:2604.03191): when the action model uses diffusion (continuous actions), the vision encoder is the binding constraint, not the action model size.
Implication for dataset design: the community’s focus on increasing total demonstration count is likely misplaced for single-task generalization. The primary leverage point is environment-object diversity. A dataset with 32 diverse environments × 50 demos strictly dominates one with 8 environments × 200 demos, even holding total demonstrations constant.
Open Questions
- Kinesthetic teaching cannot scale to 21-DOF dexterous hands: is there a hybrid where kinesthetic captures finger forces only and teleoperation provides configuration?
- What is the minimum operator skill threshold below which teleoperation data is net-negative? How do you detect and discard bad demonstrations online?
- Can passive observation (internet video) become a first-class data source once embodiment gap is solved? What would “solved” look like?
- What is the minimum annotation density (metadata richness) needed to prevent the behavior-averaging failure mode when training on heterogeneous multi-source data?