Simulation-Based Human Demonstration Collection
Sources: Research synthesis, 2026-04-05; Session notes, 2026-04-05 Raw: Human Demo Collection in Simulation: Cross-Domain Comparison; Gaussian Splatting for Robot Data Collection Updated: 2026-08-12
Overview
Three research communities approach simulation-based human demonstration collection differently: Robotics uses it for scalable imitation learning with automatic data amplification; Computer Vision uses it to extract robot-compatible demonstrations from natural human video; HRI uses simulation to study human perception and preferences rather than trajectory capture. The differences in purpose drive fundamentally different tool choices.
The Robotics Approach: Teleoperation + Amplification
Direct Simulation Teleoperation
Isaac Lab Mimic is the most complete open-source platform, providing the full flow from teleoperation device to behavior cloning training.
Supported input devices (by precision):
| Device | Cost | DOF | Best for |
|---|---|---|---|
| Keyboard | $0 | 6-DOF discrete | Quick testing |
| SpaceMouse | $150 | 6-DOF continuous | Tabletop manipulation (most common in research) |
| Meta Quest 3 (CloudXR) | $500 | Full hand tracking | Dexterous manipulation |
| Apple Vision Pro | $3,500 | Hand + eye tracking | Highest precision |
Workflow: human teleoperates (~10–200 demos, HDF5 format) → annotate subtask boundaries → Isaac Lab Mimic auto-generates more → Robomimic BC training.
Key guidance from Isaac Lab documentation:
- Keep demonstrations short — fewer decision points increases auto-generation success rate
- Do not pause — smooth continuous motion is better than precise-but-hesitant motion
- Take direct paths — roundabout trajectories reduce auto-generation success rate
MimicGen: 250× Amplification
Core mechanism: decompose each demo into “object-centric segments” (action sequences relative to one object), apply rigid SE(3) transformation to adapt each segment to new object placements, reconnect via IK + interpolation, filter failures via physics simulation.
Numbers: 200 human demonstrations → 50,000+ synthetic demonstrations (250× amplification).
2024 extensions: DexMimicGen (22-DOF humanoid bimanual), SkillMimicGen (GPU-accelerated motion planning), SoftMimicGen (deformable objects).
Limitations: rigid objects only (cloth and liquids unsupported); success rate drops sharply for tasks with more than 5 subtasks; synthetic quality is lower than human demonstrations (but volume compensates).
Quest 3 + Isaac Sim Full-Body Tracking
A $500 Quest 3 + ALVR + SteamVR + Isaac Sim pipeline can capture 63 degrees of freedom (9 body joints × 7-DOF) with WiFi latency under 50ms. Isaac Lab 2.3 (2025) officially supports Quest 3 and Manus gloves, simplifying this workflow.
The CV Approach: Human Video as Demonstration Data
EgoMimic: Egocentric Video from Wearable Glasses
Key claim: 1 hour of human hand motion data from an egocentric camera is more valuable than 1 hour of robot teleoperation data for imitation learning.
Hardware: Meta Project Aria research glasses (not consumer), with built-in 3D hand tracking via SLAM + depth.
Workflow: researcher wears Aria glasses while performing tasks naturally (no robot present) → records RGB-D video + 3D hand joint positions → post-process: camera extrinsic calibration, retarget hand joints to robot end-effector pose → mix with small number of robot demonstrations.
Implication: if this generalizes, the field should invest more in human data collection infrastructure than in faster teleoperation.
ARCap: AR Feedback for Non-Expert Demonstrators
Problem: without feedback, even motivated users produce demonstrations that exceed robot joint limits or cause collisions — only discovered at replay time.
Solution: AR passthrough mode overlays a virtual robot, providing three feedback layers:
- Visual: velocity violations turn yellow, collisions turn blue
- Haptic: controller vibration for collision warnings
- Kinematic visualization: virtual robot arm shows what the robot can actually reach
Result: replay success rate +40%; enables non-experts to produce deployment-quality demonstrations.
Cost: approximately $2,500 total (Quest 3 + Rokoko gloves + RealSense camera + laptop).
The HRI Approach: Studying Human Perception in Simulation
HRI’s use of simulation is fundamentally different: the goal is not collecting trajectory demonstrations but studying how humans perceive and respond to robots.
Typical design: participants interact with pre-programmed robots in VR environments; researchers record head pose, gaze direction, reaction times, verbal feedback. “Time freeze” technique: pause the scene mid-interaction and query participants for subjective ratings within the VR display, avoiding retrospective recall bias.
Distributed collection: VR simulation allows researchers, participants, and control servers to be in different locations, dramatically reducing the cost of large-scale user studies.
Potential crossover: HRI-collected data (attention, preference signals, corrections) could feed robot policy learning via HITL/DAgger mechanisms or RLHF-style preference comparison.
Gaussian Splatting for Data Augmentation
3D Gaussian Splatting (3DGS) reduces the marginal cost of demonstrations: the scanning cost is fixed (one-time), but the number and diversity of synthesized demonstrations can be very large.
Robo-Pearls: Language-Guided Synthesis
Scans a real scene into 3DGS, then uses language instructions to edit the scene (swap objects, change camera angle, change lighting). Automatically generates diverse synthetic demonstrations without re-teleoperation. Used in the Data Augmentation category of arXiv:2510.10903 §7.1.2.
RoboSplat: One-Shot → Many Variants
Directly edits 3DGS reconstructions starting from a single human demonstration. Synthesizes variants across objects, poses, viewpoints, lighting conditions, and robot embodiments. Substantially boosts one-shot visuomotor policy generalization. Described in arXiv:2510.10903 §5.6 as a 3D representation planner.
Core pattern: scan once → edit programmatically → train on synthetic variants. Addresses the teleoperation bottleneck by shifting from linear collection cost to amortized reconstruction cost.
Other 3DGS applications (non-data-collection): GaussianGrasper/GraspSplats/SparseGrasp extract grasp points from scene geometry; Splat-MOVER uses editable 3DGS as a digital twin for open-vocabulary manipulation planning; ManiGaussian uses dynamic 3DGS as a multi-task world model.
Decision Framework
| Goal | Tool |
|---|---|
| IL policy training, limited budget | Isaac Lab + SpaceMouse or Quest 3 |
| Need >1,000 diverse demonstrations | MimicGen / SkillMimicGen |
| No physical robot available | EgoMimic (Aria glasses) or ARCap (virtual robot in AR) |
| Non-expert operators, quality concern | ARCap (AR real-time feedback) |
| Studying HRI rather than training policies | VR simulation platform + time-freeze questionnaire |
| Dexterous hand / full-body humanoid data | Quest 3 + Isaac Lab 2.3, or DexMimicGen |
Open Questions
- Can ARCap’s AR feedback and MimicGen’s amplification be integrated into a single pipeline?
- Can EgoMimic’s approach extend to contact-rich dexterous tasks (not just gross manipulation)?
- Can HRI preference data (human judgments during robot execution) serve as RLHF-style training signal for robot policies?