MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations
Abstract
We present MimicGen, a system for automatically synthesizing large-scale, diverse datasets from a small number of human demonstrations. MimicGen adapts source human demonstrations to new scene configurations by decomposing demonstrations into object-centric segments and recomposing them through rigid-body transformations.
Key Contributions
MimicGen addresses a critical bottleneck: large-scale robot learning requires massive labeled datasets, but human teleoperation is slow and expensive. The system generates over 50,000 demonstrations across 18 tasks from fewer than 200 human demonstrations — a 250x+ data multiplier.
Pipeline
- Seed collection: Human operator collects ~10–200 demonstrations in simulation via teleoperation
- Demonstration decomposition: Each demo is split into object-centric subtask segments
- Scene variation: Each segment is rigidly transformed to match new object poses/configurations
- Segment recomposition: Segments are stitched together via motion planning (IK + interpolation)
- Physics validation: Generated trajectories are executed in simulator; failures are discarded
- Dataset output: Successful trajectories stored in HDF5 format for downstream IL training
Supported Tasks (18 total)
Tasks span complexity levels:
- Simple: nut assembly, pick-and-place
- Medium: coffee preparation, kitchen manipulation
- Complex: multi-part gear assembly, stack 3 objects
Key Results
- 50K+ demonstrations from <200 human demos
- BC policies trained on generated data: 59–96% success depending on task
- Generated vs human demo comparison (Square task): 79% vs 84% — comparable quality
- 10 human demos on one mug → 1,000 examples across 12 different mugs
Extensions (2024)
- DexMimicGen: extends to bimanual dexterous manipulation with humanoid hands
- SkillMimicGen: skill-conditioned variant for compositional manipulation
- SoftMimicGen: deformable object manipulation variant
Implementation
- Built on Robosuite simulation framework (MuJoCo backend)
- Open-source: github.com/NVlabs/mimicgen
- HDF5 dataset format compatible with Robomimic IL framework
- Published at CoRL 2023; extended work at CoRL 2024
Limitations
- Requires simulation environment (MuJoCo/IsaacGym)
- Object-centric assumptions: complex contact (cloth, liquids) poorly supported
- Generated data quality degrades for long-horizon tasks requiring many decision points
- “Mixed-quality” trajectories: auto-generated data quality lower than expert human demo