Deep Generative Models in Robotics: Learning from Multimodal Demonstrations
Authors: Julen Urain, Ajay Mandlekar, Yilun Du, Mahi Shafiullah, Danfei Xu, Katerina Fragkiadaki, Georgia Chalvatzaki, Jan Peters Submission: IEEE Transactions on Robotics (TRO)
Overview
Surveys how deep generative models are used to learn robot behaviors from demonstration data, addressing the limitation of classical methods that fail to capture complex multimodal action distributions or don’t scale to large datasets.
Generative Model Families Surveyed
| Model | Strengths | Limitations |
|---|---|---|
| Energy-Based Models (EBMs) | Flexible distribution modeling | Training stability |
| Diffusion Models | High-quality multimodal output | Slow inference |
| Action Value Maps | Spatial reasoning | Limited to discrete actions |
| GANs | Fast sampling | Mode collapse, training instability |
Why Multimodal Demonstrations Matter
Demonstrations collected from multiple humans (or same human across trials) exhibit multimodality: different strategies for the same task. Classical Behavior Cloning averages these modes, producing behaviors that don’t match any single demonstration. Generative models capture the full distribution.
Applications
- Grasp generation: diverse, valid grasps for novel objects
- Trajectory generation: smooth, physically plausible paths
- Cost learning: inferring reward functions from demonstration
Key Challenge
Out-of-distribution generalization: generative models trained on demonstrations fail when the robot encounters states not covered by the dataset.