From Human Videos to Robot Manipulation: A Survey on Scalable VLA Learning with Human-Centric Data
Authors: Zhiyuan Feng, Qixiu Li, Huizhi Liang, Rushuai Yang, Yichao Shen, Zhiying Du, Zhaowei Zhang, Yu Deng, Li Zhao, Hao Zhao, Zongqing Lu, Oier Mees, Marc Pollefeys, Jiaolong Yang, Baining Guo
Submitted: May 18, 2026 | arXiv:2606.00054 [cs.RO, cs.AI, cs.CV]
Abstract
Examines how human video data can be leveraged to train Vision-Language-Action (VLA) models for robot control. Human videos are abundant and capture rich interactions, yet direct application faces challenges due to embodiment differences and missing task annotations.
The survey organizes approaches into four categories:
- Latent action representations — encoding action information in a latent space without explicit labels
- Predictive world models — using video prediction as a surrogate for action supervision
- Explicit 2D supervision — deriving 2D spatial signals (keypoints, optical flow) as action proxies
- Explicit 3D reconstruction — recovering 3D scene/hand structure to bridge video and robot action
Three Open Challenges
- Converting unstructured human videos into training-ready episodes
- Bridging the embodiment gap: video supervision → robot-executable actions across different morphologies
- Developing evaluation protocols for real-world deployment and cross-embodiment transfer effectiveness
A curated resource list is maintained at the authors’ GitHub.