Quartz 5

Robots Need More Than VLAs & World Models

Properties2
descriptionA position paper arguing that robot generalization depends not only on scaling VLAs and world models, but on grounding mechanisms that convert unstructured behavioral data into robot supervision.
tagsclippings, research, robotics, ai

3 min read

Robots Need More Than VLAs & World Models

Authors: Elis Karcini, Faisal Mehrban, Quang Nguyen (Motoniq.ai); Mac Schwager (Stanford / Motoniq.ai); Arash Ajoundani (Istituto Italiano di Tecnologia); César Cadena, Marco Hutter (ETH Zurich); Jan Peters (TU Darmstadt); Haitham Bou-Ammar (UCL Centre for AI)

arXiv: 2606.06556v1 — June 4, 2026

Abstract

The authors contend that robot generalization relies on more than scaling vision-language-action (VLA) models. The central challenge involves converting “unstructured behavioral data into grounded robot supervision.” They identify four critical missing components: autolabeling mechanisms for unstructured behavior, embodiment interfaces for translating human motion to robot actions, physics-grounded reasoning for 3D prediction, and reward interfaces for inferring task progress from videos and language.

Key Contributions

1. Problem Framing. Rather than viewing robotics solely as a policy-learning challenge, the paper reframes it as a grounding problem — making diverse physical experience usable for robot learning. While VLAs have advanced substantially, they depend on upstream mechanisms for data transformation.

2. Robot-Native Supervision Analysis. The survey details progress in cross-embodiment datasets (Open X-Embodiment, DROID, RH20T) and foundation models (RT-1, RT-2, OpenVLA, π₀). However, these systems still require experience already expressed in robot-compatible formats with explicit actions and task labels.

3. Weak Grounding from Video. The paper reviews latent-action approaches (LAPA, UniVLA) and task-progress signals (PROGRESSOR, ReWiND, TimeRewarder) that extract value from passive video. These methods demonstrate that videos contain behavioral information, yet this signal remains “not yet robot supervision” without proper grounding mechanisms.

4. Experience Generation. Discussion of simulation routes (RLBench, ManiSkill, CALVIN), synthetic data generation (MimicGen, RoboCasa365), and learned world models reveals how generated experience scales robustly only when preserving physically relevant quantities like contact, geometry, and dynamics.

Proposed Four Missing Components

Physical Data Engines and Embodied Autolabeling. The system should “recover hidden labels automatically,” converting heterogeneous inputs (video, wearable sensors, tactile streams, logs) into structured supervision: object states, contacts, task phases, latent actions, rewards, and success indicators.

Task-Preserving Retargeting. Mapping latent physical actions across embodiments while “preserving the intended effect on the world” rather than merely copying human kinematics. This involves maintaining goal-relevant physical outcomes despite morphological differences.

Physics-Grounded World Models. Models that predict “not only what the world may look like after an action, but what physically changes and why,” reasoning about geometry, contact, force, constraints, and material properties rather than only visual plausibility.

Self-Improving Deployment Loops. Converting real-world execution traces into structured supervision through task-conditioned reward grounding, enabling systems to learn from successes, failures, and human corrections in closed-loop fashion.

Technical Formalism

The paper introduces mathematical notation for heterogeneous episodes with asynchronous streams (video, motion-capture, tactile, robot logs, language), temporal alignment variables, and latent physical event sequences. It describes inference over object states, contacts, task phases, latent actions, and reward signals without requiring complete manual annotation.

  • Foundation Models: GR00T N1, Gemini Robotics, Figure’s Helix
  • Representation Learning: R3M, VIP, MVP, VC-1
  • Video-Based Methods: AVID, XIRL, DVD, Adapt2Reward
  • World Modeling: Dreamer variants, Genie, V-JEPA 2, ContactGaussian-WM
  • Physics-Informed Models: Deep Lagrangian Networks, Hamiltonian Neural Networks, Gaussian World Models
  • Humanoid Robotics: LeVERB, WholeBodyVLA, HEX

Main Thesis

“The central bottleneck is not only policy learning, but the absence of mechanisms that convert the world’s abundant unstructured behavioral data into grounded robot supervision.” Future systems should organize around these grounding layers rather than purely around model scaling.

Graph View

Backlinks