PREPRINT SUBMITTED TO IEEE TPAMI 1
An Anatomy of Vision-Language-Action Models:
From Modules to Milestones and Challenges
Chao Xu, Suyu Zhang, Yang Liu, Baigui Sun, Weihong Chen, Bo Xu, Qi Liu, Juncheng Wang,
Shujun Wang, Shan Luo, Jan Peters, Athanasios V. Vasilakos, Stefanos Zafeiriou, Jiankang Deng
Abstract—Vision-Language-Action (VLA) models are driving a revolution in robotics, enabling machines to understand instructions
and interact with the physical world. This field is exploding with new models and datasets, making it both exciting and challenging to
keep pace with. This survey offers a clear and structured guide to the VLA landscape. We design it to follow the natural learning path of
a researcher: we start with the basic Modules of any VLA model, trace the history through key Milestones, and then dive deep into the
core Challenges that define recent research frontier. Our main contribution is a detailed breakdown of the five biggest challenges in: (1)
Representation, (2) Execution, (3) Generalization, (4) Safety, and (5) Dataset and Evaluation. This structure mirrors the developmental
arXiv:2512.11362v3 [cs.RO] 19 Dec 2025
roadmap of a generalist agent: establishing the fundamental perception-action loop, scaling capabilities across diverse embodiments
and environments, and finally ensuring trustworthy deployment—all supported by the essential data infrastructure. For each of them,
we review existing approaches and highlight future opportunities. We position this paper as both a foundational guide for newcomers
and a strategic roadmap for experienced researchers, with the dual aim of accelerating learning and inspiring new ideas in embodied
intelligence. A live version of this survey, with continuous updates, is maintained on our project page.
Index Terms—Vision-Language-Action Model, Artificial Intelligence, Embodied Intelligence, Robotics, Foundation models
✦
1 I NTRODUCTIONPaper Structure
Sec A.1: Applications
The quest for general-purpose Secrobots
5: Application that can operate in
real-world human environments is a central goal of artificial Family & Industry……
intelligence. In recent years, Familya new approach has emerged
Industry Autinomous Vehicle
as one of the most promising paths toward this goal: Vision- Sec 4: Challenges & Solutions & Future Directions
Language-Action (VLA) models. Solution,
Sec 4: Challenge, By connecting Future Direction vision, lan- 4.1 Multi-Modal
Alignment and Physical 4.3 From
guage, and physical action, these models have catalyzed
4.1 Multi-modal and physical world
World Modeling
4.2 Instruction Following,
Generalization to
Continuous Adaptation
representation 4.3 Generalization and continuous Planning, and Robust
rapid progress, making the field of embodied
4.2 Complex task planinng and
Adaptationintelligence Real-Time Execution 4.5 Data Construction
and Benchmarking
Challenge
both exciting and increasingly complex. long-term reasoning
4.5 Data, benchmark, Evaluation
4.4 Safety, Interpretability
and Reliable Interaction
Standards
To help navigate this rapidly growing
4.4 Safety, explanation, interaction landscape, numer-
ous survey papers have recently emerged, covering the field
Sec 3: Evolution & Milestones
from various perspectives. On the one hand, several works
Sec 3: Timeline & Milestone Architectures
provide focused, in-depth reviews on specific technical
3.1 End-to-End 3.2 Hierarchical VLA VLMs/LLMs Policy
subareas, such as action tokenization [1], efficient training
VLA Policy
VLMs/LLMs Perception+Brain+Action Perception+Brain Action
paradigms [2], and post-training methodologies [3], offering
Percetion+Planning+Control Percetion+Planning Control
granular insights into individual system components. On Time Line
OpenVLA … 𝝅𝟎
the other hand, broader surveys [4]–[9] offer comprehensive Time Line
OpenVL … 𝝅𝟎
system overviews. These works A typically serve as structured
taxonomies, organizing the VLA landscape by model archi- Sec 2: Basic Modules
tectures, input modalities, or Sectraining
2: Basic Component objectives, providing 2.2 Robot Perception A.2.1 Training Strategy
A.2.2 Dataset RL
readers with a systematic list of the core components. Input 2.3 Robot Brain
BC
PM
2.1 Robot Percetion 2.2 Robot Brain 2.3 Robot Action
Planning
Sample A.2.3 Evaluation
C. Xu, S. Zhang, Y. Liu, B. Sun, W. Chen, B. Xu, Q. Liu are with IROOTECH 2.4 Robot Action
Real World
2.4 & 2.5 Training Strategy & Dataset
Update
TECHNOLOGY (e-mail: chaoxuxc@gmail.com, sunbaigui85@gmail.com). Simulation
(Evaluation)
C. Xu, S. Zhang, Y. Liu, B. Sun are with Wolf 1069 b Lab, Sany Group.
Y. Liu and S. Luo are with the Department of Engineering, King’s College
London (e-mail: yang.15.liu@kcl.ac.uk, shan.luo@kcl.ac.uk). Fig. 1: The structure of this survey in a pyramid format. Section 2 lays
J. Wang and S. Wang are with the Hong Kong Polytechnic University (e-mail: the foundational knowledge by deconstructing the core components of
wjc2830@gmail.com, shu-jun.wang@polyu.edu.hk). any VLA model. Building upon this, the second stage, Section 3, traces
J. Peters is with the Computer Science Department of the Technische Univer- the historical evolution of the field through its most representative
sität Darmstadt (e-mail: peters@ias.tu-darmstadt.de). works, providing context and intuition. The deepest stage, Section 4,
A. Vasilakos is with Department of ICT and Center for AI Research, Univer- serves as the intellectual core, offering an in-depth analysis of the grand
sity of Agder (UiA) (e-mail: th.vasilakos@gmail.com). open problems and outlining actionable future research directions. The
S. Zafeiriou and J. Deng are with the Department of Computing, Imperial final section depicts the various applications, which are included in
College London (e-mail: j.deng16@imperial.ac.uk, s.zafeiriou@imperial.ac.uk). Appendix A.1.
C. Xu, S. Zhang and Y. Liu contributed equally to this work.
PREPRINT SUBMITTED TO IEEE TPAMI 2
However, we identify two key gaps that this survey aims inputs for planning, and the action module executes motor
to address. First, existing surveys often relegate research commands. Recently, these components are undergoing a challenges to a concluding section—a high-level overview fundamental shift: Perception (Sec. 2.2) is evolving from appended at the end of the paper. The field still lacks a standard visual backbones to Language-Aligned Transform- unified resource that places these challenges at its core, ers (e.g., SigLIP) to bridge the semantic gap, increasingly systematically breaking them down, comparing alternative augmented by geometric representations (e.g., DINOv2) to solution paths, and charting clear directions for future work. ensure manipulation precision. The Brain (Sec. 2.3) is con- For researchers aiming to make novel contributions, a mere verging toward pre-trained VLMs, leveraging internet-scale list of problems is insufficient; what is needed is a deep, knowledge to enable zero-shot generalization and unified structured analysis of the problem space. Second, the struc- token processing. Finally, Action (Sec. 2.4) is pivoting from ture of most surveys does not align how researchers learn a discrete tokenization towards continuous generative model- new field. Most existing works simply list and group meth- ing (e.g., Diffusion), achieving smooth, multi-modal distri- ods by category—like grouping visual-based approaches in bution modeling. Notably, to prioritize the in-depth analysis one chapter and control strategies in another. While this of challenges (Sec. 4), we provide a streamlined overview facilitates quick reference, it presents a fragmented view here due to limited space. For detailed architectural tax- of the field. It provides extensive information but fails onomies, we recommend other specialized surveys [4], [8]. to illustrate how these pieces integrate into a coherent, evolving research timeline. Consequently, such surveys do 2.2 Robot Perception not guide newcomers from foundational concepts to recent breakthroughs along a clear, progressive learning trajectory. 2.2.1 Vision Encoders in VLA This survey makes two core contributions to address (1) Convolutional Networks (CNNs). CNNs [10] remain these gaps. Our primary contribution is a deep and sys- indispensable in VLA due to their strong local feature tematic analysis of the core challenges in VLA research. extraction and translation equivariance, making them effec- Rather than appearing as a brief concluding section, our tive visual encoders in real-time and resource-constrained challenge analysis forms the central pillar of this survey. settings. Modern architectures such as ResNet and Efficient- We identify five key challenges following the developmental Net [11], [12] are widely adopted. CNNs commonly serve as roadmap of VLA: (1) Multi-Modal Alignment and Physical visual backbones in end-to-end policies by encoding RGB or World Modeling, (2) Instruction Following, Planning, and depth observations into compact features for downstream Robust Real-Time Execution, (3) From Generalization to decision-making; representative systems such as Diffusion Continuous Adaptation, (4) Safety, Interpretability, and Re- Policy [13] and SPECI [14] use ResNet-based encoders. liable Interaction, (5) Data Construction and Benchmarking CNNs also integrate naturally into hierarchical designs, Standards. For each, we provide an in-depth review of where lightweight models handle high-frequency percep- competing solutions and outline concrete avenues for future tion, as in HiRT [12], [15], which employs EfficientNet-B3. research. Our goal is twofold: to help researchers efficiently As world-model-based VLA frameworks grow in complex- navigate the vast landscape of existing work and to position ity, CNNs increasingly act as compact encoders for high- this section as a direct catalyst for novel research ideas. dimensional observations. For example, LUMOS [16] uses Our second contribution is the unique structure of this a CNN front-end to produce latent features consumed by survey, designed to mirror the natural learning journey of a RSSM [17] for prediction and planning. researcher. We intentionally structure this survey as a step- (2) Vision Transformers (ViT). ViT [18] and its variants by-step roadmap. We begin with a detailed breakdown of have become the dominant perception backbone in modern the foundational Modules that constitute any VLA model, VLA systems. Their self-attention captures global context establishing a shared vocabulary. We then trace the histori- and long-range dependencies, and patch tokenization aligns cal evolution through key Milestones, providing context for visual inputs with Transformer-based language models, how the field has arrived at its current state. This journey making ViT well suited for end-to-end VLA pipelines [19]. culminates in our deep dive into the core Challenges, demon- Contemporary VLA frameworks therefore rely heavily on strating recent trends and pointing out future directions. large-scale pretrained ViT encoders, typically fine-tuned for This structure allows newcomers to build expertise from the stronger generalization and efficiency. ViT-based visual en- ground up, while allowing experienced researchers to access coders in VLA generally follow four structural paradigms: the sections most relevant to their interests. The structure of a) Language-Supervised Visual Encoders. Models such this survey is illustrated in Fig. 1. This work is designed as CLIP [20] and SigLIP [21] learn vision features aligned as a living resource, and project page will be continuously to human semantics via contrastive learning from internet- updated to reflect advances at the research frontier. scale image–text pairs. Adopting such encoders is now standard practice: for example, π0 [22], RDT-1B [23], TriVLA [24], and ForceVLA [25] use SigLIP as their vision 2 BASIC M ODULES backbone, while many others rely on CLIP, e.g., DeeR- 2.1 Overall and Architectural Trend VLA [26], RationalVLA [27], MinD [28]. Some works inno- Vision-Language-Action (VLA) systems integrate percep- vate in their use, for instance, OTTER [29] extracts features tion, reasoning, and control to translate abstract instruc- from the final layer of a frozen CLIP ViT to obtain strongly tions into physical actions. Typically, a VLA system com- language-aligned visual representations. prises three core modules: the perception module extracts b) Self-Supervised Visual Encoders. These models, ex- grounded observations, the brain module fuses multimodal emplified by DINOv2 [30], avoid textual labels and learn PREPRINT SUBMITTED TO IEEE TPAMI 3
robust visual representations from large unlabeled corpora, systems including InstructVLA [62], FlowVLA [63], and capturing fine-grained geometry and spatial structure that others, also utilize VLMs as their language encoders. make them particularly effective for contact-rich manipu- lation tasks requiring precise physical cues. For example, 2.2.3 Proprioceptive Encoders in VLA LexVLA [31] employs a frozen DINOv2 encoder and a lightweight adapter to map local visual features into sparse, Proprioceptive inputs are provided by onboard sensors and language-aligned lexical representations. typically include (i) joint states: per-joint position, velocity, c) Hybrid Architectures. To combine the semantic and effort/torque; (ii) end-effector states: the 6-DoF pose strengths of language-supervised encoders with the geo- (x, y, z, roll, pitch, yaw), optionally with linear/angular ve- metric precision of self-supervised ones, an increasingly locities, in the world or base frames [64], [65]; and (iii) grip- common strategy is to adopt a hybrid approach. Recent VLA per status: opening width/state and applied force. These frameworks, including OpenVLA [32], OpenVLA-OFT [33], data are low-dimensional, structured vectors. GraspVLA [34], UniVLA [35], and VLA-RL [36] often em- Given the low-dimensional, structured nature of propri- ploy a SigLIP+DINOv2 hybrid to attain strong performance oception, MLPs are the standard, efficient encoders, whose on both semantic understanding and geometric reasoning. outputs are fused with vision and language via concatena- d) Vision-Language Models (VLMs). The most inte- tion or conditioning (e.g., FiLM [66]). Many VLA models grated paradigm directly adopts pretrained VLMs as high- follow this design. TriVLA [24] employs an embodiment- level visual encoders, producing language-conditioned vi- specific MLP, RDT-1B [23] encodes low-D robot states with sual embeddings rather than raw pixel features. Exam- an MLP, SPECI [14] trains an MLP from scratch on joint ples include PaLI-X [37] in RT-H [38]; PaliGemma [39] in angles and gripper states, and systems such as OpenVLA- Hume [40] and Hi Robot [41]; Qwen-VL [42] in VTLA [43], OFT [33] and the GR series [67], [68] similarly include MLP CombatVLA [44], and OpenHelix [45], which leverage modules for proprioceptive fusion. VLMs’ fused vision–language context to provide higher- level inputs for policy and planning. 2.3 Robot Brain 2.2.2 Language Encoders in VLA The robot brain is the core of a VLA system, responsible Language instructions form the semantic core of VLA sys- for fusing multimodal representations from input modules, tems, defining task objectives and providing high-level con- performing reasoning and planning, and ultimately generat- text. The language encoder has evolved alongside advances ing action intentions. Current architectures primarily follow in natural language processing, and in practice falls into four mainstream technical directions: three main categories: (1) Transformer. The Transformer serves as a core VLA (1) Transformer-Based Language Encoders. The earliest architecture by tokenizing vision, language, and proprio- approach involves the use of standard Transformer-based ception inputs and using self-attention to fuse multimodal language encoders. These VLA systems adopt text-only tokens and learn an end-to-end perception-to-action map- Transformers (e.g., BERT [46], T5 [47]) pretrained on large ping. A Generalist Agent [69] demonstrates the capacity of corpora to encode instructions, providing strong semantics a decoder-only Transformer to handle multiple modalities as the entry point of the control stack. Classical examples and tasks, and models such as VIMA [70] and GR-1/GR- include RDT-1B [23] with T5-XXL [47], RoboBERT [48] with 2 [67], [68] further adopt Transformer-based generalist poli- BERT, and early partial implementations of Octo [49] that cies. Other approaches, such as SPECI [14], apply temporal rely on such modules. Transformers across both high-level reasoning and low-level (2) Large Language Models (LLMs). With the rise of execution. Beyond Transformers, recent alternatives also LLMs, VLA systems increasingly adopt billion-parameter emerge; RoboMamba [71] adapts the Mamba [72] architec- models as their language backbone, leveraging their richer ture to VLA for more efficient long-sequence processing. world knowledge and commonsense reasoning to inter- (2) Diffusion Transformer (DiT). Unlike Transformer-only pret ambiguous and compositional instructions. Representa- policies that predict actions directly, this paradigm uses a tive choices include Llama-family models (e.g., OpenVLA- diffusion model as the generative core, with a Transformer OFT [33], VLA-RL [36] with Llama-2 [50] 7B; HiRT [15] guiding the denoising process. Diffusion models are well with Llama-based InstructBLIP [51]), Gemma-family mod- suited for robot control because they model complex con- els [52] (e.g., π0 [22]/π0.5 [53] with Gemma 2B [54]), and tinuous distributions and produce smooth, natural motion InternLM2 [55] (e.g., GraspVLA [34]). trajectories. Diffusion Policy [13] provides early evidence (3) Vision-Language Models (VLMs). The recent trend of the effectiveness of denoising-based generation, helping is to adopt native VLMs, where the language module establish diffusion as a strong policy-learning paradigm. is no longer a standalone component but is jointly pre- More recent methods, such as RDT-1B [23] and TriVLA [24], trained with vision for end-to-end multimodal understand- integrate diffusion on top of Transformer backbones to map ing. For example, several VLA systems explicitly adopt semantics to actions through multi-step denoising. well-known vision–language models: DeeR-VLA [26] and (3) Hybrid Architectures. These models pair Transformer- RoboFlamingo [56] build on OpenFlamingo [57]; Diffusion- based semantic reasoning with a diffusion [73] or flow- VLA [58] instead employs Qwen-VL [42] while Memo- matching [74] head for high-frequency, smooth control. ryVLA [59] is developed upon the 7B Prismatic VLM [60]. π0 [22] exemplifies this design by using a pretrained VLM In contrast, Dexbotic [61] pretrains its own dedicated model, as the Transformer backbone for perception and a separate DexboticVLM, tailored to dexterous manipulation. Other Flow Matching head for action generation. Octo [49] and PREPRINT SUBMITTED TO IEEE TPAMI 4
ConRFT [75] follow a similar pattern, combining a Trans- 2.4.2 Action Decoding former backbone with a generative action head. Diffusion- (1) Autoregressive Decoding. In autoregressive (AR) de- VLA [58] injects LLM reasoning into the diffusion process coding, the policy emits actions step by step with causal to coordinate high-level planning with low-level execution. masking, and each prediction conditions on all previously MinD [28] adopts a hierarchical hybrid structure, using generated actions and observations, enabling modeling of distinct diffusion models for low-frequency video prediction long-range temporal dependencies. AR remains standard and high-frequency action control. in early and many recent VLA models (e.g., A Generalist (4) Vision-Language Models (VLMs). This paradigm treats Agent [69], VIMA [70], RT-H [38], SafeVLA [79], GR-2 [68], a full pretrained vision-language model as the core robot 3D-VLA [83], UniVLA [35], OpenVLA [32], TraceVLA [81], brain, leveraging its perception, multimodal fusion, com- CombatVLA [44]). monsense reasoning, and sequence modeling, while inte- (2) Non-Autoregressive Decoding. To reduce latency, non- grating robot-specific proprioception and action spaces on AR decoders predict an action horizon in one or a few top. RT-2 [76] is a milestone in this direction, extending the passes. One path replaces causal attention with bidirec- VLM’s (i.e., PaLI-X [37]/PaLM-E [77]) output space to in- tional attention to infer all steps jointly (e.g., OpenVLA- clude action tokens, effectively creating an embodied agent. OFT [33]). Another uses inherently non-AR generators such Nearly all current SOTA VLA models, including Open- as diffusion or flow matching that iteratively denoise or VLA [32], π0.5 [53], CoT-VLA [78], SafeVLA [79], DeeR- transform the whole sequence in parallel (e.g., Diffusion Pol- VLA [26], GraspVLA [34], VTLA [43], UniVLA [35], VLA- icy [13], TriVLA [24], RDT-1B [23], π0 [22], RoboBERT [48], RL [36], WorldVLA [80], TraceVLA [81], PointVLA [82], 3D- Hume [40], DeeR-VLA [26]). VLA [83], and BridgeVLA [84] build their decision-making (3) Hybrid Decoding. A practical compromise is chunk- on strong pretrained VLMs. In hierarchical systems, VLMs ing: the policy operates autoregressively over coarse time often act as high-level planners or span both high- and low- (emitting chunks), but non-autoregressively within each level policies, as in A Dual Process VLA [85], Hi Robot [41], chunk (parallel refinement), which improves both stabil- HAMSTER [86], and HiRT [15]. ity and throughput. A representative example is π0.5 [53], which performs AR semantic decisions with parallel low- 2.4 Robot Action level chunk generation. CoT-VLA [78], UniVLA [35], and Robot action is the VLA system’s final execution interface, WorldVLA [80] follow the same design, which support long- translating abstract decisions from the robot brain into con- horizon coherence with efficient local rollout. crete, low-level control commands. Its design directly deter- mines action precision, smoothness, real-time performance, and generalization. 3 E VOLUTION & M ILESTONES The evolution of Vision-Language-Action (VLA) models is 2.4.1 Action Representation driven by the need to overcome the brittleness of traditional Action space representation defines the target language modular pipelines and achieve the broad generalization that the model predicts. Representing typically high- seen in foundation models. This evolution reflects a steady dimensional, continuous robot actions involves a key trade- shift from passive multimodal perception to active, embod- off between performance and learnability. ied reasoning and control. An overview of VLA milestones (1) Discrete Spaces. Continuous controls are discretized into is shown in Fig. 2 and Appendix Tab. S3. bins and cast as a next-token classification problem, nat- From 2017 to 2019, the Vision-and-Language Navigation urally reusing Transformer stacks for sequence prediction. (VLN) benchmark [88] pioneers large-scale evaluation of This is common in generalist Transformer agents (e.g., A agents aligning linguistic instructions with visual environ- Generalist Agent [69], VIMA [70], RT-H [38], SafeVLA [79]) ments for physical navigation. EmbodiedQA [89] advances and in many recent VLA systems (e.g., UniVLA [35], VLA- this direction by defining embodied intelligence through a RL [36], WorldVLA [80], TraceVLA [81], CombatVLA [44]). closed perception–action loop, establishing an early theo- (2) Continuous Spaces. Actions are regressed directly in retical foundation. Follow-up work such as BabyAI [90], normalized continuous domains (e.g., joint angles, end- RCM [91], and Point-Cloud EQA [92] further refine the effector velocities), yielding smoother, higher-precision con- paradigm by improving language-to-action learning and trol at the cost of high demands on model learning ability. introducing early forms of 3D geometric reasoning. This aligns naturally with diffusion or flow-matching poli- The period from 2020 to 2021 marks a shift toward long- cies (e.g., Diffusion Policy [13], TriVLA [24], RDT-1B [23], horizon reasoning and language-conditioned embodied control. π0 [22]) and with continuous variants of prior discrete ALFRED [93] introduces the first interactive benchmark models (e.g., OpenVLA-OFT [33]). Other systems such as combining high-level goals, step-by-step instructions, and iRe-VLA [87], GraspVLA [34], and Hume [40] also adopt object–environment interactions, establishing realistic long- continuous control. horizon tasks. ALFWorld [94] extends this direction by link- (3) Hybrid Spaces. To combine strengths, hybrids as- ing symbolic reasoning with visually grounded execution, sign discrete and continuous encodings to different control and BEHAVIOR [95] standardizes long-horizon household facets: BridgeVLA [84] uses continuous translation with evaluation in high-fidelity simulation. A pivotal milestone discretized rotation. HiRT [15] treats EE pose as continuous of this era is CLIPort [96], which integrates pretrained while gripper open/close is discrete. Hierarchical models visual representations into a language-conditioned policy, often keep high-level skills discrete and low-level execution demonstrating that internet-scale knowledge enables zero- continuous (e.g., Hi Robot [41], HAMSTER [86], π0.5 [53]). shot generalization in robotic manipulation. PREPRINT SUBMITTED TO IEEE TPAMI 5
Fig. 2: The timeline of VLA models, datasets, and evaluation benchmarks from 2022 to 2025. The top row presents major VLA models introduced each year. The bottom row displays key datasets used to train and benchmarks to evaluate these models, grouped by release year.
Since 2022, VLA enters the era of large models and gen- to full-body humanoid control. Another direction targets
eralized learning. SayCan [97] is the first to introduce a open-world autonomy, emphasizing deeper understanding hierarchical framework that separates LLM-based high-level and reasoning. PointVLA [82] injects point-cloud features planning from low-level skill execution, using affordance without retraining the core model, enabling faithful 3D un- and value estimates from the robot to ground candidate derstanding for open-world settings. Cosmos-Reason1 [103] subtasks and select feasible actions. Inner Monologue [98] is the first to standardize physically grounded reasoning for for the first time embeds language models within contin- VLAs, unifying ontologies and benchmarks into an open uous multimodal feedback loops, achieving self-reflection reasoning pipeline and shifting the field toward plug-and- and dynamic behavioral adjustment. RT-1 and RT-2 [76], play, physics-constrained reasoning. CoT-VLA [78] intro- [99] realize end-to-end learning from vision and language duces the first explicit visual chain-of-thought, predicting to action via Transformer architectures, marking the birth of subgoal images as intermediate reasoning before action a truly unified VLA framework. generation. At the core, some models aim to unify prior In 2023, multiple advances emerge, most notably in uni- advances by integrating hierarchy, reasoning, and control. fied multimodal backbones, generative action modeling, and cross- π0.5 [53] unifies high-level reasoning and low-level con- embodiment data scaling. PaLM-E [77] embeds visual and trol via hierarchical Transformers, enabling long-horizon state representations directly into pretrained LLMs, achiev- operation without target-specific robot data. LUMOS [16] ing for the first time a unified multimodal input space. The integrates a learned world model with on-policy RL into introduction of Diffusion Policy [13] applies generative dif- a single system. VLA-RL [36] scales online RL to pre- fusion models to action modeling, bringing greater stability trained VLAs, addressing imitation learning’s OOD limita- and expressiveness to high-dimensional continuous control, tions. GEN-0 [104] offers early evidence for scaling laws in and marking a key paradigm shift in policy generation for robotics, showing that large-scale interaction data enables VLA. Open X-Embodiment [100] represents a meaningful phase transitions in cross-embodiment generalization. milestone in robotic learning, providing large-scale and diverse cross-robot data with open access, and driving the field toward more general and powerful embodied models. 4 C HALLENGES & S OLUTIONS & F UTURE D IREC - TIONS Building on the previous year’s breakthroughs, 2024 broadens the frontier across open-source scaling, generalist Fig. 3 provides an overview of the five core challenges policies, flow/denoising action generation, web-scale video pre- addressed in this section, along with their respective sub- training, and 3D world modeling. Octo [49] establishes a gen- challenges and the relevant papers involved. eralist policy capable of cross-platform, multi-task control. OpenVLA [32] becomes the first fully open-source 7B VLA 4.1 Multi-Modal Alignment and Physical World Model- model, lowering the barrier for large-scale research and ing deployment. π0 [22] is the first to combine pretrained VLMs with flow-matching action generation, setting a new archi- Fig. 4 illustrates the three levels of this challenge, which are tectural reference point for general and precise control. GR- elaborated in detail below. 2 [68] systematizes web-scale generative video pretraining for VLA, enabling broad generalization without propor- 4.1.1 The GAP between Semantics, Perception, and Phys- tional robot labels. 3D-VLA [83] marks a shift toward full 3D ical Interaction world modeling by coupling a generative 3D world model Vision-Language-Action (VLA) tasks center on three core with VLA for plan-by-imagination. components: vision for perceiving the world, language for By 2025, VLA research enters a stage of pluralistic evo- conveying high-level instructions, and action for interacting lution, where diverse embodiments, modalities, and learn- with the physical environment. Together, they form an inte- ing paradigms co-evolve toward general robotic intelligence. grated embodied framework linking perception, reasoning, Humanoid-VLA [101] and GR00T N1 [102] extend VLA and execution. The central challenge is bridging the gap PREPRINT SUBMITTED TO IEEE TPAMI 6
gap remains when grounding this understanding into phys-
ical action [109]. One direction is end-to-end fine-tuning,
which reformulates control as sequence generation by dis-
cretizing the action space into tokens and fine-tuning a VLM
to generate these action tokens in the same way it generates
words. RT-2 [76] demonstrates the feasibility of this ap-
proach, and subsequent works such as Prompt-a-Robot-to-
Walk [110], Grounding MLLMs in Actions [111], and Open-
VLA [32] adopt similar paradigms. Another line of work
introduces shared intermediate representations between lan-
guage and action. CLIP-RT [112] extends vision–language
alignment to action generation, and Humanoid-VLA [101]
performs language–action pretraining to narrow the seman-
tic–motor gap. VoxPoser [113] leverages LLM reasoning to
produce intermediate programs and 3D affordance maps
Fig. 3: Taxonomy of VLA challenges, encompassing 5 primary chal- lenges and 15 sub-challenges, with representative works listed. Please that ground perceptual semantics into spatial actions. Re- zoom in for more details. cent studies further mitigate this mismatch by introduc- ing hierarchical architectures [109], [114], [115] that insert an Basic Alignment Spatial Geometry and Dynamic Predictive World Model explicit intermediate layer between language and action, where a VLM serves as a high-level planner and a separate image World low-level controller executes high-frequency motion. vision Language Model Pixel 2D→3D …… V- L GAP (3) Multi-modal Sensory Fusion. As VLA systems evolve, How to Represent? Latent feature perception extends beyond RGB images and language. Depth Pointcloud Voxel Action Vision Language 3D → 4D World Model short-term predict For precise manipulation, vision and instruction alone VL-A GAP Policy enhance Policy are insufficient for accurate physical interaction and fine- force feedback How to Represent? Trajectory grained control [116]. Incorporating additional modalities World How to Integrate? action vision language tactile sensation Model Long-term Rollout such as tactile, force, and audio sensing is therefore essen- auditory Explicit planning sensation
Multi-Modal Sensory Fusion action vision language How to Utilize?
Policy tial for achieving more reliable and comprehensive percep-
tion [117]–[119], yet it significantly increases the complexity
Fig. 4: The challenge of Multi-Modal Alignment and Physical World of modality alignment and model optimization. Modeling. First, Section 4.1.1 addresses the fundamental disalignment A common solution is to build specialized encoders for at the interface of information. Building upon this, Section 4.1.2 focuses on the construction of the world’s geometric and dynamic structure. each sensory modality and align them with language using Section 4.1.3 represents the highest level of understanding, as embodied contrastive learning. TLA [116] integrates tactile perception in dynamic predictive capabilities. to improve contact-rich manipulation, and OmniVTLA [120] constructs a semantically aligned tactile encoder that links tactile feedback with linguistic concepts. After obtaining between abstract semantics and grounded physical reality, effective representations, the challenge shifts to fusion, rang- which can be decomposed into three subproblems: ing from deep fusion across the full pipeline, as in Tactile- (1) Vision Language Gap. Vision provides high- VLA [121], to modular mixture-of-experts fusion that pre- dimensional perceptual input, while language offers ab- serves VLM representations, as in ForceVLA [25]. Due to the stract symbolic semantics. Establishing a precise mapping high cost of collecting real multimodal data, simulation-based between these distinct modalities is essential for ground- generation is emerging as a promising alternative. Multi- ing visual understanding and goal reasoning in the phys- Gen [122] explores this direction by generating visual scenes ical world [45], [105]. Some approaches address this chal- in the simulator and synthesizing additional modalities such lenge by enhancing visual representations to make them more as audio to pretrain or enhance real-world policies. responsive to language conditioning. OTTER [29] intro- duces text-aware feature extraction that preserves semantics 4.1.2 From 2D Images to Spatial-Temporal Representa- aligned with task descriptions, while LIV [106] employs a tions contrastive framework on robot-control data to construct Bridging the semantic–perceptual–physical gap requires a joint vision–language embedding space, enabling visual spatial grounding, meaning that VLA models must accu- features to become inherently sensitive to linguistic cues. rately capture the 3D structure of the environment. Yet most A recent paradigm bridges the vision–language gap via pretrained VLMs are trained on 2D internet images, creating symbolic reasoning with natural language as an intermediate a core limitation: their reliance on RGB inputs restricts representation, powered by LLMs. ACT-LLM [107] trans- the spatial reasoning needed for real-world robotic oper- lates visual observations into structured state descriptions ation [82]. Enabling a 2D-native model to acquire spatio- for symbolic reasoning. Look Leap [108] pushes this fur- temporal understanding is therefore a central challenge. ther by generating full structured action plans, elevating (1) Constructing Spatio-Temporal Representations. Build- vision–language alignment to a higher cognitive level and ing spatio-temporal understanding begins with selecting reframing the problem as one of reasoning. a representation capable of expressing geometric structure (2) Vision–Language Action Gap. Although multimodal and dynamics. Existing approaches primarily follow three models achieve strong perception–semantics alignment, a directions. A straightforward option is to augment RGB PREPRINT SUBMITTED TO IEEE TPAMI 7
inputs with 2.5D depth maps, which provide per-pixel dis- code to directly impose linguistic constraints on spatial tance information and align naturally with 2D images. geometry, and Gemini Robotics [114] infers 3D structure Depth Helps [123] uses depth as a supervision to learn spa- through large-scale multimodal reasoning. Finally, to opera- tial perception without real sensors, while RoboFlamingo- tionalize the 4D perspective within VLA, recent work injects Plus [124] fuses preprocessed depth with RGB features tracked motion as temporal context. TraceVLA [81] overlays to strengthen spatial awareness. These results show that tracked keypoint trajectories as spatial memory, and Spatial even simple 2.5D cues can significantly enhance geometric Traces [140] fuses tracked points with depth maps to encode reasoning. Then, point clouds preserve full 3D geometry structure and motion within a unified input. and offer lossless 3D representation [125]. PointVLA [82] integrates point cloud inputs into pretrained VLA mod- 4.1.3 Dynamic and Predictive World Models els to improve spatial reasoning without modifying the A truly embodied world representation cannot stop at backbone. Later systems, such as An Embodied Generalist static geometry or semantics, it must capture dynamics and Agent in a 3D World [126] and GeoVLA [125], unify 2D causality, i.e., construct an internal, predictive world model and 3D modalities, while FP3 [127] rebuilds the percep- capable of answering the fundamental question: if the agent tion–decision pipeline around point cloud representations executes an action, what happens next? Predictive world under a pretraining–finetuning paradigm. Beyond pure ge- modeling forms the foundation for counterfactual reason- ometry, other studies aim to infuse semantics into point ing, long-horizon planning, and physical understanding. clouds. SoFar [128] constructs semantic 3D scene graphs, (1) Representation Space. A key design choice is how Weakly-Supervised 3D Visual Grounding [129] transfers future states should be represented. One option is to pre- 2D–text alignment to 3D by leveraging CLIP, and LMM- dict directly in the observation space by generating future 3DP [130] fuses back-projected 2D semantic features with pixel-level frames, which provides a high-fidelity, human- geometric point clouds to form unified semantic–geometric interpretable imagination of future states. TriVLA [24] ex- representations. To address the irregular structure of point tends video diffusion models for multi-step visual fore- clouds, other work discretizes 3D space into voxels or oc- casting, while UP-VLA [141] and CoT-VLA [78] generate cupancy grids, enabling structured spatial reasoning. Oc- key subgoal images that indicate the next salient task cLLaMA [131] assigns semantic labels to 3D voxels, while state. DreamVLA [142] enriches prediction with task-critical RoboMM [132] incorporates multi-view temporal cues to cues such as dynamic regions, depth, and affordances, and construct unified 3D occupancy grids. Finally, since real- FlowVLA [63] introduces a visual chain-of-thought mech- world operation is dynamic, a static 3D snapshot is insuf- anism to synthesize physically consistent future scenes. ficient. ARM4R [133] captures spatio-temporal evolution by WorldVLA [80] further models object motion, contact, and predicting the 4D trajectory of 3D point motion, extending state transitions to simulate low-level physical evolution static perception to a time-aware formulation [81]. via learned world dynamics. A complementary approach (2) Architectural Integration. Once a spatio-temporal rep- is prediction in a latent space. This strategy first encodes resentation is chosen, the next challenge is incorporating high-dimensional visual observations into a compact, low- geometric information into VLA models without disrupting dimensional latent space and then learns a simpler model pretrained alignment. A common strategy is augmentation to predict the evolution of this latent state [143]. This is and injection through specialized adapters that introduce 3D more computationally efficient and robust to irrelevant vi- features while preserving the backbone’s integrity as much sual noise. For instance, VLM-in-the-Loop [144] explicitly as possible. PointVLA [82] directly augments 2D models leverages a pretrained latent world model to predict future with point cloud inputs, while GeoVLA [125] processes latent states, while MinD [28] proposes a hierarchical world 2D and 3D streams in parallel. SpatialVLA [134] projects model that performs predictions in dynamic feature spaces 2D semantic features into 3D coordinates using positional at multiple levels of abstraction. WMPO [145] generates encoding and spatial grids to form explicit space–action internally in latent space while aligning policy and opti- graphs. In contrast, implicit approaches avoid modifying mization in pixel space. the backbone by attaching external geometric modules, as (2) Utilization Paradigms. One paradigm is policy enhance- in Evo-0 [135] with VGGT [136], or by using diffusion-based ment, where the world model is tightly integrated with conditioning to model depth reliability, as in AC-DiT [137]. the policy. Short-term future predictions serve as auxiliary Another line of work circumvents direct 3D modeling by inputs or auxiliary training signals [146], providing the reprojecting 3D data into the 2D domain. BridgeVLA [84] policy with forward-looking intuition for more informed renders point clouds into multi-view images, and OG- action selection. Most observation-space models, including VLA [138] generates orthographic projections to recover TriVLA [24], CoT-VLA [78], and DreamVLA [142], follow 3D pose. Some systems predict in 2D and then lift results this strategy by conditioning their action decoders on pre- into 3D, such as A0 [138], which first predicts interaction dicted future states. The second paradigm is explicit plan- points and trajectories in 2D and then lifts them into 3D via ning, in which the world model serves as a decoupled depth projection, and RoboPoint [139], which back-projects internal simulator. In this think-before-you-act framework, 2D keypoints into 3D to create structured action cues. These the agent performs multi-step rollouts of candidate action methods preserve the strengths of large-scale 2D pretraining sequences within the model, evaluates their long-horizon while retaining essential 3D awareness. A third direction outcomes, and chooses the best plan. This deliberative ap- avoids explicit reconstruction altogether by relying on the proach, used in systems such as LUMOS [16], VLM-in-the- reasoning ability of large multimodal models. VoxPoser [113] Loop [144], and MinD [28], is particularly effective for tasks generates dense voxel-value maps from language-guided requiring long-term foresight and trade-offs. PREPRINT SUBMITTED TO IEEE TPAMI 8
4.1.4 Future Directions (2) Ambiguous Instructions. Another line of work fo- Summary & Trends: Current VLA architectures struggle cuses on endowing the model with deeper reasoning and with two fundamental disconnects, which existing methods interactive clarification capabilities. When facing ambiguous address via a patchwork strategy. (1) Regarding the Modal- commands, ThinkAct [149] infers and verifies the in- ity Disconnect, the prevailing trend relies on modular Late tended target via scene parsing and feedback, while Deep- Fusion, where separate encoders process inputs in isolation ThinkVLA [150] resolves ambiguity with causal chain-of- before concatenation. (2) Regarding the Physical Disconnect, thought and aligns subgoals with correct execution through researchers currently introduce auxiliary modules or rely on outcome-driven RL. When spatial information is underspec- state forecasting to approximate dynamics. However, these ified, InSpire [151] explicitly prompts the policy to answer approaches remain superficial: late fusion limits deep cross- “where is the target relative to the robot?” before acting, modal reasoning, while dynamic prediction often mimics thereby auto-filling missing cues. Taking this a step further, physics without understanding causality. AskToAct [152] trains an ambiguity-recognition module on Directions: To bridge these gaps, the field must simultane- synthetically incomplete queries and uses large-scale clari- ously advance toward Native Multimodal Architecture. This fication dialogues to teach the agent to proactively request means converting visual and physical data into tokens at missing details when an instruction is underspecified. the very beginning of training. By placing all modalities into the same language and shared space, the model does 4.2.2 Hierarchical Planning and Task Decomposition not need complex alignment steps. It can simply reason over all data types together, leading to a more natural and While many VLA frameworks are optimized for short- direct understanding of the physical world. An important horizon skills, executing long-horizon operations remains a next step is to develop a hybrid Latent-Physics-Semantic largely unresolved challenge [153]. Agents must decompose World Model. Such a model would internally represent 3D high-level instructions into structured subgoals to act ro- geometry, physical dynamics, semantic attributes and af- bustly. Pure end-to-end models, which directly map inputs fordances. Given vision, optional depth/point-cloud/tactile to low-level actions without explicit intermediate reasoning, input and a language instruction, the system encodes a often struggle with multi-step planning and compositional unified world state, simulates candidate future states (e.g., tasks [154], [155]. To address this, hierarchical decomposi- object motion, contact, stability, affordance changes), and tion is a dominant paradigm [15], [40], [85], [86], [156]. Based plans by reasoning jointly over semantics and physics. This on the type of intermediate representation they employ integration grounds high-level semantic intent in physics- to bridge high-level intent and low-level control, current aware simulation, helping to close the gap between semantic approaches can be broadly categorized into three families. understanding, perception, and physical interaction. (1) Language-Driven Planning. These methods adopt a modular hierarchical paradigm, leveraging language to decom- pose tasks in semantic space. π0.5 [53] embeds hierarchical 4.2 Instruction Following, Planning, and Robust Real- reasoning within a single inference chain: the model first Time Execution proposes explicit language-level sub-tasks from vision and Fig. 5 illustrates the four levels of this challenge, which are instructions, then conditions continuous control on these elaborated in detail below. sub-tasks. OneTwoVLA [157] performs structured textual reasoning at key decision points, generating scene descrip- 4.2.1 Parsing Complex Instructions tions, high-level plans, and next-step instructions, to decom- Task instructions for VLA are often multimodal and un- pose tasks within the semantic space. Hi Robot [41] em- derspecified, and failures in understanding propagate to ploys a two-layer scheme where a VLM parses instructions perception, planning, and control. We highlight two primary into atomic sub-tasks, and a VLA controller handles low- sources of difficulty: (i) Open-ended, multimodal instruction level execution. Other methods use end-to-end hierarchical forms. Instructions are no longer plain text, as they may paradigm, like LoHoVLA [158], using a common VLM back- mix language with images, scene cut-outs, internet photos, bone to jointly produce language sub-steps and continuous or hand-drawn sketches. (ii) Ambiguity and underspecifi- actions, enabling long-horizon reasoning without a strict cation. Commands like “help me” or “clean this up” omit planner-executor split. crucial task parameters (i.e., what, where, how, when). (2) Planning via Multimodal Intermediates. These methods (1) Open-Ended Instruction. To handle open-ended, mixed- perform planning via multimodal intermediates, using non- modality prompts, recent methods attempt to interleave linguistic representations like visual goals or affordances as images and text into a single sequence, and use the same the stepping stones for decomposition. On the vision-driven sequence modeling mechanism for understanding and con- side, CoT-VLA [78] employs pixel-level subgoal images as trol. OE-VLA [147] adopts a shared visual encoder for all explicit intermediates [78], while Embodied-SlotSSM [159] images and a text tokenizer for all text, converting them into employs slot-based [160] object-centric representations to token streams that are strictly interleaved to preserve the create structured visual intermediates. HiP [161] further original instruction order. Similarly, Interleave-VLA [148] in- extends this idea with a three-tier pipeline in which an LLM troduces special tags to its tokenizer, allowing image feature generates abstract subgoals, a video diffusion model pro- vectors to be seamlessly inserted within a text sequence. duces physically feasible visual trajectories, and an inverse These approaches enable the policy to understand non- dynamics model converts these trajectories into actions. On text instructions and improve direct cross-modal grounding the affordance-driven side, RT-Affordance [162] plans tasks without relying on standardized phrasing. by decomposing complex robotic manipulation into man- PREPRINT SUBMITTED TO IEEE TPAMI 9
Complex Instructions Hierarchical Planning Error Detection and Recovery Real-Time Execution
Instruction: Put the <IMG> in <IMG>
1. Put A Dynamic Optimization
Static Optimization
2. Put B
…… …
Transformer Transformer
Image Video text Language-Driven Turn left
Open-Ended, Mixed-Modalities Visual Affordance
…
VLA architecture
1. 1.
2. 2. … Human-in-the-Loop Correction
no … …
• If …, else, … If Transformer
yes
MultiModal
• Put A behind B
A
B
Action Optimization Paradigm Optimization
• Clean the room 1.
2.
…
…… Transformer Transformer
Ambiguous Skill Libraries Self-Correction
Fig. 5: The Challenge of Instruction Following, Planning, and Robust Real-Time Execution. The flow begins with Section 4.2.1, where the model understands what a human wants, even if the instructions are unclear or mixed with images. This understanding then moves to Section 4.2.2, where big goals are broken down into smaller, workable steps or plans. For the third step, Section 4.2.3, the robot carries out its plan, watches for problems, and fixes them if something goes wrong. Lastly, Section 4.2.4 is a rule for the whole process, requiring every step to happen quickly.
ageable affordance plans. CoA-VLA [163] internalizes an OneTwoVLA [157], for example, incorporates active human affordance chain at each step as an implicit planning signal. clarification as a key component, proactively querying for (3) Compositional Planning with Skill Libraries. These user input to resolve uncertainty before acting. methods decompose long-horizon tasks into reusable (2) Self-Correction. A more effective strategy is to enable the atomic skills and compose them into higher-level behav- model to autonomously detect anomalous states and cor- iors for efficient and interpretable task execution. For the rect them. Specifically, CorrectNav [170] enables self-recovery explicit skill usage, VLP [164] builds a fine-grained library without extra modules by iteratively collecting the model’s for data-efficient reuse of manipulation patterns. Agentic own error trajectories, automatically identifying deviations, Robot [165] derives a short, semantically clear subgoal and generating corrective actions and visual data to con- sequence from the library, decomposing a task into 2–5 tinuously fine-tune the model. Similarly, FPC-VLA [171] verifiable atomic steps prior to execution. RoboBrain [166] uses a VLM to assess the semantic appropriateness of key also employs a hierarchical paradigm that expands human- actions and, when necessary, generates natural language understandable abstract instructions into executable atomic feedback with corrective directions. Agentic Robot [165] fo- action sequences, achieving an intent–plan–action map- cuses on the architectural level, which achieves autonomous ping through the joint learning of data and models. Other correction via a standardized plan–act–verify closed loop: a works explore the emergence of implicit skills. For instance, vision–language validator dynamically assesses subgoal DexVLA [167] learns to automatically annotate semantic completion and, upon failure, triggers predefined recovery sub-steps within long-horizon action sequences through strategies, effectively suppressing error accumulation. temporal alignment. AgiBot World [168] serves as a transi- 4.2.4 Real-Time Execution and Computing Efficiency tion, using explicit skills during data collection but learning a policy that implicitly compresses high-dimensional control The powerful capabilities of VLA come at the cost of sub- into semantic latent action tokens, enabling the emergence stantial computational overhead. Yet, physical-world inter- of composable behaviors. action is highly sensitive to latency, especially in complex and long-horizon tasks. Bridging the compute-latency gap between model capability and real-time performance is thus 4.2.3 Error Detection and Autonomous Recovery critical to the practical deployment of VLA systems. To Long-horizon VLA deployments are inherently vulnerable address these issues, recent works focus on four directions: to execution interruptions, perception drift, and actuation (1) Static Optimization of Architecture. A line of work failures. Without timely, on-policy correction, small mis- focuses on static architectural optimization, which reduces takes can compound into cascading failures that derail the inherent computational complexity by refining the model’s entire task. To address this, research efforts have largely structure. A common solution is compression and quantiza- followed two main lines of inquiry: tion. BitVLA [172] achieves ultra-low-precision efficiency (1) Human-in–the-Loop Correction. These methods lever- via ternary 1-bit compression and distillation, while Evo- age a human user as an external source of intelligence to 1 [173] offers a similar lightweight design with only 77M guide recovery. This can be reactive, where the human pro- parameters. SQAP-VLA [174] introduces perceptual prun- vides corrective signals during execution. For instance, Yell ing strategies on the basis of quantization, and achieves At Your Robot [169] integrates real-time human language a nearly two times inference speedup and half memory feedback as corrective signals for immediate behavioral reduction. Besides, some methods directly adopt lightweight adjustment, while CLIP-RT [112] treats human language backbones, like NORA [175] and TinyVLA [176], while VLA- feedback as an ideal action template and embeds it into Adapter [177] introduces lightweight adapters to graft the decision process via similarity matching for efficient, knowledge from a large model onto a smaller policy net- retrain-free correction. This approach can also be proactive, work. Other approaches fundamentally replace the compu- where the agent solicits help when it detects ambiguity. tationally expensive Transformer attention mechanism with PREPRINT SUBMITTED TO IEEE TPAMI 10
linear attention. SARA-RT [178] converts high-cost Trans- so the model can take shortcuts at inference time. For in- former policies into linear-attention variants to cut inference stance, ECoT-Lite [194] uses reasoning traces during training delay. RoboMamba [71] replaces the Transformer with the but completely bypasses explicit reasoning steps during Mamba, attaining linear-time scaling and faster inference inference. V-JEPA 2 [143] reduces planning overhead by without explicit quantization or specialized accelerators. predicting compressed semantic representations instead of raw (2) Dynamic Optimization of Decoding Process and In- pixels. Meanwhile, Fast-in-Slow [195] employs an elegant ference Strategies. Beyond static architectural changes, this dual-system architecture within a single model, enabling tight line focuses on runtime adaptivity, dynamically adjusting coordination between slow, deliberate reasoning and fast, compute budgets during decoding and inference based on reactive execution. At the highest system level, some works task complexity, thereby reducing latency and computa- elevate optimization to the operating system or distributed tion while maintaining accuracy. One strategy is to cre- learning. For example, AMS [196] introduces OS-level action ate dynamic inference paths, which dynamically skip certain context caching and replay mechanisms, and FedVLA [197] computation layers or terminate inference early at shallow explores efficient distributed training of VLA models under depths, based on the complexity of the current input. For a federated learning framework. example, MoLe-VLA [179] leverages layer skipping to re- duce FLOPs, while CEED-VLA [180] and DeeR-VLA [26] 4.2.5 Future Directions design early exit mechanisms. Another is to perform dy- Summary & Trends: To handle complex tasks, the commu- namic token processing through token pruning or caching. nity is currently divided into rigid hierarchical systems (us- VLA-Cache [181] designs adaptive caching strategies that ing LLMs as high-level planners for code generation or sub- treat static and dynamic tokens differently. SpecPrune- goal decomposition) for long-horizon reasoning, or massive VLA [182] performs action-aware pruning conditioned on end-to-end policies via instruction tuning for reactive skills. history and current observations. CogVLA [183] also re- However, the former suffers from severe information loss duces computation through instruction-driven visual to- between modules, while the latter lacks the reasoning ca- ken sparsification. Furthermore, methods employ accelerated pability for multi-stage correction, resulting in open-loop decoding to overcome the sequential bottleneck of tradi- execution without introspection. tional approaches. For instance, Accelerating VLA [184] and Directions: Future architectures must break this dichotomy OpenVLA-OFT [33] generate an entire action chunk in a sin- by becoming Adaptive. Just like a human, the model should gle forward pass through parallel decoding. Spec-VLA [185] decide how much to think based on the task. For simple adopts speculative decoding to emit candidate action tokens tasks like grabbing a cup, it should act instantly. For com- in a single forward pass with relaxed acceptance. plex tasks like assembling furniture, it should automatically (3) Optimization of Action Representation and Generation activate deeper reasoning skills to plan steps. To do this, Paradigm. This type of method posits that the bottleneck one direction is to use Unified Decision Tokens. By treating in inference efficiency stems largely from how actions are seeing, thinking, and acting as a single stream of data, represented and generated. By rethinking and optimizing the model can naturally switch between fast action and action representations, efficiency can be fundamentally im- deep thought without needing separate, rigid modules. This proved. One strategy is efficient action tokenization, which creates a true end-to-end unified mind that handles both designs more compact and information-dense action to- simple reflexes and long-term planning. Beyond just acting kens to reduce the number of prediction steps. For ex- efficiently, robots need to change how they understand their ample, FAST [186] compresses action sequences to reduce own actions. Today’s robots are passive, i.e., they just follow training cost and wall time. XR-1 [187] leverages discrete instructions without asking why. Future VLA models must visual–motor representations learned by VQ-VAE [188] to evolve toward Self-Awareness. The goal is an agent that not guide policy learning, while VQ-VLA [189] extends this idea only knows what to do, but also understands why it is doing by using a VQ-VAE tokenizer to compress long trajecto- it. Models should shift from open-loop execution to closed- ries into a small set of discrete tokens. Another strategy loop resilient autonomy, dynamically switching between is asynchronous execution and inference, where the system replanning and reflex adjustment to autonomously recover predicts the next action chunk while the current one is from failures without intervention. being executed, as seen in SmolVLA [190] and Real-Time Action Chunking [191]. A third strategy focuses on acceler- ating diffusion policies by reducing the number of required 4.3 From Generalization to Continuous Adaptation sampling iterations. Time-Diffusion Policy [192] replaces Fig. 6 illustrates the four levels of this challenge, which are the traditional time-varying denoising process with a fixed, elaborated in detail below. direction-consistent unified velocity field. Discrete Diffu- sion VLA [193] discretizes actions into tokens and employs masked diffusion with parallel prediction, alleviating the 4.3.1 Open-World Generalization autoregressive decoding bottleneck. Despite strong cross-modal understanding and manipula- (4) Optimization of Training Paradigm and System. This tion in closed settings, large VLA models often generalize kind of work emphasizes the design of the training process poorly when deployed in open, dynamic real-world envi- and the implementation of the system to further reduce ronments. Conventional imitation learning relies on large the inference overhead and improve the execution effi- human-annotated datasets and fails to cover the long tail of ciency. A common principle among these approaches is real scenes. Therefore, achieving robust open-world gener- to leverage additional knowledge or data during training alization is a pivotal challenge. PREPRINT SUBMITTED TO IEEE TPAMI 11
Open-World Generalization Continual Learning Sim-to-Real Gap Reinforcement Learning
Forgetting Pretrained
In-context 𝜋&
Concept GAP VLA
Web A C A D D Guidance
Human generalization ……
…… B C C Simulation Psim
robot
Real-World P'()*
Knowledge Innovative Learning Optimization
Transfer Paradigm A Enhance Simulation
B
C + Overlap State&Reward
C A B
Open world Slow & Hard to
Memory unstable design
buffer
Psim → Psim ′ env
A Action
Data Adaptive
Diversity B Psim ′
Architecture 𝜋&
State
Reward
Hierarchical
Generative A C A
……
VLM env
B C B
…… Reward Generation
Isolation Replay Data-Driven Simulator
Fig. 6: The Challenge of From Generalization to Continuous Adaptation. This diagram illustrates how VLA models operate continuously in dynamic, open-world environments, highlighting four key enabling strategies. Section 4.3.1 represents the initial ability to perform well in settings not seen during training. Building on this, Section 4.3.2 focuses on how agents can continuously acquire new skills throughout their operational lifetime without forgetting old ones. Section 4.3.3 addresses the critical challenge of transferring learned policies from virtual environments to the physical world. Finally, Section 4.3.4 highlights how agents refine Security their behaviors and learn Interpretability from real-time Trustworthy experience. Interaction If danger No, I can Is this heavy？ I know why did it I must stop. do it. do that and what First，…… the next action. Second，…… (1) Knowledge Transfer and Utilization. The most dom- emergent compositionality, where methods like TRA [204] use ……
inant approach posits that the key to generalization lies a temporal contrastive loss to imbue the learned represen- not in learning from scratch, but in effectively transfer- tation space with a compositional structure, allowing the ring vast prior knowledge from large-scale data sources. model to automatically combine learned skills into new This is pursued in two main ways. Multi-task/multi-robot tasks. A more profound shift is toward conceptual gener- pretraining involves training on massive robotic datasets to alization, which moves beyond imitating actions to under- learn a general, hardware-agnostic prior over behaviors. standing semantic concepts. ObjectVLA [205] jointly trains For example, Octo [49] pretrains a Transformer on about on robot trajectories and box-labeled VL corpora to achieve 800k robot trajectories to acquire general manipulation reg- zero-shot manipulation of unseen objects, while LERF [206] ularities and uses lightweight adapters for efficient fine- fuses CLIP with 3D NeRFs for natural-language localization tuning, enabling rapid adaptation to new sensors and action and grasping of novel objects. Finally, to achieve robust spaces under limited data and compute. DexVLA [167] deployment, new adaptation paradigms emerge. Align-Then- introduces billion-parameter diffusion action experts that Steer [207] proposes a non-invasive adaptation method that pretrain across robot morphologies and adopts a three- steers a frozen VLA model’s outputs using a lightweight, stage curriculum to realize task-agnostic language–action latent-space adapter. Robot Utility Models (RUM) [208] pair mapping. RoboCat [198] pretrains on heterogeneous multi- large-scale home demonstrations with multimodal LLM robot data and continually improves on real trajectories reasoning for runtime verification and automatic retries, for sustained task transfer. Dita [199] leverages the large achieving zero-shot deployment in new environments. OXE dataset [100] and diffusion Transformers to learn cross- (3) Enhancing Data Diversity. Given the high cost of collect- environment behaviors, adapting with as few as 10 real ing real-world data, recent work expands the data distribu- demonstrations. EO-1 [200] further scales this paradigm tion using generative models and semantic priors to build by pretraining a shared backbone on the 1.5M-EO-Data large-scale, more diverse training data at low robot cost. For dataset to achieve knowledge transfer and enhance open- data augmentation, CACTI [209] scales multi-task imitation world understanding. The second method is internet/human by using Stable Diffusion for zero-shot inpainting of expert video knowledge transfer, which leverages data sources vastly images to increase layout and appearance diversity without larger than robotic datasets. Following CLIP [20], R3M [201] additional robot rollouts. GenAug [210] employs text-to- extends this paradigm to robotics by pretraining visual en- image synthesis conditioned on a few demonstrations and coders on massive collections of human first-person videos prompts to produce visually diverse yet functionally consis- (e.g., Ego4D [202]), thereby transferring general interaction tent scenes, improving robustness to unseen environment knowledge into robotic policies. In addition, the GR series shifts. For semantic augmentation, ROSIE [211] distills knowl- (e.g., GR-1 [67], GR-2 [68]) stands as a representative line edge from internet-scale VLMs into robot training, exposing of work in this direction that pre-train on massive human policies to richer semantic combinations and task variants egocentric video datasets to transfer general physical and to strengthen open-set generalization. interaction knowledge into robotic policies. (4) Adaptive Architectural Design. Beyond the above (2) Paradigm-Level Innovations. Beyond knowledge trans- approaches, the design of the model architecture itself pro- fer from pretrained models or web-scale data, a growing foundly influences its generalization capability. Specifically, body of work explores how models learn, not just what they hierarchical designs enhance generalization by decomposing learn, which is a key to achieving robust generalization. For tasks into high-level planning and low-level execution. The example, ICIL [203] follows the in-context learning paradigm high-level planner can leverage abstract knowledge learned that trains the model to infer tasks from a few demon- from large-scale data, while the low-level executor focuses strations provided in the prompt at test time, enabling on acquiring reusable skills [212]. Multimodal fusion frame- rapid, retrain-free adaptation. Another direction focuses on works that dynamically fuse multimodal sensor inputs can PREPRINT SUBMITTED TO IEEE TPAMI 12
significantly enhance robustness in complex environments, visual fidelity of the simulator’s rendering. ManiSkill3 [219] like BAKU [213]. Meanwhile, generative diversity methods leverages GPU-parallel rendering, domain randomization, like StructDiffusion [214] use language-guided diffusion and background composition to narrow the appearance to generate multiple physically plausible action structures gap and enable zero-shot transfer. Another alternative to instead of a single deterministic plan, improving robustness improving the simulation is to make the policy more robust to unseen object sizes and shapes. by learning a stable intermediate representation. SLIM [218], for instance, compresses high-dimensional RGB images into 4.3.2 Continual Learning and Incremental Skill Acquisition segmentation and depth maps, thereby filtering out task- An embodied agent’s learning process should not end at irrelevant visual differences between sim and real. deployment. It must continually acquire new skills through- (2) Data-driven Simulators. Recognizing that classical out its lifetime to adapt to evolving environments and user physics engines cannot fully capture real-world complex- needs. However, recent studies reveal a critical issue: as ity, a complementary line sidesteps explicit sim modeling new tasks are learned, the parameters supporting previ- by learning from or generating experiences using real- ously acquired skills are often overwritten, leading to sharp world data. One direction is generative augmentation on real- performance regressions and the erosion of multimodal world data, which attempts to expand a small set of real reasoning capabilities inherited from backbones [14], [62]. robot trajectories to enhance data diversity. For instance, To solve this, existing efforts broadly follow two routes. GenAug [210] leverages web-scale image generative models (1) Parameter Isolation and Expansion. These methods to synthesize visually diverse but functionally consistent allocate dedicated parameter space for new skills or adopt images from a few real robot demonstrations and semantic modular designs that safeguard existing weights, thereby prompts, bypassing simulators entirely by exploiting the fundamentally preventing weight conflicts between old model’s prior over real-world visuals to generate highly and new tasks and mitigating cross-task interference at realistic scenes. Another mainstream direction redefines its source. One prominent approach is Prompt-Based and physics-based simulation as data-driven prediction: it trains Codebook-Based Learning, which encodes skill knowledge into a powerful world model to learn physical dynamics and a set of discrete, composable prompts or codebook entries. causal relationships directly from massive amounts of real- When acquiring a new skill, the system simply adds a world data, such as DreamGen [220]. RynnVLA-001 [221] new prompt or codebook entry without modifying existing further advances this direction through large-scale video components [14], [215]. The other approach uses modular and generation pretraining combined with human-centric tra- expert-based architectures to isolate knowledge. For example, jectory perception modeling, enabling implicit transfer of InstructVLA [62] adopts a two-stage training paradigm and human manipulation skills to robotic control. a Mixture-of-Experts architecture to intelligently route be- tween reasoning and action modules, avoiding direct mod- 4.3.4 Online Interaction and Reinforcement Learning ification of its backbone. Similarly, the scalable PerceiverIO Imitation learning allows VLA models to quickly learn basic proposed in iManip [216] falls into this category by adding skills from offline data, but is limited by distributional shift new, skill-specific weights while freezing old ones. and a performance ceiling capped by human demonstrators. (2) Replay-based Knowledge Consolidation. Inspired by Reinforcement Learning (RL) addresses these by enabling human review, these methods rehearse a subset of past exam- autonomous exploration, yet its application to large VLA ples while learning new tasks to reinforce retained knowl- models in high-dimensional continuous action spaces is hin- edge. Since storing and replaying all historical data is im- dered by low sample efficiency [222]–[225] and the difficulty practical, the core challenge lies in intelligently selecting of designing effective rewards [36], [226], [227]. To tackle the most informative samples for replay. ExpReS-VLA [217] these challenges, researchers integrate RL with VLA models’ addresses this by introducing compressed experience replay strong priors, primarily through two directions: to mitigate catastrophic forgetting in robotic VLA systems, (1) Optimizing the Learning Process. Rather than letting RL while iManip [216] proposes a temporal replay strategy that explore from scratch, this approach injects or distills the rich avoids random sampling and instead replays critical frames knowledge and structural priors already learned by VLA during skill execution. models into the RL policy, addressing the slow and unstable nature of RL training. For knowledge transfer, RLDG [223] 4.3.3 Sim-to-Real Gap in Deployment first trains task-specialist RL policies, then distills their high- The sim-to-real gap remains a core obstacle for deploying quality trajectories into a general VLA, improving precise VLA policies, as discrepancies between simulated and real- control and generalization without fragile end-to-end RL world dynamics (e.g., friction, latency, actuation response) fine-tuning. Refined Policy Distillation [225] adds a sim- and perception (e.g., illumination, textures, sensor noise) ple MSE constraint so that VLA action distributions guide severely degrade policy transfer despite the low-cost, large- the RL agent, maintaining stability under sparse rewards scale data provided by simulators [218]. To address this and viewpoint changes. iRe-VLA [222] alternates phases: it challenge, researchers have explored a variety of strategies: freezes the large backbone and trains a lightweight action (1) Enhancing Simulation Fidelity and Robustness. The head during RL for stability; it then unfreezes and fine-tunes goal of this class of methods is to improve the direct with successful/expert trajectories under supervision to re- transferability of policies, either by making the simulation gain capacity. Beyond the above, some approaches optimize environment more closely resemble the real world or by the internal structure of RL algorithms. For example, CO- making the policy robust to the discrepancies between simu- RFT [224] designs a chunked temporal-difference learning lation and reality. A straightforward solution is to enhance the mechanism that feeds entire action sequences into the critic Open-World Generalization Continual Learning Sim-to-Real Gap Reinforcement Learning Forgetting Pretrained In-context 𝜋& Concept GAP VLA Web A C A D D Guidance Human generalization …… …… B C C Simulation Psim robot PREPRINT SUBMITTED TO IEEE TPAMI Real-World P’()* 13 Knowledge Innovative Learning Optimization Transfer Paradigm A Enhance Simulation to predict multi-step returns, aligning with VLA’s chunkedOpenvirtuous world closedC loop ofC “Deployment + A B → Discovery → Evolu- B Slow & State&Reward Overlap
Hard to
structure and significantly improving training stability and tion,” allowingA the systemMemory to buffer continuously refine its world unstable env Psim → Psim ′ Action design Data Adaptive sample efficiency under limited data. Diversity model and expand Architecture B its capabilities without human. 𝜋 Psim ′ & State Reward (2) Automating Reward Generation. Instead of costly hand- Hierarchical A C Generative A …… VLM env B C B crafted rewards or labor-intensive preference labels, recent …… 4.4 Safety, Interpretability and Reliable Interaction Reward Generation Isolation Replay Data-Driven Simulator work leverages VLM/LLM perception and reasoning to automatically derive dense, high-quality rewards directly Fig. 7 illustrates the two levels of this challenge, which are from observations and goals. One direction infers rewards elaborated in detail below. through perceptual alignment by measuring similarity be- Security Interpretability Trustworthy Interaction tween the current visual state and the goal description in No, I can Is this heavy？ If danger I know why did it
a shared embedding space. VLM-RMs [228] introduces this I must stop. do that and what the next action. First，…… Second，…… …… do it.
idea, and RoboCLIP [229] extends it to video trajectories by computing video–language similarity for sparse rewards. Affordance-Guided RL [230] converts VLM-predicted grasp Fig. 7: The challenge of Safety, Interpretability and Reliable Interac- This diagram shows how VLA systems build human trust, broken points and target trajectories into continuous dense rewards tion. into three key layers. Section 4.4.1 is about making sure the robot is that guide policy optimization. A second direction uses physically safe and works reliably. Moving up, Section 4.4.2 focuses VLMs as critics to rank trajectories or states rather than relying on helping humans understand why the robot makes certain decisions, on direct similarity scores. RL-VLM-F [231] employs GPT- making the robot’s actions easy to understand and predict, leading to smooth collaboration. 4V to compare observation pairs and infer preferences for training a reward function without human labels, while GRAPE [226] decomposes tasks and generates stage-wise preferences for structured, multi-objective rewards. A third 4.4.1 Reliability and Safety Assurance direction leverages LLMs’ zero-shot code generation and high- VLA models, particularly end-to-end deep learning sys- level reasoning to produce reward functions. Eureka [232] tems, usually lack transparency in their decision-making prompts an LLM with environment code and task speci- and exhibit unpredictable behavior. When deployed in fications to generate executable rewards, VIP [233] views unstructured, human-shared physical environments, they reward learning as implicit value optimization from video, may execute hazardous actions due to perception errors, and VLA-RL [36] fine-tunes a VLM into a structured pro- generalization failures, or misinterpretation of instructions, cess–reward model that transforms sparse feedback into potentially endangering humans, the environment, or them- next-action-token supervision. selves. Consequently, establishing a reliable and verifiable safety assurance mechanism is a critical prerequisite for the real-world deployment of VLA systems. To address this 4.3.5 Future Directions challenge, two directions are explored: Summary & Trends: To achieve generalization, the dom- (1) Constraint-Based Safety Paradigms. This paradigm in- inant approach currently hinges on Scaling Laws, i.e., ag- jects explicit rule systems, inside or outside the model, to gregating massive, heterogeneous datasets to train large- hard-bound the action space and avoid unsafe behaviors. scale transformers via passive imitation learning. While Specifically, applying rule-based explicit constraints is the most this has significantly improved task-level success rates on straightforward approach. AutoRT [234] introduces a robot seen distributions, models remain hardware-dependent and constitution via structured prompting to encode multi-level temporally static. They are frozen after training, lacking constraints for behavior bounding in the wild. Alternatively, the agency to actively explore or adapt to novel robot some works directly internalize safety constraints as an in- morphologies without extensive fine-tuning. tegral part of the model’s learning process. SafeVLA [79] Directions: To realize “GPT moment” in embodied intelli- explicitly models physically hazardous behaviors as a cost gence, the paradigm must shift from training fragmented, function within a constrained Markov decision process, robot-specific policies toward developing Morphology- where the training objective is to maximize task reward Agnostic Representations. By logically disentangling high- while ensuring the cumulative cost remains below a pre- level semantic planning from low-level proprioceptive con- defined safety threshold. trol, a unified brain can transfer manipulation skills across (2) Learning-based Alignment Paradigms. Since scenarios vastly different embodiments—from quadrupeds to hu- in the real world are highly complex and cannot be fully manoids—via lightweight, modular adapters. This would covered by a finite set of handcrafted rules, some methods enable true Zero-Shot Cross-Embodiment Transfer, where a aim to internalize a human-aligned safety intuition and judg- new robot is treated simply as a new peripheral for a ment, enabling models to proactively detect and avoid risks. universal policy. Furthermore, this generalization must be For example, Gemini Robotics [114] applies Constitutional sustained through time via Autonomous Open-Ended Evo- AI post-training on safety data, ensuring that policies fol- lution. We envision a shift from static training sets to a low human-centric principles and thereby internalize safety self-reinforcing data engine, where agents exhibit intrinsic intuition. Beyond passively adhering to predefined rules, motivation to act as curious explorers. By combining self- the model must actively assess the uncertainty and potential supervised exploration with online reinforcement learning, risks of the current situation and adapt its behavior accord- future VLAs will transition from passive imitators to active ingly. GPI [154] integrates confidence estimation, probabilis- learners, identifying their own knowledge gaps and gener- tic action generation, and language-guided backtracking to ating high-quality training data in the wild. This creates a pause, seek help, or replan under uncertainty. Furthermore, PREPRINT SUBMITTED TO IEEE TPAMI 14
RationalVLA [27] introduces a learnable refusal token to providing a layer of protection, these reactive measures reject unsafe/invalid commands, adding a rational safety are separated from the policy’s core decision process, often layer between high-level semantics and low-level control. failing to prevent intrinsic model hallucinations or confident but wrong actions in real-time. 4.4.2 Interpretability and Trustworthy Interaction Direction: To build truly trustworthy embodied agents, the Most VLA models follow the end-to-end deep learning field must evolve beyond imposing static safety rules to- paradigm, which is inherently a black box and offers little ward cultivating Intrinsic Uncertainty Awareness. In unstruc- mechanistic insight [235]. When a robot acts, its inability to tured open worlds, absolute safety cannot be guaranteed by explain its rationale to the user impedes debugging, erodes pre-defined constraints alone. Instead, future VLA models trust, and hinders efficient human-robot collaboration. Thus, require a System 2 reflective layer that actively estimates a core challenge for VLA systems is to make decision logic epistemic uncertainty, endowing the agent with a sense of more transparent and behavior more predictable. Research doubt. This enables a paradigm shift from reactive emer- efforts are therefore shifting toward two aspects: gency stops to Proactive Risk Aversion, where the agent au- (1) Enhancing Process Interpretability. The aim is to tonomously pauses to solicit human clarification or replans expose the model’s abstract neural states as explicit, when it detects ambiguity or potential hazard. Furthermore, human-understandable intermediates at each step of the trust relies on establishing Shared Mental Model through think–decide–act chain. Chain-of-thought reasoning is a well- intervention-ready transparency. Interpretability should not known approach to enhancing interpretability. The inter- be merely a post-hoc debugging tool, but an integral part mediate reasoning process can be expressed either in lin- of the execution loop. We envision agents that visualize guistic form or in visual form. For the former, Diffusion- their thought process, such as future trajectories, attention VLA [58] conditions a diffusion policy on natural-language heatmaps, or subgoal decompositions, before physical ac- reasoning, exposing step-wise intent. ECoT [236] outputs tion is taken. Crucially, this transparency must be action- editable step-by-step rationales that users can correct via able: it should empower users to not only anticipate robot language. For the latter, CoT-VLA [78] adds visual subgoal behavior but also intervene effectively. By allowing humans images to render intermediate plans observable. Moreover, to correct the robot’s reasoning chain via natural language or in hierarchical architectures, the intermediate instructions gen- gestures, we can close the loop of Interactive Safety, ensuring erated by the high-level planner inherently serve as a natural that VLA systems are not just compliant, but genuinely source of interpretability. For example, RT-H [38] separates aligned with human intent. language–action generation from execution, enabling self- explanation and language-level intervention. HiRobot [41] 4.5 Data Construction and Benchmarking Standards outputs readable low-level commands from a high-level Fig. 8 illustrates the two levels of this challenge, which are planner, making task decomposition transparent. GraSP- elaborated in detail below. VLA [237] explicitly converts visual inputs into symbolic states and performs planning in this symbolic space, making Dataset Evaluation its intermediate process inherently interpretable. Besides, Limitations Limitations
recent efforts aim to decode the internal, hidden symbolic sim vs. real different
states from trained, black-box VLA models. A representative Heterogeneity Different standards Failing to test advanced task Failing to probe generalization
work is DIARC-OpenVLA [238], which trains linear probes Representation-Level Data-Level Comprehensiveness and Standardization Expanding the Breadth and Depth of Tasks on hidden layers to explicitly map neural activations to G generative data augmentation Diverse tasks symbolic states, providing a monitorable layer of decision Sample Breadth transparency without altering the original model. shared action space Long-horizon (2) Behavioral Predictability. Beyond explaining why a Action data mixture optimization Depth planning
Standardization
decision is made, it is equally important to design robot More challenging tests
behaviors that are inherently intuitive and aligned with hu- …
man expectations, thereby fostering trust directly through shared semantic space
interaction. CrayonRobo [239] externalizes the model’s in- Standardization Data Simple Hard
ternal decision logic using structured, semantically explicit visual prompts, creating a shared, interpretable language Fig. 8: The challenge of Data Construction and Benchmarking Stan- dards. Section 4.5.1 addresses the critical bottleneck of acquiring and that lets humans intuitively understand and even design the unifying diverse training resources to construct large-scale datasets. prompts for deeper collaboration. Another critical aspect is Section 4.5.2 focuses on the standardization and increasing complexity predictable responses to dynamic instructions. SwitchVLA [240] of assessment protocols. introduces structured task switching: upon mid-execution instruction changes, the agent rolls back conflicting actions before smoothly transitioning to the new goal, yielding 4.5.1 Multi-Source Heterogeneous Data natural, predictable behavior in open-ended interaction. The capabilities and generalization of VLA models are fun- damentally constrained by the scale, diversity, and qual- 4.4.3 Future Directions ity of their training data. However, acquiring and unify- Summary & Trends: Currently, safety is predominantly ing high-quality, large-scale, and diverse data presents a handled by extrinsic guardrails (e.g., rule-based shields or formidable challenge, primarily due to the inherent hetero- constitution-based filtering like AutoRT) or post-hoc ratio- geneity of data sources (e.g., sim vs. real, different robot nalization (prompting VLMs to caption their actions). While embodiments) and their respective control interfaces. To PREPRINT SUBMITTED TO IEEE TPAMI 15
address this, the research community systematically initiates laboratories. Another effort involves collecting and aligning explorations across three interconnected levels: human-centric and multi-view data, which is necessary for (1) Representation-Level Unification and Alignment. The robots operating in human environments. Representative core idea here is to model heterogeneous data within a examples include Ego4D [202] and EPIC-KITCHENS [252], shared, semantically consistent latent space, thereby elim- with Ego-Exo4D [253] further integrating first- and third- inating heterogeneity at the cognitive level rather than di- person viewpoints to support learning skilled activities rectly handling raw discrepancies. This is achieved through from multiple perspectives. The broader ambition is cross- two complementary strategies. The first aligns behaviors domain standardization, where heterogeneous datasets are in a latent action space by learning a unified discrete repre- aligned at scale to form unified fusion benchmarks. Open sentation that maps continuous, high-dimensional motions X-Embodiment (OXE) [100] marks a major milestone by from different robots or human videos into semantically aggregating dozens of datasets into a single benchmark for consistent action tokens. This filters out low-level control cross-embodiment generalization. RoboMM [132] advances differences and aligns behaviors at a higher semantic level. it through a three-level semantic alignment framework that LAPA [241], Moto [242], and UniVLA [35] learn such task- enables joint training across multiple datasets. centric latent action representations through unsupervised or self-supervised video learning. A more holistic strategy constructs a shared semantic space across all modalities and 4.5.2 Evaluation Benchmarks embodiments, extending beyond action correspondence to unify perception, reasoning, and control. RDT-1B [23] and Standardized benchmarks play a pivotal role in embod- AgiBot World [168] map diverse robot actions into unified ied intelligence by establishing common evaluation proto- physical or latent vectors, while Scaling Cross-Embodied cols that enable fair comparison and reproducible research. Learning [243] tokenizes heterogeneous visual and propri- However, as VLA models advance rapidly, the yardsticks oceptive inputs for a shared Transformer to handle multi- used to measure them struggle to keep pace, revealing ple morphologies. At the multimodal level, methods such several critical limitations [254]. First, a lack of unified as RT-1 [99], GR-2 [68], ViSA-Flow [244], and Humanoid- standards in metrics and experimental setups makes fair VLA [101] achieve consistent VLA grounding through uni- comparison difficult. Second, many existing benchmarks fied tokenization, semantic alignment, or self-supervised are limited to simple, short-horizon tasks, failing to test learning. Human-to-robot transfer approaches, including advanced cognitive reasoning. Third, they often lack a sys- EgoVLA [245] and DexWild [246], further align human tematic way to probe frontier generalization capabilities. To and robot motion using MANO hand models and inverse address these gaps, the community actively develops a new kinematics, enabling cross-domain embodied transfer. generation of benchmarks and evaluations. (2) Data-Level Augmentation and Optimization. Rather A primary direction of this effort is the pursuit of than altering the model’s latent space, this line of work comprehensiveness and standardization. The work on Bench- directly operates on raw data. The first strategy, generative marking VLAs [255] provides a blueprint by emphasizing data augmentation, creates expanded data distributions using unified I/O, metrics, and multi-robot coverage, shifting large pretrained generative models. This substantially in- the focus from tasks to metrics. EUQ [256] introduces a creases visual diversity at low cost and improves robustness human-assessed, multi-dimensional scoring system to cap- to appearance variations in heterogeneous real-world data. ture process quality beyond binary success. At the infras- CACTI [209] and GenAug [210] augment robot data via tructure level, simulation platforms like ManiSkill3 [219] inpainting or restyling, ROSIE [211] enriches data at the and robosuite [257] contribute standardized APIs and task semantic level using VLM priors, and Models with Data suites, providing a reproducible foundation for fair and Generation via Residual RL [247] generate additional sam- scalable evaluation. A second major direction focuses on ples through RL to further strengthen downstream VLA expanding the breadth and depth of tasks to assess more com- performance. The second strategy, automated data mixture plex capabilities. CALVIN [258] is designed to require the optimization, focuses on making better use of existing hetero- execution of long-horizon sequences of language-guided geneous datasets by treating data fusion as an optimization operations. LIBERO [259] is introduced as the first bench- problem. Re-Mix [248] adjusts sampling weights of het- mark specifically for lifelong learning in robotics, featuring erogeneous data subsets based on performance feedback, standardized metrics for knowledge transfer and forgetting. enabling the model to focus on informative samples and Furthermore, Ego-Exo4D [253] pioneers the synchronization achieve efficient cross-domain fusion. of first- and third-person recordings for multi-perspective (3) Standardization and Benchmark Construction. This skill analysis. Finally, a third direction aims to design more line of work reduces the heterogeneity of the data at the challenging tests that focus on frontier generalization and source by establishing standardized data collection proto- reasoning capabilities. The From Intention to Execution [260] cols, synchronization mechanisms, and unified benchmarks. suite is introduced to probe the intention-execution gap A major focus is unified acquisition and synchronization within and systematically covers challenges in object diversity, individual datasets to ensure high quality and internal linguistic complexity, and visual-language reasoning. To consistency. RH20T [118] enforces strict temporal alignment specifically assess the abilities of instruction-tuned models, across multimodal sensors, and BridgeData V2 [249] orga- InstructVLA [62] releases the SimplerEnv-Instruct bench- nizes diverse data types into a standardized format. In simu- mark, a comprehensive suite of 80 zero-shot tasks featuring lation, RoboCasa [250] and CoVLA [251] provide large-scale, multilingual expressions, novel objects, and implicit inten- high-fidelity environments that act as standardized digital tions to evaluate contextual reasoning and generalization. PREPRINT SUBMITTED TO IEEE TPAMI 16
4.5.3 Future Directions [6] R. Sapkota, Y. Cao, K. I. Roumeliotis, and M. Karkee, “Vision- language-action models: Concepts, progress, applications and Summary & Trends: Driven by the pursuit of scaling challenges,” arXiv:2505.04769, 2025. laws, the field is currently preoccupied with aggregating [7] R. Shao et al., “Large vlm-based vision-language-action models massive, heterogeneous real-world datasets to fuel models. for robotic manipulation: A survey,” arXiv:2508.13073, 2025. On the evaluation front, the standard remains simplistic, [8] K. Kawaharazuka, J. Oh, J. Yamada, I. Posner, and Y. Zhu, “Vision-language-action models for robotics: A review towards relying heavily on binary success rates in controlled settings. real-world applications,” IEEE Access, 2025. However, real-world collection is inherently unscalable and [9] Y. Ma, Z. Song, Y. Zhuang, J. Hao, and I. King, “A noisy, and binary metrics fail to capture the nuances of survey on vision-language-action models for embodied ai,” robustness, often masking critical failure modes. arXiv:2405.14093, 2024. [10] K. O’shea and R. Nash, “An introduction to convolutional neural Directions: To scale embodied intelligence, the field networks,” arXiv:1511.08458, 2015. must transition towards a Simulation-First, Failure-Centric [11] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning Paradigm. Relying solely on real-world data is unscalable; for image recognition,” in Proceedings of the IEEE conference on instead, we envision Simulated Universes acting as infinite computer vision and pattern recognition, 2016, pp. 770–778. [12] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for data factories that generate diverse, labeled trajectories with convolutional neural networks,” in International conference on perfect ground truth. The core challenge will be bridging machine learning. PMLR, 2019, pp. 6105–6114. the Sim-to-Real gap for perception and physics, allowing [13] C. Chi et al., “Diffusion policy: Visuomotor policy learning via real-world data to serve efficiently as a high-quality align- action diffusion,” The International Journal of Robotics Research, vol. 44, no. 10-11, pp. 1684–1704, 2025. ment set to calibrate the simulator’s physics and render- [14] J. Xu and X. Nie, “Speci: Skill prompts based hierarchical contin- ing fidelity, rather than being the primary training source. ual imitation learning for robot manipulation,” arXiv:2504.15561, Equally important is a shift in how we treat errors. Current 2025. pipelines often discard failed trajectories, wasting critical [15] J. Zhang et al., “Hirt: Enhancing robotic control with hierarchical robot transformers,” arXiv:2410.05273, 2024. information. Future systems must Turn Failure into Signal, [16] I. Nematollahi, B. DeMoss, A. L. Chandra, N. Hawes, W. Burgard, treating mistakes as gold mines for negative mining and and I. Posner, “Lumos: Language-conditioned imitation learning contrastive learning. By explicitly training on what not to with world models,” 2025. do and diagnosing why failures occur, agents can learn not [17] D. Hafner et al., “Learning latent dynamics for planning from pixels,” in International conference on machine learning. PMLR, only to avoid risks but also to autonomously recover from 2019, pp. 2555–2565. inevitable execution errors. Finally, evaluation must evolve [18] A. Dosovitskiy, “An image is worth 16x16 words: Transformers from simple binary success rates to Comprehensive Diagnostic for image recognition at scale,” arXiv:2010.11929, 2020. Stress Testing. Benchmarks should utilize high-fidelity sim- [19] Y. Haruna, S. Qin, A. H. Adama Chukkol, A. A. Yusuf, I. Bello, and A. Lawan, “Exploring the synergies of hybrid convolu- ulation proxies to assess holistic capabilities—quantifying tional neural network and vision transformer architectures for not only task completion but also safety margins, efficiency, computer vision: A survey,” Engineering Applications of Artificial and resilience to perturbations—thereby prioritizing robust Intelligence, vol. 144, p. 110057, Mar. 2025. adaptability over rote execution of memorized trajectories. [20] A. Radford et al., “Learning transferable visual models from nat- ural language supervision,” in International conference on machine learning. PmLR, 2021, pp. 8748–8763. [21] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss 5 CONCLUSION for language image pre-training,” in Proceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 11 975–11 986. This survey presents a comprehensive anatomy of Vision- [22] K. Black et al., “π 0: A visionlanguage-action flow model for Language-Action (VLA) models, structured to guide readers general robot control, 2024a,” arXiv:2410.24164, 2024. from basic modules and historical milestones to the core [23] S. Liu et al., “Rdt-1b: a diffusion foundation model for bimanual challenges at the research frontier. We provide a detailed manipulation,” arXiv:2410.07864, 2024. [24] Z. Liu, Y. Gu, S. Zheng, X. Xue, and Y. Fu, “Trivla: A unified triple- analysis of the five key problem areas: representation, execu- system-based unified vision-language-action model for general tion, generalization, safety, and dataset evaluation, review- robot control,” arXiv:2507.01424, 2025. ing current solutions, and highlighting future opportunities [25] J. Yu et al., “Forcevla: Enhancing vla models with a force-aware for each. We hope that this work serves as a foundational moe for contact-rich manipulation,” arXiv:2505.22159, 2025. [26] Y. Yue et al., “Deer-vla: Dynamic inference of multimodal large roadmap, helping both newcomers and experienced re- language models for efficient robot execution,” Advances in Neural searchers navigate and advance the rapidly evolving field Information Processing Systems, vol. 37, pp. 56 619–56 643, 2024. of embodied intelligence. [27] W. Song et al., “Rationalvla: A rational vision-language-action model with dual system,” arXiv:2506.10826, 2025. [28] L. Wang, R. Shelim, W. Saad, and N. Ramakrishnan, “Dmwm: Dual-mind world model with long-term imagination,” R EFERENCES arXiv:2502.07591, 2025. [1] Y. Zhong et al., “A survey on vision-language-action models: An [29] H. Huang et al., “Otter: A vision-language-action model with action tokenization perspective,” arXiv:2507.01925, 2025. text-aware visual feature extraction,” arXiv:2503.03734, 2025. [2] W. Guan, Q. Hu, A. Li, and J. Cheng, “Efficient vision-language- [30] M. Oquab et al., “Dinov2: Learning robust visual features without action models for embodied manipulation: A systematic survey,” supervision,” arXiv:2304.07193, 2023. arXiv:2510.17111, 2025. [31] Y. Li, Y. Wang, Y. Fu, D. Ru, Z. Zhang, and T. He, “Unified lexical [3] T.-Y. Xiang et al., “Parallels between vla model post-training representation for interpretable visual-language alignment,” Ad- and human motor learning: Progress, challenges, and trends,” vances in Neural Information Processing Systems, vol. 37, pp. 1141– arXiv:2506.20966, 2025. 1161, 2024. [4] D. Zhang et al., “Pure vision language action (vla) models: A [32] M. J. Kim et al., “Openvla: An open-source vision-language- comprehensive survey,” arXiv:2509.19012, 2025. action model,” arXiv:2406.09246, 2024. [5] M. U. Din, W. Akram, L. S. Saoud, J. Rosell, and I. Hussain, [33] M. J. Kim, C. Finn, and P. Liang, “Fine-tuning vision-language- “Vision language action models in robotic manipulation: A sys- action models: Optimizing speed and success,” arXiv:2502.19645, tematic review,” arXiv:2507.10672, 2025. 2025. PREPRINT SUBMITTED TO IEEE TPAMI 17
[34] S. Deng et al., “Graspvla: a grasping foundation model pre- [62] S. Yang et al., “Instructvla: Vision-language-action instruction trained on billion-scale synthetic action data,” arXiv:2505.03233, tuning from understanding to manipulation,” arXiv:2507.17520, 2025. 2025. [35] Q. Bu et al., “Univla: Learning to act anywhere with task-centric [63] Z. Zhong et al., “Flowvla: Thinking in motion with a visual chain latent actions,” arXiv:2505.06111, 2025. of thought,” arXiv, pp. arXiv–2508, 2025. [36] G. Lu et al., “Vla-rl: Towards masterful and general [64] D. Lim, M.-J. Kim, J. Cha, D. Kim, and J. Park, “Proprioceptive robotic manipulation with scalable reinforcement learning,” external torque learning for floating base robot and its applica- arXiv:2505.18719, 2025. tions to humanoid locomotion,” in 2023 IEEE/RSJ International [37] X. Chen et al., “Pali-x: On scaling up a multilingual vision and Conference on Intelligent Robots and Systems (IROS). IEEE, 2023, language model,” arXiv:2305.18565, 2023. pp. 8510–8517. [38] S. Belkhale et al., “Rt-h: Action hierarchies using language,” [65] Z. Zhang et al., “Ta-vla: Elucidating the design space of torque- arXiv:2403.01823, 2024. aware vision-language-action models,” arXiv:2509.07962, 2025. [39] L. Beyer et al., “Paligemma: A versatile 3b vlm for transfer,” [66] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville, arXiv:2407.07726, 2024. “Film: Visual reasoning with a general conditioning layer,” in [40] H. Song et al., “Hume: Introducing system-2 thinking in visual- Proceedings of the AAAI conference on artificial intelligence, vol. 32, language-action model,” arXiv:2505.21432, 2025. no. 1, 2018. [41] L. X. Shi et al., “Hi robot: Open-ended instruction following with [67] H. Wu et al., “Unleashing large-scale video generative pre- hierarchical vision-language-action models,” arXiv:2502.19417, training for visual robot manipulation,” arXiv:2312.13139, 2023. 2025. [68] C.-L. Cheang et al., “Gr-2: A generative video-language-action [42] X. Chu et al., “Qwen look again: Guiding vision-language reason- model with web-scale knowledge for robot manipulation,” ing models to re-attention visual information,” arXiv:2505.23558, arXiv:2410.06158, 2024. 2025. [69] S. Reed et al., “A generalist agent,” arXiv:2205.06175, 2022. [43] C. Zhang, P. Hao, X. Cao, X. Hao, S. Cui, and S. Wang, “Vtla: [70] Y. Jiang et al., “Vima: General robot manipulation with multi- Vision-tactile-language-action model with preference learning for modal prompts,” arXiv:2210.03094, vol. 2, no. 3, p. 6, 2022. insertion manipulation,” arXiv:2505.09577, 2025. [71] J. Liu et al., “Robomamba: Efficient vision-language-action model [44] P. Chen et al., “Combatvla: An efficient vision-language-action for robotic reasoning and manipulation,” Advances in Neural model for combat tasks in 3d action role-playing games,” Information Processing Systems, vol. 37, pp. 40 085–40 110, 2024. arXiv:2503.09527, 2025. [72] A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with [45] C. Cui et al., “Openhelix: A short survey, empirical analysis, and selective state spaces,” ArXiv, vol. abs/2312.00752, 2023. open-source dual-system vla model for robotic manipulation,” [73] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic arXiv:2505.03912, 2025. models,” Advances in neural information processing systems, vol. 33, [46] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre- pp. 6840–6851, 2020. training of deep bidirectional transformers for language under- [74] Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le, standing,” in Proceedings of the 2019 conference of the North Amer- “Flow matching for generative modeling,” arXiv:2210.02747, ican chapter of the association for computational linguistics: human 2022. language technologies, volume 1 (long and short papers), 2019, pp. [75] Y. Chen, S. Tian, S. Liu, Y. Zhou, H. Li, and D. Zhao, “Conrft: 4171–4186. A reinforced fine-tuning method for vla models via consistency [47] C. Raffel et al., “Exploring the limits of transfer learning with policy,” arXiv:2502.05450, 2025. a unified text-to-text transformer,” Journal of machine learning [76] B. Zitkovich et al., “Rt-2: Vision-language-action models transfer research, vol. 21, no. 140, pp. 1–67, 2020. web knowledge to robotic control,” in Conference on Robot Learn- [48] S. Wang, S. Liu, W. Wang, J. Shan, and B. Fang, “Robobert: ing. PMLR, 2023, pp. 2165–2183. An end-to-end multimodal robotic manipulation model,” [77] D. Driess et al., “Palm-e: An embodied multimodal language arXiv:2502.07837, 2025. model,” 2023. [49] O. M. Team et al., “Octo: An open-source generalist robot policy,” [78] Q. Zhao et al., “Cot-vla: Visual chain-of-thought reasoning for arXiv:2405.12213, 2024. vision-language-action models,” in Proceedings of the Computer [50] H. Touvron et al., “Llama 2: Open foundation and fine-tuned chat Vision and Pattern Recognition Conference, 2025, pp. 1702–1713. models,” arXiv:2307.09288, 2023. [79] B. Zhang et al., “Safevla: Towards safety alignment of vision- [51] W. Dai et al., “Instructblip: Towards general-purpose vision- language-action model via safe reinforcement learning,” arXiv, language models with instruction tuning,” Advances in neural pp. arXiv–2503, 2025. information processing systems, vol. 36, pp. 49 250–49 267, 2023. [80] J. Cen et al., “Worldvla: Towards autoregressive action world [52] G. Team et al., “Gemma: Open models based on gemini research model,” arXiv:2506.21539, 2025. and technology,” arXiv:2403.08295, 2024. [81] R. Zheng et al., “Tracevla: Visual trace prompting enhances [53] Z. Zhou, Y. Zhu, J. Wen, C. Shen, and Y. Xu, “Vision-language- spatial-temporal awareness for generalist robotic policies,” action model with open-world embodied reasoning from pre- arXiv:2412.10345, 2024. trained knowledge,” arXiv:2505.21906, 2025. [82] C. Li, J. Wen, Y. Peng, Y. Peng, F. Feng, and Y. Zhu, “Pointvla: [54] G. Team et al., “Gemma 2: Improving open language models at a Injecting the 3d world into vision-language-action models,” practical size,” arXiv:2408.00118, 2024. arXiv:2503.07511, 2025. [55] Z. Cai et al., “Internlm2 technical report,” arXiv:2403.17297, 2024. [83] H. Zhen et al., “3d-vla: A 3d vision-language-action generative [56] X. Li et al., “Vision-language foundation models as effective robot world model,” arXiv:2403.09631, 2024. imitators,” arXiv:2311.01378, 2023. [84] P. Li et al., “Bridgevla: Input-output alignment for effi- [57] A. Awadalla et al., “Openflamingo: An open-source frame- cient 3d manipulation learning with vision-language models,” work for training large autoregressive vision-language models,” arXiv:2506.07961, 2025. arXiv:2308.01390, 2023. [85] B. Han, J. Kim, and J. Jang, “A dual process vla: Efficient robotic [58] J. Wen et al., “Diffusion-vla: Generalizable and inter- manipulation leveraging vlm,” arXiv:2410.15549, 2024. pretable robot foundation model via self-generated reasoning,” [86] Y. Li et al., “Hamster: Hierarchical action models for open-world arXiv:2412.03293, 2024. robot manipulation,” arXiv:2502.05485, 2025. [59] H. Shi et al., “Memoryvla: Perceptual-cognitive memory [87] Y. Guo et al., “Improving vision-language-action model with in vision-language-action models for robotic manipulation,” online reinforcement learning,” arXiv:2501.16664, 2025. arXiv:2508.19236, 2025. [88] P. Anderson et al., “Vision-and-language navigation: Interpreting [60] S. Karamcheti, S. Nair, A. Balakrishna, P. Liang, T. Kollar, and visually-grounded navigation instructions in real environments,” D. Sadigh, “Prismatic vlms: Investigating the design space of in Proceedings of the IEEE conference on computer vision and pattern visually-conditioned language models,” in Forty-first International recognition, 2018, pp. 3674–3683. Conference on Machine Learning, 2024. [89] A. Das, S. Datta, G. Gkioxari, S. Lee, D. Parikh, and D. Batra, [61] B. Xie et al., “Dexbotic: Open-source vision-language-action tool- “Embodied question answering,” in Proceedings of the IEEE con- box,” arXiv:2510.23511, 2025. ference on computer vision and pattern recognition, 2018, pp. 1–10. PREPRINT SUBMITTED TO IEEE TPAMI 18
[90] M. Chevalier-Boisvert et al., “Babyai: A platform to study [117] J. Jones, O. Mees, C. Sferrazza, K. Stachowicz, P. Abbeel, the sample efficiency of grounded language learning,” and S. Levine, “Beyond sight: Finetuning generalist robot arXiv:1810.08272, 2018. policies with heterogeneous sensors via language grounding,” [91] X. Wang et al., “Reinforced cross-modal matching and self- arXiv:2501.04693, 2025. supervised imitation learning for vision-language navigation,” [118] H.-S. Fang et al., “Rh20t: A comprehensive robotic dataset for in Proceedings of the IEEE/CVF conference on computer vision and learning diverse skills in one-shot,” arXiv:2307.00595, 2023. pattern recognition, 2019, pp. 6629–6638. [119] J. Bi, K. Y. Ma, C. Hao, M. Z. Shou, and H. Soh, “Vla-touch: [92] E. Wijmans et al., “Embodied question answering in photorealis- Enhancing vision-language-action models with dual-level tactile tic environments with point cloud perception,” in Proceedings of feedback,” arXiv:2507.17294, 2025. the IEEE/CVF Conference on Computer Vision and Pattern Recogni- [120] Z. Cheng et al., “Omnivtla: Vision-tactile-language-action model tion, 2019, pp. 6659–6668. with semantic-aligned tactile sensing,” arXiv:2508.08706, 2025. [93] M. Shridhar et al., “Alfred: A benchmark for interpreting [121] J. Huang, S. Wang, F. Lin, Y. Hu, C. Wen, and Y. Gao, “Tactile-vla: grounded instructions for everyday tasks,” in Proceedings of the Unlocking vision-language-action model’s physical knowledge IEEE/CVF conference on computer vision and pattern recognition, for tactile generalization,” arXiv:2507.09160, 2025. 2020, pp. 10 740–10 749. [122] R. Wang et al., “The sound of simulation: Learning multimodal [94] M. Shridhar, X. Yuan, M.-A. Côté, Y. Bisk, A. Trischler, and sim-to-real robot policies with generative audio,” in 9th Annual M. Hausknecht, “Alfworld: Aligning text and embodied envi- Conference on Robot Learning, 2025. ronments for interactive learning,” arXiv:2010.03768, 2020. [123] X. Pang et al., “Depth helps: Improving pre-trained rgb-based [95] S. Srivastava et al., “Behavior: Benchmark for everyday house- policy with depth information injection. in 2024 ieee,” in RSJ hold activities in virtual, interactive, and ecological environ- International Conference on Intelligent Robots and Systems (IROS), ments,” in Conference on robot learning. PMLR, 2022, pp. 477–490. 2024, pp. 7251–7256. [96] M. Shridhar, L. Manuelli, and D. Fox, “Cliport: What and where [124] S. Wang, “Roboflamingo-plus: Fusion of depth and rgb percep- pathways for robotic manipulation,” in Conference on robot learn- tion with vision-language models for enhanced robotic manipu- ing. PMLR, 2022, pp. 894–906. lation,” arXiv:2503.19510, 2025. [97] M. Ahn et al., “Do as i can, not as i say: Grounding language in [125] L. Sun, B. Xie, Y. Liu, H. Shi, T. Wang, and J. Cao, “Geovla: Em- robotic affordances,” arXiv:2204.01691, 2022. powering 3d representations in vision-language-action models,” [98] W. Huang et al., “Inner monologue: Embodied reasoning through arXiv:2508.09071, 2025. planning with language models,” arXiv:2207.05608, 2022. [126] J. Huang et al., “An embodied generalist agent in 3d world,” [99] A. Brohan et al., “Rt-1: Robotics transformer for real-world control arXiv:2311.12871, 2023. at scale,” arXiv:2212.06817, 2022. [127] R. Yang, G. Chen, C. Wen, and Y. Gao, “Fp3: A 3d foundation [100] Q. Vuong et al., “Open x-embodiment: Robotic learning datasets policy for robotic manipulation,” arXiv:2503.08950, 2025. and rt-x models,” in Towards Generalist Robots: Learning Paradigms [128] Z. Qi et al., “Sofar: Language-grounded orientation bridges spa- for Scalable Skill Acquisition@ CoRL2023, 2023. tial reasoning and object manipulation,” arXiv:2502.13143, 2025. [101] P. Ding et al., “Humanoid-vla: Towards universal humanoid [129] X. Xu et al., “Weakly-supervised 3d visual grounding based on control with visual integration,” arXiv:2502.14795, 2025. visual language alignment,” IEEE Transactions on Multimedia, [102] J. Bjorck et al., “Gr00t n1: An open foundation model for general- 2025. ist humanoid robots,” arXiv:2503.14734, 2025. [130] Y. Li, G. Yan, A. Macaluso, M. Ji, X. Zou, and X. Wang, “In- [103] A. Azzolini et al., “Cosmos-reason1: From physical common tegrating lmm planners and 3d skill policies for generalizable sense to embodied reasoning,” arXiv:2503.15558, 2025. manipulation,” arXiv:2501.18733, 2025. [104] G. A. Team, “Gen-0: Embodied foundation models that [131] J. Wei, S. Yuan, P. Li, Q. Hu, Z. Gan, and W. Ding, “Occllama: scale with physical interaction,” Generalist AI Blog, 2025, An occupancy-language-action generative world model for au- https://generalistai.com/blog/preview-uqlxvb-bb.html. tonomous driving,” arXiv:2409.03272, 2024. [105] W. Li et al., “Semanticvla: Semantic-aligned sparsification and en- [132] F. Yan et al., “Robomm: All-in-one multimodal large model for hancement for efficient robotic manipulation,” arXiv:2511.10518, robotic manipulation,” arXiv:2412.07215, 2024. 2025. [133] D. Niu et al., “Pre-training auto-regressive robotic models with [106] Y. J. Ma, V. Kumar, A. Zhang, O. Bastani, and D. Jayaraman, 4d representations,” arXiv:2502.13142, 2025. “Liv: Language-image representations and rewards for robotic [134] D. Qu et al., “Spatialvla: Exploring spatial representations for control,” in International Conference on Machine Learning. PMLR, visual-language-action model,” arXiv:2501.15830, 2025. 2023, pp. 23 301–23 320. [135] T. Lin et al., “Evo-0: Vision-language-action model with implicit [107] J. Bi, L. B. Wen, Z. Liu, and C. Xu, “Actllm: Action consistency spatial understanding,” arXiv:2507.00416, 2025. tuned large language model,” arXiv:2506.21250, 2025. [136] J. Wang, M. Chen, N. Karaev, A. Vedaldi, C. Rupprecht, and [108] Y. Hu, F. Lin, T. Zhang, L. Yi, and Y. Gao, “Look before you D. Novotny, “Vggt: Visual geometry grounded transformer,” in leap: Unveiling the power of gpt-4v in robotic vision-language Proceedings of the Computer Vision and Pattern Recognition Confer- planning,” arXiv:2311.17842, 2023. ence, 2025, pp. 5294–5306. [109] D. Driess et al., “Knowledge insulating vision-language-action [137] S. Chen et al., “Ac-dit: Adaptive coordination diffusion trans- models: Train fast, run fast, generalize better,” arXiv:2505.23705, former for mobile manipulation,” arXiv:2507.01961, 2025. 2025. [138] R. Xu et al., “A0: An affordance-aware hierarchical model for [110] Y.-J. Wang, B. Zhang, J. Chen, and K. Sreenath, “Prompt a robot to general robotic manipulation,” arXiv:2504.12636, 2025. walk with large language models,” in 2024 IEEE 63rd Conference [139] W. Yuan et al., “Robopoint: A vision-language model for spatial on Decision and Control (CDC). IEEE, 2024, pp. 1531–1538. affordance prediction for robotics,” arXiv:2406.10721, 2024. [111] A. Szot, B. Mazoure, H. Agrawal, R. D. Hjelm, Z. Kira, and A. To- [140] M. A. Patratskiy, A. K. Kovalev, and A. I. Panov, “Spatial traces: shev, “Grounding multimodal large language models in actions,” Enhancing vla models with spatial-temporal understanding,” Advances in Neural Information Processing Systems, vol. 37, pp. arXiv:2508.09032, 2025. 20 198–20 224, 2024. [141] J. Zhang, Y. Guo, Y. Hu, X. Chen, X. Zhu, and J. Chen, “Up- [112] G.-C. Kang, J. Kim, K. Shim, J. K. Lee, and B.-T. Zhang, “Clip- vla: A unified understanding and prediction model for embodied rt: Learning language-conditioned robotic policies from natural agent,” arXiv:2501.18867, 2025. language supervision,” arXiv:2411.00508, 2024. [142] W. Zhang et al., “Dreamvla: a vision-language-action [113] W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei, model dreamed with comprehensive world knowledge,” “Voxposer: Composable 3d value maps for robotic manipulation arXiv:2507.04447, 2025. with language models,” arXiv:2307.05973, 2023. [143] M. Assran et al., “V-jepa 2: Self-supervised video models enable [114] G. R. Team et al., “Gemini robotics: Bringing ai into the physical understanding, prediction and planning,” arXiv:2506.09985, 2025. world,” arXiv:2503.20020, 2025. [144] Y. Wu, R. Tian, G. Swamy, and A. Bajcsy, “From foresight to fore- [115] H. Fu et al., “Orion: A holistic end-to-end autonomous driving thought: Vlm-in-the-loop policy steering via latent alignment,” framework by vision-language instructed action generation,” arXiv:2502.01828, 2025. arXiv:2503.19755, 2025. [145] F. Zhu, Z. Yan, Z. Hong, Q. Shou, X. Ma, and S. Guo, “Wmpo: [116] P. Hao et al., “Tla: Tactile-language-action model for contact-rich World model-based policy optimization for vision-language- manipulation,” arXiv:2503.08548, 2025. action models,” arXiv:2511.09515, 2025. PREPRINT SUBMITTED TO IEEE TPAMI 19
[146] Q. Lv et al., “F1: A vision-language-action model bridging under- [174] H. Fang, Y. Liu, Y. Du, L. Du, and H. Yang, “Sqap-vla: A synergis- standing and generation to actions,” arXiv:2509.06951, 2025. tic quantization-aware pruning framework for high-performance [147] W. Zhao, G. Li, Z. Gong, P. Ding, H. Zhao, and D. Wang, vision-language-action models,” arXiv:2509.09090, 2025. “Unveiling the potential of vision-language-action models with [175] C.-Y. Hung et al., “Nora: A small open-sourced generalist vision open-ended multimodal instructions,” arXiv:2505.11214, 2025. language action model for embodied tasks,” arXiv:2504.19854, [148] C. Fan et al., “Interleave-vla: Enhancing robot manipulation with 2025. interleaved image-text instructions,” arXiv:2505.02152, 2025. [176] J. Wen et al., “Tinyvla: Towards fast, data-efficient vision- [149] C.-P. Huang, Y.-H. Wu, M.-H. Chen, Y.-C. F. Wang, and F.-E. language-action models for robotic manipulation,” IEEE Robotics Yang, “Thinkact: Vision-language-action reasoning via reinforced and Automation Letters, 2025. visual latent planning,” arXiv:2507.16815, 2025. [177] Y. Wang et al., “Vla-adapter: An effective paradigm for tiny-scale [150] C. Yin et al., “Deepthinkvla: Enhancing reasoning capability of vision-language-action model,” arXiv:2509.09372, 2025. vision-language-action models,” 2025. [178] I. Leal et al., “Sara-rt: Scaling up robotics transformers with self- [151] J. Zhang et al., “Inspire: Vision-language-action models with adaptive robust attention,” in 2024 IEEE International Conference intrinsic spatial reasoning,” arXiv:2505.13888, 2025. on Robotics and Automation (ICRA). IEEE, 2024, pp. 6920–6927. [152] X. Zhang et al., “Asktoact: Enhancing llms tool use via self- [179] R. Zhang et al., “Mole-vla: Dynamic layer-skipping vision lan- correcting clarification,” arXiv:2503.01940, 2025. guage action model via mixture-of-layers for efficient robot ma- [153] Y. Fan et al., “Long-vla: Unleashing long-horizon capability nipulation,” arXiv:2503.20384, 2025. of vision language action model for robot manipulation,” [180] W. Song et al., “Ceed-vla: Consistency vision-language-action arXiv:2508.19958, 2025. model with early-exit decoding,” arXiv:2506.13725, 2025. [154] S. Kanta et al., “Toward general physical intelligence for resilient [181] S. Xu, Y. Wang, C. Xia, D. Zhu, T. Huang, and C. Xu, “Vla-cache: agile manufacturing automation,” arXiv:2508.11960, 2025. Towards efficient vision-language-action model via adaptive to- [155] C.-P. Huang, Y.-H. Wu, M.-H. Chen, Y.-C. F. Wang, and F.-E. ken caching in robotic manipulation,” arXiv:2502.02175, 2025. Yang, “Thinkact: Vision-language-action reasoning via reinforced visual latent planning,” arXiv:2507.16815, 2025. [182] H. Wang, J. Xu, J. Pan, Y. Zhou, and G. Dai, “Specprune-vla: Ac- [156] A. Abdolmaleki et al., “Gemini robotics 1.5: Pushing the frontier celerating vision-language-action models via action-aware self- of generalist robots with advanced embodied reasoning, think- speculative pruning,” arXiv:2509.05614, 2025. ing, and motion transfer,” arXiv:2510.03342, 2025. [183] W. Li, R. Zhang, R. Shao, J. He, and L. Nie, “Cogvla: Cognition- [157] F. Lin, R. Nai, Y. Hu, J. You, J. Zhao, and Y. Gao, “Onetwovla: A aligned vision-language-action model via instruction-driven unified vision-language-action model with adaptive reasoning,” routing & sparsification,” arXiv:2508.21046, 2025. arXiv:2505.11917, 2025. [184] W. Song et al., “Accelerating vision-language-action model [158] Y. Yang, J. Sun, S. Kou, Y. Wang, and Z. Deng, “Lohovla: A unified integrated with action chunking via parallel decoding,” vision-language-action model for long-horizon embodied tasks,” arXiv:2503.02310, 2025. arXiv:2506.00411, 2025. [185] S. Wang et al., “Spec-vla: speculative decoding for vision- [159] N. Chung et al., “Rethinking progression of memory language-action models with relaxed acceptance,” in Proceedings state in robotic manipulation: An object-centric perspective,” of the 2025 Conference on Empirical Methods in Natural Language arXiv:2511.11478,2025. Processing, 2025, pp. 26 916–26 928. [160] J. Jiang, F. Deng, G. Singh, M. Lee, and S. Ahn, “Slot state [186] K. Pertsch et al., “Fast: Efficient action tokenization for vision- space models,” Advances in Neural Information Processing Systems, language-action models,” arXiv:2501.09747, 2025. vol. 37, pp. 11 602–11 633, 2024. [187] S. Fan et al., “Xr-1: Towards versatile vision-language-action [161] A. Ajay et al., “Compositional foundation models for hierarchi- models via learning unified vision-motion representations,” cal planning,” Advances in Neural Information Processing Systems, arXiv:2511.02776, 2025. vol. 36, pp. 22 304–22 325, 2023. [188] A. Van Den Oord, O. Vinyals et al., “Neural discrete representa- [162] S. Nasiriany et al., “Rt-affordance: Affordances are versatile in- tion learning,” Advances in neural information processing systems, termediate representations for robot manipulation,” in 2025 IEEE vol. 30, 2017. International Conference on Robotics and Automation (ICRA). IEEE, [189] Y. Wang, H. Zhu, M. Liu, J. Yang, H.-S. Fang, and T. He, “Vq- 2025, pp. 8249–8257. vla: Improving vision-language-action models via scaling vector- [163] J. Li et al., “Coa-vla: Improving vision-language-action mod- quantized action tokenizers,” arXiv:2507.01016, 2025. els via visual-text chain-of-affordance,” in Proceedings of the [190] M. Shukor et al., “Smolvla: A vision-language-action model for IEEE/CVF International Conference on Computer Vision, 2025, pp. affordable and efficient robotics,” arXiv:2506.01844, 2025. 9759–9769. [191] K. Black, M. Y. Galliker, and S. Levine, “Real-time execution of [164] D. Li et al., “An atomic skill library construction method for data- action chunking flow policies,” arXiv:2506.07339, 2025. efficient embodied manipulation,” arXiv:2501.15068, 2025. [192] Y. Niu, S. Zhou, Y. Li, Y. Den, and L. Wang, “Time-unified diffu- [165] Z. Yang et al., “Agentic robot: A brain-inspired frame- sion policy with action discrimination for robotic manipulation,” work for vision-language-action models in embodied agents,” arXiv:2506.09422, 2025. arXiv:2505.23450, 2025. [193] Z. Liang et al., “Discrete diffusion vla: Bringing discrete dif- [166] Y. Ji et al., “Robobrain: A unified brain model for robotic manip- fusion to action decoding in vision-language-action policies,” ulation from abstract to concrete,” in Proceedings of the Computer arXiv:2508.20072, 2025. Vision and Pattern Recognition Conference, 2025, pp. 1724–1734. [194] W. Chen et al., “Training strategies for efficient embodied reason- [167] J. Wen, Y. Zhu, J. Li, Z. Tang, C. Shen, and F. Feng, “Dexvla: ing,” arXiv:2505.08243, 2025. Vision-language model with plug-in diffusion expert for general robot control,” arXiv:2502.05855, 2025. [195] H. Chen et al., “Fast-in-slow: A dual-system foundation [168] Q. Bu et al., “Agibot world colosseo: A large-scale manipula- model unifying fast manipulation within slow reasoning,” tion platform for scalable and intelligent embodied systems,” arXiv:2506.01953, 2025. arXiv:2503.06669, 2025. [196] W. Zheng, B. Li, B. Xu, E. Feng, J. Gu, and H. Chen, [169] L. X. Shi et al., “Yell at your robot: Improving on-the-fly from “Leveraging os-level primitives for robotic action management,” language corrections,” arXiv:2403.12910, 2024. arXiv:2508.10259, 2025. [170] Z. Yu et al., “Correctnav: Self-correction flywheel empow- [197] C. Miao et al., “Fedvla: Federated vision-language-action learning ers vision-language-action navigation model,” arXiv:2508.10416, with dual gating mixture-of-experts for robotic manipulation,” in 2025. Proceedings of the IEEE/CVF International Conference on Computer [171] Y. Yang et al., “Fpc-vla: A vision-language-action framework Vision, 2025, pp. 6904–6913. with a supervisor for failure prediction and correction,” [198] K. Bousmalis et al., “Robocat: A self-improving generalist agent arXiv:2509.04018, 2025. for robotic manipulation,” arXiv:2306.11706, 2023. [172] H. Wang, C. Xiong, R. Wang, and X. Chen, “Bitvla: 1- [199] Z. Hou et al., “Diffusion transformer policy,” arXiv:2410.15959, bit vision-language-action models for robotics manipulation,” 2024. arXiv:2506.07530, 2025. [200] D. Qu et al., “Embodiedonevision: Interleaved vision-text-action [173] T. Lin et al., “Evo-1: Lightweight vision-language-action model pretraining for general robot control,” arXiv, pp. arXiv–2508, with preserved semantic alignment,” arXiv:2511.04555, 2025. 2025. PREPRINT SUBMITTED TO IEEE TPAMI 20
[201] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta, [228] J. Rocamonde, V. Montesinos, E. Nava, E. Perez, and D. Lindner, “R3m: A universal visual representation for robot manipulation,” “Vision-language models are zero-shot reward models for rein- arXiv:2203.12601, 2022. forcement learning,” arXiv:2310.12921, 2023. [202] K. Grauman et al., “Ego4d: Around the world in 3,000 hours of [229] S. Sontakke et al., “Roboclip: One demonstration is enough to egocentric video,” in Proceedings of the IEEE/CVF conference on learn robot policies,” Advances in Neural Information Processing computer vision and pattern recognition, 2022, pp. 18 995–19 012. Systems, vol. 36, pp. 55 681–55 693, 2023. [203] L. Fu et al., “In-context imitation learning via next-token predic- [230] O. Y. Lee, A. Xie, K. Fang, K. Pertsch, and C. Finn, tion,” arXiv:2408.15980, 2024. “Affordance-guided reinforcement learning via visual prompt- [204] V. Myers, B. C. Zheng, A. Dragan, K. Fang, and S. Levine, ing,” arXiv:2407.10341, 2024. “Temporal representation alignment: Successor features en- [231] Y. Wang et al., “Rl-vlm-f: Reinforcement learning from vision able emergent compositionality in robot instruction following,” language foundation model feedback,” arXiv:2402.03681, 2024. arXiv:2502.05454, 2025. [232] Y. J. Ma et al., “Eureka: Human-level reward design via coding [205] M. Zhu et al., “Objectvla: End-to-end open-world object manipu- large language models,” arXiv:2310.12931, 2023. lation without demonstration,” arXiv:2502.19250, 2025. [233] Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and [206] J. Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik, A. Zhang, “Vip: Towards universal visual reward and represen- “Lerf: Language embedded radiance fields,” in Proceedings of tation via value-implicit pre-training,” arXiv:2210.00030, 2022. the IEEE/CVF international conference on computer vision, 2023, pp. [234] M. Ahn et al., “Autort: Embodied foundation models for large 19 729–19 739. scale orchestration of robotic agents,” arXiv:2401.12963, 2024. [207] Y. Zhang et al., “Align-then-steer: Adapting the vision- [235] B. Häon, K. Stocking, I. Chuang, and C. Tomlin, “Mechanis- language action models through unified latent guidance,” tic interpretability for steering vision-language-action models,” arXiv:2509.02055, 2025. arXiv:2509.00328, 2025. [208] H. Etukuru et al., “Robot utility models: General policies for zero- [236] M. Zawalski, W. Chen, K. Pertsch, O. Mees, C. Finn, and shot deployment in new environments,” in 2025 IEEE Interna- S. Levine, “Robotic control via embodied chain-of-thought rea- tional Conference on Robotics and Automation (ICRA). IEEE, 2025, soning,” arXiv:2407.08693, 2024. pp. 8275–8283. [237] M. Neau, Z. Falomir, P. E. Santos, A.-G. Bosser, and C. Buche, [209] Z. Mandi, H. Bharadhwaj, V. Moens, S. Song, A. Rajeswaran, and “Grasp-vla: Graph-based symbolic action representation for V. Kumar, “Cacti: 256 a framework for scalable multi-task multi- long-horizon planning with vla policies,” arXiv:2511.04357, 2025. scene visual imitation learning. 257,” arXiv:2212.05711, vol. 258, [238] H. Lu, H. Li, P. S. Shahani, S. Herbers, and M. Scheutz, “Probing a 2022. vision-language-action model for symbolic states and integration [210] Z. Chen, S. Kiami, A. Gupta, and V. Kumar, “Genaug: Retargeting into a cognitive architecture,” arXiv:2502.04558, 2025. behaviors to unseen situations via generative augmentation,” [239] X. Li et al., “Object-centric prompt-driven vision-language-action arXiv:2302.06671, 2023. model for robotic manipulation,” in Proceedings of the Computer [211] T. Yu et al., “Scaling robot learning with semantically imagined Vision and Pattern Recognition Conference, 2025, pp. 27 638–27 648. experience,” arXiv:2302.11550, 2023. [240] M. Li et al., “Switchvla: Execution-aware task switching for [212] Y. Li, G. Yan, A. Macaluso, M. Ji, X. Zou, and X. Wang, “In- vision-language-action models,” arXiv:2506.03574, 2025. tegrating lmm planners and 3d skill policies for generalizable [241] S. Ye et al., “Latent action pretraining from videos,” manipulation,” arXiv:2501.18733, 2025. arXiv:2410.11758, 2024. [213] S. Haldar, Z. Peng, and L. Pinto, “Baku: An efficient transformer [242] Y. Chen et al., “Moto: Latent motion token as the bridging lan- for multi-task policy learning,” Advances in Neural Information guage for learning robot manipulation from videos,” in Proceed- Processing Systems, vol. 37, pp. 141 208–141 239, 2024. ings of the IEEE/CVF International Conference on Computer Vision, [214] W. Liu, Y. Du, T. Hermans, S. Chernova, and C. Paxton, “Structd- 2025, pp. 19 752–19 763. iffusion: Language-guided creation of physically-valid structures [243] R. Doshi, H. Walke, O. Mees, S. Dasari, and S. Levine, “Scaling using unseen objects,” arXiv:2211.04604, 2022. cross-embodied learning: One policy for manipulation, naviga- tion, locomotion and aviation,” arXiv:2408.11812, 2024. [215] Y. Yao et al., “Think small, act big: Primitive prompt learning for lifelong robot manipulation,” in Proceedings of the Computer Vision [244] C. Chen, Q. Yang, X. Xu, N. Fazeli, and O. Andersson, “Visa-flow: and Pattern Recognition Conference, 2025, pp. 22 573–22 583. Accelerating robot skill learning via large-scale video semantic action flow,” arXiv:2505.01288, 2025. [216] Z. Zheng et al., “imanip: Skill-incremental learning for robotic manipulation,” in Proceedings of the IEEE/CVF International Con- [245] R. Yang et al., “Egovla: Learning vision-language-action models ference on Computer Vision, 2025, pp. 13 890–13 900. from egocentric human videos,” arXiv:2507.12440, 2025. [246] T. Tao, M. K. Srirama, J. J. Liu, K. Shaw, and D. Pathak, “Dexwild: [217] S. N. Syed, Y. Ahuja, A. Jakobsson, and J. Ichnowski, “Expres-vla: Dexterous human interactions for in-the-wild robot policies,” Specializing vision-language-action models through experience arXiv:2505.07813, 2025. replay and retrieval,” arXiv:2511.06202, 2025. [247] W. Xiao et al., “Self-improving vision-language-action models [218] H. Zhang et al., “Slim: Sim-to-real legged instructive manipula- with data generation via residual rl,” arXiv:2511.00091, 2025. tion via long-horizon visuomotor learning,” arXiv:2501.09905. [248] J. Hejna, C. Bhateja, Y. Jiang, K. Pertsch, and D. Sadigh, “Re- [219] S. Tao et al., “Maniskill3: Gpu parallelized robotics simulation mix: Optimizing data mixtures for large scale imitation learning,” and rendering for generalizable embodied ai,” arXiv:2410.00425. arXiv:2408.14037, 2024. [220] J. Jang et al., “Dreamgen: Unlocking generalization in robot [249] H. R. Walke et al., “Bridgedata v2: A dataset for robot learning at learning through video world models,” arXiv:2505.12705, 2025. scale,” in Conference on Robot Learning. PMLR, 2023, pp. 1723– [221] Y. Jiang et al., “Rynnvla-001: Using human demonstrations to 1736. improve robot manipulation,” arXiv:2509.15212, 2025. [250] S. Nasiriany et al., “Robocasa: Large-scale simulation of everyday [222] Y. Guo et al., “Improving vision-language-action model with tasks for generalist robots,” arXiv:2406.02523, 2024. online reinforcement learning,” arXiv:2501.16664, 2025. [251] H. Arai et al., “Covla: Comprehensive vision-language-action [223] C. Xu, Q. Li, J. Luo, and S. Levine, “Rldg: Robotic generalist dataset for autonomous driving,” in 2025 IEEE/CVF Winter Con- policy distillation via reinforcement learning,” arXiv:2412.09858, ference on Applications of Computer Vision (WACV). IEEE, 2025, 2024. pp. 1933–1943. [224] D. Huang, Z. Fang, T. Zhang, Y. Li, L. Zhao, and C. Xia, “Co-rft: [252] D. Damen et al., “The epic-kitchens dataset: Collection, challenges Efficient fine-tuning of vision-language-action models through and baselines,” IEEE Transactions on Pattern Analysis and Machine chunked offline reinforcement learning,” arXiv:2508.02219, 2025. Intelligence, vol. 43, no. 11, pp. 4125–4141, 2020. [225] T. Jülg, W. Burgard, and F. Walter, “Refined policy distillation: [253] K. Grauman et al., “Ego-exo4d: Understanding skilled human From vla generalists to rl experts,” arXiv:2503.05833, 2025. activity from first-and third-person perspectives,” in Proceedings [226] Z. Zhang et al., “Grape: Generalizing robot policy via preference of the IEEE/CVF Conference on Computer Vision and Pattern Recog- alignment,” arXiv:2411.19309, 2024. nition, 2024, pp. 19 383–19 400. [227] C.-Y. Hung et al., “Nora-1.5: A vision-language-action model [254] M. U. Din, W. Akram, L. S. Saoud, J. Rosell, and I. Hussain, trained using world model- and action-based preference re- “Vision language action models in robotic manipulation: A sys- wards,” arXiv:2511.14659,2025. tematic review,” arXiv:2507.10672, 2025. PREPRINT SUBMITTED TO IEEE TPAMI 21
[255] P. Guruprasad, H. Sikka, J. Song, Y. Wang, and P. P. Liang, [281] S. James, Z. Ma, D. R. Arrojo, and A. J. Davison, “Rlbench: “Benchmarking vision, language, & action models on robotic The robot learning benchmark & learning environment,” IEEE learning tasks,” arXiv:2411.05821, 2024. Robotics and Automation Letters, vol. 5, no. 2, pp. 3019–3026, 2020. [256] P. Valle, C. Lu, S. Ali, and A. Arrieta, “Evaluating uncer- [282] T. Mu et al., “Maniskill: Generalizable manipulation skill bench- tainty and quality of visual language action-enabled robots,” mark with large-scale demonstrations,” arXiv:2107.14483, 2021. arXiv:2507.17049, 2025. [283] J. Gu et al., “Maniskill2: A unified benchmark for generalizable [257] Y. Zhu et al., “robosuite: A modular simulation framework and manipulation skills,” arXiv:2302.04659, 2023. benchmark for robot learning,” arXiv:2009.12293, 2020. [284] A. Shukla, S. Tao, and H. Su, “Maniskill-hab: A benchmark [258] O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard, “Calvin: for low-level manipulation in home rearrangement tasks,” A benchmark for language-conditioned policy learning for long- arXiv:2412.13211, 2024. horizon robot manipulation tasks,” IEEE Robotics and Automation [285] A. Mandlekar et al., “What matters in learning from offline hu- Letters, vol. 7, no. 3, pp. 7327–7334, 2022. man demonstrations for robot manipulation,” arXiv:2108.03298, [259] B. Liu et al., “Libero: Benchmarking knowledge transfer for 2021. lifelong robot learning,” Advances in Neural Information Processing [286] L. Zheng, F. Yan, F. Liu, C. Feng, Z. Kang, and L. Ma, “Robocas: A Systems, vol. 36, pp. 44 776–44 791, 2023. benchmark for robotic manipulation in complex object arrange- [260] I. Fang, J. Zhang, S. Tong, and C. Feng, “From intention to execu- ment scenarios,” arXiv:2407.06951, 2024. tion: Probing the generalization boundaries of vision-language- [287] R. Yang et al., “Embodiedbench: Comprehensive benchmarking action models,” arXiv:2506.09930, 2025. multi-modal large language models for vision-driven embodied [261] Z. Zhou et al., “Chatvla: Unified multimodal understanding and agents,” arXiv:2502.09560, 2025. robot control with vision-language-action model,” in Proceedings [288] H. Yue et al., “Ewmbench: Evaluating scene, motion, and seman- of the 2025 Conference on Empirical Methods in Natural Language tic quality in embodied world models,” arXiv:2505.09694, 2025. Processing, 2025, pp. 5377–5395. [289] T. Chen et al., “Robotwin 2.0: A scalable data generator and [262] T. Shen et al., “The journey/dao/tao of embodied intelligence: benchmark with strong domain randomization for robust biman- From large models to foundation intelligence and parallel intelli- ual robotic manipulation,” arXiv:2506.18088, 2025. gence,” IEEE/CAA Journal of Automatica Sinica, vol. 11, no. 6, pp. 1313–1316, 2024. [263] Q. Li et al., “Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipu- lation,” arXiv:2411.19650, 2024. [264] S. Li, Y. Gao, D. Sadigh, and S. Song, “Unified video action model,” arXiv:2503.00200, 2025. [265] S. Ye et al., “Latent action pretraining from videos,” arXiv:2410.11758, 2024. [266] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv:1707.06347, 2017. [267] J. Xiao et al., “World-env: Leveraging world model as a virtual environment for vla post-training,” arXiv:2509.24948, 2025. [268] H. Zhang, S. Zhang, J. Jin, Q. Zeng, R. Li, and D. Wang, “Robustvla: Robustness-aware reinforcement post-training for vision-language-action models,” arXiv:2511.01331, 2025. [269] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in International conference on machine learning. Pmlr, 2018, pp. 1861–1870. [270] J. Luo et al., “Serl: A software suite for sample-efficient robotic reinforcement learning,” in 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 16 961–16 969. [271] J. Luo, C. Xu, J. Wu, and S. Levine, “Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning,” Science Robotics, vol. 10, no. 105, p. eads5033, 2025. ∗ : A vla that learns from experience,” https: [272] P. Intelligence, “π0.6 //pi.website/blog/pistar06, accessed: 2025-02-18,2025. [273] H. Zhang, N. Zantout, P. Kachana, Z. Wu, J. Zhang, and W. Wang, “Vla-3d: A dataset for 3d semantic scene understanding and navigation,” arXiv:2411.03540, 2024. [274] A. Khazatsky et al., “Droid: A large-scale in-the-wild robot ma- nipulation dataset,” arXiv:2403.12945, 2024. [275] T. Perrett et al., “Hd-epic: A highly-detailed egocentric video dataset,” in Proceedings of the Computer Vision and Pattern Recogni- tion Conference, 2025, pp. 23 901–23 913. [276] Y. Liu et al., “Hoi4d: A 4d egocentric dataset for category- level human-object interaction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 21 013–21 022. [277] A. Padmakumar et al., “Teach: Task-driven embodied agents that chat,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 2, 2022, pp. 2017–2025. [278] L. Yu, X. Chen, G. Gkioxari, M. Bansal, T. L. Berg, and D. Batra, “Multi-target embodied question answering,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 6309–6318. [279] B. Jia, T. Lei, S.-C. Zhu, and S. Huang, “Egotaskqa: Under- standing human tasks in egocentric videos,” Advances in Neural Information Processing Systems, vol. 35, pp. 3343–3360, 2022. [280] Z. Cheng et al., “Embodiedeval: Evaluate multimodal llms as embodied agents,” arXiv:2501.11858, 2025. PREPRINT SUBMITTED TO IEEE TPAMI 1
A A PPENDIX This evolution is proceeding along three major direc- tions: (1) Incorporating physical perception via tactile and A.1 Applications force sensors (e.g., Tactile-VLA [121], VTLA [43]); (2) De- The true measure of Vision-Language-Action (VLA) mod- veloping industrial-grade Reasoning for complex processes els lies in their ability to solve real-world problems. By (e.g., ForceVLA [25], CogACT [263] ); (3) Ensuring Safety integrating perception, reasoning, and control, these mod- and Reliability through mechanisms like safe reinforcement els are uniquely equipped to translate abstract human in- learning (e.g., SafeVLA [79]). tent into grounded, executable actions, bridging the long- The future of VLA in industry hinges on bridging the standing gap between high-level cognition and low-level gap between flexible intelligence and the rigorous demands robotics. Leveraging large-scale pretraining, VLA-driven of production environments. The next wave of innovation robots demonstrate unprecedented capabilities in general- will likely focus on certification-ready safety and formal ver- ization and adaptation, surpassing conventional modular ification, moving beyond empirical safety to provide prov- pipelines in both autonomy and efficiency. This section able guarantees on robot behavior. Another critical direction surveys their transformative impact across two primary is zero-shot adaptation to new tasks and parts in highly cus- domains: household robotics and industrial automation. tomized manufacturing, where reprogramming a robot for every new product is economically infeasible. This requires A.1.1 Embodied Manipulation and Household Robotics VLA models to learn from CAD files, technical manuals, The unstructured, dynamic, and human-centric nature of and video demonstrations of human workers. Finally, the household environments makes them a major proving integration of VLA into multi-agent systems enables fleets ground for VLA models. Unlike structured factory settings, of robots to collaboratively perform complex assembly or homes require robots to understand natural language, han- logistics tasks, coordinated by a central language-based dle a vast diversity of unseen objects, and perform complex, understanding of the overall production goal. long-horizon tasks. VLA models excel in this domain precisely because their A.2 Basic Modules core architecture is well-suited to these challenges. Their A.2.1 Training Strategy ability to leverage internet-scale knowledge allows them to Current VLA training follows three largely complementary recognize and interact with a near-infinite variety of house- routes that are often combined in practice: hold items without task-specific training (e.g., SayCan [97], (1) Behavioral Cloning (BC). In current VLA research, RT-1 [99], RT-2 [76]). The evolution of benchmarks from BC is the dominant paradigm: it formulates control as ALFRED [93] to real-world validations like ChatVLA [261] supervised imitation, learning a mapping from multimodal confirms their robustness in sequential reasoning. Further- observations (i.e., vision, language, proprioception) to ex- more, the hierarchical reasoning inherent in many VLA pert actions by minimizing the prediction demonstration systems (e.g., Helix [262]) enables the decomposition of discrepancy. In practice, BC underpins a broad spectrum vague commands like “clean the kitchen” into concrete, of VLA systems across architectures, from diffusion-based executable subtasks. controllers to multimodal Transformer generalists (e.g., Dif- Looking ahead, the next frontier for household VLA fusion Policy [13], TriVLA [24], VIMA [70], Octo [49], RDT- systems lies in achieving true personalization and collabora- 1B [23], RT-H [38], Hi Robot [41], GR-2 [68], 3D-VLA [83], tive intelligence. Future models must move beyond simply RoboMM [132]). Beyond flat policies, BC is also employed executing one-off commands and learn to understand a as a pre- or post-training stage in hierarchical pipelines, e.g., user’s long-term preferences, habits, and implicit intentions. adapting continuous control in π0.5 [53] and driving the fast This requires a deeper integration of interactive learning, S-Sys1 executor in Dual-Process VLA [85]. where robots can learn from real-time verbal feedback, ask (2) Predictive Modeling. Instead of imitating actions, pre- clarifying questions when faced with ambiguity, and even dictive modeling learns to anticipate the world in future proactively suggest actions based on learned routines. observations or latent dynamics, providing powerful self- Furthermore, to become truly ubiquitous, these systems supervised signals that internalize physics and causal- must operate on low-power, on-device hardware, necessi- ity. World models exemplify this idea: WorldVLA [80], tating breakthroughs in model efficiency and compression. LUMOS [16], and UVA [264] train with predictive and The ultimate goal is to transform household robots from self-supervised objectives, enabling effective learning from simple instruction-followers into proactive, adaptive, and unstructured data and strong performance on complex, truly personalized domestic assistants. long-horizon robotic tasks. Additionally, other approaches learn discrete latent actions from unlabeled video as A.1.2 Industrial and Field Robotics in self-supervised prediction like world modeling (e.g., Following their success in household scenarios, VLA models LAPA [265]). now extend to industrial domains, where they promise to (3) Reinforcement Learning (RL). RL moves beyond bring unprecedented flexibility to manufacturing, logistics, demonstrations to optimize policies through interaction and field operations. Industrial environments, however, im- and reward feedback, and in VLA it is often built pose far stricter demands on precision, reliability, and safety. upon BC-pretrained backbones to enhance robustness The evolution of VLA models for industrial use is therefore and long-horizon performance. On-policy methods (e.g., characterized by a clear focus on enhancing their physical PPO [266]) update from freshly collected rollouts (e.g., LU- intelligence and robustness. MOS [16], VLA-RL [36], RoboCLIP [229],World-Env [267], PREPRINT SUBMITTED TO IEEE TPAMI 2
RobustVLA [268], EUREKA [232], Refined Policy Distil- they are crucial for grounding perception in human expe- lation [225]); off-policy methods (e.g., SAC [269]) exploit rience and for learning to infer human intent. The primary replay for sample efficiency (e.g., ConRFT [75], SERL [270], approach involves large-scale, egocentric video collection. RL-VLM-F [231]), with HIL-SERL [271] further leveraging Ego4D [202] provides thousands of hours of egocentric human demos and online corrections. Beyond these on- and video that support pretraining visual representations for ∗ off-policy paradigms, the latest π0.6 [272] introduces RE- human-object interaction, which can be effectively trans- CAP (RL with Experience and Corrections via Advantage- ferred to robotic policies. More specialized datasets further Conditioned Policies), a scalable RL framework for large enhance this capability. HD-EPIC [275] offers detailed, anno- VLA models that incorporates advantage-conditioned pol- tated egocentric recordings of unscripted kitchen activities, icy extraction into flow-matching/diffusion-based VLAs, and HOI4D [276] captures 4D human interactions with enabling stable and scalable training without relying on diverse objects, enabling fine-grained modeling of interac- complex RL objectives such as PPO. tion dynamics. TEACH [277] shifts the focus to instruction following by collecting dialogue-driven task execution data, A.2.2 Dateset making it a valuable resource for training agents that can collaborate with humans and resolve ambiguities through Recent progress in embodied intelligence is driven by a communication. shift toward data-centric development, where the scale, (4) Embodied Visual Question Answering Datasets. Em- diversity, and quality of training data largely determine a bodied VQA datasets pair visual scenes with language- VLA model’s generalization and robustness. VLA datasets based question-answer supervision and are increasingly form a diverse and evolving ecosystem, each providing used to train VLA models that require semantic alignment complementary supervision signals for different aspects of and environment-level reasoning. MT-EQA [278] provides embodied reasoning and control. Tab. S1 is an overview of 19,287 QA pairs for multi-target embodied question answer- representative embodied datasets. This section categorizes ing, requiring agents to navigate 3D indoor environments major datasets by their core properties and primary research and infer object attributes through active exploration. Ego- roles. TaskQA [279] expands cognitive scope with 368K generated (1) Simulation-Centered Datasets. These datasets are col- questions refined into 40K high-quality pairs covering de- lected in controlled virtual environments, which support scription, prediction, explanation, and counterfactual rea- large-scale, safe, and reproducible data generation with full soning. EmbodiedEval [280] further broadens task diver- access to state information. This makes them well suited for sity with 328 embodied tasks across 125 scenes, including studying high-level reasoning and long-horizon planning. Attribute QA on object and scene properties and Spatial ALFRED [93] provides expert demonstrations for 25,000 QA that evaluates spatial reasoning through interaction and language-grounded household tasks in AI2-THOR and em- observation. phasizes long-horizon compositionality. LIBERO [259] tar- gets lifelong robot learning by offering procedurally var- A.2.3 Evaluation ied tasks that evaluate incremental skill acquisition and Standardized benchmarks are central to embodied intel- retention. Recent datasets such as VLA-3D [273] incorporate ligence research because they define common evaluation detailed 3D scene representations paired with language protocols that support fair comparison, systematic diagnosis instructions to support the development of 3D-aware vi- of model limitations, and reproducible experimentation. The sion–language–action models. current VLA benchmark ecosystem is diverse, with plat- (2) Real-World Robotic Manipulation Datasets. These forms tailored to assess different dimensions of embodied datasets are collected from real robotic systems and capture competence, from basic skills to advanced cognitive abil- the full complexity of real-world sensing, dynamics, and ities. Tab. S2 is an overview of representative embodied environmental variability. They are essential for training benchmarks. This section reviews major benchmarks and policies that remain robust under uncertainty and can gener- categorizes them by the primary capabilities they aim to alize to unstructured settings. BridgeData V2 [249] provides evaluate. large-scale multi-task demonstrations collected across insti- (1) Language-Conditioned Manipulation. This category tutions using a standardized single-arm platform and serves evaluates a model’s ability to follow natural language in- as a central resource for generalist manipulation learning. structions and produce precise manipulation actions. RL- DROID [274] expands task and environment diversity by of- Bench [281] provides over 100 language-annotated tasks fering more than 350K in-the-wild trajectories gathered from with motion-planned demonstrations and serves as a stan- 50+ real environments with a low-cost mobile manipulator. dard benchmark for imitation and reinforcement learning. AgiBot World [168] further increases scale with a million- The ManiSkill series (ManiSkill [282], ManiSkill2 [283], level corpus spanning broad task and object variations to ManiSkill-HAB [284]) offers large-scale simulation envi- support large VLA model training. Open X-Embodiment ronments designed to assess multi-task manipulation and (OXE) [100] aggregates over 60 datasets across 22 robot policy generalization, with ManiSkill-HAB providing high- embodiments and currently represents the most comprehen- fidelity home-environment tasks. RoboMimic [285] eval- sive resource for studying cross-morphology transfer and uates offline learning methods using human demonstra- the emergence of generalist policies. tions and highlights key challenges in leveraging human- (3) Human-Centric and Egocentric Datasets. These datasets generated data for manipulation policies. capture data from a first-person human perspective. Al- (2) Long-Horizon and Interactive Task Completion. This though they typically do not include robotic action labels, category evaluates tasks that require sequential reasoning, PREPRINT SUBMITTED TO IEEE TPAMI 3
memory, and sustained interaction with the environment or new skills and retains prior ones across a task sequence. a human user. ALFRED [93] assesses long-horizon composi- RoboCAS [286] probes embodied cognition in cluttered and tional household tasks involving irreversible state changes, physically unstable scenes, exposing the limitations of cur- which challenge planning, memory, and instruction follow- rent models in physical reasoning, spatial understanding, ing. CALVIN [258] links language commands with continu- and robust interaction with unpredictable environments. ous control and evaluates an agent’s ability to execute long (4) Evaluation of Embodied Foundation Models. This sequences of language-guided operations in unseen envi- category shifts the evaluation focus from single-task agents ronments while maintaining state and performing sequen- to the holistic and emergent capabilities of large pretrained tial reasoning. TEACH [277] advances toward interactive multimodal systems. EmbodiedBench [287] evaluates mul- task execution by introducing dialogue-based instruction timodal large language models such as GPT-4o across high- following, where the agent must seek clarification and re- level semantic planning and low-level physical control to cover from errors through natural language communication. diagnose their end-to-end embodied competence. EWM- (3) Advanced Cognitive Capabilities. This category in- Bench [288] measures the physical realism of generative cludes benchmarks designed to probe higher-level cognitive world models by assessing the motion and semantic con- functions beyond basic instruction following, such as life- sistency of their predicted futures. RoboTwin [289] targets long learning and physical reasoning. LIBERO [259] quanti- cross-robot generalization and evaluates policies on dual- fies lifelong learning dynamics through forward and back- arm collaborative tasks, emphasizing their ability to transfer ward transfer metrics that measure how an agent acquires from large-scale synthetic data. PREPRINT SUBMITTED TO IEEE TPAMI 4
TABLE S1: An overview of representative embodied datasets. We exhibit different facets of these datasets, including embodiment, perspective, episodes, scenes, tasks&skills, and collection. More details are discussed in Section A2.2.
Name (Year) Embodiment Perspective Episodes Scenes Tasks & Skills Collection
Simulation-Centered Datasets
Simulated human 8,055 expert ∼120 indoor 8 composite Simulation
ALFRED [93] (2020) First-person agent demonstrations scenes household activities (AI2-THOR) 4 simulated LIBERO [259] (2022) Simulated robot arm First-person ∼6,500 130 skills Simulation (Robosuite) domains 11.5k Simulation Virtual agent in 3D Spatial navigation & VLA-3D [273] (2024) Third-person 9.7M referential pairs reconstructed 3D (Matterport3D / scenes grounding rooms ScanNet)
Real-World Robotic Manipulation Datasets
Real robot (VR
BridgeData V2 [249] Mixed (first- & 24 real 13 core manipulation Robot arm (WidowX) 60,096 trajectories teleoperation + (2023) third-person) environments skills scripted) Real robot (VR Robot arm (Franka Mixed (wrist & 564 distinct real DROID [274] (2024) ∼76k (≈350 hours) 86 tasks teleoperation by 50 Emika Panda) external cameras) scenes operators) Open Mixed (first- & 160k+ unified Web-scale aggregation X-Embodiment [100] 22 robot types 1M+ trajectories 527 skills third-person) scenes of real-robot data (2023) 5 domains (home, AgiBot World [168] Dual-arm humanoid retail, office, Real robot (multi-robot First-person 1M+ trajectories 217 tasks (2024) robot fleet restaurant, facility) industry)
Human-Centric and Egocentric Datasets
∼3,700 hours (∼1M 74 locations across Real human egocentric
Ego4D [202] (2021) Human First-person Multi-activity clips) 9 countries video Human commander + Mixed (first- & ∼3k dialog-based ∼200 simulated 17 composite Human teleoperation TEACh [277] (2021) embodied agent third-person) episodes homes household tasks in simulation 54 tasks across all 16 Head-mounted dual HOI4D [276] (2022) Human First-person ∼4,000 sequences 610 indoor scenes categories RGB-D ∼4,881 object 9 Real kitchen Wearable sensors HD-EPIC [275] (2025) Human First-person – itineraries scenes (Project Aria glasses)
Embodied Visual Question Answering Datasets
61 unique object in 8
MT-EQA [278] (2019) – First-person ∼19,287 QA pairs 588 environments Simulation (House3D) unique room EgoTaskQA [279] 48 relationships and Head-mounted Human First-person ∼40K QA pairs Kitchen (2022) 14 object attributes egocentric RGB video EmbodiedEval [280] Navigation – First-person 328 tasks 125 unique scenes – (2025) Spatial,Attribute, … PREPRINT SUBMITTED TO IEEE TPAMI 5
TABLE S2: An overview of representative embodied benchmarks. We exhibit different facets of these benchmarks, including task type, evaluation metrics, and environment/platform. More details are discussed in Section A2.3.
Name (Year) Task Type Evaluation Metric Environment / Platform
Language-Conditioned Manipulation & Control
RLBench [281] (2020) Multi-task tabletop Success rate PyRep / CoppeliaSim manipulation ManiSkill Multi-task object-centric Success / completion rate (per task) SAPIEN (ManiSkill), Habitat-based (ManiSkill- Series [282]–[284] manipulation HAB) RoboMimic [285] (2021) Multi-stage robot manipulation Success rate MuJoCo
Long-Horizon and Interactive Task Completion
ALFRED [93] (2020) Vision–language instruction Success rate, Goal-Condition Success ALFRED simulator following CALVIN [258] (2022) Language-guided multi-step Success rate, zero-shot generalization Simulated tabletop (4 scenes) manipulation TEACh [277] (2021) Dialog-driven embodied task Success rate, EDH / TfD / TATC AI2-THOR completion
Advanced Cognitive Capabilities
LIBERO [259] (2023) Continual multi-task Success rate, Fwd/Bwd Transfer, AUC Robosuite manipulation RoboCAS [286] (2024) Multi-object arrangement & Success under spatial/clearance constraints Custom arrangement scenes (SAPIEN) long-horizon manipulation
Evaluation of Embodied Foundation Models
EmbodiedBench [287] Vision-driven embodied agent Success rate, Subgoal success rate AI2-THOR, Habitat 2.0, CoppeliaSim (2025) evaluation EWM Bench [288] (2025) World-model evaluation Scene consistency, motion correctness, semantic Synthetic + real embodied datasets alignment RoboTwin [289] (2025) Multi-robot imitation, Success rate, sim ↔ real transfer rate, latency Isaac Gym / PyBullet cross-embodiment manipulation PREPRINT SUBMITTED TO IEEE TPAMI 6
TABLE S3: An overview of VLA milestones. We exhibit different facets of these methods, including robot perception, brain, action, training strategy, primary dataset, and evaluation, which corresponding to the subsections in Section 3.
Name Perception (Visual/Language) Brain Action Training Primary Dataset Evaluation By 2021 EmbodiedQA [89] CNN/LSTM LSTM+FNN Discrete(Autoregressive) BC EQA dataset EQA v1 VLN [88] ResNet-152 / LSTM LSTM Discrete(Autoregressive) BC R2R R2R RCM [91] ResNet-152/LSTM LSTM Discrete(Autoregressive) BC + RL R2R R2R Point-Cloud PointNet++ & ResNet50/LSTM RRN+GRU-RNN Discrete (Sequentia) BC MP3D-EQA Matterport3D EQA [92] ALFWorld [94] Mask R-CNN / Seq2Seq Seq2Seq Discrete(Autoregressive) BC TextWorld ALFRED benchmark CLIPort [96] ResNet-50 / Transformer FCN + Affordances Discrete(Autoregressive) BC Ravens Ravens 2022 SayCan [97] Resnet-18 / LLM LLM Discrete(Autoregressive) BC + RL – – Inner Monologue [98] LLM LLM Discrete(Autoregressive) BC Everyday Robots, Ravens Ravens & Ravens RT-1 [99] EfficientNet-B3 / USE Transformer Discrete(Autoregressive) BC Self-collected Self-built benchmark RT-2 [76] PaLI-X + PaLM-E VLM Discrete(Autoregressive) BC + co- WebLI + RT-1 Real-world finetuning General- ization Benchmark 2023 PaLM-E [77] ViT / PaLM VLM Discrete(Autoregressive) Multimodal WebLI,… OK-VQA,… SFT Diffusion Policy [13] ResNet-18 Transformer/DiT Continuous(DDPM) BC Human demonstra- Robomimic,Push- tion data T,… 2024 3D-VLA [83] VLM 3D-LLM Continuous(trajectory seg- co-finetuning OXE, RLBench, … RLBench, ment) RoboVQA,… Octo [49] CNN / T5 Transformer Continuous(DDPM) BC OXE Policy gener- alization, SR OpenVLA [32] SigLip + Dino Transformer Discrete(Autoregressive) BC OXE Libero GR-2 [68] VQGAN / CLIP Transformer Continuous Predictive HowTo100M, CALVIN, … modeling Ego4D,… π0 [22] VLM Transformer Continuous(Flow BC OXE, Bridge v2,… – Matching) 2025 Humanoid- Transformer Transformer Continuous(Autoregressive)BC Humanoid-S, HumanML3D, VLA [101] AMASS Humanoid- S,… GR00T N1 [102] Eagle-2 VLM VLM + DiT Continuous(Flow BC GR00T N1 dataset, GR-1 Matching) OXE,… Tabletop Tasks,… PointVLA [82] CNN (3D) / VLM VLM Continuous(DDPM) BC Self-collected data RoboTwin CoT-VLA [78] Transformer LLM Discrete BC OXE Libero, Brige v2, … π0.5 [53] VLM Transformer Hybrid(Flow Matching) BC + Predic- OXE Real family tive modeling scenes LUMOS [16] CNN / Sentence-BERT Goal-conditioned actor- Continuous/ BC – MLPerf critic Training Benchmarks VLA-RL [36] Siglip + Dino / Llama-2 Transformer Discrete(Autoregressive) RL Online-collected LIBERO Cosmos-R1 [103] ViT/LLM LLM Discrete(Autoregressive) BC + RL BridgeData RoboFail,AgiBot,… v2,RoboVQA,…