Imitation Learning for Robot Manipulation
Sources: Research synthesis, 2026-03-26; HITL-HRI survey, 2026-06-15; Session notes, 2026-05-02; arXiv digests, 2026-W07–W32; Urain et al. IEEE TRO 2024; TRI LBM Team arXiv:2507.05331; Xu et al. (Columbia), arXiv:2506.16685, 2025-12 Raw: Robotic Teleoperation for Dexterous Manipulation; HITL-HRI Survey 2024-2026; Robot Learning Concepts VLA Diffusion; W07; W15; W18; VLA-0; [Deep Generative Models Survey](../../raw/clippings/Deep Generative Models Learning from Multimodal Demonstrations.md); LBM; CR-DAgger Updated: 2026-08-12
Overview
Imitation learning (IL) converts human demonstration data into executable robot policies. Three paradigms dominate: behavioral cloning (BC), diffusion policies, and interactive imitation learning (HITL). Diffusion policies have become the standard for contact-rich manipulation. HITL methods (DAgger and variants) directly address BC’s core failure mode — covariate shift — but require continued human attention at deployment. The frontier is Vision-Language-Action (VLA) models that unify perception, language, and action under a single foundation model.
Behavioral Cloning
The simplest approach: supervised learning that maps observed states to actions, trained directly on demonstration trajectories.
Failure mode — covariate shift: small execution errors compound because the policy was never trained on the recovery states those errors create. The robot visits states the expert never visited; the policy has no guidance in these states; more errors accumulate. For short-horizon tasks with high-quality demonstrations, BC can still be surprisingly effective. For long-horizon tasks, failure is often spectacular.
Action chunking (ACT): predict multiple future actions at once rather than one action per timestep. Reduces effective decision frequency, mitigating compounding error. ACT (52M params) achieves 90% on fixed-setup tasks with ~5ms inference on an RTX 4090.
Diffusion Policies
Diffusion policies model the complete action distribution as a denoising process — starting from Gaussian noise and iteratively denoising to a valid action sequence.
Why they work for manipulation: dexterous tasks have multimodal action distributions. There are often multiple valid ways to grasp an object; a deterministic policy that averages over modes produces a trajectory that executes none of them correctly. Diffusion policies sample from the distribution rather than averaging it, producing committed, crisp actions.
Flow Matching Policies (2024-2025): a faster variant that defines action generation as a deterministic flow (ODE) rather than a stochastic diffusion process. Single-step inference matches diffusion quality. π₀, SmolVLA, and other frontier VLA models adopt this architecture. This is replacing diffusion policies in the most recent systems.
Generative model families for LfD (Urain et al. IEEE TRO, arXiv:2408.04380):
| Model | Strengths | Limitations |
|---|---|---|
| Energy-Based Models (EBMs) | Flexible distribution modeling | Training stability |
| Diffusion Models | High-quality multimodal output | Slow inference |
| Action Value Maps | Spatial reasoning | Limited to discrete actions |
| GANs | Fast sampling | Mode collapse, training instability |
All four address the same core problem: classical BC averages over multimodal demonstrations, producing behaviors matching none of them. Generative models sample from the distribution. OOD generalization — failure when robot encounters states outside the training dataset — is the shared open challenge.
Interactive Imitation Learning (HITL)
Core problem addressed: covariate shift — the distribution mismatch between demonstration states (where the expert was) and deployment states (where the policy actually goes).
DAgger (Ross et al., 2011): the canonical algorithm. At deployment, the human provides corrections in the states the robot actually visits. This data is added to the training set and the policy is retrained. Each round shifts the training distribution to better match the deployment distribution. All HITL systems are DAgger variants.
RoboCopilot (2025): a deployed HITL system using bilateral teleoperation for hardware-in-the-loop DAgger. The human can intervene mid-task; every intervention becomes training data. Deployed on real dexterous manipulation hardware.
CR-DAgger — Compliant Residual DAgger (Xu et al., arXiv:2506.16685, Dec 2025): addresses two DAgger failure modes — how to collect informative corrections and how to update policies from them.
- Compliant Intervention Interface: leverages compliance control for kinesthetic delta corrections without interrupting the ongoing robot policy. Human applies force directly; feels correction magnitude; smooth transition between correction/no-correction mode.
- Compliant Residual Policy: learns from corrections with force feedback and force control; the residual formulation avoids catastrophic forgetting of the base policy.
- Results: 64% improvement in base policy success rates on 4 contact-rich tasks (book flipping, belt assembly, cable routing, gear insertion). Outperforms both retraining-from-scratch and fine-tuning approaches on these tasks.
Cost-benefit tradeoff: interactive IL requires human attention at deployment time — not just at data collection time. The economic calculation: when does continuous human correction pay off vs. simply collecting more initial demonstrations? For safety-critical tasks, the answer favors HITL. For high-volume repetitive tasks, more initial demonstrations may be more economical.
Vision-Language-Action Models
VLAs treat robot actions as just another output modality of a large transformer backbone — the same architecture as multimodal language models, with an action head attached.
Key characteristics:
- Conditioned on natural language task descriptions (enables some generalization to novel tasks)
- Pretrained on internet-scale data (images, video, text) before fine-tuning on robot demonstrations
- Action representation: autoregressive token sequences or diffusion/flow matching over continuous action spaces
VLA paradigm taxonomy (from arXiv:2509.19012, 300+ papers):
- Autoregressive: actions as discrete tokens (like language tokens)
- Diffusion-based: continuous action distributions via denoising (e.g., π₀)
- RL fine-tuning: VLA pretrained on demonstrations, then optimized with RL
- Hybrid: combining multiple paradigms
- Specialized: task-specific architectures built on VLA foundations
VLA-0 (NVIDIA, 2025): challenges the assumption that VLAs require architectural modification. Represents actions directly as text — zero modification to the base VLM. Key finding: the simplest approach is surprisingly powerful. VLA-0 achieves best performance among models without large-scale pretraining (94.7% average across spatial/object/goal/long-horizon tasks on SO-100 platform), beating SmolVLA 2.25B (88.8%) and OpenVLA-OFT (91.9%). Categorizes existing VLAs into three families: discrete token, generative head, and simple text — VLA-0 takes the simple path.
Large Behavior Models (LBMs) (TRI, arXiv:2507.05331, 2025): extends Diffusion Policy paradigm to multi-task pretraining at scale. Key findings: multi-task pretraining makes policies more successful, more robust, and enables teaching complex new tasks with a fraction of the data vs single-task baselines. Performance predictably increases with pretraining scale and diversity — one of the few results that suggests a partial scaling law for robot manipulation.
The data implication: VLA training requires language annotations alongside trajectories. Teleoperation systems must now support annotation during demonstration collection, not just trajectory recording.
Diffusion Policy as VLA action head: In the π₀ architecture (the clearest current exemplar), the VLM (PaLI-X, LLaMA, or similar) handles language and scene understanding (“the brain”), while the diffusion policy generates continuous action trajectories (“the hand”). The VLM’s semantic vector conditions the diffusion denoising process; both components are jointly trained.
Autoregressive vs. diffusion — fundamental difference: Autoregressive VLAs (e.g., OpenVLA) must discretize action tokens, losing precision and struggling to represent multimodal action distributions. Diffusion-based VLAs directly generate continuous action chunks and excel at tasks where the same goal has multiple valid execution strategies.
VLA Scaling: Bottlenecks and Test-Time Compute
Compression Gap principle (arXiv:2604.03191, 2026): scaling behavior in a VLA pipeline is governed by the tightest information bottleneck, not the largest component. Two regimes:
- Continuous actions (Diffusion Policy): the vision encoder is the binding constraint. Upgrading the encoder improves performance by +21 percentage points on LIBERO.
- Discretized actions (OAT, fixed-capacity codebook): the codebook becomes the bottleneck. Encoder upgrades produce substantially attenuated improvements — gains are absorbed by the codebook’s limited capacity.
Implication: VLA scaling cannot be planned around a single component. The pipeline must be analyzed for its tightest bottleneck first. This explains why diffusion-based VLAs (π₀, SmolVLA) have outpaced autoregressive token-based VLAs at equivalent parameter counts.
RD-VLA (arXiv:2602.07845, Recurrent-Depth VLA, 2026): scales test-time compute without the memory penalty of Chain-of-Thought token generation. Adds a recurrent weight-tied action head that performs latent iterative refinement at inference:
- Training: truncated backpropagation through time (TBPTT)
- Inference: adaptive stopping criterion based on latent convergence — more compute allocated only when task demands it
- Memory: does not increase memory relative to single-pass inference
- Results: tasks with 0% success at single-iteration inference reach >90% success at four iterations; up to 80× speedup over prior reasoning-based VLA models using Chain-of-Thought
The latent iterative refinement paradigm is distinct from token-level scaling and diffusion denoising — it refines in the action head’s latent space.
7-DOF arm as research standard: Franka Panda (7-DOF) has become the de facto research platform for dexterous manipulation and LfD. Its redundant DOF enables self-motion — reconfigurating the arm without moving the end-effector — which avoids kinematic singularities and obstacles. 6-DOF IK has closed-form solutions; 7-DOF requires optimization to select the best configuration from a 1D manifold of solutions.
Sim-to-real gap mitigation: MIT RialTo (Real-to-Sim-to-Real) uses mobile phone scanning to build a digital twin of the target scene, trains in the high-fidelity simulation, then deploys. This approach reduces the sim-to-real gap by raising simulation scene fidelity rather than relying purely on domain randomization.
The Representation Alignment Problem
Bobu et al. (arXiv:2302.01928) provide the most principled account of why HITL and IL remain hard: human and robot representations are misaligned at the feature level. Even a perfect learning algorithm will produce wrong behavior if the robot’s internal features do not capture what the human cares about. Human feedback functions as a probe of representation gaps, not just a task value signal.
Implication for VLAs: internet-scale pretraining may produce representations partially aligned with human concepts, but explicit alignment objectives are likely needed for reliable task performance. This is the theoretical underpinning for RLHF-style approaches in robotics.
Survey Consensus (2023-2026)
Across five major survey papers, the same diagnosis recurs:
- Covariate shift is the root problem; DAgger closes the loop; diffusion models handle multimodality
- VLA surveys treat human data as a scaling input, not an interactive feedback mechanism — a gap the HITL community has identified but not bridged
- No unified framework covers safe RL, offline RL, and HITL simultaneously
- Data quality dominates model sophistication (ML6 empirical finding)
Open Questions
- When VLAs are fine-tuned with RL, what is the right role for human feedback vs. environment reward? Is RLHF for robot VLAs the next major paradigm?
- How does HITL correction cost scale economically? Is there a task complexity threshold below which collecting more demonstrations beats interactive correction?
- Can representation alignment (Bobu et al.) be made practical for 7B-parameter VLAs? Probing large models for feature alignment is methodologically challenging.