本文由 AI 分析生成
建立時間： 2026-04-30 來源： https://arxiv.org/html/2507.05331v1
Summary
TRI (Toyota Research Institute) rigorously evaluates Large Behavior Models (LBMs) — visuomotor foundation models trained on large-scale multitask datasets — across 1,800 real-world trials and simulation experiments using blind A/B testing with statistical significance analysis. They find that multitask pretraining consistently improves performance and robustness over single-task baselines, with benefits amplifying under distribution shift. Performance predictably scales with pretraining data scale and diversity.
TRI 團隊透過 1,800 次真實世界盲測（含統計顯著性分析）嚴格評估 LBM（大型行為模型）。核心發現：多任務預訓練使策略在分佈偏移下更具魯棒性，且所需任務特定數據更少。性能隨預訓練規模與多樣性可預測地提升。
Prerequisites
- Behavior cloning and Diffusion Policy basics
- Understanding of sim2real gap and distribution shift evaluation
- Familiarity with visuomotor policy training
Core Idea
Extends Diffusion Policy across ~1,700 hours of robot demonstrations (500+ internally collected tasks + public data). Uses a rigorous evaluation protocol: blind A/B real-world trials, 50 rollouts per task/policy/condition in real world, 200 in simulation, task-specific rubrics for partial credit (Task Completion metric), and Bayesian posterior violin plots for uncertainty visualization. Compares pretrained-only LBM, finetuned LBM, and single-task baseline.
Results
- Finetuned LBMs outperform single-task baselines when aggregating over tasks (statistically significant in all aggregate plots)
- Under distribution shift, finetuned LBM goes from outperforming single-task in 3/16 sim tasks (nominal) to 10/16 (distribution shift) — robustness benefit amplifies
- Pretrained LBM (no finetuning) performs comparably or worse than single-task on most individual tasks, especially in real world
- Finetuned LBMs require fewer task-specific demonstrations to match single-task performance
- One task (TurnCupUpsideDown) showed the finetuned LBM failing 89/200 rollouts without moving — likely a task-specific normalization artifact
- Data normalization error discovered post-evaluation in pretraining — may partially explain pretrained LBM underperformance
Limitations
- Author-stated: no architectural novelty studied — fixed Diffusion Policy architecture throughout; evaluation designed for ~50% success rate which may not generalize to easier/harder task distributions
- Unstated: data normalization bug discovered after evaluation casts uncertainty on pretrained (non-finetuned) LBM results; real-world and simulation distribution shifts are not directly comparable; long-horizon complex tasks evaluated with TC metric which is subjective
Reproducibility
- Code: project page at toyotaresearchinstitute.github.io/lbm1
- Datasets: ~1,700 hours internal + publicly available robot data (DROID, Open X-Embodiment components)
- Compute: not detailed in abstract section; requires significant compute for 1,800+ real-world rollouts
Insights
The introduction of Task Completion (TC) alongside Success Rate (SR) is methodologically important — a policy that “almost succeeds” provides more useful signal than a binary SR metric alone. The finding that distribution shift amplifies LBM benefits (3/16 → 10/16 tasks) is the most compelling result, suggesting pretrained representations generalize robustness in a way that cannot be attributed to in-distribution overfitting.
Connections
Raw Excerpt
“We find that multi-task pretraining makes the policies more successful and robust, and enables teaching complex new tasks more quickly, using a fraction of the data when compared to single-task baselines. Moreover, performance predictably increases as pretraining scale and diversity grows.”