本文由 AI 分析生成
建立時間： 2026-08-18 來源： https://arxiv.org/abs/2407.09829
Summary
VLMPC integrates VLMs into Model Predictive Control for robot manipulation by using a VLM as a hierarchical cost function evaluating candidate action sequences. Candidate actions are sampled conditioned on a goal image or language instruction; a lightweight video prediction model generates future frames for each candidate; and a two-level VLM cost (pixel-level visual alignment + semantic task completion) selects the best sequence. Presented at RSS 2024.
VLMPC 將 VLM 作為 MPC 的階層式代價函數，對候選動作序列進行視覺對齊（像素級）與語義評估（VLM 任務完成度），透過輕量影片預測模型生成候選未來幀，不需任務特定的代價函數設計（RSS 2024）。
Prerequisites
- Model Predictive Control (MPC) — the optimization loop: sample candidate actions → simulate outcomes → select best candidate based on a cost function; VLMPC replaces the cost function with a VLM.
- Video prediction models — action-conditioned video generation is needed to produce the future frames the VLM evaluates; the quality of these frames bounds the quality of VLM evaluation.
- VLMs as evaluators — the VLM is used not as a planner but as a scorer/judge of predicted future states against a goal description.
Core Idea
The central contribution is replacing the hand-designed MPC cost function with a VLM-based hierarchical evaluation: pixel-level cost ensures the predicted future visually resembles the goal, while knowledge-level cost asks the VLM whether the semantic task intent is being achieved (e.g., “is the red block now on the plate?”). This decomposition handles both precision (geometric alignment) and semantics (task success) without manual cost engineering. The action-conditioned video prediction is the enabling piece: without future frame synthesis, the VLM cannot evaluate candidate actions at all.
Results
| Task | VLMPC | Prior Methods | Delta |
|---|---|---|---|
| Tabletop manipulation (RSS 2024) | Outperforms language-conditioned BC baselines | Behavior cloning | Superior generalization |
| Traj-VLMPC (follow-up) | Trajectory-conditioned variant | VLMPC | Improved for complex motions |
Limitations
- Author-stated: Depends on quality of the video prediction model; inaccurate future frame predictions mislead VLM evaluation.
- Unstated: VLM evaluation at each MPC step adds significant latency; real-time control may require model distillation or fewer candidates. Task not safety-focused.
Reproducibility
- Code: Not confirmed as released
- Datasets: Tabletop manipulation tasks (simulation + real)
- Compute: VLM inference per candidate per MPC step; multiplied by number of candidates and horizon length
Insights
VLMPC demonstrates that VLMs can function as general-purpose MPC cost functions without task-specific engineering — the VLM’s world knowledge fills in what the cost function would otherwise need to explicitly specify. This is a major step toward sample-efficient manipulation: the VLM encodes the success criterion in natural language, eliminating reward shaping.
The architectural pattern (predict future visual state → query VLM-grounded cost) is directly extensible to safety: replace the task-completion VLM query with a risk/danger VLM query (as done in Semantic-Metric Bayesian Risk Fields), yielding safety-aware MPC without separate safety training.
Connections
- semantic-metric-bayesian-risk-fields-vlm-robot-safety — uses the same VLM-as-cost architecture for safety rather than task performance
- safedreamer-safe-reinforcement-learning-world-models — world model imagination for action selection; VLMPC uses video prediction similarly
- robot-manipulation-unified-survey-2510-10903 — manipulation survey context
Raw Excerpt
“A hierarchical VLM cost function selects the optimal action: pixel-level cost (visual alignment to goal) + knowledge-level cost (VLM semantic evaluation of the predicted video).”