VLMPC: Vision-Language Model Predictive Control for Robotic Manipulation

Authors: Wentao Zhao, Jiaming Chen, Ziyu Meng, Donghui Mao, Ran Song, Wei Zhang

Venue: RSS 2024

arXiv: 2407.09829

Abstract

VLMPC integrates VLMs into Model Predictive Control by using the VLM to evaluate candidate action sequences. A conditional action sampling module generates candidate actions from a goal image or language instruction; a lightweight video prediction model generates future frames for each candidate; a hierarchical VLM cost function selects the optimal action.

Method

• Hierarchical cost function: pixel-level cost (visual alignment to goal) + knowledge-level cost (VLM semantic evaluation of the predicted video)
• Video prediction: action-conditioned video model generates future frames for each candidate action sequence
• Selection: optimal action sequence chosen by minimizing the combined cost
• RSS 2024 follow-up (Traj-VLMPC): adds motion trajectory conditioning

Connection to Semantic-Metric Bayesian Risk Fields

Both papers use VLM evaluation of visual predictions as a cost/risk signal. VLMPC uses the VLM to score task completion quality; Semantic-Metric Bayesian Risk Fields uses a VLM prior to score spatial safety risk. The architectures are convergent: predict future visual state → query VLM-grounded cost → filter/select actions.

Notes

VLMPC is not safety-focused — it’s task-performance focused. But the VLM-as-cost-function architecture directly enables safety by replacing the task-completion VLM query with a risk/danger VLM query.