Semantic-Metric Bayesian Risk Fields: Learning Robot Safety from Human Videos with a VLM Prior

Authors: Timothy Chen, Marcus Dominguez-Kuhne, Aiden Swann, Xu Liu, Mac Schwager (Stanford)

arXiv: 2512.08233 (December 2025)

Abstract

A framework for learning robot safety from human demonstration videos by combining a VLM common-sense prior with a learned likelihood function. Outputs pixel-dense risk maps that feed into robot planners — either as value predictors for visuomotor policies or projected into 3D for trajectory optimization. Outperforms state-of-the-art VLMs in producing human-aligned risk assessments.

Method

• Bayesian formulation: prior = VLM query on scene semantics (e.g., “is it risky to move near the hot stove?”); likelihood = trained ViT mapping DINO features to pixel-aligned risk values
• Supervision: safe human demonstration videos — regions humans avoid map to high risk
• Output: pixel-dense risk image; can be projected into 3D point cloud for integration with MPC/trajectory optimization
• No explicit cost function needed: risk is learned from human behavior, not manually specified

Safety Relevance

VLM provides semantic context that pure geometric safety methods lack — it understands that a kitchen knife is risky even if it is geometrically close in the same way as a harmless object. The learned likelihood modulates the prior to produce spatially precise, context-sensitive risk estimates.

Results

• Risk-aware trajectories outperform VLM-only baselines in human alignment
• Context-sensitive decisions (e.g., treating a knife differently from a spatula) emerge from VLM prior
• Compatible with downstream classical planners (no special policy training needed)