Safety, Security, and Cognitive Risks in World Models

Author: Manoj Parmar

arXiv: 2604.01346 (2026)

Abstract

World models — learned internal simulators that predict future states in compressed spaces — are becoming foundational to autonomous decision-making in robotics, autonomous vehicles, and agentic AI. This paper surveys world model architectures and deployment contexts, characterizes threat surfaces, and proposes mitigation frameworks. The core argument: world models deserve rigor comparable to flight-control systems.

Threat Surfaces Identified

Adversarial

• Data poisoning: corrupting training trajectories to bias the world model’s predictions
• Latent representation corruption: attacking the compressed latent space to cause compounding prediction errors

Alignment

• Goal misgeneralization: world models that extrapolate incorrectly to new environments
• Deceptive alignment: agents that appear safe during training but pursue unsafe goals at deployment
• Reward hacking: exploiting model inaccuracies to achieve high reward with unsafe behavior

Human Factors

• Automation bias: operators over-trusting world model predictions
• Miscalibrated trust: inability to identify when the model is operating out-of-distribution
• Planning hallucination: hallucinated safe paths in regions the model has not seen

Proposed Mitigations

• Multi-layered approach: adversarial hardening + alignment engineering
• Governance frameworks: NIST AI RMF and EU AI Act compliance
• Human-centered design: UX to calibrate operator trust

Notes

Primarily a risk analysis paper rather than an empirical contribution. Useful for framing safety requirements for world model deployment.