Next-token prediction in large language models excels at passive sequence modeling over finite discrete token spaces. However, this paradigm is insufficient on its own for scaling to grounded, action-conditioned intelligence, especially in robotics and long-horizon agentic systems. The required approach is hierarchical, objective-driven world modeling that predicts future states in learned representation spaces conditioned on actions.
1. Limits of Next-Token Prediction
An LLM approximates the conditional distribution over a fixed vocabulary. This works when the output space is enumerable. Real-world sensory streams (e.g., video) are continuous and noisy, with high conditional entropy components that make pixel-level next-frame prediction inefficient and misaligned with the desired invariances. The objective must instead focus on learning invariants and predictable subspaces.
2. Abstraction Hierarchies as the Scientific Template
LeCun draws an analogy to physics. Airflow is modeled by discretizing space into finite volumes, assigning velocity, pressure, and density to each volume, then solving the Navier–Stokes equations. Each level of science (quantum field theory → fluid dynamics → macro-scale behavior) is itself a world model at a different abstraction granularity. Neural systems must learn encoders that map observations into such representation spaces where future states remain predictable.
3. JEPA: Predicting Representations Conditioned on Actions
The Joint Embedding Predictive Architecture (JEPA) implements this principle by predicting future or missing representations in latent space rather than reconstructing observations directly.
Let z_t denote the representation of the current state produced by the online encoder from a corrupted or masked input. The target representation z_{t+1} is produced by a target encoder (typically with EMA updates) on the clean future input. The predictor learns:
where \( a_t \) denotes either explicit actions or implicit temporal context in passive video. This explicitly makes prediction action-conditioned, turning the model into a true world model rather than a passive video predictor.
A good representation must discard unpredictable information (high-conditional-entropy components such as random leaf motion or illumination noise) while preserving variables that are predictive under action. Collapse is prevented by explicit regularization on variance and covariance of the embeddings. Consequently, DINOv3—trained purely self-supervised with this paradigm—produces highly competitive or superior features on several vision benchmarks compared with supervised models such as CLIP.
4. Self-Supervised Learning as Foundation, Not Complete Intelligence
Self-supervised learning (predicting the invisible from the visible) is the common successful mechanism behind both LLMs (masked language modeling) and modern vision (DINO/JEPA). In practice, the overwhelming majority of pre-training compute is devoted to self-supervision, with only minimal supervised fine-tuning for downstream tasks.
LeCun emphasizes that self-supervised representation learning alone does not produce intelligence. The complete structure is a three-layer “cake”:
1. Self-supervised pre-training (the bulk of compute)
2. Supervised adaptation (small)
3. Reinforcement learning / objective-driven interaction (the cherry)
5. Why LLMs Do Not Naturally Extend to Robotics or Agentic Planning
A genuine agent requires four components: (i) a grounded state representation, (ii) hypothetical actions, (iii) a world model predicting the next state conditioned on action, and (iv) a planner optimizing action sequences in latent space. LLMs do not natively implement this structure. Generation is stochastic, requiring repeated sampling plus external verification, and error propagates on complex multi-step problems. Chain-of-Thought increases token length and compute but does not constitute causal reasoning. Most successes of LLMs are better characterized as sophisticated information retrieval than genuine planning or physical reasoning. Architectures built on the same autoregressive token paradigm (including current VLM/VLA) inherit these limitations.
6. The Central Open Problem: Scalable Hierarchical World Models
The primary remaining bottleneck is not data scale or compute, but learning hierarchical world models that operate stably across multiple temporal and abstraction scales. Lower layers handle fine-grained motion and geometry; higher layers handle intent and long-horizon plans (e.g., “walk → hail taxi → fly”). We know the desired structure—action-conditioned prediction at each level—but do not yet possess a scalable training recipe that prevents collapse or exponential divergence over long rollouts due to compounding errors and chaos. Long-horizon prediction is inherently unstable in open-loop settings, motivating closed-loop planning mechanisms that repeatedly re-anchor predictions to observed states. This remains the key unsolved technical challenge.
7. Data Reality Check
Scaling video pre-training to the equivalent of 20 trillion tokens is necessary. A 4-year-old child receives visual input at approximately 2 GB/s (10–14 bytes per frame at standard frame rates), accumulating roughly 16,000 hours by age 4. Language is a thin abstraction layer built atop this rich sensory experience; text alone cannot supply the required grounded representations.
8. Objective-Driven Training and Emergent Common Sense
In JEPA-style video training, the model observes the first half of a clip and predicts the representation of the second half. Prediction error provides a signal from which common-sense regularities can be learned (e.g., a wall disappears). This signal induces an early form of common sense and object permanence without explicit supervision. The loss function automatically directs attention toward causally relevant, predictable elements.
9. Consciousness as an Engineering Consequence
Visual attention emerges naturally in deep networks. The subjective experience of consciousness arises largely because the brain possesses only a single “engine cortex” capable of running one complex world-model simulation at a time. System-1 behaviors (RL-style policies) run subconsciously and in parallel; System-2 planning is serial. The illusion of unified consciousness is the executive process that configures which simulation occupies that single engine.
10. Industrial Implication
Foundation models are becoming a commodity, analogous to Linux. Long-term value will shift from closed base models to open-source community development, applications, and specialized agents.
Summary
LLMs are powerful estimators of token distributions over language abstractions, but grounded intelligence requires action-conditioned latent world models (such as JEPA) that learn equivalence classes of future states under physical invariances, organized hierarchically, and trained to minimize objective-driven prediction error.
