本文由 AI 分析生成
建立時間： 2026-05-30 來源： https://x.com/carnot_cyclist/status/2041479655035679163
Summary
A principled framework for defining continual learning (CL) in LLMs, arguing that most existing approaches (self-distillation, replay, regularization, KV caches) fail to solve the right problem. The author (from Prime Intellect) proposes five desiderata: preserving general capabilities, sequential (not multi-task) learning, handling distribution shift, efficiency (no full-data replay), and compositional transfer between skills learned at different times. The core argument is that in-weights parametric learning has compounding potential that harness-based memory cannot match at scale.
為 LLM 持續學習提出五項原則性要求的框架，批評現有方法未解決正確問題。核心論點是：參數化知識（in-weights）具有複利潛力，而基於外掛的記憶（RAG、SKILL 文件）在規模化後會因上下文腐化和檢索瓶頸而遇到天花板。
Key Points
- Five desiderata: (1) preserve general capabilities, (2) sequential learning (not multi-task), (3) handle distribution shift, (4) efficiency (no full-replay), (5) compositional transfer across time-separated skills
- Harness-based memory (RAG, SKILL files) hits a ceiling: growing knowledge bank → context rot + retrieval difficulties → diminishing returns
- In-weights learning enables automatic cross-domain recombination (e.g., coding knowledge aids general reasoning without explicit retrieval)
- Cartridges (corpus-specific KV caches) is principled but acts like external neural memory at scale — compositional properties with 1000 KV caches are unknown
- “Automaticity” argument: neural knowledge enables intuitive recombination that retrieval-based systems cannot replicate for creative/cross-domain tasks
- Key open question: can scaffold training (agentic RL with external harness) achieve parametric CL properties without modifying weights?
Insights
The harness-vs-weights debate is framed here in terms of “scaling constructively vs. destructively” — a growing knowledge base should make the system smarter, not slower. The harness approach scales destructively because retrieval quality degrades as the knowledge base grows. This is the fundamental tension between Claude Code’s SKILL.md-based architecture (currently optimized) and truly continual systems. The “automaticity” argument (coding knowledge improves general reasoning without code-specific retrieval) is well-grounded in the cognitive science literature on expertise and represents the strongest theoretical case for in-weights learning.
Connections
Raw Excerpt
Parametric knowledge fundamentally changes the amount of intelligence per forward pass, which holds significantly greater compounding potential.