本文由 AI 分析生成
建立時間： 2026-07-30 來源： https://lilianweng.github.io/posts/2026-06-24-scaling-laws/
Summary
Lilian Weng reviews the history and nuances of neural scaling laws, reconciling the apparent Kaplan (2020) vs. Chinchilla (2022) conflict and extending the framework to data-limited regimes. The post is a practitioner’s guide to fitting pitfalls—what to hold constant, how to avoid precision artifacts—and shows how to reason about training when token repetition is unavoidable.
Lilian Weng 回顧神經網路縮放律的歷史與細節，協調 Kaplan（2020）與 Chinchilla（2022）的表面衝突，並將框架延伸至資料受限情境。本文是一份關於擬合陷阱的實用指南，說明如何在不可避免的 token 重複訓練下推斷最佳分配。
Key Points
- Kaplan (2020): N_opt ∝ C^0.73 (grow model faster than data); based on 768M–1.5B parameter models
- Chinchilla (2022): N_opt ∝ C^0.5 (equal scaling); based on 400+ models up to 16B+ parameters
- Reconciliation (Pearce & Song 2024): embedding parameter count inflates N in the small-model regime; excluding embedding params makes Kaplan’s local exponent g converge to Chinchilla’s in the large-scale limit
- Data-limited extension (Muennighoff 2023): token value decays exponentially with repetition — D’ = U_D + U_D·r_D·(1−exp(−R_D/r_D))
- Data-limited extension (Lovelace 2026): adds explicit penalty term P·R_D^δ·(N/U_D)^κ for repeated tokens
- Fitting pitfalls: precision rounding errors, loss averaging vs. summing, sensitivity to fit region, must hold architecture and optimizer constant across runs
Insights
The embedding-parameter exclusion fix (Pearce & Song) has practical impact for anyone fitting scaling laws on small models: embedding params dominate parameter count at small scale, making small models appear more parameter-efficient than they are, which explains why Kaplan’s small-model results diverge from Chinchilla’s large-scale results. The data-limited extensions matter as internet-scale data runs out—they provide a principled way to decide whether to repeat tokens (and how many times) or stop training before exhausting the repetition budget. The fitting pitfalls section is particularly valuable for ML practitioners who run their own scaling experiments.
Connections
- llm-training
- How To Train Your Own ChatGPT from Scratch (Complete Builder’s Guide) — Practical small-scale training at the extreme low end of scaling curves
- research
Raw Excerpt
“Scaling laws are one of the most critical empirical findings in deep learning. The observation is simple in form: the training loss L decreases predictably as we scale up model size N, dataset size D, and compute C, following a power-law curve, which appears as a straight line on a log-log plot.”