How To Train Your Own ChatGPT from Scratch (Complete Builder's Guide)

本文由 AI 分析生成

建立時間： 2026-07-30 來源： https://x.com/Av1dlive/status/2070548049680892003

Summary

@Av1dlive documents training a personal LLM from scratch using Karpathy’s nanochat for approximately $100 on an 8×H100 cloud node. Two phases—pretraining on ClimbMix and SFT on personal data (chat exports, notes converted to Q&A, synthetic persona conversations)—produce a personalized assistant in roughly 3 hours of compute. Key practical findings: oversampling personal data at ×3–4 is optimal, and retaining SmolTalk in the fine-tuning mix prevents catastrophic forgetting.

@Av1dlive 記錄了使用 Karpathy 的 nanochat 在 8×H100 節點上以約 $100 美元訓練個人 LLM 的全程。兩個階段——在 ClimbMix 上預訓練加上個人資料的 SFT——約 3 小時即可完成。關鍵發現：個人資料的過採樣倍率 ×3–4 最佳；保留 SmolTalk 在微調資料集中可防止災難性遺忘。

Key Points

• Total cost: ~$48–$100 on a cloud 8×H100 node (~3 hours pretraining + minutes for SFT)
• Pretraining: ClimbMix dataset, 240 shards, ~24GB; depth-26 model; CORE score target >0.256525
• SFT data sources: 3,800 AI chat exports + notes converted to Q&A via LLM + 1,500 synthetic persona conversations
• Oversampling ratio: ×3–4 is optimal (×2 too faint, ×6 causes memorization)
• Must retain SmolTalk alongside personal data to prevent catastrophic forgetting of general capabilities
• Always download checkpoints to local machine before terminating the cloud instance

Insights

The oversampling finding (×3–4) is practically useful and difficult to find documented elsewhere—it gives a concrete starting point for personal data SFT without expensive ablation runs. The catastrophic forgetting warning (keep SmolTalk) reflects a common pitfall in fine-tuning: removing general-purpose data to increase personal data density destroys the base model’s capabilities. The $100 cost threshold makes personal model training genuinely accessible to individual developers for the first time.

Connections

• Scaling Laws, Carefully — Practical illustration of training dynamics at the extreme low end of scaling curves (depth-26 models)
• llm-training
• personal-ai

Raw Excerpt

“You don’t need billions to train the next ChatGPT. All you need is a $100 and Andrej’s Karpathy’s Nanochat. I used it for the last week here is what I found.”