本文由 AI 分析生成
建立時間： 2026-05-25 來源： https://x.com/alin_zone/status/2040018737181630537
Summary
Karpathy open-sourced autoresearch — an automated experiment loop where an AI agent iterates on a single modifiable artifact, measures improvement against a binary eval, keeps changes that improve the metric, and rolls back those that don’t. Originally designed for ML training code, it ran 700 experiments in 2 days and auto-discovered 20 effective optimizations. Someone ported the pattern to Claude Skills (SKILL.md as the artifact), improving a landing-page skill from 56% to 92% accuracy in 4 automated rounds. The Shopify CEO applied the same pattern to the Liquid template engine: 93 auto-commits, 53% faster rendering, 61% less memory.
Karpathy 開源 autoresearch：AI 代理人自動迭代單一目標檔案，以二元評估指標衡量改善，保留有效變更、回滾無效變更。2 天跑了 700 次實驗，自動發現 20 個有效優化。移植到 Claude Skills 後，落地頁技能準確率從 56% 升至 92%，全程自動。
Key Points
- autoresearch loop: change one thing → run test → check metric → keep if improved, revert if not → repeat
- Original constraints: agent can only modify
train.py, fixed 5-minute training runs, val_bpb as the metric - Ported to Claude Skills: SKILL.md is the artifact, yes/no binary eval is the metric (3-6 criteria like “does the headline have a specific number?”, “are there buzzwords?”, “is the CTA clear?”)
- Binary eval quality is the critical dependency — a poorly defined eval causes the agent to optimize the checklist, not real quality
- Risks: eval drift (agent learns to game checklist), overfitting to small test samples, inapplicable to purely subjective creative tasks
Insights
The key reframe is “prompt tuning from gut feel to experiment.” A one-time prompt is a snapshot that ages; an iteratable asset built through autoresearch compounds. The pattern is essentially A/B testing made fully autonomous — the same discipline that runs product experiments applied to prompt/skill optimization. The limitation about binary evals is important: tasks that require holistic judgment resist this method, which is why the examples cluster around measurable outputs (accuracy, rendering speed, memory).
Connections
- agentic-workflow
- claude-code
- How to finally trust Claude’s advice (using Karpathy’s LLM Council method)
- Lessons from Building Claude Code How We Use Skills 1
Raw Excerpt
一次性 Prompt，只会越来越旧。可迭代资产，才会越用越强。 (A one-time prompt only grows stale. An iterable asset grows stronger with use.)