本文由 AI 分析生成
建立時間： 2025-12-25 來源： https://x.com/ahall_research/status/2003892036001566871
Summary
Inspired by Karpathy’s “LLM council,” the author ran a small experiment testing four governance procedures for multi-model decision-making, evaluated on GSM8K and TruthfulQA with cheap models. The procedures were: (1) majority vote with no communication, (2) vote then deliberate then re-vote, (3) vote-deliberate-revote then a Chairman decides, and (4) vote, evaluate each other, then Chairman decides (Karpathy’s setup). All four beat the best individual model, but option 2 — voting with deliberation — performed best, though without statistical significance. The author notes prior CS literature (e.g., Du et al. 2023) already shows councils outperform individuals and calls for a “science of collective decision-making with AI agents.”
受 Karpathy「LLM 議會」啟發，作者做了一個小實驗，測試四種多模型決策的治理程序，並在 GSM8K 與 TruthfulQA 上以廉價模型評估。四種程序為：(1) 無溝通多數決、(2) 投票後審議再投票、(3) 投票審議再投票後由主席裁決、(4) 投票互評後由主席裁決（Karpathy 的做法）。四者皆勝過最佳單一模型，但選項 2（含審議的投票）表現最佳，惟未達統計顯著。作者指出既有 CS 文獻（如 Du et al. 2023）已顯示議會優於個體，並呼籲建立「AI 代理集體決策的科學」。
Key Points
- Four council governance procedures tested on GSM8K (math) and TruthfulQA, using cheap models to limit cost.
- All four procedures beat the best single model.
- Option 2 (vote → deliberate → re-vote) performed best; no Chairman needed for the win.
- Results not run long enough for statistical significance — explicitly caveated.
- Prior work (Du et al. 2023) already established that councils outperform individuals; framed as an opening for social-science governance research.
Insights
The standout finding is that deliberation, not hierarchy, drove the gain — adding a Chairman (options 3 and 4) did not beat plain deliberative re-voting, suggesting the value comes from cross-model information exchange rather than an arbiter. This connects multi-agent AI to collective-intelligence and voting-theory literature, hinting that mechanism design from political science may transfer directly to agent orchestration. The honest “I didn’t discover anything new” framing makes it a useful pointer to the established multi-agent-debate literature.
Connections
Raw Excerpt
it looks like all four approaches do a little bit better than the best model on its own, but option 2, voting with deliberation, does the best.