本文由 AI 分析生成
建立時間： 2026-09-17
來源： https://x.com/walterzhu8/status/2100255999365964113?s=46&t=a6mO9d6LiCSXZ-_SiOkjIQ
Summary
An X thread (adapted from an EIT HAI group talk) analyzing what GPT-6 Astra’s 3D/inverse-graphics and computer-use abilities imply for embodied AI. The author frames both code agents and embodied agents as instances of one “perception-action loop,” and argues the field’s remaining bottlenecks split cleanly into infrastructure (simulator fidelity, sensor hardware, hardware control) versus model/connector problems (4D reasoning, reading new signals, emitting high-DoF actions).
一篇 X 討論串(改編自 EIT HAI 組會演講),分析 GPT-6 Astra 的 3D/逆圖形學與電腦操作能力對具身智能(embodied AI)的意義。作者將 coding agent 與 embodied agent 都視為同一種「感知-行動迴路」的實例,並主張該領域剩下的瓶頸可清楚分為兩類:基礎設施類(模擬器精確度、感測器硬體、硬體控制)與模型/連接器類(4D 推理、讀取新訊號、輸出高自由度動作)。
Key Points
- Astra’s headline 3D capability is inverse graphics: given an image/video, write (Blender) code, render, inspect, correct, repeat — structurally the same loop as a coding agent, just running inside a 3D engine instead of a compiler/sandbox.
- Inverse physics (recovering mass/friction/damping etc. so a simulation reproduces motion, not just appearance) is the harder unsolved piece; some parameters are only observable once the agent acts on the object and feels the response, so this loop cannot stay purely inside a simulator.
- For embodied AI, a strong foundation model can already act as an orchestrator over lower-level tools/SDKs, and the level of abstraction it operates at keeps dropping — from calling policy/navigation models to producing direct control signals (e.g. end-effector positions) itself.
- Predicted near-term path: SFT on robot/human interaction data (absorbing VLA into the base model) followed by RL through real or simulated physical interaction — “perception-action loop is all you need.”
- Four bottlenecks are named and split by kind: (1) 4D reasoning [model], (2) simulator fidelity [infrastructure], (3) richness of sensing incl. tactile [infrastructure/hardware], (4) expressiveness/DoF of the action space [infrastructure + model-connector].
- Author’s practical takeaway: pure data-driven specialized-model training may be “already overrun”; the remaining open ground is lower-level — rendering/physics algorithms for the 3D side, hardware and control for the embodied side.
Insights
The framing that a coding agent and a Blender-scripting agent and a robot policy are all the same perception-action loop, just with different environments and action spaces, is a useful lens for the vault’s VLA/LfD interests: it suggests “embodiment” is not a special category requiring a fundamentally different architecture, but rather a case where the environment (physical world) imposes harder-to-observe state (contact-rich dynamics, tactile) and a richer action space (high-DoF control) than a code sandbox does. This reframes the VLA research question as largely an infrastructure problem (simulator fidelity, sensors, low-level control APIs) rather than a modeling problem — which has implications for where individual researchers (vs. frontier labs with Astra-scale compute) can still contribute: down at rendering/physics algorithms or hardware/control, rather than at training bigger foundation models.
Connections
Raw Excerpt
Perception-action loop is all you need. For an agent, this is the fundamental formulation: perception, action, and a closed loop with an environment. With that loop and an environment, the agent can keep improving through RL at training time, or by iterating and correcting itself at test time (which I call agentic scaling).