There is a popular view that if you collect enough human ego-data and throw it at a robot, the problem is solved. You would believe it too, if all you saw was the GEN-1 demo reel claiming 500K hours of human data and 1h of task-specific data without going into details about data engineering. Thank god for the Physical Intelligence team putting out the π0.7 paper.
A small aside on the title. “Emergent Capabilities” is overselling. Most of the gains in the paper come from specific, nameable engineering choices.
- All kinds of data (but not sim data)
- You have PD control loops for joint angles.
- Inverse kinematics for end effectors
- Affordances, Affordances, affordances: subgoal images. Task breakdowns
- Dropouts during training but on affordances & instructions
The data
π0.7 uses every kind of data except sim data. Teleop demos, autonomous rollouts, RL specialist trajectories, outright failures, egocentric human video, web data.
The heavy lifting happens in the prompt itself. Subgoal images, subtask instructions, episode metadata. All of them get dropped out during training so the model learns to cope with any subset at test time.
Why naive scaling does not work
Annotation density of how the episode went is what unlocks scale.
Without metadata, naive training averages across conflicting strategies. The paper’s phrase is “averaging together different behaviors.” You pour in more data, the model gets worse, because the strategies inside the data disagree with each other. Scale without context is a curse.
Adding a data-quality score to the metadata during training is the fix. You can now pour in a bigger, messier dataset and counter the data-quality curse at the same time. Metadata carries the signal that tells the model which examples to imitate and which to merely learn the state distribution from.
Subgoal conditioning as a bootstrapped World Model
Running a world model to roll out full future trajectories is expensive. π0.7’s workaround is to discretise this continuous future. Predict a single future frame, the subgoal, and condition the policy on that.
With a SuSIE-style subgoal image in the prompt, action prediction stops being open-loop planning and becomes inverse dynamics: “what action gets me from the current observation to this future observation?” The conditioning collapses the hypothesis space.
Why instruction following finally works
People have complained that π0 and π0.5 broke the moment you changed “drop x on the left” to “drop x on the right,” because most of the training data drifted one direction. The policy learned the task distribution, not the language.
π0.7’s answer is not just more linguistically diverse tasks. It is that subgoal image conditioning has a stronger effect than language in overriding the task prior. You tell the model what the world should look like after the action, and it will get there even when the language says something the training data rarely said.
Cross-embodiment transfer
The claim is “no task-specific data.” It does not mean no embodiment data. The policy saw the target robot doing other things. It just never saw that robot doing this task. Nevertheless pretty cool and ties into compositional generalization.
Compositional generalization is the north star
This is the goal the whole field is now chasing. Russ Tedrake has been campaigning it for years. Physical Intelligence, Generalist, Skild, Sunday, basically every serious robotics company is pointing at the same target. The disagreement is only about which data source gets you there. π0.7’s bet is that the answer is not more data of one kind, but much better-annotated data of many kinds. As a robot data company, you need to start building and training your own robots, and then make your learnings, annotations and models available. Only then the companies will buy data from you. Even Nvidia had a build an end-to-end selfing driving model and demo car in 2016 to sell their GPUs and services auto industry.