My X feed might be cooked with just robotics, but I find myself and others arguing constantly about which data actually matters for robot intelligence. Egocentric video. UMI. Teleop. Rollouts. Internet-scale video scraped off YouTube. Every one has a camp, and the camps argue past each other: partly out of conviction, but mostly because of what they’re funded to believe. Nobody has put them on the same map. The field has a version of it. The Data Pyramid.
Check out @tanayj and his breakdown of the Robotics Data Pyramid at https://www.tanayj.com/p/the-robot-data-pyramid
Analyse the pyramid with me. Width is scalability and Height is fidelity. Internet video at the base because there’s an endless supply of it, deployment at the apex because there’s almost none. The shape is the argument: a lot of the cheap stuff, a little of the expensive stuff, roughly in the proportions drawn. Some mixture of this must work. So most opportunists think lets go all in on ego data and make some money.
7月28日
i think this framing is valid if you left it at fidelity but i’m wondering how you define scalability? simulation should be strictly more scalable than egocentric video data because theres no human time limited scaling law right?
This reply got me thinking that simulation doesn’t fit on that axis at all. How is it getting squished between Human and Robot Data? It’s more scalable and it can be higher fidelity. Sim has perfect state, perfect actions, perfect labels, and effectively infinite volume. By any reading of the two axes it belongs at both ends at once. The pyramid has no room for that, so sim gets shoved somewhere arbitrary and everyone moves on.
This framework produces a nonsense ranking but isn’t ranking wrong. It’s measuring more than one thing and reporting a single number.
Distance
Break the sources down and there’s a real axis underneath.
Ego video gives you the first-person view of a human doing a task. How they approach an object, what they reach for, what order they do things in. Priors, affordances, context. This is genuinely valuable and it’s going to stay valuable.
It has no actions. And there’s no clean way to add them, for two separate reasons. Most deployed robots are parallel-jaw grippers, so there’s nowhere for five human fingers to go. Multi-fingered hands exist¹ but aren’t mainstream enough to change that soon. The gap isn’t annotation effort. There’s no shared action space to annotate into.
UMI-style capture moves closer. Still first-person, still a human doing the task, but the end effector in frame is the robot’s embodiment: a handheld gripper matching what’s actually deployed. The human’s hand stops mattering. What gets recorded is already in the robot’s action space, so retargeting is close to free.²
Teleop moves closer again. Now it’s the full embodiment. Forward kinematics, joint limits, reachability, the actual arm solving the actual task. Nothing is left to translate.
That’s the real axis under the pyramid: action-space distance. How far the captured data sits from the thing the robot will execute. Ego is far. UMI is closer. Teleop is zero.
The Fence
Look at the diagram above again. For all data types above the dotted line, a human is acting.
Ego, UMI, teleop, are all different distances, same acting agent. Every frame is a person’s choice about what to do next. The distribution you collect is the human operator’s distribution.
Rollouts are different in acting agent, not action space. The policy acts. What gets recorded is what the model does. The successes, the failures, and the random jittery motions, all of it.
This is a hard line. Teleop sits at the top of the ladder but on the wrong side of the fence.
Distance tells you how close the data is to the embodiment. The fence tells you whose behavior it is.
Where does Simulation Fit?
Sim never fit the pyramid because it was never on it. It’s the same map, mirrored. The physical world on one side, simulated on the other. Atoms and bits.
So the map has two binary splits, not one. Acting agent and substrate. Four quadrants, all of them can exist. Physical human demos, real-world rollouts, simulated teleop, simulated rollouts. Which means you can simulate any data type. Sim ego, sim UMI, sim teleop, sim rollouts.
Most of those are pointless. Simulating ego or UMI buys nothing when the real version is cheap to collect. Sim earns its cost at the expensive end.
On the human side of the fence, that means calibration. Simulate teleop, compare it to real teleop, and you get a direct read on whether your sim and your world agree because you have both versions of the same source.
On the policy side it stops being a check and becomes load-bearing. Rollouts are the only physical source past the fence, and every sample burns a real robot doing something real, often something wrong. Sim is how you get volume of on-policy behavior without paying that.
This doesn’t work today. The Sim2Real gap is an active blocker. But the alignment work on the human side is exactly what would earn the right to trust the policy side.
My Bet: Other side of the Fence Wins
You find out a policy is broken by running it and watching it fail.
That failure data is the most valuable thing in the pipeline and it only exists on one side of the fence. No amount of human demonstration produces it. A teleoperator doesn’t make the mistakes your policy makes. Scale ego video to a million hours and you still won’t have a single example of your model reaching for the wrong bin.
And that’s the data that actually changes how a robot moves. Everything on the human side teaches a policy what a task looks like. Only rollouts tell it what it gets wrong. Which is the signal that turns a demonstration-shaped model into something that recovers.
Real-world rollout data is valuable but scarce and unscalable. They can’t supply their own volume. Something else must supply that volume while maintaining policy operated distribution. It is Sim Rollouts.
The data pyramid is compressing 3 properties on 1 axis. The Robotics Data Map actually looks closer to this:
Distance hits zero exactly at the fence. Rollouts are at zero by construction, because the policy’s own action space is what generated them. The fence is the origin from where distance is measured.
I truly believe the dotted line shows the scaling path. It runs from Ego -> UMI -> TeleOp -> Sim Rollouts (crossing from atoms to bits, human to policy). And this skipping of dimensions is quite literally where the Sim2Real Gap lives. Now to put it in perspective. Everyone is buying hours on the far side of the first quadrant, furthest distance from robot action space and calling it progress.
- Check out @mimicrobotics to see cool multi-fingered hands
- Cheng Chi (@chichengcc) , who built UMI, went on to co-found Sunday Robotics, which trained a humanoid on laundry: more about that in ACT-2. . UMI misses the forward kinematics of the embodiment.