Before you scale multi-modal, get this right. Physical AI is racing toward sensorimotor human data. Synchronization is the foundation underneath. It hasn’t been built.
The Bet on Human Data
Physical AI is shifting fast toward training on human behavioral data. UMI (Stanford / Columbia / TRI, RSS 2024) showed that manipulation policies can be trained entirely from human demonstrations, no robot needed. EgoVerse (2026) scaled this to 1,362 hours, 80,000 episodes, and 2,087 demonstrators, showing that policy performance scales with the amount of human data. Humans are general-purpose robots deployed at a scale of 8 billion. Just as LLMs learned from humanity’s accumulated text, Physical AI will learn from humanity’s accumulated behavior. The difference: text already existed on the internet. Behavioral data must be collected from scratch.
World models raise the bar further. Learning physical causality requires multiple viewpoints and modalities capturing the same scene simultaneously. Video prediction is emerging as the core paradigm, with models like Meta’s V-JEPA 2 showing that video-scale human data can be distilled into genuine physical understanding. For all of these, temporal quality directly determines the accuracy of the physical understanding the model can learn.
The Problem Most Teams Haven’t Hit Yet
Right now, most teams collecting human data for Physical AI are capturing single-stream egocentric video. One camera, one file, no sync needed.
But the field is moving toward multi-modal sensorimotor data. Ego camera plus wrist camera plus IMU plus tactile sensors plus force feedback. This is where the real value lies, because manipulation is not just about what you see. It is about the relationship between vision, proprioception, and contact.
The moment you add a second sensor, synchronization becomes your problem. And most teams will discover this the hard way, because nothing in their pipeline will warn them that it is broken.
The sync errors described in this post are not edge cases that surface occasionally. They are structural. They exist in every multi-device recording by default, and they will silently corrupt your training data unless explicitly solved. This is the danger waiting in multi-modal human data collection, and the field is walking toward it.
Why Human Data Has This Problem (and Robot Data Doesn’t)
In robot data collection, synchronization is barely a concern. A single controller governs all sensors. Joint encoders, force/torque sensors, cameras: all wired to one host, sharing one system clock. ROS provides a unified time reference. Synchronization is built into robot systems by design.
Human data is structurally different, and the reason is not a technical limitation. It is a prerequisite for good data.
Human data is valuable because people behave naturally. Using tools, opening drawers, moving objects without constraint. To capture this, the setup must not interfere with movement. Cables, heavy rigs, tethered sensors all destroy the naturalism that makes the data worth collecting.
So human data collection takes this form:
- Lightweight egocentric camera on the head (GoPro, Project Aria)
- Wrist-mounted camera
- IMU sensors on body or hands (wireless BLE)
- External tripod cameras (third-person view)
- Optionally, tactile sensors, eye trackers
clock domain difference between robot and human
Each device has its own hardware, its own clock, its own storage. They cannot be wired to a single host without destroying the freedom of movement that makes the data valuable. The paradox: the properties that make human data worth collecting are exactly what create the synchronization problem.
Why Most Teams Underestimate Sync
Most teams assume synchronization is straightforward. Here is why every common approach falls short.
“NTP will handle it”
NTP accuracy: ±1–10ms on LAN, ±10–50ms on WiFi. At 30fps, one frame is 33.3ms. WiFi NTP carries 1–2 frames of error by default. More fundamentally, NTP synchronizes host wall-clocks. GoPros do not use NTP. BLE IMUs are not network-connected. Without a common time reference between devices, NTP has nothing to align.
“Just compare timestamps”
“Timestamp” means different things on different devices. GoPro stamps frames against its own internal clock, never exposed to any host. iPhone CMSampleBuffer presentation timestamps live on the iOS media clock. BLE IMU samples carry their own sensor clock, unrelated to host receive time. Even if every device reports a timestamp, those timestamps live in different clock domains and cannot be compared directly. Resolution isn’t the problem, the absence of a shared reference is.
“Use PTP or hardware sync”
PTP achieves ±microsecond precision, but requires dedicated hardware, doesn’t work over WiFi, and costs thousands per setup. Hardware sync (genlock) works between industrial cameras but not on consumer wearables. Precise sync technology exists. It is incompatible with human data collection.
“30fps means 33.3ms intervals”
It doesn’t. OS scheduling, USB contention, and sensor exposure variance make frame intervals vary. Many consumer cameras are effectively VFR. Computing time = frame_number × (1/fps) introduces per-frame errors that accumulate differently on every device.
Why Synchronization Is Hard: Four Core Problems
Multi-modal synchronization decomposes into four distinct problems. Each has a different physical cause and requires a different solution.
4 core problems of multi-modal data synchronization
Problem 1) Offset: The time axes start at different points
Each device defines “time zero” independently. To compare them, you need the offset between their axes. On the same host, a shared system clock makes this possible. Across hosts (body-worn cameras vs. external tripods on separate machines), no shared reference point exists. Asking “what time is it?” over a network doesn’t help, because the network itself introduces timing uncertainty on every packet.
Problem 2) Drift: The time axes run at different speeds
Even after correcting offset, crystal oscillators vibrate at slightly different frequencies (±10–100 ppm for consumer-grade). At 40 ppm relative drift: ~24ms after 10 minutes, ~72ms after 30 minutes, ~144ms after an hour. Drift is an error that grows with time. It requires rescaling the time axis, not just shifting it.
Problem 3) Jitter: Frame intervals are not uniform
Even after correcting offset and drift, frame_number × (1/fps) is wrong. Actual capture intervals vary per frame due to OS scheduling, USB contention, and sensor exposure variance. Many cameras are effectively VFR. You need actual per-frame timestamps, not nominal calculations.
Problem 4) Rate Mismatch: Streams run at different frequencies
A 30fps camera, a 200Hz IMU, a 1000Hz tactile sensor. By what standard do you even define “synchronized”? Do you anchor everything to the lowest rate, or the highest? A single video frame overlaps ~7 IMU samples and ~33 tactile readings. Which samples “belong” to that frame, and how do you map one to the other?
This is the frame mapping problem.
Compounding Errors
These four problems do not occur in isolation. They compound.
In a typical setup (ego cam, wrist cam, two external cameras, BLE IMU), every stream pair faces a different combination. Same-host cameras: drift + jitter + rate mismatch. Cross-host cameras: add offset. Camera-to-sensor: all four.
At scale, variance explodes. EgoVerse-level collection: 2,087 demonstrators, different devices, different environments, different firmware. No single calibration works everywhere.
Foundation models ingest this data wholesale, unable to distinguish temporally corrupted samples from clean ones. Contact events in manipulation happen on the order of tens of milliseconds. World models need precise temporal correspondence between visual contact and force/IMU readings to learn correct physical causality. In this regime, tens of milliseconds of misalignment shifts the apparent cause-and-effect a model sees during training, particularly for contact-rich tasks where the signal window is itself only tens of milliseconds wide.
All of this is invisible. Human perception tolerates tens of milliseconds of audio-visual asynchrony without noticing (playback looks fine). But models learn pixel- and sample-level correspondence, encoding misalignment as pattern. The feedback loop from collection to training failure is long enough that sync is rarely identified as the cause.
The Case for Infrastructure
Today, most teams handle sync through ad-hoc methods: handclaps, LED flashes, per-project scripts. These work at small scale. They do not survive production.
The cost is real. Undetected misalignment causes policy failures that take weeks to diagnose. Scripts break when device setups change. Quality is assumed, never verified. When multiple sites contribute to a shared dataset, there is no standard for temporal consistency.
Some groups have invested seriously. UMI built a pipeline around GoPro GPMF telemetry. Meta built Project Aria with custom timing hardware. But UMI works only with GoPros. Aria depends on purpose-built hardware with timing subsystems most teams cannot replicate.
Most teams have no choice but consumer devices. No general-purpose synchronization infrastructure exists for them.
What is missing is not another script. It is an infrastructure layer: device-agnostic, producing verifiable alignment with per-stream confidence metrics, handling mixed sampling rates, scaling from one session to thousands of hours without manual intervention.
Everything built on human data (foundation models, world models, manipulation policies) depends on the reliability of this layer.
We have spent the past year building it. In the next post, we share how.
OpenGraph Labs