Large-Scale Robot Manipulation Datasets
Sources: Research synthesis, 2026-05-14; Dataset comparison notes, 2026-05-14 Raw: DROID and Teleoperation Datasets Comparison; Robotic Dataset Comparison Table; Robotic Dataset Survey in Vault Updated: 2026-08-12
Overview
Robot manipulation datasets split along the collection method axis: teleoperation (operator controls robot via VR/interface), natural demonstration (human demonstrates without robot), and passive observation (egocentric video of human activity). Teleoperation datasets (DROID, BridgeData V2, RoboMIND, AgiBot World) dominate IL and VLA pretraining. Natural demonstration and passive observation datasets are the growth area for 2025-2026, particularly for dexterous manipulation where teleoperation interfaces are inadequate.
Collection Method Taxonomy
| Method | Examples | End-effector | Scale (typical) |
|---|---|---|---|
| Teleoperation | DROID, BridgeData V2, RoboMIND, AgiBot World | Gripper, mixed | 10k–1M+ trajectories |
| Natural demonstration | UMI, DexWild, GenRobot | Gripper, dexterous | 5k–10k+ |
| Passive observation / egocentric | EgoDex, EgoVerse, VITRA, Xperience-10M | Dexterous | 80k–10M+ clips |
Teleoperation Design Philosophy Taxonomy
Three design schools have emerged:
Scene diversity (DROID): maximize the number of distinct environments to build policies that generalize to novel settings. The bet is that a policy seen in 564 different scenes will handle a 565th.
Task diversity (RoboMIND): maximize the number of distinct tasks to build policies with broad skill coverage. The bet is that 479 tasks covers the space of manipulation better than 86 tasks in more scenes.
Scale (AgiBot World): maximize trajectory count to unlock data-driven scaling behavior. The bet is that 1M+ trajectories, even across fewer scenes, provide enough diversity through sheer volume.
Dataset Profiles
BridgeData V2 (CoRL 2023, UC Berkeley)
- Scale: 60,000 trajectories, 24 environments, 13 skill types
- Robot: WidowX 250 (~$3k), 6-DOF
- Interface: VR controllers
- Significance: Founding document for the democratized, reproducible teleoperation dataset approach. Any researcher could replicate the hardware and collection setup. Core contributor to Open X-Embodiment.
- Limitation: 24 environments are all kitchen/tabletop; WidowX’s 6-DOF limits task complexity ceiling.
DROID (RSS 2024, 13 institutions)
- Scale: 76,000 trajectories (350 hours), 86 tasks, 564 distinct scenes
- Robot: Franka Panda, 7-DOF, 18 units
- Interface: Oculus Quest 2 VR controllers (6-DOF continuous control)
- Camera: Two ZED2 wrist/external cameras + ZED Mini wrist camera (stereo + depth)
- Significance: First truly distributed in-the-wild dataset. 50 collectors across 52 buildings. VR controller interface enabled non-specialist data collection. Three-camera setup (wide + wrist) became a reference standard.
- Limitation: Higher hardware cost than BridgeData (Franka is expensive); task count (86) limited relative to scene count.
RH20T (NeurIPS 2023 Workshop)
- Scale: 110,000+ trajectories, 147 tasks
- Unique feature: 6-DOF force-torque sensor data on every trajectory — rare in public datasets
- Significance: Only major dataset with tactile/force data, making it uniquely valuable for assembly, insertion, and contact-rich tasks where force matters.
RoboMIND (RSS 2025)
- Scale: 107,000 trajectories, 479 tasks
- Robots: Franka Panda, UR-5e, AgileX bimanual, Tiangong humanoid (4 platforms)
- Unique feature: 5,000 annotated failure trajectories (with failure reason classification); Isaac Sim digital twin matching real collection scenes
- Significance: Highest task count of any public dataset; first to include a structured failure demonstration library. Multi-embodiment under a unified collection protocol. Failure data supports safety-aware policy training.
AgiBot World (IROS 2025 Finalist)
- Scale: 1M+ trajectories (2,976 hours), 217 task types, humanoid robot
- Unique feature: Multi-stage human quality verification pipeline; GO-1 policy (latent action representation) trained on this data outperforms Open-X pretrained baseline by 30% on complex tasks
- Distribution: Alpha subset (92k, high-quality curated) + Beta full (1M+); includes depth and language annotations
- Significance: Largest scale publicly available robot manipulation dataset. Quantity-over-diversity tradeoff: 217 tasks vs. DROID’s 564 scenes.
RoboMIND 2.0 (2025, arXiv:2512.24653)
- Scale: 310k+ trajectories, 739 tasks, 6 robot platforms
- Extension over RoboMIND: adds 12,000 tactile episodes (force/tactile sensor data), 20k mobile manipulation episodes, 20k simulation episodes
- Significance: Now includes force/tactile data, making it (alongside RH20T) one of the only public datasets with this modality. Largest public task count at 739.
Natural Demonstration Datasets
Natural demonstration datasets capture humans performing tasks without robot hardware, then transfer demonstrations to robots.
UMI (2024)
- Scale: 10,000+ trajectories, 22 tasks
- Method: Handheld cup-shaped gripper (replicates robot gripper shape) operated freely in the wild
- Significance: No robot required during collection. FastUMI-100K extended version contains 100k+ episodes.
- Best for: High-variety in-the-wild collection by non-specialists; gripper-only (no dexterous).
DexWild (2025)
- Scale: 9,500 trajectories (33 hours), 5 tasks, 93 different environments
- Method: Bimanual dexterous capture with Leap Motion + Manus gloves; 4+ RGB cameras + ZED stereo depth; HDF5 format (2.14 TB on HuggingFace)
- Significance: Currently the most comprehensive open-source bimanual dexterous natural demonstration dataset. 93 environments provides in-the-wild diversity.
- Best for: Dexterous hand research requiring real-world environment diversity.
GenRobot / RealOmni (2025)
- Scale: 1M+ video clips
- Method: DAS (Dynamic Action Space) Gripper / DAS Fingers / DAS Dex — three device tiers including dexterous option
- Data: Freely available on HuggingFace; quantized as clips rather than trajectories.
HoMMI (2024)
- Scale: Unreleased; Stanford + Toyota Research Institute
- Method: Whole-body mobile manipulation using UMI gripper with egocentric sensing (ego + wrist cameras)
- Status: Data not released as of 2026.
Passive Observation / Egocentric Video Datasets
Egocentric datasets capture human hand activity from a head-mounted perspective. They represent the largest available scale but require retargeting to robot morphology. All 2025-2026 major entries focus on dexterous manipulation.
EgoDex (Apple, 2026)
- Scale: 338,000 demonstrations (829 hours), 194 tasks, 90M frames
- Collection: Apple Vision Pro; includes precise 3D hand joint tracking
- Significance: Largest publicly available labeled dexterous egocentric dataset from a commercial device; depth + 3D joints.
EgoVerse (2026, GTech/Stanford/UCSD/ETH/MIT/Meta/Scale AI)
- Scale: 80,000 demonstrations (1,362 hours), 1,965 tasks
- Collection: Smartphone (low barrier)
- Significance: Highest task count of any dataset (1,965). Multi-institution collaboration across academia and industry.
VITRA (Microsoft, 2026)
- Scale: 1M+ episodes
- Method: Post-processes in-the-wild human hand video into VLA-compatible format
- Access: GitHub (microsoft/VITRA)
- Significance: Demonstrates conversion of uncontrolled internet video into robot training data.
Xperience-10M (Ropedia, 2025)
- Scale: 10M demonstrations (10,000 hours), ~1 PB
- Collection: 6-camera rig (4 fisheye + 2 stereo) + IMU + MANO full-body mocap; 16M language captions; 720M stereo frames
- Access: Non-commercial research only; requires DocuSign agreement
- Significance: Largest dataset by trajectory count and hours; stereo depth at scale.
EgoScale (NVIDIA GEAR Lab, arXiv:2602.16710, 2026)
- Scale: 20,000+ hours of egocentric human hand video
- Method: 22-DoF dexterous hand; VLA architecture; not yet released (Coming Soon as of mid-2026)
- Significance: If released, would be largest by duration.
Being-H0 / UniHand-2.5M (2025)
- Scale: 2.5M samples
- Note: This is a VLA pretrained model, not a public dataset — action labels are human hand poses. The underlying dataset is not released.
Scale and Dimension Comparison
Ranked by dataset size (hours):
| Dataset | Hours | Tasks | Dexterous | Depth | Force/Tactile | Open | Year |
|---|---|---|---|---|---|---|---|
| EgoScale | 20,000+ | — | ✓ | ✓ | ✗ | Coming Soon | 2026 |
| Xperience-10M | 10,000 | — | ✓ | ✓ | ✗ | Non-commercial | 2025 |
| AgiBot World | 2,976 | 217 | ✗ | ✓ | ✗ | ✓ | 2025 |
| EgoVerse | 1,362 | 1,965 | ✓ | ✗ | ✗ | ✓ | 2026 |
| EgoDex | 829 | 194 | ✓ | ✓ | ✗ | ✓ | 2026 |
| DROID | 350 | 86 | ✗ | ✓ | ✗ | ✓ | 2024 |
| DexWild | 33 | 5 | ✓ | ✓ | ✗ | ✓ | 2025 |
Force/tactile confirmed: RH20T (6-DOF F/T), RoboMIND 2.0 (partial in original, full in 2.0). LeRobot format confirmed: DROID (✓), BridgeData V2 (✓).
Interface Trends
VR hand-held controllers (Oculus Quest 2, Quest 3) have become the dominant interface across 2024-2025 datasets, displacing SpaceMouse and exoskeletons. Reasons: lower operator training requirement, intuitive 6-DOF control, and accessibility for non-robotics-specialist collectors.
Language annotation was retrofitted into DROID and RoboMIND to satisfy VLA pretraining requirements — originally these datasets used only goal images as conditioning signals. Future datasets will need language annotations from collection time.
Dataset Selection Guide
| Research Goal | Best Dataset | Reason |
|---|---|---|
| Real-environment generalization | DROID | 564 scenes, in-the-wild diversity |
| Broad task coverage | RoboMIND | 479 tasks, widest skill variety |
| VLA pretraining at scale | AgiBot World | 1M+ trajectories |
| Low-cost replication | BridgeData V2 | WidowX hardware accessible |
| Contact-rich/assembly tasks | RH20T | Only dataset with force-torque data |
| Safety / failure learning | RoboMIND | 5k annotated failure demonstrations |
| Multi-embodiment generalization | RoboMIND / Open X-Embodiment | Cross-platform coverage |
Open Questions
- Does AgiBot World’s 1M scale already supersede Open-X as the preferred VLA pretraining base?
- Do RoboMIND’s 5,000 failure demonstrations provide measurable improvement in practice? No published systematic comparison exists.
- Does humanoid robot data (RoboMIND Tiangong, AgiBot World) transfer to standard 6-DOF arm policies?
- What comes next? Likely: bimanual operation datasets, tactile sensing integration, and open data flywheels (contributor networks resembling LAION for images).