DexWild: Dexterous Human Interactions for
In-the-Wild Robot Policies
Tony Tao∗ , Mohan Kumar Srirama∗ , Jason Jingzhou Liu, Kenneth Shaw, Deepak Pathak
Carnegie Mellon University
∗ Equal contribution
In-the-Wild Human Data Generalize to Unseen Environments
Novel Objects
Novel Scenes
arXiv:2505.07813v1 [cs.RO] 12 May 2025
️ Cross-Task
Cross-Embodiment
DexWild
Limited Robot Data
Fig. 1: DexWild enables dexterous policies to generalize to new objects, scenes, and embodiments. This is achieved by leveraging large-scale,
real-world human embodiment data collected in many scenes and co-trained with a smaller robot embodiment dataset for grounding.
Abstract—Large-scale, diverse robot datasets have emerged as challenge. While there have been many breakthroughs in large
a promising path toward enabling dexterous manipulation policies language models (LLMs) [53, 51, 3] and vision language
to generalize to novel environments, but acquiring such datasets models (VLMs) [24, 48], the key to their success lies in
presents many challenges. While teleoperation provides high-
fidelity datasets, its high cost limits its scalability. Instead, what if harnessing vast datasets. In contrast, robotics faces a critical
people could use their own hands, just as they do in everyday life, hurdle: large-scale, diverse robot datasets needed to train
to collect data? In DexWild, a diverse team of data collectors uses foundation models do not yet exist.
their hands to collect hours of interactions across a multitude In recent years, a key approach to collecting robot datasets
of environments and objects. To record this data, we create has been through teleoperation, which provides high-precision,
DexWild-System, a low-cost, mobile, and easy-to-use device. The
DexWild learning framework co-trains on both human and robot high-quality action data that a policy can directly train on.
demonstrations, leading to improved performance compared to [8, 21, 54]. However, acquiring this data requires highly-
training on each dataset individually. This combination results in trained human operators working with specialized robot setups.
robust robot policies capable of generalizing to novel environments, Gathering data in diverse environments presents additional
tasks, and embodiments with minimal additional robot-specific challenges such as physically relocating the robot to each new
data. Experimental results demonstrate that DexWild significantly
improves performance, achieving a 68.5% success rate in unseen location. This data collection process is both labor-intensive
environments—nearly four times higher than policies trained and expensive, making it difficult to scale to the volume of data
with robot data only—and offering 5.8× better cross-embodiment needed for dexterous generalization in unseen environments.
generalization. Video results, codebases, and instructions at https: Another approach to scaling robot datasets is to leverage
//dexwild.github.io internet-scale video data from platforms like YouTube, which
provide vast and diverse visual grounding in real-world environ-
I. I NTRODUCTION
ments [15, 10]. However, utilizing this data effectively presents
Roboticists have long dreamed of creating robots that can significant challenges. First, publicly available videos often
perform tasks with the same dexterity and adaptability as lack the fine-grained accuracy needed to capture detailed hand
humans. We would like robots to deftly generalize to many states because vision-based body detection modules are noisy
different objects, environments, and embodiments-yet this and unreliable. Additionally, these videos are not inherently
vision of truly versatile robot behaviors remains a formidable structured with categorized episodes for task-specific learn-
ing, further complicating their direct application in robotics. B. Data Generation for Robot Manipulation [18, 1, 40]. While some data collection efforts exist with more Overcoming the robot data bottleneck has become a central accurate and structured data, [60, 2], they do not have enough challenge in robot learning. environment diversity. We seek to collect data with tracking One approach leverages internet videos to extract action accuracy and environment diversity to enable generalizable information. Several works, such as VideoDex [40] and HOP dexterous behavior. [42], utilize large scale human videos to learn an action prior To overcome these barriers, some have explored collecting through retargeting, which they use to bootstrap policy training. accurate in-the-wild human demonstrations by equipping Others, such as LAPA [57], use unlabelled videos to generate users with a wearable gripper that directly maps their hand latent action representations that can be used for downstream movements to robot actions [7]. However, this approach is tasks. While these video-based schemes enjoy vast visual cumbersome, ill-suited for natural, everyday interactions, and diversity, they typically fall short at capturing the precise, low- constrains the collected data to a specific embodiment. Otherlevel motor commands needed for real-world manipulation. works [55] propose using dexterous hands and gloves, but they Simulation enables rapid generation of action data at scale. do not scale to in-the-wild environments. However, creating diverse, realistic environments for many In this paper, we present DexWild, a system that enables tasks and addressing the sim-to-real gap is challenging. Recent effective learning of robust dexterous manipulation policies successes in transferring manipulation policies from simulation through co-training on human and robot demonstrations. Our [43] have been confined to tabletop settings and lack the key contributions include: generalization needed for deployment in diverse environments. Direct teleoperation on physical robots yields the highest
-
Scalable Data Collection System: A novel human- fidelity, but scales poorly. Recent works have shown impressive embodiment DexWild-System that enables untrained op- dexterity and efficient learning in fixed scenarios [59, 56, 41, erators to quickly collect 9,290 demonstrations across 19], yet collecting enough demonstrations to generalize across 93 diverse environments, achieving 4.6× speedup over diverse scenes quickly becomes prohibitively expensive. conventional robot-based methods Recently, there has been a growing body of work that utilizes
-
Efficient Co-training Framework: An approach that purpose-collected high quality human embodiment data without optimally combines human and robot demonstrations, the tedious teleoperation. We discuss these approaches in the significantly improving policy generalization to achieve next section. 68.5% success rate in novel environments, nearly four times higher than robot-only policies. C. Human Action Tracking Systems
-
Strong Cross Embodiment and Cross Task Perfor- In order to acquire high-quality data from human motions, mance: Our data collection system combined with our accurate hand and wrist tracking is of paramount importance. To co-training framework achieves of 5.8× improvement in bypass the complexities of hand pose estimation, several works cross-embodiment transfer over baselines and effective equip users with handheld robot grippers [7, 12, 46]. While skill transfer across tasks. this approach simplifies retargeting, it constrains users to the specific morphology of the robot gripper, limiting the diversity II. R ELATED W ORKS of captured behavior. Moreover, many of these systems rely on SLAM-based wrist tracking, which can fail in feature-sparse A. Generalization for Imitation Learning environments or when occlusions occur [7, 23]—such as during drawer opening or tool use. Learning generalizable policies for robot manipulation has Other approaches aim to estimate both hand and wrist poses seen rapid progress, driven largely by advances in visual directly from visual input [29, 35, 5, 45, 28, 20, 32]. These representation learning and imitation learning from large-scale methods are easy to deploy and require no instrumentation, but datasets. On the visual side, embodied representation learning their performance degrades significantly under occlusion—an has benefited from egocentric datasets such as Ego4D [15] and unavoidable situation in manipulation. Alternative strategies EPIC-KITCHENS [10], with recent methods [27, 11, 47, 39] for wrist tracking, such as IMU-based [9, 50] and outside- leveraging these datasets to train scalable visual encoders. in optical systems [30], come with their own limitations: However, these approaches still require substantial downstream IMUs are lightweight and portable but prone to drift, while robot demonstrations to train control policies. optical systems are accurate yet require laborious calibration In parallel, robot-only demonstration datasets have grown and controlled environments. DexWild leverages calibration- significantly in scale and diversity [21, 8, 54], fueling research free Aruco tracking—significantly improving reliability and in behavior cloning and enabling generalist policy architec- minimizing setup time as it requires a single monocular camera. tures [49, 8, 22]. While these policies show impressive per- While vision-based methods often attempt to track both the formance across many tasks, they often struggle to generalize wrist and fingers simultaneously, many recent systems decouple to unseen object categories, scene layouts, or environmental the two to improve accuracy. Kinematic exoskeleton gloves conditions [25]. This lack of robustness remains a key limitation can provide high-fidelity joint measurements and even haptic of current systems. feedback [58], but are bulky and uncomfortable for long-term Human Demonstration Setup Robot Setup
DexWild Camera Tracking Camera Mini-PC Mocap Glove
Egocentric Exocentric Franka Panda xArm
Fig. 2: Left: DexWild efficiently capture high-fidelity data using an individual’s own hands across various environments. Right: Robot hands are equipped with cameras aligned with the human cameras. We test DexWild on two distinct robot hands and robot arms.
use. Instead, DexWild, along with prior works [41, 55], adopts DexWild-System is designed around three core objectives: a lightweight glove-based solution that uses electromagnetic • Portability: Allow rapid, large-scale data collection field (EMF) sensing to estimate fingertip positions. This allows across diverse environments without requiring complex for accurate, real-time hand tracking that is robust to occlusions calibration procedures. and readily retargetable to a wide range of robot hands. • High Fidelity: Accurately capture fine-grained hand and
III. D EX W ILD environment interactions essential for training precise
dexterous policies.
Many believe that leveraging large, high-quality datasets is • Embodiment-Agnostic: Enable seamless retargeting from the key for creating dexterous robot policies that generalize human demonstrations to a wide variety of robot hands. [8, 49, 40, 11]. We introduce DexWild-System, a user-friendly, high-fidelity platform for efficiently gathering natural human Portability: hand demonstrations across diverse real-world settings. Com- To collect data in diverse real-world settings, a system must pared to traditional teleoperation-based approaches, DexWild- be portable, robust, and usable by anyone. We design DexWild- System enables 4.6× faster data acquisition at scale. System with these goals in mind: it is lightweight, easy to carry, Building on this system, we propose DexWild, an imitation and can be set up in just a few minutes—enabling scalable learning framework that co-trains on large-scale DexWild- data collection across many locations. System human demonstrations alongside a small number of As shown in Figure 2, DexWild-System consists of only three robot demonstrations. This approach combines the diversity components: a single tracking camera for wrist pose estimation, and richness of human interactions with the grounding of the a battery-powered mini-PC for onboard data capture, and a robot embodiment, enabling policies to robustly generalize custom sensor pod comprising a motion-capture glove and across new objects, environments, and embodiments. Figure 1 synchronized palm-mounted cameras. displays our high level approach. Unlike traditional motion capture systems [60, 13, 4, 52] that often rely on complex outside-in tracking setups that require A. Data Collection System calibration, DexWild-System is truly calibration free, making it A scalable data collection system for dexterous robot learning versatile for any scenario and foolproof for untrained operators. must enable natural, efficient, and high-fidelity collection across This is achieved by adopting a relative state-action rep- diverse environments. To this end, we design DexWild-System: resentation, where each state and action is captured as the a portable, user-friendly system that captures human dexterous relative difference from the previous time step’s pose. This behavior with minimal setup and training. While previous eliminates any need for a global coordinate frame, allowing in-the-wild data collection approaches have typically relied the tracking camera to be freely placed—either egocentrically on sensorized grippers, we aimed to create a more intuitive or exocentrically. Additionally, the palm cameras are rigidly hardware interface that mirrors how humans naturally interact mounted in fixed positions across both human and robot with the world. From delicate fine-motor actions to powerful embodiments. This ensures visual observations are aligned grasps, humans possess dexterity across a wide range of across domains, eliminating the need for further calibration at manipulation tasks. By learning from this intrinsic capability, deployment. The external tracking camera, when carefully DexWild-System captures rich, diverse data applicable to a positioned, can also capture supplementary environmental broad range of robot embodiments. context useful for learning robust policies. High Fidelity: To learn dexterous behaviors, fine-grained, nuanced motions must be captured in the training dataset. Although DexWild- System consists of only a few portable components, we make no compromises on data fidelity. Our system is designed to accurately capture both hand and wrist actions, paired with high-quality visual observations. Human For wrist and hand tracking, vision-only methods are easy to setup. However, what they gain in portability, they often lose in accuracy and robustness—yielding noisy pose estimates that degrade policy learning [41, 14, 32, 7]. For hand pose estimation, we use motion capture gloves, which offer high accuracy, low latency, and robustness against occlusions [41]. For wrist tracking, we mount ArUco markers on the glove and track them using an external camera. This Robot avoids the fragility of SLAM-based wrist tracking, which often fails in feature-sparse environments or during occlusion-heavy Fig. 3: DexWild aligns the visual observations between humans and robots to bridge the embodiment gap. This incentivizes the model to tasks (e.g., drawer opening). learn a task-centric rather than embodiment-centric representation. Unlike many datasets that rely on egocentric or distant external cameras, we place two global-shutter cameras directly on the palm. As illustrated in Figure 2, these stereo cameras we use the data collected by DexWild to enable dexterous capture detailed, localized interaction views with minimal policies to generalize to in-the-wild scenarios. motion blur and a wide field of view. This wide field of view enables policies to operate using only the onboard palm B. Training Data Modalities and Preprocessing cameras, without any reliance on static viewpoints. Generalization in dexterous manipulation demands both Embodiment-Agnostic: scale and embodiment grounding. With this goal, DexWild To ensure the longevity and versatility of DexWild data, collects two complementary datasets: a large-scale human we aim for it to remain useful across different robot embod- demonstration dataset DH using DexWild-System, and a iments—even as hardware platforms evolve. Achieving this smaller teleoperated robot dataset DR . Human data offers broad goal requires careful alignment of both the observation space task diversity and ease of collection in real-world settings, but and the action space between humans and robots. lacks embodiment alignment. Robot data, while limited in scale, We begin by standardizing the observation space. Although provides crucial grounding in the robot’s action and observation our palm-mounted cameras have a wide field of view, we spaces. To harness the strengths of both, we co-train policies intentionally position them to focus primarily on the environ- using a fixed ratio of human and robot data within a batch, ment, minimizing the visibility of the hand itself. Importantly, (wh , wr )—balancing diversity with embodiment grounding to the camera placement is mirrored between the human and enable robust generalization during deployment. robot hands. As shown in Figure 3, this design yields visually At each training iteration, we sample a batch consisting of consistent observations across embodiments, allowing the transitions xh and xr from DH and DR , respectively, according policy to learn a shared visual representation that generalizes to the co-training weights. Each transition xi at timestep i across both human and robot domains. contains: For action space alignment, we build on insights from prior work [17, 44], optimizing robot hand kinematics to match the • Observation oi : An observation at a given timestep
fingertip positions observed in human demonstrations. We note consists of two synchronized palm camera images that this method is general and can work for any robot hand Ipinky and Ithumb captured at the current timestep, embodiment. It operates with fixed hyperparameters across as well as a sequence of historical states, sampled users and is robust to variations in hand size—eliminating the at a step size up a given horizon H, comprising of need for user-specific tuning. {∆pi , ∆pi−step , …, ∆pi−H }. Each ∆p consists of relative Collecting data using natural human hands offers benefits historical end-effector positions. beyond ease of use. The diversity in hand morphology across • Action ai:i+n−1 : An action chunk of size n that includes
human demonstrators introduces useful variation, which we actions {ai , ai+1 , … , ai+n−1 }, where ai is the action at hypothesize helps policies learn more generalizable grasping the current timestep. Specifically, ai is a 26-dimensional strategies—particularly important given the inherent mismatch vector consisting of: between human and robot hand kinematics. – aarm : A 9-dimensional vector describing relative end- In summary, DexWild is a portable, high quality, human- effector position (3D) and orientation (6D). centric system that can be worn by any operator to collect – ahand : A 17-dimensional vector describing the finger human data in real-world environments. Next, we explain how joint position targets of the robot hand. Spray Bottle Toy Cleanup Pouring
Florist Clothes Folding
Fig. 4: Using DexWild-System, humans can effortlessly collect accurate data with their own hands across a wide range of environments. This data is directly used to train any robot hand to perform dexterous manipulation in a human-like way in any environment. We validate this approach on five representative tasks. Please see videos of these tasks on our website at https://dexwild.github.io
For bimanual tasks, the observation and action spaces are action distributions more effectively than alternatives such
duplicated, and the inter-hand pose is appended to the as Gaussian Mixture Models (GMMs) or transformers. This
observation to facilitate coordination. capability becomes increasingly important in DexWild, where
While our retargeting procedure brings human and robot demonstrations are collected from multiple humans with diverse trajectories into a shared action space, a few additional steps strategies, resulting in inherently multi-modal behaviors. As are necessary to make the human and robot datasets compatible the dataset scales, modeling this variability becomes critical for for joint training: robust policy learning. Specifically, DexWild uses a diffusion U-Net model [6] to generate action chunks. • Action Normalization: The actions of human and robot data are normalized separately to account for inherent Concretely, the training procedure is outlined in Algorithm 1. distribution mismatches. • Demo Filtering: Since human demonstrations are col- Algorithm 1 DexWild Imitation Learning Procedure lected by untrained operators in uncontrolled environments, Require: Human dataset DH , Robot dataset DR , Co-training weights we apply a heuristic-based filtering pipeline to automati- {ωh , ωr } 1: Initialize policy πθ with ViT encoder ϕvit cally detect and remove low-quality or invalid trajectories. 2: while not converged do This filtering step significantly improves dataset quality 3: Sample a batch of transitions {xh }, {xr } from DH , DR using without manual labeling. weights {ωh , ωr } 4: for each transition xi in the batch do C. Policy Training 5: Extract observation oi 6: Encode images: Zi = ϕvit (oi ) Through the careful design of our hardware, observation, and 7: Extract ground truth action chunk ai:i+n−1 = action interfaces, we are able to train dexterous robot policies {ai , … , ai+n−1 } using a simple behavior cloning (BC) objective [31, 37, 36]. To 8: Sample noise scale t ∼ U(1, T ) effectively learn from our multimodal, diverse data, our training 9: Add noise ϵt ∼ N (0, σt ) to ai:i+n−1 pipeline leverages large-scale pre-trained visual encoders and 10: Predict noise ϵ̂θ = πθ (Zi , ai:i+n−1 + ϵt , t) 11: Compute diffusion loss Lθ = ∥ϵt − ϵ̂θ ∥22 shows strong performance across different policy architectures. 12: end for Visual Encoder: Training on DexWild data exposes our 13: Update policy parameters θ policy to significant visual diversity—across scenes, objects, 14: end while and lighting—requiring an encoder that generalizes well to such variability. To address this, we adopt a pre-trained Vision An important finding in our training framework is that tuning Transformer (ViT) backbone, which has shown superior perfor- the human-to-robot data weighting significantly affects real- mance over ResNet-based encoders on in-the-wild manipulation world performance. We discuss these effects in Section V-A. tasks [16, 23]. Pre-trained ViTs, especially those trained on large internet-scale datasets, are particularly effective at IV. E XPERIMENTS extracting rich, transferable features [27, 33, 47, 11], making them well-suited for our setting. Our experimental evaluation encompasses extensive real- Policy Class: While several imitation learning architectures world deployment across diverse environments and robots, uti- have been proposed recently [59, 6], we adopt a diffusion- lizing both human demonstrations and robot teleoperation data. based policy. Diffusion models are particularly well-suited Below, we outline our data collection process, experimental for dexterous manipulation, as they can capture multi-modal setup, and evaluation tasks. Train
Test
Fig. 5: We collect data using a diverse set of objects across categories. Spray Bottle Task – 25 Train, 11 Test; Toy Cleanup Task – 64 Train, 9 Test; Pour Task – 35 Train, 5 Test; Florist Task - 6 Train, 2 Test; Clothes Folding Task - 17 Train, 6 Test.
A. Scaling up Data Collection These tasks systematically evaluate DexWilds functional Our hardware system was deployed to 10 untrained users to grasping capabilities, generalization across object types, trans- collect data across a wide range of real-world environments. feral of skills across tasks, coordination between arms, and These settings included indoor and outdoor locations, day and adaptability to deformable objects. Success requires the policy night conditions, crowded cafeterias and quiet study areas, to adapt to varying object properties, environmental conditions, with varied tables, objects, and lighting setups. The collectors and task constraints. themselves varied in hand sizes and demonstration styles, C. Evaluation Environments enabling us to learn from a wide distribution of environments and interactions. For robot experiments, we employed an xArm robot and We constructed two datasets through our collection efforts: Franka system, both equipped with either LEAP hand or LEAP DH (human-collected data) and DR (robot-collected data). The hand V2 Advanced [38, 41]. Unless explicitly mentioned, xArm human dataset DH comprises 9,290 demonstrations across five and LEAP hand V2 Advanced was used. We evaluate our tasks: 3,000 demonstrations from 30 different environments approach across three scenarios: for each of the Spray Bottle and Toy Cleanup tasks, 621 1) In-Domain: Environments where robot training data was trajectories from 6 environments for the Pour task, 1,545 collected, testing with novel objects demonstrations from 15 environments for the Florist task, and 2) In-the-Wild: Environments present in DexWild but absent 1,124 demonstrations from 12 environments for the Clothes from robot training data Folding task. 3) In-the-Wild Extreme: Unseen environments absent from The robot dataset DR includes 1,395 demonstrations: 388 both datasets. for Spray Bottle, 370 for Toy Cleanup, 111 for Pour, 236 for V. A NALYSIS AND R ESULTS Florist, and 290 for Clothes Folding tasks. Robot data was In our evaluations, we seek to investigate the following key collected using an xArm and LEAP hand V2 Advanced. Our questions: training and test objects are detailed in Figure 5. 1) How effectively does DexWild leverage human data to B. Evaluation Tasks achieve strong in-the-wild performance? We evaluate our approach on five diverse manipulation 2) Does DexWild enable policy transfer across tasks and tasks, each designed to assess specific aspects of dexterous robot embodiments? manipulation: functional grasping, long-horizon planning, cross- 3) Does policy performance scale effectively with increasing task transfer, bimanual coordination, and deformable object amounts of DexWild-System data? manipulation. A task visualization is provided in Figure 4. Please see videos of our results at https://dexwild.github.io. In the Spray Bottle task, the robot grasps a spray bottle by the handle and sprays a target cloth, testing functional grasping A. Zero Shot In the Wild Policies w/ DexWild and affordance understanding. In Toy Cleanup, the robot DexWild enables strong policy generalization in novel picks up scattered toys and places them in a bin, evaluating scenes. We evaluate policies in environments with increasing generalization and long-horizon planning. The Pouring task novelty to assess their generalization. As shown in Figure 6, involves tilting a bottle to pour into a container, demonstrating policies trained exclusively on robot data perform well in in- skill transfer from the spray bottle task. In Bimanual Florist, domain settings (64.7% success rate) but degrade significantly the robot hands over a flower between its arms and inserts it in more challenging scenarios—in-the-wild (28.5%) and in- into a vase, testing precise bimanual coordination. Finally, in the-wild extreme (22.0%). This 36-point performance drop Bimanual Clothes Folding, the robot uses both hands to fold a suggests that robot-only policies overfit to environment-specific clothing item, assessing manipulation of deformable objects. features and fail to develop robust, transferable representations. Full task specifications and scoring criteria for all tasks are In contrast, policies trained only on human data learn high- provided in Appendix VII-A. level object affordances and approach objects reliably, even In-Domain Performance In the Wild Performance In the Wild Extreme Performance 0.8 0.8 0.6 0.6 0.6 Score
0.4
0.4 0.4
0.2 0.2 0.2
0.0 Spray Toy Average 0.0 Spray Toy Average 0.0 Spray Toy Florist Clothes Average
Cleanup Cleanup Cleanup Folding
Robot Only Co-train 1:1 Co-train 1:2 (Ours) Co-train 1:5 Human Only
Fig. 6: How does co-training help with scaling up in the wild performance? We evaluate our policy across three scenarios: (a) In-Domain scenes where robot training data was collected but with novel objects, (b) In-the-Wild scenes present in DexWild but not in robot data, and (c) In-the-Wild Extreme scenes absent from both datasets. Displayed ratio is Robot:Human.
in complex scenes. However, without robot-specific action transfer using the pouring task, which shares many motion grounding, they struggle to execute precise manipulation, primitives with the spray task. Crucially, we use no robot data resulting in poor performance across all scenarios (3.6% in- for pouring and instead combine human (DexWild-System) domain, 7.3% in-the-wild). demonstrations of pouring with robot demonstrations from To combine the strengths of both modalities, we adopt a spraying. This setup enables zero-shot generalization to co-training strategy—jointly training on both robot and human pouring in in-the-wild extreme environments. Using a 1:2 data—a method validated in prior works [8, 49, 21, 20, 32]. robot-to-human co-training ratio, our policy achieves a 94% This encourages the policy to learn task-relevant features rather success rate, far exceeding policies trained with only robot than overfitting to specific embodiments or environments. We (0%) or only human data (11%). experiment with different robot-to-human data ratios (1:1 to DexWild enables transfer across robot embodiments. 1:5) per training batch. Our empirical analysis reveals that a Since DexWild data is not tied to any specific embodiment, it 1:2 ratio yields optimal performance across all scenarios: naturally supports cross-platform transfer. This prolongs the
-
In Domain: 79.8% vs. 64.7% (robot-only) value of our data, as collecting platform-specific data for every
-
In-the-wild: 75.1% vs. 28.5% (robot-only) new robot is resource-intensive and impractical. We test two
-
In-the-wild Extreme: 62.7% vs. 22.0% (robot-only) transfer scenarios in in-the-wild extreme scenes: Interestingly, increasing the human data ratio further (e.g., • Cross-arm: Transferring from an xArm to a Franka Panda 1:5) degrades performance (54.5% in-domain, 50.9% in-the- arm. We achieve a 37.5% success rate, compared to 4.5% wild), indicating that robot data remains essential for grounding for the robot-only baseline—an 8.3× improvement. fine-grained control. • Cross-hand: Transferring from the LEAP Hand V2 DexWild extends to complex bimanual coordination Advanced to the original LEAP Hand. We achieve 65.3% tasks. To evaluate whether DexWild generalizes beyond single- success versus 13.3% for the baseline, showing that arm tasks, we test it on bimanual tasks that demand precise DexWild generalizes not only across arms, but across coordination between two hands. We compare co-trained dexterous hands as well. policies (1:2 ratio) against robot-only policies in in-the-wild These results, shown in Figure 7, demonstrate that DexWild extreme settings. DexWild policies achieve a strong 68.1% enables zero-shot generalization to new tasks and hardware average success rate, compared to just 13% for the robot- embodiments without any additional robot-specific data, only baseline. Even when failures occur, DexWild policies making it an efficient and general framework for dexterous exhibit meaningful attempts at task execution—while robot- policy learning on many robots. only policies often produce erratic or unstructured behavior. These results demonstrate that DexWild not only enables C. Scalability of DexWild robust generalization across environments but also scales to more complex manipulation behaviors. Policy performance scales with dataset size. To understand how data scale impacts policy performance in the wild, we B. Robust Cross-Task and Cross-Embodiment Generalization randomly sample subsets of the full human dataset at varying DexWild enables transfer of low-level skills across tasks. sizes and evaluate the resulting policies. We fix the size of Many manipulation tasks share foundational motor skills—such the robot dataset. As shown in Figure 7, there is a clear as lifting, orienting, and rotating objects—which opens the positive correlation between dataset size and average task door to skill reuse across related tasks. For example, opening performance—rising from 28.7% at 20% dataset size to 67.8% a microwave and opening a cupboard both involve similar with the full dataset, marking a 2.36× improvement. Interest- coordination and control. We evaluate this form of cross-task ingly, the learning curve is nonlinear, with especially steep gains 1.0 Cross-Task Performance 1.0 Cross-Embodiment Performance 1.0 Scaling Performance 0.8 0.8 0.8 0.6 0.6 0.6 Score
0.4 0.4 0.4 0.2 0.2 0.2 0.0 0.0 Cross-Hand Cross-Arm 0.0 0 25 50 100 Type DexWild Dataset Scale (%) Robot Only Co-train 1:1 Co-train 1:2 (Ours) Co-train 1:5 Human Only
Fig. 7: Left: Cross-Task Performance – Evaluating DexWild on the pour task using robot data exclusively from the spray task. Middle: Cross-Embodiment Performance – Testing DexWild policy on the Original LEAP hand and a Franka robot arm. Right: Scaling Performance – Demonstrating improved DexWild performance as dataset size increases. Displayed ratio is Robot:Human.
in the 25–50% range, suggesting a critical threshold where the DexWild not only demonstrates strong scaling trends with policy begins to reliably learn generalizable behaviors. increasing data volume, but also offers a practical and efficient Importantly, performance continues to improve all the way path to collecting diverse, high-quality data at scale—crucial to 100% data usage, indicating that the system has not yet for real-world generalization. plateaued. This suggests that even more capable policies could VI. C ONCLUSION AND L IMITATIONS be learned with continued data collection. DexWild-System enables fast and scalable data collection. We introduce DexWild, a scalable framework for learning Given the observed benefits of scaling, we evaluate the data dexterous manipulation policies that effectively generalize to collection efficiency of DexWild-System via a comparative new tasks, environments, and robot embodiments. We introduce user study measuring demonstrations per hour. As shown in DexWild-System, a portable, human-centric data collection Figure 8, DexWild-System achieves an average collection rate device that significantly accelerates dataset creation (4.6× of 201 demos/hour across five representative tasks—nearly faster than conventional robot teleoperation). We propose matching the rate of demonstrations collected using bare hands DexWild cotraining method, which leverages large scale human and 4.6× faster than a traditional robot teleoperation system demonstrations alongside minimal robot data to achieve robust based on Gello [41, 56], which achieves just 43 demos/hour. generalization-reaching a success rate of 68.5% in completely We identify three key limitations of Gello-based collection unseen environments, nearly four times higher than methods that our system overcomes: using robot data only. Furthermore, DexWild’s embodiment- agnostic design enables strong cross-embodiment and cross-task
-
Lack of haptic feedback: Operators cannot feel objects, transfer capabilities, reducing the need for robot-specific data. making fine manipulation difficult for certain tasks. Despite these strengths, several limitations remain that
-
Scene reset: Resetting the environment is cumbersome motivate future research: First, our approach still depends on and often requires a second operator or pauses in data a limited number of teleoperated robot data to bridge the gap collection. between human and robot actions. Future work could explore
-
Hardware setup overhead: Robots are heavy and require improved retargeting or online policy adaptation to remove time-consuming setup at each new location, whereas the need for teleoperated data. Next, because humans typically DexWild-System is portable and can be set up in minutes. perform these tasks successfully, their demonstrations seldom include error recovery—causing trained policies to struggle to Data Collection Speed by Method recover from unexpected failures. Adding recovery examples or 250 adaptive strategies could boost real-world robustness. Finally, our method uses only visual and kinematic data, which limits its 200 Demos per hour
performance in contact-rich tasks. Incorporating tactile or haptic 150 sensing could improve the handling of delicate interactions.1: 100 In summary, DexWild represents a significant step toward 50 scalable, generalizable robot manipulation policies. Our results 0 highlight the promise of leveraging human interaction data at Spray Toy Pour Florist Clothes Average Cleanup Folding scale, offering an exciting avenue toward truly dexterous and Robot Ours Human versatile robots operating in diverse, real-world environments.
Fig. 8: DexWild-System offers 4.6× improvement over robot data Videos, code, and hardware instructions are available on our collection speed and nearly matches the human bare hands data website at https://dexwild.github.io collection speed. ACKNOWLEDGMENTS Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kemb- havi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit We would like to thank Yulong Li, Hengkai Pan, and Sandeep Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Routray for thoughtful discussions. We’d also like to thank Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Andrew Wang for setting up compute and Yulong Li for helping Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu with robot system setup. Lastly, we’d like to express thanks to Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang, Hengkai Pan, Andrew Wang, Adam Kan, Ray Liu, Mingxuan Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Li, Lukas Vargas, Jose German, Laya Satish, Sri Shasanka Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Madduri for helping collect data. This work was supported Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne in part by AFOSR FA9550-23-1-0747 and Apple Research Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Award. Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Foster, Fangchen Liu, Federico R EFERENCES Ceola, Fei Xia, Feiyu Zhao, Felipe Vieira Frujeri, Freek [1] Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salho- and Deepak Pathak. Affordances from human videos as tra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, a versatile representation for robotics. 2023. 2 Gregory Kahn, Guangwen Yang, Guanzhi Wang, Hao Su, [2] Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Shreyas Hampali, Shangchen Han, Fan Zhang, Linguang Amor, Henrik I Christensen, Hiroki Furuta, Homanga Zhang, Jade Fountain, Edward Miller, Selen Basol, et al. Bharadhwaj, Homer Walke, Hongjie Fang, Huy Ha, Igor Hot3d: Hand and object tracking in 3d from egocentric Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, multi-view videos. arXiv preprint arXiv:2411.19167, Jad Abou-Chakra, Jaehyung Kim, Jaimyn Drake, Jan 2024. 2 Peters, Jan Schneider, Jasmine Hsu, Jay Vakil, Jeannette [3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Neelakantan, Pranav Shyam, Girish Sastry, Amanda Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Askell, et al. Language models are few-shot learners. Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna, Advances in neural information processing systems, 33: Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi 1877–1901, 2020. 1 Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang, [4] Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Van Wyk, Umar Iqbal, Stan Birchfield, et al. Dexycb: Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin A benchmark for capturing hand grasping of objects. In Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Proceedings of the IEEE/CVF Conference on Computer Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Vision and Pattern Recognition, pages 9044–9053, 2021. Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle 3 Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, [5] Xuxin Cheng, Jialong Li, Shiqi Yang, Ge Yang, and Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi ”Jim” Fan, Xiaolong Wang. Open-television: Teleoperation with Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion immersive active visual feedback. arXiv preprint Lepert, Marius Memmel, Masayoshi Tomizuka, Masha arXiv:2407.01512, 2024. 2 Itkina, Mateo Guaman Castro, Max Spero, Maximilian [6] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Du, Michael Ahn, Michael C. Yip, Mingtong Zhang, Cousineau, Benjamin Burchfiel, and Shuran Song. Dif- Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit fusion policy: Visuomotor policy learning via action Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, diffusion. In Proceedings of Robotics: Science and Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Systems (RSS), 2023. 5, 14 Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah, [7] Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Sanketi, Patrick ”Tree” Miller, Patrick Yin, Paul Wohlhart, Shuran Song. Universal manipulation interface: In- Peng Xu, Peter David Fagan, Peter Mitrano, Pierre the-wild robot teaching without in-the-wild robots. In Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Proceedings of the IEEE International Conference on Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Robotics and Automation (ICRA), 2024. 2, 4 Roberto Mart’in-Mart’in, Rohan Baijal, Rosario Scalise, [8] Open X-Embodiment Collaboration, Abby O’Neill, Abdul Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khaz- Sonawani, Shubham Tulsiani, Shuran Song, Sichun Xu, atsky, Anant Rai, Anchit Gupta, Andrew Wang, Andrey Siddhant Haldar, Siddharth Karamcheti, Simeon Ade- bola, Simon Guist, Soroush Nasiriany, Stefan Schaal, (CVPR), pages 11743–11753, 2022. 1, 2 Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, [16] Huy Ha, Yihuai Gao, Zipeng Fu, Jie Tan, and Shuran Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Song. UMI on legs: Making manipulation policies mobile Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, with manipulation-centric whole-body controllers. In Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Proceedings of the 2024 Conference on Robot Learning, Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. 2024. 5 Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, [17] Ankur Handa, Karl Van Wyk, Wei Yang, Jacky Liang, Vidhi Jain, Vikash Kumar, Vincent Vanhoucke, Wei Zhan, Yu-Wei Chao, Qian Wan, Stan Birchfield, Nathan Ratliff, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu and Dieter Fox. Dexpilot: Vision-based teleoperation Chen, Xiaolong Wang, Xinghao Zhu, Xinyang Geng, of dexterous robotic hand-arm system. In 2020 IEEE Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yansong Pang, International Conference on Robotics and Automation Yao Lu, Yecheng Jason Ma, Yejin Kim, Yevgen Chebotar, (ICRA), pages 9164–9170. IEEE, 2020. 4 Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan [18] Yafei Hu, Quanting Xie, Vidhi Jain, Jonathan Francis, Wang, Yonatan Bisk, Yongqiang Dou, Yoonyoung Cho, Jay Patrikar, Nikhil Keetha, Seungchan Kim, Yaqi Xie, Youngwoon Lee, Yuchen Cui, Yue Cao, Yueh-Hua Wu, Tianyi Zhang, Zhibo Zhao, et al. Toward general-purpose Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang, robots via foundation models: A survey and meta-analysis. Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka arXiv preprint arXiv:2312.08782, 2023. 2 Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen [19] Aadhithya Iyer, Zhuoran Peng, Yinlong Dai, Irmak Guzey, Zhang, Zipeng Fu, and Zipeng Lin. Open X-Embodiment: Siddhant Haldar, Soumith Chintala, and Lerrel Pinto. Robotic learning datasets and RT-X models, 2023. 1, 2, OPEN TEACH: A versatile teleoperation system for 3, 7 robotic manipulation. arXiv preprint arXiv:2403.07870, [9] Juan Antonio Corrales, Francisco A Candelas, and Fer- 2024. 2 nando Torres. Hybrid tracking of human operators using [20] Simar Kareer, Dhruv Patel, Ryan Punamiya, Pranay imu/uwb data fusion by a kalman filter. In Proceedings Mathur, Shuo Cheng, Chen Wang, Judy Hoffman, and of the 3rd ACM/IEEE international conference on Human Danfei Xu. Egomimic: Scaling imitation learning via robot interaction, pages 193–200, 2008. 2 egocentric video, 2024. 2, 7 [10] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, [21] Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ash- Sanja Fidler, Antonino Furnari, Evangelos Kazakos, win Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yun- Price, and Michael Wray. Epic-kitchens: A large-scale liang Chen, Kirsty Ellis, et al. Droid: A large-scale dataset for recognizing, anticipating, and retrieving hand- in-the-wild robot manipulation dataset. arXiv preprint object interactions. In Proceedings of the European arXiv:2403.12945, 2024. 1, 2, 7 Conference on Computer Vision (ECCV), pages 802–819, [22] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted 2018. 1, 2 Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, [11] Sudeep Dasari, Mohan Kumar Srirama, Unnat Jain, and Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: Abhinav Gupta. An unbiased look at datasets for visuo- An open-source vision-language-action model. arXiv motor pre-training. In Conference on Robot Learning. preprint arXiv:2406.09246, 2024. 2 PMLR, 2023. 2, 3, 5, 14 [23] Fanqi Lin, Yingdong Hu, Pingyue Sheng, Chuan Wen, [12] Haritheja Etukuru, Norihito Naka, Zijin Hu, Seungjae Jiacheng You, and Yang Gao. Data scaling laws in Lee, Julian Mehu, Aaron Edsinger, Chris Paxton, Soumith imitation learning for robotic manipulation. In Conference Chintala, Lerrel Pinto, and Nur Muhammad Mahi Shafiul- on Robot Learning (CoRL), 2024. 2, 5 lah. Robot utility models: General policies for zero-shot [24] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae deployment in new environments, 2024. 2 Lee. Visual instruction tuning. In NeurIPS, 2023. 1 [13] Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed [25] Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Kocabas, Manuel Kaufmann, Michael J Black, and Otmar Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Hilliges. Arctic: A dataset for dexterous bimanual hand- Savarese, Yuke Zhu, and Roberto Martı́n-Martı́n. What object manipulation. In Proceedings of the IEEE/CVF matters in learning from offline human demonstrations Conference on Computer Vision and Pattern Recognition, for robot manipulation. arXiv preprint arXiv:2108.03298, pages 12943–12954, 2023. 3 2021. 2 [14] Authors from UC San Diego and MIT. Open-television: [26] Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, An open-source immersive teleoperation system with Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, stereo visual feedback. The Robot Report, 2024. 4 Yunrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck [15] Kristen Grauman, Michael Ryoo, Aljoša Smolić, Minh Babich, Gavriel State, Marco Hutter, and Animesh Garg. Vo, and et al. Ego4d: Around the world in 3,000 hours Orbit: A unified simulation framework for interactive of egocentric video. In Proceedings of the IEEE/CVF robot learning environments. IEEE Robotics and Au- Conference on Computer Vision and Pattern Recognition tomation Letters, 8(6):3740–3747, June 2023. ISSN 2377-3774. doi: 10.1109/lra.2023.3270034. URL http: Karen Liu, Dana Kulic, and Jeff Ichnowski, editors, //dx.doi.org/10.1109/LRA.2023.3270034. 14 Proceedings of The 6th Conference on Robot Learning, [27] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, and volume 205 of Proceedings of Machine Learning Re- Chelsea Finn. R3M: A universal visual representation search, pages 654–665. PMLR, 14–18 Dec 2023. 2 for robot manipulation. arXiv preprint arXiv:2203.12601, [40] Kenneth Shaw, Shikhar Bahl, and Deepak Pathak. 2022. 2, 5 Videodex: Learning dexterity from internet videos. In [28] Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Conference on Robot Learning, pages 654–665. PMLR, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. 2023. 2, 3 Hamer: Hand mesh recovery for the egoexo4d hand pose [41] Kenneth Shaw, Yulong Li, Jiahui Yang, Mohan Kumar challenge. 2 Srirama, Ray Liu, Haoyu Xiong, Russell Mendonca, and [29] Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Deepak Pathak. Bimanual dexterity for complex tasks. Angjoo Kanazawa, David Fouhey, and Jitendra Malik. In 8th Annual Conference on Robot Learning, 2024. 2, Reconstructing hands in 3d with transformers. In 3, 4, 6, 8, 14 Proceedings of the IEEE/CVF Conference on Computer [42] Himanshu Gaurav Singh, Antonio Loquercio, Carmelo Vision and Pattern Recognition (CVPR), 2024. 2 Sferrazza, Jane Wu, Haozhi Qi, Pieter Abbeel, and [30] Alexandra Pfister, Alexandre M West, Shaw Bronner, and Jitendra Malik. Hand-object interaction pretraining from Jack Adam Noah. Comparative abilities of microsoft videos, 2024. URL https://arxiv.org/abs/2409.08273. 2 kinect and vicon 3d motion capture for gait analysis. [43] Ritvik Singh, Arthur Allshire, Ankur Handa, Nathan Journal of medical engineering & technology, 38(5):274– Ratliff, and Karl Van Wyk. Dextrah-rgb: Visuomotor 280, 2014. 2 policies to grasp anything with dexterous hands. arXiv [31] Dean A Pomerleau. Alvinn: An autonomous land vehicle preprint arXiv:2412.01791, 2024. 2 in a neural network. Advances in neural information [44] Aravind Sivakumar, Kenneth Shaw, and Deepak Pathak. processing systems, 1, 1988. 5 Robotic telekinesis: Learning a robotic hand imitator by [32] Ri-Zhao Qiu, Shiqi Yang, Xuxin Cheng, Chaitanya watching humans on youtube, 2022. 4 Chawla, Jialong Li, Tairan He, Ge Yan, David J. Yoon, [45] Aravind Sivakumar, Kenneth Shaw, and Deepak Pathak. Ryan Hoque, Lars Paulsen, Ge Yang, Jian Zhang, Sha Robotic telekinesis: Learning a robotic hand imitator by Yi, Guanya Shi, and Xiaolong Wang. Humanoid policy ˜ watching humans on youtube. RSS, 2022. 2 human policy. arXiv preprint arXiv:2503.13441, 2025. 2, [46] Shuran Song, Andy Zeng, Johnny Lee, and Thomas 4, 7 Funkhouser. Grasping in the wild: Learning 6dof closed- [33] Ilija Radosavovic, Tete Xiao, Stephen James, Pieter loop grasping from low-cost demonstrations. Robotics Abbeel, Jitendra Malik, and Trevor Darrell. Real-world and Automation Letters, 2020. 2 robot learning with masked visual pre-training. CoRL, [47] Mohan Kumar Srirama, Sudeep Dasari, Shikhar Bahl, and 2022. 5 Abhinav Gupta. HRP: Human affordances for robotic [34] Nathan D. Ratliff, Jan Issac, Daniel Kappler, Stan Birch- pre-training. In Proceedings of Robotics: Science and field, and Dieter Fox. Riemannian motion policies, 2018. Systems, Delft, Netherlands, 2024. 2, 5 URL https://arxiv.org/abs/1801.02854. 14 [48] Andreas Steiner, André Susano Pinto, Michael Tschan- [35] Yu Rong, Takaaki Shiratori, and Hanbyul Joo. Frankmo- nen, Daniel Keysers, Xiao Wang, Yonatan Bitton, cap: A monocular 3d whole-body pose estimation system Alexey Gritsenko, Matthias Minderer, Anthony Sher- via regression and integration. In Proceedings of the bondy, Shangbang Long, Siyang Qin, Reeve Ingle, IEEE/CVF International Conference on Computer Vision, Emanuele Bugliarello, Sahar Kazemzadeh, Thomas Mes- pages 1749–1759, 2021. 2 nard, Ibrahim Alabdulmohsin, Lucas Beyer, and Xiaohua [36] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A Zhai. PaliGemma 2: A Family of Versatile VLMs for reduction of imitation learning and structured prediction to Transfer. arXiv preprint arXiv:2412.03555, 2024. 1 no-regret online learning. In Proceedings of the fourteenth [49] OM Team, D Ghosh, H Walke, K Pertsch, K Black, international conference on artificial intelligence and O Mees, S Dasari, J Hejna, C Xu, J Luo, et al. Octo: statistics, pages 627–635. JMLR Workshop and Confer- An open-source generalist robot policy. Proceedings of ence Proceedings, 2011. 5 Robotics: Science and Systems, Delft, Netherlands, 2023. [37] Stefan Schaal. Is imitation learning the route to humanoid 2, 3, 7 robots? Trends in cognitive sciences, 3(6):233–242, 1999. [50] Yushuang Tian, Xiaoli Meng, Dapeng Tao, Dongquan 5 Liu, and Chen Feng. Upper limb motion tracking with [38] Kenneth Shaw, Ananye Agarwal, and Deepak Pathak. the integration of imu and kinect. Neurocomputing, 159: Leap hand: Low-cost, efficient, and anthropomorphic hand 207–218, 2015. 2 for robot learning. Robotics: Science and Systems (RSS), [51] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier 2023. 6 Martinet, Marie-Anne Lachaux, Timothée Lacroix, Bap- [39] Kenneth Shaw, Shikhar Bahl, and Deepak Pathak. tiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Videodex: Learning dexterity from internet videos. In et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 1 [52] Valve Corporation. https://store.steampowered.com/ steamvr. [Virtual reality platform]. 3 [53] A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 1 [54] Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen- Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning (CoRL), 2023. 1, 2 [55] Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang, Li Fei-Fei, and Karen Liu. Dexcap: Scalable and portable mocap data collection system for dexterous manipulation. In Robotics: Science and Systems (RSS), 2024. 2, 3 [56] Philipp Wu, Yide Shentu, Zhongke Yi, Xingyu Lin, and Pieter Abbeel. Gello: A general, low-cost, and intuitive teleoperation framework for robot manipulators, 2023. 2, 8 [57] Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, et al. Latent action pretraining from videos. arXiv preprint arXiv:2410.11758, 2024. 2 [58] Han Zhang, Songbo Hu, Zhecheng Yuan, and Huazhe Xu. Doglove: Dexterous manipulation with a low-cost open-source haptic force feedback glove. arXiv preprint arXiv:2502.07730, 2025. 2 [59] Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In Proceedings of Robotics: Science and Systems (RSS), 2023. 2, 5, 14 [60] Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. Freihand: A dataset for markerless capture of hand pose and shape from single rgb images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 813– 822, 2019. 2, 3 VII. A PPENDIX Videos of our results, code to recreate our system, and hardware instructions are available on our website at https: //dexwild.github.io A. Detailed Task Description and Scoring Criteria: We evaluate five dexterous manipulation tasks, each designed to assess different capabilities such as functional grasping, long-horizon planning, precision, bimanual coordination, and deformable object manipulation. Each task is scored according to a structured rubric based on discrete completion milestones. The task scoring criteria are designed to quantify the performance of different robot tasks based on specific com- pletion milestones. Each task has a set of defined actions with corresponding point values. Higher scores are assigned to more complex or functionally successful actions, while partial Fig. 9: DexWild-System features a simple and easy-to-use interface completions and failed attempts receive lower scores. This for deployment by untrained data collectors. structured scoring system allows for consistent evaluation and comparison of task performance. – 0.75: Grasp the bouquet, handover Spray Bottle – 1.00: Grasp the bouquet, handover, insert into vase This task evaluates functional grasping and affordance under- standing. The robot must grasp a spray bottle and orient it to Clothes Folding spray over a target cloth. This task tests manipulation of deformable objects using both hands. The robot must fold a clothing item placed on a surface. – 0.00: Nothing – 0.15: Tries functional grasp but fails – 0.00: Nothing – 0.25: Grasp bottle – 0.25: Tries grasp but fails – 0.75: Grasp bottle, orient over cloth – 0.50: Grasp with one hand – 0.75: Grasp bottle, use functional grasp – 0.75: Grasp with both hands – 1.00: Grasp bottle, use functional grasp, orient over cloth – 1.00: Grasp and fold Toy Cleanup B. Data Collection Procedure This task tests long-horizon planning and generalization. The To deploy DexWild-System with untrained data collectors, robot must collect scattered toys and deposit them in a we provide a one-page instruction sheet outlining the task, designated bin. object setup, and system startup/shutdown. DexWild-System – 0.00: Nothing includes three core components: a wrist-tracking camera, a – 0.25: Tries for grasp but fails battery-powered mini-PC for onboard data capture, and a – 0.50: Grasp object custom sensor pod with a motion-capture glove and palm- – 1.00: Grasp object, drop into bin mounted cameras. At a new site, users simply wear the mocap Pouring glove and power on the mini-PC with a provided power This task assesses precise motion control and transfer learning bank. For egocentric tracking, a headstrap holds the tracking from the spray bottle task. The robot must pour liquid from a camera; for exocentric tracking, we provide a collapsible bottle into a container. tripod. Once booted, users launch our custom desktop app – 0.00: Nothing and control recording via a Bluetooth clicker or foot pedal. – 0.15: Tries functional grasp but fails The UI (Fig. 9) shows sensor status, SLAM recording, and – 0.25: Grasp bottle data capture indicators, along with buttons to view the tracking – 0.75: Grasp bottle, pour into container camera feed and delete the last episode. Collectors gather 100 – 0.75: Grasp bottle, use functional grasp episodes per location. After the day is finished, we upload the – 1.00: Grasp bottle, use functional grasp, pour into container data to our remote machine for processing.
Bimanual Florist C. Downstream Data Processing This task evaluates coordinated control of both hands. The Each episode is stored in its own folder, with subfolders robot must pick up a flower, hand it to the other arm, and organizing individual actions and observations. SVO recordings insert it into a vase. from the Zed Mini camera—used for SLAM and wrist pose – 0.00: Nothing tracking—are saved separately, with each file covering five – 0.15: Tries grasp but fails episodes. To begin data processing, we use the Zed SDK – 0.25: Grasp the bouquet to decode these SVO files, reconstruct the camera’s motion, and perform ArUco cube tracking and wrist pose estimation G. Cross Hand Extended Results using both the left image and stereo depth data. We then Does DexWild generalize across different robot hands? apply a filtering pipeline to assess tracking quality; episodes Table II reports LEAP Hand performance under both In the Wild are discarded if the wrist pose cannot be reliably tracked and In the Wild Extreme conditions. In every case, DexWild for more than 75% of the duration. Next, we compute the co-training substantially outperforms the robot-only baseline. action distribution and clip outliers outside the 2nd and 97th These results highlight the effectiveness of DexWild in cross percentiles. We smooth the trajectories using interpolation and embodiment generalization even when using a completely Gaussian filtering to ensure fluid motion. Hand motions are different robot hand. then retargeted using inverse kinematics in PyBullet, following the method in [41]. The entire pipeline is parallelized using H. Scaling Extended Results Ray for efficiency. Does DexWild improve as more DexWild data is added? Table III shows steady gains as we scale from 0% to 100% of the DexWild dataset. Performance increases steadily with more D. Behavior Cloning Policy Architecture and Training Hyper- human demonstrations, with a notable jump between 25% and Parameters 50% of the dataset. These results demonstrate that DexWild Our behavior cloning policy takes as input RGB images enables scalable learning, where even comparably smaller data and relative state history. We obtain tokens for the image scales yields substantial gains, and additional data continues observation via a ViT and tokens for relative states via linear to enhance generalization layers. The weights of ViT is initialized from the Soup 1M I. Cotraining Extended Results model from [11]. We decide to include relative states as we How does DexWild react to different cotraining ratios? found it greatly increases the robustness of the policy, and Table IV groups all three raw metrics: (a) In-Domain, (b) In- enables smoother motions. In particular, for bimanual tasks, the-Wild, and (c) In-the-Wild Extreme. All evaluations were we find that including the interhand pose (pose of left hand run on xArm + LEAP Hand V2 Advanced. relative to right hand) greatly increases success rate in tasks like Florist We implement both Action Chunking Transformer [59] and Diffusion U-Net [6] as policy classes, which output a sequence of actions. The network outputs actions which consists of relative end effector actions and absolute hand joint angles. Task Policy Class In the Wild In the Wild Extreme We list the hyper-paramaters that we used for policy training Robot Only 1:2 Robot Only 1:2 using behavior-cloning in this Table V ACT 0.000 0.680 0.115 0.395 Spray Diffusion 0.050 0.628 0.120 0.520 ACT 0.458 0.583 0.125 0.458 E. Low Level Motion Control Toy Cleanup Diffusion 0.521 0.875 0.500 0.625
For optimal smoothness of our policies and safety, we employ ACT 0.025 0.508 0.000 0.350 Pour (Cross Task) Diffusion 0.000 0.958 0.000 0.917 a Riemannian Motion Policy (RMP) [34] implemented in Isaac Lab [26], where the RMP dynamically generates joint- TABLE I: DexWild Performance on Different Policy Classes space targets given end effector targets. RMP also has the added benefit of incorporating real-time collision avoidance, In the Wild In the Wild Extreme preventing self-collision between the arms and a set table Task Robot Only 1:2 Robot Only 1:2 height. Although our policies does not rely on RMP to prevent collisions, the peace of mind is appreciated. Spray 0.305 0.805 0.150 0.600 Toy Cleanup 0.500 0.656 0.250 0.542 Pour (Cross Task) 0.050 0.917 0.000 0.817 F. Comparing Policy Classes TABLE II: LEAP Hand Performance on In-the-Wild and In- Does DexWild work with different behavior cloning the-Wild Extreme Tasks. Ratio is Robot:Human policy classes? Table I compares the performance of ACT and Diffusion—across both the In-the-Wild and In-the-Wild Extreme settings. Each policy is evaluated in a robot-only Scale 0% 25% 50% 100% setting and a co-trained (1:2) setting using the DexWild dataset. Spray 0.060 0.260 0.605 0.565 Notably, Diffusion policies benefit more from DexWild co- Toy Cleanup 0.514 0.442 0.440 0.792 training, achieving the highest scores in all tasks, including substantial improvements on the Pour task where the policy Average 0.287 0.351 0.523 0.678 must generalize across tasks. These results suggest that Std 0.321 0.129 0.116 0.160 DexWild co-training enables stronger generalization, especially when paired with expressive policy architectures like Diffusion. TABLE III: Performance Scaling with DexWild Dataset Size Task Robot 1:1 1:2 1:5 Human Hyperparameter Value Spray 0.690 0.630 0.763 0.381 0.030 Training Configuration Toy Cleanup 0.604 0.792 0.833 0.708 0.042 Optimizer AdamW Average 0.647 0.711 0.798 0.545 0.036 Base Learning Rate 3e-4 Std 0.061 0.114 0.050 0.232 0.008 Optimizer Momentum β1 , β2 = 0.95, 0.999 Learning Rate Schedule Cosine (diffusers) (a) Performance Across Cotrain Ratios for Varying Deployment Warmup Steps 2000 Conditions. Ratio is Robot:Human Total Steps 70000 Batch Size 256 Task Robot 1:1 1:2 1:5 Human Environment Frequency 30 Hz Spray 0.050 0.625 0.628 0.393 0.063 Observation Settings Toy Cleanup 0.521 0.646 0.875 0.625 0.083 1 (Spray, Toy, Pour) Average 0.285 0.635 0.751 0.509 0.073 Proprioception Horizon 3 (Florist, Clothes) Std 0.333 0.015 0.175 0.164 0.015 Image Horizon 1 (all tasks) Observation Resolution 224×224 (b) In-the-Wild Task Performance 9 (Spray, Toy, Pour) Observation Dim 27 (Florist, Clothes) Task Robot 1:2 26 (Spray, Toy, Pour) Action Dimension Spray 0.120 0.520 52 (Florist, Clothes) Toy Cleanup 0.500 0.625 Action Chunk Size 48 Bimanual Florist 0.063 0.623 Action Chunking Transformer Bimanual Clothes Folding 0.198 0.740 # Encoder Layers 4 Average 0.220 0.627 # Decoder Layers 6 Std 0.195 0.090 # MHSA Heads 8 Feed-Forward Dim 3200 (c) In-the-Wild Extreme Task Performance Hidden Dim (Token Dim) 768 TABLE IV: Performance for different cotrain ratios Dropout 0.1 Feature Norm LayerNorm Diffusion U-Net Policy Train Diffusion Steps 100 Eval Diffusion Steps 16 Down Channels [256, 512, 1024] Kernel Size 3 Groups (GN) 8 Dropout 0.1 Feature Norm None
TABLE V: Full training and architecture settings used across our
experiments.