DexWild: Dexterous Human Interactions for
In-the-Wild Robot Policies
Tony Tao∗ , Mohan Kumar Srirama∗ , Jason Jingzhou Liu, Kenneth Shaw, Deepak Pathak
Carnegie Mellon University
∗ Equal contribution
In-the-Wild Human Data Generalize to Unseen Environments
Novel Objects
Novel Scenes
arXiv:2505.07813v1 [cs.RO] 12 May 2025
️ Cross-Task
Cross-Embodiment
DexWild
Limited Robot Data
Fig. 1: DexWild enables dexterous policies to generalize to new objects, scenes, and embodiments. This is achieved by leveraging large-scale,
real-world human embodiment data collected in many scenes and co-trained with a smaller robot embodiment dataset for grounding.
Abstract—Large-scale, diverse robot datasets have emerged as challenge. While there have been many breakthroughs in large
a promising path toward enabling dexterous manipulation policies language models (LLMs) [53, 51, 3] and vision language
to generalize to novel environments, but acquiring such datasets models (VLMs) [24, 48], the key to their success lies in
presents many challenges. While teleoperation provides high-
fidelity datasets, its high cost limits its scalability. Instead, what if harnessing vast datasets. In contrast, robotics faces a critical
people could use their own hands, just as they do in everyday life, hurdle: large-scale, diverse robot datasets needed to train
to collect data? In DexWild, a diverse team of data collectors uses foundation models do not yet exist.
their hands to collect hours of interactions across a multitude In recent years, a key approach to collecting robot datasets
of environments and objects. To record this data, we create has been through teleoperation, which provides high-precision,
DexWild-System, a low-cost, mobile, and easy-to-use device. The
DexWild learning framework co-trains on both human and robot high-quality action data that a policy can directly train on.
demonstrations, leading to improved performance compared to [8, 21, 54]. However, acquiring this data requires highly-
training on each dataset individually. This combination results in trained human operators working with specialized robot setups.
robust robot policies capable of generalizing to novel environments, Gathering data in diverse environments presents additional
tasks, and embodiments with minimal additional robot-specific challenges such as physically relocating the robot to each new
data. Experimental results demonstrate that DexWild significantly
improves performance, achieving a 68.5% success rate in unseen location. This data collection process is both labor-intensive
environments—nearly four times higher than policies trained and expensive, making it difficult to scale to the volume of data
with robot data only—and offering 5.8× better cross-embodiment needed for dexterous generalization in unseen environments.
generalization. Video results, codebases, and instructions at https: Another approach to scaling robot datasets is to leverage
//dexwild.github.io internet-scale video data from platforms like YouTube, which
provide vast and diverse visual grounding in real-world environ-
I. I NTRODUCTION
ments [15, 10]. However, utilizing this data effectively presents
Roboticists have long dreamed of creating robots that can significant challenges. First, publicly available videos often
perform tasks with the same dexterity and adaptability as lack the fine-grained accuracy needed to capture detailed hand
humans. We would like robots to deftly generalize to many states because vision-based body detection modules are noisy
different objects, environments, and embodiments-yet this and unreliable. Additionally, these videos are not inherently
vision of truly versatile robot behaviors remains a formidable structured with categorized episodes for task-specific learn-
ing, further complicating their direct application in robotics. B. Data Generation for Robot Manipulation
[18, 1, 40]. While some data collection efforts exist with more Overcoming the robot data bottleneck has become a central
accurate and structured data, [60, 2], they do not have enough
challenge in robot learning.
environment diversity. We seek to collect data with tracking One approach leverages internet videos to extract action
accuracy and environment diversity to enable generalizable information. Several works, such as VideoDex [40] and HOP
dexterous behavior. [42], utilize large scale human videos to learn an action prior
To overcome these barriers, some have explored collecting through retargeting, which they use to bootstrap policy training.
accurate in-the-wild human demonstrations by equipping Others, such as LAPA [57], use unlabelled videos to generate
users with a wearable gripper that directly maps their hand latent action representations that can be used for downstream
movements to robot actions [7]. However, this approach is tasks. While these video-based schemes enjoy vast visual
cumbersome, ill-suited for natural, everyday interactions, and
diversity, they typically fall short at capturing the precise, low-
constrains the collected data to a specific embodiment. Otherlevel motor commands needed for real-world manipulation.
works [55] propose using dexterous hands and gloves, but they Simulation enables rapid generation of action data at scale.
do not scale to in-the-wild environments. However, creating diverse, realistic environments for many
In this paper, we present DexWild, a system that enables tasks and addressing the sim-to-real gap is challenging. Recent
effective learning of robust dexterous manipulation policies successes in transferring manipulation policies from simulation
through co-training on human and robot demonstrations. Our [43] have been confined to tabletop settings and lack the
key contributions include: generalization needed for deployment in diverse environments.
Direct teleoperation on physical robots yields the highest
-
Scalable Data Collection System: A novel human-
fidelity, but scales poorly. Recent works have shown impressive
embodiment DexWild-System that enables untrained op-
dexterity and efficient learning in fixed scenarios [59, 56, 41,
erators to quickly collect 9,290 demonstrations across
19], yet collecting enough demonstrations to generalize across
93 diverse environments, achieving 4.6× speedup over
diverse scenes quickly becomes prohibitively expensive.
conventional robot-based methods
Recently, there has been a growing body of work that utilizes -
Efficient Co-training Framework: An approach that
purpose-collected high quality human embodiment data without
optimally combines human and robot demonstrations,
the tedious teleoperation. We discuss these approaches in the
significantly improving policy generalization to achieve
next section.
68.5% success rate in novel environments, nearly four
times higher than robot-only policies. C. Human Action Tracking Systems -
Strong Cross Embodiment and Cross Task Perfor- In order to acquire high-quality data from human motions,
mance: Our data collection system combined with our accurate hand and wrist tracking is of paramount importance. To
co-training framework achieves of 5.8× improvement in bypass the complexities of hand pose estimation, several works
cross-embodiment transfer over baselines and effective equip users with handheld robot grippers [7, 12, 46]. While
skill transfer across tasks. this approach simplifies retargeting, it constrains users to the
specific morphology of the robot gripper, limiting the diversity
II. R ELATED W ORKS of captured behavior. Moreover, many of these systems rely on
SLAM-based wrist tracking, which can fail in feature-sparse
A. Generalization for Imitation Learning environments or when occlusions occur [7, 23]—such as during
drawer opening or tool use.
Learning generalizable policies for robot manipulation has Other approaches aim to estimate both hand and wrist poses
seen rapid progress, driven largely by advances in visual directly from visual input [29, 35, 5, 45, 28, 20, 32]. These
representation learning and imitation learning from large-scale methods are easy to deploy and require no instrumentation, but
datasets. On the visual side, embodied representation learning their performance degrades significantly under occlusion—an
has benefited from egocentric datasets such as Ego4D [15] and unavoidable situation in manipulation. Alternative strategies
EPIC-KITCHENS [10], with recent methods [27, 11, 47, 39] for wrist tracking, such as IMU-based [9, 50] and outside-
leveraging these datasets to train scalable visual encoders. in optical systems [30], come with their own limitations:
However, these approaches still require substantial downstream IMUs are lightweight and portable but prone to drift, while
robot demonstrations to train control policies. optical systems are accurate yet require laborious calibration
In parallel, robot-only demonstration datasets have grown and controlled environments. DexWild leverages calibration-
significantly in scale and diversity [21, 8, 54], fueling research free Aruco tracking—significantly improving reliability and
in behavior cloning and enabling generalist policy architec- minimizing setup time as it requires a single monocular camera.
tures [49, 8, 22]. While these policies show impressive per- While vision-based methods often attempt to track both the
formance across many tasks, they often struggle to generalize wrist and fingers simultaneously, many recent systems decouple
to unseen object categories, scene layouts, or environmental the two to improve accuracy. Kinematic exoskeleton gloves
conditions [25]. This lack of robustness remains a key limitation can provide high-fidelity joint measurements and even haptic
of current systems. feedback [58], but are bulky and uncomfortable for long-term
Human Demonstration Setup Robot SetupDexWild Camera Tracking Camera Mini-PC Mocap Glove
Egocentric Exocentric Franka Panda xArm
Fig. 2: Left: DexWild efficiently capture high-fidelity data using an individual’s own hands across various environments. Right: Robot hands
are equipped with cameras aligned with the human cameras. We test DexWild on two distinct robot hands and robot arms.
use. Instead, DexWild, along with prior works [41, 55], adopts DexWild-System is designed around three core objectives:
a lightweight glove-based solution that uses electromagnetic • Portability: Allow rapid, large-scale data collection
field (EMF) sensing to estimate fingertip positions. This allows across diverse environments without requiring complex
for accurate, real-time hand tracking that is robust to occlusions calibration procedures.
and readily retargetable to a wide range of robot hands. • High Fidelity: Accurately capture fine-grained hand and
III. D EX W ILD environment interactions essential for training precise
dexterous policies.
Many believe that leveraging large, high-quality datasets is
• Embodiment-Agnostic: Enable seamless retargeting from
the key for creating dexterous robot policies that generalize
human demonstrations to a wide variety of robot hands.
[8, 49, 40, 11]. We introduce DexWild-System, a user-friendly,
high-fidelity platform for efficiently gathering natural human Portability:
hand demonstrations across diverse real-world settings. Com- To collect data in diverse real-world settings, a system must
pared to traditional teleoperation-based approaches, DexWild- be portable, robust, and usable by anyone. We design DexWild-
System enables 4.6× faster data acquisition at scale. System with these goals in mind: it is lightweight, easy to carry,
Building on this system, we propose DexWild, an imitation and can be set up in just a few minutes—enabling scalable
learning framework that co-trains on large-scale DexWild- data collection across many locations.
System human demonstrations alongside a small number of As shown in Figure 2, DexWild-System consists of only three
robot demonstrations. This approach combines the diversity components: a single tracking camera for wrist pose estimation,
and richness of human interactions with the grounding of the a battery-powered mini-PC for onboard data capture, and a
robot embodiment, enabling policies to robustly generalize custom sensor pod comprising a motion-capture glove and
across new objects, environments, and embodiments. Figure 1 synchronized palm-mounted cameras.
displays our high level approach. Unlike traditional motion capture systems [60, 13, 4, 52] that
often rely on complex outside-in tracking setups that require
A. Data Collection System calibration, DexWild-System is truly calibration free, making it
A scalable data collection system for dexterous robot learning versatile for any scenario and foolproof for untrained operators.
must enable natural, efficient, and high-fidelity collection across This is achieved by adopting a relative state-action rep-
diverse environments. To this end, we design DexWild-System: resentation, where each state and action is captured as the
a portable, user-friendly system that captures human dexterous relative difference from the previous time step’s pose. This
behavior with minimal setup and training. While previous eliminates any need for a global coordinate frame, allowing
in-the-wild data collection approaches have typically relied the tracking camera to be freely placed—either egocentrically
on sensorized grippers, we aimed to create a more intuitive or exocentrically. Additionally, the palm cameras are rigidly
hardware interface that mirrors how humans naturally interact mounted in fixed positions across both human and robot
with the world. From delicate fine-motor actions to powerful embodiments. This ensures visual observations are aligned
grasps, humans possess dexterity across a wide range of across domains, eliminating the need for further calibration at
manipulation tasks. By learning from this intrinsic capability, deployment. The external tracking camera, when carefully
DexWild-System captures rich, diverse data applicable to a positioned, can also capture supplementary environmental
broad range of robot embodiments. context useful for learning robust policies.
High Fidelity:
To learn dexterous behaviors, fine-grained, nuanced motions
must be captured in the training dataset. Although DexWild-
System consists of only a few portable components, we make
no compromises on data fidelity. Our system is designed to
accurately capture both hand and wrist actions, paired with
high-quality visual observations. Human
For wrist and hand tracking, vision-only methods are easy
to setup. However, what they gain in portability, they often
lose in accuracy and robustness—yielding noisy pose estimates
that degrade policy learning [41, 14, 32, 7].
For hand pose estimation, we use motion capture gloves,
which offer high accuracy, low latency, and robustness against
occlusions [41]. For wrist tracking, we mount ArUco markers
on the glove and track them using an external camera. This
Robot
avoids the fragility of SLAM-based wrist tracking, which often
fails in feature-sparse environments or during occlusion-heavy Fig. 3: DexWild aligns the visual observations between humans and
robots to bridge the embodiment gap. This incentivizes the model to
tasks (e.g., drawer opening). learn a task-centric rather than embodiment-centric representation.
Unlike many datasets that rely on egocentric or distant
external cameras, we place two global-shutter cameras directly
on the palm. As illustrated in Figure 2, these stereo cameras we use the data collected by DexWild to enable dexterous
capture detailed, localized interaction views with minimal policies to generalize to in-the-wild scenarios.
motion blur and a wide field of view. This wide field of
view enables policies to operate using only the onboard palm B. Training Data Modalities and Preprocessing
cameras, without any reliance on static viewpoints.
Generalization in dexterous manipulation demands both
Embodiment-Agnostic:
scale and embodiment grounding. With this goal, DexWild
To ensure the longevity and versatility of DexWild data,
collects two complementary datasets: a large-scale human
we aim for it to remain useful across different robot embod-
demonstration dataset DH using DexWild-System, and a
iments—even as hardware platforms evolve. Achieving this
smaller teleoperated robot dataset DR . Human data offers broad
goal requires careful alignment of both the observation space
task diversity and ease of collection in real-world settings, but
and the action space between humans and robots.
lacks embodiment alignment. Robot data, while limited in scale,
We begin by standardizing the observation space. Although
provides crucial grounding in the robot’s action and observation
our palm-mounted cameras have a wide field of view, we
spaces. To harness the strengths of both, we co-train policies
intentionally position them to focus primarily on the environ-
using a fixed ratio of human and robot data within a batch,
ment, minimizing the visibility of the hand itself. Importantly,
(wh , wr )—balancing diversity with embodiment grounding to
the camera placement is mirrored between the human and
enable robust generalization during deployment.
robot hands. As shown in Figure 3, this design yields visually
At each training iteration, we sample a batch consisting of
consistent observations across embodiments, allowing the
transitions xh and xr from DH and DR , respectively, according
policy to learn a shared visual representation that generalizes
to the co-training weights. Each transition xi at timestep i
across both human and robot domains.
contains:
For action space alignment, we build on insights from prior
work [17, 44], optimizing robot hand kinematics to match the • Observation oi : An observation at a given timestep
fingertip positions observed in human demonstrations. We note consists of two synchronized palm camera images
that this method is general and can work for any robot hand Ipinky and Ithumb captured at the current timestep,
embodiment. It operates with fixed hyperparameters across as well as a sequence of historical states, sampled
users and is robust to variations in hand size—eliminating the at a step size up a given horizon H, comprising of
need for user-specific tuning. {∆pi , ∆pi−step , …, ∆pi−H }. Each ∆p consists of relative
Collecting data using natural human hands offers benefits historical end-effector positions.
beyond ease of use. The diversity in hand morphology across • Action ai:i+n−1 : An action chunk of size n that includes
human demonstrators introduces useful variation, which we actions {ai , ai+1 , … , ai+n−1 }, where ai is the action at
hypothesize helps policies learn more generalizable grasping the current timestep. Specifically, ai is a 26-dimensional
strategies—particularly important given the inherent mismatch vector consisting of:
between human and robot hand kinematics. – aarm : A 9-dimensional vector describing relative end-
In summary, DexWild is a portable, high quality, human- effector position (3D) and orientation (6D).
centric system that can be worn by any operator to collect – ahand : A 17-dimensional vector describing the finger
human data in real-world environments. Next, we explain how joint position targets of the robot hand.
Spray Bottle Toy Cleanup Pouring
Florist Clothes Folding
Fig. 4: Using DexWild-System, humans can effortlessly collect accurate data with their own hands across a wide range of environments.
This data is directly used to train any robot hand to perform dexterous manipulation in a human-like way in any environment. We validate
this approach on five representative tasks. Please see videos of these tasks on our website at https://dexwild.github.io
For bimanual tasks, the observation and action spaces are action distributions more effectively than alternatives such
duplicated, and the inter-hand pose is appended to the as Gaussian Mixture Models (GMMs) or transformers. This
observation to facilitate coordination. capability becomes increasingly important in DexWild, where
While our retargeting procedure brings human and robot demonstrations are collected from multiple humans with diverse
trajectories into a shared action space, a few additional steps strategies, resulting in inherently multi-modal behaviors. As
are necessary to make the human and robot datasets compatible the dataset scales, modeling this variability becomes critical for
for joint training: robust policy learning. Specifically, DexWild uses a diffusion
U-Net model [6] to generate action chunks.
• Action Normalization: The actions of human and robot
data are normalized separately to account for inherent Concretely, the training procedure is outlined in Algorithm 1.
distribution mismatches.
• Demo Filtering: Since human demonstrations are col- Algorithm 1 DexWild Imitation Learning Procedure
lected by untrained operators in uncontrolled environments, Require: Human dataset DH , Robot dataset DR , Co-training weights
we apply a heuristic-based filtering pipeline to automati- {ωh , ωr }
1: Initialize policy πθ with ViT encoder ϕvit
cally detect and remove low-quality or invalid trajectories.
2: while not converged do
This filtering step significantly improves dataset quality 3: Sample a batch of transitions {xh }, {xr } from DH , DR using
without manual labeling. weights {ωh , ωr }
4: for each transition xi in the batch do
C. Policy Training 5: Extract observation oi
6: Encode images: Zi = ϕvit (oi )
Through the careful design of our hardware, observation, and 7: Extract ground truth action chunk ai:i+n−1 =
action interfaces, we are able to train dexterous robot policies {ai , … , ai+n−1 }
using a simple behavior cloning (BC) objective [31, 37, 36]. To 8: Sample noise scale t ∼ U(1, T )
effectively learn from our multimodal, diverse data, our training 9: Add noise ϵt ∼ N (0, σt ) to ai:i+n−1
pipeline leverages large-scale pre-trained visual encoders and 10: Predict noise ϵ̂θ = πθ (Zi , ai:i+n−1 + ϵt , t)
11: Compute diffusion loss Lθ = ∥ϵt − ϵ̂θ ∥22
shows strong performance across different policy architectures. 12: end for
Visual Encoder: Training on DexWild data exposes our 13: Update policy parameters θ
policy to significant visual diversity—across scenes, objects, 14: end while
and lighting—requiring an encoder that generalizes well to
such variability. To address this, we adopt a pre-trained Vision
An important finding in our training framework is that tuning
Transformer (ViT) backbone, which has shown superior perfor-
the human-to-robot data weighting significantly affects real-
mance over ResNet-based encoders on in-the-wild manipulation
world performance. We discuss these effects in Section V-A.
tasks [16, 23]. Pre-trained ViTs, especially those trained
on large internet-scale datasets, are particularly effective at
IV. E XPERIMENTS
extracting rich, transferable features [27, 33, 47, 11], making
them well-suited for our setting. Our experimental evaluation encompasses extensive real-
Policy Class: While several imitation learning architectures world deployment across diverse environments and robots, uti-
have been proposed recently [59, 6], we adopt a diffusion- lizing both human demonstrations and robot teleoperation data.
based policy. Diffusion models are particularly well-suited Below, we outline our data collection process, experimental
for dexterous manipulation, as they can capture multi-modal setup, and evaluation tasks.
Train
Test
Fig. 5: We collect data using a diverse set of objects across categories. Spray Bottle Task – 25 Train, 11 Test; Toy Cleanup Task – 64 Train,
9 Test; Pour Task – 35 Train, 5 Test; Florist Task - 6 Train, 2 Test; Clothes Folding Task - 17 Train, 6 Test.
A. Scaling up Data Collection These tasks systematically evaluate DexWilds functional
Our hardware system was deployed to 10 untrained users to grasping capabilities, generalization across object types, trans-
collect data across a wide range of real-world environments. feral of skills across tasks, coordination between arms, and
These settings included indoor and outdoor locations, day and adaptability to deformable objects. Success requires the policy
night conditions, crowded cafeterias and quiet study areas, to adapt to varying object properties, environmental conditions,
with varied tables, objects, and lighting setups. The collectors and task constraints.
themselves varied in hand sizes and demonstration styles, C. Evaluation Environments
enabling us to learn from a wide distribution of environments
and interactions. For robot experiments, we employed an xArm robot and
We constructed two datasets through our collection efforts: Franka system, both equipped with either LEAP hand or LEAP
DH (human-collected data) and DR (robot-collected data). The hand V2 Advanced [38, 41]. Unless explicitly mentioned, xArm
human dataset DH comprises 9,290 demonstrations across five and LEAP hand V2 Advanced was used. We evaluate our
tasks: 3,000 demonstrations from 30 different environments approach across three scenarios:
for each of the Spray Bottle and Toy Cleanup tasks, 621 1) In-Domain: Environments where robot training data was
trajectories from 6 environments for the Pour task, 1,545 collected, testing with novel objects
demonstrations from 15 environments for the Florist task, and 2) In-the-Wild: Environments present in DexWild but absent
1,124 demonstrations from 12 environments for the Clothes from robot training data
Folding task. 3) In-the-Wild Extreme: Unseen environments absent from
The robot dataset DR includes 1,395 demonstrations: 388 both datasets.
for Spray Bottle, 370 for Toy Cleanup, 111 for Pour, 236 for V. A NALYSIS AND R ESULTS
Florist, and 290 for Clothes Folding tasks. Robot data was
In our evaluations, we seek to investigate the following key
collected using an xArm and LEAP hand V2 Advanced. Our
questions:
training and test objects are detailed in Figure 5.
1) How effectively does DexWild leverage human data to
B. Evaluation Tasks achieve strong in-the-wild performance?
We evaluate our approach on five diverse manipulation 2) Does DexWild enable policy transfer across tasks and
tasks, each designed to assess specific aspects of dexterous robot embodiments?
manipulation: functional grasping, long-horizon planning, cross- 3) Does policy performance scale effectively with increasing
task transfer, bimanual coordination, and deformable object amounts of DexWild-System data?
manipulation. A task visualization is provided in Figure 4. Please see videos of our results at https://dexwild.github.io.
In the Spray Bottle task, the robot grasps a spray bottle by
the handle and sprays a target cloth, testing functional grasping A. Zero Shot In the Wild Policies w/ DexWild
and affordance understanding. In Toy Cleanup, the robot DexWild enables strong policy generalization in novel
picks up scattered toys and places them in a bin, evaluating scenes. We evaluate policies in environments with increasing
generalization and long-horizon planning. The Pouring task novelty to assess their generalization. As shown in Figure 6,
involves tilting a bottle to pour into a container, demonstrating policies trained exclusively on robot data perform well in in-
skill transfer from the spray bottle task. In Bimanual Florist, domain settings (64.7% success rate) but degrade significantly
the robot hands over a flower between its arms and inserts it in more challenging scenarios—in-the-wild (28.5%) and in-
into a vase, testing precise bimanual coordination. Finally, in the-wild extreme (22.0%). This 36-point performance drop
Bimanual Clothes Folding, the robot uses both hands to fold a suggests that robot-only policies overfit to environment-specific
clothing item, assessing manipulation of deformable objects. features and fail to develop robust, transferable representations.
Full task specifications and scoring criteria for all tasks are In contrast, policies trained only on human data learn high-
provided in Appendix VII-A. level object affordances and approach objects reliably, even
In-Domain Performance In the Wild Performance In the Wild Extreme Performance
0.8
0.8 0.6
0.6 0.6
Score
0.4
0.4 0.4
0.2 0.2 0.2
0.0 Spray Toy Average 0.0 Spray Toy Average 0.0 Spray Toy Florist Clothes Average
Cleanup Cleanup Cleanup Folding
Robot Only Co-train 1:1 Co-train 1:2 (Ours) Co-train 1:5 Human Only
Fig. 6: How does co-training help with scaling up in the wild performance? We evaluate our policy across three scenarios: (a) In-Domain
scenes where robot training data was collected but with novel objects, (b) In-the-Wild scenes present in DexWild but not in robot data, and
(c) In-the-Wild Extreme scenes absent from both datasets. Displayed ratio is Robot:Human.
in complex scenes. However, without robot-specific action transfer using the pouring task, which shares many motion
grounding, they struggle to execute precise manipulation, primitives with the spray task. Crucially, we use no robot data
resulting in poor performance across all scenarios (3.6% in- for pouring and instead combine human (DexWild-System)
domain, 7.3% in-the-wild). demonstrations of pouring with robot demonstrations from
To combine the strengths of both modalities, we adopt a spraying. This setup enables zero-shot generalization to
co-training strategy—jointly training on both robot and human pouring in in-the-wild extreme environments. Using a 1:2
data—a method validated in prior works [8, 49, 21, 20, 32]. robot-to-human co-training ratio, our policy achieves a 94%
This encourages the policy to learn task-relevant features rather success rate, far exceeding policies trained with only robot
than overfitting to specific embodiments or environments. We (0%) or only human data (11%).
experiment with different robot-to-human data ratios (1:1 to DexWild enables transfer across robot embodiments.
1:5) per training batch. Our empirical analysis reveals that a Since DexWild data is not tied to any specific embodiment, it
1:2 ratio yields optimal performance across all scenarios: naturally supports cross-platform transfer. This prolongs the
-
In Domain: 79.8% vs. 64.7% (robot-only) value of our data, as collecting platform-specific data for every
-
In-the-wild: 75.1% vs. 28.5% (robot-only) new robot is resource-intensive and impractical. We test two
-
In-the-wild Extreme: 62.7% vs. 22.0% (robot-only) transfer scenarios in in-the-wild extreme scenes:
Interestingly, increasing the human data ratio further (e.g., • Cross-arm: Transferring from an xArm to a Franka Panda
1:5) degrades performance (54.5% in-domain, 50.9% in-the- arm. We achieve a 37.5% success rate, compared to 4.5%
wild), indicating that robot data remains essential for grounding for the robot-only baseline—an 8.3× improvement.
fine-grained control. • Cross-hand: Transferring from the LEAP Hand V2
DexWild extends to complex bimanual coordination Advanced to the original LEAP Hand. We achieve 65.3%
tasks. To evaluate whether DexWild generalizes beyond single- success versus 13.3% for the baseline, showing that
arm tasks, we test it on bimanual tasks that demand precise DexWild generalizes not only across arms, but across
coordination between two hands. We compare co-trained dexterous hands as well.
policies (1:2 ratio) against robot-only policies in in-the-wild These results, shown in Figure 7, demonstrate that DexWild
extreme settings. DexWild policies achieve a strong 68.1% enables zero-shot generalization to new tasks and hardware
average success rate, compared to just 13% for the robot- embodiments without any additional robot-specific data,
only baseline. Even when failures occur, DexWild policies making it an efficient and general framework for dexterous
exhibit meaningful attempts at task execution—while robot- policy learning on many robots.
only policies often produce erratic or unstructured behavior.
These results demonstrate that DexWild not only enables C. Scalability of DexWild
robust generalization across environments but also scales to
more complex manipulation behaviors. Policy performance scales with dataset size. To understand
how data scale impacts policy performance in the wild, we
B. Robust Cross-Task and Cross-Embodiment Generalization randomly sample subsets of the full human dataset at varying
DexWild enables transfer of low-level skills across tasks. sizes and evaluate the resulting policies. We fix the size of
Many manipulation tasks share foundational motor skills—such the robot dataset. As shown in Figure 7, there is a clear
as lifting, orienting, and rotating objects—which opens the positive correlation between dataset size and average task
door to skill reuse across related tasks. For example, opening performance—rising from 28.7% at 20% dataset size to 67.8%
a microwave and opening a cupboard both involve similar with the full dataset, marking a 2.36× improvement. Interest-
coordination and control. We evaluate this form of cross-task ingly, the learning curve is nonlinear, with especially steep gains
1.0 Cross-Task Performance 1.0 Cross-Embodiment Performance 1.0 Scaling Performance
0.8 0.8 0.8
0.6 0.6 0.6
Score0.4 0.4 0.4 0.2 0.2 0.2 0.0 0.0 Cross-Hand Cross-Arm 0.0 0 25 50 100 Type DexWild Dataset Scale (%) Robot Only Co-train 1:1 Co-train 1:2 (Ours) Co-train 1:5 Human OnlyFig. 7: Left: Cross-Task Performance – Evaluating DexWild on the pour task using robot data exclusively from the spray task. Middle:
Cross-Embodiment Performance – Testing DexWild policy on the Original LEAP hand and a Franka robot arm. Right: Scaling Performance
– Demonstrating improved DexWild performance as dataset size increases. Displayed ratio is Robot:Human.
in the 25–50% range, suggesting a critical threshold where the DexWild not only demonstrates strong scaling trends with
policy begins to reliably learn generalizable behaviors. increasing data volume, but also offers a practical and efficient
Importantly, performance continues to improve all the way path to collecting diverse, high-quality data at scale—crucial
to 100% data usage, indicating that the system has not yet for real-world generalization.
plateaued. This suggests that even more capable policies could
VI. C ONCLUSION AND L IMITATIONS
be learned with continued data collection.
DexWild-System enables fast and scalable data collection. We introduce DexWild, a scalable framework for learning
Given the observed benefits of scaling, we evaluate the data dexterous manipulation policies that effectively generalize to
collection efficiency of DexWild-System via a comparative new tasks, environments, and robot embodiments. We introduce
user study measuring demonstrations per hour. As shown in DexWild-System, a portable, human-centric data collection
Figure 8, DexWild-System achieves an average collection rate device that significantly accelerates dataset creation (4.6×
of 201 demos/hour across five representative tasks—nearly faster than conventional robot teleoperation). We propose
matching the rate of demonstrations collected using bare hands DexWild cotraining method, which leverages large scale human
and 4.6× faster than a traditional robot teleoperation system demonstrations alongside minimal robot data to achieve robust
based on Gello [41, 56], which achieves just 43 demos/hour. generalization-reaching a success rate of 68.5% in completely
We identify three key limitations of Gello-based collection unseen environments, nearly four times higher than methods
that our system overcomes: using robot data only. Furthermore, DexWild’s embodiment-
agnostic design enables strong cross-embodiment and cross-task
-
Lack of haptic feedback: Operators cannot feel objects,
transfer capabilities, reducing the need for robot-specific data.
making fine manipulation difficult for certain tasks.
Despite these strengths, several limitations remain that -
Scene reset: Resetting the environment is cumbersome
motivate future research: First, our approach still depends on
and often requires a second operator or pauses in data
a limited number of teleoperated robot data to bridge the gap
collection.
between human and robot actions. Future work could explore -
Hardware setup overhead: Robots are heavy and require
improved retargeting or online policy adaptation to remove
time-consuming setup at each new location, whereas
the need for teleoperated data. Next, because humans typically
DexWild-System is portable and can be set up in minutes.
perform these tasks successfully, their demonstrations seldom
include error recovery—causing trained policies to struggle to
Data Collection Speed by Method recover from unexpected failures. Adding recovery examples or
250 adaptive strategies could boost real-world robustness. Finally,
our method uses only visual and kinematic data, which limits its
200
Demos per hourperformance in contact-rich tasks. Incorporating tactile or haptic 150 sensing could improve the handling of delicate interactions.1: 100 In summary, DexWild represents a significant step toward 50 scalable, generalizable robot manipulation policies. Our results 0 highlight the promise of leveraging human interaction data at Spray Toy Pour Florist Clothes Average Cleanup Folding scale, offering an exciting avenue toward truly dexterous and Robot Ours Human versatile robots operating in diverse, real-world environments.Fig. 8: DexWild-System offers 4.6× improvement over robot data Videos, code, and hardware instructions are available on our
collection speed and nearly matches the human bare hands data website at https://dexwild.github.io
collection speed.
ACKNOWLEDGMENTS Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kemb-
havi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit
We would like to thank Yulong Li, Hengkai Pan, and Sandeep
Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna,
Routray for thoughtful discussions. We’d also like to thank
Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim,
Andrew Wang for setting up compute and Yulong Li for helping
Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu
with robot system setup. Lastly, we’d like to express thanks to
Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang,
Hengkai Pan, Andrew Wang, Adam Kan, Ray Liu, Mingxuan
Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine
Li, Lukas Vargas, Jose German, Laya Satish, Sri Shasanka
Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline
Madduri for helping collect data. This work was supported
Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne
in part by AFOSR FA9550-23-1-0747 and Apple Research
Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler,
Award.
Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh,
Edward Johns, Ethan Foster, Fangchen Liu, Federico
R EFERENCES
Ceola, Fei Xia, Feiyu Zhao, Felipe Vieira Frujeri, Freek
[1] Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salho-
and Deepak Pathak. Affordances from human videos as tra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth,
a versatile representation for robotics. 2023. 2 Gregory Kahn, Guangwen Yang, Guanzhi Wang, Hao Su,
[2] Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben
Shreyas Hampali, Shangchen Han, Fan Zhang, Linguang Amor, Henrik I Christensen, Hiroki Furuta, Homanga
Zhang, Jade Fountain, Edward Miller, Selen Basol, et al. Bharadhwaj, Homer Walke, Hongjie Fang, Huy Ha, Igor
Hot3d: Hand and object tracking in 3d from egocentric Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang,
multi-view videos. arXiv preprint arXiv:2411.19167, Jad Abou-Chakra, Jaehyung Kim, Jaimyn Drake, Jan
2024. 2 Peters, Jan Schneider, Jasmine Hsu, Jay Vakil, Jeannette
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Bohg, Jeffrey Bingham, Jeffrey Wu, Jensen Gao, Jiaheng
Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo,
Neelakantan, Pranav Shyam, Girish Sastry, Amanda Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu,
Askell, et al. Language models are few-shot learners. Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna,
Advances in neural information processing systems, 33: Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi
1877–1901, 2020. 1 Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang,
[4] Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan
Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra
Van Wyk, Umar Iqbal, Stan Birchfield, et al. Dexycb: Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin
A benchmark for capturing hand grasping of objects. In Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran
Proceedings of the IEEE/CVF Conference on Computer Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan,
Vision and Pattern Recognition, pages 9044–9053, 2021. Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle
3 Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen,
[5] Xuxin Cheng, Jialong Li, Shiqi Yang, Ge Yang, and Lerrel Pinto, Li Fei-Fei, Liam Tan, Linxi ”Jim” Fan,
Xiaolong Wang. Open-television: Teleoperation with Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion
immersive active visual feedback. arXiv preprint Lepert, Marius Memmel, Masayoshi Tomizuka, Masha
arXiv:2407.01512, 2024. 2 Itkina, Mateo Guaman Castro, Max Spero, Maximilian
[6] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Du, Michael Ahn, Michael C. Yip, Mingtong Zhang,
Cousineau, Benjamin Burchfiel, and Shuran Song. Dif- Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit
fusion policy: Visuomotor policy learning via action Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen,
diffusion. In Proceedings of Robotics: Science and Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning
Systems (RSS), 2023. 5, 14 Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah,
[7] Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R
Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Sanketi, Patrick ”Tree” Miller, Patrick Yin, Paul Wohlhart,
Shuran Song. Universal manipulation interface: In- Peng Xu, Peter David Fagan, Peter Mitrano, Pierre
the-wild robot teaching without in-the-wild robots. In Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen,
Proceedings of the IEEE International Conference on Quan Vuong, Rafael Rafailov, Ran Tian, Ria Doshi,
Robotics and Automation (ICRA), 2024. 2, 4 Roberto Mart’in-Mart’in, Rohan Baijal, Rosario Scalise,
[8] Open X-Embodiment Collaboration, Abby O’Neill, Abdul Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang,
Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian,
Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan
Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham
Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khaz- Sonawani, Shubham Tulsiani, Shuran Song, Sichun Xu,
atsky, Anant Rai, Anchit Gupta, Andrew Wang, Andrey Siddhant Haldar, Siddharth Karamcheti, Simeon Ade-
bola, Simon Guist, Soroush Nasiriany, Stefan Schaal, (CVPR), pages 11743–11753, 2022. 1, 2
Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, [16] Huy Ha, Yihuai Gao, Zipeng Fu, Jie Tan, and Shuran
Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Song. UMI on legs: Making manipulation policies mobile
Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, with manipulation-centric whole-body controllers. In
Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Proceedings of the 2024 Conference on Robot Learning,
Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. 2024. 5
Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, [17] Ankur Handa, Karl Van Wyk, Wei Yang, Jacky Liang,
Vidhi Jain, Vikash Kumar, Vincent Vanhoucke, Wei Zhan, Yu-Wei Chao, Qian Wan, Stan Birchfield, Nathan Ratliff,
Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu and Dieter Fox. Dexpilot: Vision-based teleoperation
Chen, Xiaolong Wang, Xinghao Zhu, Xinyang Geng, of dexterous robotic hand-arm system. In 2020 IEEE
Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yansong Pang, International Conference on Robotics and Automation
Yao Lu, Yecheng Jason Ma, Yejin Kim, Yevgen Chebotar, (ICRA), pages 9164–9170. IEEE, 2020. 4
Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan [18] Yafei Hu, Quanting Xie, Vidhi Jain, Jonathan Francis,
Wang, Yonatan Bisk, Yongqiang Dou, Yoonyoung Cho, Jay Patrikar, Nikhil Keetha, Seungchan Kim, Yaqi Xie,
Youngwoon Lee, Yuchen Cui, Yue Cao, Yueh-Hua Wu, Tianyi Zhang, Zhibo Zhao, et al. Toward general-purpose
Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang, robots via foundation models: A survey and meta-analysis.
Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka arXiv preprint arXiv:2312.08782, 2023. 2
Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen [19] Aadhithya Iyer, Zhuoran Peng, Yinlong Dai, Irmak Guzey,
Zhang, Zipeng Fu, and Zipeng Lin. Open X-Embodiment: Siddhant Haldar, Soumith Chintala, and Lerrel Pinto.
Robotic learning datasets and RT-X models, 2023. 1, 2, OPEN TEACH: A versatile teleoperation system for
3, 7 robotic manipulation. arXiv preprint arXiv:2403.07870,
[9] Juan Antonio Corrales, Francisco A Candelas, and Fer- 2024. 2
nando Torres. Hybrid tracking of human operators using [20] Simar Kareer, Dhruv Patel, Ryan Punamiya, Pranay
imu/uwb data fusion by a kalman filter. In Proceedings Mathur, Shuo Cheng, Chen Wang, Judy Hoffman, and
of the 3rd ACM/IEEE international conference on Human Danfei Xu. Egomimic: Scaling imitation learning via
robot interaction, pages 193–200, 2008. 2 egocentric video, 2024. 2, 7
[10] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, [21] Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ash-
Sanja Fidler, Antonino Furnari, Evangelos Kazakos, win Balakrishna, Sudeep Dasari, Siddharth Karamcheti,
Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yun-
Price, and Michael Wray. Epic-kitchens: A large-scale liang Chen, Kirsty Ellis, et al. Droid: A large-scale
dataset for recognizing, anticipating, and retrieving hand- in-the-wild robot manipulation dataset. arXiv preprint
object interactions. In Proceedings of the European arXiv:2403.12945, 2024. 1, 2, 7
Conference on Computer Vision (ECCV), pages 802–819, [22] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted
2018. 1, 2 Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov,
[11] Sudeep Dasari, Mohan Kumar Srirama, Unnat Jain, and Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla:
Abhinav Gupta. An unbiased look at datasets for visuo- An open-source vision-language-action model. arXiv
motor pre-training. In Conference on Robot Learning. preprint arXiv:2406.09246, 2024. 2
PMLR, 2023. 2, 3, 5, 14 [23] Fanqi Lin, Yingdong Hu, Pingyue Sheng, Chuan Wen,
[12] Haritheja Etukuru, Norihito Naka, Zijin Hu, Seungjae Jiacheng You, and Yang Gao. Data scaling laws in
Lee, Julian Mehu, Aaron Edsinger, Chris Paxton, Soumith imitation learning for robotic manipulation. In Conference
Chintala, Lerrel Pinto, and Nur Muhammad Mahi Shafiul- on Robot Learning (CoRL), 2024. 2, 5
lah. Robot utility models: General policies for zero-shot [24] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae
deployment in new environments, 2024. 2 Lee. Visual instruction tuning. In NeurIPS, 2023. 1
[13] Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed [25] Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush
Kocabas, Manuel Kaufmann, Michael J Black, and Otmar Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio
Hilliges. Arctic: A dataset for dexterous bimanual hand- Savarese, Yuke Zhu, and Roberto Martı́n-Martı́n. What
object manipulation. In Proceedings of the IEEE/CVF matters in learning from offline human demonstrations
Conference on Computer Vision and Pattern Recognition, for robot manipulation. arXiv preprint arXiv:2108.03298,
pages 12943–12954, 2023. 3 2021. 2
[14] Authors from UC San Diego and MIT. Open-television: [26] Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu,
An open-source immersive teleoperation system with Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh,
stereo visual feedback. The Robot Report, 2024. 4 Yunrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck
[15] Kristen Grauman, Michael Ryoo, Aljoša Smolić, Minh Babich, Gavriel State, Marco Hutter, and Animesh Garg.
Vo, and et al. Ego4d: Around the world in 3,000 hours Orbit: A unified simulation framework for interactive
of egocentric video. In Proceedings of the IEEE/CVF robot learning environments. IEEE Robotics and Au-
Conference on Computer Vision and Pattern Recognition tomation Letters, 8(6):3740–3747, June 2023. ISSN
2377-3774. doi: 10.1109/lra.2023.3270034. URL http: Karen Liu, Dana Kulic, and Jeff Ichnowski, editors,
//dx.doi.org/10.1109/LRA.2023.3270034. 14 Proceedings of The 6th Conference on Robot Learning,
[27] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, and volume 205 of Proceedings of Machine Learning Re-
Chelsea Finn. R3M: A universal visual representation search, pages 654–665. PMLR, 14–18 Dec 2023. 2
for robot manipulation. arXiv preprint arXiv:2203.12601, [40] Kenneth Shaw, Shikhar Bahl, and Deepak Pathak.
2022. 2, 5 Videodex: Learning dexterity from internet videos. In
[28] Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Conference on Robot Learning, pages 654–665. PMLR,
Angjoo Kanazawa, David Fouhey, and Jitendra Malik. 2023. 2, 3
Hamer: Hand mesh recovery for the egoexo4d hand pose [41] Kenneth Shaw, Yulong Li, Jiahui Yang, Mohan Kumar
challenge. 2 Srirama, Ray Liu, Haoyu Xiong, Russell Mendonca, and
[29] Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Deepak Pathak. Bimanual dexterity for complex tasks.
Angjoo Kanazawa, David Fouhey, and Jitendra Malik. In 8th Annual Conference on Robot Learning, 2024. 2,
Reconstructing hands in 3d with transformers. In 3, 4, 6, 8, 14
Proceedings of the IEEE/CVF Conference on Computer [42] Himanshu Gaurav Singh, Antonio Loquercio, Carmelo
Vision and Pattern Recognition (CVPR), 2024. 2 Sferrazza, Jane Wu, Haozhi Qi, Pieter Abbeel, and
[30] Alexandra Pfister, Alexandre M West, Shaw Bronner, and Jitendra Malik. Hand-object interaction pretraining from
Jack Adam Noah. Comparative abilities of microsoft videos, 2024. URL https://arxiv.org/abs/2409.08273. 2
kinect and vicon 3d motion capture for gait analysis. [43] Ritvik Singh, Arthur Allshire, Ankur Handa, Nathan
Journal of medical engineering & technology, 38(5):274– Ratliff, and Karl Van Wyk. Dextrah-rgb: Visuomotor
280, 2014. 2 policies to grasp anything with dexterous hands. arXiv
[31] Dean A Pomerleau. Alvinn: An autonomous land vehicle preprint arXiv:2412.01791, 2024. 2
in a neural network. Advances in neural information [44] Aravind Sivakumar, Kenneth Shaw, and Deepak Pathak.
processing systems, 1, 1988. 5 Robotic telekinesis: Learning a robotic hand imitator by
[32] Ri-Zhao Qiu, Shiqi Yang, Xuxin Cheng, Chaitanya watching humans on youtube, 2022. 4
Chawla, Jialong Li, Tairan He, Ge Yan, David J. Yoon, [45] Aravind Sivakumar, Kenneth Shaw, and Deepak Pathak.
Ryan Hoque, Lars Paulsen, Ge Yang, Jian Zhang, Sha Robotic telekinesis: Learning a robotic hand imitator by
Yi, Guanya Shi, and Xiaolong Wang. Humanoid policy ˜ watching humans on youtube. RSS, 2022. 2
human policy. arXiv preprint arXiv:2503.13441, 2025. 2, [46] Shuran Song, Andy Zeng, Johnny Lee, and Thomas
4, 7 Funkhouser. Grasping in the wild: Learning 6dof closed-
[33] Ilija Radosavovic, Tete Xiao, Stephen James, Pieter loop grasping from low-cost demonstrations. Robotics
Abbeel, Jitendra Malik, and Trevor Darrell. Real-world and Automation Letters, 2020. 2
robot learning with masked visual pre-training. CoRL, [47] Mohan Kumar Srirama, Sudeep Dasari, Shikhar Bahl, and
2022. 5 Abhinav Gupta. HRP: Human affordances for robotic
[34] Nathan D. Ratliff, Jan Issac, Daniel Kappler, Stan Birch- pre-training. In Proceedings of Robotics: Science and
field, and Dieter Fox. Riemannian motion policies, 2018. Systems, Delft, Netherlands, 2024. 2, 5
URL https://arxiv.org/abs/1801.02854. 14 [48] Andreas Steiner, André Susano Pinto, Michael Tschan-
[35] Yu Rong, Takaaki Shiratori, and Hanbyul Joo. Frankmo- nen, Daniel Keysers, Xiao Wang, Yonatan Bitton,
cap: A monocular 3d whole-body pose estimation system Alexey Gritsenko, Matthias Minderer, Anthony Sher-
via regression and integration. In Proceedings of the bondy, Shangbang Long, Siyang Qin, Reeve Ingle,
IEEE/CVF International Conference on Computer Vision, Emanuele Bugliarello, Sahar Kazemzadeh, Thomas Mes-
pages 1749–1759, 2021. 2 nard, Ibrahim Alabdulmohsin, Lucas Beyer, and Xiaohua
[36] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A Zhai. PaliGemma 2: A Family of Versatile VLMs for
reduction of imitation learning and structured prediction to Transfer. arXiv preprint arXiv:2412.03555, 2024. 1
no-regret online learning. In Proceedings of the fourteenth [49] OM Team, D Ghosh, H Walke, K Pertsch, K Black,
international conference on artificial intelligence and O Mees, S Dasari, J Hejna, C Xu, J Luo, et al. Octo:
statistics, pages 627–635. JMLR Workshop and Confer- An open-source generalist robot policy. Proceedings of
ence Proceedings, 2011. 5 Robotics: Science and Systems, Delft, Netherlands, 2023.
[37] Stefan Schaal. Is imitation learning the route to humanoid 2, 3, 7
robots? Trends in cognitive sciences, 3(6):233–242, 1999. [50] Yushuang Tian, Xiaoli Meng, Dapeng Tao, Dongquan
5 Liu, and Chen Feng. Upper limb motion tracking with
[38] Kenneth Shaw, Ananye Agarwal, and Deepak Pathak. the integration of imu and kinect. Neurocomputing, 159:
Leap hand: Low-cost, efficient, and anthropomorphic hand 207–218, 2015. 2
for robot learning. Robotics: Science and Systems (RSS), [51] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier
2023. 6 Martinet, Marie-Anne Lachaux, Timothée Lacroix, Bap-
[39] Kenneth Shaw, Shikhar Bahl, and Deepak Pathak. tiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar,
Videodex: Learning dexterity from internet videos. In et al. Llama: Open and efficient foundation language
models. arXiv preprint arXiv:2302.13971, 2023. 1
[52] Valve Corporation. https://store.steampowered.com/
steamvr. [Virtual reality platform]. 3
[53] A Vaswani. Attention is all you need. Advances in Neural
Information Processing Systems, 2017. 1
[54] Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim,
Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-
Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan
Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2:
A dataset for robot learning at scale. In Conference on
Robot Learning (CoRL), 2023. 1, 2
[55] Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang,
Li Fei-Fei, and Karen Liu. Dexcap: Scalable and portable
mocap data collection system for dexterous manipulation.
In Robotics: Science and Systems (RSS), 2024. 2, 3
[56] Philipp Wu, Yide Shentu, Zhongke Yi, Xingyu Lin, and
Pieter Abbeel. Gello: A general, low-cost, and intuitive
teleoperation framework for robot manipulators, 2023. 2,
8
[57] Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo,
Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben
Tan, Yu-Wei Chao, Bill Yuchen Lin, et al. Latent action
pretraining from videos. arXiv preprint arXiv:2410.11758,
2024. 2
[58] Han Zhang, Songbo Hu, Zhecheng Yuan, and Huazhe
Xu. Doglove: Dexterous manipulation with a low-cost
open-source haptic force feedback glove. arXiv preprint
arXiv:2502.07730, 2025. 2
[59] Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea
Finn. Learning fine-grained bimanual manipulation with
low-cost hardware. In Proceedings of Robotics: Science
and Systems (RSS), 2023. 2, 5, 14
[60] Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan
Russell, Max Argus, and Thomas Brox. Freihand: A
dataset for markerless capture of hand pose and shape
from single rgb images. In Proceedings of the IEEE/CVF
International Conference on Computer Vision, pages 813–
822, 2019. 2, 3
VII. A PPENDIX
Videos of our results, code to recreate our system, and
hardware instructions are available on our website at https:
//dexwild.github.io
A. Detailed Task Description and Scoring Criteria:
We evaluate five dexterous manipulation tasks, each designed
to assess different capabilities such as functional grasping,
long-horizon planning, precision, bimanual coordination, and
deformable object manipulation. Each task is scored according
to a structured rubric based on discrete completion milestones.
The task scoring criteria are designed to quantify the
performance of different robot tasks based on specific com-
pletion milestones. Each task has a set of defined actions
with corresponding point values. Higher scores are assigned to
more complex or functionally successful actions, while partial Fig. 9: DexWild-System features a simple and easy-to-use interface
completions and failed attempts receive lower scores. This for deployment by untrained data collectors.
structured scoring system allows for consistent evaluation and
comparison of task performance.
– 0.75: Grasp the bouquet, handover
Spray Bottle – 1.00: Grasp the bouquet, handover, insert into vase
This task evaluates functional grasping and affordance under-
standing. The robot must grasp a spray bottle and orient it to Clothes Folding
spray over a target cloth. This task tests manipulation of deformable objects using both
hands. The robot must fold a clothing item placed on a surface.
– 0.00: Nothing
– 0.15: Tries functional grasp but fails – 0.00: Nothing
– 0.25: Grasp bottle – 0.25: Tries grasp but fails
– 0.75: Grasp bottle, orient over cloth – 0.50: Grasp with one hand
– 0.75: Grasp bottle, use functional grasp – 0.75: Grasp with both hands
– 1.00: Grasp bottle, use functional grasp, orient over cloth – 1.00: Grasp and fold
Toy Cleanup B. Data Collection Procedure
This task tests long-horizon planning and generalization. The To deploy DexWild-System with untrained data collectors,
robot must collect scattered toys and deposit them in a we provide a one-page instruction sheet outlining the task,
designated bin. object setup, and system startup/shutdown. DexWild-System
– 0.00: Nothing includes three core components: a wrist-tracking camera, a
– 0.25: Tries for grasp but fails battery-powered mini-PC for onboard data capture, and a
– 0.50: Grasp object custom sensor pod with a motion-capture glove and palm-
– 1.00: Grasp object, drop into bin mounted cameras. At a new site, users simply wear the mocap
Pouring glove and power on the mini-PC with a provided power
This task assesses precise motion control and transfer learning bank. For egocentric tracking, a headstrap holds the tracking
from the spray bottle task. The robot must pour liquid from a camera; for exocentric tracking, we provide a collapsible
bottle into a container. tripod. Once booted, users launch our custom desktop app
– 0.00: Nothing and control recording via a Bluetooth clicker or foot pedal.
– 0.15: Tries functional grasp but fails The UI (Fig. 9) shows sensor status, SLAM recording, and
– 0.25: Grasp bottle data capture indicators, along with buttons to view the tracking
– 0.75: Grasp bottle, pour into container camera feed and delete the last episode. Collectors gather 100
– 0.75: Grasp bottle, use functional grasp episodes per location. After the day is finished, we upload the
– 1.00: Grasp bottle, use functional grasp, pour into container data to our remote machine for processing.
Bimanual Florist C. Downstream Data Processing
This task evaluates coordinated control of both hands. The Each episode is stored in its own folder, with subfolders
robot must pick up a flower, hand it to the other arm, and organizing individual actions and observations. SVO recordings
insert it into a vase. from the Zed Mini camera—used for SLAM and wrist pose
– 0.00: Nothing tracking—are saved separately, with each file covering five
– 0.15: Tries grasp but fails episodes. To begin data processing, we use the Zed SDK
– 0.25: Grasp the bouquet to decode these SVO files, reconstruct the camera’s motion,
and perform ArUco cube tracking and wrist pose estimation G. Cross Hand Extended Results
using both the left image and stereo depth data. We then Does DexWild generalize across different robot hands?
apply a filtering pipeline to assess tracking quality; episodes Table II reports LEAP Hand performance under both In the Wild
are discarded if the wrist pose cannot be reliably tracked and In the Wild Extreme conditions. In every case, DexWild
for more than 75% of the duration. Next, we compute the co-training substantially outperforms the robot-only baseline.
action distribution and clip outliers outside the 2nd and 97th These results highlight the effectiveness of DexWild in cross
percentiles. We smooth the trajectories using interpolation and embodiment generalization even when using a completely
Gaussian filtering to ensure fluid motion. Hand motions are different robot hand.
then retargeted using inverse kinematics in PyBullet, following
the method in [41]. The entire pipeline is parallelized using H. Scaling Extended Results
Ray for efficiency. Does DexWild improve as more DexWild data is added?
Table III shows steady gains as we scale from 0% to 100% of
the DexWild dataset. Performance increases steadily with more
D. Behavior Cloning Policy Architecture and Training Hyper-
human demonstrations, with a notable jump between 25% and
Parameters
50% of the dataset. These results demonstrate that DexWild
Our behavior cloning policy takes as input RGB images enables scalable learning, where even comparably smaller data
and relative state history. We obtain tokens for the image scales yields substantial gains, and additional data continues
observation via a ViT and tokens for relative states via linear to enhance generalization
layers. The weights of ViT is initialized from the Soup 1M I. Cotraining Extended Results
model from [11]. We decide to include relative states as we
How does DexWild react to different cotraining ratios?
found it greatly increases the robustness of the policy, and
Table IV groups all three raw metrics: (a) In-Domain, (b) In-
enables smoother motions. In particular, for bimanual tasks,
the-Wild, and (c) In-the-Wild Extreme. All evaluations were
we find that including the interhand pose (pose of left hand
run on xArm + LEAP Hand V2 Advanced.
relative to right hand) greatly increases success rate in tasks
like Florist We implement both Action Chunking Transformer
[59] and Diffusion U-Net [6] as policy classes, which output
a sequence of actions. The network outputs actions which
consists of relative end effector actions and absolute hand joint
angles. Task Policy Class In the Wild In the Wild Extreme
We list the hyper-paramaters that we used for policy training Robot Only 1:2 Robot Only 1:2
using behavior-cloning in this Table V ACT 0.000 0.680 0.115 0.395
Spray
Diffusion 0.050 0.628 0.120 0.520
ACT 0.458 0.583 0.125 0.458
E. Low Level Motion Control Toy Cleanup
Diffusion 0.521 0.875 0.500 0.625
For optimal smoothness of our policies and safety, we employ ACT 0.025 0.508 0.000 0.350
Pour (Cross Task)
Diffusion 0.000 0.958 0.000 0.917
a Riemannian Motion Policy (RMP) [34] implemented in
Isaac Lab [26], where the RMP dynamically generates joint- TABLE I: DexWild Performance on Different Policy Classes
space targets given end effector targets. RMP also has the
added benefit of incorporating real-time collision avoidance, In the Wild In the Wild Extreme
preventing self-collision between the arms and a set table Task Robot Only 1:2 Robot Only 1:2
height. Although our policies does not rely on RMP to prevent
collisions, the peace of mind is appreciated. Spray 0.305 0.805 0.150 0.600
Toy Cleanup 0.500 0.656 0.250 0.542
Pour (Cross Task) 0.050 0.917 0.000 0.817
F. Comparing Policy Classes
TABLE II: LEAP Hand Performance on In-the-Wild and In-
Does DexWild work with different behavior cloning the-Wild Extreme Tasks. Ratio is Robot:Human
policy classes? Table I compares the performance of ACT
and Diffusion—across both the In-the-Wild and In-the-Wild
Extreme settings. Each policy is evaluated in a robot-only Scale 0% 25% 50% 100%
setting and a co-trained (1:2) setting using the DexWild dataset.
Spray 0.060 0.260 0.605 0.565
Notably, Diffusion policies benefit more from DexWild co-
Toy Cleanup 0.514 0.442 0.440 0.792
training, achieving the highest scores in all tasks, including
substantial improvements on the Pour task where the policy Average 0.287 0.351 0.523 0.678
must generalize across tasks. These results suggest that Std 0.321 0.129 0.116 0.160
DexWild co-training enables stronger generalization, especially
when paired with expressive policy architectures like Diffusion. TABLE III: Performance Scaling with DexWild Dataset Size
Task Robot 1:1 1:2 1:5 Human Hyperparameter Value
Spray 0.690 0.630 0.763 0.381 0.030 Training Configuration
Toy Cleanup 0.604 0.792 0.833 0.708 0.042
Optimizer AdamW
Average 0.647 0.711 0.798 0.545 0.036 Base Learning Rate 3e-4
Std 0.061 0.114 0.050 0.232 0.008 Optimizer Momentum β1 , β2 = 0.95, 0.999
Learning Rate Schedule Cosine (diffusers)
(a) Performance Across Cotrain Ratios for Varying Deployment
Warmup Steps 2000
Conditions. Ratio is Robot:Human
Total Steps 70000
Batch Size 256
Task Robot 1:1 1:2 1:5 Human Environment Frequency 30 Hz
Spray 0.050 0.625 0.628 0.393 0.063 Observation Settings
Toy Cleanup 0.521 0.646 0.875 0.625 0.083
1 (Spray, Toy, Pour)
Average 0.285 0.635 0.751 0.509 0.073 Proprioception Horizon
3 (Florist, Clothes)
Std 0.333 0.015 0.175 0.164 0.015 Image Horizon 1 (all tasks)
Observation Resolution 224×224
(b) In-the-Wild Task Performance
9 (Spray, Toy, Pour)
Observation Dim
27 (Florist, Clothes)
Task Robot 1:2 26 (Spray, Toy, Pour)
Action Dimension
Spray 0.120 0.520 52 (Florist, Clothes)
Toy Cleanup 0.500 0.625 Action Chunk Size 48
Bimanual Florist 0.063 0.623 Action Chunking Transformer
Bimanual Clothes Folding 0.198 0.740
# Encoder Layers 4
Average 0.220 0.627 # Decoder Layers 6
Std 0.195 0.090 # MHSA Heads 8
Feed-Forward Dim 3200
(c) In-the-Wild Extreme Task Performance
Hidden Dim (Token Dim) 768
TABLE IV: Performance for different cotrain ratios Dropout 0.1
Feature Norm LayerNorm
Diffusion U-Net Policy
Train Diffusion Steps 100
Eval Diffusion Steps 16
Down Channels [256, 512, 1024]
Kernel Size 3
Groups (GN) 8
Dropout 0.1
Feature Norm None
TABLE V: Full training and architecture settings used across our
experiments.