Compliant Residual DAgger: Improving Real-World
Contact-Rich Manipulation with Human Corrections
Xiaomeng Xu∗ Yifan Hou∗ Chendong Xin Zeyi Liu Shuran Song
Abstract
We address key challenges in Dataset Aggregation (DAgger) for real-world contact-rich manipulation: how to collect
informative human correction data and how to effectively update policies with this new data. We introduce Compliant
Residual DAgger (CR-DAgger), which contains two novel components: 1) a Compliant Intervention Interface that
leverages compliance control, allowing humans to provide gentle, accurate delta action corrections without interrupting
the ongoing robot policy execution; and 2) a Compliant Residual Policy formulation that learns from human corrections
while incorporating force feedback and force control. Our system significantly enhances performance on precise
contact-rich manipulation tasks using minimal correction data, improving base policy success rates by 64% on
arXiv:2506.16685v5 [cs.RO] 25 Dec 2025
four challenging tasks (book flipping, belt assembly, cable routing, and gear insertion) while outperforming both
retraining-from-scratch and finetuning approaches. Through extensive real-world experiments, we provide practical
guidance for implementing effective DAgger in real-world robot learning tasks. Result videos are available at:
https://compliant-residual-dagger.github.io/
Keywords
DAgger, Imitation Learning, Manipulation, Compliance Control
1 Introduction done with Reinforcement Learning Ankile et al. (2024); Yuan
et al. (2024) or Imitation Learning Bharadhwaj et al. (2024),
Learning from human demonstrations has seen many recent both require a large number of samples.
successes in real-world robotic tasks Chi et al. (2024); Hou
In this work, we address these questions by proposing
et al. (2025); Wu et al. (2025); Xu et al. (2025); Xiong
an improved system Compliant Residual DAgger (CR-
et al. (2025). However, to obtain a successful policy, human
DAgger) consisting of two critical components:
demonstrators often have to repeatedly deploy a policy and
observe its failure cases, then collect more data to update the
policy until it succeeds. This process is broadly referred to • Compliant Intervention Interface. We propose an on-
as Dataset Aggregation (DAgger) in Ross et al. (2011); Kelly policy correction system based on kinesthetic teaching
et al. (2019). However, doing DAgger effectively for real- to collect delta action without interrupting the current
world robotic problems still faces the following challenges: robot policy. Leveraging compliance control, the interface
How to collect informative human correction data? Ross lets humans directly apply force to the robot and feel
et al. (2011) shows that DAgger is most effective when the the magnitude of their instantaneous correction. Unlike
correction data is within the original policy’s induced state- take-over corrections, our design allows smooth transition
action distribution. In practice, the common approach is either between correction/no correction mode, while providing
(1) collecting offline demonstrations that cover the policy’s direct control of correction magnitudes.
typical failure scenarios Chi et al. (2023), or (2) human taking
over robot control during policy deployment Spencer et al. • Compliant Residual Policy. Leveraging the force
(2020); Mandlekar et al. (2020). However, in both cases, the feedback from our Compliant Intervention Interface, we
human demonstrator has no access to the original policy’s propose a residual policy formulation that takes in an
behavior and may deviate excessively from it. Human taking extra force modality and predicts both residual motions
over additionally introduces force discontinuity when they do and target forces, which can fully describe the human
not instantly reproduce the exact same robot force. This is correction behavior. The Compliant Residual Policy is
partially due to the lack of effective correction interfaces that force-aware, even when the base policy is position-only.
support precise and instantaneous intervention. We show that our residual policy formulation learns
How to effectively update the policy with new data? effective correction strategies using the data collected from
Prior methods for improving a pretrained policy with our Compliant Intervention Interface.
additional data include (1) retraining the policy from scratch
with the aggregated dataset Kelly et al. (2019), which can Together, our system significantly improves the success rate
be computationally expensive; (2) finetuning the policy with of precise contact-rich robot manipulation tasks using a small
only the additional data Wu et al. (2025); He et al. (2025); amount of additional data. We demonstrate the efficacy of our
Chen et al. (2025), which is sensitive to the quality of the method on four challenging tasks involving long horizons
new data Yuan et al. (2024), and (3) training a residual policy and sequences of contacts: book flipping, belt assembly,
separately on top of the pretrained policy, which is typically cable routing, and gear insertion. We improve over the base
2 CR-DAgger
Figure 1. CR-DAgger. To improve a robot manipulation policy, we propose a compliant intervention interface (a) for collecting human correction data, and use this data to update a compliant residual policy (b), and thoroughly study their effects by deploying the updated policy on two contact-rich manipulation tasks in the real world (c).
policy success rate by 64%, while also outperforming retrain- the demonstrator to directly label actions generated by the from-scratch and finetuning under the same data budgets. In policy. In robotics, a practical variation is to let the human summary, our contributions are: take over the robot control and provide correct action directly Kelly et al. (2019). Such human correction motion can be • A Compliant Intervention Interface, a system that allows recorded with spacemouse Chen et al. (2025); Liu et al. humans to provide accurate, gentle, and smooth corrections (2022), joystick Spencer et al. (2020), smartphone Mandlekar in both position and force to a running robot policy without et al. (2020), or arm-based teleoperation system Hoque et al. interrupting it. (2021a,b); Wu et al. (2025). We instead proposes a novel • A Compliant Residual Policy, a policy formulation that kinesthetic teaching system with compliance controller that seamlessly integrates additional force modality inputs and allows the demonstrator to apply delta corrections while the predicts residual motions and forces. robot policy is still running, and additionally records force • A practical guide for efficient DAgger based on extensive feedback. Our results show that both the delta correction data real-world studies for critical but often overlooked design and the force data are crucial to the success of the learned choices, such as batch size and sampling strategy. Our policy. hardware design, training data, and policy code can be found here. Improving Pretrained Robot Policies with New Data. The most direct approach to improving a pretrained policy This work is an extended version of the conference with new correction data is to retrain the policy on the paper Xu* et al. (2025). We expand the content of this paper aggregated dataset, combining prior demonstrations with in the following ways: new feedback Mandlekar et al. (2020); Spencer et al. (2020). Alternatively, Reinforcement Learning (RL) offers a • Enrich the experimental results with two additional contact- framework to incorporate both offline and online data, either rich tasks as detailed in Sec. 4: a peg-in-hole style gear by warm-starting replay buffers Luo et al. (2024); Ball et al. insertion task, which requires sub millimeter accuracy and (2023) or by using offline data to guide online fine-tuning Yin tests our system for small-magnitude correction motions; et al. (2025); Xu et al. (2022). When policies are trained and a cable routing task, which contains large variations in on large-scale human demonstration datasets Black et al. item physical properties (cable stiffness). (2024); Team et al. (2024); Kim et al. (2024); O’Neill et al. (2024); Khazatsky et al. (2024); Xu et al. (2023a, 2024a), • Identify and explain the intention-misinterpretation retraining becomes impractical, especially when the original phenomenon in correction data collection in Sec. 3.1, data is inaccessible. In such cases, fine-tuning with only caused by robot tracking errors. We provide a solution the new data is a common solution, using either imitation to the problem by reducing tracking error both spatially learning Liu et al. (2022); He et al. (2025); Wu et al. (2025) and temporally and include additional analysis on the effect or RL Mark et al. (2024); Chen et al. (2025). Another line of of tracking error in Sec. 4.5. work introduces an additional residual model on top of the • Analyze the quality improvement of human correction original policy. These residual policies can be trained with data from our proposed system comparing with traditional RL in simulation Yuan et al. (2024); Ankile et al. (2024); manipulation data collection method in Sec. 4.4. Haldar et al. (2023a), but suffers from sim-to-real challenges. Training residual policy in the real world usually requires a large number of samples Bharadhwaj et al. (2024); Johannink 2 Related Work et al. (2019), intermediate scene representation Guzey et al. Human-in-the-Loop Corrections for Robot Policy Learn- (2024b), or consistent visual observations between training ing. The original DAgger work Ross et al. (2011) requires and testing Guzey et al. (2024a); Haldar et al. (2023b), making 3
the approach hard to adopt in practice. In this work, we humans can monitor policy execution and intervene on the introduce a practical data collection system and an efficient spot when failures occur or are anticipated. This approach residual policy learning algorithm for long-horizon, contact- allows humans to provide corrections more targeted to the rich manipulation tasks. Our approach requires only a small base policy’s failure cases. However, challenges still exist for amount of real-world correction data and supports integration an intervention system: of additional sensory modalities not present in the original model, leading to improved policy performance. • Non-smooth transitions. Intervention in robotics is Compliance and Compliance Control. Compliance is typically implemented by take-over correction: letting a motor property that describes how motion responds to human take complete control and overwrite robot policy. force. For example, traditional industrial robots typically have As the underlying control abruptly switches between robot position control with very low compliance to achieve precise policy and human intention, disturbances are introduced tracking under heavy load. On the contrary, high compliance due to policy inference and human response latency, lets a robot retreat when it experiences external forces. A high especially when the robot is withholding external forces. compliance or a variable compliance profile Mason (2007) The recorded data thus may include undesired actions that may be preferred for tasks involving interactions with the do not reflect the human’s intention. environment, since they can increase execution robustness or • Distribution shift. The human-intervened state-action avoid huge contact forces during the interactions Hou and distribution may deviate significantly from the original Mason (2019); Hou et al. (2020). distribution. Additionally, the non-smooth transition above Compliance control refers to feedback control techniques could bring in disturbances and add to the distribution shift. that give a robot high compliance or spatial-temporally varying compliance profiles. Compliance control can be • Indirect correction brings errors. Correction is implemented by Impedance Control Hogan (1984) if the robot commonly implemented via teleoperation interfaces such is back-drivable, or by Admittance Control with external as spacemouse or joysticks He et al. (2025); Chen et al. force sensors if the robot cannot be back-driven, such as (2025). With spatial mismatch and teleoperation latency, it most industrial robots. A more through review of compliance is hard for the demonstrator to instantly provide accurate controller implementations can be found in Lynch and Park corrections upon intervention starts without going through (2017) and Villani and De Schutter (2016). a short adjustment period.
• Missing information. The recorded correction data need
3 CR-DAgger Method to fully describe the human’s intended action. Simply Our goal is to improve a pretrained robot policy with a small recording the robot’s position is not sufficient, since it may amount of human correction data. To achieve this, we propose be under the influence of human correction force and will a Compliant Intervention Interface (§ 3.1) that enables precise cause different result when testing without human. and intuitive on-policy human correction data collection, and a Compliant Residual Policy (§ 3.2) that efficiently learns the We propose a Compliant Intervention Interface with the correction behaviors to be used on top of the pretrained policy. following designs to solve those challenges: Throughout the paper, we use the term base policy to refer to the pretrained policy without online improvements. • Delta correction instead of take-over correction. Unlike take-over correction, where the demonstrator has no idea 3.1 Compliant Intervention Interface of the policy’s original intention once taking over, we propose a novel on-policy delta correction method: we let Correction data is collected by human demonstrators to rectify the robot policy executes continuously while the human policy failures. Unlike initial demonstrations that establish applies forces to the robot with a handle mounted on the end baseline behaviors, correction data specifically targets failure effector, resulting in delta actions on top of the policy action. modes observed during policy deployment. Correction data The human demonstrator can always sense the policy’s is most effective when it corrects failures in policy-induced intention through haptic feedback, and easily control the state distributions Ross et al. (2011). The interface through magnitude of intervention by the amount of force applied to which these corrections are collected significantly impacts the handle. As a result, delta correction ensures smooth the quality of correction data, which should be intuitive intervention data and limits the human from providing for demonstrators, capture critical corrective information at very large corrections. The approach is also intuitive as precise moments of failures, and facilitates collecting data the human can directly move the robot towards desired close to the base policy state-action distribution. correction directions. There are two types of correction collection methods: Off-policy correction is when humans observe failures of • Correction interface with compliance control. In order to the base policy during deployment, and then recollect extra apply delta correction over a running policy, we provide a offline demonstrations to address failure cases. This approach compliant interface that allows humans to safely intervene is most commonly used for improving Behavior Cloning and apply force to the robot to affect its behaviors at any policy performance due to its simplicity - it requires no time, as shown in Fig. 2. We design a kinesthetic correction additional infrastructure beyond the original data collection hardware setup with a detachable handle for human to setup. However, the resulting demonstrations may fail to cover hold when correcting, and allows easy tool-swapping for all the failure cases or deviate from the original state-action different tasks. We run a compliance controller (specifically distribution. We focus on on-policy correction instead, where admittance control) in the background to respond to both 4 CR-DAgger
Figure 2. Compliant Intervention Interface characterized by a kinesthetic correction hardware setup where humans hold on the handle and apply forces to correct robot execution, providing on-policy delta corrections.
learn the wrong correction motion. This is illustrated in Fig. 3.
Feedback With our Compliant Intervention Interface, human correction
Recorded
Correction motion is computed as the difference between robot position
feedback and the base policy output. A subtle assumption
Correction
Command
(A) starts here is that the robot position before intervention accurately (B) Recorded reflects the base policy behavior, i.e. the tracking error should Command Correction be negligible. When the tracking error is larger than the delta corrective motion from human intervention, the computed correction motion could be in the opposite direction of the Feedback human’s true intended correction direction. To avoid intention misinterpretation even when human correction is small, we Figure 3. Illustration of the intention misinterpretation caused by implement two features on top of Xu* et al. (2025) to reduce tracking error. In both cases, the robot feedback (and thus the the tracking error: 1) we add a velocity tracking term in the human correction motion) are exactly the same. However, the recorded correction motion direction is very different. (A) compliance control law in addition to the position tracking recorded correction is correct when the tracking error is small. (B) term used in 1: recorded correction motion has wrong direction when there is large tracking error caused by both control and latency. M q̈ = K(qre f − q) + D(q̇re f − q̇) + F, (2)
where the velocity reference q̇re f is computed via finite
contact forces and human correction forces, allowing the difference from the position reference qre f . This reduces the
human to influence but not completely override the policy tracking error spatially. 2) We added a look-ahead time to
execution. The admittance controller forms a virtual spring- robot feedback when computing the correction motion qdelta :
mass-damper system around the base policy output qre f :
qdelta [t] = q[t] − qre f [t − ∆t], (3)
M q̈ = K(qre f − q) − Dq̇ + F, (1)
where ∆t is empirically set by the estimation of hardware
where M, K and D denotes the virtual inertia, stiffness latency. The look-ahead time reduces the tracking error
and damping of the compliant system, F represents the temporally. Together, the largest tracking error reduces from
combined force feedback from both external contacts and 30 mm to below 5 mm during fastest robot motion in our
human correction. The admittance controller uses a constant experiments.
stiffness ∼1000 N/m to allow easy human intervention and
ensure accurate tracking. 3.2 Compliant Residual Policy
• Correction recording with buttons and force sensor. Given the correction data, there are multiple ways to update Our interface additionally includes an ATI 6-D force the policy. Common practices include retraining the base sensor to directly measure contact forces, and a single-key policy from scratch with both initial data and correction keyboard placed on the handle to record the exact timings of data, and finetuning the base policy with only the correction correction starts/ends. Both the policy’s original commands data. However, retraining is costly as it requires updating and the human’s delta corrections are recorded, along with the entire base policy network from scratch with all the force sensor readings during the interaction. available data. It also requires access to the base policy’s initial training data, which might not be accessible for many Additionally, comparing with the conference version of open source pretrained models. The amount of correction this paper Xu* et al. (2025), we identified and fixed an issue data is significantly smaller than the initial training data, of intention misinterpretation: When the human correction thus simply mixing them together makes the policy hard to motion is smaller than the robot tracking error, the human gain effective corrective behaviors. While finetuning allows intention can be misinterpreted and the residual policy would updating partial policy network parameters with new data 5
Retrain or Finetune
Compliant Residual Policy
(Prior Work) (Ours)
img, proprio poses (0.1s)
Retrain Finetune
delta poses (0.02s)
Admittance
Robot
Controller
forces target forces (0.02s)
Figure 4. Policy Update Methods. Left: Common policy update methods - retraining and finetuning. Right: Ours. The base policy runs at 1 Hz. It takes in images It and proprioceptions Pt and predicts 32 frames of end-effector poses Atp = {Atp0 , Atp1 , …, AtpTa } spaced 0.1 Seconds apart. The Compliant Residual Policy runs at 50 Hz. It takes in additional force inputs Ft and predicts 5 frames of delta f p p poses ∆Atp = {∆Atp0 , ∆Atp0 +1 , …, ∆Atp0 +Tr } and target forces At spaced 0.02 Seconds apart. The combined poses of At and ∆At , and target f forces At are taken by an admittance controller to command the robot.
only, its training stability can be easily affected by the by collecting a small amount of correction data with force distribution mismatch between the correction data and initial modality. training data. Moreover, both retraining and finetuning can • High-frequency inference. The light-weight residual policy only update the policy with its fixed network architecture runs at a higher frequency than the base policy, while being unable to incorporate new inputs and outputs. incorporating high-frequency force feedback and enabling We propose a compliant residual policy trained only on the reactive corrective behaviors. This reactivity is particular correction data to refine base policy’s position actions and important for error correction during contact events. predict additional force actions. Compliant residual policy formulation. Our policy Training strategy. In prior work, a residual policy is directly learns corrective behavior from the human delta trained either in simulation with RL Ankile et al. (2024); correction data, as shown in Fig. 4. It takes as input the Yuan et al. (2024) to give it sufficient coverage of the state same visual and proprioceptive feedback as the base policy distribution, or in the real world with pre-collected behavior but with a shorter horizon. It also takes in an extra force cloning data Luo et al. (2024). In this work, we train the modality, which is available using our Compliant Intervention Compliant Residual Policy completely on the small amount of Interface. The policy outputs five frames of actions at a time, new real-world correction data with the following strategies: corresponding to 0.1 s of execution time when running at • Ensure sufficient coverage of in-distribution data. Human 50 Hz. The action space is 15-dimensional: the first nine correction tends to be frequent around a few key moments dimensions represent the SE3 delta pose from the base policy of the task. A residual trained on correction data alone action to the robot pose command Chi et al. (2023), while the can extrapolate badly around states where no correction is later six dimensions represent the expected wrench (force and provided. To help the residual policy understand when not torque) the robot should feel from external contacts. Both the to provide corrections, we: (1) include the no correction data robot pose command and the expected wrench are sent to a for training but label it as zero residual actions; (2) collect standard admittance controller for execution with compliance. a few trajectories where the demonstrator always holds the The residual policy directly uses the base policy’s frozen handle and marks the whole trajectory as correction even image encoder Radford et al. (2021); Amir et al. (2021); when the correction is small or zero. Details are in § A.3. Xu et al. (2023b) to extract an image embedding, a temporal convolution network Van Den Oord et al. (2016) to encode the • Prioritize correction data over no-correction (zero residual force vectors, followed by fully-connected layers to decode action) data. Similar to Liu et al. (2022), we alter the actions. data sample frequency during training based on whether Advantages. This formulation provides the following they have human correction or not. Specifically, since the advantages: moment of correction start indicates where the current policy performs badly followed by immediate action • Sample-efficient learning. The residual policy’s network is to fix it, we sample data more frequently for a short light-weight (∼2MB trainable weights) and only requires period immediately after correction starts. Our real-world a small amount of correction data to train (50∼100 ablations (§ 4.5) demonstrate that our training strategies demonstrations). improve the quality of the residual policy and the overall success rate. • Incorporating new sensor modality. Compared to retrain- ing and finetuning methods that are limited to the base policy’s network architecture, residual policy can incorpo- 4 Evaluation rate new sensor modality. This allows taking any position- For each task, we train a diffusion policy Chi et al. (2023) based pretrained policy and turning it force-aware simply with 150∼400 demonstrations as the base policy. We first 6 CR-DAgger
deploy the base policy and observe its performance and failure generalization capability. We ran 20 rollouts across five cables modes. Next, from the same base policy, we collect 50∼100 and four different fixed end locations. correction episodes with each data collection method. Then, Gear Insertion The task involves inserting a plastic gear we update the policy using each network updating method and onto a metal axis while mating it to a nearby gear, which is training procedure. Finally, we deploy the updated policies also a part of the NIST board assembly challenge Kimble et al. and evaluate their performance under the same test cases. (2020). The base policy is trained with 150 demonstrations. Details of tasks and comparisons are described below. The task is challenging due to the tiny tolerance between the gear and the axis. It is also different than the previous tasks 4.1 Contact-Rich Manipulation Tasks as the human correction motions often have small magnitudes. Book Flipping: As shown in Fig. 5 (a), this task is to flip up We introduce additional difficulties by elevating the base books on a shelf. Starting with one or more books lying flat board by 3 cm. We ran 20 rollouts across five different base on the shelf, the robot first insert fingers below the book, then locations. rotate the book up and push them firmly against the shelf to let them stand on their own. The base policy is trained with 4.2 Base Policy and its Failure Modes 150 demonstrations. We trained a diffusion policy Chi et al. (2023) that takes This task is challenging as it involves rich use of physical in past images from a wrist-mounted camera and robot contacts and forceful strategies Hou et al. (2020). A position- proprioception observations, and predicts a future position- only strategy always fails immediately by triggering large based action trajectory. To isolate the contribution of force forces, so we execute all policies through the same admittance inputs versus human corrections, we trained diffusion policies controller. The task success requires high precision in both with and without force inputs as baselines for the belt motion and force to accurately align the fingers with the gap assembly task. upon insertion, and to provide enough force to rotate heavy The book flipping base policy achieves a 40% success rate books and make the books stand firmly. with the following common failure cases (Fig. 5 (c)): (1) Each evaluation includes 20 rollouts on 4 test cases (5 Missed insertion. The fingers initially go too high above the rollouts each), as shown in Fig. 5 (b): 1) flipping a single book or aims for the gap between the two books, failing to book (three seen and two unseen books), initially far from the properly insert beneath the books. (2) Incomplete flipping. At shelf edge; 2) flipping a single book close to the shelf edge; the last stage, the policy retracts the blade before the book 3) flipping two books together (combinations of three seen can stand stably, causing it to fall back. and three unseen books), initially far from the shelf edge; 4) The belt assembly base policy achieves a 15.6% success flipping two books close to the shelf edge. We use the same rate. Adding force input increases the base policy success initial configurations for all evaluations. rate to 43.8%. Common failure cases include (Fig. 6 (c)): (1) Belt Assembly: As shown in Fig. 6 (a), this task is to Missed slotting: the fingertip goes too high or too low, causing assemble a thin belt onto two pulleys, which is part of the the belt to miss the slot on the big pulley. (2) Belt slippage: NIST board assembly challenge Kimble et al. (2020). Starting the fingers pull the belt in the wrong direction, causing the with the belt grasped by the gripper, the robot needs to first belt to tilt and slip off the pulley. thread the belt over the small pulley, next move down while stretching the belt to thread its other side on the big pulley, 4.3 Baselines then rotate 180◦ around the big pulley to tighten the belt, and finally pull up to release the belt from the gripper. The base We compare CR-DAgger with baselines across two policy is trained with 405 demonstrations. dimensions: correction method and policy update method. The task is challenging as it requires both position and We present the quantitative results in Fig. 9, and explain key force accuracy throughout the process. Specifically, the belt findings in § 4.4. is thin and soft so the initial alignments onto the pulleys are Correction data collection methods. We compare visually ambiguous. Also, since the belt is not stretchable, our Compliant Intervention Interface with the two most there is more resistance force and less position tolerance as commonly used correction data collection strategies: the belt approaches the second pulley, requiring a policy with • Observe-then-Collect includes two steps: first, the policy good force-position coordination and adaptation. We ran 32 is deployed and human demonstrators observe the initial rollouts across four different initial board positions and four settings that could cause failures; then, demonstrators grasp locations (Fig. 6 (b)). provide completely new demonstrations starting from Cable Routing The task includes routing a USB cable similar initial settings. As explained in § 3.1, this type around three clips, as shown in Fig. 7 (a). The USB cable has of offline correction potentially misses critical timing one fixed end and a free end. The cable starts being grasped, information, and the resulting demonstrations may deviate and can slide easily through a groove on the finger as the hand from the policy’s original behavior distribution. moves. The base policy is trained with 200 demonstrations. We make this task challenging by using cables with • Take-over-Correction (HG-DAgger) Kelly et al. (2019) different appearances and stiffness than those seen by the is another common correction strategy where human base policy. A change in cable position or physical property demonstrators monitor policy execution and take complete can easily cause the cable to be too high and miss a clip control when failures are anticipated. We implement Take- (Fig. 8 (b)), or too low and get stuck by a clip. We additionally over-Correction on our Compliant Intervention Interface by introduce difficulty by using clips that are visually different cleaning up command buffer to the compliance controller from the one used in base policy training, testing our system’s and switching stiffness to zero upon correction starts, so the 7
Figure 5. Book Flipping Task. (a) Policy rollout of [Compliant Residual] policy trained with [On-Policy Delta] data, demonstrating accurate insertion motions and forceful pushing strategy. (b) Different test scenarios. (c) Typical failure cases of the base policy: inserting too high above the book and missing the gap; retracting the fingers before the books can steadily stand.
(a) Base + Compliant Residual Policy Rollout
Start Thread on 1st pulley Thread on 2nd pulley Pull over 2nd pulley Release
(b) Test Configs (c) Base Policy Failure Cases
Missed the slot Slipped off the pulley
belt
height
Figure 6. Belt Assembly Task. (a) Policy rollout of [Compliant Residual] policy trained with [On-Policy Delta] data, demonstrating accurate force-position coordination and adaptation. (b) Different test scenarios. (c) Typical failure cases of the base policy: missing the slot by going too high above the pulley; tilting the belt and causing it to slip off the pulley.
(a) Base + Compliant Residual Policy Rollout (b) Base Policy Failure Case
Figure 7. Cable Routing Task. (a) Policy rollout of [Compliant Residual] policy trained with [On-Policy Delta] data. (b) Typical failure case of the base policy: missing a clip.
robot policy does not affect the robot during correction. Policy update methods. We compare with two common However, as explained in § 3.1, take-over correction policy update methods: introduces an abrupt transition around control authority • Retrain Policy: Retrain the base policy using both the switching, which may cause distributional discontinuities original training data and the correction data from scratch. in the training data. As explained in § 3.2, this approach is reliable but may require access to the orignal data and large amount of new • On-Policy Delta (Ours): the details are described in § 3.1. data to work well. 8 CR-DAgger
(a) Base + Compliant Residual Policy Rollout (b) Base Policy Failure
Figure 8. Gear Insertion Task. (a) Policy rollout of [Compliant Residual] policy trained with [On-Policy Delta] data. (b) Typical failure case of the base policy: misaligning the gear with the axis and failing to mate with the nearby gear.
• Finetune Policy: Finetune the base policy using only the the pulley to find the slot when the finger touches the top of correction data (freezing visual encoders). As explained in the pulley. § 3.2, this approach can be sensitive to data quality and Finding 3: Smooth On-Policy Delta data enables stable distribution shifts. residual policy. [Compliant Residual] policy has 45% higher success rate when trained on [On-Policy Delta] data instead • Fintune Policy with KL Regularization: A recent of [Take-over-Correction] data on the book flipping task. method Fan et al. (2023) that stabilizes finetuning Residual policy trained with [Take-over-Correction] data training by encouraging the predicted action to be close to sometimes exhibits large noisy motions that trigger task the training data distribution. failures, such as retracting the fingers too early in the book flipping task. On the contrary, the residual policy trained • Residual Policy: an ablation of our method where force is with [On-policy Delta] data has much smoother action removed from both input and outputs. trajectories and better reflects human’s correction intentions, providing suitable magnitudes of corrections. To see the • Compliant Residual Policy (Ours): Residual policy update improvements more clearly, we plot robot velocity data with additional force input and outputs, see details in § 3.2. around correction starts and ends from 100 episodes of data collections in Fig. 10. [Take-over-Correction] data contains 4.4 Key Findings from Comparisons larger robot velocity magnitude right after control authority Finding 1: Compliant Residual Policy is able to improve switches, especially when robot takes over control from base policy by a large margin. As shown in Fig. 9, human. Moreover, the demonstrator can directly perceive the [Compliant Residual] policy trained with [On-Policy Delta] base policy’s intention and the extent of correction being data improves the base policy success rate by 60% and applied through the resistance force when employing the 50% on the two tasks respectively. It effectively learns [On-Policy Delta] kinesthetic teaching style correction, thus useful corrective strategies from the limited demonstrations. preventing correction demonstrations from deviating too For example, in the book flipping task, the policy learns to much from the base policy’s state-action distribution. Fig. 11 pitch the fingers down more before finger insertion to increase shows that [On-Policy Delta] introduces less distribution shift the insertion success; in the belt assembly task, the policy compared to [Take-over-Correction]. learns to correct the height of the belt when misaligned to Finding 4: Retraining base policy is stable but learns the pulley slot. Results are best viewed in our supplementary correction behavior slowly. Retraining from scratch with video. the initial and correction data together leads to policies with Finding 2: Residual policy allows additional useful stable motions. However, its behavior is less affected by the modality to be added during correction. [Compliant small amount of correction data compared to the dominant Residual] policy performs significantly better than other portion of initial data, leading to insignificant improvements methods without force (45% higher success rate than the best over the base policy (1.67% success rate drop on the book position-only baseline on the book task and 53% higher on the task averaged across all data collection methods, 18.8% belt task) as it can both take in force feedback that indicates success rate improve on the belt task, both are much less critical task information and predict adequate contact forces improvements than [Compliant Residual]). to apply. For example, the last stage of the book flipping task Finding 5: Finetuning base policy is unstable. Policy requires the robot to firmly push the book against the shelf finetuning with either correction data has the worst wall to let it stand on its own. [Compliant Residual] policy performance across all policy update methods and even predicts large pushing forces at this stage to make the books underperforms the base policy (30% success rate drop on the stand stably with a 100% success rate, while [Residual]’s book task averaged across all data collection methods, 15.6% success rate drops from 70% to 35% (§ A.2). The second drop on the belt task). The finetuned policy predicts unstable stage of the belt assembly task (threading the belt on the and noisy motions, quickly leading to out-of-distribution large pulley) requires delicate belt height adjustments under states, such as inserting too high in the book flipping task ambiguous visual information due to occlusions and the lack and drifting away from the board in the belt assembly task. of depth. [Compliant Residual] policy learns to move along This is likely due to the distribution mismatch between the 9
Book Flipping Belt Assembly
Observe-then-Collect 100 100 96.8
100 Take-over-Correction
On-policy Delta
Success Rate (%)
Success Rate (%)
80 Base Policies 80
60 55 55 60
50
43.8
40 40 40 34.4
40 35
28.1
20
20 20 20 15.6
5 5 3.1
0 0
0
e
in
e
l
e
e
in
e
L
l
)
)
ua
ua
urs
urs
/K
orc
n
orc
rc
n
tra
tra
etu
etu
sid
sid
Fo
ew
(O
(O
oF
oF
Re
Re
Fin
Fin
Re
Re
w/
CR
CR
n
w/
w/
etu
se
se
se
Ba
Fin
Ba
Ba
Gear Insertion Cable Routing
100
90 100
100 95
80 85
Success Rate (%)
Success Rate (%)
80
70
60
50 60
40 40
40 35 40
25 25
20 20
5 5
0 0
e
e
in
ne
al
e
e
KL
al
rs)
ne
KL
rs)
orc
rc
orc
rc
u
u
tra
etu
etu
u
u
sid
sid
Fo
w/
Fo
w/
(O
(O
oF
oF
Re
Fin
Fin
Re
Re
ne
ne
w/
w/
CR
CR
w/
w/
etu
etu
se
se
se
se
Ba
Ba
Fin
Fin
Ba
Ba
Figure 9. Results. We compare CR-DAgger across two dimensions: correction method and policy update method. The result shows that our [Compliant Residual (CR)] policy trained with [On-Policy Delta] data is able to improve upon base policies on both tasks and outperforms other variations. X Position (m)
Large Deviation From
On-policy Delta Base Motion
Take-over-Correction
Velocity (m/s)
Correction
starts
Y Position (m)
Base Motion
On-policy Delta
Take-over-Correction
Z Position (m)
Velocity (m/s)
Correction
Ends
Figure 11. [On-Policy Delta] introduces less distribution
shift. Here we compare the distribution of fingertip trajectories
among 1) base policy training data, 2) correction episodes from
[On-Policy Delta], and 3) correction episodes from
Figure 10. [On-Policy Delta] enables smoother trajectories. [Take-Over-Correction]. For intuitive visualization, all trajectories Here compares velocity magnitude within 1.5 s of the corrections are stretched to the same duration and normalized to (0,1). The starts/ends. [On-Policy Delta] velocity magnitudes are smaller range of the vertical axis is 0.55m for all three axes. [On-Policy and more consistent, [Take-Over Correction] has notably larger Delta] data’s distribution is better aligned with base policy training magnitude and variations, demonstrating that [On-Policy Delta] data’s distribution than [Take-Over-Correction] data. encourages smoother trajectories.
base policy training data and correction data, causing training Training frequency and batch size. One important instabilities. Adding KL-regularization effectively reduced parameter in DAgger is the batch size between policy updates. the noisy behavior, however, the overall success rate is still With a smaller batch size, the policy is updated more lower than other baselines. frequently, then new correction data can better reflect the most recent policy behavior. However, DAgger with small batch sizes is known to suffer from catastrophic forgetting 4.5 Ablations Kirkpatrick et al. (2017); Goodfellow et al. (2013) since We study the effect of important design decisions in our it finetunes neural networks on data with non-stationary method with ablation studies. distribution. Common solutions include retraining the residual 10 CR-DAgger
Training Frequency Training Sample
direction of correction. Accurate delta action predictions right
110
100
100
100
after correction starts are important for reactive corrective
100
Success Rate (%)
Success Rate (%)
80
behaviors. We investigate three strategies for sampling from
90
60
online correction data during training: 1. Uniform sample,
40 75 80
where the whole episode is sampled uniformly. 2. Denser
70
20 sample around the start of a human intervention, and 3. denser
70
0 0 sample only after the human intervention starts. For 2 and 3,
60
batch size=10 batch size=50 Uniform Dense Around Dense After
we uniformly increase the sample frequency four times for a
Figure 12. Effect of Training Frequency and Sample. Batch fixed period before and/or after intervention starts. size=50 leads to more stable training and dense sampling after Finding: Sampling denser right after intervention starts correction starts achieves better performance on the book leads to more reactive and accurate corrections. As shown in flipping task. Fig. 12 (right), the best performance comes from densely sampling after the beginning of interventions. Sampling 100
96.8 denser around the start of a human intervention also adds
98
more samples right before the intervention starts, which is
Success Rate (%)
96
where humans observe signs of failures. These are mostly
94
negative data, and using them for training decreases the policy
92 90.6
success rate.
90
87.5 Intention Misinterpretation and tracking error. As
88
explained in Sec. 3, the correction data can misinterpret
86
the human correction behavior when tracking error is large.
1 batch 2 batch 3 batch During development of our method, we observed this
phenomenon on the gear insertion task, where the human
Figure 13. Multi-Batch Update Result. The compliant residual policy trained on two batches of correction data achieves the best correction motion is small in magnitude and mostly aligns result on the belt assembly task, indicating that more batches of with the base policy motion, both of which moves towards correction data do not guarantee better performance. the pole. As a result, the residual policy learns to pull the gear away from the pole instead of moving it towards the pole, causing a persistent failure. With our fixes that reduces policy at the end of DAgger using all available correction data tracking error (equation 2 and 3), the correction data can collected from all the intermediate residual policies Wu et al. correctly reflect the human intention and train a residual policy (2025). Another way is to rely on the base policy training data that improves success rate. as a normalizer He et al. (2025). In this work, we empirically The problem does not affect the book flipping task and found that larger batch sizes can effectively stabilize residual the belt assembly task even before our fixes are applied. training. With batch size = 50, the book flipping task reach For the book flipping task, the correction motion magnitude 100% with one batch, while the belt assembly task performs is typically much larger than the robot tracking error. The better with two batches. We compare our method with a belt assembly task uses minute correction motion around smaller batch size on the book flipping task, where we warm the second pulley, however, the human correction motion is up the residual with 20 episodes of initial correction data, mostly vertical, in which direction the robot has little motion then update every ten more episodes for three times. We also and tracking error. experiment with multiple batches on the belt assembly task to validate if the improvement is monotonic. Finding: Large-batch DAgger is more suitable for training 5 Limitation and Future Work Compliant Residual Policy. The small-batch training becomes Base/residual trade-off. The base policy should have a unstable and the demonstrator needs to provide large reasonable success rate for the residual policy to learn magnitudes of corrections as the number of iterations effectively. From our experiments, we recommend starting increases. During evaluation, the final policy always fails to collect correction data for the residual policy when the by inserting too high, while our single-batch policy achieves a base policy has at least 10% ∼ 20% success rate. A practical 100% success rate with the same amount of data and training question is whether to spend resources on improving the base epochs. policy or the residual policy, given a challenging new task. Finding: More batches does not always bring better A future direction is to derive theoretical guidelines for such performance. Fig. 13 shows the change of success rate on trade-off between the base and residual improvements. the belt assembly task as the number of batches increases Incorporate more data. Our work learns corrections from one to three. The policy trained with three batches of efficiently using 50∼100 episodes. The cost of this efficiency data performs slightly worse than the one trained with two is the lack of extreme robustness under large variations, which batches, indicating that more correction data does not always does require more information about the problem dynamics. bring better performance. This could be caused by distribution There are two interesting questions to ask for future work: 1) shift as we observed different failure modes during each new how can more data/information be absorbed by the residual batch of data collection, and the 50 additional episodes of policy to make it adapt to more diverse scenarios? Can we data was not sufficient to cover the new distribution. use a world model/dynamics model to guide the correction Sampling strategy during training. The start of a human behavior under novel perturbations? An example work in this intervention contains critical information of the timing and direction is Sun and Song (2025). 2) Is there better policy 11
architecture than a shallow MLP? Throughout this work, we Maria Bauza, Antonia Bronars, Yifan Hou, Ian Taylor, Nikhil use a MLP as the action head of our Compliant Residual Chavan-Dafle, and Alberto Rodriguez. Simple, a visuotactile Policy and directly regress the actions. We briefly explored method learned in simulation to precisely pick, localize, regrasp, popular alternatives such as a probabilistic MLP Duan et al. and place objects. Science Robotics, 9(91):eadi8808, 2024. (2016); Haarnoja et al. (2018) or Flow Matching Lipman Homanga Bharadhwaj, Roozbeh Mottaghi, Abhinav Gupta, and et al. (2022); Black et al. (2024) in Sec. A.4, but found no Shubham Tulsiani. Track2act: Predicting point tracks from significant difference. Although a simple MLP already works internet videos enables generalizable robot manipulation. In well in our tasks, we believe it may experience difficulty European Conference on Computer Vision, pages 306–324. for complex tasks that involve more distinctive action multi- Springer, 2024. modalities, for which more expressive policy formulations Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael might be useful. Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol More flexible/intuitive data collection interface. Our data Hausman, Brian Ichter, et al. π0 : A vision-language-action collection system is based on kinesthetic teaching. Although flow model for general robot control. arXiv preprint it provides richer data with higher quality than teleoperation arXiv:2410.24164, 2024. as explained in the paper, it may require more labor during Yuhui Chen, Shuai Tian, Shugao Liu, Yingting Zhou, Haoran Li, and training data collection since the demonstrator needs to Dongbin Zhao. Conrft: A reinforced fine-tuning method for vla grasp the handle on the robot. Two interesting future work models via consistency policy. arXiv preprint arXiv:2502.05450, directions include 1) develop teleoperation systems with the 2025. same intuitive and smooth correction data collection, and 2) Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun develop intuitive interface for manipulation with more DoFs, Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. such as bi-manual or dexterous hands manipulation. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 6 Conclusion 02783649241273668, 2023. Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin In this work, we evaluate practical design choices for DAgger Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. in real-world robot learning, and provide a system, CR- Universal manipulation interface: In-the-wild robot teaching DAgger, to effectively collect human correction data with without in-the-wild robots. arXiv preprint arXiv:2402.10329, a Compliant Intervention Interface and improve the base 2024. policy with a Compliant Residual Policy. We demonstrate Hojung Choi, Jun En Low, Tae Myung Huh, Gabriela A Uribe, the effectiveness of our designs by comparing them with a Seongheon Hong, Kenneth AW Hoffman, Julia Di, Tony G variety of alternatives on four contact-rich manipulation tasks. Chen, Andrew A Stanley, and Mark R Cutkosky. Coinft: A coin-sized, capacitive 6-axis force torque sensor for robotic Acknowledgements applications. arXiv preprint arXiv:2503.19225, 2025. We would like to thank Eric Cousineau, Huy Ha, and Benjamin Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Burchfiel for thoughtful discussions on the proposed method, thank Abbeel. Benchmarking deep reinforcement learning for Mandi Zhao, Maximillian Du, Calvin Luo, Mengda Xu, and all continuous control. In International conference on machine REALab members for their suggestions on the experiment setup and learning, pages 1329–1338. PMLR, 2016. the manuscript. Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Funding Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in This work was supported in part by the NSF Award #2143601, Neural Information Processing Systems, 36:79858–79885, 2023. #2037101, and #2132519, Sloan Fellowship, Toyota Research Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Institute, and Samsung. We would like to thank Google and TRI Yoshua Bengio. An empirical investigation of catastrophic for the UR5 robot hardware. The views and conclusions contained forgetting in gradient-based neural networks. arXiv preprint herein are those of the authors and should not be interpreted as arXiv:1312.6211, 2013. necessarily representing the official policies, either expressed or implied, of the sponsors. Irmak Guzey, Yinlong Dai, Ben Evans, Soumith Chintala, and Lerrel Pinto. See to touch: Learning tactile dexterity through visual incentives. In 2024 IEEE International Conference on Robotics References and Automation (ICRA), pages 13825–13832. IEEE, 2024a. Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep Irmak Guzey, Yinlong Dai, Georgy Savva, Raunaq Bhirangi, and vit features as dense visual descriptors. arXiv preprint Lerrel Pinto. Bridging the human to robot dexterity gap through arXiv:2112.05814, 2(3):4, 2021. object-oriented rewards. arXiv preprint arXiv:2410.23289, Lars Ankile, Anthony Simeonov, Idan Shenfeld, Marcel Torne, and 2024b. Pulkit Agrawal. From imitation to refinement–residual rl for Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey precise assembly. arXiv preprint arXiv:2407.16677, 2024. Levine. Soft actor-critic: Off-policy maximum entropy deep Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. reinforcement learning with a stochastic actor. In International Efficient online reinforcement learning with offline data. In conference on machine learning, pages 1861–1870. Pmlr, 2018. International Conference on Machine Learning, pages 1577– Siddhant Haldar, Vaibhav Mathur, Denis Yarats, and Lerrel Pinto. 1594. PMLR, 2023. Watch and match: Supercharging imitation with regularized 12 CR-DAgger
optimal transport. In Conference on Robot Learning, pages James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel
32–43. PMLR, 2023a. Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan,
Siddhant Haldar, Jyothish Pari, Anant Rai, and Lerrel Pinto. John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Teach a robot to fish: Versatile imitation from one minute of et al. Overcoming catastrophic forgetting in neural networks. demonstrations. arXiv preprint arXiv:2303.01497, 2023b. Proceedings of the national academy of sciences, 114(13):3521– Zhanpeng He, Yifeng Cao, and Matei Ciocarlie. Uncertainty comes 3526, 2017. for free: Human-in-the-loop policies with diffusion models. Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, arXiv preprint arXiv:2503.01876, 2025. and Matt Le. Flow matching for generative modeling. arXiv Neville Hogan. Impedance control: An approach to manipulation. preprint arXiv:2210.02747, 2022. In 1984 American control conference, pages 304–313. IEEE, Huihan Liu, Soroush Nasiriany, Lance Zhang, Zhiyao Bao, and Yuke 1984. Zhu. Robot learning on the job: Human-in-the-loop autonomy Ryan Hoque, Ashwin Balakrishna, Ellen Novoseller, Albert Wilcox, and learning during deployment. The International Journal of Daniel S Brown, and Ken Goldberg. Thriftydagger: Budget- Robotics Research, page 02783649241273901, 2022. aware novelty and risk gating for interactive imitation learning. Yun Liu, Xiaomeng Xu, Weihang Chen, Haocheng Yuan, He Wang, arXiv preprint arXiv:2109.08273, 2021a. Jing Xu, Rui Chen, and Li Yi. Enhancing generalizable 6d Ryan Hoque, Ashwin Balakrishna, Carl Putterman, Michael Luo, pose tracking of an in-hand object with tactile sensing. IEEE Daniel S Brown, Daniel Seita, Brijen Thananjeyan, Ellen Robotics and Automation Letters, 9(2):1106–1113, 2023. Novoseller, and Ken Goldberg. Lazydagger: Reducing context Jianlan Luo, Charles Xu, Jeffrey Wu, and Sergey Levine. Precise switching in interactive imitation learning. In 2021 IEEE 17th and dexterous robotic manipulation via human-in-the-loop international conference on automation science and engineering reinforcement learning. arXiv preprint arXiv:2410.21845, 2024. (case), pages 502–509. IEEE, 2021b. Kevin M Lynch and Frank C Park. Modern robotics. Cambridge Yifan Hou and Matthew T Mason. Robust execution of contact- University Press, 2017. rich motion plans by hybrid force-velocity control. In 2019 Ajay Mandlekar, Danfei Xu, Roberto Martín-Martín, Yuke International Conference on Robotics and Automation (ICRA), Zhu, Li Fei-Fei, and Silvio Savarese. Human-in-the-loop pages 1933–1939. IEEE, 2019. imitation learning using remote teleoperation. arXiv preprint Yifan Hou, Zhenzhong Jia, and Matthew Mason. Manipulation with arXiv:2012.06733, 2020. shared grasping. In Robotics: Science and Systems, 2020. Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Yifan Hou, Zeyi Liu, Cheng Chi, Eric Cousineau, Naveen Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, and Kuppuswamy, Siyuan Feng, Benjamin Burchfiel, and Shuran Aviral Kumar. Policy agnostic rl: Offline rl and online Song. Adaptive compliance policy: Learning approximate rl fine-tuning of any class and backbone. arXiv preprint compliance for diffusion guided control. In 2025 IEEE arXiv:2412.06685, 2024. International Conference on Robotics and Automation (ICRA), Matthew T Mason. Compliance and force control for computer pages 4829–4836, 2025. doi: 10.1109/ICRA55743.2025. controlled manipulators. IEEE Transactions on Systems, Man, 11128452. and Cybernetics, 11(6):418–432, 2007. Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow, Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, and Sergey Levine. Residual reinforcement learning for robot Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x- control. In 2019 international conference on robotics and embodiment: Robotic learning datasets and rt-x models: Open automation (ICRA), pages 6023–6029. IEEE, 2019. x-embodiment collaboration 0. In 2024 IEEE International Michael Kelly, Chelsea Sidrane, Katherine Driggs-Campbell, and Conference on Robotics and Automation (ICRA), pages 6892– Mykel J Kochenderfer. Hg-dagger: Interactive imitation learning 6903. IEEE, 2024. with human experts. In 2019 International Conference on Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Robotics and Automation (ICRA), pages 8077–8083. IEEE, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, 2019. Pamela Mishkin, Jack Clark, et al. Learning transferable visual Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, models from natural language supervision. In International Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, conference on machine learning, pages 8748–8763. PmLR, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, 2021. et al. Droid: A large-scale in-the-wild robot manipulation Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction dataset. arXiv preprint arXiv:2403.12945, 2024. of imitation learning and structured prediction to no-regret Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, online learning. In Proceedings of the fourteenth international Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan conference on artificial intelligence and statistics, pages 627– Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An 635. JMLR Workshop and Conference Proceedings, 2011. open-source vision-language-action model. arXiv preprint Jonathan Spencer, Sanjiban Choudhury, Matthew Barnes, Matthew arXiv:2406.09246, 2024. Schmittle, Mung Chiang, Peter Ramadge, and Siddhartha Kenneth Kimble, Karl Van Wyk, Joe Falco, Elena Messina, Yu Sun, Srinivasa. Learning from interventions: Human-robot Mizuho Shibata, Wataru Uemura, and Yasuyoshi Yokokohji. interaction as both explicit and implicit feedback. In 16th Benchmarking protocols for evaluating small parts robotic robotics: science and systems, RSS 2020. MIT Press Journals, assembly systems. IEEE robotics and automation letters, 5 2020. (2):883–889, 2020. Zhanyi Sun and Shuran Song. Latent policy barrier: Learning robust visuomotor policies by staying in-distribution. arXiv preprint 13
arXiv:2508.05941, 2025. A Technical Appendices and
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Supplementary Material Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. A.1 Controller Architecture arXiv preprint arXiv:2405.12213, 2024. Our software system consists of three independent loops: Aaron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, 1. The base policy loop that runs the diffusion policy. Andrew Senior, Koray Kavukcuoglu, et al. Wavenet: The base policy loop updates at about 1Hz, each time A generative model for raw audio. arXiv preprint predicts 32 frames of actions corresponding to 3.2s arXiv:1609.03499, 12, 2016. of future robot positions. In this work we have not Luigi Villani and Joris De Schutter. Force control. In Springer optimized the implementation for computation speed. handbook of robotics, pages 195–220. Springer, 2016. 2. The residual policy loop that runs at approximately Philipp Wu, Yide Shentu, Qiayuan Liao, Ding Jin, Menglong Guo, 50Hz, each time predicts five frames of delta actions Koushil Sreenath, Xingyu Lin, and Pieter Abbeel. Robocopilot: corresponding to 0.1s of delta positions. The delta Human-in-the-loop interactive imitation learning for robot actions are added to the corresponding base policy manipulation. arXiv preprint arXiv:2503.07771, 2025. actions based one time, before being sent to the low- Haoyu Xiong, Xiaomeng Xu, Jimmy Wu, Yifan Hou, Jeannette Bohg, level compliance controller for execution. The loop rate and Shuran Song. Vision in action: Learning active perception is limited by our current implementation and can be from human demonstrations. arXiv preprint arXiv:2506.15666, improved if needed. 2025. Mengda Xu, Manuela Veloso, and Shuran Song. Aspire: Adaptive 3. The admittance controller loop that runs at exactly skill priors for reinforcement learning. Advances in Neural 500Hz. This loop implements 6D Cartesian compliance Information Processing Systems, 35:38600–38613, 2022. on the robot. It takes reference positions and forces Mengda Xu, Zhenjia Xu, Cheng Chi, Manuela Veloso, and Shuran from the residual policy. When there is zero reference Song. Xskill: Cross embodiment skill discovery. In Conference force and no external force, the admittance controller on robot learning, pages 3536–3555. PMLR, 2023a. lets the robot track the reference position precisely. Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, Gordon Wetzstein, When external force exists, the robot will deviate from Manuela Veloso, and Shuran Song. Flow as the cross-domain the reference position like a spring centered on the base manipulation interface. arXiv preprint arXiv:2407.15208, policy output position. 2024a. Apart from the above controller/policy loops, each hardware Xiaomeng Xu, Yanchao Yang, Kaichun Mo, Boxiao Pan, Li Yi, (e.g. camera, force-torque sensor) has a standalone driver and Leonidas Guibas. Jacobinerf: Nerf shaping with mutual loop maintaining 1. communication with the hardware, and 2. information gradients. In Proceedings of the IEEE/CVF buffers for action and feedback for this hardware. Conference on Computer Vision and Pattern Recognition, pages We choose admittance control to implement compliance 16498–16507, 2023b. in this work, since the Universal Robot we use has high Xiaomeng Xu, Huy Ha, and Shuran Song. Dynamics-guided position-tracking accuracy and is not back-drivable. We open- diffusion model for robot manipulator design. arXiv preprint sourced our implementation at https://github.com/ arXiv:2402.15038, 2024b. yifan-hou/force_control. A more through review Xiaomeng Xu, Dominik Bauer, and Shuran Song. Robopanoptes: of compliance controller implementations can be found The all-seeing robot with whole-body dexterity. arXiv preprint in Lynch and Park (2017) and Villani and De Schutter (2016). arXiv:2501.05420, 2025. Xiaomeng Xu*, Yifan Hou*, Chendong Xin, Zeyi Liu, and Shuran A.2 Stage-Wise Success Rate Song. Compliant residual dagger: Improving real-world contact- rich manipulation with human corrections. In The 39th We report the success rate of the book flipping task into three Annual Conference on Neural Information Processing Systems key stages. Fig. 14 below is a more detailed version of Fig. 9, (NeurIPS), 2025. which reports all the task success rates by stages. Patrick Yin, Tyler Westenbroek, Simran Bagaria, Kevin Huang, Ching-an Cheng, Andrey Kobolov, and Abhishek Gupta. A.3 Correction Data Decomposition Rapidly adapting policies to the real world via simulation- As mentioned in the “Training strategy” part of § 3.2, we guided fine-tuning. arXiv preprint arXiv:2502.02705, 2025. used two strategies to ensure the residual policy behaves Wenzhen Yuan, Siyuan Dong, and Edward H Adelson. Gelsight: stably around low correction data regions. The first strategy High-resolution robot tactile sensors for estimating geometry is to include the no correction portion of online data for and force. Sensors, 17(12):2762, 2017. training and label them with zero residual actions. The second Xiu Yuan, Tongzhou Mu, Stone Tao, Yunhao Fang, Mengke Zhang, strategy is to collect a few trajectories (15 out of the 50 total and Hao Su. Policy decorator: Model-agnostic online refinement correction episodes) in which the demonstrator marks the for large policy model. arXiv preprint arXiv:2412.13630, 2024. whole trajectory as correction, even when the correction is small or zero. In practice, we find that the first strategy works better when the base policy is more stable and has a higher success rate, while the second strategy works better otherwise. In our experiments, we use the first strategy for the book 14 CR-DAgger
Observe-then-Collect Take-over-Correction On-Policy Delta
Book Flipping: Insertion Book Flipping: Rotation (>80°) Book Flipping: Release & Stands
100
100 100 100
100 100
100
90
85 85
80
Success rate (%)
80 80 80
Base (65%) Base (65%) 70
60 60 60
60 60 60 55 55
50
45 Base (40%)
40 40
40 40 40 35
25 25 25
20 20 20
20 20 15 20
10
5 5 5
0 0 0
l ) l ) l )
Retrain Finetune Residua CR (Ours Retrain Finetune Residua CR (Ours Retrain Finetune Residua CR (Ours
Figure 14. Assembly: Hook
Belt Stage-wise on Small
Success PulleyEach row
rates. 80
Belt Assembly:
represents the Hook
results onePulley
Large
onfor task, while Belt Assembly: Finish & Release
80 each column shows the success rate up
to the 100100
100 corresponding stage. 95
100 70 70
90 70 70
Success rate (%)
80 60 60
flipping task and use the second strategy for the50 other three capacitive
50 F/T sensors Choi
50 et al. (2025) and 50vision-based
Base (65%) 65 50 50
tasks. 60 tactile sensors Yuan et al. (2017); Liu et al. (2023); 40
Bauza
60 40
50 40
et al. (2024). 40
40 30 30
A.440 Comparison of Policy Structures 25 25 25
Base (20%) 2020 20
Base (20%) 20
15 15
To 20further explore alternative residual policy 10 formulations 10 beyond the standard MLP0 action head used in our experiments,0 0 0 0 0 0 0 0 l ) l ) l ) Retrain Finetune Residua CR (Ours Retrain Finetune Residua CR (Ours Retrain Finetune Residua CR (Ours we tested two additional popular policy structures on the belt assembly task: a probabilistic MLP Duan et al. (2016); Haarnoja et al. (2018) and a Flow-Matching Transformer Lipman et al. (2022); Black et al. (2024). The probabilistic MLP extends the original deterministic MLP by outputting both the mean and log-variance of a Gaussian distribution over actions and sampling actions from this distribution. The Flow Matching Transformer uses a transformer-based conditional flow model that predicts action velocities along a continuous time trajectory, enabling ODE- based sampling over the residual action distribution instead of one-step prediction. We trained residual policies with these two models using the same data under identical configurations as the original MLP structure on the belt assembly task. The probabilistic MLP achieves a success rate of 100%, and the Flow Matching Transformer achieves 90% (each ablation was evaluated over 10 rollouts). Compared to the success rate of 96.8% achieved by the original MLP, the improvements from more expressive policy structures are inconclusive. The failure modes in this task are relatively homogeneous. We hypothesis that the benefits of expressive policy structure could be more evident for tasks that exhibit more distinctive action multi-modality and leave the more detailed exploration to future work.
A.5 Experiments Compute Resources
We use a desktop with a NVIDIA GeForce RTX 4090 GPU
for training and deployment.
A.6 Hardware Design
Our kinesthetic correction hardware setup features a tool
interface that allows task-specific tool swapping. For the
book flipping task, we designed a customized fork-shaped
tool that can easily insert under the books and flip them.
For the belt assembly task, we used a standard WSG-50
gripper and fin-ray fingers Chi et al. (2024). An interesting
future direction is to leverage generative models for automatic
manipulator design Xu et al. (2024b). Future work can also
incorporate other types of force or tactile sensors, such as