Cross-Embodiment Robot Manipulation Skill Transfer using Latent
Space Alignment
Tianyu Wang Dwait Bhatt Xiaolong Wang Nikolay Atanasov
Abstract— This paper focuses on transferring control policies
between robot manipulators with different morphology. While
reinforcement learning (RL) methods have shown successful re-
sults in robot manipulation tasks, transferring a trained policy
from simulation to a real robot or deploying it on a robot with
different states, actions, or kinematics is challenging. To achieve
arXiv:2406.01968v1 [cs.RO] 4 Jun 2024
cross-embodiment policy transfer, our key insight is to project
the state and action spaces of the source and target robots
to a common latent space representation. We first introduce
encoders and decoders to associate the states and actions of the
source robot with a latent space. The encoders, decoders, and a
latent space control policy are trained simultaneously using loss Fig. 1: Policy transfer to different robot embodiments using latent
functions measuring task performance, latent dynamics consis- space alignment. In the source domain (left), we train a simulated
tency, and encoder-decoder ability to reconstruct the original Panda robot to pick and place objects. Our approach allows trans-
states and actions. To transfer the learned control policy, we ferring the policy to different target domains (right), such as a
only need to train target encoders and decoders that align a new simulated Sawyer robot (top right) or a real xArm6 robot (bottom
target domain to the latent space. We use generative adversarial right), without requiring additional task-specific training data.
training with cycle consistency and latent dynamics losses
without access to the task reward or reward tuning in the target
domain. We demonstrate sim-to-sim and sim-to-real manipula- alignment assumes different robots can solve the same task
tion policy transfer with source and target robots of different at roughly the same speed. However, paired trajectories
states, actions, and embodiments. The source code is avail- collected by a pretrained policy or human labelling are chal-
able at https://github.com/ExistentialRobotics/
cross_embodiment_transfer.
lenging to obtain. In self-supervised learning, latent repre-
sentations are learned from pixel observations for prediction
I. I NTRODUCTION and reasoning in downstream supervised learning and RL
Reinforcement learning (RL) has achieved remarkable suc- applications. This is usually done with autoencoders using
cess in solving sequential decision-making problems where pixel reconstruction losses [8]–[10] or contrastive learning
an agent improves its performance through interactions with [11]–[13] with energy-based losses [14]. However, pixel
the environment. Despite its success, one challenge that pre- reconstruction aims at decoding the original observations
vents the real-world application of RL is sample efficiency. accurately, which may ignore visually small but important
RL training typically requires millions of interactions to features, such as bullets in Atari games [15], or could focus
obtain a policy specialized for a single task. On the other on predicting irrelevant features, such as background. In
hand, humans exhibit the ability to learn from third-person contrastive learning, the number of samples needed to con-
observations of different embodiments [1]–[3]. For example, struct a well-shaped energy surface may scale exponentially,
if a human is able to pour a cup of tea with their right hand, requiring significant computation to train the system [16].
they should need little practice to do it with their left hand. We focus on learning behavior correspondence across
However, if a robot is trained with RL to pick an object up robots of different embodiments, including different num-
with one arm, the learned policy cannot be easily reused if bers of joints, kinematics, and dynamics. For example, we
the arm joint locations, link lengths, or arm dynamics chance. consider training a pick-and-place policy for a 7 degrees of
Transfer learning is a promising methodology to reuse freedom (DoF) Panda arm and transferring it to a 6 DoF
robot skills in different domains. This paper considers trans- xArm6 robot. Our approach is divided into three stages as
fer learning between robots of different morphologies exe- shown in Fig 2: (i) source domain policy learning, (ii) target
cuting the same task (see Fig 1). Recently, many works have domain latent alignment, and (iii) target domain deployment.
proposed solutions for domain discrepancies using paired In the first stage, we project the source states and actions to
and aligned demonstrations [4]–[7]. For example, temporal an embodiment-independent latent space that is used later
to relate to the target states and actions. We train encoders
This work was supported by the Technology Innovation Program
20018112 (Development of autonomous manipulation and gripping tech- and decoders for the latent space projection simultaneously
nology using imitation learning based on visual and tactile sensing) funded with a latent-space policy using loss functions measuring
by the Ministry of Trade, Industry & Energy (MOTIE), Korea. The au- task performance, latent dynamics consistency, and encoder-
thors are with the Department of Electrical and Computer Engineering,
University of California San Diego, La Jolla, CA 92093, USA (e-mails: decoder ability for reconstruction of the original source states
{tiw161,dhbhatt,xiw012,natanasov}@ucsd.edu). and actions. In the second stage, we train target encoders
Fig. 2: Approach overview: (a) The source robot learns encoders and decoders Fs , Gs , F̃s , G̃s for state-action projections between its own space and a latent space. The source robot learns a latent policy π z simultaneously with encoders and decoders with RL. (b) During latent alignment, the source encoder decoder functions are frozen while the target encoder decoder are trained to match latent distributions as well as to satisfy cycle consistency and latent dynamics constraints. (c) During target deployment, we compose the target encoder and decoder functions trained in (b) with the latent policy trained in (a).
and decoders that align a new target robot to the same latent latent input is compared against the nearest neighbor latents space obtained in the first stage. We optimize an adversarial from the demonstrations to query the action. training objective such that the latent distribution of target Learning visual and state representations is useful for samples matches that of the source samples. Additionally, we training better robot manipulation policies or aligning with introduce a cycle consistency loss such that a state-action human demonstrations. Zhang et al. [5] use teleoperation to sample from one domain, projected to the other domain align human demonstrations with robot arms for imitation through the latent space, is close to itself when projected learning. Wang et al. [20] consider a visual manipulation back to its original domain. The cycle consistency constraint planning problem where a causal InfoGAN model [21] can leverage unpaired, unaligned, randomly-collected data generates future visual observations with latent planning from the two domains allowing us to relax the strong assump- and a learned inverse dynamics model tracks the predicted tion that paired correspondences between the two domains observation sequence. Nair at el. [22] pre-train latent visual are available. In the third stage, we compose the target representations using time-contrastive learning [7] and video- encoder and decoder functions from (ii) with the latent policy language alignment [23] to be used in robot manipulation (i) to obtain a target-domain policy. This zero-shot target tasks. Das et al. [24] use visual keypoints as latent features to policy does not require access to a target-domain reward imitate human demonstrations for robot manipulation tasks. function or expert demonstrations, nor does it optimize the Transfer learning leverages the knowledge learned on RL objective in the target domain. In summary, this paper one task to finetune on a second related task. Zhang et makes the following contributions. al. [25] learn invariant representations through bisimulation • We develop a novel approach for cross-embodiment metrics [26] where states are considered similar if their robot manipulation skill transfer. Our key idea is to train immediate rewards and state distributions are similar under a policy in a latent space that facilitates alignment with the same action sequence. Wulfmeier et al. [27] finetune robots with different state and action spaces. a source policy on a target robot of the same type but • We enable transfer without policy retraining or fine- with different dynamics by encouraging a similar state tuning by aligning the target domain to the latent distributions. Zakka et al. [28] use a temporal consistency space via adversarial training and cycle consistency with constraint with paired source and target samples for cross- unpaired, unaligned, randomly-collected target data. embodiment imitation learning. Hejna et al. [29] achieve • We demonstrate that policies for different manipulation cross-morphology transfer by finetuning hierarchical policies tasks trained on a simulated Panda robot can be trans- with a Kullback–Leibler divergence constraint. Zhang et al. ferred to simulated Sawyer and real xArm6 robots. [30] learn a direct state and action correspondence between source and target domains. However, they do not construct II. R ELATED W ORK a latent space so the correspondence has to be re-trained for each new domain. Stadie et al. [3] consider imitation learning Learning invariant features has shown promise for solving under viewpoint mismatch by training viewpoint-invariant RL and domain transfer tasks. Guo et al. [17] and Hafner features. Kim et al. [31] consider matching state distributions et al. [10] independently propose to learn a latent recurrent via generative adversarial networks across domains in an state model from pixel observations and bootstrap latent imitation learning setting. In contrast, our work does not dynamics to learn latent predictions of future observations. require expert demonstrations in target domains. Yin et al. Pari et al. [18] decouple latent representation learning from [32] learn a latent invariant representation for a robot with robot behavior learning. First, they use offline visual data different physical parameters (e.g. link length). However, to train a bootstrap-style self-supervised model [19] where the method cannot be applied if the source and target the encoder learns to project augmented versions of the same robots have different morphology (e.g., different number of image to similar latent representations. During inference, the links). Yoneda et al. [33] align latent features from source and target samples with adversarial training and dynamics A. Latent Policy Training with Source Domain Alignment consistency constraint. While their approach only considers Training a policy on a given robot system directly does visual adaptation for the same robot, we consider a more not help build a representation that allows transferring the general setting of aligning robots of different embodiments. learned skill to a new system. To enable skill transfer, we pa- Our work is inspired by cycle consistency techniques. rameterize a source-domain policy in Ms via a latent policy CycleGAN [34] uses a cycle consistency loss for unpaired in Mz using a state encoder F s and an action decoder G̃s : image-to-image translation. Rao et al. [35] extend CycleGAN π s (ss ) = G̃s (ss , π z (F s (ss ))). Instead of directly predicting to reinforcement learning with an additional Q-function a source action from a source state, we project the source supervision loss and show sim2real transfer for vision-based state ss to a latent state sz = F s (ss ), use the latent policy grasping. Recently, [36] proposed a transformer model with to predict a latent action az = π z (sz ), and project the latent tokenized embeddings to encode visual observations and action back to a source action as = G̃s (ss , az ). generalize across different robot embodiments. We train the encoders, decoders, and latent policy in the source domain simultaneously using three loss functions: III. P ROBLEM F ORMULATION task, latent dynamics, and reconstruction. The task loss A Markov decision process (MDP) M = {S, A, r, T, γ} captures the objective that the policy needs to optimize. consists of a continuous state space S, a continuous ac- Since the latent-space MDP Mz does not correspond to a tion space A, a reward function r : S × A → R, a real system, the latent dynamics loss ensures that the latent probabilistic transition function T : S × A × S → [0, 1], transitions are consistent forward and backward in time. The and a discount factor γ ∈ [0, 1]. We consider a source reconstruction loss ensures that the state-action encoders and MDP Ms = {S s , As , rs , T s , γ} and a target MDP Mt = decoders are inverses of each other. These source-domain {S t , At , rt , T t , γ} with different state and action spaces. training loss functions are described in detail below. We aim to align the source and the target domains by Task Loss. The first objective is to optimize the source- defining a latent-space MDP Mz = {S z , Az , rz , T z , γ} domain encoders, decoders, and latent policy to perform and encoder and decoder functions that relate the source a desired task. We optimize the state encoder F s and the and target domains to the latent-space MDP. We introduce action decoder G̃s jointly with the latent policy π z using a a state encoder F s : S s → S z and an action encoder deterministic policy gradient algorithm, e.g., TD3 [37]–[39]. Gs : S s × As → Az to map source state-action pairs to Recall that deterministic policy gradient algorithms minimize the latent MDP. We also introduce decoders F̃ s : S z → S s expected cumulative cost (or equivalently maximize expected and G̃s : S s × Az → As to map latent state-action pairs cumulative reward) of a parameterized policy πθs : back. Similarly, we define state-action encoders F t , Gt and X ∞ k decoders F̃ t , G̃t between the target MDP andthe latent MDP. Ltask (θ) = −Eπθs γ r(sk , ak ) . (1) Assumethat random transitions Ds = (ssk , ask , ssk+1 ) k=0 and Dt = (stk , atk , stk+1 ) are available from the source and The gradient of the task loss is: target domains. Our goal is to learn the state-action encoders h s i and decoders F {s,t} , F̃ {s,t} , G{s,t} , G̃{s,t} such that a source ∇θ Ltask (θ) = −E ∇θ πθs (s)∇a Qπθ (s, a)|a=πθs (s) , (2) policy parameterized through the latent space, π s (ss ) = where the action-value (Q) function of πθs is G̃s (ss , π z (F s (ss ))), can be transferred to the target domain ∞ by keeping the latent policy π z fixed and only replacing the X πθs k Q (s, a) = Es∼Tπθ ,a∼πθs γ r(sk , ak )|s0 = s, a0 = a , embedding functions, i.e., π t (st ) = G̃t (st , π z (F t (st )). We k=0 consider a deterministic latent policy π z : S z → Az for simplicity of presentation. i.e., the expected sum of rewards when choosing action a in In the context of robotics, the encoders and decoders state s and following πθs afterwards. In our approach, since provide a common latent space in which different robot emo- the policy is a composition of the encoder F s , decoder G̃s bodiments are aligned. A trained latent policy can be reused and latent policy π z , the policy gradient in (2) is backprop- when a new target robot is introduced without learning a new agated through each component to update its parameters. target policy from scratch or requiring new task-specific data. Latent Dynamics Loss. The task loss above optimizes the latent state and action representations to achieve good task IV. C ROSS E MBODIMENT R EPRESENTATION A LIGNMENT performance but does not ensure that the learned represen- tations correspond to consistent latent-space transitions. We In this section, we present an approach to train the source- introduce a self-supervision signal based on latent dynamics domain encoders and decoders as well as a latent space prediction [40] to learn consistent forward and inverse dy- policy to achieve a desired task. Next, we show how to namics models T z : S z × Az → S z and T̃ z : S z × S z → Az train target-domain encoders and decoders by aligning target- in the latent space: domain samples to the latent space constructed in the first stage. This allows transferring the latent space policy learned Ldyn (T z , T̃ z , F s , Gs ) = E(ssk ,ask ,ssk+1 )∼Ds in the source domain without requiring additional task- h i ∥T z (szk , azk ) − szk+1 ∥22 + ∥T̃ z (szk , szk+1 ) − azk ∥22 (3) specific training data in the target domain. Fig. 3: Overview of target domain alignment losses: (left) the adversarial loss ensures that the state-action distributions in the source and target domain match, (middle) the cycle consistency loss regularizes state-action samples to be close to themselves when translated to the other domain and back, (right) the latent dynamics loss enforces consistent forward and inverse latent transitions.
Algorithm 1 Source domain policy training decoders need to be trained. We use three loss functions to 1: Initialize replay buffer D with source samples D s . train the target-domain encoders and decoders, as shown in 2: loop Fig. 3: adversarial, cycle consistency, and latent dynamics. 3: Select action a = G̃s (ss , π z (F s (ss )) + ϵ with exploration The adversarial loss ensures that state-action distributions noise ϵ ∼ N (0, σ 2 ) 4: Observe reward r, next state s′ and store (s, a, r, s′ ) in D obtained in the latent domain from the source encoders and 5: Sample a minibatch {(ssk , ask , rks , ssk+1 )} from D from the target encoders are similar. The cycle consistency 6: Update G̃s , π z , F s with deterministic policy gradient in (2) loss ensures that translating any state-action pair from the 7: Update latent dynamics T z , T̃ z and encoders F s , Gs with source to the target domain and back recovers the same state- latent dynamics loss in (3) action pair. This loss is critical for training the encoders and 8: Update encoders F s , Gs and decoders F̃ s , G̃s with recon- decoders with unpaired samples in the two domains. The struction loss in (4) 9: Output: Source encoders and decoders F s , F̃ s , Gs , G̃s , latent latent dynamics consistency loss is the same as in the source dynamics T z , T̃ z , latent policy π z . training stage and ensures that the encoded target samples follow the same dynamics in the latent space. Adversarial Loss. The encoders and decoders learned where szk = F s (ssk ) and azk = Gs (ssk , ask ). We use deter- in the source domain fix the latent distribution of the ministic latent dynamics for ease of implementation but a source-domain transitions. We aim to train the target-domain stochastic dynamics model can also be considered [41]. The encoders and decoders F t , F̃ t , Gt , G̃t such that the latent dynamics loss is optimized simultaneously with the task loss. distribution of the target-domain transitions Dt matches Reconstruction Loss. Finally, we introduce a reconstruc- the latent distribution of the source-domain transitions Ds . tion loss that ensures that the state and action encoders and We consider an adversarial learning approach, where a decoders are consistent in the sense of being inverses of each discriminator Dz : S z × Az → [0, 1] tries to distinguish other. Reconstruction losses are not always used with visual between (F s (ss ), Gs (ss , as )) from the source domain and observations to avoid reconstructing distractions and noise (F t (st ), Gt (st , at )) from the target domain. The distribution but, if the source domain captures robot joint configuration, of (F s (ss ), Gs (ss , as )) is fixed because F s , Gs are trained this information should be retained accurately in the latent in Sec.IV-A and are frozen during target domain alignment. space. Therefore, we require that F s and F̃ s , as well as Gs The target domain encoders F t , Gt act as a generator aiming and G̃s , are inverse mappings of each other: to synthesize latent state-action pairs F t (st ), Gt (st , at ) that Lrec (F s , F̃ s , Gs , G̃s ) = E(ssk ,ask )∼Ds (4) are indistinguishable from the source latents. The adversarial h i loss in the latent space is defined as: ∥F̃ s (F s (ssk )) − ssk ∥22 + ∥G̃s (ssk , G(ssk , ask )) − ask ∥22 . max z min t t Lzgan (F t , Gt , Dz ) = Training Algorithm. The pseudo code for learning a D F ,G
source-domain policy with latent-space parametrization is E(ss ,as )∼Ds [log Dz (F s (ss ), Gs (ss , as ))] + shown in Alg. 1. We first initialize a replay buffer D E(st ,at )∼Dt log(1 − Dz (F t (st ), Gt (st , at ))) . (5) containing the random source transitions Ds . We update the F s , π z , G̃s with the task policy gradient in (2) and use the We may also match the translated state-action distributions same minibatch samples to optimize the latent dynamics in the source and in the target domains. For example, in the T z , T̃ z with the dynamics loss in (3) and to enforce the target domain, a state-action pair translated from the source encoder-decoder reconstruction loss in (4). domain is s̄t = F̃ t (F s (ss )) and āt = G̃t (s̄t , Gs (ss , as )). With another discriminator Dt : S t × At → [0, 1] which B. Target Domain Alignment distinguishes real target pairs (st , at ) from generated ones Next, we consider aligning a target domain to the latent (s̄t , āt ), the adversarial loss in the target domain is: domain constructed during policy training in the source max min Ltgan (F̃ t , G̃t , Dt ) = domain. This allows us to construct a target policy π t (st ) = Dt F̃ t ,G̃t G̃t (st , π z (F t (st )), in which the task-dependent latent policy E(st ,at )∼Dt [log Dt (st , at )] + E(ss ,as )∼Ds [log(1 − Dt (s̄t , āt )] . (6) π z is fixed (after the source-domain training in Sec. IV-A) and only the task-independent target-domain encoders and Similarly, we can construct a source-domain discriminator Algorithm 2 Target domain policy transfer 1: Freeze learned models F s , F̃ s , Gs , G̃s , T z , T̃ z , π z from Alg. 1 2: loop Sample {(ssk , ask , ssk+1 )} ∼ Ds and (stk , atk , stk+1 ) ∼ Dt 3: z s t 4: Update discriminators D , D , D by maximizing (8) Fig. 4: Robosuite simulation tasks (from left to right): Reach, Lift, 5: Update target encoders and decoders F t , F̃ t , Gt , G̃t by PickPlace and Stack. minimizing (8), (9), (10) 6: Output: Target policy: π t (st ) = G̃t (st , π z (F t (st ))) Ds which distinguishes the translated target distribution in the source domain: V. E XPERIMENTS max min Lsgan (F t , Gt , Ds ) = In this section, we evaluate the ability of our method for D s F t ,Gt policy transfer to simulated and real robot manipulators. E(ss ,as )∼Ds [log Ds (ss , as )] + E(st ,at )∼Dt [log(1 − Ds (s̄s , ās )], (7) A. Simulation Results where s̄s = F̃ s (F t (st )) and ās = G̃s (s̄s , Gt (st , at )). We evaluated our policy-transfer approach and conducted Combining the adversarial objectives in the latent, source, ablation studies in four robot manipulation tasks, Reach, and target domains leads to the complete adversarial loss: Lift, PickPlace, and Stack, in the Robosuite simulation max min Lzgan + Lsgan + Ltgan . (8) environment [42], illustrated in Fig. 4. Details about the tasks D z ,D s ,D t F t ,F̃ t ,Gt ,G̃t and the simulation setup are presented in Appendix VII. Cycle Consistency Loss. Inspired by CycleGAN [34], Reach Task. In the first experiment, we considered the we construct a cycle consistency loss such that translating same Panda robot of different state spaces on the Reach task. a state-action pair from one domain to the other and back Specifically, the source domain was a Panda with a 3-D end- recovers the same state-action pair. This loss leverages effector position state space and 3-D end-effector position unpaired samples since it only requires samples from one displacement action space. The target domain was the same domain, obviating the need for paired samples from both Panda robot with the same action space but using 7-D joint domains. Specifically, if we have a translated target state s̄t = angles as states. We set the latent-space MDP Mz equal to F̃ t (F s (ss )) from a source state, the reconstructed source the source-domain MDP Ms , i.e., F s , F̃ s , Gs,t , G̃s,t were state from it should be close to itself, i.e., ¯s̄s = F̃ s (F t (s̄t )) ≈ identity functions, and learned the target state encoder de- ss . The cycle consistency objective also applies to translated coder F t , F̃ t only. In this setting, F t , F̃ t should theoretically actions, i.e., ā ¯s = G̃s (F̃ s (F t (s̄t )), Gt (s̄t , āt )) ≈ as . The approximate the forward and inverse kinematics between the full cycle consistency loss for both domains is: joint angles and the end-effector position. As an evaluation metric, we computed the ℓ1 -distance be- Lcyc (F t , F̃ t , Gt , G̃t ) = E(ss ,as )∼Ds [∥¯s̄s − ss ∥1 + ∥ā ¯s − as ∥1 ] tween the predicted and ground-truth end-effector positions + E(st ,at )∼Dt ∥¯s̄t − st ∥1 + ∥ā ¯t − at ∥1 . (9) to assess the state alignment. We also evaluated the RL performance by training a source policy π s using TD3 [39] Latent Dynamics Loss. In the previous section, we (π s = π z since F s = G̃s = I) and transferring it to a target trained latent dynamics models T z and T̃ z using source policy π t (st ) = π z (F t (st )) (since G̃t = I). samples. During alignment, we train target encoders such that We compared with the following baselines: (i) invariance the latent dynamics consistency still holds for target samples: through latent alignment (ILA) [33] which performs state Ldyn,t (F t , Gt ) = E(stk ,atk ,stk+1 )∼Dt alignment in the source domain and does not require cycle h i consistency; (ii) our model trained without a latent dynamics ∥T z (szk , azk ) − szk+1 ∥22 + ∥T̃ z (szk , szk+1 ) − azk ∥22 , (10) loss; (iii) a strongly supervised model that is trained on paired end-effector and joint-angle data; (iv) an oracle model where szk = F t (stk ), azk = Gt (stk , atk ). Here T z , T̃ z are fixed obtained by training an RL policy directly in the target during the target latent alignment stage since we want the domain; (v) a random policy in the target domain. Table target samples to follow the same latent transitions induced I shows that the cycle consistency and dynamics consistency by the source domain training in Sec. IV-A. losses are important for improving the performance of our Transfer Algorithm. After training the target-domain model over ILA [33]. The strongly supervised model is encoders and decoders F t , F̃ t , Gt , G̃t with the above losses, marginally better than ours even though it uses paired source the source and target samples Ds , Dt are aligned to the com- and target data while our model uses unpaired data. mon latent space Mz . Alg. 2 summarizes the target-domain Reach Task Ablation. Next, we performed an ablation alignment procedure. During target deployment, the latent study on the dimensions of the latent state and action spaces policy π z from Sec. IV-A is reused while the state encoder for the Reach task. We used a 7 DoF Panda arm as source and action decoder are replaced with F t and G̃t to obtain and a 7 DoF Sawyer arm and a 6 DoF xArm6 arm as target the target-domain policy π t (st ) = G̃t (st , π z (F t (st ))). Our domains. The source and target state dimensions, represented approach does not require paired source and target data or in sine and cosine functions of the joint angles, were 14 reward supervision for target-domain transfer. for the Panda and the Sawyer and 12 for the xArm6. Joint TABLE I: Transfer learning on Reach task with Panda end-effector TABLE II: RL reward for a source policy trained on Panda and space source and Panda joint space target. The results are averaged transferred to Sawyer and xArm6 robots. The reward of an oracle over 5 runs. Lower is better for the ℓ1 error (cm). Higher is policy trained directly on the target robot is shown in parenthesis. better for the RL score. Our model performs better using a latent Task Lift PickPlace Stack dynamics loss, and outperforms ILA [33] which does not use cycle Sawyer 122 ± 41 (181 ± 4) 70 ± 35 (86 ± 15) 85 ± 40 (121 ± 18) consistency. The strong supervision model requires paired training xArm6 132 ± 23 (171 ± 7) 74 ± 36 (84 ± 9) 78 ± 49 (116 ± 23) data. The oracle is an RL policy trained in the target domain. Ours Strong TABLE III: Transfer results of 10 episodes from simulated Panda to Ours ILA [33] Oracle Random real xArm6 with joint velocity control. Our model can successfully w/o dyn. supervision ℓ1 error 0.7 ± 0.1 4.7 ± 0.2 3.9 ± 0.3 0.3 ± 0.1 - - transfer from sim to real. The success rate on the real robot is lower RL score 163 ± 10 41 ± 27 53 ± 27 166 ± 7 172 ± 11 13 ± 8 due to dropped objects or collisions, which cause a safety stop. Task Success Collision Drop Lift 70% 20% 10% PickPlace 60% 20% 20%
task, which helps avoid jerky motions when transferring the
control policy to the real robot. Since ground-truth object
information is not available in the real Lift and PickPlace
Fig. 5: Ablation on latent state and action dimensions for policy experiments, we used an RGBD camera to estimate the transfer from Panda to Sawyer and xArm6 robots on the Reach task. The lowest state and action dimensions with reasonable object position in real time. Details about the object tracking performance are 4. are described in Appendix VII-C. We computed the success rate for each task over 10 test episodes as an evaluation velocities were used as the action space for each robot with metric. The success conditions are defined in Appendix VII- dimension equal to the DoF. Fig. 5 shows the performance A for each task. While simulation environments are quite of a control policy transferred from the Panda source domain tolerant to collisions, on the real robot collisions cause a to the Sawyer and xArm6 target domains with different safety stop and are counted as failures. latent state sz and action az dimensions. We found that the Our method transferred Lift and PickPlace control policies transfer performance drops significantly when the state or trained on the simulated Panda robot to the real xArm6 robot action dimensions are too small. This makes sense as end- with reasonable success rate, as shown in Table III. Fig. 8 effector position control takes place in 3D space and latent shows a successful trajectory of the transferred policy for trajectories in a lower dimensional space cannot recover the the PickPlace task, while Fig. 9 shows some failure modes. 3D motions. In the subsequent experiments, we choose the When the alignment is not perfect, the gripper would collide latent state and action dimensions to both be 4, corresponding with the cube or the table. We also observed that the gripper to the smallest dimensions that achieve good performance. may sometimes drop the cube before it reaches the target. Lift, PickPlace, Stack Tasks. Following a similar setup The robot is not able to re-grasp the dropped cube if its as in the Reach ablation study, we used joint velocity control landing position is outside that of the training distribution. to transfer control policies for Lift, PickPlace, and Stack tasks from a Panda arm source domain to Sawyer and VI. C ONCLUSION xArm6 target domains in the Robosuite simulator [42]. The This paper introduced an approach to learn a common state variables included robot joint angles, gripper width, latent state and action representation across different manipu- gripper touch signal, and object and object goal positions. lators to enable policy transfer. We simultaneously train state The experiment settings are described in detail in Appendix and action encoders and decoders to project from source to VII-A. Fig. 6 shows a Lift task example of the transferred latent domain, as well as a latent policy using RL. During policy from Panda to Sawyer and xArm6 robots. Table II policy transfer, the target domain encoders and decoders shows quantitatively that our model learns a meaningful learn to project to the same latent space as the source domain mapping between robots of different embodiments. However, using unpaired data from adversarial training with cycle manipulation tasks require precise alignment to correctly consistency and latent dynamics constraints. Combining the grasp objects and, sometimes, the transferred policy cannot latent policy with the target-domain encoders and decoders complete the tasks successfully. allows the policy to be deployed without accessing the reward function or additional demonstrations in the target B. Robot Experiments domain. Our policy transfer approach performs better than We evaluated our model’s capability for sim-to-real skill baselines that do not use cycle consistency and latent dynam- transfer for Lift and PickPlace tasks. We used a simulated ics constraints in both simulated and real robot experiments. Panda arm as the source robot and a real xArm6 arm (setup shown in Fig. 7) as the target robot. In either the source or VII. A PPENDIX target domain, the state space consists of joint angles, gripper A. Experiment Settings width, gripper touch signal, and object and object goal posi- We considered Reach, Lift, PickPlace, and Stack tasks tions and the action space is the joint velocity controls. The from the Robosuite simulator [42] as shown in Fig. 4. source-domain policy was trained using behavioral cloning In Reach, the robot end-effector has to reach a target 3D [43], [44] on 100 human demonstrated trajectories in each position. In Lift, it lifts a cube to a target position. In Fig. 6: Examples of transferring Panda robot policy (top row) to Sawyer robot (middle row) and xArm6 robot (bottom row) for the Lift task in the Robosuite simulator. We learned encoders to map the source robot states and actions to a latent space and simultaneously a latent policy to perform the Lift task. The latent space can be used to align different types of target robots (Sawyer or xArm6) and successfully transfer the learned policy without finetuning with task-specific data in the target domain.
Fig. 7: Real-world experiment setup with an xArm6 robot and an RGBD camera for object position estima- tion. Force sensing resistor (FSR) Fig. 8: Sim to real transfer for PickPlace task. The source policy is trained with behavior cloning sensors (bottom right) are attached to in simulation on a Panda robot (top row) and transferred to a real xArm6 robot (bottom row). grippers to obtain pressure signals.
layers of size [256, 256, 256]. We used ReLU activations
for all hidden layers and hyperbolic tangent for the final
layer on all models except for discriminators where leaky
ReLU activations were used. All models were trained with
Fig. 9: Examples of failure modes including cube collisions, table the Adam optimizer using decay rates β1 = 0.9, β2 = 0.999. collisions, or permanently dropping the cube. C. Object Tracking PickPlace, it picks up an object and places it in a bin. In Real-time object tracking was performed with an Intel Stack, it stacks a red cube on top of a green cube. The RealSense D435 stereo camera. We used color-based classifi- state space consists of robot-related states, including sine and cation to find the pixel location of object centroids. The color cosine functions of joint angles or the gripper end-effector range for the green cube in HSV space was from [30, 80, 50] position and gripper opening width, and object-related states, to [90, 255, 255]. The 3D position of the centroid pixel was including object and goal positions. The action space consists obtained by querying the corresponding depth value, where of joint velocities for joint velocity control or delta positions post-processing filters on the disparity and time were applied for end-effector position control, and gripper open or close. to reduce noise. Finally, we obtained the 3D object position For Lift, PickPlace, and Stack tasks, we also used touch in the robot frame by calibrating camera extrinsic parameters sensor signal which was obtained by checking collisions on using hand-eye calibration with ArUco markers. gripper in simulation and by using force sensing resistors on D. Dataset Collection the real xArm6. We collected random trajectories from Panda, Sawyer, and B. Model Architecture xArm6 robots in the Robosuite simulator for state-action The state and action encoders and decoders F s,t , F̃ s,t , alignment. Each robot was placed such that the gripper Gs,t , G̃s,t , latent dynamics functions T z , T̃ z , and discrimina- initial position is at [−0.2, 0, 1.05]. In each episode, the robot tors Ds,t,z were implemented as neural networks with hidden gripper moved in straight line to a randomly sampled position in a 3D rectangular region bounded [−0.2, −0.25, 0.8] to [22] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta, “R3m: A [0.2, 0.25, 1.2] without resetting to initial position. We col- universal visual representation for robot manipulation,” Conference on Robot Learning, 2022. lected 10000 episodes of length 200 for each robot. This [23] S. Nair, E. Mitchell, K. Chen, S. Savarese, C. Finn, et al., “Learning sampling strategy covers the robot workspace better than language-conditioned robot behavior from offline data and crowd- sourced annotation,” in Conference on Robot Learning, 2022, pp. randomly sampling actions at each step. 1303–1315. R EFERENCES [24] N. Das, S. Bechtle, T. Davchev, D. Jayaraman, A. Rai, and F. Meier, “Model-based inverse reinforcement learning from visual demonstra- [1] G. Rizzolatti and L. Craighero, “The mirror-neuron system,” Annual tions,” in Conference on Robot Learning, 2021, pp. 1930–1942. Review of Neuroscience, 2004. [25] A. Zhang, R. McAllister, R. Calandra, Y. Gal, and S. Levine, “Learning [2] M. Iacoboni, R. Woods, M. Brass, H. Bekkering, J. Mazziotta, and invariant representations for reinforcement learning without recon- G. Rizzolatti, “Cortical mechanisms of human imitation,” Science, struction,” arXiv preprint arXiv:2006.10742, 2020. 1999. [26] N. Ferns and D. Precup, “Bisimulation metrics are optimal value [3] B. Stadie, P. Abbeel, and I. Sutskever, “Third-person imitation learn- functions.” in UAI, 2014, pp. 210–219. ing,” International Conference on Learning Representations, 2017. [27] M. Wulfmeier, I. Posner, and P. Abbeel, “Mutual alignment transfer [4] A. Gupta, C. Devin, Y. Liu, P. Abbeel, and S. Levine, “Learning learning,” in Conference on Robot Learning, 2017, pp. 281–290. invariant feature spaces to transfer skills with reinforcement learning,” [28] K. Zakka, A. Zeng, P. Florence, J. Tompson, J. Bohg, and D. Dwibedi, International Conference on Learning Representations, 2017. “Xirl: Cross-embodiment inverse reinforcement learning,” in Confer- [5] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and ence on Robot Learning, 2022, pp. 537–546. P. Abbeel, “Deep imitation learning for complex manipulation tasks [29] D. Hejna, L. Pinto, and P. Abbeel, “Hierarchically decoupled imitation from virtual reality teleoperation,” in IEEE International Conference for morphological transfer,” in International Conference on Machine on Robotics and Automation (ICRA), 2018. Learning, 2020, pp. 4159–4171. [6] P. Florence, L. Manuelli, and R. Tedrake, “Self-supervised correspon- [30] Q. Zhang, T. Xiao, A. Efros, L. Pinto, and X. Wang, “Learning cross- dence in visuomotor policy learning,” IEEE Robotics and Automation domain correspondence for control with dynamics cycle-consistency,” Letters, 2019. International Conference on Learning Representations, 2021. [7] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, [31] K. Kim, Y. Gu, J. Song, S. Zhao, and S. Ermon, “Domain adaptive S. Levine, and G. Brain, “Time-contrastive networks: Self-supervised imitation learning,” in International Conference on Machine Learning, learning from video,” in IEEE International Conference on Robotics 2020, pp. 5286–5295. and Automation (ICRA), 2018, pp. 1134–1141. [32] Z.-H. Yin, L. Sun, H. Ma, M. Tomizuka, and W.-J. Li, “Cross domain [8] M. Watter, J. Springenberg, J. Boedecker, and M. Riedmiller, “Embed robot imitation with invariant representation,” in IEEE International to control: A locally linear latent dynamics model for control from raw Conference on Robotics and Automation (ICRA), 2022, pp. 455–461. images,” Advances in neural information processing systems, 2015. [33] T. Yoneda, G. Yang, M. R. Walter, and B. Stadie, “Invariance through [9] D. Ha and J. Schmidhuber, “World models,” Advances in Neural latent alignment,” Robotics: Science and Systems (RSS), 2022. Information Processing Systems (NeurIPS), 2018. [34] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image- [10] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and to-image translation using cycle-consistent adversarial networks,” in J. Davidson, “Learning latent dynamics for planning from pixels,” in IEEE International Conference on Computer Vision (ICCV), 2017, International Conference on Machine Learning (ICML), 2019. pp. 2223–2232. [11] A. van den Oord, Y. Li, and O. Vinyals, “Representation learning with [35] K. Rao, C. Harris, A. Irpan, S. Levine, J. Ibarz, and M. Khansari, contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018. “RL-CycleGAN: Reinforcement learning aware simulation-to-real,” in [12] B. Eysenbach, T. Zhang, S. Levine, and R. R. Salakhutdinov, “Con- IEEE/CVF Conference on Computer Vision and Pattern Recognition, trastive learning as goal-conditioned reinforcement learning,” Ad- 2020, pp. 11 157–11 166. vances in Neural Information Processing Systems, 2022. [36] K. Bousmalis, G. Vezzani, D. Rao, C. Devin, A. X. Lee, M. Bauza, [13] A. Anand, E. Racah, S. Ozair, Y. Bengio, M.-A. Côté, and R. D. T. Davchev, Y. Zhou, A. Gupta, A. Raju, et al., “Robocat: A self- Hjelm, “Unsupervised state representation learning in atari,” Advances improving foundation agent for robotic manipulation,” arXiv preprint in neural information processing systems, vol. 32, 2019. arXiv:2306.11706, 2023. [14] Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, and F. Huang, “A [37] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour, “Policy gradi- tutorial on energy-based learning,” Predicting structured data, 2006. ent methods for reinforcement learning with function approximation,” [15] L. Kaiser, M. Babaeizadeh, P. Miłos, B. Osiński, R. H. Campbell, in Advances in Neural Information Processing Systems, 1999. K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine, A. Mo- [38] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, hiuddin, R. Sepassi, G. Tucker, and H. Michalewski, “Model based D. Silver, and D. Wierstra, “Continuous control with deep reinforce- reinforcement learning for Atari,” in International Conference on ment learning,” arXiv preprint arXiv:1509.02971, 2015. Learning Representations (ICLR), 2020. [39] S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approxi- [16] Y. LeCun, “A path towards autonomous machine intelligence version mation error in actor-critic methods,” in International Conference on 0.9. 2, 2022-06-27,” Open Review, vol. 62, no. 1, 2022. Machine Learning, 2018. [17] Z. D. Guo, B. A. Pires, B. Piot, J.-B. Grill, F. Altché, R. Munos, and [40] N. Hansen, R. Jangir, Y. Sun, G. Alenyà, P. Abbeel, A. A. Efros, M. G. Azar, “Bootstrap latent-predictive representations for multitask L. Pinto, and X. Wang, “Self-supervised policy adaptation during reinforcement learning,” in International Conference on Machine deployment,” arXiv preprint arXiv:2007.04309, 2020. Learning, 2020, pp. 3875–3886. [41] A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine, “Stochastic latent [18] J. Pari, N. M. Shafiullah, S. P. Arunachalam, and L. Pinto, “The actor-critic: Deep reinforcement learning with a latent variable model,” surprising effectiveness of representation learning for visual imitation,” in Neural Information Processing Systems (NeurIPS), 2020. Robotics: Science and Systems (RSS), 2021. [42] Y. Zhu, J. Wong, A. Mandlekar, R. Martı́n-Martı́n, A. Joshi, S. Nasiri- [19] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, any, and Y. Zhu, “robosuite: A modular simulation framework and E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. G. Azar, benchmark for robot learning,” in arXiv preprint arXiv:2009.12293, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko, “Bootstrap your 2020. own latent-a new approach to self-supervised learning,” Advances in [43] M. Bain and C. Sammut, “A framework for behavioural cloning.” in Neural Information Processing Systems (NeurIPS), 2020. Machine Intelligence 15, 1995, pp. 103–129. [20] A. Wang, T. Kurutach, K. Liu, P. Abbeel, and A. Tamar, “Learning [44] S. Ross, G. Gordon, and D. Bagnell, “A reduction of imitation learning robotic manipulation through visual planning and acting,” Proceedings and structured prediction to no-regret online learning,” in International of Robotics: Science and Systems (RSS), 2019. Conference on Artificial Intelligence and Statistics, 2011, pp. 627–635. [21] T. Kurutach, A. Tamar, G. Yang, S. J. Russell, and P. Abbeel, “Learning plannable representations with causal infogan,” Advances in Neural Information Processing Systems, vol. 31, 2018.