DOGlove: Dexterous Manipulation with a Low-Cost
Open-Source Haptic Force Feedback Glove
Han Zhang1 , Songbo Hu1 , Zhecheng Yuan1,2,3 , Huazhe Xu1,2,3
1
Tsinghua University, 2 Shanghai Qi Zhi Institute, 3 Shanghai AI Lab
https://do-glove.github.io/
Motion Capture Haptic Force Feedback
21 DoFs 5 DoFs
arXiv:2502.07730v1 [cs.RO] 11 Feb 2025
DOGlove Shadow Hand LEAP Hand
a) Teleoperation without Visual Feedback b) Regulating the Sauce Flow c) In-Hand Rotation with Force-Based Extrinsic Dexterity
Fig. 1: DOGlove, a haptic force feedback glove designed for precise teleoperation and dexterous manipulation. It features 21-DoF motion capture and 5-DoF
haptic force feedback. By leveraging action and force retargeting, it enables the teleoperation of dexterous hands for complex, contact-rich tasks, including:
a) without visual feedback, adjusting contact force with a bottle during teleoperation, b) regulating the flow of condensed milk, and c) performing in-hand
rotation by using haptic force feedback to adjust friction.
Abstract—Dexterous hand teleoperation plays a pivotal role I. I NTRODUCTION
in enabling robots to achieve human-level manipulation dex-
Imitation learning (IL) has shown significant promise in ad-
terity. However, current teleoperation systems often rely on
expensive equipment and lack multi-modal sensory feedback, dressing complex manipulation tasks [7, 8, 46, 45]. However,
restricting human operators’ ability to perceive object properties it often necessitates a substantial amount of task-specific data
and perform complex manipulation tasks. To address these to train a generalizable learning policy. Efficiently collecting
limitations, we present DOGlove, a low-cost, precise, and haptic and ensuring the high quality of such demonstrations remains a
force feedback glove system for teleoperation and manipulation.
persistent and challenging problem for the robotic community.
DOGlove can be assembled in hours at a cost under 600 USD.
It features a customized joint structure for 21-DoF motion Teleoperation is among the most commonly used methods
capture, a compact cable-driven torque transmission mechanism for collecting demonstrations, often involving the development
for 5-DoF multidirectional force feedback, and a linear resonate of a wide range of devices tailored to meet diverse data
actuator for 5-DoF fingertip haptic feedback. Leveraging action acquisition requirements. These devices enable the transfer
and haptic force retargeting, DOGlove enables precise and im-
of human manipulation behaviors to various robotic plat-
mersive teleoperation of dexterous robotic hands, achieving high
success rates in complex, contact-rich tasks. We further evaluate forms [13, 12, 6, 10, 15]. However, when it comes to dexterous
DOGlove in scenarios without visual feedback, demonstrating hands, their high degrees of freedom (DoFs) and inherent
the critical role of haptic force feedback in task performance. In complexity impose even stricter demands on operational pre-
addition, we utilize the collected demonstrations to train imitation cision and the accuracy of human motion capture. Hence, it is
learning policies, highlighting the potential and effectiveness of
crucial to design an intuitive, responsive, and highly precise
DOGlove. DOGlove’s hardware and software system will be fully
open-sourced at https://do-glove.github.io/. device specifically suited for dexterous hand teleoperation
applications.
a) Teleoperate the LEAP Hand to squeeze condensed milk onto the bread
b) Grasp the bottle without relying on visual feedback c) Identify objects without visual feedback
Fig. 2: Teleoperation demos. a) While squeezing condensed milk, the operator regulates the flow using haptic force feedback from DOGlove. b) The operator grasps a slipping bottle without visual feedback. c) The user identifies object pairs solely through haptic force feedback.
Vision-based methods are primarily used for tracking the integration of force and haptic feedback creates an immersive human hand in dexterous hand teleoperation. A simple ap- and responsive interface for dexterous manipulation. proach involves using RGB cameras [25, 35], but the accuracy Action and haptic force retargeting: We propose a general of hand gesture capture is often questioned and may be further retargeting framework. For action retargeting, the rigid con- limited by visual obstacles during hand-object interactions. straints of the glove allow fingertip positions to be mapped Motion capture (MoCap) systems [23, 36, 27, 38] provide sta- from the human hand to the target robotic hand. For haptic ble hand tracking. However, relying solely on visual feedback force retargeting, the combination strategy enables users to for teleoperation makes intuitive control challenging for the perceive contact information during teleoperation. human operator. The resulting system, DOGlove, provides precise hand pose Haptic force feedback offers additional perceptual charac- motion capture and the ability to sense interactions with ma- teristics beyond those provided by vision, such as the ability nipulated objects. This enables human operators to intuitively to sense an object’s weight, friction, and softness. Integrating and efficiently teleoperate dexterous hands. As shown in Fig. 2, haptic feedback into teleoperation can enrich the feedback it further supports the completion of complex manipulation available during interaction and enable the completion of tasks. We evaluate the necessity of haptic force feedback more challenging tasks. Recently, some commercialized force through a user study and further assess the teleoperation effi- feedback gloves [9, 29, 21, 14] have shown promise for ciency and data accuracy of DOGlove in several quantitative enabling intuitive teleoperation. However, these solutions are experiments. often prohibitively expensive and require significant integra- Finally, we demonstrate that DOGlove seamlessly integrates tion efforts to work with existing robot learning frameworks. with existing methods in robot learning. We use DOGlove to In this paper, we introduce DOGlove, a low-cost, fully teleoperate the LEAP Hand mounted on a Franka robot arm, open-sourced, and easy-to-manufacture haptic force feedback collecting data to train imitation learning policies. To foster glove for dexterous manipulation. The glove can be assembled further research, we will open-source the mechanical designs, in hours for a total cost of 600 USD. Key features of DOGlove circuit designs, embedded code, assembly instructions, URDF include: models, retargeting methods, and MuJoCo simulation environ- 21-DoF motion capture: DOGlove features an anthropo- ment at https://do-glove.github.io/. morphic design resembling the human hand, providing precise II. R ELATED W ORK motion capture and a comfortable wearing experience. More- over, we propose a customized joint structure that integrates a A. Data Collection from Human Demonstrations compact, low-cost yet accurate joint encoder, with the entire A substantial amount of task-specific data is essential for assembly measuring less than 15 mm in thickness. imitation learning. In dexterous manipulation, obtaining high- 5-DoF haptic force feedback: DOGlove leverages a cable- quality hand motion data is critical for training effective poli- driven mechanism to deliver force feedback to each finger cies. Prior work includes extracting demonstration data from while maintaining a compact and cost-effective design. Addi- human videos [32, 40, 41, 2] and hand trajectories [37, 42]. tionally, each fingertip is also equipped with a linear resonant While these approaches are accessible and have shown promis- actuator (LRA) to provide realistic haptic feedback. This ing results, the significant visual gap between recorded human demonstrations and the robot’s perception often makes real- While ensuring these functionalities, the glove is optimized for world transfer challenging. An alternative is using dedicated accessibility by the research community, focusing on low cost, hardware for data collection to bridge this gap. Hand-held ease of manufacturing, and high performance. To achieve these grippers [33, 8, 26] have proven effective in capturing robot goals, DOGlove incorporates the following design principles: manipulation data. However, these systems are primarily de- signed for parallel grippers. Another widely used approach A. Low cost is MoCap systems, which record human demonstrations and Commercial products such as the SenseGlove Nova [29] extract hand motion data. These systems include camera-based and Manus VR [21] cost more than 5,000 USD, making methods [25, 49], glove-based tracking systems [38, 19, 20], them prohibitively expensive for many researchers. In contrast, marker-based tracking [48], and commercial MoCap solu- DOGlove provides a low-cost solution under 600 USD. tions [34, 11]. While MoCap offers high-precision tracking, B. Ease of manufacturing bridging the embodiment gap between human and robotic hands remains a persistent challenge. All parts of DOGlove are either readily available for pur- chase online or manufacturable using standard methods. The B. Dexterous Hand Teleoperation glove’s main body can be 3D-printed using a commodity 3D Collecting high-quality human demonstrations through printer, while the remaining electronics and servos are easily robotic teleoperation systems [12, 6, 10, 15] also plays a sourced. The entire glove can be assembled within 6 hours. critical role for advancing dexterous manipulation. Existing C. Performance Sufficiency research has explored teleoperation from various perspectives, including leader-follower setups such as ALOHA [46, 47, To ensure precise fingertip position tracking, the glove’s 13, 1]. However, teleoperating dexterous hands remains a encoders deliver joint angle data with an error range of ±7.2°, significant challenge. OpenTelevision [6] leverages VR devices which can be further minimized through careful calibration. to capture hand poses and streams the pose information For intuitive haptic force feedback, the servos provide suffi- for retargeting to robotic hands. BiDex [31], on the other cient stall torque to halt human finger movement, while the hand, implements a teleoperation system based on commercial haptic engine supports multiple haptic waveforms to enhance motion capture gloves [21] and leader arms. Compared to these tactile sensations. frameworks and other glove-based systems [19, 20], DOGlove D. Low latency offers distinct advantages. It eliminates the need for expensive equipment while precisely capturing fingertip positions and The MoCap system operates at a maximum frequency of delivering richer haptic force feedback to the operator. This 120 Hz, while the haptic force feedback system achieves a system achieves accurate dexterous hand teleoperation with a maximum frequency of 30 Hz. Together with the retargeting low-cost setup, making it an efficient alternative. algorithm, the system ensures seamless operation at a mini- mum frequency of 30 Hz, providing a smooth and responsive C. Teleoperation with Haptic Force Feedback teleoperation experience. While recent studies rely on visual information to capture IV. H ARDWARE S YSTEM environmental characteristics, vision alone inherently limits the richness of available sensory data. In contrast, haptic force A. Kinematic Design feedback enhances the teleoperation experience by providing The kinematic design of DOGlove refers to the use of con- greater immersion and improving perception of the robot’s straints to achieve desired movements, emulating the natural status and movement compared to vision-based methods. motion of a human hand. To ensure precise MoCap capabilities Bunny-VisionPro [10] and Liu et al. [20] apply real-time haptic and a comfortable wearing experience, DOGlove are designed feedback to enable more accurate manipulation. Xu et al. [39] to closely resemble the anthropomorphic structure of the build a bilateral isomorphic bimanual telerobotic system using human hand. a commercial force feedback glove [9] to enhance percep- Several studies [3, 4] model the hand skeleton as a kine- tion and improve performance in complex tasks. NimbRo- matic chain, represented by a hierarchical structure of rigidly Avatar [28] and Mosbach et al. [22] integrate commercial connected joints. As shown in Figure 3, the kinematic structure force feedback glove [29] into dexterous teleoperation systems. of the human hand primarily consists of two types of joints: However, these approaches rely on specialized or expensive hinge joints and ball joints. equipment. In contrast, DOGlove provides a highly accurate In the index, middle, ring, and pinky finger, distal inter- teleoperation system with integrated haptic force feedback at phalangeal (DIP) and proximal interphalangeal (PIP) joints a significantly lower cost and can be widely used in dexterous are hinge joints with 1-DoF, allowing only flexion-extension manipulation. movements. In contrast, the metacarpophalangeal (MCP) joint is a ball joint with 2-DoF, allowing both flexion-extension and III. G LOVE D ESIGN O BJECTIVES adduction-abduction movements. DOGlove is designed to precisely capture human hand poses The thumb differs slightly in its structure. The interpha- and provide haptic force feedback for intuitive teleoperation. langeal (IP) and metaphalangeal (MCP) joints are hinge joints Ring Pinky Ring Pinky Middle Middle
Index
Index
PIP
DIP
DIP
PIP
MCP_B
MCP (B+S)
MCP_S
Thumb
Thumb Wrist_R IP Wrist
MCP MCP
IP TM (B+S)
TM_B
B = Bend (flexion-extension)
TM_S S = Split (adduction-abduction)
R = Rotate (pronation–supination)
Fig. 3: The kinematic structure of DOGlove, designed to replicate the kinematics of a human hand. The MCP (B+S) and TM (B+S) joints are modeled as ball joints using a combination of two rotary joints. The right figure from [3] illustrates the simplified human hand kinematics.
with 1 DoF, while the additional trapeziometacarpal (TM) joint is a ball joint that supports both flexion-extension and adduction-abduction movements. To further enhance dexterity, an additional DoF at the wrist allows the thumb to perform pronation-supination movements. To implement these joints in DOGlove, hinge joints are modeled as two linkages connected by a rotary joint, with a joint encoder installed on the rotary axis to capture motion. Ball joints are designed as a combination of two orthogonal rotary joints, each equipped with a joint encoder on its respective rotary axis. The linkage lengths in DOGlove are designed to accommo- Proper linkage lengths
date the majority of adult human sizes. To achieve this, a stan- dard human finger length was first modeled, and the glove’s linkage parameters were simulated to ensure an optimal range of motion. As shown in Figure 4, improper linkage lengths can obstruct the natural flexion-extension of the fingers, leading to discomfort and reduced MoCap performance. Furthermore, DOGlove features a modular design where all fingers share a common structural framework. This modularity enables users to replace linkages with customized sizes as needed, enhancing Improper linkage lengths both adaptability and usability. Fig. 4: Improper linkage lengths can cause collisions between the human B. Finger Design finger (link f, g) and the glove (link m), restricting finger movements and As shown in the exploded-view in Figure 3, DOGlove is leading to discomfort and poor MoCap performance. composed of the thumb, index, middle, ring, and pinky finger assemblies, along with the palm base structure. The design of The exploded view of a single finger is illustrated in each finger assembly follows a modular approach, ensuring Figure 5. The highlighted area indicates the basic components consistent structural elements across all fingers. of a rotary joint. Each rotary joint is constructed using an Fixed Pulley B M3 Locknut Fixed Pin M3*20 Headless Clevis Pin Joint Encoder with Retaining Ring Groove
Fixed Pulley A
Servo Pulley
Ball Bearing
M4*15 Shoulder Screw
M4*15 Shoulder Screw
Joint Holder
Finger Proximal Finger Pulley B
Finger Distal Finger Middle
Dynamixel XC/XL330
Finger Pulley A
LRA M4*20 Headless Clevis Pin Motor Shaft
with Retaining Ring Groove
Ball Bearing
LRA Holder
Fig. 5: Exploded view of the finger assembly, with the highlighted area indicating the basic components of a rotary joint.
M4×15 shoulder screw to connect the finger linkages, ball to the actual joint angle. These voltage signals are read bearing, and joint encoder, secured with an M3 locknut. This by an Analog-Digital Converter (ADC) module. For precise design ensures smooth and reliable joint rotation. The main conversion, we use the TI ADS1256, a low-noise 24-bit body of the finger, colored white and gold, is 3D printed using ADC operating at 30k samples per second. The converted PETG material for ease of fabrication and durability. signals are then sent to a microcontroller unit (MCU), the Given the limited space on the back of the human hand, ST Electronics STM32F042K6T6, which operates at a the finger assembly’s width is constrained to less than 26 clock speed of 48 MHz. To optimize system performance and mm. Simultaneously, to provide effective force feedback, the reduce OS scheduling overhead, the STM32’s Direct Memory actuator must deliver a stall torque of at least 0.5 N·m. Access (DMA) feature is utilized to accelerate joint encoder Additionally, adjustable stiffness requires the actuator’s current readings. Finally, the processed joint data is transmitted to the to be regulated. Since the actuator is directly connected to host machine via a serial port on the STM32. the pulley system as a rotary joint for M CPB , it is essential The voltage readings are mapped directly to joint an- to measure its rotary position in real time to achieve precise gles under the assumption that the supply voltage of the joint angle control. Considering these design requirements, the STM32 (approximately 3.3 V) corresponds to 360°, while the Dynamixel XC/XL330 servo motors were selected as the ground voltage (0 V) corresponds to 0°. Using the ADC output actuators for force feedback. It fulfills the torque, size, and voltage, the joint angle is calculated as: real-time position measurement needs, making it a suitable VADC choice for DOGlove. αjoint = · 360 (1) VCC
- Joint Encoders: The primary error in this conversion comes from the lin- To integrate joint encoders into the finger linkages, the earity error of the encoder which is ±2% according to its encoders must be compact while maintaining high precision. datasheet. This results in an angular error of ±7.2° when mea- Additionally, as 16 encoders are required in combination with suring joint angles. To mitigate this, we employ a calibration 5 servo motors to achieve 21-DoF MoCap capabilities, the process. Using an external high-precision joint encoder, we cost of each encoder needs to be affordable. Considering map the voltage reading to an actual joint angle, creating a these constraints, we selected the Alps RDC506018A rotary correction table for each encoder. With this calibration, the sensor as the joint encoder. This compact encoder (W11 mm error can be reduced to within ±1°. × L14.9 mm × H2.2 mm) is easily integrated into the 3D- printed joint structures. The encoder operates as a variable 2) Cable-Driven Force Feedback Structure: resistor, changing its resistance as the shaft rotates. To provide force feedback on the human fingers, the output The resistance changes are converted into voltage sig- torque of the Dynamixel servo must be transmitted to the nals using a simple voltage divider circuit. Due to the en- glove’s finger linkage system. As illustrated in Figure 5, the coder’s linear response, the voltage output is proportional rotary axis of the servo and the rotary axis of the M CPB joint are misaligned. Consequently, a transmission mechanism fingertip in DOGlove is equipped with a tactile actuator. is required to transfer the torque effectively. Traditional haptic actuators include eccentric rotating Although a bevel gear system could serve as a potential mass (ERM) motors and linear resonate actuators (LRAs). solution, its implementation would require significant space to Limited by the inertia of the rotating mass, ERM motors accommodate the two orthogonal gears. Additionally, transmit- are slow to start and stop, making it challenging to produce ting large torque through gears can cause deformation in the complex waveforms needed for subtle tactile sensations. On gear shaft, leading to gear slippage. In contrast, a cable-driven the contrary, LRAs offer linear motion, resulting in a cleaner mechanism offers a more compact design while ensuring stable and more precise tactile output. torque transmission. In DOGlove, we use LRAs with a diameter of 8 mm and Traditional cable-driven systems typically provide unidi- a height of 2.5 mm, installed close to the fingertips. These rectional force transmission on the tension side, relying on LRAs provide vibration stimuli by resonating at approximately a spring to generate force in the opposite direction. How- 240 Hz along Z axis, which is orthogonal to the fingertip ever, this approach introduces unrealistic feedback sensations. surface. Operating at 1.2 Vrms , the LRAs generate high- While using two servos per finger could resolve this issue, it quality haptic waveforms. To fully leverage the potential of would significantly increase the glove’s weight and cost. the LRA, we employ the TI DRV2605L motor driver, which includes the licensed Immersion TouchSense© 2200 Servo Pulley haptic library. This driver supports over 100 pre-programmed waveforms, allowing DOGlove to deliver realistic and refined haptic feedback. Fixed Pulley B C. Wrist Localization In DOGlove, we design a shell with a 1/4 inch screw con- nector to accommodate external wrist localization devices. For Cable B Fixed Pulley A our experiments, we use the HTC Vive Tracker for real- Finger Pulley B time wrist position tracking. However, the design is compatible Cable A with other solutions, depending on the user’s requirement. V. R ETARGETING A. Action Retargeting To map human hand gestures to a robotic hand, it is essential to perform action retargeting, which converts motion data Finger Pulley A from the glove into robotic hand movements. This process addresses both the embodiment gap and motion discrepancies. Fig. 6: Pulley system of the cable-driven mechanism. Previous studies [38, 11, 34] highlight the significance of fingertips, as they are the primary contact area during object To address these challenges, DOGlove utilizes a pulley interactions. Building on this insight, we apply the 5-DoF system to provide the bi-directional force feedback, as shown haptic force feedback to the human operators’ fingertips and in Figure 6. DOGlove uses a 0.6 mm stainless steel braided adopt a retargeting method focused on fingertip positions. wire as the cable, chosen for its strength and durability. The Our approach combines Forward Kinematics (FK) Servo Pulley connects the servo to the finger linkage (Finger to compute human fingertip positions and Inverse Middle) via the Finger Pulley, maintaining a 1:1 transmission Kinematics (IK) to calculate the corresponding robotic ratio. To minimize friction during transmission, the Fixed hand positions. When wearing DOGlove, the human operator Pulley is used to redirect the cable’s path. When the Servo secures their fingertips inside the finger caps. Since the glove Pulley rotates clockwise, the tension on Cable B increases, acts as a rigid body, the relative positions of the fingertips with causing Finger Pulley to rotate clockwise. The extra slack respect to the glove’s origin can be accurately calculated. With on the Cable A side is taken up by the Servo Pulley A. DOGlove’s anthropomorphic kinematic design and precise Since the finger linkage is fixed to the Finger Pulley, it also MoCap capabilities, fingertip positions are effortlessly deter- rotates clockwise, resulting in the extension movement of mined using the glove’s built-in FK. To map these positions the M CPB joint. Similarly, when the Servo Pulley rotates to a robotic hand, we utilize Mink [44], a differential inverse counterclockwise, the tension shifts to Cable A, producing a kinematics library, to generate smooth and feasible motions flexion movement of the M CPB joint. for the robotic hand. This configuration enables bi-directional torque transmis- A size discrepancy often exists between the human hand and sion while maintaining a simple, compact, and cost-effective the target robotic hand. To address this, we introduce a scaling design. factor when calculating IK, allowing adaptation to different
- Fingertip Haptic Feedback: robotic hand sizes. This ensures an intuitive teleoperation To further enhance the operator’s tactile experience, each experience where the robotic hand naturally mirrors the human LEAP Hand Shadow Hand Inspire Hand Allegro Hand
Fig. 7: Action retargeting results: Teleoperating the LEAP Hand to grasp a toy in the real world and teleoperating the Shadow Hand, Inspire Hand, and Allegro Hand in simulation.
hand’s gestures. For instance, when the human operator opens from noise, we set the first threshold at 10 g to initiate their hand, the robotic hand open proportionally. Similarly, haptic feedback. During a user study without visual feedback when the human operator brings their thumb and index finger (Section VI-A), we observe that human operators are highly together, the robotic hand’s thumb and index finger also touch. sensitive to force feedback. To create a more realistic expe- This ability for precise fingertip alignment is critical for tasks rience, force feedback is applied only after the force sensor like grasping small objects. readings exceed 50 g, which serves as the second threshold. In our experiments, we deploy the system on the LEAP Furthermore, observations from the bottle-slipping experiment hand [30] in real-world scenarios and test it with various (Section VI-B) reveal that continuous haptic feedback during robotic hands in the MuJoCo simulator. Figure 7 presents the teleoperation can create a misleading sensation, interfering retargeting results of DOGlove in both simulation and real- with the operator’s ability to perceive the subtle properties world environments. of the object’s surface. To address this, a third threshold is set at 100 g, where haptic feedback stops, leaving only force B. Haptic Force Retargeting feedback active. To provide the haptic force feedback, it is first necessary For force feedback, the Dynamixel servos operate to sense tactile or force information at the robotic hand’s in current-based position control mode. The fingertips. This can be achieved using a simple tactile sensor, force readings from the LEAP Hand fingertips are clamped such as the force sensing resistor (FSR) sensor [10, 16], or to the range [0g, 3000g], and mapped linearly to the KP for better performance, by utilizing an F/T sensor [17, 5] or a gain of the Dynamixel servos. For haptic feedback, we use vision-based tactile sensor [43, 18]. waveform ID 56 from the haptic engine library, corre- In our experimental setup, we install a 1-D force sponding to Pulsing Sharp 1-100%. sensor on each fingertip of the LEAP Hand, with a mea- This combination strategy for haptic force retargeting en- surement range of 3 kg and a precision of 1 g. During our ables human operators to distinguish object shape, size and quantitative experiments (Section VI), we identify a com- softness without visual feedback. It also improves performance bination strategy for integrating haptic and force feedback in complex, contact-rich manipulation tasks. Further details are that optimizes performance. This strategy along with the provided in Section VI. corresponding thresholds and feedback patterns is summarized in Table I. VI. E XPERIMENTS Force Sensor Readings (g) Haptic Feedback Force Feedback <10 ✗ ✗ In this section, we use DOGlove to teleoperate the LEAP 10–50 ✓ ✗ Hand [30] mounted on the Franka Robot Arm to evaluate 50–100 ✓ ✓ its effectiveness through a series of challenging tasks across >100 ✗ ✓ three key aspects: Table I: The combination strategy for haptic force feedback in DOGlove. • Haptic Force Perception: Without visual feedback, how ef- When the robotic hand touches an object, the force sen- fectively can DOGlove assist human operators in perceiving sor readings increase. To effectively distinguish these signal object properties through haptic force feedback? • Teleoperation Efficiency: Does integrating haptic force five object pairs, selected based on factors such as shape, size, feedback improve vision-based teleoperation success rates and softness. and reduce task completion time? Can DOGlove enable Metrics: Users’ ability to distinguish object pairs is evaluated human operators to perform challenging, contact-rich ma- based on their success rate. nipulation tasks? Challenges: The five object pairs are intentionally chosen • IL Compatibility: Can the data collected via DOGlove be based on the following considerations: leveraged to train IL policies for dexterous manipulation? • Pair 1: Basic Pair, different shape. The ball and the box have Evaluation Setup: To evaluate the effectiveness of haptic distinctly different shapes (Fig 8b). force perception, we conduct a user study (Section VI-A) and • Pair 2: Basic Pair, similar shape, different size. The peanut a quantitative experiment (Section VI-B). Teleoperation effi- bottle and the coffee paper cup share a similar cylindrical ciency is assessed in Experiment VI-C, while IL compatibility shape, but their diameters differ slightly (Fig 8c). is evaluated in Experiment VI-D. • Pair 3: Basic Pair, similar softness, different size. The two toys have similar softness and shapes but vary in Comparisons: All experiments share the following compar- size (Fig 8d). ison conditions, although a subset of these may be selected • Pair 4: Challenging Pair, similar size and shape, different depending on the specific task setup: softness. Two identical bottles are used, one filled with pure • Only Force: Force feedback is enabled only when the water (soft) and the other filled with carbonated cola, shaken force sensor readings exceed 10 g. to increase its hardness (Fig 8e). • Only Haptic: Haptic feedback is enabled only when the • Pair 5: Challenging Pair, similar shape, different size and force sensor readings exceed 10 g. softness. A toy cabbage (softer, larger) and a real cab- • Haptic+Force: A combined feedback strategy is applied, bage (Fig 8f). as detailed in Section V-B. • No Haptic/Force: DOGlove is used solely for MoCap, Pair 1 Pair 2 Pair 3 Pair 4 Pair 5 with no feedback provided. Only Force 5/5 5/5 5/5 4/5 0/5 • Baseline: AnyTeleop [25], a widely recognized vision- Only Haptic 5/5 5/5 5/5 3/5 3/5 based hand retargeting method, is used as the MoCap Haptic+Force 5/5 5/5 5/5 3/5 2/5 baseline. Table II: Success rates in the user study. All feedback modes perform well for the basic pairs. For the challenging pairs, force feedback is more sensitive A. User Study: Object Perception w/o Visual Feedback to softness, while haptic feedback is more sensitive to shape.
Performance: As shown in Table II, even without visual
and auditory feedback, all participants effortlessly distinguish
basic pairs 1-3. For challenging pair 4, most participants can
Earphones perceive softness using only force feedback. Some also discern Eyemask d) softness using only haptic feedback by evaluating the duration of contact during deformation. For challenge pair 5, when the robotic hand grasps the softer toy cabbage, it deforms to resemble the size of the real cabbage. This deformation increases its perceived softness, a) e) making it difficult for participants to distinguish using force feedback alone. For both challenge pairs, combining haptic and force feed- back slightly reduces user sensitivity, leading to a marginally lower accuracy. b) c) f) B. Bottle-Slipping Fig. 8: User Study. (a) Experiment setup: Users wear an eyemask and 1) Teleoperation w/o Visual Feedback headphones to eliminate visual and auditory feedback. (b)–(f) Object pairs Task: In this experiment, the human operator must per- tested in the study. form a bottle-slipping action relying solely on feedback from Task: Five untrained human operators participate in this user DOGlove. A 15-second countdown timer is set for each trial. study. During the experiment, they are required to distinguish If the bottle successfully slips without falling within the 15 between five pairs of objects solely through feedback from seconds, the trial is denoted as successful. DOGlove, without any visual or auditory input (achieved by Metrics: The success rate. wearing an eyemask and headphones). In each trial, a pair of Challenges: Without any visual or auditory input (achieved objects is randomly selected, and users provide their answers by wearing an eyemask and headphones), the operator must immediately after experiencing feedback from DOGlove for determine if the bottle is slipping at the right speed or too both objects. Figure 8 illustrates the experiment setup and the quickly, risking a fall. w/o Visual with Visual a) Bottle-Slipping Experiment
b) Rotating and Placing the Carton
Fig. 9: Teleoperation experiments and quantitative results. a) Without visual feedback, force feedback significantly improves the task success rate. With visual feedback, it enhances precise control. b) In in-hand rotation, the challenge is to slightly release the fingers, allowing the carton to rotate without slipping out (as shown in the middle two images).
Performance: As shown in Fig 9a, force feedback signifi- also teleoperation precision. To minimize slipping deviation, cantly improves the success rate of this task. Additionally, operators are instructed to control the LEAP Hand carefully incorporating haptic feedback further enhances overall perfor- and optimally. As shown in Fig 9a, similar to previous results, mance. However, since the fingers of the LEAP Hand maintain haptic feedback does not provide additional information and continuous contact with the bottle during the task, haptic may even interfere with task precision. However, force feed- feedback does not provide additional information beyond using back enables operators to minimize slipping deviation more the glove solely as a MoCap device, resulting in the same effectively. While using DOGlove solely as a MoCap device success rate for both conditions. achieves the same success rate as with haptic force feedback, Due to differences in retargeting strategies, even a slight it results in a larger average slipping deviation. change in human finger position can lead to a significant C. Rotating and Placing the Carton deviation in the LEAP Hand’s movements. As a result, AnyTeleop [25] struggles to perform the slipping task effec- Success Rate Average Completion Time (s) tively. Only Force 9/10 18.92 2) Teleoperation with Visual Feedback Only Haptic 9/10 21.16 Task: Unlike the previous blindfolded experiment, this ex- Haptic+Force 10/10 19.89 No Haptic/Force 4/10 24.76 periment allows operators to have visual feedback. To further AnyTeleop 1/10 54.85 evaluate the operator’s control ability, they are required to slip the bottle to a specified distance (9 cm). A trial is denoted Table III: Quantitative experiment results. Haptic force feedback enables operators to achieve a higher success rate and a faster average completion as successful if the bottle slips without falling. Additionally, time, as haptic feedback provides contact information, while force feedback We measure the deviation between the actual slipping distance indicates the proper timing for in-hand rotation. and the target distance (9 cm). Task: This is a long-horizon contact-rich task. As shown in Metrics: Performance is evaluated using two metrics: Fig 9b, the operator must first pick up the carton horizontally, • Success Rate: A trial is denoted as successful if the bottle then perform an in-hand rotation, orienting the carton verti- slips without falling. cally before placing it into a small bucket. • Slipping Deviation: This measures the difference between Metrics: Performance is evaluated using two metrics: the target sliding distance (9 cm) and the actual slipping • Success Rate: A trial is denoted as successful if the carton distance, with a smaller deviation indicating greater opera- rotates more than 45 degrees and is successfully placed into tional accuracy. the bucket. Challenges: Operators must precisely control the bottle to • Completion Time: The total time taken to complete the achieve the desired distance. While a greedy approach often entire process. causes the bottle to fall and results in failure, a conservative Challenges: approach leads to an unsatisfactory distance deviation. • Precise Manipulation: The operator must accurately teleop- Performance: This task evaluates not only success rate but erate to rotate the carton while preventing it from falling. a) Press and Move Box
b) Pick and Place Teddy Bear
Fig. 10: The imitation learning experiment. (a) The robot must first locate the correct position of the box and then apply adequate force to press it. Excessive force prevents movement, while insufficient force causes the fingers to slip. (b) The robot must first locate the bear, then open its hand to grasp it. Due to the bear’s size, precise grasping control is required. An inaccurate grasp deforms the bear and causes it to slip out of the fingertips.
• Visual Obstacle: Grasping the carton is hindered by visual to the teddy bear slipping out of the robotic hand when not obstacles, as the operator cannot see the contact points grasped firmly. between the robotic hand’s fingers and the carton. Rotating and Placing the Carton. This task follows the same Performance: Table III shows that both haptic and force setup as Section VI-C. For this contact-rich task, we use 3 feedback significantly improve the teleoperation success rate human-collected demonstrations to train the policy. Across 10 and reduce completion time. While force feedback alone trials, the success rate is 90% (9/10). results in a comparable average completion time, haptic force VII. L IMITATIONS AND F UTURE W ORK feedback achieves a higher success rate. The vision-based Mo- Cap method AnyTeleop [25] struggles with in-hand rotation DOGlove is a powerful haptic force feedback glove for in this task. dexterous manipulation, but several limitations remain. First, the weight of DOGlove is inevitably high, as it utilizes 5 com- D. Imitation Learning mercial servos, bringing the total weight to 550g. Additionally, in agile teleoperation scenarios, performance is constrained We show DOGlove is capable of collecting high-quality by the servos’ maximum speed and torque output. To address demonstrations. 3D Diffusion Policy (DP3) [45] is selected these issues, we are investigating the use of lighter servos with as our imitation learning algorithm, and we use Realsense smaller reduction ratios and designing a customized reduction L515 to acquire the point cloud inputs, which are then down- mechanism to balance speed and torque more effectively. sampled to 1024 points using farthest point sampling [24]. The Second, although DOGlove is designed to accommodate most data collected by DOGlove is used to train policies for various hand sizes, it may be uncomfortable for some users. To downstream tasks. We evaluate imitation learning performance enhance adaptability and wearability, we are developing CAD on 2 basic contact-rich tasks and 1 long-horizon task: files for linkages in multiple sizes, enabling customization for Press and Move Box: As shown in Fig 10a, the robot must various hand dimensions. continuously press down on a box and move it to a specified target location. During data collection, the box is randomly VIII. C ONCLUSION placed within a 30×20 cm area, and DOGlove collects 40 In this paper, we present DOGlove, a low-cost, open- demonstrations to train the policy. In evaluation, the box is source haptic force feedback glove designed for dexterous also randomly placed in the same area. Across 20 trials, the manipulation. DOGlove enables precise and efficient execution success rate is 85% (17/20). of long-horizon, contact-rich tasks. Experimental results show Pick and Place Teddy Bear: As shown in Fig 10b, the that DOGlove enhances the operator’s immersive teleoperation robot must grasp a teddy bear and place it into a designated experience while also serving as an effective tool for training box. During data collection, the teddy bear’s initial position is imitation learning policies. Moreover, the user study demon- randomized within a 30×20 cm area, and DOGlove collects strates that DOGlove provides precise perception of object 40 demonstrations to train the policy. In evaluation, the bear properties through its integrated haptic force feedback. To is again randomly placed in the same area. Across 20 trials, support further research and contribute to the community, all the success rate is 70% (14/20), with failures primarily due hardware designs and code will be open-sourced. ACKNOWLEDGMENT Proceedings of Robotics: Science and Systems (RSS), We would like to thank Zhengrong Xue, Gu Zhang, Changyi 2024. Lin, Mengda Xu, and Yifan Hou for their invaluable advice [9] Dexta Robotics. Dexta robotics official website. https: and fruitful discussions on hardware design and learning //www.dextarobotics.com/, 2025. policies. We also appreciate Wenhao Ding and Laixi Shi [10] Runyu Ding, Yuzhe Qin, Jiyue Zhu, Chengzhe Jia, for their insightful discussions and feedback. Additionally, Shiqi Yang, Ruihan Yang, Xiaojuan Qi, and Xiaolong we thank Yichuan Gao, Xiaoyan Yang, Xinyao Qin, and Wang. Bunny-visionpro: Real-time bimanual dexterous Botian Xu for their assistance with the user study. Special teleoperation for imitation learning. arXiv preprint thanks to Skyentific, Gennady Plyushchev, for their innovative arXiv:2407.03162, 2024. contributions to the unconventional cable-driven joint design. [11] Zicong Fan, Omid Taheri, Dimitrios Tzionas, We are also grateful to Tiansheng Sun and Guanghan Pan for Muhammed Kocabas, Manuel Kaufmann, Michael J their open-source repository of the HTC Vive Tracker Python Black, and Otmar Hilliges. Arctic: A dataset for API. Lastly, we extend our appreciation to Yitong Wang for dexterous bimanual hand-object manipulation. In her help in creating elegant graphic renderings of the hardware Proceedings of the IEEE/CVF Conference on Computer design. Vision and Pattern Recognition, pages 12943–12954, 2023. R EFERENCES [12] Hongjie Fang, Hao-Shu Fang, Yiming Wang, Jieji Ren, [1] Jorge Aldaco, Travis Armstrong, Robert Baruch, Jingjing Chen, Ruo Zhang, Weiming Wang, and Cewu Jeff Bingham, Sanky Chan, Kenneth Draper, De- Lu. Airexo: Low-cost exoskeletons for learning whole- bidatta Dwibedi, Chelsea Finn, Pete Florence, Spencer arm manipulation in the wild. In 2024 IEEE International Goodrich, et al. Aloha 2: An enhanced low-cost Conference on Robotics and Automation (ICRA), pages hardware for bimanual teleoperation. arXiv preprint 15031–15038. IEEE, 2024. arXiv:2405.02292, 2024. [13] Zipeng Fu, Tony Z. Zhao, and Chelsea Finn. Mobile [2] Homanga Bharadhwaj, Abhinav Gupta, Vikash Kumar, aloha: Learning bimanual mobile manipulation with low- and Shubham Tulsiani. Towards generalizable zero-shot cost whole-body teleoperation. In Conference on Robot manipulation via translating human interaction plans. In Learning (CoRL), 2024. 2024 IEEE International Conference on Robotics and [14] HaptX. Haptx official website. https://haptx.com/, 2025. Automation (ICRA), pages 6904–6911. IEEE, 2024. [15] Aadhithya Iyer, Zhuoran Peng, Yinlong Dai, Irmak [3] Ilaria Cerulo, Fanny Ficuciello, Vincenzo Lippiello, and Guzey, Siddhant Haldar, Soumith Chintala, and Lerrel Bruno Siciliano. Teleoperation of the schunk s5fh under- Pinto. OPEN TEACH: A versatile teleoperation system actuated anthropomorphic hand using human hand mo- for robotic manipulation. In 8th Annual Conference on tion tracking. Robotics and Autonomous Systems, 89: Robot Learning, 2024. 75–84, 2017. [16] Zhanat Kappassov, Juan-Antonio Corrales, and [4] Pietro Cerveri, Elena De Momi, N Lopomo, Gabriel Véronique Perdereau. Tactile sensing in dexterous Baud-Bovy, RML Barros, and Giancarlo Ferrigno. Finger robot hands. Robotics and Autonomous Systems, 74: kinematic modeling and real-time hand motion estima- 195–220, 2015. tion. Annals of biomedical engineering, 35:1989–2002, [17] Uikyum Kim, Heeyeon Jeong, Hyunmin Do, Jongwoo 2007. Park, and Chanhun Park. Six-axis force/torque finger- [5] Claire Chen, Zhongchun Yu, Hojung Choi, Mark tip sensor for an anthropomorphic robot hand. IEEE Cutkosky, and Jeannette Bohg. Dexforce: Extract- Robotics and Automation Letters, 5(4):5566–5572, 2020. ing force-informed actions from kinesthetic demon- [18] Changyi Lin, Han Zhang, Jikai Xu, Lei Wu, and Huazhe strations for dexterous manipulation. arXiv preprint Xu. 9dtact: A compact vision-based tactile sensor for arXiv:2501.10356, 2025. accurate 3d shape reconstruction and generalizable 6d [6] Xuxin Cheng, Jialong Li, Shiqi Yang, Ge Yang, and force estimation. IEEE Robotics and Automation Letters, Xiaolong Wang. Open-television: Teleoperation with im- 2023. mersive active visual feedback. In 8th Annual Conference [19] Hangxin Liu, Xu Xie, Matt Millar, Mark Edmonds, Feng on Robot Learning, 2024. Gao, Yixin Zhu, Veronica J Santos, Brandon Rothrock, [7] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, and Song-Chun Zhu. A glove-based system for study- Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran ing hand-object manipulation via joint pose and force Song. Diffusion policy: Visuomotor policy learning via sensing. In 2017 IEEE/RSJ International Conference on action diffusion. The International Journal of Robotics Intelligent Robots and Systems (IROS), pages 6617–6624. Research, page 02783649241273668, 2023. IEEE, 2017. [8] Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, [20] Hangxin Liu, Zhenliang Zhang, Xu Xie, Yixin Zhu, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Yue Liu, Yongtian Wang, and Song-Chun Zhu. High- Shuran Song. Universal manipulation interface: In- fidelity grasping in virtual reality using a glove-based the-wild robot teaching without in-the-wild robots. In system. In 2019 international conference on robotics and automation (icra), pages 5180–5186. IEEE, 2019. loop grasping from low-cost demonstrations. IEEE [21] Manus Meta. Manus meta robotics official website. https: Robotics and Automation Letters, 5(3):4978–4985, 2020. //www.manus-meta.com/robotics, 2025. [34] Omid Taheri, Nima Ghorbani, Michael J Black, and [22] Malte Mosbach, Kara Moraw, and Sven Behnke. Accel- Dimitrios Tzionas. Grab: A dataset of whole-body human erating interactive human-like manipulation learning with grasping of objects. In Computer Vision–ECCV 2020: gpu-based simulation and high-quality demonstrations. 16th European Conference, Glasgow, UK, August 23–28, In 2022 IEEE-RAS 21st International Conference on 2020, Proceedings, Part IV 16, pages 581–600. Springer, Humanoid Robots (Humanoids), pages 435–441. IEEE, 2020. 2022. [35] Ultraleap. Ultraleap official website. https://www. [23] OptiTrack. OptiTrack Motion Capture Systems. ultraleap.com/, 2025. https://www.optitrack.com/, 2025. [36] Vicon Motion Systems. Vicon Motion Capture Systems. [24] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J https://www.vicon.com/, 2025. Guibas. Pointnet: Deep learning on point sets for 3d [37] Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, classification and segmentation. In Proceedings of the Li Fei-Fei, Danfei Xu, Yuke Zhu, and Anima Anand- IEEE conference on computer vision and pattern recog- kumar. Mimicplay: Long-horizon imitation learning by nition, pages 652–660, 2017. watching human play. In 7th Annual Conference on [25] Yuzhe Qin, Wei Yang, Binghao Huang, Karl Van Wyk, Robot Learning, 2023. Hao Su, Xiaolong Wang, Yu-Wei Chao, and Dieter Fox. [38] Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Anyteleop: A general vision-based dexterous robot arm- Zhang, Li Fei-Fei, and C Karen Liu. Dexcap: Scalable hand teleoperation system. In Proceedings of Robotics: and portable mocap data collection system for dexterous Science and Systems (RSS), 2023. manipulation. In Proceedings of Robotics: Science and [26] Felipe Sanches, Geng Gao, Nathan Elangovan, Ricardo V Systems (RSS), 2024. Godoy, Jayden Chapman, Ke Wang, Patrick Jarvis, and [39] Han Xu, Mingqi Chen, Gaofeng Li, Lei Wei, Shichi Minas Liarokapis. Scalable. intuitive human to robot Peng, Haoliang Xu, and Qiang Li. An immersive virtual skill transfer with wearable human machine interfaces: reality bimanual telerobotic system with haptic feedback. On complex, dexterous tasks. In 2023 IEEE/RSJ Inter- arXiv preprint arXiv:2501.00822, 2025. national Conference on Intelligent Robots and Systems [40] Mengda Xu, Zhenjia Xu, Cheng Chi, Manuela Veloso, (IROS), pages 6318–6325. IEEE, 2023. and Shuran Song. XSkill: Cross embodiment skill [27] Gaspare Santaera, Emanuele Luberto, Alessandro Serio, discovery. In 7th Annual Conference on Robot Learning, Marco Gabiccini, and Antonio Bicchi. Low-cost, fast and 2023. accurate reconstruction of robotic and human postures [41] Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, via imu measurements. In 2015 IEEE International Gordon Wetzstein, Manuela Veloso, and Shuran Song. Conference on Robotics and Automation (ICRA), pages Flow as the cross-domain manipulation interface. In 8th 2728–2735. IEEE, 2015. Annual Conference on Robot Learning, 2024. [28] Max Schwarz, Christian Lenz, Andre Rochow, Michael [42] Jingyun Yang, Junwu Zhang, Connor Settle, Akshara Rai, Schreiber, and Sven Behnke. Nimbro avatar: Interactive Rika Antonova, and Jeannette Bohg. Learning periodic immersive telepresence with force-feedback telemanipu- tasks from human demonstrations. In 2022 International lation. In 2021 IEEE/RSJ International Conference on Conference on Robotics and Automation (ICRA), pages Intelligent Robots and Systems (IROS), pages 5312–5319. 8658–8665. IEEE, 2022. IEEE, 2021. [43] Wenzhen Yuan, Siyuan Dong, and Edward H Adelson. [29] SenseGlove. Senseglove official website. https://www. Gelsight: High-resolution robot tactile sensors for esti- senseglove.com/, 2025. mating geometry and force. Sensors, 17(12):2762, 2017. [30] Kenneth Shaw, Ananye Agarwal, and Deepak Pathak. [44] Kevin Zakka. Mink: Python inverse kinematics based on Leap hand: Low-cost, efficient, and anthropomorphic MuJoCo, July 2024. URL https://github.com/kevinzakka/ hand for robot learning. Robotics: Science and Systems mink. Software. (RSS), 2023. [45] Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, [31] Kenneth Shaw, Yulong Li, Jiahui Yang, Mohan Kumar Muhan Wang, and Huazhe Xu. 3d diffusion policy: Srirama, Ray Liu, Haoyu Xiong, Russell Mendonca, and Generalizable visuomotor policy learning via simple 3d Deepak Pathak. Bimanual dexterity for complex tasks. representations. In Proceedings of Robotics: Science and In 8th Annual Conference on Robot Learning, 2024. Systems (RSS), 2024. [32] Aravind Sivakumar, Kenneth Shaw, and Deepak Pathak. [46] Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Robotic telekinesis: Learning a robotic hand imitator by Finn. Learning fine-grained bimanual manipulation with watching humans on youtube. In Robotics: Science and low-cost hardware. In Proceedings of Robotics: Science Systems, 2022. and Systems (RSS), 2023. [33] Shuran Song, Andy Zeng, Johnny Lee, and Thomas [47] Tony Z Zhao, Jonathan Tompson, Danny Driess, Pete Funkhouser. Grasping in the wild: Learning 6dof closed- Florence, Seyed Kamyar Seyed Ghasemipour, Chelsea Finn, and Ayzaan Wahid. Aloha unleashed: A simple recipe for robot dexterity. In 8th Annual Conference on Robot Learning, 2024. [48] Wenping Zhao, Jinxiang Chai, and Ying-Qing Xu. Com- bining marker-based mocap and rgb-d camera for acquir- ing high-fidelity hand motion data. In Proceedings of the ACM SIGGRAPH/eurographics symposium on computer animation, pages 33–42, 2012. [49] Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. Freihand: A dataset for markerless capture of hand pose and shape from single rgb images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 813–822, 2019.