AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System

Authors: Yuzhe Qin, Wei Yang, Binghao Huang, Karl Van Wyk, Hao Su, Xiaolong Wang, Yu-Wei Chao, Dieter Fox Submitted: July 2023 (v3: May 2024) Venue: arXiv 2307.04577

Abstract

A unified vision-based teleoperation system that supports multiple different robot arms, hands, simulation environments, and camera configurations within a single system — addressing the limitation that prior systems were engineered for specific hardware.

Key Features

• Vision-only hand tracking: no wearable gloves or EMF sensors required
• Cross-platform: works with multiple arm+hand combinations without re-engineering
• Sim + real: unified interface for both simulation training and real-world deployment
• Single camera setup sufficient for hand pose estimation

Results

• Outperforms hardware-specific teleoperation systems on dexterous manipulation task success rates
• Better imitation learning policy outcomes in simulation vs hardware-specific predecessors
• Enables rapid transfer to new robot configurations

Significance

Removes the major friction point of hardware-specific teleoperation: any researcher with a camera can use it for dexterous hand data collection, regardless of their robot platform.