Open-TeleVision: Teleoperation with Immersive Active Visual Feedback
Authors: Xuxin Cheng, Jialong Li, Shiqi Yang, Ge Yang, Xiaolong Wang Submitted: July 2024 Venue: arXiv 2407.01512
Abstract
An immersive teleoperation system that allows operators to actively perceive a robot’s surroundings in a stereoscopic manner. The system replicates the operator’s arm and hand movements onto the robot — “the operator’s mind is transmitted to a robot embodiment.”
System Design
- Stereoscopic cameras on robot head stream to VR headset → operator sees from robot’s perspective
- Arm/hand movement captured from operator → retargeted to robot joints
- Active visual feedback: operator can control robot head orientation for better viewpoint
- Works across different humanoid robot platforms
Validation
Four complex tasks evaluated:
- Can Sorting
- Can Insertion
- Folding
- Unloading
Tested on two different humanoid robots. Data collected via this system used to train imitation learning policies with strong results.
Fully open-source.