OpenTouch: Bringing Full-Hand Touch to Real-World Interaction
Authors: Yuxin Ray Song, Jinzhou Li, Rao Fu, Devin Murphy, Kaichen Zhou, Rishi Shiv, Yaqi Li, Haoyu Xiong, Crystal E. Owens, Yilun Du, Yiyue Luo, Xianyi Cheng, Antonio Torralba, Wojciech Matusik, Paul Pu Liang
Affiliations: MIT, Duke University, Brown University, University of Washington, Harvard University
arXiv: 2512.16842v1
Abstract
The research presents the first in-the-wild egocentric full-hand tactile dataset, containing 5.1 hours of synchronized video-touch-pose data and 2,900 curated clips with detailed text annotations. The team developed wearable sensing hardware combining a flexible printed circuit-based (FPC) tactile glove with hand-tracking and video capture systems. They established benchmarks for cross-sensory retrieval and tactile classification, demonstrating that multimodal inputs offer complementary information that reduces retrieval ambiguity, and that tactile signals effectively support grasp recognition.
Hardware Innovation
The custom FPC-based tactile glove features 169 taxels that uniformly cover the fingers and palmar surface, offering low-cost, reproducible tactile sensing superior to traditional conductive textile approaches.
Key Findings
Multimodal Advantage: Combining video, pose, and tactile data yields substantially higher performance than individual modalities. Video provides global scene context, pose encodes kinematics, and tactile captures local contact and force.
Tactile Encoder Efficiency: Lightweight CNN encoders outperform ResNet-18 backbones for tactile processing, suggesting tactile signals — unlike natural images — are sparse and highly structured, benefiting from compact architectures.
Cross-Dataset Application: Models trained on OpenTouch successfully retrieve relevant tactile patterns from Ego4D video queries, indicating generalization potential.
Dataset Composition
- 5.1 hours of synchronized video-touch-pose data
- 2,900 curated clips with detailed text annotations
- Collected in-the-wild with egocentric perspective
- Benchmarks: cross-sensory retrieval and tactile classification tasks