本文由 AI 分析生成
建立時間： 2026-05-28 來源： https://arxiv.org/html/2512.16842v1
Summary
OpenTouch introduces the first in-the-wild egocentric full-hand tactile dataset, comprising 5.1 hours of synchronized video, touch, and hand-pose data with 2,900 annotated clips. The system pairs a custom FPC-based tactile glove (169 taxels covering the full hand) with egocentric video and hand tracking, enabling benchmarks for cross-sensory retrieval and grasp classification. Results show multimodal fusion of vision, pose, and tactile consistently outperforms any single modality, and lightweight CNN encoders outperform ResNet backbones for sparse tactile signals.
OpenTouch 發布首個真實場景下的全手觸覺資料集，包含 5.1 小時同步的視頻、觸覺與手部姿態數據及 2,900 個帶文字標注的片段。研究自製基於 FPC 的觸覺手套（169 個觸感元件覆蓋整手），建立跨感測模態檢索與抓握分類基準。實驗結果顯示多模態融合穩定優於單一模態，輕量 CNN 編碼器在稀疏觸覺信號上表現優於 ResNet。
Prerequisites
- Tactile sensing fundamentals — familiarity with taxel arrays, capacitive/resistive sensing, and why spatial coverage of the full hand matters for dexterous tasks.
- Egocentric video datasets (Ego4D) — the paper evaluates cross-dataset generalization against Ego4D; understanding that benchmark’s scope helps interpret the retrieval results.
- Multimodal contrastive learning — the cross-sensory retrieval benchmark uses contrastive embedding alignment (similar to CLIP); understanding how joint embedding spaces are trained and evaluated with Recall@K is necessary.
Core Idea
The central claim is that full-hand touch is a complementary sensing modality that reduces retrieval ambiguity when combined with egocentric video and hand pose. The FPC glove design is the enabling contribution: prior wearable tactile sensors either used conductive textiles (low spatial resolution, inconsistent manufacturing) or rigid sensor arrays (poor finger coverage). The 169-taxel FPC glove achieves uniform palmar and finger coverage at low cost with high reproducibility. The dataset collection methodology — in-the-wild rather than lab-constrained — is intentional: it captures the contact diversity needed to train generalizable encoders. The architectural finding (lightweight CNN > ResNet for tactile) is explained by the structural sparsity of tactile signals: unlike natural images, tactile maps have localized, high-contrast activation patterns that small kernels capture better than deep residual networks.
Results
| Modality Combination | Cross-Sensory Retrieval R@1 | Grasp Classification Acc. |
|---|---|---|
| Vision only | baseline | baseline |
| Pose only | lower than vision | competitive |
| Tactile only | lower than vision | strong |
| Vision + Pose | +Δ over vision | +Δ |
| Vision + Pose + Tactile | best | best |
- 5.1 hours, 2,900 annotated clips collected in-the-wild.
- Lightweight CNN encoder outperforms ResNet-18 on tactile classification.
- Cross-dataset: OpenTouch-trained models successfully retrieve tactile patterns from Ego4D video queries.
(Note: exact delta values not reported in the fetched content — refer to paper Table 2 for precise numbers.)
Limitations
- Author-stated: the dataset is egocentric and wearable-sensor-constrained; does not cover bimanual tasks or tool-use scenarios explicitly.
- Unstated: the 5.1-hour scale is modest compared to large-scale video datasets; generalization to non-hand robotic grippers is unclear. The FPC glove requires custom fabrication — reproducibility for external labs depends on manufacturing access. Publication date is approximate (arXiv 2512.x = December 2025).
Reproducibility
- Code: not confirmed from fetched content; check project page linked from arXiv abstract.
- Dataset: OpenTouch dataset released (5.1h video-touch-pose); availability confirmed in abstract.
- Compute: lightweight CNN encoders — low compute requirement implied; exact GPU specs not reported in fetched content.
Insights
The key insight for robot learning is that full-hand tactile data collected in-the-wild (not in constrained lab setups) is necessary for building generalizable contact encoders. The cross-dataset retrieval result — OpenTouch models generalizing to Ego4D — is the strongest evidence of practical value: it suggests the tactile representations learned are not dataset-specific artifacts. For dexterous manipulation research, this dataset fills a gap between fingertip-only tactile sensors (common in robotics) and vision-only egocentric datasets (Ego4D, EPIC-Kitchens). The lightweight CNN finding is a useful architectural prior: before reaching for ResNet or ViT backbones for tactile processing, compact CNNs should be the default baseline.
Connections
Raw Excerpt
The first in-the-wild egocentric full-hand tactile dataset, containing 5.1 hours of synchronized video-touch-pose data and 2,900 curated clips with detailed text annotations. Video provides global scene context, pose encodes kinematics, and tactile captures local contact and force.