Hoi! - A Multimodal Dataset for Force-Grounded, Cross-View Articulated
Manipulation
Tim Engelbracht1 René Zurbrügg1 Matteo Wohlrapp2 Martin Büchner3
Abhinav Valada3 Marc Pollefeys1,4 Hermann Blum5 Zuria Bauer1
1 2
ETH Zurich Technical University of Munich 3 University of Freiburg
4
Microsoft 5 University of Bonn
arXiv:2512.04884v2 [cs.RO] 10 Feb 2026
Corresponding author: tengelbracht@ethz.ch
Figure 1. Overview of the Hoi! Dataset: A multimodal dataset for force-grounded, cross-view articulated manipulation in wild indoor
environments. The dataset captures human interactions with common articulated objects (drawers, doors, fridges, dishwashers) with
synchronized RGB, depth, force, tactile sensing, and multi-view videos from egocentric and exocentric perspectives. Each interaction is
annotated with articulation parameters (e.g., opening angles, displacements, peak forces), supporting research on multimodal perception,
manipulation learning, and embodied reasoning.
Abstract ing from video, enabling researchers to evaluate how well
methods transfer between human and robotic viewpoints,
but also investigate underexplored modalities such as force
We present a dataset for force-grounded, cross-view artic- sensing and prediction. Further information can be found
ulated manipulation that couples what is seen with what is on the Website.
done and what is felt during real human interaction. The
dataset contains 3048 sequences across 381 articulated ob- 1. Introduction
jects in 38 environments. Each object is operated under
four embodiments - (i) human hand, (ii) human hand with a In recent years, computer vision has moved from purely
wrist-mounted camera, (iii) handheld UMI gripper, and (iv) perceptive domains to a dynamic and interactive era, with
a custom Hoi! gripper - where the tool embodiment pro- research increasingly aiming to interpret how objects can
vide synchronized end-effector forces and tactile sensing. be used, moved or interacted with. Progress in this direc-
Our dataset offers a holistic view of interaction understand- tion has been largely fueled by data-driven methods and the
availability of large-scale datasets such as [7, 17, 18, 37] sequences over 381 objects and 38 environments, span- capturing diverse human and object interactions. Such ning four main embodiments. Each sequence provides datasets enable learning-based models to generalize across time-synchronized vision, pose, force, and haptic streams different tasks and environments, thereby moving toward together with precise scene-level ground truth. generalizable, zero-shot actionable understanding. A closer • Interaction-centric annotations supporting benchmarks examination, however, reveals a fundamental discrepancy in in force-from-vision, articulation estimation, cross-view the nature of interactions studied across different research transfer, and state-change prediction. domains. While human–centric video datasets emphasize long-horizon activities such as cooking, furniture assem- 2. Related Work bly, and sports, robotics datasets predominantly target short- horizon primitives like pick-and-place, wiping, or drawer In the following, we analyze previous efforts in articulation opening. This data gap makes it hard to investigate interest- and video understanding and delineate how our Hoi! dataset ing transfer questions: Do interaction-force predictors gen- connects the fields of physical reasoning and perceptive in- eralize to humans videos? Do articulation-tracking methods teraction understanding. remain effective from an exocentric robotic viewpoint? Can Articulation Understanding. Early works modeled ar- interactions demonstrated by a human hand be re-targeted ticulations as kinematic graphs and estimated joint pa- to a two-finger robotic gripper? Among the many forms rameters [45, 46], but relied on known coordinate frames of interaction, articulated furniture provides an especially in controlled settings. Later efforts introduced large- rich yet understudied case. Despite its prevalence in ev- scale simulated environments [33, 50] and real-world part eryday human activity, curated video data of people inter- databases [29]. However, these resources either remain acting with furniture is sparse. On the robotics side, ar- fully simulated—limiting transfer to real manipulation—or ticulated furniture manipulation is emerging as a tractable provide object models without any interaction data. Going challenge, with perception identified as the main obstacle to beyond pure visual perception, datasets such as RBO [32] further progress [19]. Existing articulation datasets [9, 20] provide real RGB-D sequences of humans manipulating ar- offer valuable labels but are largely constructed from static ticulated objects and include limited force measurements, scans, lacking the paired motion data needed to ground but remain small in scale and lack multi-view or multi- these annotations in real interactions. To bridge this gap, we embodiment coverage. Learning-based approaches like introduce a dataset for force-grounded, cross-view, multi- FlowBot3D [11] estimate dense articulation flow fields for embodiment manipulation of articulated furniture. Each planning, yet rely largely on simulated training data. Com- object is operated under four embodiments: (i) a human plementary work on force prediction, such as object-centric hand, (ii) a human hand with a wrist-mounted camera, (iii) models of everyday forces [23] and text-guided force rea- a handheld UMI gripper, and (iv) a custom Hoi! gripper soning for mobile manipulation [6], demonstrates the value equipped with tactile sensors and a force-torque sensor. To of modeling contact-force distributions for articulated tasks. anchor perception and interaction to high-fidelity scene ge- Vision-only methods [21, 44] and digital-twin approaches ometry, we scan each environment before and after manip- such as Ditto [24] or ArtGS [31] focus on reconstruct- ulation with a Leica RTC360 laser scanner, yielding dense ing articulated objects from video, but omit force or tac- 3D point clouds that capture both static structure and geom- tile interaction. In-the-wild articulation estimation has also etry changes induced by the interaction. We augment the emerged, with ArtiPoint [49] inferring articulations from raw data with interaction annotations, including sequence- egocentric RGB-D. However, key gaps remain, as summa- level state-change labels (e.g., open/close progression), ob- rized in Tab. 1: (1) few public datasets combine artic- ject metadata (category, articulation type), and joint param- ulated object manipulation with force/tactile sensing, (2) eters needed for kinematic reasoning. This combination even fewer cover multiple viewpoints, e.g., ego, exo, wrist: of forces, egocentric cues, calibrated third-person context, and (3) almost none provide aligned recordings of both hu- and scene-level ground truth provides a unified substrate for man and robotic embodiments performing the same artic- studying the multimodality and entire sensor range of in- ulated interactions across these modalities. As a conse- teraction on everyday articulated objects. In summary, our quence, it is difficult to systematically study how an action’s contributions are: visual appearance relates to its physical patterns, or how human-demonstrated skills transfer to robot embodiments. • A data capture pipeline for human demonstrations in mo- Our proposed dataset “Hoi!” is explicitly designed to close bile manipulation that couples multi-view RGB-D obser- this gap by jointly capturing vision, force, and tactile signals vations with end-effector forces and tactile signals, en- from humans and robots interacting with the same articu- abled by a novel hand-held robotic gripper for recording lated objects, thereby enabling research that tightly links in-the-wild interaction forces. visual perception to haptic action and facilitates effective • A multi-view dataset for articulated manipulation: 3048 transfer of manipulation skills from human to robot. Table 1. Commonly used datasets for human interactions and articulated environments. Views: Numbers indicate available camera viewpoints; columns show Egocentric (Ego), Exocentric (Exo), and Wrist-mounted (Wrist). Embodiments: Columns show Human (H), Robot (R), and Tool/Gripper (T). Modalities: RGB, Depth, Force/Torque, Haptic/Tactile, Hand Tracking, Audio, 3 Joint States, Eyetracking, 3D Model / Digital Twin, Language. Dataset Class Tasks/Objects Environment Views Embodiments Modalities Ego Exo Wrist H R T ∼1 h of interactions, RBO [32] articulation ✗ 1×✓ ✗ ✓ ✗ ✗ articulated meshes 14 articulated objects pick & place, toy playing RH20T [13] ∼916 h, 110k demos tabletop ✗ 6×✓ ✓ ✓ 4×✓ ✗ 3 very few articulations EgoExo4D [18] cooking, assembly, sports 1286 h, 690 actions 123 scenes ✓ 4×✓ ✗ ✓ ✗ ✗ ForceMimic [30] cooking 30 k zucchini peeling sequences tabletop ✗ 1×✓ ✗ ✓ ✓ ✓ ArticuBot [47] articulation 322 articulated parts, 42k demos simulation Multi. ✗ ✓ ✗ sim assets 3 Kaiwu [41] assembly 40 h, 30 objects and 11 k interactions tabletop ✗ 1×✓ ✗ ✓ ✗ ✗ haptic glove EpicKitchens [7] cooking 100 h of cooking activities 48 kitchens ✓ ✗ ✗ ✓ ✗ ✗ articulation, pick & place, DROID [27] 188 h teleoperation 52 buildings ✗ 2×✓ ✓ ✗ 3×✓ ✗ 3 cleaning Arti4D [48] articulation 1 h of interacting with 85 objects 4 scenes ✓ ✗ ✗ ✓ ✗ ✗ reconstruction AgiBot World [1] pick & place ∼16 kh teleoperation 106 scenes ✓ ✗ ✓ ✗ ✓ ✗ visuo-tactile 3 OpenFunGraph [54] articulation 0.5 h of interaction with 201 elements 14 scenes ✓ ✗ ✗ ✓ ✗ ✗ 3D scans HDEpic [37] cooking 41 h of cooking activities 9 kitchens ✓ ✗ ✗ ✓ ✗ ✗ digital twin 48 h of interactions, digit tactile Hoi! (ours) articulation 38 scenes ✓ 2×✓ ✓ ✓ (✓) 2×✓ 381 articulated parts 3gripper only 3D scans
Figure 2. Locations of the Hoi! dataset. A diverse collection of real-world indoor environments featuring kitchens, bathrooms, offices, and living spaces, were each has RGB-D sequences, GT, panoramic images, and various articulated objects that have interactions with multiple grippers and users.
Video Understanding. Egocentric benchmarks such as et al. [25] accelerates imitation learning through easier-to- [7, 17, 18, 37] have enabled progress on action recog- collect human demonstrations. These approaches illustrate nition, anticipation, and activity understanding in every- the growing trend of leveraging human video data to teach day environments. However, while they excel at depict- robots, effectively narrowing the gap between computer vi- ing ”what happened” in videos, they offer no information sion and robotics. However, a crucial limitation is their lim- on the forces applied or contact feedback, making it hard ited generalization: Most of these works are restricted to to translate insights to the physical realm. In robotics, re- tabletop settings and a low domain gap between demon- cent efforts have leveraged large video collections to im- strations and deployment. Partially, this is because they prove policy learning. Nair et al. [34] learn universal visual strongly rely on visual inputs and omit the multimodal rich- representations from egocentric video, substantially accel- ness (like forces and tactile cues) that is central to physical erating downstream manipulation learning. This demon- manipulation. ForceMimic [30] and RH20T [13] demon- strates that representations learned from human video can strated that incorporating multimodal signals, particularly accelerate robot policy learning. Similarly, [51] and [28] force measurements, can significantly improve robotic ma- derive robot policies directly from human video demonstra- nipulation performance. As summarized in Sec. 3, prior tions, eliminating the need for robot-collected data. Kareer video datasets provide wide semantic coverage of human Manipulation Gripper Cameras Additional Modalities Zed Camera Mode Wrist Wrist Force- Finger Motor Aria Glasses 2 × Exo Ego Wrist IMU Depth Torque Haptics Torque Force-Torque Hand only 5-finger ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ Sensor Hand + Wrist Cam 5-finger ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ UMI Gripper Antipodal ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ Hoi! Gripper (ours) Antipodal ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ (Spot) Antipodal ✓ ✗ ✓ ✗ ✓ ✗ ✗ ✗
Table 2. Recording setup. Each manipulation condition comprises several recording modules producing multiple time-aligned data streams.
activity, but they do not provide the integrated multimodal Dynamixel Digit Loadcell sensing or cross-embodiment correspondence needed for Tactile Sensors Motor grounded manipulation research. In particular, without data that couples “what is seen” with “what is done” across both Figure 3. Hoi! Gripper. The 2-finger parallel gripper is operated through the load cell, where the measured load is translated into gripping force. In- human and robotic embodiments, we cannot easily research teraction force and tactile contact pressure are measured through the Digit an action’s visual appearance correlated with the forces ap- and Force-Torque sensors respectively. Aria Glasses and a stereo camera plied or how a skill demonstrated by a person might transfer provide pose estimation and wrist-view observations. We will release the design as open source. to a robot’s morphology. Motivated by this shortcoming, we jointly capture vision, force, and tactile signals across both human and robotic executions of identical tasks to bridge data during interaction. The gripper features two oppos- this gap. By providing aligned multimodal recordings , ing GelSight Digit sensors for high-resolution tactile imag- the dataset aims to enable research that tightly links visual ing and an antipodal mechanism inspired by the ALOHA perception to haptic action, ultimately facilitating effective design [55]. The gripper is powered by a Dynamixel learning and transfer of manipulation skills across embodi- XM430-W350-T motor and features a Bota SensONE 6- ments. DoF force–torque sensor to measure interaction forces at the wrist. To support human-operated demonstrations, the assembly is mounted on a handheld “gripper-on-a-stick,” 3. Hoi! Dataset with gripping force modulated via a calibrated load cell We introduce a dataset for force-grounded, cross-view, and in the handle. A wrist-mounted ZED Mini stereo camera cross-embodiment articulated manipulation. The dataset and Project Aria device provide pose tracking and RGB-D captures humans operating everyday articulated objects in wrist observations. This setup allows us to couple visual realistic furnished rooms, with each interaction performed observations with precise force–torque and tactile signals, under four embodiments: (i) a human hand, (ii) a human providing a rich basis for investigating real articulated in- hand with a wrist-mounted camera, (iii) a handheld UMI [5] teractions. As illustrated in Fig. 4, force profiles vary sub- gripper, and (iv) a custom Hoi! gripper equipped with stantially across different articulated objects, highlighting force–torque and tactile sensing. We record a small subset how force signals offer complementary information beyond of the interactions using a teleoperated Spot robot, equipped visual cues. The full system runs on a battery-powered with body cameras and a wrist-mounted Aria. An overview NVIDIA Jetson Orin Nano carried in a backpack, enabling of the articulated parts is shown in Fig. 5. Across all em- fully mobile data collection. The sensor suite is fully cal- bodiments, interactions are recorded from multiple view- ibrated and the gripper is gravity-compensated. Detailed points: an egocentric camera (Project Aria [12]) providing specifications are provided in the supplementary. RGB, SLAM, eye gaze, and hand pose, and two static third- Data Collection. We record the dataset primarily in the person views captured with iPhone 13 Pro devices (RGB + exhibition area of a furniture store, with additional se- LiDAR depth). Tab. 2 summarizes the data streams under quences captured in a university lab and private apartments. each condition, and Fig. 7 shows all viewpoints. All modal- A team of seven human demonstrators performs all inter- ities are temporally and spatially aligned, enabling direct actions, each under the four manipulation conditions de- comparison of how the same articulated object is operated scribed above and in Tab. 2. To ensure consistent capture across different embodiments. To anchor these interactions and simplify post-processing, we follow a structured col- in accurate geometry, we additionally capture before/after lection protocol. Each recording session covers between 3 high-resolution 3D scans of each environment using a Leica and 11 articulated parts using a single manipulation condi- RTC360 laser scanner. This fully aligned and multimodal tion (e.g., hand only). Once all devices start recording, we setup creates a unique foundation for studying how visual present a dynamic QR code encoding the current timestamp observations relate to physical interaction forces across hu- to every video stream. We then mark the start of each indi- man and robotic embodiments. vidual interaction by briefly showing a static QR code to one Hoi! Gripper. We design and open-source a custom exocentric camera. The operator proceeds through all artic- gripper (Fig. 3) to capture high-quality tactile and force ulated parts in alternating open/close order. For each envi- Table 3. Trajectory Evaluation Trajectory error of Aria-derived quanti- ties, evaluated against Qualisys motion-capture trajectories.
Metric Head Wrist Gripper
RMSE position [m] ↓ 0.005 0.005 0.006
RMSE rotation [rad] ↓ 0.016 0.012 0.012
Acc@1cm, 1◦ ↑ 0.810 0.886 0.810
Acc@5cm, 5◦ ↑ 0.998 1.000 1.000
Acc@10cm, 10◦ ↑ 1.000 1.000 1.000
all modalities on a shared timeline. For typical frame rates
of 30–60 Hz, this procedure results in a temporal alignment
accuracy of ca. 10–25 ms.
Pose Data. We use Aria Machine Perception Services
(MPS) to generate 6-DoF device poses as well as hand poses
and eye gaze data from the egocentric Aria video stream.
For the wrist camera and the Hoi! Gripper we also use the
attached Aria devices and extrinsic calibrations to pose all
sensors. For the UMI gripper, we follow their setup and use
Figure 4. Example of the measured interaction forces for several artic- ORB-SLAM3 [2] to track the gripper-mounted GoPro cam- ulated elements. Each curve corresponds to a different component (high- era and globally refine the trajectory using GTSAM [10]. lighted in matching colors below), illustrating how force magnitudes vary Because the third-person iPhone cameras remain static, we across types of articulated parts. can directly measure their poses in a common reference frame during spatial calibration, as described next. Spatial Alignment. We spatially align all recording de- vices into a common reference frame using visual localiza- tion against the high-resolution 3D scans. We first construct a reference set of dense 2D-3D correspondences from the point cloud and corresponding panoramic images captured with the scanner. We assume zero drift and global consis- tency of the per-device SLAM, which reduces the global registration problem to a single rigid 3D-3D alignment. We automatically select a set of high-quality keyframes for robust registration. We estimate the 6-DoF pose of each keyframe through hloc [43] with the 2D-3D database con- Figure 5. Distribution of environments and articulated interaction cat- structed from the Leica scan and robustly estimate a single egories in the Hoi! dataset. The bar chart depicts the relative frequency of rigid transformation Tquery world between each sensor trajectory human interactions across articulated categories, while the inset pie chart and the shared world frame. summarizes the proportion of environments involved in the interactions. Alignment Accuracy. We evaluate trajectory accuracy using a Qualisys motion capture (mocap) system. Mocap ronment, we also capture high-resolution 3D point clouds markers attached to the Aria frame provide the ground- with a Leica RTC360 laser scanner. We first scan the unar- truth trajectory, while the Aria MPS gives the correspond- ticulated scene, followed by scans in which as many artic- ing device trajectory. The rigid transform between the mo- ulated parts as possible are opened without occluding each cap body frame and the Aria device frame is obtained via other, resulting in 2-5 pointclouds per location. These scans hand–eye calibration [15]. Using this calibration, we ex- provide ground-truth geometry for all environments and ar- press both trajectories in a common frame, align them and ticulations and serve as a shared reference frame for spatial directly compare them for evaluation. Trajectory errors are calibration across devices. listed in Tab. 3. Time Alignment. Since the different recording modules Annotations. For each articulated object, we gather all run on independent internal clocks, we align all streams individual interaction recordings based on automatic video in post-processing. During recording, we display a QR cutting with the QR code as described above. We then man- code encoding the current Unix timestamp at 25 Hz into ually verify using a light-weight annotation tool. In addi- each camera stream i. Detecting and decoding this QR tion, we use the tool of [49] to annotate the articulation type code yields a time offset for each video stream with re- (prismatic or revolute) and axis. We further extend spect to a common reference time, allowing us to express the annotation tool to add a 3D mask of the object part and a Figure 6. Overview of the dataset collection setup. The dataset consists of 3048 multimodal sequences capturing human interactions with 381 articulated objects across 38 locations using multiple viewpoints (egocentric and third-person cameras) and manipulation conditions (hand, gripper-based). Ground truth data includes trajectories, contacts, haptic feedback, force measurements, and high-resolution 3D point clouds of each environment.
estimating their kinematics) in Sec. 4.1, tactile force estima-
tion (inferring contact forces from touch) in Sec. 4.2, and vi-
sual force estimation (predicting the force needed to achieve
a desired interaction from visual input) in Sec. 4.3.
4.1. In-the-Wild Articulated Object Estimation
We consider articulation understanding as a key prerequi-
site for manipulation: before an agent can reason about
the forces required to act on an object, it must first infer
Figure 7. Viewpoint recordings. Recorded viewpoints for articulated part how that its motion is governed. Our dataset enables this interactions. Each row corresponds to a different setup, showing synchro- by providing posed RGB observations (egocentric and exo- nized exocentric (left), egocentric (center), and wrist-mounted (right) per- centric), 3D scene reconstructions, and manually annotated spectives for both human and robot executions. articulation parameters, allowing us to evaluate articulation estimation methods under realistic in-the-wild conditions. language description of the part. To generate the mask, we We evaluate three representative approaches: ArtiPoint [49] prompt SAMv2 [39] on the panoramic images and lift the and the Gaussian-Splatting–based ArtGS [31] predict both predicted mask to 3D using the corresponding pointcloud. articulation type and motion parameters. In addition, we evaluate GPT-5 as a VLM to infer articulation types directly 4. Evaluations from single RGB views - both egocentric and third-person Since Aria glasses do not provide dense depth, we gener- Transferring manipulation skills across different embodi- ate depth inputs using MapAnything [26]. Predicted depth ments, for instance from a human hand demonstration to maps are aligned to metric scale by rendering depth from a robot gripper, requires a fundamental understanding of our 3D scene meshes at the corresponding camera pose and what interactions each object affords and the object’s phys- estimating a global scale factor via the mode of the per-pixel ical properties. An agent must deduce how an object can be depth-ratio histogram. This compensates for regions where manipulated (e.g. does a handle pull outwards or twist, does predicted depth and rendered geometry differ (e.g., hands a door swing left or right) and recognize constraints that dic- or moving articulated parts). More details are included in tate the required forces or motions, e.g., a drawer might be the supplementary material. We observe that methods such heavier or latched, requiring more force to open). Our eval- as ArtiPoint and ArtGS exhibit significantly lower perfor- uation examines embodied object understanding via three mance on our dataset compared to Arti4D [48] as indicated complementary tasks with corresponding benchmarks: ar- in Tab. 4. ArtGS exhibits lower performance in in-the-wild- ticulated object estimation (perceiving how parts move and settings due to clutter and non-robust object segmentation Dataset Method Articulation Type Motion Parameters pris Pprismatic [%] Prevolute [%] θerr [deg] dpris L2 [m] rev [deg] θerr GPT-5 (egocentric) 96.9 75.9 - - - Arti4D [48] ArtGS 100 0 52.59 56.82 0.25 ArtiPoint [48] 68 98 14.54 17.14 0.07 Hoi! (ours) GPT-5 (egocentric) 88.5 74.3 - - - GPT-5 (exocentric) 87.5 70.3 - - - ArtGS [31] 100.0 0.00 58.39 49.11 0.321 ArtiPoint [48] 26.90 57.10 47.06 63.76 0.540
Table 4. Articulation Estimation. Given a single image before interaction (for GPT) or the egocentric video (for ArtGS and ArtiPoint), methods estimate the type of articulation as well as the exact articulation axis in 3D.
on both Arti4D and Hoi!. However, ArtiPoint performs Table 5. Interaction Force Prediction. Based on measurements from the DIGIT tactile pressure sensor.RMSE (95% CI) in Newtons, averaged over worse on Hoi! when faced with scaled monocular depth, all validation environments. which yields stochastic non-steady noise throughout inter- actions, severely limiting its 3D lifting and trajectory filter- Method Tangential Normal Combined ing. In addition, we find that mere articulation type predic- Sparsh[22] w/ DINO 3.07[2.87, 3.26] 3.45[3.24, 3.66] 3.86[3.62, 4.11] tion is surprisingly robust when relying on GPT-5 on both Sparsh[22] w/ DINOv2 3.18[2.99, 3.38] 3.79[3.61, 3.96] 4.11[3.90, 4.33] Hoi! and Arti4D. We conclude that current articulation esti- mation methods are either too dependent on accurate depth across two opposing Digit sensors - this large increase in or fail in the presence of clutter and hands. error is noteworthy. We attribute this degradation primar- ily to out-of-distribution contact geometries (the model was 4.2. Tactile Force Estimation trained on a small set of simple indenters, whereas real han- dles, edges, and furniture parts present far more complex We investigate force prediction from gel-based tactile im- contact shapes) and to out-of-distribution load regimes that ages using data collected with our dataset. The goal is naturally arise during in-the-wild human operation. This to estimate the normal and tangential forces acting on highlights that tactile models which excel in controlled lab the gripper solely from the tactile images produced by settings struggle to generalize to unconstrained real-world the Hoi! gripper’s GelSight Digit sensors during interac- interactions, highlighting the value our data have for future tion. This task directly reflects the multimodality enabled research. by our dataset: tactile images from the Digits are tem- porally aligned with the corresponding end-effector forces 4.3. Visual Force Estimation from the force–torque (FT) sensor and the gripper’s motor- induced gripping forces, allowing us to construct reliable We evaluate the utility of our dataset for visual force esti- ground-truth labels. We evaluate two versions of the Sparsh mation, where a model predicts a 3D interaction force (and model [22], the state-of-the-art method for self-supervised affordance) from an RGB-D observation given a manipu- tactile representation learning, using both the DINO [3] lation goal. For example, an RGB-D image of a drawer and DINOv2 [35] decoders with the force-estimation head together with “open the drawer” should yield where to in- (“Task 1”) from the original work. To obtain ground-truth teract and what force to apply. We benchmark the Force- force components, we decompose interaction forces into Sight model [6], which predicts force goals for text-guided normal and tangential directions and express all measure- manipulation and has shown that such goals can improve ments in a common interaction frame aligned with the Digit robotic performance. Our dataset provides the necessary sensors. External forces measured by the FT sensor are multimodality for this task: the Hoi! gripper supplies per- rotated into this frame, while the gripper’s internal grip- image force–torque measurements, each sequence includes ping force is estimated from the torque–current relation- a language goal derived from our annotations, and 3D ship, its Jacobian, and a load-dependent calibration factor. ground-truth trajectories allow accurate alignment across We aggregate left/right sensor contributions to obtain com- frames. Because raw force–torque readings may include bined normal, tangential, and total force magnitudes. Since operator-induced forces unrelated to the actual articulation, Sparsh is trained on forces within a known range, we clip we report results on both the raw signals and a motion- the combined force magnitudes to remain consistent with aligned version. We project measured forces and torques the model’s expected distribution, avoiding evaluation-time onto the gripper’s linear and rotational velocity directions, extrapolation. Full details are provided in the supplemen- removing components that do not align with the intended tary material. We report the RMSE of the estimated forces motion. Following the evaluation protocol of the origi- in Tab. 5 and observe errors on the order of several New- nal ForceSight paper, we evaluate the method in a zero- tons, whereas Sparsh achieves millinewton-level accuracy shot setting on our dataset. While the model achieves an on its original benchmark. Although a direct comparison is RMSE of 0.404 N on the original dataset, its performance not fully equivalent - our setup evaluates forces aggregated degrades noticeably when applied to our data. As shown in Tab. 6, it particularly struggles in locations containing 6. Conclusions multiple articulated objects that require higher operating forces (e.g. kitchen 7 with an RMSE of 3.531 N, which in- We present Hoi!, a multimodal dataset designed to bridge cludes a fridge and an oven, or office 1 with an RMSE of the longstanding gap between human-centric and robot- 2.325 N, which features magnetic drawers; see Fig. 4). This centric interaction data. Central to our approach is the Hoi! suggests limited prior exposure to stiff or force-demanding gripper, which allows human operators to produce natural demonstrations with robot-grade physical sensing, captur- articulated mechanisms, an underrepresented class in exist- ing datasets and prior work. We also observe a noticeable ing force, torque, tactile, and visual signals in real-world improvement when using the projected forces and torques, manipulation tasks. This setup enables a unified representa- indicating that the method is less robust to real-world oper- tion of interaction that is transferable across embodiments. ation disturbances present in the raw measurements. Again, Our evaluation shifts focus towards an object-centric view of manipulation: we assess how well agents can infer what we highlight the value of our dataset to this underexplored line of research. an object affords, how it moves, and what physical effort is required to interact with it. Across three tasks: tactile force estimation, articulated motion prediction, and visual force Table 6. visual force prediction. Given an RGB-D observation and a estimation, we demonstrate how understanding object con- manipulation goal (e.g., “open the drawer”), the model predicts the 3D interaction force required to perform the action. We report the force RMSE straints is essential for generalizable manipulation and how (in N, lower is better) of ForceSight [6] across different locations in our current methods show significant room for improvement. dataset. Projected denotes evaluation on force components aligned with By contributing a richly annotated, cross-embodiment and the gripper’s motion direction. cross-view dataset, we aim to support the development of Location RMSE Projected [N] RMSE Raw [N] agents that not only observe but truly understand and act upon the physical world. bathroom 2 1.21 2.09 bedroom 4 1.33 1.85 bedroom 6 2.10 2.43 References kitchen 7 3.53 3.64 [1] Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, office 1 2.33 3.69 Siyuan Feng, Shenyuan Gao, Xindong He, Xu Huang, Shu livingroom 1 1.09 1.74 Jiang, et al. Agibot world colosseo: A large-scale manipula- Hoi! (Ours) 2.23 2.57 tion platform for scalable and intelligent embodied systems. ForceSight Dataset – 0.40 CoRR, 2025. 3 [2] Carlos Campos, Richard Elvira, Juan J. Gomez Rodriguez, Jose M. M. Montiel, and Juan D. Tardos. Orb-slam3: An accurate open-source library for visual, visual–inertial, and 5. Limitations & Future Work multimap slam. IEEE Transactions on Robotics, 37(6): 1874–1890, 2021. 5 [3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Hoi! represents a first step toward bridging human and Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- robot embodiments through force-grounded, cross-view in- ing properties in self-supervised vision transformers. CoRR, teraction data, but several limitations remain. First, al- abs/2104.14294, 2021. 7, 5 though human demonstrations with the Hoi! gripper mimic [4] Tianyi Cheng, Dandan Shan, Ayda Sultan, Richard E. L. robotic end-effector interactions, they are still generated Higgins, and David F. Fouhey. Towards a richer 2d under- by a human operator and therefore do not fully capture standing of hands at scale. In Proceedings of the 37th Inter- the kinematic and dynamic constraints of real manipu- national Conference on Neural Information Processing Sys- lators.This hybrid embodiment simplifies skill transfer in tems, Red Hook, NY, USA, 2023. Curran Associates Inc. 7 some way but complicates it in others since true general- [5] Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Ben- ization across full-body morphology is still an open chal- jamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran lenge. Second, while our dataset spans a wide variety of Song. Universal manipulation interface: In-the-wild robot articulated household objects, it does not yet cover the full teaching without in-the-wild robots, 2024. 4 range of mechanical complexities or rare edge-case mech- [6] Jeremy A. Collins, Cody Houff, You Liang Tan, and Charles C. Kemp. Forcesight: Text-guided mobile manip- anisms found in real-world environments. Finally, our cur- ulation with visual-force goals, 2023. 2, 7, 8, 6 rent benchmarks are object-centric and focus on perception [7] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, tasks such as force estimation and articulation reasoning. Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide These serve as foundational capabilities, but extending them Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and toward end-to-end policy learning, where perception and Michael Wray. Rescaling egocentric vision: Collection, action are trained jointly, remains an important direction for pipeline and challenges for epic-kitchens-100. International future work. Journal of Computer Vision (IJCV), 130:33–55, 2022. 2, 3 [8] Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan sion and pattern recognition, pages 18995–19012, 2022. 2, Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima 3, 7 Damen. Epic-kitchens visor benchmark: Video segmenta- [18] Kristen Grauman, Andrew Westbury, Lorenzo Torresani, tions and object relations, 2022. 7 Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar [9] Alexandros Delitzas, Ayca Takmaz, Federico Tombari, Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, Robert Sumner, Marc Pollefeys, and Francis Engelmann. Eugene Byrne, Zach Chavis, Joya Chen, Feng Cheng, Fu- SceneFun3D: Fine-Grained Functionality and Affordance Jen Chu, Sean Crane, Avijit Dasgupta, Jing Dong, Maria Understanding in 3D Scenes. In IEEE/CVF Conference on Escobar, Cristhian Forigua, Abrham Gebreselasie, Sanjay Computer Vision and Pattern Recognition (CVPR), 2024. 2 Haresh, Jing Huang, Md Mohaiminul Islam, Suyog Jain, [10] Frank Dellaert and GTSAM Contributors. borglab/gtsam, Rawal Khirodkar, Devansh Kukreja, Kevin J Liang, Jia- 2022. 5 Wei Liu, Sagnik Majumder, Yongsen Mao, Miguel Martin, Effrosyni Mavroudi, Tushar Nagarajan, Francesco Ragusa, [11] Ben Eisner and Harry Zhang. Flowbot3d: Learning 3d ar- Santhosh Kumar Ramakrishnan, Luigi Seminara, Arjun So- ticulation flow to manipulate articulated objects. Robotics mayazulu, Yale Song, Shan Su, Zihui Xue, Edward Zhang, Science and Systems 2022, 2022. 2 Jinxu Zhang, Angela Castillo, Changan Chen, Xinzhu Fu, [12] Jakob Engel, Kiran Somasundaram, Michael Goesele, Al- Ryosuke Furuta, Cristina Gonzalez, Prince Gupta, Jiabo bert Sun, Alexander Gamino, Andrew Turner, Arjang Talat- Hu, Yifei Huang, Yiming Huang, Weslie Khoo, Anush Ku- tof, Arnie Yuan, Bilal Souti, Brighid Meredith, Cheng Peng, mar, Robert Kuo, Sach Lakhavani, Miao Liu, Mi Luo, Chris Sweeney, Cole Wilson, Dan Barnes, Daniel DeTone, Zhengyi Luo, Brighid Meredith, Austin Miller, Oluwatumin- David Caruso, Derek Valleroy, Dinesh Ginjupalli, Duncan inu Oguntola, Xiaqing Pan, Penny Peng, Shraman Praman- Frost, Edward Miller, Elias Mueggler, Evgeniy Oleinik, Fan ick, Merey Ramazanova, Fiona Ryan, Wei Shan, Kiran So- Zhang, Guruprasad Somasundaram, Gustavo Solaira, Harry masundaram, Chenan Song, Audrey Southerland, Masatoshi Lanaras, Henry Howard-Jenkins, Huixuan Tang, Hyo Jin Tateno, Huiyu Wang, Yuchen Wang, Takuma Yagi, Mingfei Kim, Jaime Rivera, Ji Luo, Jing Dong, Julian Straub, Yan, Xitong Yang, Zecheng Yu, Shengxin Cindy Zha, Chen Kevin Bailey, Kevin Eckenhoff, Lingni Ma, Luis Pesqueira, Zhao, Ziwei Zhao, Zhifan Zhu, Jeff Zhuo, Pablo Arbelaez, Mark Schwesinger, Maurizio Monge, Nan Yang, Nick Char- Gedas Bertasius, David Crandall, Dima Damen, Jakob En- ron, Nikhil Raina, Omkar Parkhi, Peter Borschowa, Pierre gel, Giovanni Maria Farinella, Antonino Furnari, Bernard Moulon, Prince Gupta, Raul Mur-Artal, Robbie Pennington, Ghanem, Judy Hoffman, C. V. Jawahar, Richard Newcombe, Sachin Kulkarni, Sagar Miglani, Santosh Gondi, Saransh Hyun Soo Park, James M. Rehg, Yoichi Sato, Manolis Savva, Solanki, Sean Diener, Shangyi Cheng, Simon Green, Steve Jianbo Shi, Mike Zheng Shou, and Michael Wray. Ego- Saarinen, Suvam Patra, Tassos Mourikis, Thomas Whe- exo4d: Understanding skilled human activity from first- and lan, Tripti Singh, Vasileios Balntas, Vijay Baiyya, Wilson third-person perspectives, 2024. 2, 3 Dreewes, Xiaqing Pan, Yang Lou, Yipu Zhao, Yusuf Man- [19] Arjun Gupta, Michelle Zhang, Rishik Sathua, and Saurabh sour, Yuyang Zou, Zhaoyang Lv, Zijian Wang, Mingfei Yan, Gupta. Opening articulated structures in the real world, Carl Ren, Renzo De Nardi, and Richard Newcombe. Project 2025. 2 aria: A new tool for egocentric multi-modal ai research, [20] Anna-Maria Halacheva, Yang Miao, Jan-Nico Zaech, Xi 2023. 4 Wang, Luc Van Gool, and Danda Pani Paudel. Articulate3d: [13] Hao-Shu Fang, Hongjie Fang, Zhenyu Tang, Jirong Liu, Holistic understanding of 3d scenes as universal scene de- Chenxi Wang, Junbo Wang, Haoyi Zhu, and Cewu Lu. scription. In Proceedings of the IEEE/CVF International Rh20t: A comprehensive robotic dataset for learning diverse Conference on Computer Vision (ICCV), 2025. 2 skills in one-shot, 2023. 3 [21] Nick Heppert, Muhammad Zubair Irshad, Sergey Zakharov, [14] Martin A. Fischler and Robert C. Bolles. Random sample Katherine Liu, Rares Andrei Ambrus, Jeannette Bohg, Ab- consensus: a paradigm for model fitting with applications to hinav Valada, and Thomas Kollar. Carto: Category and joint image analysis and automated cartography. Commun. ACM, agnostic reconstruction of articulated objects. In Proceed- 24(6):381–395, 1981. 4 ings of the IEEE/CVF Conference on Computer Vision and [15] Fadri Furrer, Marius Fehr, Tonci Novkovic, Hannes Sommer, Pattern Recognition, pages 21201–21210, 2023. 2 Igor Gilitschenski, and Roland Siegwart. Evaluation of Com- [22] Carolina Higuera, Akash Sharma, Chaithanya Krishna Bod- bined Time-Offset Estimation and Hand-Eye Calibration on duluri, Taosha Fan, Patrick Lancaster, Mrinal Kalakrishnan, Robotic Datasets. Springer International Publishing, Cham, Michael Kaess, Byron Boots, Mike Lambeta, Tingfan Wu, 2017. 5 and Mustafa Mukadam. Sparsh: Self-supervised touch rep- [16] T. Gonzalez. Clustering to minimize the maximum inter- resentations for vision-based tactile sensing. In 8th Annual cluster distance. Theoretical Computer Science, 38:293–306, Conference on Robot Learning, 2024. 7, 5 1985. 4 [23] Advait Jain and Charles C. Kemp. Improving robot manip- [17] Kristen Grauman, Andrew Westbury, Eugene Byrne, ulation with data-driven object-centric models of everyday Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson forces. Autonomous Robots, 35(2):143–159, 2013. 2 Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: [24] Zhenyu Jiang, Cheng-Chun Hsu, and Yuke Zhu. Ditto: Around the world in 3,000 hours of egocentric video. In Building digital twins of articulated objects from interaction, Proceedings of the IEEE/CVF conference on computer vi- 2022. 2 [25] Simar Kareer, Dhruv Patel, Ryan Punamiya, Pranay Mathur, Kanazawa, David Fouhey, and Jitendra Malik. Reconstruct- Shuo Cheng, Chen Wang, Judy Hoffman, and Danfei Xu. ing hands in 3D with transformers. In CVPR, 2024. 6 Egomimic: Scaling imitation learning via egocentric video. [37] Toby Perrett, Ahmad Darkhalil, Saptarshi Sinha, Omar In 2025 IEEE International Conference on Robotics and Au- Emara, Sam Pollard, Kranti Kumar Parida, Kaiting Liu, Pra- tomation (ICRA), pages 13226–13233. IEEE, 2025. 3 jwal Gatti, Siddhant Bansal, Kevin Flanagan, et al. Hd-epic: [26] Nikhil Keetha, Norman Müller, Johannes Schönberger, A highly-detailed egocentric video dataset. In Proceedings Lorenzo Porzi, Yuchen Zhang, Tobias Fischer, Arno of the Computer Vision and Pattern Recognition Conference, Knapitsch, Duncan Zauss, Ethan Weber, Nelson Antunes, pages 23901–23913, 2025. 2, 3 Jonathon Luiten, Manuel Lopez-Antequera, Samuel Rota [38] Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Point- Bulò, Christian Richardt, Deva Ramanan, Sebastian Scherer, net++: Deep hierarchical feature learning on point sets in a and Peter Kontschieder. Mapanything: Universal feed- metric space, 2017. 4 forward metric 3d reconstruction, 2025. 6, 5 [39] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang [27] Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Bal- Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman akrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junt- Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang ing Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Chen, Kirsty Ellis, et al. Droid: A large-scale in-the-wild Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feicht- robot manipulation dataset. In RSS 2024 Workshop: Data enhofer. Sam 2: Segment anything in images and videos, Generation for Robotics. 3 2024. 6 [28] Marion Lepert, Jiaying Fang, and Jeannette Bohg. Phantom: [40] Joern Rehder, Janosch Nikolic, Thomas Schneider, Timo Training robots without robots using only human videos, Hinzmann, and Roland Siegwart. Extending kalibr: Cali- 2025. 3 brating the extrinsics of multiple imus and of individual axes. [29] Liu Liu, Wenqiang Xu, Haoyuan Fu, Sucheng Qian, Qiao- pages 4304–4311, 2016. 1 jun Yu, Yang Han, and Cewu Lu. Akb-48: A real-world [41] Ruochen Ren, Zhipeng Wang, Chaoyun Yang, Jiahang Liu, articulated object knowledge base. In Proceedings of the Rong Jiang, Yanmin Zhou, Shuo Jiang, and Bin He. Enhanc- IEEE/CVF Conference on Computer Vision and Pattern ing robotic skill acquisition with multimodal sensory data: A Recognition, pages 14809–14818, 2022. 2 novel dataset for kitchen tasks. Scientific Data, 12(1):476, [30] Wenhai Liu, Junbo Wang, Yiming Wang, Weiming Wang, 2025. 3 and Cewu Lu. Forcemimic: Force-centric imitation learning [42] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary with force-motion capture system for contact-rich manipula- Bradski. Orb: An efficient alternative to sift or surf. In 2011 tion, 2025. 3 International Conference on Computer Vision, pages 2564– [31] Yu Liu, Baoxiong Jia, Ruijie Lu, Junfeng Ni, Song-Chun 2571, 2011. 4 Zhu, and Siyuan Huang. Artgs: Building interactable repli- [43] Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and cas of complex articulated objects via gaussian splatting, Marcin Dymczyk. From coarse to fine: Robust hierarchical 2025. 2, 6, 7, 5 localization at large scale. In CVPR, 2019. 5, 4 [32] Roberto Martı́n-Martı́n, Clemens Eppner, and Oliver Brock. [44] Chaoyue Song, Jiacheng Wei, Chuan-Sheng Foo, Guosheng The rbo dataset of articulated objects and interactions, 2018. Lin, and Fayao Liu. Reacto: Reconstructing articulated ob- 2, 3 jects from a single video, 2024. 2 [33] Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna [45] Jürgen Sturm. Learning kinematic models of articulated ob- Tripathi, Leonidas J Guibas, and Hao Su. Partnet: A large- jects. Springer Tracts in Advanced Robotics, pages 65–111, scale benchmark for fine-grained and hierarchical part-level 2013. 2 3d object understanding. In Proceedings of the IEEE/CVF [46] J ürgen Sturm, Vijay Pradeep, Cyrill Stachniss, Christian conference on computer vision and pattern recognition, Plagemann, Kurt Konolige, and Wolfram Burgard. Learn- pages 909–918, 2019. 2 ing kinematic models for articulated objects. 2 [34] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea [47] Yufei Wang, Ziyu Wang, Mino Nakura, Pratik Bhowal, Chia- Finn, and Abhinav Gupta. R3m: A universal visual repre- Liang Kuo, Yi-Ting Chen, Zackory Erickson, and David sentation for robot manipulation. In 6th Annual Conference Held. Articubot: Learning universal articulated object ma- on Robot Learning. 3 nipulation policy via large scale simulation, 2025. 3 [35] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy [48] Abdelrhman Werby, Martin Büchner, Adrian Röfer, Chen- Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, guang Huang, Wolfram Burgard, and Abhinav Valada. Ar- Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mah- ticulated object estimation in the wild. In Conference on moud Assran, Nicolas Ballas, Wojciech Galuba, Russell Robot Learning (CoRL), 2025. 3, 6, 7 Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael [49] Abdelrhman Werby, Martin Büchner, Adrian Röfer, Chen- Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Je- guang Huang, Wolfram Burgard, and Abhinav Valada. Ar- gou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr ticulated object estimation in the wild, 2025. 2, 5, 6 Bojanowski. Dinov2: Learning robust visual features with- [50] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao out supervision, 2024. 7, 4, 5 Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, [36] Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo He Wang, et al. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11097– 11107, 2020. 2 [51] Haoyu Xiong, Haoyuan Fu, Jieyi Zhang, Chen Bao, Qiang Zhang, Yongxi Huang, Wenqiang Xu, Animesh Garg, and Cewu Lu. Robotube: Learning household manipulation from human videos with simulated twin environments. In 6th An- nual Conference on Robot Learning, 2022. 3 [52] Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao- gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth any- thing v2, 2024. 4 [53] Yi Yang and Deva Ramanan. Articulated human detection with flexible mixtures of parts. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 35(12):2878–2890, 2013. 6 [54] Chenyangguang Zhang, Alexandros Delitzas, Fangjinhua Wang, Ruida Zhang, Xiangyang Ji, Marc Pollefeys, and Francis Engelmann. Open-vocabulary functional 3d scene graphs for real-world indoor spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19401–19413, 2025. 3 [55] Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware, 2023. 4 Hoi! - A Multimodal Dataset for Force-Grounded, Cross-View Articulated Manipulation Supplementary Material
Contents
-
Introduction 1
-
Related Work 2
-
Hoi! Dataset 4
-
Evaluations 6 4.1. In-the-Wild Articulated Object Estimation … . . 6 4.2. Tactile Force Estimation … … … … . . 7 4.3. Visual Force Estimation … … … … … 7
-
Limitations & Future Work 8
-
Conclusions 8 Supplementary Figure 8. Hoi! Gripper. The 2-finger parallel gripper is operated through the load cell, where the measured load is translated into A. Hoi! Gripper Calibration Details 1 gripping force. Interaction force and tactile contact pressure are measured through the Digit and Force-Torque sensors respectively. Aria Glasses and A.1. Motor Calibration … … … … … … 1 a stereo camera provide pose estimation and wrist-view observations. We A.2. Inter-Sensor Calibration … … … … … 1 will release the design as open source. A.3. Gripper Gravity Compensation … … … . . 2
B. Alignment of Sensors in the Hoi! Dataset Recordings 2 B.1. Time Alignment … … … … … … . 2 with k1 = 1.769 and k2 = −0.2214, as stated in the data B.2. Spatial Alignment … … … … … … 3 sheet. Second, the gripping force is derived via the jaco- bian of the kinematic relationship F = J(q) τ , where τ is C. Spot Recordings 4 calculated as a function of the lever angle q: D. Evaluations 4 D.1. In-The-Wild Articulation Estimation … … . . 5 D.2. Tactile Force Estimation … … … … . . 5 L12 sin q cos q J(q) = 2−L1 sin q − q 2 . D.3. Visual Force Estimation … … … … … 6 L22 − L1 sin q D.4. Hand Pose Estimation … … … … … . 6 To account for efficiency variations due to load- dependent motor performance and friction, we introduce a A. Hoi! Gripper Calibration Details load-dependent calibration factor. This factor is obtained In the following we give a detailed description of the cali- empirically by gripping a force sensor and recording pairs brations done for the Hoi! gripper depicted in Supplemen- of measured gripping forces and corresponding motor cur- tary Fig. 8. rents. Using least-squares estimation, we determine η(I) across multiple load regimes. A.1. Motor Calibration A.2. Inter-Sensor Calibration The Hoi! gripper’s gripping force is modeled as We use Kalibr [40] to calibrate the gripper in the following (grip) order: We first calibrate the intrinisics of the ZED camera, Fi = g(J(q), η(I), I) and then the visual-inertial extrinsics to the Zed’s IMU. We where I denotes the motor current, J(q) the gripper Jaco- then use visual-inertial calibration between the ZED images bian as a function of the motor position q, and η(I) a load- and the IMU in the force-torque sensor to find the extrin- dependent efficiency factor. First, the motor torque is ex- sics between stereo camera and FT sensor. The Aria device pressed as a proportional function of the current runs its own intrinsic calibration, and we find the extrinsics through stereo calibration between one Aria and one Zed τ (ImA ) = k1 I + k2 . camera in their overlapping field-of-view Supplementary Figure 9. Gravity Compensation. We depict the uncompensated (left), compensated (middle) and filtered compensated (right) forces (upper row) and torques (lower row) of an examplary gripper recording (bathroom 2). The no-load windows are stylized in blue.
A.3. Gripper Gravity Compensation measurement windows by subtracting the median filtered force magnitude from the raw force magnitude and thresh- To measure the isolated interaction forces between grip- olding the result. We then estimate the biases by solving the per and furniture, we need to compensate for gravitational following least-squares problem: forces acting on the endeffector, as well as internal biases of the FT sensor. The governing equation is: N 2 meas − R⊤ fS,k W S,k m gW − bf X fSmeas = fSext + fSg + bf , min , bf ,bτ τ S,k − rSC × (R⊤ meas W S,k m gW ) − bτ g k=1 τ meas S = τ ext S + τ S + bτ , where N is the number of no-load samples. Given the es- where timated biases and the known mass and center of mass, we fSg = RS←W m gW , can now compute the external forces and torques during in- teraction as τ gS = rC←S × fSg , fSext = fSmeas − R⊤ W S m gW − bf , τ ext ext S = rP ←S × fS , τ ext meas S = τS − rSC × (R⊤ W S m gW ) − bτ . where rSP is the vector from the sensor origin to the con- . The results of gravity compensation are depicted in Sup- tact point. Here, fSmeas and τ meas S are the measured forces plementary Fig. 8. We also apply a Butterworth filter of and torques in the sensor frame S, fSext and τ ext S are the ex- degree 4 to filter noise. ternal forces and torques acting on the sensor, fSg and τ gS are the gravitational forces and torques acting on the sen- B. Alignment of Sensors in the Hoi! Dataset sor, bf and bτ are the internal biases of the force-torque Recordings sensor, RS←W is the rotation matrix from the world frame W to the sensor frame S, m is the mass of the endeffector We give a more in-depth explantion of both temporal and assembly, gW is the gravity vector in the world frame, and spatial alignment of the multiple sensor streams in the Hoi! rC←S is the vector from the sensor origin to the center of dataset. This process allows us to capture interactions mass of the endeffector assembly. We measure the mass of over time and across multiple perspectives, as depicted in the endeffector using a scale, while the center of mass is es- Fig. 10. timated from the CAD model of the endeffector. RS←W is taken from the Aria SLAM and extrinsic calibration. Inter- B.1. Time Alignment nal biases are estimated during no-load conditions, where As the different recording modules run independently of external forces and torques are zero. We estimate no load each other on different internal clocks, we need to align Supplementary Figure 10. Spatial and Temporal Alignment We both temporally and spatially align all recording modules. This allows us to capture interactions over time across multiple viewpoints.
the time frames of the recordings in post processing. Dur- corresponds to a temporal alignment accuracy of roughly ing recording, we display a QR code encoding the current 10−25 ms per stream. In a representative example, when Unix timestamp at 25 Hz into each camera stream i. We the iPhone records at 60 Hz (Ti ≈ 16.7 ms) and the Aria detect and decode the first QR code, yielding a time pair at 30 Hz (TAria ≈ 33.3 ms), the resulting uncertainty is (tinternal , tref ), where tinternal is the internal timestamp of σ∆i ≈ 10.8 ms, the video stream i at which the QR code was captured, and B.2. Spatial Alignment tref is the corresponding reference Unix timestamp encoded in the QR code. Detecting and decoding this QR code yields We spatially align all recording devices into a common a time offset for each video stream with respect to a com- reference frame using visual localization against the high- mon reference time, in our case the Aria egocentric camera resolution 3D scans. We first construct a reference set of stream, as dense 2D-3D correspondences from the point cloud and corresponding panoramic images captured with the scanner. ∆i = (tref,i − tinternal,i ) − (tref,Aria − tinternal,Aria ), We rectify the panoramic images into multiple perspective images with virtual camera parameters K, R, t. We fur- where ∆i denotes the relative clock offset of stream i ther convert the point cloud into a mesh using the Leica with respect to the Aria reference. The uncertainty of this proprietary software and render depth maps from the same single-shot estimate is dominated by frame-timing quanti- virtual camera poses. The rendered depth maps are then zation: assuming the QR update occurs uniformly δi ∼ back-projected into 3D space using the known virtual cam- Uniform(0, Ti ), i ∈ {ArEgo, Sensor}, within the exposure era intrinsics K and extrinsics (R, t) as of the first detected frame, the standard deviation of the alignment error is ud P = R−1 K−1 v d − R−1 t, q 2 T2 TAria σ∆i = 12i + 12 , d yielding a 95% confidence interval of approximately where (u, v) are pixel coordinates in the image plane and ±1.96 σ∆i . For typical frame rates of 30−60 Hz, this d is the corresponding depth value at that pixel. This With the latter, we filter out frames where the device is very close to the furniture, where usually no good references are in view. For Aria and GoPro cameras, which do not provide depth maps, we use DepthAnything v2 [52]. On this fil- tered subset, we extract DINOv2 [35] features and perform farthest-point sampling [16, 38] in feature space to select N diverse and representative keyframes per trajectory. We es- timate the 6-DoF pose of each keyframe through hloc [43] with the 2D-3D database constructed from the Leica scan and robustly estimate a single rigid transformation Tqueryworld between each sensor trajectory and the shared world frame. Since we also have the corresponding query poses Tcam query , i
with query ∈ {iPhone, Aria, UMI}, we can estimate a sin-
gle rigid transformation Tquery
world aligning each query trajec-
tory to the common world frame by
cami −1
Tquery cami
world = Tworld Tquery .
After localizing each keyframe, we compute the Frobenius
distances between all pairwise combinations of the N esti-
mated transformations Tqueryworld and reject outliers based on a
threshold relative to the median distance. We then average
the remaining inlier transformations to obtain a final robust
estimate of Tdevice
world . This simple outlier rejection strategy
is sufficient, as Hierarchical Localization already performs
RANSAC [14]-based PnP pose estimation internally. Fi-
nally, we transform all Aria poses and hand poses into the
common world frame using the estimated transformation
TAria
world , yielding globally aligned 6-DoF trajectories for the
Aria head, hands, and Hej gripper. The iPhone cameras are
statically mounted, and we therefore directly measure their
poses in the world frame during spatial calibration. The
UMI gripper poses are transformed into the world frame
using the estimated transformation TUMI world .
Supplementary Figure 11. Time Alignment. The recording is started for all recording modules of a single recording and a QR code encoding the C. Spot Recordings current time is shown to all video streams, so that the individual clocks can be aligned in post processing. As mentioned in the main paper, we collect robotic data for a subset of interactions using a Boston Dynamics Spot Robot. Here, we record the robot’s joint states, surrounding operation transforms each depth pixel into a 3D point P RGB-D cameras, the gripper RGB-D camera as well as Aria in the global coordinate frame, thereby establishing dense data using a wrist-mounted Aria, imitating the wrist view- 2D-3D correspondences between the rendered depth maps point of gripper and wrist recordings. As shown in Fig. 12, and the rectified panoramic images. We assume zero drift the robot is teleoperated using a Meta Quest 3. Here the and global consistency of the per-device SLAM, which operator controls the robot base using the joystick on the reduces the global registration problem to a single rigid Quest remote, while the remote’s 6-DoF pose is retargeted 3D-3D alignment. While in principle a single localized to the Spot gripper. The opening angle is also controlled via frame would suffice for this alignment, we automatically button on the remote. select a set of high-quality keyframes for robust registra- tion. To obtain these keyframes, we filter all frames of a D. Evaluations trajectory according to feature density, sharpness, and scene depth. Specifically, we retain frames with a high number of In the following we give more in-depth description of how ORB [42] keypoints, a high variance of the Laplacian (in- we created the evaluation groundtruth used in our evalua- dicating low motion blur) and a high mean estimated depth. tions from the Hoi! dataset. D.2. Tactile Force Estimation We evaluate the utility of our dataset for the task tactile force estimation from gel-based tactile sensors. This task aims to estimate contact forces acting on the sensor’s sur- face solely from the tactile images captured by the sensor during contact. We specifically focus on estimating the nor- mal and tangential forces, as these are most relevant for ma- nipulation tasks. We combine the tactile images provided by the Hoi! gripper’s Gelsight Digit sensors with the cor- responding forces provided by the gripper’s force-torque sensor and the gripper’s gripping forces into ground-truth labels. We evaluate two versions the Sparsh model [22], the SOTA method for self-supervised tactile representa- tions. We evaluate both the DINO [3] and DINOv2 [35] de- coder, with a fine-tuned force estimation decoder (referred Supplementary Figure 12. Spot Teleoperation The Spot Robot is teleop- erated using a Meta Quest 3. We retarget the remote’s 6-DoF pose to the to as ’Task 1’ in the original Sparsh paper). The interac- gripper and control the base using the remote mounted Joystick. tion forces during grasping can be decomposed into two (grip) components: a normal preload Fi resulting from the motor-torque–induced gripping force, and an external reac- (ext) D.1. In-The-Wild Articulation Estimation tive force Fi exerted by the environment. The external (ext) In this section, we evaluate a set of different approaches to force is measured by the force–torque (FT) sensor as Fs (grip) articulation estimation. This task concerns recovering ar- in the sensor’s local coordinate frame. The preload Fi , ticulation parameters (axis and position) as well as the ar- while sensed by the tactile sensors (Digits), represents an ticulation type from visual observations. We first evaluate internal force and is therefore not captured by the FT sen- ArtiPoint [49], a recent method for articulation estimation sor. To express all forces in a consistent coordinate system, in the wild. We also include the Gaussian-Splatting-based we define an interaction frame i, whose axes are aligned approach ArtGS [31]. Furthermore, we investigate how with the Digit frames. The interaction frame is defined to GPT-5, as a state-of-the-art vision-language model (VLM), be coaxial with the Digit frames; however, since the two can infer articulation types from both egocentric and ex- Digits face each other, corresponding axes have opposing ocentric observations. For this analysis, we make use of directions (xL = −xR , zL = −zR , yL = yR ). This does not our dataset’s posed RGB frames (both egocentric and exo- affect our analysis, as we only consider the sum of absolute centric), articulation annotations, and provided 3D ground- force magnitudes. The FT-sensor measurements are rotated (ext) (ext) truth.As the Aria does not provide dense depth, we gener- into this frame as Fi = Ri←s Fs , where Ri←s de- ate dense RGB-D data, we employ MapAnything [26] to notes the rotation from the sensor frame s to the interaction predict dense depth for each posed frame in our interaction frame i. All subsequent equations and force components sequences. We convert these depth estimates into metric are defined in this frame. Considering the hardware config- scale by rendering depth maps from our 3D ground-truth uration of the Hoi gripper, the combined absolute forces are meshes under the corresponding camera pose. We then ro- computed as bustly compute a global scale factor by forming a per-pixel (tang) X (ext) scale histogram and selecting the scale as the mode Fi = |Fi, k,{x,y} | , 2 k∈{L, R}
s = arg max h(s) (norm)
X (ext)
s Fi = |Fi, k,z |, (1)
k∈{L, R}
, where h(s) denotes the histogram of per-pixel ratios be- q tween predicted and rendered depths. This is necessary be- (comb) (tang) 2 (norm) 2 Fi = Fi + Fi . cause certain regions in the predicted depth differ from the rendered depth (e.g., the operator’s hand or articulated parts The gripping force is estimated from the gripper’s present in the predicted depth but absent in the mesh render- torque–current relationship, its Jacobian, and a load- (grip) ing). We finally apply the scale factor to obtain dense, met- dependent calibration factor as Fi = g(J(q), η(I), I). rically accurate depth for each frame. The actual ground Since the Sparsh models are trained on force data within truth articulation parameters are provided using a light- the range of [4, 4, 5] N along the x, y, and z axes, respec- weight manual annotation tool presented in [49]. tively, we clip the combined ground-truth magnitudes to (norm) (tang) √ Fmax = 10 N and Fmax = 32 N to ensure consis- mance of hand-pose estimation across these viewpoints. tency between training and evaluation distributions. This Because the Aria MPS hand keypoints are automatically avoids extrapolation to unseen force magnitudes and pro- generated-derived from stereo and globally optimized tra- vides a fair comparison of model performance. The grip- jectories but not from manual annotations or motion-capture ping force is estimated using the gripper’s torque-current systems, we treat this evaluation as an exploratory analysis relationship, its Jacobian and a load dependent calibration rather than a definitive benchmark. factor Fgrip = g(J(q), η(I), I). As the Sparsh models We employ the method of Pavlakos et al. [36], which are trained on force data within therange of [4, 4, 5] N for has shown strong in-the-wild performance. Using the Aria x, y, and z, respectively, we clip our combined ground- MPS trajectories, we project hand keypoints into egocentric truth magnitudes to√Fmax,normal =√ 2 × 5 = 10 N and frames and compute the commonly used PCK metric [53]. Fmax,tangential = 42 + 42 = 32 N to ensure con- While the results ( Tab. 8) show a noticeable perfor- sistency between training and testing distributions. This mance gap relative to controlled benchmarks, this is ex- avoids extrapolation to unseen force magnitudes and pro- pected given the challenging characteristics of our real- vides a fair assessment of model performance. The ex- world setting - fast hand motions, natural manipulation be- tended evaluation results are depicted in Tab. 7. haviors, and lower-resolution egocentric imagery. Rather As depicted in Fig. 13, we observe that the force predic- than indicating deficiencies, these findings highlight the dif- tions generally under-predict the tactile forces, even though ficulty of egocentric manipulation scenes and underline the the GT is clipped to be within the training range. opportunity for future methods to better leverage the rich multimodal signals present in our dataset. D.3. Visual Force Estimation We evaluate the utility of our dataset to the task of visual force estimation. In this task a 3D interaction force is esti- mated alongside an affordance (interaction type) in order to complete a given manipulation goal. In our context the in- put might be an RGB-D image of a drawer alongside the prompt ”open the drawer” and the model would predict where to interact and what force to apply. We evaluate the ForceSight model [6], a model that aims to predict forces as part of visual-force goals for robotic manipulation, demon- strating that force goals can significantly increase robotic manipulation performance. The ForceSight dataset consists of interaction sequences that include posed RGB-D obser- vations, per-frame force–torque (FT) readings, and gripping forces. Each sequence is paired with an open-language goal derived from our interaction annotations. Using the Hoi! gripper’s FT sensor, we generate per-image force–torque la- bels and leverage the 3D ground-truth trajectories provided by our dataset. We evaluate the model across a diverse sub- set of six environments. Because raw force-torque signals may include operator-induced forces in directions unrelated to the articulation (e.g., internal stresses not required for the intended motion), we report results on both the raw data and a motion-aligned variant. For fair comparison, we project the measured force vector f onto the gripper’s linear ve- v locity v using f∥ = f · ∥v∥ , and similarly project the torque ω vector τ onto the rotational velocity ω as τ∥ = τ · ∥ω∥ . This removes force and torque components that do not contribute to the articulated interaction, resulting in a fairer evaluation signal.
D.4. Hand Pose Estimation As our dataset captures not only gripper interactions but also hand interactions, we additionally explore the perfor- Supplementary Table 7. Tactile Force Estimation. We show the evaluation results for our tactile force estimation evaluation per location and split into tangential, normal and combined forces over 2 digit images.
Tangential Normal Combined
Location DINO DINOv2 DINO DINOv2 DINO DINOv2 bathroom 2 2.07 [1.88, 2.25] 2.02 [1.81, 2.23] 4.81 [4.54, 5.09] 4.97 [4.69, 5.26] 4.92 [4.64, 5.21] 5.17 [4.88, 5.47] bedroom 4 3.23 [3.09, 3.37] 2.77 [2.63, 2.92] 3.22 [3.07, 3.38] 3.72 [3.60, 3.85] 3.41 [3.22, 3.60] 3.96 [3.81, 4.11] bedroom 6 3.39 [3.19, 3.59] 3.77 [3.52, 4.00] 3.48 [3.23, 3.71] 3.43 [3.22, 3.63] 4.31 [4.04, 4.57] 4.45 [4.24, 4.66] kitchen 7 3.05 [2.82, 3.27] 3.83 [3.67, 4.00] 2.76 [2.60, 2.92] 3.50 [3.39, 3.61] 3.46 [3.28, 3.65] 3.19 [2.97, 3.41] office 1 3.63 [3.41, 3.86] 3.76 [3.52, 4.01] 3.88 [3.62, 4.14] 4.07 [3.86, 4.27] 4.61 [4.30, 4.92] 4.91 [4.67, 5.17] livingroom 1 2.54 [2.31, 2.77] 2.61 [2.40, 2.82] 3.63 [3.34, 3.91] 3.71 [3.46, 3.95] 3.89 [3.56, 4.21] 3.90 [3.61, 4.19] Overall 3.07 [2.87, 3.26] 3.18 [2.99, 3.38] 3.45 [3.24, 3.66] 3.79 [3.61, 3.96] 3.86 [3.62, 4.11] 4.11 [3.90, 4.33]
Supplementary Figure 13. Tactile force Estimation. We depict the GT forces, clipped GT forces and the estiamted forces (DINOv2) for an examplary recording.
Supplementary Table 8. Hand Pose Estimation Average PCK@0.15 on our evaluation locations and compared to the Hamer baseline performace on New Days, VISOR and Ego4D datasets.
Location / Dataset PCK
bathroom 2 0.757
bedroom 4 0.764
bedroom 6 0.708
kitchen 7 0.535
office 1 0.732
livingroom 1 0.748
Overall 0.699
New Days [4] 0.888
VISOR [8] 0.893
Ego4D [17] 0.844