VAM-HRI 2024 Research Report
7th Workshop on Virtual, Augmented, and Mixed Reality for Human-Robot Interaction HRI 2024 · Boulder, Colorado, USA · March 2024 官方頁面：https://vam-hri.github.io/previous/2024/program OpenReview：https://openreview.net/group?id=humanrobotinteraction.org%2FHRI%2F2024%2FWorkshop%2FVAM-HRI
Workshop Overview
VAM-HRI 2024 為第七屆，共收錄 21 篇論文，以 Lightning Talk 格式分四組呈現。相較於 2025 年版，本屆研究廣度更大，涵蓋使用者辨識、輔助機器人、聽障兒童、工業部署、機器人競賽 AR 化，以及 ROS/Unity 通訊效能比較等多樣主題。
VAM-HRI 2024 is the 7th edition, featuring 21 papers in four Lightning Talk groups. Compared to the 2025 edition, this year covers a broader range of topics including operator identification, assistive robotics, accessibility for deaf children, industrial deployment, robotics competition AR overlays, and ROS/Unity communication benchmarking.
Paper Summaries
Lightning Talk Group 1
#1 AR-Based UI for Improving IO Setup in Robot Deployment Process
Authors: Lauritsen, Andersen, Zielinski, Kjærgaard Source: https://openreview.net/forum?id=18dfsIIr41
中文摘要： 以行動裝置 AR 應用程式提供情境感知的引導，協助非專業人員將電線正確接入機器人的 IO 埠。用戶測試結果顯示，相比傳統方法，心理負荷降低 25%，可用性評分提高 15%。
English: Uses a mobile AR app to provide contextually-aware wire-connection guidance for robot IO port setup. User testing showed 25% lower mental demand and 15% higher usability score vs. traditional methods.
Key Contribution: Quantitative AR-guided IO setup improvement with user study validation
#2 Hear Here: Sonification as a Design Strategy for Robot Teleoperation Using Virtual Reality
Authors: Simmons, Bown, Bremner, McIntosh, Mitchell Institution: University of the West of England, Bristol / Sellafield partnership Source: https://openreview.net/forum?id=1ZPQtSzYAP
中文摘要： 在 VR 模擬核設施場景中，以聽覺化（sonification）技術將機器人感測器資料轉換為聲音，輔助操作者辨識輻射、溫度、可燃氣體等危害。設計原則包含認知對齊隱喻、可讀性和使用者舒適度。
English: In a VR-simulated nuclear facility, robot sensor data (radiation, temperature, flammable gas) is sonified using cognitively aligned metaphors to help operators identify hazards. Developed in partnership with Sellafield nuclear facility operators.
Key Contribution: Principled sonification design framework for safety-critical robot teleoperation in VR
#3 Helping Humans Become Better Teachers for Robots with Augmented Reality
Authors: Cleaver, Sinapov Institution: Tufts University
中文摘要： 研究如何透過 AR 介面讓人類示範者成為更好的機器人教師。探索「不完美示範」的特性，並設計 AR 工具幫助使用者在示範時提供對機器學習更有益的動作資料。
English: Investigates how AR interfaces can help humans become better demonstration teachers for robot learning. Explores how “imperfect” teaching can be leveraged and how AR tools can guide users to provide more ML-beneficial demonstration data.
Key Contribution: AR-guided improvement of human teaching behavior for robot LfD
#4 Operator Identification in a VR-Based Robot Teleoperation Scenario Using Head, Hands, and Eyes Movement Data
Authors: Ritola, Giaretta, Kiselev Source: https://openreview.net/forum?id=Xals4UE6ZS
中文摘要： 在 VR 機器手臂遙操作場景中，結合頭部、眼部追蹤及 Leap Motion 手部追蹤資料進行操作者身份辨識。實驗顯示在 70/30 訓練/測試分割下辨識準確率接近 100%，具備 VR 身份認證應用潛力。
English: Identifies VR teleoperation operators using head, eye, and hand tracking data in a simulated robot arm scenario. Aggregated session data achieves near-100% identification accuracy with 70/30 train/test split, demonstrating VR biometric authentication potential.
Key Contribution: Near-perfect VR operator identification using multimodal movement biometrics
#5 Towards a Gaze-Driven Assistive Neck Exoskeleton via Virtual Reality Data Collection
Authors: Thompson, Zhang, Brown Source: https://openreview.net/forum?id=6DxA9CMfvJ
中文摘要： 針對神經退化性疾病造成的「垂頭症候群」，透過 VR 收集健康受試者的眼部與頭部聯動資料，訓練機器學習模型以眼動預測使用者意圖頭部運動，用於驅動輔助頸部外骨骼。
English: Addresses dropped head syndrome in neurodegenerative diseases by using VR to collect coupled eye-head movement data from healthy individuals. Trains an ML model to predict intended head movement from gaze alone, targeting an assistive neck exoskeleton controller.
Key Contribution: VR as data collection platform for gaze-to-head motion mapping in assistive exoskeleton design
Lightning Talk Group 2
#6 Comparing Dashboard and Virtual Reality Wizard-of-Oz Setups in a Human-Robot Conversational Task
Authors: Miniotaite, Torubarova, Pereira
中文摘要： 比較在人機對話任務中，傳統 Dashboard 介面與 VR Wizard-of-Oz 設定的差異，探討兩種方式對實驗設計和互動品質的影響。
English: Compares traditional dashboard vs. VR Wizard-of-Oz experimental setups for human-robot conversational tasks, examining how interface choice affects experimental validity and interaction quality.
Key Contribution: Methodological comparison of WoZ setups for HRI research design
#7 Interaction Design of the Mixed Reality Application for Deaf Children
Authors: Kydyrbekova, Kenzhekhan, Omirbayev, Oralbayeva, Imashev, Sandygulova
中文摘要： 為聽障兒童設計混合現實應用程式，研究 MR 互動設計在無障礙教育中的應用，結合手語辨識技術提供更具包容性的學習體驗。
English: Designs a mixed reality application for deaf children, exploring how MR interaction design can create more inclusive educational experiences, integrating sign language recognition for accessible learning.
Key Contribution: Accessible MR design for deaf children, bridging HRI and special education
#8 RobARtics: Using Augmented Reality to Enhance Robotics Competitions
Authors: Marques, Alves, Pedrosa, Cabral, Silva, Santos
中文摘要： 將 AR 技術引入機器人競賽場景，提升觀眾與參賽者的互動體驗，透過 AR 覆蓋層視覺化機器人狀態、評分資訊和競賽進度，使機器人競賽更具娛樂性與教育性。
English: Introduces AR overlays to robotics competitions to enhance spectator and participant engagement, visualizing robot status, scoring, and competition progress. Makes competitions more entertaining and educational.
Key Contribution: AR enhancement of robotics competitions for broader public engagement
#9 Towards Encountered-Type Haptic Interaction for Immersive Bilateral Telemanipulation
Authors: Kim, Anastasi, Deshpande Institution: Italian Institute of Technology (IIT)
中文摘要： 探索將「遭遇型觸覺裝置」（Encountered-Type Haptic Device）整合進沉浸式雙向遠端操控系統，使操作者在 VR 環境中能感受到真實機器人端的物理接觸力回饋，提升遠端操控的具身感。
English: Explores integrating encountered-type haptic devices into immersive bilateral telemanipulation, enabling operators to physically feel contact forces from the remote robot side within a VR environment, enhancing embodied telepresence.
Key Contribution: Encountered-type haptics for bilateral telemanipulation — more immersive force feedback than wearable haptics
#10 Unlocking the Potential of Virtual Reality in Human Robot Interaction: Insights for User Studies
Authors: Zodo, Gallhuber, Zafari, Puthenkalam, Sackl, Tscheligi Source: https://openreview.net/forum?id=y9Xpz2pydt
中文摘要： 系統性探討 VR 在 HRI 使用者研究中的應用潛力，整理設計考量與最佳實踐，提供研究者在 VR 環境中進行 HRI 使用者研究的方法論指引。
English: Systematically examines the potential of VR for HRI user studies, compiling design considerations and best practices. Provides methodological guidance for researchers conducting HRI user studies in virtual environments.
Key Contribution: Methodological framework for VR-based HRI user studies
Lightning Talk Group 3
#11 Comparing Performance between Different Implementations of ROS for Unity
Authors: Allspaw, LeMasurier, Yanco Institution: UMass Lowell Robotics Lab Source: https://openreview.net/forum?id=WH3yhsbBjj
中文摘要： 對 ROS#（WebSocket/JSON）、ROS-TCP-Connector（TCP/binary）、ROS.NET 三種 Unity↔ROS 通訊實作進行效能 benchmark。圖像傳輸測試顯示 TCP Connector 約 0.6 秒，ROS# 約 10 秒，差距達 16 倍。
English: Benchmarks ROS# (WebSocket/JSON), ROS-TCP-Connector (TCP/binary), and ROS.NET for Unity↔ROS communication. Image transmission tests show TCP Connector at ~0.6s vs ROS# at ~10s — a 16× difference attributable to JSON serialization and the Python rosbridge intermediary.
Key Contribution: Empirical performance comparison guiding Unity-ROS integration decisions
#12 Effect of Environment-Aware AR Interfaces on Task Performance in a Workspace Setting
Authors: Techasarntikul, Owaki, Shimonishi
中文摘要： 研究環境感知 AR 介面（能感知並回應工作場景中物件與人員狀態）對作業效率的影響，探討 AR 資訊覆蓋的情境適應性如何改善人機協作任務執行。
English: Studies how environment-aware AR interfaces (those that perceive and respond to workspace objects and personnel states) affect task performance in HRC settings, examining whether context-adaptive AR overlays improve collaborative task execution.
Key Contribution: Empirical evaluation of environment-aware AR on HRC task performance
#13 First Encounters with a Robot: The Value of Augmented Reality when Learning about Mobile Robots
Authors: Cleaver, Chen, Sinapov
中文摘要： 研究 AR 在使用者首次接觸行動機器人時的教育價值，探討 AR 視覺化如何幫助新使用者快速建立對機器人能力、意圖與限制的正確心智模型。
English: Studies AR’s educational value when users first encounter mobile robots, examining how AR visualizations help new users rapidly build accurate mental models of robot capabilities, intentions, and limitations.
Key Contribution: AR as onboarding tool for new robot users — mental model formation study
#14 Lessons from a Small-Scale Robot Joining Experiment in VR
Authors: Higgins, Barron, Engel, Matuszek
中文摘要： 在 VR 環境中進行小規模機器人「加入」群體實驗，研究機器人加入既有人類群體互動時的社會動態，提取 VR 實驗設計的經驗教訓與方法論發現。
English: Conducts small-scale VR experiments studying social dynamics when a robot joins an existing human group. Extracts methodological lessons for VR-based HRI experimental design.
Key Contribution: VR methodology insights for studying robot social integration in human groups
#15 MOTH: Moving Object Tracking via Head-Mounted Displays
Authors: Bösing, Puljiz, Hein Source: https://openreview.net/forum?id=p0XzP07CWT
中文摘要： 在 HoloLens 2 上實作兩種移動物體追蹤方法（體素網格法與聚類法），用於工業場所的 AR 眼鏡人機協作。實驗顯示追蹤移動人員的整體精度在 15–25 cm 之間，適合工業 AR 應用需求。
English: Implements two moving object tracking approaches (voxel-grid and cluster-based) on HoloLens 2, targeting industrial HRC via AR glasses. Tracking accuracy for moving humans is 15–25 cm, suitable for industrial AR applications.
Key Contribution: HMD-based moving object tracking for industrial HRC — 15-25 cm accuracy benchmark
#16 Testing Human-Robot Interaction in Virtual Reality: Experience from a Study on Speech Act Classification
Authors: Kaszuba, Sabbella, Leotta, Serrarens, Nardi
中文摘要： 報告在 VR 環境中進行 HRI 語音行為分類研究的經驗，探討 VR 模擬環境作為 HRI 語言互動研究平台的可行性，以及與真實場景的差距。
English: Reports experience from a VR-based HRI user study on speech act classification. Examines VR simulation as a platform for HRI language interaction research and discusses ecological validity compared to real-world settings.
Key Contribution: Practical experience report on VR for HRI speech/language studies
Lightning Talk Group 4
#17 A Text Mining Analysis of Digital Twins for HRI
Author: Camara Institution: University of York Source: https://vam-hri.github.io/papers/VAM-HRI_2023_Fanta_Camara.pdf
中文摘要： 以文字探勘技術系統性分析「數位孿生用於人機互動」的研究現狀，探索數位孿生技術在各產業中與人類互動的應用場景，以及當前 HRI 研究如何對應這些場景。
English: Applies text mining techniques to systematically analyze the research landscape of digital twins for HRI. Explores how digital twin applications across industries intersect with HRI requirements, identifying research gaps.
Key Contribution: Systematic literature landscape mapping of digital twins × HRI via text mining
#18 DARA: A Dynamic Augmented Reality Architecture for Human-Robot Interaction
Authors: Miller-Klugman, Sinapov, Cleaver Source: https://openreview.net/forum?id=VrYTExINkF
中文摘要： 提出動態 AR 架構 DARA，支援行動裝置同時連接多台不同機器人，透過資料庫管理設定並在執行時期動態建立視覺化，無需重新建置應用即可新增機器人支援。
English: Proposes DARA, a mobile AR architecture enabling simultaneous connections to multiple heterogeneous robots. Uses a database to manage configurations and instantiate visualizations at runtime, allowing new robots to be added without rebuilding the app.
Key Contribution: Runtime-configurable multi-robot AR architecture — no rebuild needed to add new robots
#19 Improving Human Legibility in Collaborative Robot Tasks through Augmented Reality and Workspace Preparation
Authors: Tung, Luebbers, Roncone, Hayes Institution: University of Colorado Boulder / CAIRO Lab Source: https://openreview.net/forum?id=pQEGt53DMx
中文摘要： 提出演算法同時優化工作空間物件配置與 AR 虛擬障礙物投影，以提升人類行為的「可讀性」（legibility），使協作機器人能更準確預測人類目標，改善任務流暢性與安全性。在桌面協作手臂與倉儲導航機器人兩個場景中驗證。
English: Algorithmic approach jointly optimizes workspace object arrangement and AR virtual obstacle projections to improve human motion legibility. Better legibility improves robot prediction of human goals, enhancing task fluency and safety. Validated in tabletop manipulation and warehouse navigation scenarios.
Key Contribution: Joint workspace + AR optimization for human legibility — fluency and safety improvement
#20 PhysicalTwin: Mixed Reality Interaction Environment for AI-Supported Assistive Robots
Authors: Pascher, Kronhardt, Gerken Source: https://openreview.net/forum?id=dz0EoKeBFW
中文摘要： 建立 MR 互動環境讓使用者透過虛擬 Kinova Jaco 手臂（含多種 AI 輔助控制方式）練習輔助機器人操作，並可透過 ROS 將虛擬手臂動作即時映射到真實機器人。提供使用者在安全虛擬環境中熟悉輔助機器人的途徑。
English: Builds a MR environment with a virtual Kinova Jaco arm supporting multiple AI-assisted control modes. The virtual arm mirrors directly to the physical robot via ROS, enabling safe training in MR before operating the real device.
Key Contribution: MR-based safe training environment for assistive robot manipulation with sim-to-real mirroring
#21 Spatial Augmented Reality User Interface for Assistive Robot Manipulation
Authors: Wilkinson, Sinclaire, Yanco Institution: UMass Lowell NERVE Center Source: https://openreview.net/forum?id=YSrYrQdZSf
中文摘要： 為安裝在電動代步車上的輔助機器手臂設計 SAR（Spatial AR）介面，以投影游標顯示可抓取物件，高亮顯示手臂可達範圍，並以視覺方式傳達機器人抓取意圖。進行中的使用者研究對比 SAR 與傳統 GUI 介面。
English: Designs a spatial AR interface for a mobility scooter-mounted assistive arm. Projects a joystick-controlled cursor into the workspace, highlights the arm’s reachable surface, and visualizes grasp intent. Ongoing user study compares SAR vs. traditional GUI.
Key Contribution: SAR interface for assistive manipulation — workspace visualization + intent communication
Thematic Analysis
Five Research Axes
| 主軸 / Theme | 相關論文 / Papers |
|---|---|
| 輔助機器人與無障礙 / Assistive Robotics & Accessibility | #5, #7, #20, #21 |
| 遙操作介面 / Teleoperation Interfaces | #2, #4, #9 |
| AR 輔助示範學習 / AR for LfD & Robot Teaching | #3, #12, #19 |
| 工具基礎建設 / Infrastructure & Tooling | #1, #11, #18 |
| HRI 方法論 / HRI Methodology | #6, #10, #13, #14, #16, #17 |
Observed Trends
-
輔助機器人是本屆最大主題： #5、#7、#20、#21 都在問「如何讓行動不便或特殊需求使用者透過 VR/AR 更好地與機器人互動」，這在 2025 年版中較少出現。
-
HRI 方法論研究密集： 超過四分之一的論文（#6、#10、#13、#14、#16）聚焦於「如何用 VR/AR 做 HRI 研究」本身，反映社群對 VR 作為研究工具的信心與反思。
-
多機構 Tufts/UMass Lowell 合作網絡： Cleaver、Sinapov（Tufts）和 Allspaw、Yanco（UMass Lowell）在多篇論文出現，形成明顯的跨年研究線。
-
工業 AR 實用化： #1（IO 接線）、#8（機器人競賽）、#15（工廠移動追蹤）均為直接可落地的工業/教育 AR 應用。
-
聽覺化（Sonification）作為設計策略： #2 是少見的以聲音為主要 AR 模態的研究，適用於視覺資訊飽和的高風險環境。
報告整理自 VAM-HRI 2024 官方 Program 頁面及各論文的 OpenReview / 作者機構頁面 Workshop 官網：https://vam-hri.github.io/previous/2024/program