Abstract
This systematic review synthesizes 86 peer-reviewed articles examining how large language models transform human-robot interaction, following PRISMA 2020 guidelines. The authors find that LLMs are reshaping foundational HRI capabilities around contextual sensing, socially-grounded interaction generation, and continuous human alignment. Current research remains largely exploratory, with varied experimental designs, methodologies, and evaluation approaches. The review identifies eight key design challenges and future research priorities at the LLM-HRI intersection.
arXiv: https://arxiv.org/abs/2602.15063
Core Research Questions
- RQ1: How do LLMs transform foundational HRI capabilities?
- RQ2: How are LLMs integrated into HRI system design?
- RQ3: What evaluation approaches work for LLM-driven HRI?
- RQ4: What opportunities and challenges exist for future research?
The Sense-Interaction-Alignment Framework
The central contribution — replacing the classical “Sense-Plan-Act” model with:
Sense: Robots ground LLM capabilities in physical/social context via multimodal perception (static prompts → dynamic sensor-to-text pipelines → vision-language models).
Interaction: LLMs enable generative, agentic collaboration beyond scripted responses — persona adaptation, social expressiveness, task co-creation.
Alignment: Continuous optimization through personalization and repair mechanisms (behavioral, emotional, ethical) ensuring robots stay congruent with human needs over time.
Nine-Dimension Taxonomy (with Inter-Rater Reliability)
| Dimension | IRR (Cohen’s κ) |
|---|---|
| Contextual Perception & Understanding | 0.720 |
| Generative & Agentic Interaction | 0.796 |
| Iterative Optimization & Alignment | 0.684 |
| Modality & Interaction Channels | 0.768 |
| Morphology | 0.894 |
| Autonomy Levels | 0.846 |
| Methodology | 0.684 |
| Evaluation Metrics | 0.832 |
| Application Domains | 0.904 |
Contextual Perception Sub-categories
- Static/semi-static context injection
- Modular perception with textual abstraction
- Integrated visual-language reasoning
- Emotional grounding
- Task intent formulation
- Human model alignment (Theory-of-Mind)
Generative Interaction Sub-categories
- Persona adaptation and conversational fluidity
- Embodied social expressiveness
- Task-oriented planning and execution
- Creative storytelling and social engagement
- Social initiation
- Anticipatory assistance
Alignment Sub-categories
- Sustained personalization
- Episodic memory integration
- Behavioral repair
- Emotional repair
- Ethical/normative alignment repair
Publication Trends
- LLM-driven HRI research began in 2019 (1 paper)
- 108 publications in 2023; 311 in 2024
- 86 of the reviewed papers concentrated in 2024–2025
- Venues: HRI (31.4%), CHI (12.8%), ROMAN (10.5%), IJSR (8.1%)
Eight Key Design Challenges
- LLM-driven understanding reliability in noisy real-world environments
- Multimodal emotional perception requiring robust fusion
- Trust calibration managing capability expectations
- Long-term engagement sustaining interaction quality
- Computational overhead balancing capability vs. resource constraints
- Algorithmic bias and data-leakage risks
- Ethical alignment in sensitive domains
- Lossless multimodal grounding (deep fusion vs. sequential processing)
Critical Research Gaps
- User modeling beyond generic personas under-investigated
- Autonomy levels and human oversight mechanisms insufficiently studied
- Fragmented evaluation with no standardized metrics
- Almost no longitudinal studies on sustained alignment
Recommendations
Methodological: Field deployments > lab-only; standardized metrics; longitudinal studies; transparent system documentation.
Design: Robust multimodal perception; repair mechanisms (behavioral/emotional/ethical); human-in-the-loop optimization; diverse populations.
Technical: Visual-language reasoning for real-time understanding; Theory-of-Mind grounded in empirical human models; memory for sustained personalization; confidence estimation for safety-critical LLM outputs.
Scope Definitions
- Robot: Physical or simulated (VR/AR) embodiment with sensors/actuators. Excludes disembodied chatbots.
- HRI types covered: supervisory control, teleoperation, automated vehicles, social interaction — grounded in empirical user studies.
- LLM: Transformer-based models with billions of parameters, including VLMs and multimodal LLMs.