Closing the Loop in Teleoperation: Episode-Level Data Quality Assessment and Feedback for High-Quality Demonstration Collection
Authors: Gokul Narayanan, Yash Shahapurkar, Melih Erdogan, Brian Zhu, Eugen Solowjow (Siemens Corporation)
arXiv: 2605.26349v1 [cs.RO] — May 25, 2026
Abstract
The paper addresses a critical challenge in robotic data collection: obtaining high-quality teleoperated demonstrations. While novice operators can complete tasks successfully, their demonstrations often contain suboptimal execution patterns—jerky motions, inefficient corrections, and operations near robot joint limits—that degrade downstream policy learning. The authors introduce the Data Quality Assessment and Feedback (DQAF) framework, which evaluates episodes immediately after completion and provides actionable natural-language guidance to operators.
Rather than offering only binary success/failure feedback, the system “explains why an episode is suboptimal and highlights specific behaviors to correct.” The framework combines semantic task progress analysis with robot telemetry metrics to identify quality issues and guide operator improvement during data collection itself.
Main Contributions
-
Multimodal Quality Assessment Framework – Integrates semantic task progress from Vision-Language Models with telemetry-based execution metrics, task context, and expert reference demonstrations.
-
Natural Language Feedback Generation – Translates assessment results into structured, actionable guidance for teleoperators grounded in semantic and telemetry evidence.
-
Empirical Validation – Two-stage evaluation comprising diagnostic validation against expert review and a pilot user study demonstrating faster operator improvement with immediate feedback.
Methodology
Pipeline Architecture
Stage 1: Semantic Analysis
- Uses Vision-Language Models (Gemini Flash 1.5) to analyze visual observations
- Estimates subtask progression and completion percentages
- Detects anomalies including semantic regressions and backtracking
- Processes frames at 2.5-second intervals against expert reference demonstrations
Stage 2: Telemetry Quality Analysis — Four Key Metrics
- Action Range Saturation – Measures how frequently commands approach joint limits
- Motion Smoothness – Quantifies jerkiness via Log Dimensionless Jerk (LDLJ)
- Gripper Chatter – Counts excessive open/close transitions
- Stalling (Static Fraction) – Identifies prolonged periods of minimal action
Stage 3: Cross-Modal Evidence Construction
- Aligns telemetry violations with semantic events
- Creates temporally-linked diagnostic records
Stage 4: Feedback Synthesis
- Constrained LLM (Gemini 3 Pro) generates natural-language feedback
- Specifies what failed, where it occurred, and how to improve
System Requirements
- R1 – Actionable post-episode guidance (what failed, when, how to correct)
- R2 – Evaluate both high-level task success and low-level execution quality
- R3 – Robustness to transients; persistent violations penalized more
- R4 – Outputs support reproducible filtering and dataset curation
- R5 – All quality decisions traceable to specific metrics and evidence
Experimental Results
Validation Study (100 episodes, Unitree G1 humanoid)
- Failure Detection Recall: 85.7% (24/28)
- False Positives: 2 successful episodes incorrectly flagged
- Processing Latency: ~43 seconds post-episode (~20s with concurrent processing)
Pilot User Study (3 novice operators)
| Condition | DQAF Score | Error Count | Success Rate |
|---|---|---|---|
| With Feedback | 8.8 ± 0.5 | 1.0 ± 0.2 | 80–93% |
| No Feedback | 5.6–8.2 | 2.9 ± 0.4 | 53–67% |
Hardware & Implementation
- Robot: Unitree G1 humanoid
- Interface: Meta Quest hand-tracking via Unitree XR teleoperation stack
- VLM: Gemini Flash 1.5 (semantic analysis), Gemini 3 Pro (feedback generation)
- Tasks: Pick-and-place, item handover
Limitations
- Semantic analysis sensitive to occlusions and ambiguous subtask boundaries
- Telemetry thresholds calibrated per-task, lack adaptive formulation
- Feedback only post-episode; no real-time intervention
- Pilot study small in scale