Imitation Learning Data Quality Metrics
Sources: Session notes, 2026-04-05 Raw: arXiv 2510.10903 Data Quality Evaluation Updated: 2026-08-12
Overview
Raw robot demonstration datasets contain noise, redundancy, and domain imbalance. §7.1.2 of arXiv:2510.10903 surveys the most complete academic treatment of data quality evaluation for imitation learning. The central difficulty: no single metric captures “quality” — joint evaluation across action consistency, distributional diversity, and domain balance is required. Traditional training loss metrics do not predict physical success rate.
Core Quality Metrics
DC-IL: Action Divergence + Transition Diversity
The most direct quantitative definition of “data quality” in the survey.
- Action divergence: measures consistency of the action distribution within demonstrations. High divergence indicates contradictory demonstrations (poor quality).
- Transition diversity: measures coverage of state-action pair space, used to suppress distribution shift.
These two metrics can assess a batch of demonstrations without executing the robot, making DC-IL the closest available proxy for “data quality score” that can be computed offline.
EIL: Temporal Cycle Consistency Filtering
Learns action-conditioned embeddings with unsupervised voting alignment. Automatically filters trajectory segments that are inconsistent with task intent. A trajectory that cannot form a temporally consistent cycle in latent space is treated as noise and removed. Unsupervised — no human labeling required.
L2D: Preference Learning
Represents heterogeneous human demonstrations as latent trajectory embeddings, then uses preference learning to rank demonstration quality. Selects higher-quality examples for offline imitation learning. Improves robustness under mixed-quality data.
Re-Mix: Minimax Domain Reweighting
Frames data curation as minimax reweighting over domain mixtures, using excess behavior-cloning loss to automatically up-weight beneficial domains and down-weight harmful ones. Operates at domain level rather than individual demonstration level — optimizes the composition of the dataset across sources.
Additional Methods
- ILID (State Discriminator Scoring): trains a state-only discriminator to score resulting states from state-action pairs; filters for high-value state-action transitions.
- MimicLabs: prioritizes camera pose diversity and spatial diversity in data composition and retrieval — diversity over dataset scale.
- UVP (Pretraining Distribution): counterintuitive finding: the distribution of pretraining images matters more than dataset size. One quality dimension is whether the data distribution matches the target task, independent of raw demonstration quality.
Five Data Utilization Strategies
Quality evaluation is one component of a broader five-strategy framework from §7.1.2:
| Strategy | Representative Methods | Core Idea |
|---|---|---|
| Data Selection | EIL, L2D, Re-Mix, DC-IL, UVP, ILID, MimicLabs | Filter noise, remove redundancy, adjust domain balance |
| Data Retrieval | VINN, SAILOR, DINOBot, STRAP | Mine task-relevant demonstrations from existing datasets |
| Data Augmentation | DAAG, GenAug, RoVi-Aug | Label relabeling, geometric augmentation, generative cross-embodiment augmentation |
| Data Expansion | JUICER, Diff-DAGger, SAFARI | Generate synthetic trajectories, decompose and recombine skills |
| Data Reweighting | FABCO, PLARE, Beliaev et al. | Weight by feasibility, VLM preference, or operator skill level |
Industry vs. Academic Quality Frameworks
The ML6 practitioner framework (from LeRobot field reports) identifies four empirical quality factors: accuracy, controlled sequences, comprehensive coverage, and robustness. These align directionally with the academic metrics: DC-IL’s action divergence corresponds to accuracy/controlled sequences; transition diversity corresponds to comprehensive coverage.
Both frameworks converge on the same conclusion: training loss is insufficient to evaluate quality; richer proxy signals are needed. L2D uses preference learning and ILID uses state discriminators as alternatives to loss-based quality estimation.
Open Questions
- Does DC-IL’s action divergence metric apply equivalently to joint-angle action spaces and end-effector action spaces, or does the metric need to be adapted?
- L2D ranks demonstration quality via preference learning — does this require human-annotated preference pairs, or can the preferences be generated automatically from task success signals?
- Re-Mix minimax reweighting: is the method stable on small datasets (<500 episodes) where the domain mixture estimate is noisy?
- ILID’s state discriminator requires positive samples (states from successful task completion) — how are these obtained in practice when success is rare during data collection?