Quartz 5

lerobot-framework

3 min read

LeRobot: HuggingFace Robot Learning Framework

Sources: Research synthesis, 2026-04-05; ML6 field report, 2026 Raw: LeRobot Tools and Use Cases Updated: 2026-08-12

Overview

LeRobot is HuggingFace’s open-source end-to-end robot learning library — the first platform to unify hardware control, demonstration collection, dataset management, policy training, and inference in a single Python framework. As of September 2025, it hosts 16,000+ datasets from 2,200+ contributors with 3.9M+ episodes, making it the largest open repository of robot learning data in the world.

Architecture

Four layers form the LeRobot stack:

  1. Hardware abstraction: a unified Robot Python class that works across diverse platforms from the €225 SO-100 arm to humanoid robots. v0.4.0’s plugin system makes adding new hardware a single pip install lerobot_robot_xyz.

  2. Dataset format (LeRobotDataset v3.0): Parquet tables for observations/actions, MP4 for video, JSON metadata. Chunked episode format enables streaming 400GB+ datasets (Open X-Embodiment) without full download. StreamingLeRobotDataset provides bounded memory usage regardless of dataset size.

  3. Policies: implements ACT, Diffusion Policy, SmolVLA, π₀/π0.5, GR00T N1.5, TD-MPC, and SERL. Same training and inference code works across all hardware platforms.

  4. Training infrastructure: single or multi-GPU via HuggingFace Accelerate. Multi-GPU gives approximately linear speedup.

Policy Comparison

PolicyTypeParamsInference latencyBest for
ACTBehavioral cloning (action chunking)52M~5ms (RTX 4090)Precise, repetitive tasks
Diffusion PolicyDiffusion-based BC~80M~50msComplex multimodal distributions
SmolVLAVLA (pretrained)450M~200msGeneral manipulation + language
π₀ / π0.5VLA (Physical Intelligence)3.5B / 4B~500msOpen-world generalization
GR00T N1.5VLA (NVIDIA)3B~500msCross-embodiment, language following

ACT vs VLA in practice: ACT achieves 90% success on fixed-setup positional tasks but fails under distribution shift. VLAs (GR00T-N1, SmolVLA) generalize to semantic variation and deformable objects but require careful latency management. Choosing between them is a task complexity vs. compute tradeoff.

The Async Inference Architecture

The key technical enabler for affordable capable robots: action prediction is decoupled from execution. The policy runs on a remote server and streams actions locally. A €225 SO-100 arm can therefore run a 3.5B parameter VLA model. Without this, cost and capability would be in direct conflict.

Community Data Flywheel

Pretraining SmolVLA on 481 SO-100 community datasets raised task success from 51.7% to 78.3% — a 26.6 point improvement from community data alone. This demonstrates that community-contributed diversity is a meaningful training signal, not just noise.

Standard Workflow

# 1. Collect demonstrations
lerobot-record --robot.type=so101_follower --teleop.type=so100_leader \
  --dataset.repo_id=user/my_task --dataset.num_episodes=50
 
# 2. Fine-tune SmolVLA
lerobot-train --policy.path=lerobot/smolvla_base \
  --dataset.repo_id=user/my_task --steps=20000
 
# 3. Evaluate
lerobot-record --robot.type=so101_follower \
  --policy.path=user/my_trained_policy --dataset.num_episodes=20

Training SmolVLA takes ~4 hours on an A100.

Critical Assessment

What the ML6 field report found (from actual deployment on SO-ARM100):

  • Loss curves do not predict physical success. A low training loss does not mean the robot will succeed.
  • Millimeter-level errors cause task failure. The margin is narrow.
  • Bad demonstrations hurt more than fewer good ones. Dataset quality dominates quantity.
  • The four critical data factors: accuracy of demonstrations, controlled motion sequences, comprehensive task variation coverage, and robustness (include error recovery demonstrations).

Structural limitations:

  • Evaluation is non-standardized — most results are lab-specific percentages with non-reproducible setup
  • ACT has zero generalization to distribution shift; production use requires extensive data engineering
  • VLA inference latency still causes stuttering without async setup

Use Cases

  • Pick-and-place (most common): ACT achieves 90% on 5-position tasks; SmolVLA 78.3% after fine-tuning on 50 episodes
  • Deformable object manipulation: GR00T-N1 handles textile spreading and towel folding at 60-80% success
  • Mobile manipulation: LeKiwi mobile base + arm + unified API
  • Research benchmarking: LIBERO (130+ VLA tasks) and Meta-World (50+ tasks) integrated directly
  • Educational robotics: €225 SO-100 + LeRobot + HuggingFace open course

See Also

Graph View

Backlinks