bot_vault

LeRobot: HuggingFace 機器人學習框架 — 工具、實作與應用場景

14 min read

LeRobot: HuggingFace Robotics Framework — Tools, Implementations, and Use Cases

Research Question

What tools and implementations does LeRobot provide, and what are its real-world use cases in robotics research and deployment?

Knowledge Map

  • Imitation Learning (模仿學習) — LeRobot’s primary training paradigm. Requires understanding behavioral cloning, action chunking (ACT), and data collection via teleoperation. All LeRobot policies are trained from human demonstrations.
  • Vision-Language-Action Models (VLA) — Foundation models that take camera images + language instructions + proprioceptive state as input and output robot actions. Essential for understanding SmolVLA, π₀, and GR00T N1.5.
  • Diffusion-Based Policy Learning — Diffusion Policy starts from Gaussian noise and iteratively denoises to generate action sequences. Contrasts with ACT’s deterministic action chunking.
  • Robot Hardware Interfaces — Understanding motor control, serial communication (USB-TTL), and the difference between position/velocity/torque control modes is prerequisite for real robot deployment with LeRobot.
  • Dataset Formats (Parquet + MP4) — LeRobotDataset stores observations as Parquet tables and video as compressed MP4. Knowledge of streaming data pipelines matters for large-scale training.
  • PyTorch + Accelerate — All LeRobot models are pure PyTorch. Multi-GPU training uses HuggingFace Accelerate. Familiarity with these is assumed.
  • Simulation Environments — LIBERO and Meta-World provide sim-to-real bridges. Understanding Gym/MuJoCo APIs is useful for simulation-based evaluation.

Sources Gathered

New sources clipped and analyzed during this research:

Existing vault notes referenced:

Key Findings

  1. LeRobot is a full-stack robotics learning platform, not just a training library. It unifies hardware control, data collection, dataset management, policy training, and inference in a single Python framework — the first OSS project to do this end-to-end.

  2. The async inference architecture solves the cost/capability mismatch. A €225 SO-100 arm can run a 3.5B parameter VLA model because action prediction is decoupled from execution — the policy runs on a remote server and streams actions locally. This is the key technical enabler for affordable capable robots.

  3. Community data is the real moat. 16,000+ datasets from 2,200+ contributors created a flywheel: pretraining SmolVLA on 481 SO-100 datasets raised task success from 51.7% to 78.3% — a 26.6 point improvement from community data alone.

  4. ACT and VLAs occupy distinct niches in practice. ACT (52M params) achieves 90% on fixed-setup positional tasks but fails on distribution shift. VLAs (GR00T-N1, SmolVLA) generalize to semantic variation and deformable objects but require careful latency management. Choosing between them is a task complexity vs. compute tradeoff.

  5. Data quality trumps model sophistication. The ML6 field report is unambiguous: loss curves don’t predict physical success, millimeter-level errors cause failure, and bad demonstrations hurt more than fewer good ones. The four critical factors are: accuracy, controlled sequences, comprehensive coverage, and robustness (include error recovery).

Open Questions

  • How does SmolVLA transfer to hardware other than SO-100/101? The pretraining is SO-100-heavy — cross-embodiment generalization is untested in public benchmarks.
  • What is the production reliability ceiling for ACT in industrial settings? ML6’s 90% on 5-position tasks is promising but evaluation methodology is non-standardized.
  • How does π0.5’s open-world generalization (trained on internet + robot data) compare to GR00T N1.5’s cross-embodiment approach for new robot types?
  • Does the plugin system’s pip install pattern actually scale? The ecosystem is new and hardware packages need maintenance as core LeRobot APIs evolve.

Report

What is LeRobot?

LeRobot is HuggingFace’s open-source end-to-end robot learning library, built in PyTorch. Released publicly in early 2024 and reaching v0.4.0 by late 2025, it aims to do for robot learning what the HuggingFace transformers library did for NLP: standardize the full pipeline from data to deployment.

The library covers four layers: (1) a unified hardware abstraction layer for real robot control, (2) the LeRobotDataset format for storing and streaming demonstration data, (3) a clean implementation of state-of-the-art policies, and (4) a training + inference stack with multi-GPU support.

As of September 2025, the platform hosts 16,000+ datasets from 2,200+ contributors with 3.9M+ episodes — making it the largest open repository of robot learning data in the world.

Tools Overview

Hardware Interface Layer

LeRobot provides a unified Robot Python class that abstracts over diverse hardware:

RobotTypeCost
SO-100 / SO-101Low-cost arm~€225
Koch v1.1Tabletop manipulator~€500
ALOHA-2Bimanual manipulation~$20k
LeKiwiMobile manipulationCustom
Unitree G1HumanoidCommercial
Reachy 2Full-body robotCommercial

The same training and inference code works across all these platforms — the hardware layer is swappable.

v0.4.0 Plugin System: Third-party hardware is now added via pip install lerobot_robot_xyz, eliminating the need to modify the core library. Phone teleoperation (iOS/Android) is available as a plugin out of the box.

Dataset Tools

LeRobotDataset format: Parquet tables for observations/actions + MP4 for video. All metadata unified in Parquet. Compatible with Hugging Face Hub for sharing.

v3.0 upgrades:

  • Chunked episode format: streams massive datasets (OXE >400GB) without full download
  • StreamingLeRobotDataset: bounded memory usage regardless of dataset size

lerobot-edit-dataset CLI:

# Merge datasets
lerobot-edit-dataset --operation.type merge --operation.repo_ids "['user/ds1', 'user/ds2']"
 
# Delete specific episodes
lerobot-edit-dataset --operation.type delete_episodes --operation.episode_indices "[0, 2, 5]"
 
# Split by fraction
lerobot-edit-dataset --operation.type split --operation.fraction 0.8

Policy Implementations

PolicyTypeParamsInferenceBest for
ACTBehavioral cloning (action chunking)52M~5ms (RTX 4090)Precise, repetitive tasks
Diffusion PolicyDiffusion-based BC~80M~50msComplex, multimodal distributions
SmolVLAVLA (pretrained)450M~200msGeneral manipulation + language
π₀ / π0.5VLA (Physical Intelligence)3.5B / 4B~500msOpen-world generalization
GR00T N1.5VLA (NVIDIA)3B~500msCross-embodiment, language following
TD-MPCModel-based RLVariable—Online RL tasks
SERLSample-efficient RLVariable—Real-world RL with resets

Training Infrastructure

# Single GPU
lerobot-train --policy.type=smolvla --dataset.repo_id=user/my_dataset --steps=20000
 
# Multi-GPU (Accelerate)
accelerate launch --multi_gpu --num_processes=4 $(which lerobot-train) \
  --dataset.repo_id=user/dataset --policy.type=act

Multi-GPU training via Accelerate gives linear speedup: 2 GPUs = ~50% time reduction.

Processors Pipeline (v0.4.0):

  • PolicyProcessorPipeline: handles batched tensors for training/inference (normalize, tokenize, move device)
  • RobotProcessorPipeline: handles individual data points for real-time control (unnormalize, action-space conversion)

Implementation Workflow

The standard LeRobot workflow for a new robot learning task:

1. Hardware Setup

pip install lerobot
# For plugin hardware:
pip install lerobot_robot_so100

Calibrate robot joints, configure cameras, set up leader/follower arm pairing for teleoperation.

2. Data Collection

lerobot-record --robot.type=so101_follower --teleop.type=so100_leader \
  --dataset.repo_id=user/my_task --dataset.num_episodes=50

Recommended: 50+ episodes, 10+ per task variation. Record at 30fps with 2–3 camera views.

3. Dataset Visualization

lerobot-visualize-dataset --repo_id=user/my_task --episode_index=0

4. Training

# Fine-tune SmolVLA (recommended for new tasks)
lerobot-train --policy.path=lerobot/smolvla_base \
  --dataset.repo_id=user/my_task --steps=20000
 
# Train ACT from scratch
lerobot-train --policy.type=act --dataset.repo_id=user/my_task --steps=80000

5. Evaluation

lerobot-record --robot.type=so101_follower \
  --policy.path=user/my_trained_policy --dataset.num_episodes=20

Use Cases

1. Pick-and-Place (Most Common)

The dominant use case in the community. ACT achieves 90% on 5-position pick-and-place with ~46k frames (~25 min teleoperation). SmolVLA reaches 78.3% on SO-100 pick-place after fine-tuning on 50 episodes.

Target industries: bin-picking, assembly, logistics.

2. Deformable Object Manipulation

VLA models (GR00T-N1) handle textile spreading and towel folding at 60–80% success. This is beyond ACT’s capability — deformable objects have too many degrees of freedom for pure position-based policies.

Target industries: laundry automation, food handling, medical supply.

3. Mobile Manipulation

LeKiwi (mobile base + arm) + LeRobot enables combined navigation and manipulation. The unified hardware API means mobile platforms use the same training infrastructure as static arms.

4. Research Benchmarking

LIBERO (130+ VLA tasks) and Meta-World (50+ tasks) are integrated directly. Researchers can train and evaluate policies in simulation with standardized metrics.

5. Educational Robotics

The €225 SO-100 arm + LeRobot provides a complete robot learning curriculum at accessible cost. HuggingFace publishes an open robotics course covering classical control, imitation learning, RL, and VLAs.

6. Teleoperation Data Collection at Scale

The community data flywheel: 16K+ datasets contributed by 2,200+ users. The standardized format means datasets from different labs are directly combinable for pretraining.

Critical Assessment

Strengths:

  • Lowest barrier-to-entry in robot learning: €225 hardware, pip install, Colab fine-tuning
  • Community data scale is unprecedented and growing
  • Async inference architecture solves the cost/compute mismatch elegantly
  • Plugin system will accelerate hardware ecosystem expansion

Limitations:

  • Evaluation is non-standardized: loss ≠ physical success; most results reported as human-judged percentages in non-reproducible lab setups
  • ACT has zero generalization to distribution shift — production robustness requires extensive data engineering
  • Inference latency for large VLAs (π₀, GR00T N1.5) still causes stuttering without careful async setup
  • Plugin ecosystem is nascent — community hardware packages may have inconsistent maintenance

Near-term trajectory: GR00T N1.5 post-training on Jetson Thor + async inference + LeRobot’s data tooling makes 2025-2026 the likely window when LeRobot-based systems first enter controlled industrial deployment.

中文版

研究問題

LeRobot 提供哪些工具與實作？在實際機器人研究和部署中的應用場景為何？

知識地圖

  • 模仿學習 — LeRobot 的主要訓練範式。需要理解行為克隆、動作分塊（ACT）和遙操作資料收集。
  • 視覺-語言-動作模型（VLA） — 接受相機影像、語言指令和本體感知狀態作為輸入並輸出機器人動作的基礎模型。
  • 擴散策略學習 — 從高斯噪聲開始，迭代去噪生成動作序列。與 ACT 的確定性動作分塊形成對比。
  • 機器人硬體介面 — 電機控制、串列通訊和位置/速度/扭矩控制模式的知識是部署前提。
  • 資料集格式（Parquet + MP4） — LeRobotDataset 以 Parquet 表格儲存觀測值，以壓縮 MP4 儲存視訊。
  • PyTorch + Accelerate — 所有 LeRobot 模型為純 PyTorch；多 GPU 訓練使用 HuggingFace Accelerate。
  • 模擬環境 — LIBERO 和 Meta-World 提供 sim-to-real 橋接。

關鍵發現

  1. LeRobot 是全棧機器人學習平台，而非單純訓練函式庫。它統一了硬體控制、資料收集、資料集管理、策略訓練和推論為單一 Python 框架。

  2. 非同步推論架構解決了成本/能力不匹配問題。一支 €225 的 SO-100 機械臂可以執行 3.5B 參數的 VLA 模型，因為動作預測與執行解耦——策略在遠端伺服器運行並將動作串流傳回本地。

  3. 社群資料是真正的護城河。16,000+ 個資料集來自 2,200+ 貢獻者，創造了飛輪效應：在 481 個 SO-100 資料集上預訓練 SmolVLA 將任務成功率從 51.7% 提升到 78.3%。

  4. ACT 和 VLA 在實踐中佔據不同生態位。ACT（52M 參數）在固定設置位置任務上達到 90% 但在分佈偏移時失敗。VLA 能泛化到語義變化和可變形物體，但需要仔細管理延遲。

  5. 資料品質勝過模型複雜度。損失曲線不預測物理成功；毫米級誤差導致操作失敗；糟糕的示範比較少的優質示範更有害。

未解問題

  • SmolVLA 如何遷移到 SO-100/101 以外的硬體？預訓練資料以 SO-100 為主，跨體態泛化在公開基準中未經測試。
  • 工業環境中 ACT 的生產可靠性上限是什麼？
  • π0.5 的開放世界泛化（在網路+機器人資料上訓練）與 GR00T N1.5 的跨體態方法對新機器人類型的比較如何？
  • 插件系統的 pip install 模式是否真的能規模化？

報告

LeRobot 是什麼？

LeRobot 是 HuggingFace 的開源端到端機器人學習函式庫，以 PyTorch 構建。2024 年初公開發布，2025 年底發布 v0.4.0，目標是為機器人學習做 HuggingFace transformers 為 NLP 所做的事：標準化從資料到部署的完整流程。

截至 2025 年 9 月，平台託管來自 2,200+ 貢獻者的 16,000+ 個資料集，擁有 3.9M+ 個 episode——是世界上最大的開放機器人學習資料庫。

工具概覽

硬體支援涵蓋從 €225 的 SO-100 低成本機械臂到 ALOHA-2 雙臂操作器、LeKiwi 移動平台和 Unitree G1 人形機器人。v0.4.0 的插件系統讓新增第三方硬體變成 pip install lerobot_robot_xyz。

策略實作從輕量到強大：ACT（52M 參數，~5ms 推論）適合精確重複任務；SmolVLA（450M 參數）是最佳的入門 VLA 選擇；π₀/π0.5（4B）和 GR00T N1.5（3B）提供開放世界泛化能力。

資料集工具（lerobot-edit-dataset CLI）支援合併、分割、刪除 episode 和新增功能。Datasets v3.0 的分塊格式使 >400GB 的大型資料集可串流使用。

實作工作流程

標準流程：(1) 安裝並校準硬體 → (2) 用 lerobot-record + 遙操作收集 50+ 個 episode → (3) 用 lerobot-train 訓練策略（SmolVLA 在 A100 上約 4 小時）→ (4) 用 lerobot-record --policy.path=... 評估和部署。

應用場景

  • 拾取放置（最普遍）：ACT 在 5 位置任務上達到 90%，SmolVLA 在 SO-100 上達到 78.3%
  • 可變形物體操作：GR00T-N1 處理紡織品鋪展（60%）和毛巾折疊（80%）
  • 移動操作：LeKiwi 移動底座 + 機械臂的組合導航與操作
  • 研究基準測試：LIBERO（130+ VLA 任務）和 Meta-World（50+ 任務）直接整合
  • 教育機器人：€225 SO-100 + LeRobot + HuggingFace 開放課程提供完整學習路徑
  • 大規模資料收集：標準化格式使不同實驗室的資料集可直接合併用於預訓練

批判性評估

優勢：最低的機器人學習門檻（€225 硬體、pip 安裝、Colab 微調）；社群資料規模空前；非同步推論架構優雅解決成本/計算不匹配。

限制：評估未標準化（損失 ≠ 物理成功）；ACT 對分佈偏移無泛化能力；大型 VLA 的推論延遲在沒有非同步設置的情況下仍導致運動抖動；插件生態系統還在起步階段。

近期發展方向：GR00T N1.5 在 Jetson Thor 上的後訓練 + 非同步推論 + LeRobot 資料工具使 2025-2026 年成為基於 LeRobot 系統首次進入受控工業部署的可能視窗。

Graph View

Backlinks