Precise Manipulation with Efficient Online RL
Overview
Physical Intelligence researchers introduce RLT (RL tokens), a method enabling robots to rapidly improve precision tasks through online reinforcement learning. The approach achieves up to 3× speed improvements on critical manipulation phases using just minutes to hours of real-world data.
Key Innovation
The method works by extracting a compressed “RL token” from Vision-Language-Action (VLA) models. This token summarizes internal representations and serves as input to lightweight actor-critic networks trained directly on the robot. As the researchers explain, “the actor receives the VLA’s predicted action as input, so it learns to edit the VLA action rather than replace it entirely.”
Technical Approach
Training Process:
- An encoder-decoder transformer learns to reconstruct VLA embeddings through a bottleneck
- The resulting RL token acts as a compact state representation
- Small actor and critic networks train via sample-efficient off-policy RL
- Updates occur at hundreds per second directly onboard
Design Decisions: The policy predicts action chunks matching the VLA’s structure, maintaining temporal consistency. Regularization keeps exploration near baseline behavior when reasonable, deviating only when improvements are identified.
Experimental Results
Evaluated on four precision tasks:
| Task | Base Model | RLT |
|---|---|---|
| Screwdriver | 1.7 successes/10min | 14 |
| Zip Tie | 2.8 | 13 |
| Ethernet | 147 | 400 |
| Charger | 136 | 600 |
RLT achieved faster execution than human teleoperation on ethernet insertion tasks, with 50% of trials exceeding all human demonstration speeds.
Real-World Applications
- Sub-millimeter precision screw alignment
- Delicate connector insertion
- Contact-rich manipulation phases
Future Direction
The researchers envision multi-level adaptation: fine-grained behavior refinement, large-scale model retraining, and RL-based reasoning improvements — enabling robots to continuously improve through real-world deployment experience.