Abstract
We introduce ParticleFormer, a Transformer-based point cloud world model for robotic manipulation involving multiple objects and materials. It uses a hybrid point cloud reconstruction loss supervising both global and local dynamics features, trained directly on real-world robot perception data without requiring elaborate 3D scene reconstruction. Validated on 6 simulation and 3 real-world tasks, it consistently outperforms GNN-based baselines in dynamics prediction and downstream visuomotor control.
Introduction
Existing 3D world models rely on Graph Neural Networks with particle representations, which are sensitive to hyperparameter tuning (especially TopK graph connectivity) and require costly dynamic Gaussian Splatting reconstruction. These methods constrain particle interaction learning to graph topology, making them inflexible for multi-material scenarios combining rigid, deformable, and granular objects.
ParticleFormer addresses these issues through:
- A Transformer-based backbone that learns particle interactions implicitly via attention — no manual topology hyperparameters
- A hybrid loss combining Chamfer Distance (local, per-point) + Hausdorff Distance (global, worst-case), applied directly to point cloud streams
Published at CoRL / PMLR 2025. Project: suninghuang19.github.io/particleformer_page/
Method
Problem Formulation: x_{t+1} = f(x_t, u_t, M) where x_t = object/end-effector point clouds, u_t = applied motion, M = material type encodings.
Observation Embedding: Stereo vision (FoundationStereo) + semantic segmentation (GroundingDINO + SAM) to extract clean per-object point clouds. Each particle embeds position, material encoding, and motion into latent representations.
Dynamics Transition: L=3 multi-head self-attention layers. All particles attend to all others — interaction structure learned implicitly, no graph topology needed.
Motion Prediction: Shared decoder predicts per-particle displacement.
Hybrid Loss: L = α·L_CD + (1-α)·L_HD. Chamfer Distance for local nearest-neighbor accuracy; Hausdorff Distance penalizes worst-case global deviations. Trained with k=5 autoregressive rollout steps.
Planning: Integrates with MPPI (Model Predictive Path Integral) for MPC-based downstream control.
Experiments
Simulation (6 tasks): NVIDIA FleX — rigid, deformable, granular material combinations
- Box Pushing (rigid)
- Cloth Gathering (deformable + granular)
- Rope Sweeping (two-arm, rope dynamics)
Real-world (3 tasks): UFACTORY xArm-6 + ZED-2i stereo cameras
Baselines: GBND (GNN-based), Ours w/o Hybrid (CD only), DINO-WM (2D pixel space)
Key Results:
- MSE: 0.0023 vs 0.0076 (GBND) on Cloth Gathering
- Consistent CD and CD+HD improvement across all tasks
- GBND sensitive to TopK hyperparameter; ParticleFormer avoids this entirely
- Real-world MPC shows lower final-state error and more fine-grained manipulation
Limitations
- Trained per-scene, not yet generalized across environments/robots
- Depends on external segmentation masks (GroundingDINO + SAM); segmentation failures propagate to dynamics prediction
Conclusion
ParticleFormer provides a capable, easy-to-use dynamics modeling framework for multi-material manipulation via Transformer attention and hybrid geometric supervision. Future work targets scene-agnostic generalization and end-to-end semantic understanding.