Abstract
DUSt3R introduces a novel approach for dense 3D reconstruction from uncalibrated image collections without requiring prior camera parameters. Rather than following traditional multi-view stereo pipelines, the method “casts the pairwise reconstruction problem as a regression of pointmaps, relaxing the hard constraints of usual projective camera models.” The system unifies monocular and binocular reconstruction through a transformer-based architecture and achieves state-of-the-art results across multiple 3D vision tasks.
Introduction
The paper identifies a fundamental problem with existing reconstruction pipelines: they decompose 3D reconstruction into sequential minimal problems (matching, triangulation, pose estimation, dense reconstruction), where “each sub-problem is not solved perfectly and adds noise to the next step.” This cascading error accumulation makes pipelines brittle, especially when facing low viewpoint counts, non-Lambertian surfaces, or insufficient camera motion.
DUSt3R proposes an alternative: a single end-to-end network that jointly processes image pairs and outputs pointmaps—dense 2D fields mapping pixels to 3D scene coordinates. This representation implicitly preserves relationships between pixels, scene geometry, and viewpoints without explicitly enforcing geometric constraints.
Core Technical Contribution: Pointmaps
A pointmap is a “dense 2D field of 3D points” X ∈ ℝ^(W×H×3) that establishes one-to-one correspondence between image pixels and 3D scene points. Critically, both output pointmaps from a stereo pair are expressed in the same coordinate frame (the first image’s frame), which differs fundamentally from traditional depth-based approaches. This allows the network to handle scene geometry and viewpoint relationships implicitly.
Network Architecture
The architecture, inspired by CroCo, follows a Siamese encoder-decoder structure:
- Encoding: Both input images pass through a shared Vision Transformer (ViT-Large) encoder, producing token representations
- Decoding: Transformer decoder blocks perform self-attention within each view and cross-attention between views, enabling information exchange
- Output: Separate heads regress pointmaps and confidence maps for both views
The shared weights and cross-attention mechanism ensure “properly aligned pointmaps” by design.
Training Objective
Training employs a simple 3D regression loss with confidence weighting:
ℒ_conf = Σ C^(v,1)_i · ℓ_regr(v,i) - α log C
Where the base regression loss normalizes predictions by average point distance to handle scale ambiguity. The confidence maps learn to identify unreliable regions (sky, translucent surfaces) without explicit supervision.
Downstream Applications
The rich pointmap representation enables effortless extraction of typical 3D vision outputs:
- Pixel Matching: Nearest-neighbor search in 3D pointmap space yields correspondences
- Intrinsic Recovery: Simple optimization over focal length using pointmap reprojection
- Relative Pose: Procrustes alignment or RANSAC-PnP between pointmap pairs
- Visual Localization: Establishes 2D-3D correspondences for absolute pose estimation
Global Alignment for Multi-View Reconstruction
For multiple image sets, a global optimization procedure aligns pairwise predictions into unified 3D space. This differs from traditional bundle adjustment by optimizing 3D projection errors rather than 2D reprojection errors, enabling fast convergence in “mere seconds on a standard GPU.”
Training Setup
The network was trained on 8.5 million image pairs from eight datasets (Habitat, MegaDepth, ARKitScenes, etc.) covering diverse scene types. Training progressed from 224×224 to 512-pixel resolution. The model was initialized with CroCo pretraining weights.
Experimental Results
- Visual Localization (7Scenes, Cambridge): comparable or superior to HLoc and DSAC*
- Multi-View Pose Estimation (CO3Dv2): RRA@15 = 96.2% vs PoseDiffusion 80.5%
- Monocular Depth: matches supervised baselines in zero-shot evaluation
- 3D Reconstruction (DTU): 2.7mm accuracy, 0.8mm completeness — without ground-truth camera parameters
Limitations
- Regresses to scale ambiguity (requires normalization)
- Lower accuracy than sub-pixel triangulation methods on calibrated benchmarks
- Assumes each camera ray hits a single 3D point — breaks for translucent surfaces
- Underperforms calibrated methods on DTU
Conclusion
DUSt3R unifies disparate 3D vision tasks under a single framework without task-specific finetuning. By eliminating the need for camera calibration and pose priors, it democratizes 3D reconstruction for “unconstrained image collections,” addressing a longstanding bottleneck in the traditional reconstruction pipeline.