Quartz 5

ml-training-infrastructure

4 min read

ML Training Infrastructure for Robotics Research

Sources: Session notes, 2026-04-05 Raw: GPU Scheduling Small Team VLA 3DGS; MLOps GPU Cluster Insights Updated: 2026-08-12

Overview

Small robotics ML teams (2–20 researchers) share 1–4 GPU workstations for VLA training, 3D Gaussian Splatting, and Isaac Sim headless simulation. The infrastructure problem has two independent layers: data storage (DVC datasets must be accessible without repeated downloads) and GPU scheduling (preventing conflicts between concurrent jobs). These layers are solved separately, then integrated.

Storage Layer: NFS + DVC

The problem: DVC-tracked datasets (VLA demonstrations, 3DGS scenes) pulled to each GPU machine independently create storage redundancy and download delays.

Solution: shared NFS storage. One NAS or Linux host serves an NFS volume mounted identically on all GPU machines:

NAS/NFS Server
    /data/
        datasets/vla/      ← DVC remote target
        datasets/3dgs/
        checkpoints/
        outputs/

GPU Machine A & B
    /mnt/data → (NFS mount)

DVC remote configured to the NFS path: dvc pull runs on LAN (fast), both machines share the cache. Hardware requirement: 10GbE internal network; TrueNAS Scale (free) for the server.

Dataset separation: VLA and 3DGS datasets are scene-specific and large. Keep them in separate DVC project repos (dvc.yaml per project) with per-project storage quotas on the NFS side to prevent one project consuming all space.

Scheduling Layer: Three Options

Open-source ML platform (Determined AI):

  • Web UI for job submission without SSH
  • Automatic GPU queue management
  • Built-in experiment tracking (no separate MLflow needed)
  • Multi-GPU and multi-machine support
pip install determined
det deploy local cluster-up
det experiment create config.yaml .

Best for: teams with mixed technical backgrounds; prefer web UI; avoid Kubernetes.

Industry-standard HPC scheduler. Scales from 2 machines to 20+ without changing tools — just add compute nodes, configure per-user GPU quotas and priorities.

# Submit training job
sbatch train_vla.sh
 
# train_vla.sh
#!/bin/bash
#SBATCH --gres=gpu:1
#SBATCH --job-name=vla_training
python train.py --config vla_config.yaml

Isaac Lab + SLURM: NVIDIA officially supports this combination. Isaac Lab’s Cluster Guide uses Singularity/Apptainer containers for headless Isaac Sim (--nv flag uses host NVIDIA driver). This lets training and simulation share the same scheduler.

SLURM cgroup isolation: SLURM is the sole GPU allocation gateway — processes not submitted through SLURM cannot acquire GPUs. Prevents the most common multi-user conflict without any coordinator tool.

Web UI by scale: slurm-web (lightweight, queue view only) → Open OnDemand (academic standard, job submission + terminal) → Grafana + prometheus-slurm-exporter (monitoring-focused, 10+ people).

Option C: Minimal (2–3 person team, transition period)

Without a scheduler, five failure modes arise: GPU OOM from concurrent jobs, no identity on who occupies GPU, no disk quota, CUDA version conflicts, no audit trail.

Minimum viable: CUDA_VISIBLE_DEVICES manual assignment + gpustat monitoring. Acceptable for 2–3 people as a short-term measure; not sustainable beyond that.

The simple-gpu-scheduler project (v0.1.4, last updated 2019) is abandoned — do not use for team infrastructure.

RTX 5090 vs Data Center GPU: Key Difference

RTX 5090 does not support MIG (Multi-Instance GPU, hardware partitioning). MIG is exclusive to A100/H100 data center GPUs. Sharing an RTX 5090 between concurrent CUDA processes requires CUDA MPS (software-layer sharing), but without VRAM isolation: two jobs summing >32GB VRAM will OOM. Queuing jobs rather than sharing is safer for consumer GPU workstations.

Experiment Tracking

MLflow: pure HTTP client-server; set MLFLOW_TRACKING_URI environment variable — zero code changes, any machine (laptop, GPU server, CI) can log to the same server. Network connectivity recommendation: Tailscale (no public port needed).

RL-specific: MLflow UI is functional but basic for RL. W&B is the best UX for RL (Isaac Lab / SB3 / RLlib native integration) but primarily SaaS. Aim is the fully open-source self-hosted W&B alternative.

Common combination: TensorBoard (live monitoring during training) + MLflow (checkpoint management and final results archiving). These are complementary, not competing.

GitLab CI + SLURM Role Split

  • GitLab CI: lint, unit tests, smoke tests (minutes); runs on a GitLab runner on the SLURM head node
  • SLURM: full training runs (hours to days); researchers submit manually via sbatch since training configuration decisions require human judgment

Integration: GitLab runner on SLURM head node, .gitlab-ci.yml calls sbatch --wait for automated jobs.

Apptainer / Singularity: the Docker equivalent for HPC environments. Runs without root privileges; imports Docker images as .sif format. Docker on multi-user GPU servers creates security problems (root daemon); Apptainer resolves this. Now a Linux Foundation project (forked from Singularity in 2021).

See Also

Graph View

Backlinks