Quartz 5

MinIO AIStor Object Lifecycle Management

Properties1
tagsreference, infrastructure, data

2 min read

Summary

MinIO AIStor’s Object Lifecycle Management implements S3-compatible rules for automatic object transition (tiering to remote storage) and expiration (deletion). Tiering moves data to cheaper cold storage (S3, GCS, Azure, another MinIO) after a specified age; expiration deletes objects after a period. Both operate through a low-priority background scanner to avoid performance spikes.

MinIO AIStor 的物件生命週期管理提供 S3 相容的自動分層儲存（transition）與過期刪除（expiration）規則，適合將老化資料自動移至低成本遠端儲存層，並透過低優先級背景掃描器執行，不影響正常工作負載效能。

Key Points

  • Transition (tiering): moves objects to a configured remote tier after N days; MinIO retains metadata and retrieves objects transparently on-access
  • Exclusive access required: manually modifying tiered objects on the remote side corrupts MinIO’s object tracking — remote storage must be treated as an opaque backend
  • Tiering ≠ backup: remote tier cannot be used for disaster recovery; use site/bucket replication for that
  • Expiration: generates a DeleteMarker on versioned buckets by default; --noncurrent-expire-days handles old versions separately
  • --purge-all-object-versions-days: permanently deletes all versions after N days — incompatible with object-locked buckets
  • Scanner timing: the low-priority scanner may apply rules slightly after their trigger date; not suitable for hard real-time expiration

Insights

The architecture enforces a clean separation between MinIO (authoritative metadata + hot access path) and the remote tier (dumb byte storage). This design is sound but creates an operational trap: teams often assume tiered data is independently accessible, then discover they cannot recover after a MinIO metadata loss without the remote data being unusable. The docs correctly emphasize this limitation but it’s easy to miss in practice.

For robotics dataset pipelines, lifecycle management is useful for automatically tiering raw teleoperation recordings to cold storage after ingest and processing, keeping the hot MinIO cluster fast while preserving long-term data for retraining.

Connections

Raw Excerpt

“MinIO AIStor requires exclusive access to the transitioned data on the remote storage tier. Object metadata on the ‘hot’ MinIO AIStor source is strongly linked to the object data on the ‘warm/cold’ remote tier.”

Graph View