Git LFS and DVC: The Ultimate Guide to Managing Large Artifacts in MLOps | by Pablo jusue | Medium

Git LFS and DVC: The Ultimate Guide to Managing Large Artifacts in MLOps

The Growing Challenge of Large Files in ML Projects

Machine learning practitioners face a unique challenge that traditional software developers rarely encounter: managing extremely large files within version control systems. While a typical software repository might consist of code files measuring in kilobytes, ML projects routinely incorporate:

• Datasets ranging from hundreds of megabytes to many gigabytes
• Trained model weights often exceeding 1GB (especially for deep learning models)
• Feature stores and preprocessed data artifacts
• Evaluation results including visualizations and benchmark outputs

This creates a fundamental tension. On one side, these artifacts are essential components that need version control just as much as code does. On the other side, Git — the de facto standard for version control — was designed for tracking text files, not multi-gigabyte binaries.

The Git Limitation Reality

Git’s architecture creates several specific problems when dealing with large files:

Repository Bloat: Git stores the complete history of every file. When you commit a 500MB dataset and then update it five times, your repository now contains 3GB of data just for that file’s history.Performance Degradation: Git operations slow to a crawl with large binaries. A simplegit clone

that normally takes seconds can take hours when large files are involved.Collaboration Friction: Team members must download the entire repository history, including all versions of all large files, creating significant bandwidth consumption and waiting times.Platform Limitations: GitHub, GitLab, and similar platforms impose size limits — GitHub, for instance, warns on files larger than 50MB and blocks files exceeding 100MB.CI/CD Pipeline Inefficiency: Continuous integration environments must repeatedly download large files, increasing build times and cloud compute costs.

This creates a practical dilemma for ML teams: they need the versioning capabilities of Git but cannot use standard Git workflows for their largest and most important artifacts.

The Integration vs. Separation Dilemma

When facing these limitations, ML teams typically consider two general approaches:

The Integration Approach

Many teams prefer keeping everything together in one repository, following GitOps principles. The advantages are clear:

Atomic Changes: Code and data modifications can be committed together, ensuring that model code always works with the corresponding data versionSimplified Workflows: Developers can use familiar Git commands and don’t need to learn additional toolsConsistent History: A single timeline of project evolution covering both code and dataEasier Reproducibility: Checking out a specific commit gets you the exact state of both code and data

This approach is especially appealing to teams transitioning from software development to ML, as it maintains familiar workflows.

The Separation Approach

The alternative is separating large files from the Git repository entirely:

Storing datasets in object storage(S3, GCS, Azure Blob)Using database systemsfor feature storesImplementing artifact registriesfor model weightsCreating custom metadata systemsto maintain relationships

While this approach solves the Git limitations problem, it introduces new challenges:

Version Synchronization: How do you ensure that code commit A works with dataset version B?Cognitive Overhead: Developers must remember which files are where and learn multiple toolsComplex Reproducibility: Recreating a specific project state requires careful coordination across systemsIncreased Infrastructure: More systems to maintain, secure, and pay for

The Middle Ground: Git Extensions

Recognizing this dilemma, several tools have emerged that offer a middle ground — extending Git’s capabilities to handle large files while preserving much of its workflow. The two most prominent are Git LFS and DVC.

These solutions aim to provide:

• The familiar Git interface for all operations
• Efficient storage and transfer of large files
• Integration with cloud storage providers
• Specialized features for machine learning workflows

In the sections that follow, we’ll dive deep into both Git LFS and DVC, exploring how they work and when to use each one. We’ll examine their architectures, provide practical setup guides, and compare them across real-world ML scenarios.

By the end, you’ll have a clear understanding of how to implement an effective strategy for managing your ML project’s largest artifacts, enabling smooth collaboration without sacrificing version control.

Git LFS: GitHub’s Native Solution for Large Files

Git Large File Storage (LFS) emerged in 2015 as GitHub’s answer to the large file dilemma. Developed originally by GitHub but now an open-source project, Git LFS has become deeply integrated into the GitHub ecosystem and many other Git platforms.

How Git LFS Works: The Technical Architecture

Git LFS employs a clever “pointer” system that maintains Git’s workflow while offloading large file storage:

Pointer Files: Instead of storing large files directly in the Git repository, Git LFS replaces them with tiny text files (typically less than 1KB) called “pointer files.” These files contain:

• The OID (Object ID): A SHA-256 hash that uniquely identifies the file’s content
• The file size
• A reference to where the actual file is stored

Content-Addressable Storage: The actual file content is stored in a separate LFS server, indexed by its content hash. This means:

• Identical files are stored only once (deduplication)
• Files can be verified for integrity
• Content is immutable and referenced by its hash

Transparent Operations: Git LFS integrates with standard Git commands, automatically:

• Replacing large files with pointers during git add

• Uploading file content to LFS storage during git push

• Downloading content and replacing pointers with actual files during git checkout

This architecture means that Git itself only tracks and transfers the small pointer files, while a separate LFS client handles the large file transfers behind the scenes.

Setting Up Git LFS: A Practical Guide

Setting up Git LFS requires a few straightforward steps:

1. Installation

Ubuntu/Debian

sudo apt-get install git-lfs# Windows (using Chocolatey)

choco install git-lfs# macOS (using Homebrew)

brew install git-lfs# Or download from https://git-lfs.github.com/

2. System-Wide Initialization

This sets up the Git LFS hooks in your global Git configuration.

git lfs install

3. Repository Configuration

# Navigate to your repository

cd my-ml-project

Start tracking specific file patterns

git lfs track “*.h5” # Keras models

git lfs track “*.pkl” # Pickled Python objects

git lfs track “*.parquet” # Parquet files

git lfs track “*.csv” # CSV datasets

git lfs track “*.zip” # Compressed archives

git lfs track “data/**” # Everything in the data directory

IMPORTANT: Commit the .gitattributes file where tracking config is stored

git add .gitattributes

git commit -m “Configure Git LFS tracking patterns”

4. Using Git LFS

Once configured, you can use Git normally. Git LFS intercepts operations involving tracked file patterns:

# Adding a large file works like normal Git

git add models/my_large_model.h5

git commit -m “Add trained model”

git push

Pulling works transparently too

git pull

Clone with LFS content

git clone https://github.com/username/repo.git

5. Managing LFS Objects

Git LFS provides commands for managing the large files:

# List all tracked files

git lfs ls-files

See tracking patterns

git lfs track

Pull all LFS objects (not just the ones needed for current checkout)

git lfs pull —all

Pre-download LFS objects for a specific branch without checking it out

git lfs fetch —recent

Clean up local LFS cache (remove files not referenced by commits)

git lfs prune

LFS Storage Options

Git LFS content can be stored in several places:

1. GitHub’s LFS Storage

GitHub provides LFS storage by default, but with limitations:

• Free plans include 1GB of LFS storage and 1GB monthly bandwidth
• Exceeding these limits incurs additional costs ($5 per month per 50GB of storage)

2. Self-Hosted LFS Server

For complete control, you can host your own LFS server:

• Git LFS Server (reference implementation)
• MinIO (compatible with S3 API)

3. Third-Party LFS Services

Several services offer Git LFS hosting:

• GitLab
• Azure DevOps
• Bitbucket

Strengths of Git LFS

For ML teams, Git LFS offers several key advantages:

Familiar Git Workflow: Team members use standard Git commands; the LFS operations happen transparently.Platform Integration: Deep integration with GitHub, GitLab, and other platforms, including web interfaces.Selective Checkout: LFS objects are only downloaded when checking out a branch that needs them, saving bandwidth and storage.Simple Setup: Adding LFS to existing repositories is straightforward.Language and Framework Agnostic: Works with any ML framework or file format.CI/CD Integration: Most CI platforms (GitHub Actions, GitLab CI, Jenkins) have built-in support for Git LFS.

Limitations for ML Workflows

However, Git LFS has limitations when applied to ML workflows:

No Pipeline Awareness: LFS doesn’t understand relationships between files (e.g., which dataset produced which model).Limited Metadata: LFS doesn’t track ML-specific metadata like training metrics or experiment parameters.Storage Costs: GitHub and similar platforms charge for LFS storage beyond free tier limits.Performance with Very Large Files: While better than plain Git, LFS can still struggle with extremely large files (10GB+).No Dataset Versioning Features: For dataset-specific operations like splits and transformations, LFS offers no specialized tools.

In the next section, we’ll explore DVC (Data Version Control), which addresses many of these ML-specific limitations while still leveraging Git’s underlying architecture.

DVC: The ML-Native Approach to Version Control

Data Version Control (DVC) emerged from the ML community’s specific needs, designed from the ground up to address the unique challenges of machine learning workflows. While Git LFS extends Git to handle large files, DVC reimagines version control for the entire ML lifecycle.

The DVC Philosophy: Beyond File Storage

DVC’s approach is fundamentally different from Git LFS, focusing on:

Data and Model Versioning: Not just storing large files, but tracking their lineage and relationshipsPipeline Management: Capturing the entire workflow from data preprocessing to model evaluationExperiment Tracking: Monitoring and comparing metrics across different model iterationsReproducibility: Ensuring any experiment can be precisely recreated

This comprehensive approach makes DVC more than just a large file storage solution — it’s a framework for managing the entire ML development process.

How DVC Works: Technical Architecture

DVC operates alongside Git, creating a powerful hybrid version control system:

Metadata in Git: DVC stores lightweight metadata files in your Git repositoryData in Remote Storage: The actual large files live in configurable remote storage (S3, GCS, Azure, etc.)Local Cache: DVC maintains a local content-addressed cache for efficient operations

When you add a file to DVC:

dvc add data/large_dataset.csv

DVC performs these operations:

• Calculates a hash of the file content
• Moves the file to the local cache, named by its hash
• Creates a small metadata file ( data/large_dataset.csv.dvc

) containing the hash - Adds the original file to .gitignore

to prevent Git from tracking it - You then commit the .dvc

file to Git

This pattern achieves several important goals:

• Git only tracks small metadata files
• Large files can be stored anywhere
• File integrity is guaranteed by content hashing
• The repository remains lightweight and fast

Setting Up DVC: A Practical Guide

1. Installation

# Using pip (recommended)

pip install dvc

For specific remote storage support

pip install dvc[s3] # For Amazon S3

pip install dvc[azure] # For Azure Blob Storage

pip install dvc[gs] # For Google Cloud Storage

pip install dvc[all] # All remote storage options

Alternative: Using conda

conda install -c conda-forge dvc

2. Repository Initialization

# Navigate to your Git repository

cd my-ml-project

Initialize DVC

dvc init

Commit DVC configuration files

git add .dvc .dvcignore

git commit -m “Initialize DVC”

3. Setting Up Remote Storage

# Add a remote storage location

dvc remote add -d myremote s3://my-bucket/dvc-storage

Or for other storage backends:

dvc remote add -d myremote gs://my-bucket/dvc-storage

dvc remote add -d myremote azure://my-container/dvc-storage

dvc remote add -d myremote /path/to/local/storage # Local directory

Commit the configuration

git add .dvc/config

git commit -m “Configure DVC remote storage”

4. Adding and Versioning Data

# Add a large file or directory to DVC

dvc add data/training_data.parquet

dvc add models/

Commit the metadata file

git add data/training_data.parquet.dvc models.dvc

git commit -m “Add training data and models”

Push the actual data to remote storage

dvc push

5. Retrieving Data

# Clone the repository (gets only code and .dvc files)

git clone https://github.com/username/repo.git

cd repo

Pull all DVC-tracked files

dvc pull

Or pull only specific files

dvc pull data/training_data.parquet

Beyond Storage: DVC’s Advanced Features

What truly sets DVC apart from Git LFS is its ML-specific features:

1. Data Pipelines

DVC allows you to define reproducible pipelines that connect data, code, and models:

# Define a pipeline stage

dvc run -n preprocess \

-d src/preprocess.py -d data/raw_data.csv \

-o data/features.parquet \

python src/preprocess.py data/raw_data.csv data/features.parquet

Define another stage that depends on the previous one

dvc run -n train \

-d src/train.py -d data/features.parquet \

-o models/model.pkl -M metrics.json \

python src/train.py data/features.parquet models/model.pkl

These commands create a dvc.yaml

file that defines your pipeline. Running dvc repro

will:

• Check which files have changed
• Re-run only the affected pipeline stages
• Update outputs and metrics

This ensures reproducibility and saves computation by only running what’s necessary.

2. Experiment Tracking

DVC tracks metrics across experiments:

# Train with different parameters

dvc exp run —set-param train.learning_rate=0.1

dvc exp run —set-param train.learning_rate=0.01

Compare experiments

dvc exp show

Apply a successful experiment

dvc exp apply exp-a123b

This provides a lightweight alternative to dedicated experiment tracking tools, all integrated with your version control system.

3. Metrics and Plots

DVC can track and visualize metrics across experiments:

# Define metrics output in your pipeline

dvc run -n evaluate \

-d src/evaluate.py -d models/model.pkl -d data/test.csv \

-M metrics.json -O plots/confusion_matrix.csv \

python src/evaluate.py

View metrics

dvc metrics show

Compare metrics across Git tags/branches

dvc metrics diff main v1.0

Generate plots

dvc plots show plots/confusion_matrix.csv

Strengths of DVC for ML Workflows

DVC offers several advantages for ML teams:

Complete ML Lifecycle: Manages data, code, models, and metrics in one systemStorage Flexibility: Works with any cloud storage provider or even local directoriesPipeline Management: Captures dependencies between artifacts for reproducibilityExperiment Tracking: Lightweight tracking of hyperparameters and metricsTeam Collaboration: Enables sharing of data and models through standard Git workflowsOpen Source: Completely free and open source with an active community

Limitations and Considerations

However, DVC does have some trade-offs:

Learning Curve: More complex than Git LFS, requiring understanding of new conceptsAdditional Tooling: Requires installing Python and DVC on all development machinesLess Platform Integration: Less integrated with GitHub/GitLab interfaces than Git LFSSeparate Commands: Requires using both Git and DVC commands in workflowsStill Emerging: While stable, the ecosystem is still evolving compared to Git

In the next section, we’ll compare Git LFS and DVC across practical ML scenarios to help you choose the right approach for your projects.

Practical Comparison: Git LFS vs. DVC in Real-World Scenarios

While understanding the theoretical differences between Git LFS and DVC is valuable, seeing how they perform in realistic ML workflows provides more actionable insights. Let’s examine three common scenarios that ML teams face and compare how each tool addresses them.

Scenario 1: Versioning a 10GB Dataset

Imagine your team works with a 10GB training dataset that undergoes periodic updates and refinements. How would each tool handle this?

With Git LFS:

# Track the dataset format

git lfs track “*.parquet”

git add .gitattributes

git commit -m “Configure LFS tracking for parquet files”

Add the dataset

git add training_data.parquet

git commit -m “Add training dataset v1”

git push

Later, update the dataset

(modify training_data.parquet)

git add training_data.parquet

git commit -m “Update dataset with new samples”

git push

A team member would get the latest version

git pull

The latest version is automatically downloaded

What’s happening behind the scenes:

• Git LFS stores each version of the dataset (10GB per version)
• The Git repository contains only small pointer files
• When pulling, Git LFS downloads only the version you need
• No explicit relationship with code or models is recorded

With DVC:

# Add the dataset to DVC

dvc add training_data.parquet

git add training_data.parquet.dvc .gitignore

git commit -m “Add training dataset v1”

git push

dvc push

Later, update the dataset

(modify training_data.parquet)

dvc add training_data.parquet # Re-add after changes

git add training_data.parquet.dvc

git commit -m “Update dataset with new samples”

git push

dvc push

A team member would get the latest version

git pull

dvc pull

What’s happening behind the scenes:

• DVC stores each version in remote storage (10GB per version)
• Git tracks the small .dvc metadata files
• DVC manages the relationship between Git commits and dataset versions
• Team members need to run both git pull

anddvc pull

Key differences in this scenario:

Command Complexity: Git LFS requires fewer commands but DVC provides more explicit controlStorage: Both store each version of the datasetVersioning Granularity: DVC offers more tools for comparing dataset versionsIntegration with Code: DVC can formally link dataset versions to code versions

Scenario 2: Managing an ML Pipeline with Changing Inputs

Now let’s consider a more complex scenario: an ML pipeline that processes raw data, generates features, trains models, and evaluates performance. How do the tools handle this workflow?

With Git LFS:

# Track all large files

git lfs track “data//*.csv” “data//.parquet” “models/.pkl”

git add .gitattributes

git commit -m “Set up LFS tracking patterns”

To run the pipeline, you’d use scripts or a workflow tool

python scripts/process_data.py # Generates features.parquet

python scripts/train_model.py # Generates model.pkl

python scripts/evaluate.py # Generates metrics.json

Commit all outputs

git add data/features.parquet models/model.pkl metrics.json

git commit -m “Run pipeline with latest data”

git push

What’s happening behind the scenes:

• Git LFS tracks large files but has no awareness of the pipeline structure
• Dependencies between inputs and outputs are not formally tracked
• You need external tools or documentation to remember pipeline steps
• If someone changes an input, they need to remember to re-run the affected pipeline steps

With DVC:

# Define the pipeline stages

dvc stage add -n process \

-d scripts/process_data.py -d data/raw_data.csv \

-o data/features.parquet \

python scripts/process_data.py

dvc stage add -n train \

-d scripts/train_model.py -d data/features.parquet \

-o models/model.pkl \

python scripts/train_model.py

dvc stage add -n evaluate \

-d scripts/evaluate.py -d models/model.pkl -d data/test.csv \

-o metrics.json \

python scripts/evaluate.py

Run the entire pipeline

dvc repro

Commit the pipeline definition and outputs

git add dvc.yaml dvc.lock data/features.parquet.dvc models/model.pkl.dvc metrics.json

git commit -m “Run pipeline with latest data”

git push

dvc push

What’s happening behind the scenes:

• DVC formally tracks the pipeline structure in dvc.yaml

• Dependencies between inputs and outputs are explicitly defined

• DVC knows which stages to re-run when inputs change

• The pipeline becomes a versioned artifact itself

Key differences in this scenario:

Dependency Tracking: DVC has native understanding of which files depend on othersSelective Execution: DVC can run only the parts of the pipeline affected by changesReproducibility: DVC pipelines are self-documenting and ensure consistent resultsComplexity: DVC requires more setup but provides more automation

Conclusion: Choosing the Right Tool for Your ML Workflows

When deciding between Git LFS and DVC for managing large artifacts in your machine learning projects, there’s no one-size-fits-all answer. Your choice should be guided by your team’s specific needs, technical context, and workflow complexity.

When to Choose Git LFS

Git LFS is likely the better choice when:

You need a simple solutionwith minimal learning curveYour team is primarily software engineersfamiliar with GitYou want tight integrationwith GitHub or GitLab platformsYour large files are relatively staticand don’t change frequentlyYour ML workflows are straightforwardwithout complex pipeline dependenciesCI/CD integrationis a primary concern

When to Choose DVC

DVC becomes the preferable option when:

You have complex ML pipelineswith multiple interdependent stagesYour team needs experiment trackingintegrated with version controlYou want flexible storage backendsbeyond what Git hosting providesReproducibilityis a critical requirementYour datasets are very large(10GB+) or change frequentlyYou need specialized ML featureslike metrics tracking and pipeline visualization

Hybrid Approaches

Many teams find success with hybrid approaches:

Using Both Tools Together: Git LFS for certain assets (documentation images, small datasets) and DVC for ML-specific artifacts (large datasets, models)Starting with Git LFS, Growing into DVC: Beginning with the simpler tool and gradually adopting DVC as workflows matureDVC for Data, Other Tools for Experiments: Using DVC for data and pipeline management while employing specialized tools like MLflow or Weights & Biases for experiment tracking

Final Recommendations

Regardless of which tool you choose, follow these best practices:

Commit to a Consistent Workflow: Once you choose a tool, establish clear team guidelinesDocument Your Approach: Create clear documentation for team members on how to use the chosen toolsRegularly Review Storage Usage: Monitor storage consumption to avoid unexpected costsConsider Pruning Historical Data: Not every version of every artifact needs to be kept foreverAutomate Where Possible: Use CI/CD to enforce consistent practices

The management of large artifacts in ML projects remains an evolving challenge, but both Git LFS and DVC offer viable solutions that significantly improve upon raw Git. By understanding their strengths and limitations, you can make an informed choice that supports rather than hinders your team’s productivity.

Remember that the ultimate goal is not perfect version control, but enabling your team to collaborate effectively, iterate rapidly, and produce reproducible, high-quality machine learning models. Choose the tool that best supports those fundamental objectives for your specific context.