AnsibleByExample

Git Large File Storage (LFS) — Handle Big Files in Git

BeginnerGitGit LFSversion controlDevOps
2024-01-014By Luca Berton

Lesson 1 of 15 in Foundations

Ansible Foundations to Production — full curriculum
  1. 1. Introduction to Ansible and Automation
  2. 2. Installing and Configuring Ansible
  3. 3. Running Ad-Hoc Commands
  4. 4. Writing Your First Playbook
  5. 5. Understanding and Using Modules
  6. 6. Variables and Facts
  7. 7. Conditionals and Loops
  8. 8. Handlers and Idempotency
  9. 9. Structuring Projects with Roles
  10. 10. Inventory Management
  11. 11. Using Ansible Vault for Secrets
  12. 12. Debugging and Troubleshooting
  13. 13. Real-World Automation Examples
  14. 14. Final Project: Deploy a Web Server
  15. 15. Bonus - What's Next with Ansible?
View path overview →

Introduction

Git is optimized for text files — source code, configs, YAML playbooks. When you push large binary files (ISOs, VM images, datasets, compiled artifacts), you'll hit the pack exceeds maximum allowed size error. Git Large File Storage (LFS) solves this by storing large files on a separate server while keeping lightweight pointers in your repo.

The Error

$ git push origin main
remote: error: pack exceeds maximum allowed size (100 MB)
fatal: the remote end hung up unexpectedly

This happens when:

  • A single file exceeds the platform's size limit (GitHub: 100MB, GitLab: varies)
  • The cumulative pack file is too large
  • Large binary files were committed to history

Quick Fix: Git LFS

Install Git LFS

# macOS
brew install git-lfs

# Ubuntu/Debian
sudo apt install git-lfs

# RHEL/Fedora
sudo dnf install git-lfs

# Windows
# Included with Git for Windows, or:
choco install git-lfs

Initialize in Your Repo

cd my-repo
git lfs install
# Updated git hooks. Git LFS initialized.

Track Large Files

# Track specific file types
git lfs track "*.iso"
git lfs track "*.tar.gz"
git lfs track "*.zip"
git lfs track "*.vmdk"
git lfs track "*.qcow2"

# Track specific files
git lfs track "data/training-set.csv"

# Commit the .gitattributes file
git add .gitattributes
git commit -m "Configure Git LFS tracking"

Verify Tracking

git lfs track
# Listing tracked patterns
#     *.iso (.gitattributes)
#     *.tar.gz (.gitattributes)

git lfs ls-files
# a1b2c3d4e5 * data/large-dataset.csv
# f6g7h8i9j0 * images/vm-image.vmdk

Understanding .gitattributes

# .gitattributes
*.iso filter=lfs diff=lfs merge=lfs -text
*.tar.gz filter=lfs diff=lfs merge=lfs -text
*.vmdk filter=lfs diff=lfs merge=lfs -text
*.qcow2 filter=lfs diff=lfs merge=lfs -text
data/*.csv filter=lfs diff=lfs merge=lfs -text

File Types to Track with LFS

CategoryExtensionsTypical Size
VM images.vmdk, .qcow2, .vdi, .ova1-50 GB
Archives.iso, .tar.gz, .zip, .7z100 MB - 10 GB
Datasets.csv, .parquet, .h5100 MB - 5 GB
Media.mp4, .mov, .wav50 MB - 5 GB
Compiled.jar, .war, .whl10 MB - 500 MB
ML models.pt, .onnx, .pb, .safetensors100 MB - 50 GB

Migrate Existing Repo to LFS

If large files are already in your Git history:

# Migrate existing tracked files to LFS
git lfs migrate import --include="*.iso,*.tar.gz,*.vmdk" --everything

# This rewrites history — force push required
git push --force origin main

Warning: migrate import rewrites Git history. Coordinate with your team before running this.

Migrate a Single File

git lfs migrate import --include="path/to/large-file.bin" --everything

Alternative: Remove Large Files from History

If you don't want LFS, remove the large files entirely:

# Using git-filter-repo (recommended over filter-branch)
pip install git-filter-repo

# Remove a specific file from all history
git filter-repo --path path/to/large-file.iso --invert-paths

# Remove all files larger than 50MB from history
git filter-repo --strip-blobs-bigger-than 50M

Prevent Future Issues

Pre-commit Hook

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      - id: check-added-large-files
        args: ['--maxkb=10000']  # 10MB limit

Git Config

# Warn about large files
git config --global core.bigFileThreshold 50m

.gitignore

# Don't track these at all
*.iso
*.vmdk
*.ova
build/
dist/
*.egg-info/

LFS with Ansible Projects

Common large files in Ansible repos:

# Track large Ansible artifacts
git lfs track "*.rpm"
git lfs track "*.deb"
git lfs track "files/*.tar.gz"
git lfs track "files/*.zip"

Platform LFS Limits

PlatformFree LFS StorageFree BandwidthPaid Plans
GitHub1 GB1 GB/month$5/50GB pack
GitLab5 GB (SaaS)UnlimitedPer plan
Bitbucket1 GB5 GB/monthPer plan

Troubleshooting

"Smudge filter lfs failed"

# Re-install LFS hooks
git lfs install
git lfs pull

LFS Files Show as Pointers

# Download actual content
git lfs fetch --all
git lfs checkout

Check LFS Status

git lfs status
git lfs env

Conclusion

Install Git LFS with git lfs install, track large files with git lfs track "*.iso", commit .gitattributes, and push normally. For existing repos, use git lfs migrate import to move large files out of Git history. Use .gitignore for files that shouldn't be tracked at all, and pre-commit hooks to prevent accidentally committing large files. Keep your Git repos lean — track code in Git, store artifacts in LFS or an artifact registry.

All Tutorials