diff options
| -rw-r--r-- | doc/git-performance.md | 227 |
1 files changed, 227 insertions, 0 deletions
diff --git a/doc/git-performance.md b/doc/git-performance.md new file mode 100644 index 0000000..cc7c3c5 --- /dev/null +++ b/doc/git-performance.md @@ -0,0 +1,227 @@ +# Git Server Performance Tuning + +Guide to optimizing clone and fetch speeds on the git server. +Especially important for large repositories like the Linux kernel. + + +## Why clones are slow + +When a client runs `git clone`, the server has to: +1. Enumerate all objects (commits, trees, blobs) in the repository +2. Compute deltas between similar objects to reduce transfer size +3. Compress everything into a pack stream and send it + +This is CPU-intensive, not network-intensive. A 10 Gbit network won't +help if the server is spending all its time packing objects. The +optimizations below reduce the work the server has to do. + + +## 1. Repack the repository + +If the repository was created with `git clone --bare` or `--mirror`, +the pack files may not be optimized. Repacking creates a single, +well-organized pack file that the server can stream directly instead +of rebuilding on every clone. + +```bash +cd /var/git/repos/<repo>.git + +# Create a single optimized pack with all objects +git repack -a -d -f --threads=0 + +# Or for maximum compression (slower, but smaller pack file) +git gc --aggressive +``` + +- `-a` — pack all objects into a single pack +- `-d` — remove old packs after repacking +- `-f` — force recomputation of deltas (even if a pack already exists) +- `--threads=0` — use all available CPU cores + +This is the single most impactful optimization. For the Linux kernel +repo, this can take 30+ minutes but the result is permanent. + + +## 2. Enable bitmap index + +Bitmaps let git skip object traversal during clone. Without bitmaps, +git must walk the entire commit graph to find which objects to send. +With bitmaps, it uses a precomputed bitmap to look up objects in O(1). +This can speed up clones by 10x or more on large repositories. + +```bash +cd /var/git/repos/<repo>.git +git config repack.writeBitmaps true +git config pack.writeBitmapHashCache true +git repack -a -d -b +``` + +The `-b` flag writes the bitmap file alongside the pack. The bitmap +must be rebuilt after a repack, so enabling `repack.writeBitmaps` +ensures it's regenerated automatically. + +Only one pack file can have a bitmap. If the repo has multiple pack +files, run `git repack -a -d -b` to consolidate into one. + + +## 3. Enable commit graph + +The commit graph file is a precomputed index that speeds up commit +traversal (used during object enumeration on clone/fetch): + +```bash +cd /var/git/repos/<repo>.git +git commit-graph write --reachable +``` + +This creates `.git/objects/info/commit-graph` (or in bare repos, +`objects/info/commit-graph`). Rebuild it after significant pushes. + + +## 4. Git config tuning + +Apply these settings to each repository you want to optimize: + +```bash +cd /var/git/repos/<repo>.git + +# Packing +git config pack.threads 0 # use all CPU cores for packing +git config pack.deltaCacheSize 256m # cache deltas in memory during packing +git config pack.windowMemory 256m # memory limit for delta search window +git config pack.packSizeLimit 0 # no limit on pack file size (single pack is fastest) + +# Compression +git config core.compression 1 # lower compression level (default is 6) + # 1 = fast, larger transfer; 9 = slow, smallest transfer + # for fast networks, lower is better (CPU is the bottleneck) + +# Transfer +git config transfer.unpackLimit 1 # keep pushed objects in packs (don't unpack to loose objects) +git config uploadpack.allowFilter true # allow partial clones (client requests only what it needs) + +# Protocol +git config protocol.version 2 # protocol v2 is more efficient for negotiation +``` + +### Compression level tradeoff + +`core.compression` controls the zlib compression level (0-9): +- `0` — no compression, fastest CPU, largest transfer +- `1` — minimal compression, very fast, slightly smaller +- `6` — default, good balance +- `9` — maximum compression, slowest CPU, smallest transfer + +On fast local networks (1 Gbit+), use `1` — the CPU time saved +outweighs the extra bytes. On slow connections, keep the default or +use `9`. + + +## 5. Protocol choice + +The protocol used for cloning affects speed: + +| Protocol | Encryption | Speed | Command | +|----------|-----------|-------|---------| +| `git://` | None | Fastest | `git clone git://swave.lol/repo.git` | +| `ssh://` | Yes (SSH) | Slower (CPU overhead) | `git clone ssh://git@swave.lol/repos/repo.git` | + +For large repos on trusted networks, `git://` is faster because +there's no encryption overhead. Use `ssh://` when you need +authentication or are on untrusted networks. + + +## 6. Client-side speedups + +The client can also reduce clone time: + +### Shallow clone (only latest commit) + +```bash +git clone --depth 1 git://swave.lol/repo.git +``` + +Downloads only the latest commit and its tree — no history. Ideal for +builds that don't need git log. Much faster and smaller. + +### Partial clone (blobless) + +```bash +git clone --filter=blob:none git://swave.lol/repo.git +``` + +Clones commits and trees but skips file contents (blobs). Blobs are +fetched on demand when you checkout files. Requires +`uploadpack.allowFilter=true` on the server. + +### Partial clone (treeless) + +```bash +git clone --filter=tree:0 git://swave.lol/repo.git +``` + +Even more aggressive — skips both trees and blobs. Only downloads +commit objects initially. Fastest initial clone, but operations like +`git log --stat` will be slower (fetches trees on demand). + + +## 7. Container CPU limits + +Check that the git-server container isn't CPU-throttled: + +```bash +docker inspect git-server --format '{{.HostConfig.CpuQuota}}' +``` + +If it returns `0`, there's no limit (good). If it returns a number, +the container is CPU-throttled and repacking/cloning will be slow. +Remove the limit or increase it in the compose file: + +```yaml +services: + git-server: + # ... + deploy: + resources: + limits: + cpus: '4' # allow 4 CPU cores +``` + + +## Quick setup script + +Apply all optimizations to a single repository: + +```bash +#!/bin/bash +# Usage: bash optimize-repo.sh /var/git/repos/<repo>.git + +REPO=$1 + +if [ -z "$REPO" ]; then + echo "Usage: $0 /path/to/repo.git" + exit 1 +fi + +cd "$REPO" || exit 1 + +echo "Configuring git settings..." +git config pack.threads 0 +git config pack.deltaCacheSize 256m +git config pack.windowMemory 256m +git config pack.packSizeLimit 0 +git config core.compression 1 +git config transfer.unpackLimit 1 +git config uploadpack.allowFilter true +git config protocol.version 2 +git config repack.writeBitmaps true +git config pack.writeBitmapHashCache true + +echo "Writing commit graph..." +git commit-graph write --reachable + +echo "Repacking with bitmaps (this may take a while)..." +git repack -a -d -f -b --threads=0 + +echo "Done. Repository optimized." +``` |