# Big Repositories, Shallow, Partial and Sparse Canonical: Clone Only the Part You Need, and Know What You Gave Up You need to fix a typo in one documentation page. The repository is large, and `git clone` is going to download every version of every file anyone ever committed. Git has three ways to take less. They sound alike, and they cut different things. - A **shallow** clone takes fewer commits. - A **partial** clone takes the commits but leaves out file contents until you need them. - A **sparse** checkout takes everything, but only puts some folders on your disk. We will build a small repository with real history and clone it all three ways. That gives us a table of what each one cost on disk, and the commands that undo each one: ```bash git clone --depth 1 git clone --filter=blob:none --sparse git sparse-checkout set docs ``` > **Note:** > Every command and output in this article comes from Git 2.45.2 on Windows 11, run in Git Bash. The minimum Git version for each flag was not checked, so if one is missing on your machine, run `git --version` first. Our stand-in for GitHub is a bare repository in a folder, with 29 commits in which the same 150 KB data file is replaced every time, plus a few text files. Folder paths in the output are shortened to `...`. ## The cost of the full clone This is the baseline. A normal clone of our stand-in remote: ```console $ git clone file:///C:/.../hub.git full $ git rev-list --count main 29 $ git count-objects -vH ... size-pack: 4.29 MiB ``` `size-pack` is the size of the packed history on disk. 4.29 MiB for a small test repository is the number to beat. > **Warning:** > Notice the `file:///` at the start of the URL. If you clone a plain folder path, Git ignores `--depth`, and says so: `warning: --depth is ignored in local clones; use file:// instead.` You only meet this when practicing with folders on your own disk. A GitHub URL needs nothing special. On Windows, the form is `file:///C:/path/to/repo.git`. ## Shallow: fewer commits `--depth 1` asks for the latest commit and nothing before it. ```console $ git clone --depth 1 file:///C:/.../hub.git shallow Cloning into 'shallow'... $ git log --oneline e9769cd Change 30 $ git rev-parse --is-shallow-repository true $ git count-objects -vH ... size-pack: 154.67 KiB ``` 154.67 KiB instead of 4.29 MiB. The cost is in that `git log` output: there is one commit. Git remembers where it stopped in a file, `.git/shallow`, which holds the id of the commit that has its parents cut off. That is fine for a build server that only needs to compile the latest version. It is not fine when you want `git blame`, `git log` on a file, or `git bisect`, because the history they need is not there. You can get more, or all of it, later: ```console $ git fetch --depth=5 $ git log --oneline | wc -l 5 $ git fetch --unshallow $ git log --oneline | wc -l 29 $ git rev-parse --is-shallow-repository false ``` `--depth=5` grows the history to five commits. `--unshallow` fetches the rest and turns the repository back into a normal one. ## Partial: all commits, no file contents A **partial clone** keeps the whole commit history but leaves out file contents, which Git calls blobs. `--filter=blob:none` says: no blobs yet. ```console $ git clone --filter=blob:none file:///C:/.../hub.git part2 Cloning into 'part2'... $ git rev-list --count main 29 $ git count-objects -vH ... size-pack: 176.46 KiB ``` All 29 commits, and 176.46 KiB on disk. A shallow clone with a single commit and this one are about the same size here, and this one still has the history. Where did the files go? Git fetches a blob when something needs it. Checking out `main` needed the current files, so those came with the clone. An old version of the data file is a different matter: ```console $ git show HEAD~20:assets/data.txt | head -c 60 /X/SMgBvhT7YnWhpJhZn6SgfWrgGUQ6Dooe1EmkpO0fQImxUNquFAriLInCn $ git count-objects -vH ... size-pack: 328.08 KiB ``` Reading a file from 20 commits ago made Git go back to the remote and fetch that one blob. `size-pack` grew by about 150 KB, which is one copy of the data file. Nothing printed, so you will not see it happen. Two things to know before you rely on this: - The server has to allow filters. With a plain bare repository, Git warned `filtering not recognized by server, ignoring` and sent everything. After `git config uploadpack.allowFilter true` on that repository, the filter worked. GitHub supports it. - The repository now depends on the remote. Our test clone shows why in its config: ```console $ git config --get-regexp "remote.origin.(promisor|partialclonefilter)" remote.origin.promisor true remote.origin.partialclonefilter blob:none ``` `promisor` means the remote has promised to supply what is missing. Offline, with an old file you never opened, that promise cannot be kept. ## Sparse: everything, but only some folders A **sparse checkout** is about your working folder, not about what you downloaded. You tell Git to put only the folders you name on disk. The two ideas combine well. `--sparse` on a clone starts with only the top-level files: ```console $ git clone --filter=blob:none --sparse file:///C:/.../hub.git sp $ ls README.md $ git count-objects -vH ... size-pack: 24.40 KiB ``` 24.40 KiB. Now add the folder we work in: ```console $ git sparse-checkout set docs $ ls README.md docs $ git sparse-checkout list docs ``` Git fetched the files in `docs`, and `size-pack` rose to 26.89 KiB. The `assets` and `src` folders are not on disk. `set` replaces the list. `add` extends it: ```console $ git sparse-checkout add src $ ls README.md docs src ``` In Git 2.45.2, `set` works in what Git calls **cone mode**: you name whole folders, and everything inside comes with them. `git config core.sparseCheckoutCone` printed `true` here. Finally, to go back to the whole tree: ```console $ git sparse-checkout disable $ ls README.md assets docs src $ git count-objects -vH ... size-pack: 179.69 KiB ``` Git fetched the files it had skipped, and the pack grew from 26.89 KiB to 179.69 KiB.
What the three left behind, side by side (optional) All from the same remote. | Clone | Commits | `size-pack` | Missing | | --- | --- | --- | --- | | Full | 29 | 4.29 MiB | nothing | | `--depth 1` | 1 | 154.67 KiB | all older history | | `--filter=blob:none` | 29 | 176.46 KiB | old file versions, until read | | `--filter=blob:none --sparse` | 29 | 24.40 KiB | files outside the sparse folders |
## What about Git LFS? Git LFS is a different tool for a different problem: a few very large files, such as models or design files, that you do not want in history at all. It stores a small text pointer in Git and the file itself elsewhere. We ran it in a scratch repository, with the file type tracked before the file was added: ```console $ git lfs install --local Updated Git hooks. Git LFS initialized. $ git lfs track "*.bin" Tracking "*.bin" $ cat .gitattributes *.bin filter=lfs diff=lfs merge=lfs -text $ git lfs ls-files 52ca684f48 * model.bin $ git cat-file -p HEAD:model.bin version https://git-lfs.github.com/spec/v1 oid sha256:52ca684f4830a78b5a958d39f26e5bd276931ae753f2e3ea8476dd9f742be4ee size 100000 ``` The commit holds three lines of text. The 100,000 bytes live in LFS. It does not help with a file you already committed. In a second test, we added `old.bin`, committed it, and only then ran `git lfs track "*.bin"`. `git lfs ls-files` printed nothing, and `git cat-file -s HEAD:old.bin` printed `100000`, the real file. Tracking applies to new commits. History that already has the big file keeps it. `git lfs migrate`, the tool for rewriting history, and GitHub's LFS storage limits were not tested here. Look those up before you rely on them. ## Try it Here is the whole tour in seven commands. The remote is `https://github.com/you/big.git`. Type them one at a time. **Practice terminal ยท take less of a big repository** Type these commands in order, in a practice folder: 1. Clone only the latest commit of https://github.com/you/big.git. Try `git clone --depth 1 https://github.com/you/big.git`. A plain `git clone` downloads all of the history. 2. Ask Git whether this clone is shallow. Try `git rev-parse --is-shallow-repository`. It prints `true`. 3. Get the rest of the history. Try `git fetch --unshallow`. The repository stops being shallow. 4. Clone again, this time with no file contents until needed, and with only the top-level files checked out. Try `git clone --filter=blob:none --sparse https://github.com/you/big.git lean`. The commits come with it; the folders do not. 5. Check out only the docs folder. Try `git sparse-checkout set docs`. Git fetches the files in docs. 6. See which folders are on your disk. Try `git sparse-checkout list`. It prints `docs`. 7. Go back to the whole tree. Try `git sparse-checkout disable`. Git fetches what it skipped. [Try the interactive Git practice terminal on the canonical article](https://mnixblog.netlify.app/big-repositories-shallow-partial-and-sparse/). Two things worth trying in the terminal: ask for a hint when you are stuck, and add `src` with `git sparse-checkout add src` before disabling. **Challenge: all the history, two folders** Make a new clone that has every commit but only `docs` and `src` on disk. Do it with a filter, not with `--depth`. Then prove that `assets` is missing from the disk and not from Git. **Expected result:** `git rev-list --count main` prints the same number as in a full clone, `ls` shows `README.md`, `docs` and `src`, `ls assets` says `No such file or directory`, and `git ls-files` still lists `assets/data.txt`.
Answer ```console $ git clone --filter=blob:none --sparse file:///C:/.../hub.git lean $ cd lean $ git sparse-checkout set docs src $ git rev-list --count main 29 $ ls assets ls: cannot access 'assets': No such file or directory $ git ls-files README.md assets/data.txt docs/guide.md src/main.c ``` We ran this on a rebuilt stand-in remote, so the file names are ours. The point is the last command: the index knows `assets/data.txt`, the disk does not. This clone needed `git config uploadpack.allowFilter true` on the remote first.
**Question:** You clone with `--depth 1` and later run `git blame` on an old file. What is the problem? 1. The older commits it needs were not downloaded 2. `--depth 1` also leaves out file contents 3. Shallow clones are read-only **Answer:** 1. The older commits it needs were not downloaded > **Explanation:** A shallow clone has the latest commit and no history behind it. `git fetch --unshallow` brings the history back. **Question:** You made a partial clone with `--filter=blob:none`, and you read an old version of a file for the first time. What does Git do? 1. It fetches that one file version from the remote at that moment 2. It reports the file as missing 3. It had already downloaded every file in the first place **Answer:** 1. It fetches that one file version from the remote at that moment > **Explanation:** In our test, `git show HEAD~20:assets/data.txt` fetched one blob, and `size-pack` rose by about 150 KB. That is why the repository keeps the remote marked as `promisor`. **Question:** You track `*.bin` with Git LFS after `old.bin` was already committed. Where is `old.bin` stored? 1. In Git history, as before. Tracking applies to new commits 2. In LFS, because the pattern now matches 3. It is deleted from history **Answer:** 1. In Git history, as before. Tracking applies to new commits > **Explanation:** `git lfs ls-files` printed nothing in our test, and `git cat-file -s HEAD:old.bin` showed the full 100000 bytes. ## What we learned - `git clone --depth 1` takes only the latest commit. `git fetch --unshallow` takes the rest. With local folders, use `file:///` or Git ignores `--depth`. - `--filter=blob:none` takes every commit and fetches file contents when something needs them. The server has to allow it, and the repository depends on the remote afterwards. - `git sparse-checkout set ` puts only those folders on your disk. `add` extends the list, `list` shows it, `disable` brings back the whole tree. - A shallow clone cuts history, a partial clone cuts file contents, a sparse checkout cuts what is on disk. They can be combined. - Git LFS keeps large files out of history from the moment you track them, and does nothing for files already committed. Next time you only need one folder, you will not wait for the rest. > **Portable fallback:** The authored description of the demo is preserved; the live demonstration remains on the canonical article. Review: Review the authored description of the demo and keep the canonical link for the live version. > **Portable fallback:** The question and answer are preserved; scoring, retry, and remembered attempts are unavailable. Review: Review the static question and answer; keep the canonical link when scoring or retries matter. > **Portable fallback:** The question and answer are preserved; scoring, retry, and remembered attempts are unavailable. Review: Review the static question and answer; keep the canonical link when scoring or retries matter. > **Portable fallback:** The question and answer are preserved; scoring, retry, and remembered attempts are unavailable. Review: Review the static question and answer; keep the canonical link when scoring or retries matter. [Join the discussion on the canonical article](https://mnixblog.netlify.app/big-repositories-shallow-partial-and-sparse/#ob-discussion)