Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Tracking huge files with Git LFS - LinuxCon 2016
Similar to Tracking huge files with Git LFS - LinuxCon 2016 (20)
Recently uploaded
Recently uploaded (20)
Tracking huge files with Git LFS - LinuxCon 2016
- 1. @kannonboy Photo: Le Monde en Vidéo @kannonboy
- 2. @kannonboy@kannonboy Photo: Le Monde en Vidéo
- 3. @kannonboy Git LFS! Git LOB! @kannonboy Photo: Le Monde en Vidéo
- 4. @kannonboy ok cool @kannonboy Photo: Le Monde en Vidéo Git LFS!
- 5. TIM PETTERSEN • SENIOR DEVELOPER • ATLASSIAN • @KANNONBOY Tracking huge files with Git LFS
- 6. @kannonboy GIT LFS THE PROBLEM WITH BIG FILES Agenda CONVERTING YOUR REPO TIPS FOR TEAMS
- 9. @kannonboy 434bb..tree bab1e..parent Tim P <kannonboy@…> 1455209277 -0800committer Tim P <kannonboy@…> 1455209277 -0800author My life is my commit message. 98ca9.. bab1e.. fad3d.. git cat-file -p 98ca9$
- 10. @kannonboy git cat-file -p 434bb$ 434bb.. 98ca9.. bab1e.. fad3d.. ace23..100644 blob .gitignore dbdbd..100644 blob README.md a0bc3..040000 tree app 33d33..040000 tree config b1de7..100755 blob deploy-prod.sh 7011e..100755 blob deploy-staging.sh typefilemode SHA-1
- 15. @kannonboy
- 16. @kannonboy
- 23. @kannonboy ☞ ☞ ☞ dabad.. 98ca9.. bab1e.. fad3d.. 86753.. 434bb.. ☞ ☞ ☞ dabad.. 98ca9.. bab1e.. fad3d.. 86753.. 434bb.. https://git-lfs.github.com/spec/v1version sha256:325ddfb…oid 29342295size git cat-file -p 4749d$ 4749d.. bdd12.. 778aa..
- 24. @kannonboy $ brew install git-lfs $ git lfs install
- 25. @kannonboy $ cat ~/.gitconfig [filter "lfs"] clean = git-lfs clean %f smudge = git-lfs smudge %f required = true
- 26. @kannonboy $ git lfs track “*.mp4” $ cat .gitattributes *.mp4 filter=lfs diff=lfs merge=lfs -text
- 27. @kannonboy massive_video.mp4 Work tree dev .git/lfs/objects Clean filter (git-lfs clean) ☞ Index massive_video.mp4 $ .git/objects git add
- 28. @kannonboy $ dev .git/lfs/objects Smudge filter (git-lfs smudge) Work tree massive_video.mp4 ☞ Commit tree massive_video.mp4 .git/objects LFS Store git checkout
- 31. @kannonboy $ git push Git LFS: (12 of 13 files, 1 skipped) 168.75 MB / 180.87 MB, 12.12 skipped Counting objects: 22, done. ...
- 32. @kannonboy $ git pull remote: Counting objects: 3, done. ... Downloading massive_video.mp4 (38.79 MB) ... 1 file changed, 2 insertions(+)
- 33. @kannonboy POST .../repo.git I need object cafebabe @kannonboy /info/lfs/objects/batch {“objects”:[ { “oid”: “cafebabe...”, “size”: 40689401 }, ... ], “operation”: “download”}
- 34. @kannonboy 200 OK {“objects”:[ {“oid”: “cafebabe…”, “size”: 40689401, “actions”: { “download”: { “href”: “https://…/lfs/cafebabe…”, @kannonboy cafebabe is over there } } ...
- 35. @kannonboy 200 OK {“objects”:[ {“oid”: “cafebabe…”, “size”: 40689401, “actions”: { “download”: { “href”: “https://…/lfs/cafebabe…”, @kannonboy cafebabe is over there “header”: { “Authorization”: “JWT eyJ0eXA…”, } } } ...
- 36. @kannonboy Converting to Git LFS ☞
- 37. @kannonboy Converting to Git LFS ☞
- 39. @kannonboy git filter-branch $ git filter-branch --force --index-filter 'git rm --cached --ignore-unmatch big_video.mp4’ --prune-empty --tag-name-filter cat -- --all DON’T DO THIS!
- 40. @kannonboy $ git filter—branch --prune-empty --tree-filter ' git config -f .gitconfig lfs.url “https://bitbucket.example.com/team/repo.git” git lfs track "*.mp4" git add .gitattributes .gitconfig for file in $(git ls-files | xargs git check-attr filter | grep "filter: lfs" | sed -r "s/(.*): filter: lfs/1/"); do git rm -f --cached ${file} git add ${file} done' --tag-name-filter cat -- --all @kannonboy DON’T DO THIS EITHER!
- 42. @kannonboy BFG Repo-Cleaner @kannonboy 10-720x faster than filter-branch built to kill history Git LFS support by @rtyley
- 43. @kannonboy $ git filter—branch --prune-empty --tree-filter ' git config -f .gitconfig lfs.url “https://bitbucket.example.com/team/repo.git” git lfs track "*.mp4" git add .gitattributes .gitconfig for file in $(git ls-files | xargs git check-attr filter | grep "filter: lfs" | sed -r "s/(.*): filter: lfs/1/"); do git rm -f --cached ${file} git add ${file} done' --tag-name-filter cat -- --all @kannonboy DON’T DO THIS EITHER!
- 44. @kannonboy $ brew install bfg $ bfg —-convert-to-git-lfs ‘*.{zip,mp4}’ --no-blob-protection
- 47. @kannonboy $ a295ef4… 102437 95372 2cc7063… 152171 140443 blob SHA size on disk average blob size generate-larger-than 50000 Identifying large objects
- 48. @kannonboy a295ef4… 102437 95372 2cc7063… 152171 140443 , PNG, 1148 x 482 , PNG, 1101 x 800 $ generate-larger-than 50000 | add-file-info Identifying large objects
- 49. @kannonboy $ generate-larger-than 50000 | add-file-info 2cc7063… 152171 140443, PNG, 1101 x 800 a295ef4… 102437 95372, PNG, 1148 x 482 order by average blob size | sort -k3nr Identifying large objects
- 50. @kannonboy $ generate-larger-than 50000 | add-file-info | sort -k3nr $ report-on-large-objects big-stuff.txt logo-hdpi.png 2cc7063… 152171 140443, PNG… logo-mdpi.png a295ef4… 102437 95372, PNG… paths > big-stuff.txt Identifying large objects
- 51. @kannonboy $ bfg —-convert-to-git-lfs ‘logo-*.png’ --no-blob-protection Identifying large objects
- 53. @kannonboy
- 55. @kannonboyBeware merge conflicts @kannonboy
- 58. @kannonboy master feature0 feature1 File locking (single branch model)
- 59. @kannonboy master feature0 feature1 File locking (single branch model)
- 60. @kannonboy master feature0 feature1 File locking (single branch model)
- 61. @kannonboy master feature0 feature1 File locking (single branch model)
- 62. @kannonboy master feature0 feature1 File locking (single branch model) LAME
- 63. @kannonboy master feature0 feature1 File locking (multi branch model)
- 64. @kannonboy master feature0 feature1 File locking (multi branch model)
- 65. @kannonboy master feature0 feature1 File locking (multi branch model)
- 66. @kannonboy master feature0 feature1 File locking (multi branch model)
- 67. @kannonboy master feature0 feature1 File locking (multi branch model)
- 68. @kannonboy master feature0 feature1 File locking (multi branch model)
- 69. @kannonboy master feature0 feature1 File locking (multi branch model)
- 70. @kannonboy master feature0 feature1 File locking (multi branch model)
- 71. @kannonboy master feature0 feature1 File locking (multi branch model)
- 72. @kannonboy master feature0 feature1 File locking (multi branch model)
- 73. @kannonboy master feature0 feature1 File locking (multi branch model)
- 74. @kannonboy master feature0 feature1 File locking (multi branch model)
- 75. @kannonboy master feature0 feature1 File locking (multi branch model)
- 76. @kannonboy $ git lfs lock <path> $ git lfs unlock <path> $ git lfs locks [-i id] [-p path]
- 79. @kannonboy $ git lfs fetch master feature0 feature1
- 80. @kannonboy $ git lfs fetch $ git config lfs.fetchrecentalways “true” master feature0 feature1 --recent T-minus 7 days
- 81. @kannonboy $ git config lfs.fetchrecentalways “true” lfs.fetchrecentrefsdays lfs.fetchrecentremoterefs lfs.fetchrecentcommitsdays (default = 7) (default = 0) $ git lfs fetch --recent
- 84. @kannonboy lfs.pruneoffsetdays lfs.pruneverifyremotealways (default = 3) Set this! $ git lfs prune
- 85. @kannonboyFetch the bare necessitiesFetch the bare necessities @kannonboy © Disney
- 86. @kannonboy # for a build that just runs the unit tests $ git lfs fetch --exclude Assets/** # for an audio engineer $ git lfs fetch --include Assets/Audio/**
- 87. @kannonboy # for a build that just runs the unit tests $ git lfs fetch --exclude Assets/** # for an audio engineer $ git lfs fetch --include Assets/Audio/** $ git config lfs.fetchexclude Assets/** $ git config lfs.fetchinclude Assets/Audio/**
- 88. @kannonboy coming very soon! Bitbucket Cloud git-lfs.github.com docs github.com/github/git-lfs source atlassian.com/bitbucket Bitbucket Server Looking for more?
- 89. @kannonboy coming very soon! Bitbucket Cloud git-lfs.github.com docs github.com/github/git-lfs source atlassian.com/bitbucket Bitbucket Server Looking for more?
- 90. @kannonboy coming very soon! Bitbucket Cloud git-lfs.github.com docs github.com/github/git-lfs source atlassian.com/bitbucket Bitbucket Server Looking for more? Follow me for occasional Git, Bitbucket & JIRA trivia