Recommended
More Related Content
Similar to Cutting the Kubernetes Monorepo in pieces
Similar to Cutting the Kubernetes Monorepo in pieces (20)
More from Stefan Schimanski
More from Stefan Schimanski (16)
Recently uploaded
Recently uploaded (20)
Cutting the Kubernetes Monorepo in pieces
- 1. Cutting the Kubernetes Monorepo in pieces Never learnt more about git Dr. Stefan Schimanski – @the_sttts – github.com/sttts sttts@redhat.com
- 3. „My other computer is a datacenter“
- 4. • Founded in 2014 by Google engineers • 100% OpenSource, Apache license • Governed by Cloud Native Computing Foundation (CNCF) • > 100 companies contributing Go
- 5. Monorepo + makes testing easier + makes releases easier + makes global changes easier + is easy to implement - is not how Golang works - is not what 3rd-parties expect - does not scale well github.com/kubernetes/kubernetes cmd pkg staging src k8s.io api apimachinery apiserver client-go apiextensions-apiserver kube-aggregator sample-apiserver sample-controller
- 6. Monorepo + makes testing easier + makes releases easier + makes global changes easier + is easy to implement - is not how Golang works - is not what 3rd-parties expect - does not scale well github.com/kubernetes/kubernetes cmd pkg staging src k8s.io api apimachinery apiserver client-go apiextensions-apiserver kube-aggregator sample-apiserver sample-controller vendor ... symlinks
- 7. Monorepo + makes testing easier + makes releases easier + makes global changes easier + is easy to implement - is not how Golang works - is not what 3rd-parties expect - does not scale well 😱 github.com/kubernetes/kubernetes cmd pkg staging src k8s.io api apimachinery apiserver client-go apiextensions-apiserver kube-aggregator sample-apiserver sample-controller vendor ... symlinks
- 8. github.com/kubernetes/kubernetes cmd pkg api cmd pkg apiserver cmd pkg apimachinery cmd pkg client-go cmd pkg aggregator cmd pkg apiextensions cmd pkg Publishing staging repos + automated, nightly + git filter-branch –subdirectory-filter - preserve history - tag across repos - always work 😱 preserve Github PR merge commits 😱 update dependencies: Godeps.json 😱 work with arbitrary merges 😱 some files live in staging repo Godeps.json, README.md, vendor/
- 9. commit author committer parents tree message $ git cat-file –p HEAD tree 4a541f0a3b26ab7ef8eb38090072558cd8aa54cd parent 3667154d8524e58db0e21b9338f335f5c53dd83e parent 951d2c060e0fa173f82bc221e388f9491709c52c author Kubernetes Submit Queue <k8s-merge-robot@users.noreply.github.c committer GitHub <noreply@github.com> 1508409674 -0700 Merge pull request #54192 from mkumatag/versioned_busybox message commit author committer parents tree message commit author committer parents
- 10. commit author committer parents tree message message commit author committer parents tree A, mode, hash B, mode, hash tree C, mode, hash D, mode, hash E, mode, hash file blob 00100101011101010 10100100101010101 10010111010100101 tree A, mode, hash B, mode, hash file blob 10100101001010111 11001010010010010 11010101001010100 file blob 01010101001010111 10101001010100101 10111101001001011 file blob 11010101001010000 01010101000101001 10010101010100100 file blob 11010101010101111 11011111111010000 11101010111110110 commit author committer parents tree message
- 11. git plumbing commands $ git hash-object –w <filename> # Copy into repository <blob-sha> $ git update-index --add --cacheinfo 100644 <blob-sha> <filename> $ git write-tree <tree-sha> $ git commit-tree -p <parent-sha> -m “Commit message“ <tree-sha> <commit-sha>
- 13. commit author committer parents tree message message commit author committer parents tree A, mode, hash B, mode, hash tree C, mode, hash D, mode, hash E, mode, hash file blob 00100101011101010 10100100101010101 10010111010100101 tree A, mode, hash B, mode, hash file blob 10100101001010111 11001010010010010 11010101001010100 file blob 01010101001010111 10101001010100101 10111101001001011 file blob 11010101001010000 01010101000101001 10010101010100100 file blob 11010101010101111 11011111111010000 11101010111110110 commit author committer parents tree message
- 14. commit author committer parents tree message message commit author committer parents tree A, mode, hash B, mode, hash tree C, mode, hash D, mode, hash E, mode, hash file blob 00100101011101010 10100100101010101 10010111010100101 tree A, mode, hash B, mode, hash file blob 10100101001010111 11001010010010010 11010101001010100 file blob 01010101001010111 10101001010100101 10111101001001011 file blob 11010101001010000 01010101000101001 10010101010100100 file blob 11010101010101111 11011111111010000 11101010111110110 commit author committer parents tree message
- 15. commit author committer parents tree message message commit author committer parents tree A, mode, hash B, mode, hash tree C, mode, hash D, mode, hash E, mode, hash file blob 00100101011101010 10100100101010101 10010111010100101 tree A, mode, hash B, mode, hash file blob 10100101001010111 11001010010010010 11010101001010100 file blob 01010101001010111 10101001010100101 10111101001001011 file blob 11010101001010000 01010101000101001 10010101010100100 file blob 11010101010101111 11011111111010000 11101010111110110 commit author committer parents tree message
- 16. commit author committer parents tree message message commit author committer parents tree C, mode, hash D, mode, hash E, mode, hash file blob 10100101001010111 11001010010010010 11010101001010100 file blob 01010101001010111 10101001010100101 10111101001001011 file blob 11010101001010000 01010101000101001 10010101010100100 commit author committer parents tree message „subdirectory filter“
- 17. commit author committer parents tree message message commit author committer parents tree C, mode, hash D, mode, hash E, mode, hash file blob 10100101001010111 11001010010010010 11010101001010100 file blob 01010101001010111 10101001010100101 10111101001001011 file blob 11010101001010000 01010101000101001 10010101010100100 commit author committer parents tree message „subdirectory filter“
- 18. message commit author committer parents tree C, mode, hash D, mode, hash E, mode, hash file blob 10100101001010111 11001010010010010 11010101001010100 file blob 01010101001010111 10101001010100101 10111101001001011 file blob 11010101001010000 01010101000101001 10010101010100100 commit author committer parents tree message „prune empty“ „subdirectory filter“
- 19. git filter-branch –subdirectory-filter staging/src/k8s.io/client-go PR merge PR merge PR merge PR merge commit commit commit commit PR merge commit commit
- 20. git filter-branch –subdirectory-filter staging/src/k8s.io/client-go PR merge PR merge PR merge PR merge commit commit commit commit PR merge commit PR merge PR merge PR merge PR merge commit commit commit commit PR merge commit commit
- 21. Merge pull request #56190 author committer parents tree message message commit author committer parents tree C, mode, hash D, mode, hash E, mode, hash file blob 10100101001010111 11001010010010010 11010101001010100 file blob 01010101001010111 10101001010100101 10111101001001011 file blob 11010101001010000 01010101000101001 10010101010100100 commit author committer parents tree message „subdirectory filter“
- 22. Merge pull request #56190 author committer parents tree message message commit author committer parents tree C, mode, hash D, mode, hash E, mode, hash file blob 10100101001010111 11001010010010010 11010101001010100 file blob 01010101001010111 10101001010100101 10111101001001011 file blob 11010101001010000 01010101000101001 10010101010100100 commit author committer parents tree message „subdirectory filter“ 😱
- 23. git filter-branch –subdirectory-filter staging/src/k8s.io/client-go instead: custom filter-branch with https://github.com/src-d/go-git „A highly extensible Git implementation in pure Go“ which prunes exactly the commits we want.
- 24. Lessons learnt • Git data structures are no rocket science • 500 custom Git code in Go vs. 700 dense bash lines working around git filter-branch • porcelain vs. plumbing do not script the former, understand what they do!
- 25. References • Git Internals: https://git-scm.com/book/en/v2/Git-Internals-Git- Objects • Native (non-cgo) go library: https://github.com/src-d/go-git