With the widespread adoption of Git, and the rollout of Git-based development workflows, large organizations must be able to scale their source code management system with their needs. In this talk we will provide practical advice to overcome the challenges in scaling Git.

1. #atlassian

2. STEFAN SAASEN • DEVELOPMENT MANAGER • ATLASSIAN • @STEFANSAASEN
Scaling Git

8. Source View - git cat-file

13. Git is fundamentally a content-addressable
filesystem with a VCS user interface written on
top of it
Pro Git Book, Section: Git Inter nals
”
“

14. GIT UNDER THE HOOD
$> tree .git/objects
.git/objects
├── info
└── pack
!
2 directories

15. GIT UNDER THE HOOD
git add some-file.txt

16. GIT UNDER THE HOOD
$> tree .git/objects
.git/objects
├── e4
│ └── 3a6ac59164adadac854d591001bbb10086f37d
├── info
└── pack
!
3 directories, 1 file
zlib compressed
SHA1

17. GIT UNDER THE HOOD
git commit -m "First commit"

18. GIT UNDER THE HOOD
$> tree .git/objects
.git/objects
├── 13
│ └── 1e360ae1a0c08acd18182c6160af6a83e0d22f
├── 31
│ └── 995f2d03aa31ee97ee2e814c9f0b0ffd814316
├── e4
│ └── 3a6ac59164adadac854d591001bbb10086f37d
├── info
└── pack
!
5 directories, 3 files

19. GIT UNDER THE HOOD
$> tree .git/objects
.git/objects
├── 13
│ └── 1e360ae1a0c08acd18182c6160af6a83e0d22f
├── 31
│ └── 995f2d03aa31ee97ee2e814c9f0b0ffd814316
├── e4
│ └── 3a6ac59164adadac854d591001bbb10086f37d
├── info
└── pack
!
5 directories, 3 files
Blob

20. GIT UNDER THE HOOD
$> tree .git/objects
.git/objects
├── 13
│ └── 1e360ae1a0c08acd18182c6160af6a83e0d22f
├── 31
│ └── 995f2d03aa31ee97ee2e814c9f0b0ffd814316
├── e4
│ └── 3a6ac59164adadac854d591001bbb10086f37d
├── info
└── pack
!
5 directories, 3 files
Blob
Tree

21. GIT UNDER THE HOOD
$> tree .git/objects
.git/objects
├── 13
│ └── 1e360ae1a0c08acd18182c6160af6a83e0d22f
├── 31
│ └── 995f2d03aa31ee97ee2e814c9f0b0ffd814316
├── e4
│ └── 3a6ac59164adadac854d591001bbb10086f37d
├── info
└── pack
!
5 directories, 3 files
Blob
Tree
Commit

22. GIT UNDER THE HOOD
echo "// Comment" >> some-file.txt

23. GIT UNDER THE HOOD
git add some-file.txt

24. GIT UNDER THE HOOD
$> tree .git/objects
.git/objects
├── 13
│ └── 1e360ae1a0c08acd18182c6160af6a83e0d22f
├── 31
│ └── 995f2d03aa31ee97ee2e814c9f0b0ffd814316
├── c1
│ └── 9e6823e34980033917b6427f3e245ce2102e6e
├── e4
│ └── 3a6ac59164adadac854d591001bbb10086f37d
!
6 directories, 4 files
Entirely new
BLOB

25. wat?

26. GIT UNDER THE HOOD
git gc

27. GIT UNDER THE HOOD
$> tree .git/objects
.git/objects
├── info
│ └── packs
└── pack
├──
pack-7475314b451a882d77b1535d215def8bad0f4306.idx
└──
pack-7475314b451a882d77b1535d215def8bad0f4306.pack
!
2 directories, 3 files

28. GIT UNDER THE HOOD
Loose Objects

29. GIT UNDER THE HOOD
1.zlib compressed
2.Delta encoded
PackfileLoose Objects

30. but...

31. CPU Replace Graph
0
200000
400000
600000
800000
1407989425 1407989450 1407989475 1407989500 1407989525 1407989550 1407989575
user syst
CPU

32. Memory
0
100
200
300
400
1407989425 1407989450 1407989475 1407989500 1407989525 1407989550 1407989575
MiB
Memory

33. IO Replace Graph
0
5
10
15
20
1407989440 1407989460 1407989480 1407989500 1407989520 1407989
read write
Disk I/O

34. What's your point?

36. git clone/fetch

37. generates a
packfile
git clone/fetch

38. git clone/fetch
every time

39. Clone
800
stash.atlassian.com
Fetch
1,200

40. This is what we
learned so far.

41. 1. SCM Cache

42. CACHE
git clone 1

43. git clone 2
CACHE

44. 2. Sizing is important

45. You need sufficient hardware

46. 768MiB
Memory budget

47. 5 GiB
Memory budget

48. 768MiB
5 GiB
Memory budget

49. 3. Limits

50. Limits

51. 4. Continuous Integration

52. What do you have?

53. This is what I've got.

54. Ok, here is what I've got. Give me
everything that's new.

55. Here you go!

56. Don't worry. I'm up to date.

57. Caption goes here

58. Avoid Polling.

59. SCM Cache can
also cache ref
advertisements

60. Consider shallow
clones.

61. 5. Update

62. Use recent
versions of

63. Just text by itself,
for impact.

64. Stash Data Center

65. RDBMS
FS

66. RDBMS
NFS

67. Performance at
scale

68. RDBMS
NFS

69. Scaling Git

70. Scaling Git
• Git hosting operations are expensive.

71. Scaling Git
• Git hosting operations are expensive.
• Properly size the hardware for the workload that you expect

72. Scaling Git
• Git hosting operations are expensive.
• Properly size the hardware for the workload that you expect
• Avoid polling when you do a lot of builds, enable caching of
ref advertisements when you can't

73. Scaling Git
• Git hosting operations are expensive.
• Properly size the hardware for the workload that you expect
• Avoid polling when you do a lot of builds, enable caching of
ref advertisements when you can't
• Prefer shallow clones

74. Scaling Git
• Git hosting operations are expensive.
• Properly size the hardware for the workload that you expect
• Avoid polling when you do a lot of builds, enable caching of
ref advertisements when you can't
• Prefer shallow clones
• Limits are in place to keep your Stash server running

75. Scaling Git
• Git hosting operations are expensive.
• Properly size the hardware for the workload that you expect
• Avoid polling when you do a lot of builds, enable caching of
ref advertisements when you can't
• Prefer shallow clones
• Limits are in place to keep your Stash server running
• Stash Data Center allows you to scale out and have high
availability

76. Sign up today!
!
Talk to me after if you’re
interested in learning more
Stash Data Center Beta Program

77. Thank you!
STEFAN SAASEN • DEVELOPMENT MANAGER • ATLASSIAN • @STEFANSAASEN