My mom told me that Git doesn’t scale by Vicent Martí

  • 627 views
Uploaded on

With over 2 million and a half repositories, GitHub is the world’s largest source code host. Since day one, we’ve faced an unique engineering problem: making terabytes of Git data always available, …

With over 2 million and a half repositories, GitHub is the world’s largest source code host. Since day one, we’ve faced an unique engineering problem: making terabytes of Git data always available, either directly or through our website. This talk offers a hopefully insightful view into the internals of Git, the way its original design affects our scalable architecture, and the many things we’ve learnt while solving this fascinating problem.

More in: Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
627
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
5
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Saturday, May 11, 13
  • 2. These are thethings you don’tcare aboutSaturday, May 11, 13
  • 3. Saturday, May 11, 13
  • 4. githubSaturday, May 11, 13
  • 5. githubGit hosting:No longer a pain in the assSaturday, May 11, 13
  • 6. githubGit hosting:No longer a pain in the assfor you.Not for us.Because, goddamnit,if I ever find the guy who inventedthis thing I’m going to hang him froma fence by his underwear and.Saturday, May 11, 13
  • 7. Let’s host someGit repos! file.c src file.h README.md COPYING.md .git Bare Repository HEAD index objects refsgit-daemonSaturday, May 11, 13
  • 8. OK, now aboutthe web...grit Ruby - GitinterfaceSaturday, May 11, 13
  • 9. OK, now aboutthe web...grit  Bare Repo Bare Repo Bare RepoRuby - GitinterfaceSaturday, May 11, 13
  • 10. 1VM grit📽storagerails appSaturday, May 11, 13
  • 11. nVM 📽storageSaturday, May 11, 13
  • 12. nVM 📽storage(GFS)Saturday, May 11, 13
  • 13. Rails was making us slow.Saturday, May 11, 13
  • 14. Saturday, May 11, 13
  • 15. Literally.Saturday, May 11, 13
  • 16. Time to move toReal HardwareSaturday, May 11, 13
  • 17. fileserversfrontends📸dbSaturday, May 11, 13
  • 18. fileserversfrontends📸db?????????Saturday, May 11, 13
  • 19. smokeSaturday, May 11, 13
  • 20. Saturday, May 11, 13
  • 21. bert(binary Erlang term)Saturday, May 11, 13
  • 22. bert(binary Erlang term)ernie(not an acronym)Saturday, May 11, 13
  • 23. 📸chimney(Redis)frontendfileserversmokegriterniegritSaturday, May 11, 13
  • 24. Vertical Scaling#realtalkSaturday, May 11, 13
  • 25. bottleneck:gritSaturday, May 11, 13
  • 26. bottleneck:gritsolution:gitshell out toSaturday, May 11, 13
  • 27. bottleneck:gitshell out toSaturday, May 11, 13
  • 28. bottleneck:gitsolution:gitshell out toshell out toSaturday, May 11, 13
  • 29. bottleneck:gitsolution:gitshell out toshell out toproperlySaturday, May 11, 13
  • 30. Saturday, May 11, 13
  • 31. GUISESaturday, May 11, 13
  • 32. GUISEGUISESaturday, May 11, 13
  • 33. GUISEGUISEGUISESaturday, May 11, 13
  • 34. GUISEGUISEGUISE...what?Saturday, May 11, 13
  • 35. Saturday, May 11, 13
  • 36. Why don’t we takeSaturday, May 11, 13
  • 37. Why don’t we takethe Git binary...Saturday, May 11, 13
  • 38. Why don’t we takethe Git binary...yeah?Saturday, May 11, 13
  • 39. Why don’t we takethe Git binary...yeah? and compile it asSaturday, May 11, 13
  • 40. Why don’t we takethe Git binary...yeah? and compile it asa librarySaturday, May 11, 13
  • 41. Why don’t we takethe Git binary...yeah? and compile it asa libraryoh... go on...Saturday, May 11, 13
  • 42. Why don’t we takethe Git binary...yeah? and compile it asa libraryoh... go on...and link that intoSaturday, May 11, 13
  • 43. Why don’t we takethe Git binary...yeah? and compile it asa libraryoh... go on...and link that intoour serverSaturday, May 11, 13
  • 44. ScientificGraph™Saturday, May 11, 13
  • 45. MemoryUsageTimeScientificGraph™Saturday, May 11, 13
  • 46. MemoryUsageTimeScientificGraph™Saturday, May 11, 13
  • 47. MemoryUsageTimeScientificGraph™Saturday, May 11, 13
  • 48. MemoryUsageTimeScientificGraph™Saturday, May 11, 13
  • 49. MemoryUsageTimeScientificGraph™Saturday, May 11, 13
  • 50. MemoryUsageTimeScientificGraph™Saturday, May 11, 13
  • 51. Well, we didn’t think aboutfreeing memory, but...Saturday, May 11, 13
  • 52. Well, we didn’t think aboutfreeing memory, but...THIS IS THE KINDOF PROBLEMWE COULD SOLVEWITH CGISaturday, May 11, 13
  • 53. Well, we didn’t think aboutfreeing memory, but...THIS IS THE KINDOF PROBLEMWE COULD SOLVEWITH CGIIN 1995Saturday, May 11, 13
  • 54. ScientificGraph™Saturday, May 11, 13
  • 55. MemoryUsageTimeScientificGraph™Saturday, May 11, 13
  • 56. MemoryUsageTimeScientificGraph™Saturday, May 11, 13
  • 57. MemoryUsageTimeScientificGraph™Saturday, May 11, 13
  • 58. MemoryUsageTimeScientificGraph™Saturday, May 11, 13
  • 59. MemoryUsageTimeScientificGraph™Saturday, May 11, 13
  • 60. MemoryUsageTimeScientificGraph™Saturday, May 11, 13
  • 61. MemoryUsageTimeScientificGraph™Saturday, May 11, 13
  • 62. What do you meanthe server died?Saturday, May 11, 13
  • 63. die("BUG: non-INDEX attr directionin a bare repo");die("a bad revision is needed");die("%s is not a validbranch name.", name); die("Empty patch.Aborted.");die("unable to read index file");What do you meanthe server died?Saturday, May 11, 13
  • 64. libgitSaturday, May 11, 13
  • 65. libgit2the “2” means this onefrees memorySaturday, May 11, 13
  • 66. libgit2the “2” means this onefrees memoryNOT ENOUGHABSTRACTFACTORIESSaturday, May 11, 13
  • 67. JGitthe “J” means this oneis in Java...not our thing.Saturday, May 11, 13
  • 68. Javaa brief timelineNew companiesdon’t use Javabecause it’snot like Unix1995New companiesuse Javabecause it’snew and shiny1997New companiesdon’t use Javabecause it’sooooooold2005New companiesuse the JVMbecause WEBSCALE2011Saturday, May 11, 13
  • 69. Javaa brief timelineNew companiesdon’t use Javabecause it’snot like Unix1995New companiesuse Javabecause it’snew and shiny1997New companiesdon’t use Javabecause it’sooooooold2005New companiesuse the JVMbecause WEBSCALE2011githubSaturday, May 11, 13
  • 70. If you think you understandthe JVM, you are either:Saturday, May 11, 13
  • 71. If you think you understandthe JVM, you are either:a) Very smartSaturday, May 11, 13
  • 72. If you think you understandthe JVM, you are either:a) Very smartb) Very wrongSaturday, May 11, 13
  • 73. If you think you understandthe JVM, you are either:a) Very smartb) Very wrongSaturday, May 11, 13
  • 74. Some people thinkthat github is aRails shopRuby shop.or even aSaturday, May 11, 13
  • 75. Some people thinkthat github is aRails shopRuby shop.or even agithub is aUnix shopand everything else isjust a detail.Saturday, May 11, 13
  • 76. libgit2So,Saturday, May 11, 13
  • 77. Good Heavens,just look at thetime.It’s NoSQL o’clockNoSQLNoSQL NoSQLNoSQLNoSQL NoSQLNoSQLNoSQLSaturday, May 11, 13
  • 78. Saturday, May 11, 13
  • 79. ...do you evenSaturday, May 11, 13
  • 80. ...do you evenmongo?Saturday, May 11, 13
  • 81. a b r i e f i n t r o d u c t i o nt o t h e G i t d a t a m o d e lSaturday, May 11, 13
  • 82. Saturday, May 11, 13
  • 83. Saturday, May 11, 13
  • 84.  file.c src file.h README.md COPYING.mdSaturday, May 11, 13
  • 85.  file.c src file.h README.md COPYING.mdtreesrc/README.mdCOPYING.mdtreefile.cfile.hblobblobblobblobSaturday, May 11, 13
  • 86. commitparenttree TmetadataSaturday, May 11, 13
  • 87. commitTcommitTcommitTcommitTcommitTcommitTBehold,a graph.Saturday, May 11, 13
  • 88. Saturday, May 11, 13
  • 89. Well that was easy.Saturday, May 11, 13
  • 90. Saturday, May 11, 13
  • 91. masterOhgodkillmeSaturday, May 11, 13
  • 92. Li le knowntorturemethods:Saturday, May 11, 13
  • 93. Saturday, May 11, 13
  • 94. warning:the rabbithole ispretty deepSaturday, May 11, 13
  • 95. Git doesn’t givea #!%$ about CAPSaturday, May 11, 13
  • 96. Saturday, May 11, 13
  • 97. Number of hops on a complex query1,000,000Saturday, May 11, 13
  • 98. Number of hops on a complex query1,000,000Required hops for a successful query1,000,000Saturday, May 11, 13
  • 99. Number of hops on a complex query1,000,000Required hops for a successful query1,000,000Replica count to ensure 100% availabilitya metric shittonSaturday, May 11, 13
  • 100. We could fix it.Saturday, May 11, 13
  • 101. We could fix it.But we won’t.Saturday, May 11, 13
  • 102. libgit2Saturday, May 11, 13
  • 103. libgit2Saturday, May 11, 13
  • 104. GitRPCSaturday, May 11, 13
  • 105. GitRPCLess.Saturday, May 11, 13
  • 106. GitRPCRuggedlibgit2serverRubyRubyCSaturday, May 11, 13
  • 107. 📸chimney(Redis)frontendfileserversmokegriternie-corngritGitRPC GitRPCSaturday, May 11, 13
  • 108. 📸chimney(Redis)frontend fileserverGitRPC GitRPCserverclientSaturday, May 11, 13
  • 109. evolutionary(disappointing?)Saturday, May 11, 13
  • 110. We’ve had alot ofSaturday, May 11, 13
  • 111. We’ve had alot of hardengineeringchallengesSaturday, May 11, 13
  • 112. We tackled them by:Saturday, May 11, 13
  • 113. Using themost reliabletools we know.Saturday, May 11, 13
  • 114. Challenging ourselves to buildthe simplest thing.Not because it’s easy,but because it works.Saturday, May 11, 13
  • 115. Innovatingwhere it really ma ers.Saturday, May 11, 13
  • 116. revolutionaryproductbuilding arevolutionarybackend.not aSaturday, May 11, 13
  • 117. Having funSaturday, May 11, 13
  • 118. Saturday, May 11, 13