This document provides an overview of GitHub and its technical architecture presented by Chris Wanstrath. Some key points: - GitHub started as a git hosting site but became a social coding platform where users can see friends' activity and leave comments. - It uses Ruby on Rails for the main codebase, Resque for background jobs, MySQL for the database, and nginx, unicorn, and memcached. - Git operations are handled by Grit and communicated to file servers via the BERT-RPC based Smoke protocol. - Caching, asset optimization, and AJAX loading are used extensively to improve performance. Monitoring tools include Nagios, Resque Web, Haystack, and CollectD.

1. hi

2. Hello.
Hi everyone.

3. My name is Chris Wanstrath. I go by @defunkt online.

4. inside
github
And today I’m going to talk about GitHub.

5. inside
github
That’s me.

6. GitHub is what we like to call “social coding.”

7. You can see what your friends are doing from your dashboard or news feed

8. Everyone has a profile showing off their code and activity

9. And you can do things like leave comments on commits.

10. But it wasn’t always like this.

11. Originally we just wanted to make a git hosting site.
In fact, that was the first tagline.

12. git repository hosting
git repository hosting.
That’s what we wanted to do: give us and our friends a place to share git repositories.

13. a brief
history
let’s start with a brief history

14. It’s not easy to setup a git repository. It never was.
But back in 2007 I really wanted to.

15. I had seen Torvalds’ talk on YouTube about git.
But it wasn’t really about git - it was more about distributed version control.
It answered many of my questions and clarified DVCS ideas.
I still wasn’t sold on the whole idea, and I had no idea what it was good for.

16. CVS
is stupid
But when Torvalds says “CVS is stupid”

17. and so are
you
“and so are you,” the natural reaction for me is...

18. To start learning git.

19. At the time the biggest and best free hosting site was repo.or.cz.

20. Right after I had seen the Torvalds video, the god project was posted up on repo.or.cz
I was interested in the project so I finally got a chance to try it out with some other people.

21. Namely this guy, Tom Preston-Werner.
Seen here in his famous “I put ketchup on my ketchup” shirt.

22. I managed to make a few contributions to god before realizing that repo.or.cz was not different.
git was not different.
Just more of the same - centralized, inflexible code hosting.

23. This is what I always imagined.
No rules. Project belongs to you, not the site. Share, fork, change - do what you want.
Give people tools and get out of their way. Less ceremony.

24. So, we set off to create our own site.
A git hub - learning, code hosting, etc.

25. We started with the code browsing and commit viewing...

26. But once we added the current version of the dashboard, we knew this was different.

27. And eventually “git repository hosting” gave way to “social coding”

28. Unleash Your Code
Join 500,000 coders with
over 1,500,000 repositories
What’s special about GitHub is that people use the site in spite of git.
Many git haters use the site because of what it is - more than a place to host
git repositories, but a place to share code with others.

29. 2007 october
The first commit was on a Friday night in October, around 10pm.

30. 2008 january
We launched the beta in January at Steff’s on 2nd street in San Francisco’s SOMA district.
The first non-github user was wycats, and the first non-github project was merb-core.
They wanted to use the site for their refactoring and 0.9 branch.

31. 2008 april
A few short months after that we launched to the public.

32. Along the way we managed to pick up Scott Chacon, our VP of R&D

33. Tekkub, our level 80 support druid

34. Melissa Severini, who keeps us all in check

35. Kyle Neath, who makes the site pretty

36. Ryan Tomayko, who helps keep the site running smoothly.

37. Zach Holman, head of enterprise

38. Rick Olson, Rails extraordinaire

39. Eston Bond, Design Generalissimo

40. Corey Donohoe, Director of Shipology

41. And Brian Lopez, our bleeding edge cowboy

42. Oh yeah, and the other founders: PJ and Tom.

43. github.com
That’s where we’re at today.
So let’s talk about the technical details of the website: github.com

44. .com as opposed to fi, which I’m not going to get into today.
You’ll have to invite PJ out if you want to hear about that.

45. We also have a store

46. A job board

47. And do git training

48. the web site
As everyone knows, a web “site” is really a bunch of different components.
Some of them generate and deliver HTML to you, but most of them don’t.
Our site consists of four major code “frameworks” or “apps”

49. rails
#
GitHub.com, Gist, etc
1

50. resque
#
Background processing, 50ish different job types currently
2

51. smoke
#
All git calls happen over the wire
3

52. utils
#
Exception logging, stats, helper apps, etc
4

53. rails
We use Ruby on Rails 2.2.2 as our web framework.
It’s kept up to date with all the security patches and includes custom patches we’ve added
ourselves, as well as patches we’ve cherry-picked from more recent versions of Rails.

54. rails
GitHub is about 20,000 lines of Rails code, not counting Rails itself, plugins, or gems.

55. We found out Rails was moving to GitHub in March 2008, after we had reached out to
them and they had turned us down.
So it was a bit of a surprise.

56. rails plugins
We currently have 27 Rails plugins installed, and that number is always changing.

57. shopify / active_merchant

58. lgn21st / s3_swf_upload

59. technoweenie /
serialized_attributes

60. query_trace

61. query_analyzer

62. rubygems
GitHub depends on about 50 RubyGems

63. albino

64. ar-extensions

65. aws-s3

66. faker

67. faraday

68. github-markup

69. rdiscount

70. jekyll

71. gollum

72. redis-rb

73. rack
One of the big features in Rails 2.3 is Rack support.

74. We badly wanted this, but didn’t want to invest the time upgrading.
So using a few open source libraries we’ve wrapped our Rails 2.2.2 instance in Rack.

75. Now we can use awesome Rack middleware like Rack::Bug in GitHub

76. Coders created and submitted dozens of Rack middleware for the Coderack competition last year.
I was a judge so I got the see the submissions already. Some of my favorite
were

77. nerdEd / rack-validate

78. webficient / rack-tidy

79. talison / rack-mobile-detect
sets the X_MOBILE_DEVICE header to the mobile device, if
recognized

80. unicorn
We use unicorn as our application server
- master / worker
- 16 workers
- preforking

81. unicorn
- instant restart after kill
- hard 30s request timeouts
- control ram growth

82. unicorn
- 0 downtime deploys
- protects against bad rails startup
- migrations handled old fashioned way

83. nginx
For serving static content and slow clients, we use nginx
nginx is pretty much the greatest http server ever
it’s simple, fast, and has a great module system

84. nginx
Limit Zone
Limit simultaneous connections from a client

85. nginx
Limit Requests
Limit frequency of connections from a client
Anti-DDOS

86. nginx
I see many people using Rack to do what the Limit modules do.
Don’t.

87. nginx
memcached
memcached support
can serve directly from memcached

88. nginx
Push Module
comet!

89. git
The next major part of GitHub is git

90. grit
We wrote an open source library called Grit
which lets us use git from Ruby

91. mojombo / grit
you can get it here
it originally shelled out to git and just parsed the responses.
which worked well for a long time.

92. grit
File.read()
Eventually we realized, however, that File.read() can be 100 times faster

93. grit
system()
Than shelling out

94. One of the first things Scott worked on was rewriting the core parts of Grit
to be pure Ruby
Basically a Ruby implementation of Git

95. mojombo / grit
And that’s what we run now

96. smoke
Kinda.
Eventually we needed to move of our git repositories off of our web servers
Today our HTTP servers are distinct from our git servers. The two communicate using smoke

97. smoke
“Grit in the cloud”
Instead of reading and writing from the disk, Grit makes Smoke calls
The reading and writing then happens on our file servers

98. bert-rpc
Rather than use Protocol Buffers or Thrift or JSON-RPC, Smoke uses BERT-RPC

99. bert-rpc
bert : erlang ::
json : javascript
BERT is an erlang-based protocol
BERT-RPC is really great at dealing with large binaries
Which is a lot of what we do

100. bert-rpc
we have four file servers, each running bert-rpc servers
our front ends and job queue make RPC calls to the backend servers

101. mojombo / bertrpc
You can grab bert-rpc on GitHub

102. mojombo / bert
Or if you just want to play with BERT

103. chimney
We have a proprietary library called chimney
It routes the smoke. I know, don’t blame me.

104. chimney
All user routes are kept in Redis
Chimney is how our BERT-RPC clients know which server to hit
It falls back to a local cache and auto-detection if Redis is down

105. chimney
It can also be told a backend is down.
Optimized for connection refused but in reality that wasn’t the real problem - timeouts were

106. proxymachine
All anonymous git clones hit the front end machines
the git-daemon connects to proxymachine, which uses chimney to proxy your
connection between the front end machine and the back end machine (which holds
the actual git repository)
very fast, transparent to you

107. mojombo / proxymachine
proxymachine can be used to proxy any kind of tcp connection
open source

108. ssh
Sometimes you need to access a repository over ssh
In those instances, you ssh to an fe and we tunnel your connection to
the appropriate backend
To figure that out we use chimney

109. node.js

110. node.js
downloads

111. node.js
downloads
http => https <img>

112. node.js
downloads
http => https <img>
event streams

113. hubot

114. jobs
We do a lot of work in the background at GitHub

115. resque
Currently we use a system called Resque.

116. defunkt / resque
You can grab it on GitHub

117. resque
- dealing with pushes
- web hooks
- creating events in the database
- generating GitHub Pages
- clearing & warmingcaches
- search indexing

118. queues
In Resque, a queue is used as both a priority and a localization technique
By localization I mean, “where your workers live”

119. queues
critical,high,low
these three run on our front end servers
Resque processes them in this order

120. queues
page
GitHub Pages are generated on their own machine using the `page` queue

121. queues
archive
And tarball and zip downloads are created on the fly using the `archive` queue
on our archiving machines

122. search
On GitHub, you can search code, repositories, and people

123. solr
Solr is basically an HTTP interface on top of Lucene. This makes it pretty simple
to use in your code.
We use solr because of its ability to incrementally add documents to
an index.

124. Here I am searching for my name in source code

125. solr
We’ve had some problems making it stable but luckily the guys at Pivotal
have given us some tips
Like bumping the Java heap size.
Whatever that means

126. database
Our database story is pretty uninteresting

127. mysql
We use mysql 5

128. master / slave
All reads and writes go to the master
We use the slave for backups and failover

129. caching
On the site we do a ton of caching
using memcached

130. fragments
We cache chunks of HTML all over
Usually they are invalidated by some action

131. fragments
Formerly we invalidated most of our fragments using a generation scheme,
where you put a number into a bunch of related keys and increment it
when you want all those caches to be missed (thus creating new cache
entries with fresh data)

132. fragments
But we had high cache eviction due to low ram and hardware constraints, and found
that scheme did more harm than good.
We also noticed some cached data we wanted to remain forever was being evicted due
to the slabs with generational keys filling up fast

133. page
We cache entire pages using nginx’s memcached module
Lots of HTML, but also other data which gets hit a lot and changes rarely:

134. page
- network graph json
- participation graph data
Always looking to stick more into page caches

135. object
We do basic object caching of ActiveRecord objects such as
repositories and users all over the place
Caches are invalidated whenever the objects are saved

136. associations
We also cache associations as arrays of IDs
Grab the array, then do a get_multi on its contents to get a list of objects
That way we don’t have to worry about caching stale objects

137. walker
We also have a proprietary caching library called Walker

138. walker
It originally walked trees and cached them when someone pushed
But now it caches everything related to git:

139. walker
- commits
- diffs
- commit listing
- branches
- tags
- everything

140. Every git-related page load hits Walker a lot

141. walker
For most big apps, you need to write a caching layer
that knows your business domain
Generic, catch-all caching libraries probably won’t do

142. events
An example of this is our events system

144. This is one fragment

145. Each of these is a fragment

146. They’re also cached as objects

147. As well as a list of ids

148. And that’s just for the dashboard...

149. optimizations
So what other optimizations have we done

150. asset servers
Well we do the common trick of serving assets from multiple subdomains

151. asset servers
assets0.github.com
assets1.github.com
and so forth

152. sha asset id
Instead of using timestamps for asset ids, which may end up hitting the disk
multiple times on each request, we set the asset id to be the sha of the last commit
which modified a javascript or css file

153. sha asset id
/css/bundle.css?197d742e9fdec3f7
/js/bundle.js?197d742e9fdec3f7
Now simple code changes won’t force everyone to re-download the css or js bundles

154. bundling
For bundling itself, we use

155. bundling
yui’s compressor for css and

156. bundling
google’s closure compiler for javascript
we don’t use the most aggressive setting because it means changing
your javascript to appease the compression gods,
which we haven’t committed to yet

157. scripty 301
Again, for most of these tricks you need to really pay
attention to your app.
One example is scriptaculous’ wiki

158. scripty 301
When we changed our wiki URL structure, we setup dynamic 301 redirects
for the old urls.
Scriptaculous’ old wiki was getting hit so much we put the redirect into nginx itself -
this took strain off our web app and made the redirects happen almost instantly

159. ajax loading
We also load data in via ajax in many places.
Sometimes a piece of information will just take too long to retrieve
In those instances, we usually load it in with ajax

162. If Walker sees that it doesn’t have all the information it needs, it kicks off a job
to stick that information in memcached.

163. We then periodically hit a URL which checks if the information is in memcached or not.
If it is, we get it and rewrite the page with the new information.

164. We use this same trick on the Network Graph

165. Fork Queue

166. ajax loading
and anywhere else it makes sense.

167. comet loading
very soon this will all be comet, though

168. monitoring
what do we use for monitoring?

169. nagios
Our support team monitors the health of our machines and core
services using nagios.
I don’t really touch the thing.

170. Here’s a screenshot from my IE browser, complete with the ICQ plugin

171. resque web
We monitor our queue using Resque’s included Sinatra app

173. haystack
We use an in-house app called Haystack to monitor arbitrary information,
tracked as JSON.

174. Here’s an example of Haystack’s “exceptions” view

175. collectd
We also use collectd to monitor load, RAM usage, CPU usage, and other
app-related metrics

176. pingdom
pingdom sends us SMSes when the site is down
it’s nice

177. tender
tender is what we use for customer support

178. it works incredibly well, and they’re constantly improving it

179. testing
Our testing setup is pretty standard

180. test unit
We mostly use Ruby’s test/unit.
We’ve experimented with other libraries including test/spec, shoulda, and RSpec, but in the end
we keep coming back to test/unit

181. git fixtures
As many of our fixtures are git repositories, we specify in the test what sha
we expect to be the HEAD of that fixture.
This means we can completely delete a git repository in one test, then have it back in
pristine state in another. We plan to move all our fixtures to a similar git-system in the future.

182. machinist
We use machinist for our fixtures

183. notahat / machinist

184. running_man
Gives us setup_once
Use it to cache machinist fixtures on a per-test-class basis

185. technoweenie / running_man

186. ci joe
We use ci joe, a continuous integration server, to run on tests after each push.
He then notifies us if the tests fail.

189. defunkt / cijoe
You can grab him at github

190. staging
We also always deploy the current branch to staging
This means you can be working on your branch, someone else can be working on theirs,
and you don’t need to worry about reconciling the two to test out a feature
One of the best parts of Git

191. security

192. github.com/
security
having a security page really helps

193. security@
github.com
we get weekly emails to our security email (that people find on the security page)
and people are always grateful when we can reassure them or a answer their question

194. regular audits
if you can, find a security consultant to poke your site for XSS vulnerabilities
having your target audience be developers helps, too

195. 24/7 monitoring
24/7 monitoring is cool too

196. backups
backups are incredibly important
don’t just make backups: ensure you can restore them, as well

197. sql
we keep nightly, off-site backups of our sql databases

198. git
and the same for all our git repositories

199. the future

200. svn

201. pull requests

202. organizations

203. ...and more

204. Questions?
thanks for coming

205. Thanks.
thanks for coming