The document discusses the technical challenges of scaling GitHub and describes GitHub's architecture. Key points include that GitHub is not on the cloud, it uses MySQL as a cache, memcached for cache misses, load balancing, and various technologies like Ruby, Rails, Unicorn, Nginx, Redis, and Resque to distribute load and improve performance.

1. GITHUB
DOWN THE RABBIT HOLE
Vagmi Mudumbai
@vagmi, @dharanasoft

2. github.com/vagmi
@vagmi, @dharanasoft

3. @vagmi, @dharanasoft

4. DISTRIBUTED
FAST
TOTALLY AWESOME
@vagmi, @dharanasoft

5. https://github.com/defunkt https://github.com/mojombo
https://github.com/pjhyett
@vagmi, @dharanasoft

6. @vagmi, @dharanasoft

7. AND A LOT MORE
@vagmi, @dharanasoft

8. Service Frontend / Proxy Balancer
Analytics/ App App App App
Server Server Server Server Queue
Monitoring
/
Services
Async
Persistence Persistence
Server
Cluster Cluster
GENERIC ARCHITECTURE
@vagmi, @dharanasoft

9. Smart HTTP HTTP
SSH Git
HTTP API Frontend
ONLY ONE PROBLEM
Multiple Service Points
@vagmi, @dharanasoft

10. • Git relies on filesystem to store its repository
• BLOB objects
• Packfiles
• Git GC
• SSH relies on filesystem (~/.ssh/authorized_keys)
• Ergo, GitHub is bloody hard to scale
TECHNICAL CHALLENGES
@vagmi, @dharanasoft

11. GITHUB IS NOT ON THE
CLOUD
@vagmi, @dharanasoft

12. @vagmi, @dharanasoft

13. A really smart
ldirectord lb1a lb1b load balancer
8 core
fe1 fe2 fe3 fe4
16 GB
u0 u1 u2 u3 u4 u5 u6 u7 16 Unicorn
u8 u9 ua ub uc ud ue uf Processes
8 core, 16 GB
db1a db1b 15k RPM SAS drives
THE HTTP STACK
@vagmi, @dharanasoft

14. MYSQL IS THE CACHE
@vagmi, @dharanasoft

15. u0 u1 u2 u3 u4 u5 u6 u7 16 Unicorn
u8 u9 ua ub uc ud ue uf Processes
16 ProxyMachine
ProxyMachine fe1 fe2 fe3 fe4
Instances
Smoke, Chimney
Maps user/repo
db1a db1b
to machine/folder
8 Core, 16 GB
Ernie fs1a fs2a fs3a fs3b 6x300 GB
Grit fs1b fs2b fs3b fs4b 15k RPM SAS
RAID 10
@vagmi, @dharanasoft

16. ALL THIS HAPPENS IF THE
CACHE IS A MISS
@vagmi, @dharanasoft

17. u0 u1 u2 u3 u4 u5 u6 u7 16 Unicorn
u8 u9 ua ub uc ud ue uf Processes
memcache1 memcache2 memcache3 memcache4
fs1b fs2b fs3b fs4b
12 GB of memcached on each fs slave
@vagmi, @dharanasoft

18. The SSH Stack
@vagmi, @dharanasoft

19. A really smart
ldirectord lb1a lb1b load balancer
Custom SSHD
Gerve fe1 fe2 fe3 fe4
Chimney
db1a db1b
fs1a fs2a fs3a fs3b
git
fs1b fs2b fs3b fs4b
SSH STACK @vagmi, @dharanasoft

20. A really smart
ldirectord lb1a lb1b load balancer
ProxyMachine fe1 fe2 fe3 fe4
Chimney
db1a db1b
fs1a fs2a fs3a fs3b
gitd
fs1b fs2b fs3b fs4b
GITD STACK @vagmi, @dharanasoft

21. WHAT ABOUT QUEUES?
@vagmi, @dharanasoft

22. HTTP://GITHUB.COM/DEFUNKT/RESQUE
@vagmi, @dharanasoft

23. WHAT ABOUT ARCHIVE
DOWNLOADS?
@vagmi, @dharanasoft

24. WHAT ABOUT THE WIKI?
+
@vagmi, @dharanasoft

25. GITHUB PAGES
https://github.com/mojombo/jekyll
@vagmi, @dharanasoft

26. SHAMELESSLY STOLEN FROM
https://github.com/blog/530-how-we-made-github-fast
@vagmi, @dharanasoft

27. @vagmi, @dharanasoft

28. http://ruby-lang.org, http://rubyonrails.org/
https://github.com/mojombo/ernie,
http://www.erlang.org/, http://bert-rpc.org/
http://git-scm.com
http://nginx.org/
http://unicorn.bogomips.org/
http://mysql.com/
ldirectord http://horms.net/projects/ldirectord/
http://www.drbd.org/
@vagmi, @dharanasoft

29. http://haproxy.1wt.eu/
http://redis.io/
http://nodejs.org/
@vagmi, @dharanasoft