Github - Down the Rabbit Hole

1,840 views

Published on

Slides for the talk given at ChennaiGeeks

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,840
On SlideShare
0
From Embeds
0
Number of Embeds
21
Actions
Shares
0
Downloads
23
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Github - Down the Rabbit Hole

    1. 1. GITHUBDOWN THE RABBIT HOLE Vagmi Mudumbai @vagmi, @dharanasoft
    2. 2. github.com/vagmi @vagmi, @dharanasoft
    3. 3. @vagmi, @dharanasoft
    4. 4. DISTRIBUTED FASTTOTALLY AWESOME @vagmi, @dharanasoft
    5. 5. https://github.com/defunkt https://github.com/mojombo https://github.com/pjhyett @vagmi, @dharanasoft
    6. 6. @vagmi, @dharanasoft
    7. 7. AND A LOT MORE @vagmi, @dharanasoft
    8. 8. Service Frontend / Proxy BalancerAnalytics/ App App App App Server Server Server Server QueueMonitoring / Services Async Persistence Persistence Server Cluster Cluster GENERIC ARCHITECTURE @vagmi, @dharanasoft
    9. 9. Smart HTTP HTTPSSH Git HTTP API FrontendONLY ONE PROBLEM Multiple Service Points @vagmi, @dharanasoft
    10. 10. • Git relies on filesystem to store its repository • BLOB objects • Packfiles • Git GC• SSH relies on filesystem (~/.ssh/authorized_keys)• Ergo, GitHub is bloody hard to scale TECHNICAL CHALLENGES @vagmi, @dharanasoft
    11. 11. GITHUB IS NOT ON THE CLOUD @vagmi, @dharanasoft
    12. 12. @vagmi, @dharanasoft
    13. 13. A really smartldirectord lb1a lb1b load balancer 8 core fe1 fe2 fe3 fe4 16 GB u0 u1 u2 u3 u4 u5 u6 u7 16 Unicorn u8 u9 ua ub uc ud ue uf Processes 8 core, 16 GB db1a db1b 15k RPM SAS drives THE HTTP STACK @vagmi, @dharanasoft
    14. 14. MYSQL IS THE CACHE @vagmi, @dharanasoft
    15. 15. u0 u1 u2 u3 u4 u5 u6 u7 16 Unicorn u8 u9 ua ub uc ud ue uf Processes 16 ProxyMachine ProxyMachine fe1 fe2 fe3 fe4 InstancesSmoke, Chimney Maps user/repo db1a db1b to machine/folder 8 Core, 16 GB Ernie fs1a fs2a fs3a fs3b 6x300 GB Grit fs1b fs2b fs3b fs4b 15k RPM SAS RAID 10 @vagmi, @dharanasoft
    16. 16. ALL THIS HAPPENS IF THE CACHE IS A MISS @vagmi, @dharanasoft
    17. 17. u0 u1 u2 u3 u4 u5 u6 u7 16 Unicorn u8 u9 ua ub uc ud ue uf Processesmemcache1 memcache2 memcache3 memcache4 fs1b fs2b fs3b fs4b 12 GB of memcached on each fs slave @vagmi, @dharanasoft
    18. 18. The SSH Stack @vagmi, @dharanasoft
    19. 19. A really smart ldirectord lb1a lb1b load balancerCustom SSHD Gerve fe1 fe2 fe3 fe4 Chimney db1a db1b fs1a fs2a fs3a fs3b git fs1b fs2b fs3b fs4b SSH STACK @vagmi, @dharanasoft
    20. 20. A really smart ldirectord lb1a lb1b load balancerProxyMachine fe1 fe2 fe3 fe4 Chimney db1a db1b fs1a fs2a fs3a fs3b gitd fs1b fs2b fs3b fs4b GITD STACK @vagmi, @dharanasoft
    21. 21. WHAT ABOUT QUEUES? @vagmi, @dharanasoft
    22. 22. HTTP://GITHUB.COM/DEFUNKT/RESQUE @vagmi, @dharanasoft
    23. 23. WHAT ABOUT ARCHIVE DOWNLOADS? @vagmi, @dharanasoft
    24. 24. WHAT ABOUT THE WIKI? + @vagmi, @dharanasoft
    25. 25. GITHUB PAGEShttps://github.com/mojombo/jekyll @vagmi, @dharanasoft
    26. 26. SHAMELESSLY STOLEN FROMhttps://github.com/blog/530-how-we-made-github-fast @vagmi, @dharanasoft
    27. 27. @vagmi, @dharanasoft
    28. 28. http://ruby-lang.org, http://rubyonrails.org/ https://github.com/mojombo/ernie, http://www.erlang.org/, http://bert-rpc.org/ http://git-scm.com http://nginx.org/ http://unicorn.bogomips.org/ http://mysql.com/ldirectord http://horms.net/projects/ldirectord/ http://www.drbd.org/ @vagmi, @dharanasoft
    29. 29. http://haproxy.1wt.eu/ http://redis.io/ http://nodejs.org/ @vagmi, @dharanasoft

    ×