From 0-1 billion in 46 days

993 views

Published on

Slides for a talk at the Colorado Software Summit in 2008 that I did about growing Bumper Sticker, a Ruby on Rails Facebook app to over a billion pageviews.

Funny thing is ... I had to bail on the conference. Had to ship product.

Published in: Technology, Education
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
993
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
8
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

From 0-1 billion in 46 days

  1. 1. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin 0-1 Million in 46 Days Scaling a Facebook Application in Rails Ikai Lan LinkedinIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 1
  2. 2. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Hi! I’m Ikai LanIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 2
  3. 3. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin What this talk is not One-size-fits all solution to scaling problems Example source code An in-depth discussion of Ruby on RailsIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 3
  4. 4. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin What this talk is A look into the challenges we faced with scaling a rapid growth web applicationIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 4
  5. 5. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin This is a detective story.Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 5
  6. 6. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Rather than telling you best practicesIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 6
  7. 7. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin We’ll talk about how we came to define them.Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 7
  8. 8. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Bumper Sticker? Investigation of the Facebook platform Investigation of Ruby on Rails as a serious development platformIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 8
  9. 9. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Reid Hoffman’s wagerIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 9
  10. 10. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin 1 million users in 45 days!Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 10
  11. 11. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Personal Badging Create and display badges Share badges Find badgesIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 11
  12. 12. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin = Bumper Sticker!Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 12
  13. 13. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Success!Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 13
  14. 14. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin … 1 million in 46 daysIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 14
  15. 15. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Trouble in ParadiseIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 15
  16. 16. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Throw hardware at it!Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 16
  17. 17. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin More hardware!Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 17
  18. 18. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin This is not working.Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 18
  19. 19. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Current state of affairs 7 million adds 20 million pageviews 3 million timeouts a dayIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 19
  20. 20. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Strategy Analyze MySQL slow queries log Application logIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 20
  21. 21. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin D’oh! No logging!Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 21
  22. 22. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Logging -> Syslog Rails does not buffer writes to filesystemIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 22
  23. 23. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Clue 1: Slaves ran several hours behind MasterIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 23
  24. 24. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Clue 2: Updates to add count One by one updatesIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 24
  25. 25. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Counter Cache SQL Updates not batched Very inefficient!Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 25
  26. 26. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Result: less slow queriesIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 26
  27. 27. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Lesson 1: Buffer everything!Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 27
  28. 28. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Still had lots of timeouts!Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 28
  29. 29. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Clue 3: 80% of traffic to static contentIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 29
  30. 30. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Next attempt: edge caching!Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 30
  31. 31. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Edge caching using fb:ref Facebook mechanism for storing dataIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 31
  32. 32. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Fb:ref on the load balancer! F5 BigIP set to conditionally respond to static URIs with fb:ref Background process to constantly update fb:ref in Facebook cacheIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 32
  33. 33. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Result: Static pages = perfectIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 33
  34. 34. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Lesson 2: Edge cache when possibleIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 34
  35. 35. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin But we were still serving millions of timeouts!Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 35
  36. 36. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Clue 4: Most CPU time accepting stickers Request taking up to 10 seconds during peakIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 36
  37. 37. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Slow database writesIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 37
  38. 38. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin 1st attempt: Background process writesIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 38
  39. 39. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Result: Timeouts dropped significantly!Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 39
  40. 40. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Success … ?Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 40
  41. 41. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin No.Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 41
  42. 42. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Application would not seem responsive to user actions and seem out of sync.Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 42
  43. 43. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin 2nd attempt: Two step writeIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 43
  44. 44. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Step 1: Synchronous write to MemCache Step 2: Asynchronous write to database (queue)Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 44
  45. 45. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Result: Success!Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 45
  46. 46. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Lesson 3: Use asynchronous processing when possibleIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 46
  47. 47. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Lesson 3.1: Keeping data in sync is not trivial!Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 47
  48. 48. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Lesson 3.2: Lack of concurrency in Rails presents problem earlierIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 48
  49. 49. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Lesson 3.3: Very few problems can be reduced to a linear equationIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 49
  50. 50. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin QED: Rails can scale.Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 50
  51. 51. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Latest stats 16 million installs 30 million pageviews a day Average response time of <80msIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 51
  52. 52. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Scaling has a price!Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 52
  53. 53. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin ComplexityIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 53
  54. 54. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin No new functionality!Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 54
  55. 55. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Key takeaways Make long running tasks asynchronous Use edge caching Buffer long operations when possibleIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 55
  56. 56. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin The bigger picture Language choice is less important than architecture Be creative – brute force doesn’t work Be ready to be wrongIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 56
  57. 57. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Questions?Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 57
  58. 58. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin Thanks for coming! ilan@linkedin.com http://www.linkedin.com/in/ikailanIkai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 58

×