Slides for a talk at the Colorado Software Summit in 2008 that I did about growing Bumper Sticker, a Ruby on Rails Facebook app to over a billion pageviews. Funny thing is ... I had to bail on the conference. Had to ship product.

1. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
0-1 Million in 46 Days
Scaling a Facebook Application in Rails
Ikai Lan
Linkedin
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 1

2. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Hi! I’m Ikai Lan
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 2

3. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
What this talk is not
One-size-fits all solution to scaling
problems
Example source code
An in-depth discussion of Ruby on
Rails
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 3

4. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
What this talk is
A look into the challenges we faced
with scaling a rapid growth web
application
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 4

5. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
This is a detective story.
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 5

6. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Rather than telling you best
practices
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 6

7. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
We’ll talk about how we came to
define them.
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 7

8. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Bumper Sticker?
Investigation of the Facebook
platform
Investigation of Ruby on Rails as a
serious development platform
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 8

9. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Reid Hoffman’s wager
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 9

10. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
1 million users in 45 days!
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 10

11. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Personal Badging
Create and display badges
Share badges
Find badges
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 11

12. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
= Bumper Sticker!
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 12

13. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Success!
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 13

14. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
… 1 million in 46 days
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 14

15. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Trouble in Paradise
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 15

16. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Throw hardware at it!
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 16

17. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
More hardware!
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 17

18. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
This is not working.
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 18

19. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Current state of affairs
7 million adds
20 million pageviews
3 million timeouts a day
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 19

20. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Strategy
Analyze MySQL slow queries log
Application log
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 20

21. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
D’oh! No logging!
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 21

22. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Logging -> Syslog
Rails does not buffer writes to
filesystem
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 22

23. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Clue 1: Slaves ran several hours
behind Master
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 23

24. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Clue 2: Updates to add count
One by one updates
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 24

25. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Counter Cache
SQL Updates not batched
Very inefficient!
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 25

26. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Result: less slow queries
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 26

27. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Lesson 1: Buffer
everything!
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 27

28. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Still had lots of timeouts!
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 28

29. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Clue 3: 80% of traffic to static
content
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 29

30. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Next attempt: edge caching!
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 30

31. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Edge caching using fb:ref
Facebook mechanism for storing
data
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 31

32. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Fb:ref on the load balancer!
F5 BigIP set to conditionally respond
to static URIs with fb:ref
Background process to constantly
update fb:ref in Facebook cache
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 32

33. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Result: Static pages =
perfect
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 33

34. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Lesson 2: Edge cache
when possible
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 34

35. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
But we were still serving
millions of timeouts!
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 35

36. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Clue 4: Most CPU time accepting
stickers
Request taking up to 10 seconds
during peak
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 36

37. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Slow database writes
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 37

38. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
1st attempt: Background process
writes
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 38

39. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Result: Timeouts dropped
significantly!
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 39

40. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Success … ?
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 40

41. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
No.
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 41

42. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Application would not
seem responsive to user
actions and seem out of
sync.
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 42

43. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
2nd attempt: Two step
write
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 43

44. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Step 1: Synchronous
write to MemCache
Step 2: Asynchronous write to
database (queue)
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 44

45. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Result: Success!
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 45

46. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Lesson 3: Use
asynchronous processing
when possible
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 46

47. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Lesson 3.1: Keeping data
in sync is not trivial!
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 47

48. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Lesson 3.2: Lack of
concurrency in Rails
presents problem earlier
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 48

49. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Lesson 3.3: Very few
problems can be reduced
to a linear equation
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 49

50. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
QED: Rails can scale.
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 50

51. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Latest stats
16 million installs
30 million pageviews a day
Average response time of <80ms
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 51

52. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Scaling has a price!
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 52

53. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Complexity
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 53

54. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
No new functionality!
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 54

55. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Key takeaways
Make long running tasks
asynchronous
Use edge caching
Buffer long operations when possible
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 55

56. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
The bigger picture
Language choice is less important
than architecture
Be creative – brute force doesn’t
work
Be ready to be wrong
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 56

57. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Questions?
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 57

58. Colorado Software Summit: October 19 – 24, 2008 © Copyright 2008, Linkedin
Thanks for coming!
ilan@linkedin.com
http://www.linkedin.com/in/ikailan
Ikai Lan — From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 58