0
Cassandra                           @FormspringTuesday, August 30, 2011
Cassandra                           @Formspring                 Yet another Knight, Dragon and Princess Story.Tuesday, Aug...
Formspring helps people find out more about each         other by sharing interesting & personal responsesTuesday, August 3...
25M usersTuesday, August 30, 2011
3.5B responsesTuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
Follow[ing|ers]Tuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
Our situation before                               CassandraTuesday, August 30, 2011
MySQL + MemcacheTuesday, August 30, 2011
MySQL + Memcache                    • Flat MySQL tableTuesday, August 30, 2011
MySQL + Memcache                    • Flat MySQL table                    • Indexes (Followers vs Following)Tuesday, Augus...
MySQL + Memcache                    • Flat MySQL table                    • Indexes (Followers vs Following)              ...
Memcache                    • Stored as JSON listTuesday, August 30, 2011
Memcache                    • Stored as JSON list                    • Can get out of syncTuesday, August 30, 2011
MySQL                    • Hundreds of millions of rowsTuesday, August 30, 2011
MySQL                    • Hundreds of millions of rows                    • m2.4xl EC2 instanceTuesday, August 30, 2011
It all started with a                                  Feature.Tuesday, August 30, 2011
QOTDTuesday, August 30, 2011
QOTDTuesday, August 30, 2011
QOTD                    • Feb: 35K FollowersTuesday, August 30, 2011
QOTD                    • Feb: 35K Followers                    • Mid-Feb: 100KTuesday, August 30, 2011
QOTD                    • Feb: 35K Followers                    • Mid-Feb: 100K                    • Growing about 30K a w...
QOTD                    • Feb: 35K Followers                    • Mid-Feb: 100K                    • Growing about 30K a w...
QOTD                    • Feb: 35K Followers                    • Mid-Feb: 100K                    • Growing about 30K a w...
Then we added more                           features . . .Tuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
Temporary solutionTuesday, August 30, 2011
Redis                    • Key : JSON blob of IDsTuesday, August 30, 2011
Redis                    • Key : JSON blob of IDs                    • ~2-3s / insertTuesday, August 30, 2011
Frontend Ask                           Mem                                   MySQL                           CacheTuesday,...
Frontend Ask                Outbox     Mem                 QW                                   MySQL                     ...
Frontend Ask                           SELECT account_id                           FROM follow                Outbox     W...
Frontend Ask                Outbox     OH SHIT   Mem                 QW                                             MySQL ...
Frontend Ask                Outbox     OH SHIT   Mem                 QW                                             MySQL ...
Frontend Ask                Outbox       OH SHIT            Mem                 QW                                        ...
Frontend Ask                Outbox           OH SHIT            Mem                 QW                                    ...
Frontend Ask                Outbox           OH SHIT            Mem                 QW                                    ...
Frontend Ask     Outbox  *HANGING*                                 OH SHIT            Mem      QW                         ...
Frontend Ask             Outbox            *OOM*                                 OH SHIT            Mem              QW   ...
Frontend Ask         Outbox         Outbox          Outbox          Outbox                 OH SHIT            Mem         ...
Frontend Ask         Outbox         Outbox          Outbox          Outbox                 OH SHIT            Mem         ...
We need a solutionTuesday, August 30, 2011
FastTuesday, August 30, 2011
Tuesday, August 30, 2011
Step 1: ResearchTuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
Our first shot with                              CassandraTuesday, August 30, 2011
All ResponsesTuesday, August 30, 2011
Tuesday, August 30, 2011
Explain our stack hereTuesday, August 30, 2011
Index of responses to                      questions on the site.Tuesday, August 30, 2011
Index of responses to                      questions on the site.Tuesday, August 30, 2011
Explain our stack hereTuesday, August 30, 2011
Tuesday, August 30, 2011
Ask FollowersTuesday, August 30, 2011
How to scale ?Tuesday, August 30, 2011
Build a Social GraphTuesday, August 30, 2011
Social GraphTuesday, August 30, 2011
Build a Social Graph                    • PythonTuesday, August 30, 2011
Build a Social Graph                    • Python                    • Cassandra (Obviously..)Tuesday, August 30, 2011
Build a Social Graph                    • Python                    • Cassandra (Obviously..)                    • PyCassa...
Build a Social Graph                    • Python                    • Cassandra (Obviously..)                    • PyCassa...
4 types of actions                    • FollowingTuesday, August 30, 2011
4 types of actions                    • Following                    • FollowersTuesday, August 30, 2011
4 types of actions                    • Following                    • Followers                    • BlocksTuesday, Augus...
4 types of actions                    • Following                    • Followers                    • Blocks              ...
ThriftTuesday, August 30, 2011
Tuesday, August 30, 2011
Tuesday, August 30, 2011
CassandraTuesday, August 30, 2011
CassandraTuesday, August 30, 2011
CassandraTuesday, August 30, 2011
CassandraTuesday, August 30, 2011
CassandraTuesday, August 30, 2011
CassandraTuesday, August 30, 2011
Python + PycassaTuesday, August 30, 2011
Python + Pycassa                           Column FamiliesTuesday, August 30, 2011
Python + Pycassa                      Get list of Follow[ers/ing]Tuesday, August 30, 2011
Python + Pycassa                           More QueriesTuesday, August 30, 2011
Python + Pycassa                           Follow userTuesday, August 30, 2011
Switch to ProductionTuesday, August 30, 2011
Simultaneous writes to                   MySQL and CassandraTuesday, August 30, 2011
Switch to Production                                     Cassandra   MySQL                             Read               ...
Idempotence is your                                 friend.Tuesday, August 30, 2011
Migration from MySQL                       to CassandraTuesday, August 30, 2011
Load batches of user                           info into Cassandra.Tuesday, August 30, 2011
It’s OK if it fails.Tuesday, August 30, 2011
Remember                           idempotence is your                                 friend.Tuesday, August 30, 2011
Adjust your capacityTuesday, August 30, 2011
From 4 instances                                  to                            a 12 instancesTuesday, August 30, 2011
New Ask Followers                             InfrastructureTuesday, August 30, 2011
Frontend Ask                           Cassandra                            ClusterTuesday, August 30, 2011
Frontend Ask                Outbox                           Cassandra                 QW                            Clust...
Frontend Ask                Outbox                            Cassandra                 QW                             Clu...
Frontend Ask                Outbox                                                           Cassandra                 QW ...
Frontend Ask                Outbox                                                             Cassandra                 Q...
Frontend Ask                Outbox                                                                     Cassandra          ...
Frontend Ask                Outbox                                                                     Cassandra          ...
Frontend Ask                Outbox                                                                 Cassandra              ...
Frontend Ask                Outbox                                                                   Cassandra            ...
Count Following          14K/min hits at peak timeTuesday, August 30, 2011
Count Followers          40K/min hits at peak timeTuesday, August 30, 2011
Reduce our #instances                           12x                    12x                           Inbox                ...
Reduce our #instances                           12x                    4x                           Inbox                 ...
Reduce our #instances                                12x                 4x                           Inbox               ...
Counts for free with                                  0.8.1Tuesday, August 30, 2011
Counting . . .Tuesday, August 30, 2011
Counting is hard                                  Let’s go Shopping.Tuesday, August 30, 2011
Counting                    • FastTuesday, August 30, 2011
Counting                    • Fast                    • Not-accurateTuesday, August 30, 2011
Counting                    • Fast                    • Not-accurate                    • Faking “Read-repair”Tuesday, Aug...
CountingTuesday, August 30, 2011
Ghetto CountTuesday, August 30, 2011
Cassandra allowed us:                    • To create new features.Tuesday, August 30, 2011
Cassandra allowed us:                    • To create new features.                    • To scale.Tuesday, August 30, 2011
Cassandra allowed us:                    • To create new features.                    • To scale.                    • To ...
Cassandra allowed us:                    • To create new features.                    • To scale.                    • To ...
Cassandra allowed us:                    • To create new features.                    • To scale.                    • To ...
Come work with us !Tuesday, August 30, 2011
Upcoming SlideShare
Loading in...5
×

Cassandra @Formspring

2,807

Published on

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,807
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
19
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Transcript of "Cassandra @Formspring"

  1. 1. Cassandra @FormspringTuesday, August 30, 2011
  2. 2. Cassandra @Formspring Yet another Knight, Dragon and Princess Story.Tuesday, August 30, 2011
  3. 3. Formspring helps people find out more about each other by sharing interesting & personal responsesTuesday, August 30, 2011
  4. 4. 25M usersTuesday, August 30, 2011
  5. 5. 3.5B responsesTuesday, August 30, 2011
  6. 6. Tuesday, August 30, 2011
  7. 7. Tuesday, August 30, 2011
  8. 8. Tuesday, August 30, 2011
  9. 9. Tuesday, August 30, 2011
  10. 10. Tuesday, August 30, 2011
  11. 11. Tuesday, August 30, 2011
  12. 12. Tuesday, August 30, 2011
  13. 13. Tuesday, August 30, 2011
  14. 14. Tuesday, August 30, 2011
  15. 15. Tuesday, August 30, 2011
  16. 16. Tuesday, August 30, 2011
  17. 17. Tuesday, August 30, 2011
  18. 18. Tuesday, August 30, 2011
  19. 19. Tuesday, August 30, 2011
  20. 20. Tuesday, August 30, 2011
  21. 21. Tuesday, August 30, 2011
  22. 22. Follow[ing|ers]Tuesday, August 30, 2011
  23. 23. Tuesday, August 30, 2011
  24. 24. Tuesday, August 30, 2011
  25. 25. Tuesday, August 30, 2011
  26. 26. Tuesday, August 30, 2011
  27. 27. Our situation before CassandraTuesday, August 30, 2011
  28. 28. MySQL + MemcacheTuesday, August 30, 2011
  29. 29. MySQL + Memcache • Flat MySQL tableTuesday, August 30, 2011
  30. 30. MySQL + Memcache • Flat MySQL table • Indexes (Followers vs Following)Tuesday, August 30, 2011
  31. 31. MySQL + Memcache • Flat MySQL table • Indexes (Followers vs Following) • MemcacheTuesday, August 30, 2011
  32. 32. Memcache • Stored as JSON listTuesday, August 30, 2011
  33. 33. Memcache • Stored as JSON list • Can get out of syncTuesday, August 30, 2011
  34. 34. MySQL • Hundreds of millions of rowsTuesday, August 30, 2011
  35. 35. MySQL • Hundreds of millions of rows • m2.4xl EC2 instanceTuesday, August 30, 2011
  36. 36. It all started with a Feature.Tuesday, August 30, 2011
  37. 37. QOTDTuesday, August 30, 2011
  38. 38. QOTDTuesday, August 30, 2011
  39. 39. QOTD • Feb: 35K FollowersTuesday, August 30, 2011
  40. 40. QOTD • Feb: 35K Followers • Mid-Feb: 100KTuesday, August 30, 2011
  41. 41. QOTD • Feb: 35K Followers • Mid-Feb: 100K • Growing about 30K a week.Tuesday, August 30, 2011
  42. 42. QOTD • Feb: 35K Followers • Mid-Feb: 100K • Growing about 30K a week. • 200K: closer to Memcache 1MB limit.Tuesday, August 30, 2011
  43. 43. QOTD • Feb: 35K Followers • Mid-Feb: 100K • Growing about 30K a week. • 200K: closer to Memcache 1MB limit. • (with gz compaction . . .)Tuesday, August 30, 2011
  44. 44. Then we added more features . . .Tuesday, August 30, 2011
  45. 45. Tuesday, August 30, 2011
  46. 46. Tuesday, August 30, 2011
  47. 47. Temporary solutionTuesday, August 30, 2011
  48. 48. Redis • Key : JSON blob of IDsTuesday, August 30, 2011
  49. 49. Redis • Key : JSON blob of IDs • ~2-3s / insertTuesday, August 30, 2011
  50. 50. Frontend Ask Mem MySQL CacheTuesday, August 30, 2011
  51. 51. Frontend Ask Outbox Mem QW MySQL CacheTuesday, August 30, 2011
  52. 52. Frontend Ask SELECT account_id FROM follow Outbox WHERE following = 1234 Mem QW MySQL CacheTuesday, August 30, 2011
  53. 53. Frontend Ask Outbox OH SHIT Mem QW MySQL CacheTuesday, August 30, 2011
  54. 54. Frontend Ask Outbox OH SHIT Mem QW MySQL CacheTuesday, August 30, 2011
  55. 55. Frontend Ask Outbox OH SHIT Mem QW MySQL Cache Gazillion messagesTuesday, August 30, 2011
  56. 56. Frontend Ask Outbox OH SHIT Mem QW MySQL Cache Inbox Inbox Inbox Gazillion messages QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QW QW QWsTuesday, August 30, 2011
  57. 57. Frontend Ask Outbox OH SHIT Mem QW MySQL Cache Inbox Inbox Inbox Gazillion messages QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QW QW Inbox QWs ObjectTuesday, August 30, 2011
  58. 58. Frontend Ask Outbox *HANGING* OH SHIT Mem QW MySQL Cache Inbox Inbox Inbox Gazillion messages QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QW QW Inbox QWs ObjectTuesday, August 30, 2011
  59. 59. Frontend Ask Outbox *OOM* OH SHIT Mem QW MySQL Cache Inbox Inbox Inbox Gazillion messages QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QW QW Inbox QWs ObjectTuesday, August 30, 2011
  60. 60. Frontend Ask Outbox Outbox Outbox Outbox OH SHIT Mem Outbox QW Outbox QW MySQL Outbox QW QW QW QW QW Cache Inbox Inbox Inbox Gazillion messages QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QW QW Inbox QWs ObjectTuesday, August 30, 2011
  61. 61. Frontend Ask Outbox Outbox Outbox Outbox OH SHIT Mem Outbox QW Outbox QW MySQL QW OOM QW QW QW Cache Inbox Inbox Inbox Gazillion messages QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QW QW Inbox QWs ObjectTuesday, August 30, 2011
  62. 62. We need a solutionTuesday, August 30, 2011
  63. 63. FastTuesday, August 30, 2011
  64. 64. Tuesday, August 30, 2011
  65. 65. Step 1: ResearchTuesday, August 30, 2011
  66. 66. Tuesday, August 30, 2011
  67. 67. Tuesday, August 30, 2011
  68. 68. Tuesday, August 30, 2011
  69. 69. Tuesday, August 30, 2011
  70. 70. Tuesday, August 30, 2011
  71. 71. Our first shot with CassandraTuesday, August 30, 2011
  72. 72. All ResponsesTuesday, August 30, 2011
  73. 73. Tuesday, August 30, 2011
  74. 74. Explain our stack hereTuesday, August 30, 2011
  75. 75. Index of responses to questions on the site.Tuesday, August 30, 2011
  76. 76. Index of responses to questions on the site.Tuesday, August 30, 2011
  77. 77. Explain our stack hereTuesday, August 30, 2011
  78. 78. Tuesday, August 30, 2011
  79. 79. Ask FollowersTuesday, August 30, 2011
  80. 80. How to scale ?Tuesday, August 30, 2011
  81. 81. Build a Social GraphTuesday, August 30, 2011
  82. 82. Social GraphTuesday, August 30, 2011
  83. 83. Build a Social Graph • PythonTuesday, August 30, 2011
  84. 84. Build a Social Graph • Python • Cassandra (Obviously..)Tuesday, August 30, 2011
  85. 85. Build a Social Graph • Python • Cassandra (Obviously..) • PyCassaTuesday, August 30, 2011
  86. 86. Build a Social Graph • Python • Cassandra (Obviously..) • PyCassa • ThriftTuesday, August 30, 2011
  87. 87. 4 types of actions • FollowingTuesday, August 30, 2011
  88. 88. 4 types of actions • Following • FollowersTuesday, August 30, 2011
  89. 89. 4 types of actions • Following • Followers • BlocksTuesday, August 30, 2011
  90. 90. 4 types of actions • Following • Followers • Blocks • BlockersTuesday, August 30, 2011
  91. 91. ThriftTuesday, August 30, 2011
  92. 92. Tuesday, August 30, 2011
  93. 93. Tuesday, August 30, 2011
  94. 94. CassandraTuesday, August 30, 2011
  95. 95. CassandraTuesday, August 30, 2011
  96. 96. CassandraTuesday, August 30, 2011
  97. 97. CassandraTuesday, August 30, 2011
  98. 98. CassandraTuesday, August 30, 2011
  99. 99. CassandraTuesday, August 30, 2011
  100. 100. Python + PycassaTuesday, August 30, 2011
  101. 101. Python + Pycassa Column FamiliesTuesday, August 30, 2011
  102. 102. Python + Pycassa Get list of Follow[ers/ing]Tuesday, August 30, 2011
  103. 103. Python + Pycassa More QueriesTuesday, August 30, 2011
  104. 104. Python + Pycassa Follow userTuesday, August 30, 2011
  105. 105. Switch to ProductionTuesday, August 30, 2011
  106. 106. Simultaneous writes to MySQL and CassandraTuesday, August 30, 2011
  107. 107. Switch to Production Cassandra MySQL Read x Write x xTuesday, August 30, 2011
  108. 108. Idempotence is your friend.Tuesday, August 30, 2011
  109. 109. Migration from MySQL to CassandraTuesday, August 30, 2011
  110. 110. Load batches of user info into Cassandra.Tuesday, August 30, 2011
  111. 111. It’s OK if it fails.Tuesday, August 30, 2011
  112. 112. Remember idempotence is your friend.Tuesday, August 30, 2011
  113. 113. Adjust your capacityTuesday, August 30, 2011
  114. 114. From 4 instances to a 12 instancesTuesday, August 30, 2011
  115. 115. New Ask Followers InfrastructureTuesday, August 30, 2011
  116. 116. Frontend Ask Cassandra ClusterTuesday, August 30, 2011
  117. 117. Frontend Ask Outbox Cassandra QW ClusterTuesday, August 30, 2011
  118. 118. Frontend Ask Outbox Cassandra QW Cluster Iterative KWTuesday, August 30, 2011
  119. 119. Frontend Ask Outbox Cassandra QW Cluster 0, N) ser_id, ers(u get_follow Iterative KWTuesday, August 30, 2011
  120. 120. Frontend Ask Outbox Cassandra QW Cluster ’, ..., ‘ 1999’] ‘1234 , ‘1238 [ Iterative KWTuesday, August 30, 2011
  121. 121. Frontend Ask Outbox Cassandra QW Cluster ’, ..., ‘ 1999’] ‘1234 , ‘1238 [ Iterative KW N MessagesTuesday, August 30, 2011
  122. 122. Frontend Ask Outbox Cassandra QW Cluster ’, ..., ‘ 1999’] ‘1234 , ‘1238 [ Iterative KW N Messages Inbox QWTuesday, August 30, 2011
  123. 123. Frontend Ask Outbox Cassandra QW Cluster ’, ..., ‘ 1999’] ‘1234 , ‘1238 [ Iterative KW Inbox N Inbox QW ObjectTuesday, August 30, 2011
  124. 124. Frontend Ask Outbox Cassandra QW Cluster +N) ser_i d, N, N u wers( get _follo Iterative KW Inbox N Inbox QW ObjectTuesday, August 30, 2011
  125. 125. Count Following 14K/min hits at peak timeTuesday, August 30, 2011
  126. 126. Count Followers 40K/min hits at peak timeTuesday, August 30, 2011
  127. 127. Reduce our #instances 12x 12x Inbox Inbox Inbox Inbox Inbox Inbox Inbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs Inbox Outbox QWs QWs QWs QWs QWs QWs QWs QWs QWsTuesday, August 30, 2011
  128. 128. Reduce our #instances 12x 4x Inbox Inbox Inbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox Inbox Inbox QWs QWsInbox Inbox Inbox Inbox QWs QWs Outbox QWs QWs QWs QWs QWs QWs QWsTuesday, August 30, 2011
  129. 129. Reduce our #instances 12x 4x Inbox Inbox Inbox QWsInbox QWsInbox QWsInbox QWsInbox QWsInbox Inbox QWsInbox QWsInbox Inbox Inbox QWsInbox Inbox Outbox QWs Inbox QWs Inbox QWsInbox QWs QWs QWsInbox Inbox QWs QWs QWs+ QWs QWs QWs Iterator QWTuesday, August 30, 2011
  130. 130. Counts for free with 0.8.1Tuesday, August 30, 2011
  131. 131. Counting . . .Tuesday, August 30, 2011
  132. 132. Counting is hard Let’s go Shopping.Tuesday, August 30, 2011
  133. 133. Counting • FastTuesday, August 30, 2011
  134. 134. Counting • Fast • Not-accurateTuesday, August 30, 2011
  135. 135. Counting • Fast • Not-accurate • Faking “Read-repair”Tuesday, August 30, 2011
  136. 136. CountingTuesday, August 30, 2011
  137. 137. Ghetto CountTuesday, August 30, 2011
  138. 138. Cassandra allowed us: • To create new features.Tuesday, August 30, 2011
  139. 139. Cassandra allowed us: • To create new features. • To scale.Tuesday, August 30, 2011
  140. 140. Cassandra allowed us: • To create new features. • To scale. • To save money.Tuesday, August 30, 2011
  141. 141. Cassandra allowed us: • To create new features. • To scale. • To save money. • To sleep. (Almost there)Tuesday, August 30, 2011
  142. 142. Cassandra allowed us: • To create new features. • To scale. • To save money. • To sleep. (Almost there) • Lots of fun :)Tuesday, August 30, 2011
  143. 143. Come work with us !Tuesday, August 30, 2011
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×