Cassandra @Formspring

Cassandra @FormspringTuesday, August 30, 2011

Cassandra @Formspring Yet another Knight, Dragon and Princess Story.Tuesday, August 30, 2011

Formspring helps people ﬁnd out more about each other by sharing interesting & personal responsesTuesday, August 30, 2011

25M usersTuesday, August 30, 2011

3.5B responsesTuesday, August 30, 2011

Tuesday, August 30, 2011

Follow[ing|ers]Tuesday, August 30, 2011

Our situation before CassandraTuesday, August 30, 2011

MySQL + MemcacheTuesday, August 30, 2011

MySQL + Memcache • Flat MySQL tableTuesday, August 30, 2011

MySQL + Memcache • Flat MySQL table • Indexes (Followers vs Following)Tuesday, August 30, 2011

MySQL + Memcache • Flat MySQL table • Indexes (Followers vs Following) • MemcacheTuesday, August 30, 2011

Memcache • Stored as JSON listTuesday, August 30, 2011

Memcache • Stored as JSON list • Can get out of syncTuesday, August 30, 2011

MySQL • Hundreds of millions of rowsTuesday, August 30, 2011

MySQL • Hundreds of millions of rows • m2.4xl EC2 instanceTuesday, August 30, 2011

It all started with a Feature.Tuesday, August 30, 2011

QOTDTuesday, August 30, 2011

QOTD • Feb: 35K FollowersTuesday, August 30, 2011

QOTD • Feb: 35K Followers • Mid-Feb: 100KTuesday, August 30, 2011

QOTD • Feb: 35K Followers • Mid-Feb: 100K • Growing about 30K a week.Tuesday, August 30, 2011

QOTD • Feb: 35K Followers • Mid-Feb: 100K • Growing about 30K a week. • 200K: closer to Memcache 1MB limit.Tuesday, August 30, 2011

QOTD • Feb: 35K Followers • Mid-Feb: 100K • Growing about 30K a week. • 200K: closer to Memcache 1MB limit. • (with gz compaction . . .)Tuesday, August 30, 2011

Then we added more features . . .Tuesday, August 30, 2011

Temporary solutionTuesday, August 30, 2011

Redis • Key : JSON blob of IDsTuesday, August 30, 2011

Redis • Key : JSON blob of IDs • ~2-3s / insertTuesday, August 30, 2011

Frontend Ask Mem MySQL CacheTuesday, August 30, 2011

Frontend Ask Outbox Mem QW MySQL CacheTuesday, August 30, 2011

Frontend Ask SELECT account_id FROM follow Outbox WHERE following = 1234 Mem QW MySQL CacheTuesday, August 30, 2011

Frontend Ask Outbox OH SHIT Mem QW MySQL CacheTuesday, August 30, 2011

Frontend Ask Outbox OH SHIT Mem QW MySQL Cache Gazillion messagesTuesday, August 30, 2011

Frontend Ask Outbox OH SHIT Mem QW MySQL Cache Inbox Inbox Inbox Gazillion messages QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QW QW QWsTuesday, August 30, 2011

Frontend Ask Outbox OH SHIT Mem QW MySQL Cache Inbox Inbox Inbox Gazillion messages QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QW QW Inbox QWs ObjectTuesday, August 30, 2011

Frontend Ask Outbox *HANGING* OH SHIT Mem QW MySQL Cache Inbox Inbox Inbox Gazillion messages QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QW QW Inbox QWs ObjectTuesday, August 30, 2011

Frontend Ask Outbox *OOM* OH SHIT Mem QW MySQL Cache Inbox Inbox Inbox Gazillion messages QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QW QW Inbox QWs ObjectTuesday, August 30, 2011

Frontend Ask Outbox Outbox Outbox Outbox OH SHIT Mem Outbox QW Outbox QW MySQL Outbox QW QW QW QW QW Cache Inbox Inbox Inbox Gazillion messages QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QW QW Inbox QWs ObjectTuesday, August 30, 2011

Frontend Ask Outbox Outbox Outbox Outbox OH SHIT Mem Outbox QW Outbox QW MySQL QW OOM QW QW QW Cache Inbox Inbox Inbox Gazillion messages QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QWInbox Inbox QW QW QW Inbox QWs ObjectTuesday, August 30, 2011

We need a solutionTuesday, August 30, 2011

FastTuesday, August 30, 2011

Step 1: ResearchTuesday, August 30, 2011

Our ﬁrst shot with CassandraTuesday, August 30, 2011

All ResponsesTuesday, August 30, 2011

Explain our stack hereTuesday, August 30, 2011

Index of responses to questions on the site.Tuesday, August 30, 2011

Explain our stack hereTuesday, August 30, 2011

Ask FollowersTuesday, August 30, 2011

How to scale ?Tuesday, August 30, 2011

Build a Social GraphTuesday, August 30, 2011

Social GraphTuesday, August 30, 2011

Build a Social Graph • PythonTuesday, August 30, 2011

Build a Social Graph • Python • Cassandra (Obviously..)Tuesday, August 30, 2011

Build a Social Graph • Python • Cassandra (Obviously..) • PyCassaTuesday, August 30, 2011

Build a Social Graph • Python • Cassandra (Obviously..) • PyCassa • ThriftTuesday, August 30, 2011

4 types of actions • FollowingTuesday, August 30, 2011

4 types of actions • Following • FollowersTuesday, August 30, 2011

4 types of actions • Following • Followers • BlocksTuesday, August 30, 2011

4 types of actions • Following • Followers • Blocks • BlockersTuesday, August 30, 2011

ThriftTuesday, August 30, 2011

CassandraTuesday, August 30, 2011

Python + PycassaTuesday, August 30, 2011

Python + Pycassa Column FamiliesTuesday, August 30, 2011

Python + Pycassa Get list of Follow[ers/ing]Tuesday, August 30, 2011

Python + Pycassa More QueriesTuesday, August 30, 2011

Python + Pycassa Follow userTuesday, August 30, 2011

Switch to ProductionTuesday, August 30, 2011

Simultaneous writes to MySQL and CassandraTuesday, August 30, 2011

Switch to Production Cassandra MySQL Read x Write x xTuesday, August 30, 2011

Idempotence is your friend.Tuesday, August 30, 2011

Migration from MySQL to CassandraTuesday, August 30, 2011

Load batches of user info into Cassandra.Tuesday, August 30, 2011

It’s OK if it fails.Tuesday, August 30, 2011

Remember idempotence is your friend.Tuesday, August 30, 2011

Adjust your capacityTuesday, August 30, 2011

From 4 instances to a 12 instancesTuesday, August 30, 2011

New Ask Followers InfrastructureTuesday, August 30, 2011

Frontend Ask Cassandra ClusterTuesday, August 30, 2011

Frontend Ask Outbox Cassandra QW ClusterTuesday, August 30, 2011

Frontend Ask Outbox Cassandra QW Cluster Iterative KWTuesday, August 30, 2011

Frontend Ask Outbox Cassandra QW Cluster 0, N) ser_id, ers(u get_follow Iterative KWTuesday, August 30, 2011

Frontend Ask Outbox Cassandra QW Cluster ’, ..., ‘ 1999’] ‘1234 , ‘1238 [ Iterative KWTuesday, August 30, 2011

Frontend Ask Outbox Cassandra QW Cluster ’, ..., ‘ 1999’] ‘1234 , ‘1238 [ Iterative KW N MessagesTuesday, August 30, 2011

Frontend Ask Outbox Cassandra QW Cluster ’, ..., ‘ 1999’] ‘1234 , ‘1238 [ Iterative KW N Messages Inbox QWTuesday, August 30, 2011

Frontend Ask Outbox Cassandra QW Cluster ’, ..., ‘ 1999’] ‘1234 , ‘1238 [ Iterative KW Inbox N Inbox QW ObjectTuesday, August 30, 2011

Frontend Ask Outbox Cassandra QW Cluster +N) ser_i d, N, N u wers( get _follo Iterative KW Inbox N Inbox QW ObjectTuesday, August 30, 2011

Count Following 14K/min hits at peak timeTuesday, August 30, 2011

Count Followers 40K/min hits at peak timeTuesday, August 30, 2011

Reduce our #instances 12x 12x Inbox Inbox Inbox Inbox Inbox Inbox Inbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs Inbox Outbox QWs QWs QWs QWs QWs QWs QWs QWs QWsTuesday, August 30, 2011

Reduce our #instances 12x 4x Inbox Inbox Inbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox QWs QWsInbox Inbox Inbox Inbox QWs QWsInbox Inbox Inbox Inbox QWs QWs Outbox QWs QWs QWs QWs QWs QWs QWsTuesday, August 30, 2011

Reduce our #instances 12x 4x Inbox Inbox Inbox QWsInbox QWsInbox QWsInbox QWsInbox QWsInbox Inbox QWsInbox QWsInbox Inbox Inbox QWsInbox Inbox Outbox QWs Inbox QWs Inbox QWsInbox QWs QWs QWsInbox Inbox QWs QWs QWs+ QWs QWs QWs Iterator QWTuesday, August 30, 2011

Counts for free with 0.8.1Tuesday, August 30, 2011

Counting . . .Tuesday, August 30, 2011

Counting is hard Let’s go Shopping.Tuesday, August 30, 2011

Counting • FastTuesday, August 30, 2011

Counting • Fast • Not-accurateTuesday, August 30, 2011

Counting • Fast • Not-accurate • Faking “Read-repair”Tuesday, August 30, 2011

CountingTuesday, August 30, 2011

Ghetto CountTuesday, August 30, 2011

Cassandra allowed us: • To create new features.Tuesday, August 30, 2011

Cassandra allowed us: • To create new features. • To scale.Tuesday, August 30, 2011

Cassandra allowed us: • To create new features. • To scale. • To save money.Tuesday, August 30, 2011

Cassandra allowed us: • To create new features. • To scale. • To save money. • To sleep. (Almost there)Tuesday, August 30, 2011

Cassandra allowed us: • To create new features. • To scale. • To save money. • To sleep. (Almost there) • Lots of fun :)Tuesday, August 30, 2011

Come work with us !Tuesday, August 30, 2011

http://us-w1.rockmelt.com	37
http://twitter.com	12
https://twitter.com	11
https://twimg0-a.akamaihd.net	1

Cassandra @Formspring

by martincozzi on Aug 30, 2011

Accessibility

Categories

Upload Details

Usage Rights

4 Embeds 61

Statistics

Cassandra @Formspring Presentation Transcript