Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Cassandra Summit 2014: Cassandra at Instagram 2014

7,694 views

Published on

Presenter: Rick Branson, Infrastructure Engineer at Instagram
As Instagram has scaled to over 200 million users, so has our use of Cassandra. We've built new features and rebuilt old on Cassandra, and it's become an extremely mission-critical foundation of our production infrastructure. Rick will deliver a refresh of our use cases and go deep on the technical challenges we faced during our expansion.

Published in: Technology
  • Be the first to comment

Cassandra Summit 2014: Cassandra at Instagram 2014

  1. 1. CASSANDRA @ INSTAGRAM 2014 Rick Branson, Software Engineer, Backend Engineering @rbranson Cassandra Summit September 11th, 2014
  2. 2. 2013 in a Nutshell • Moving some log-style data from Redis to Cassandra • Saving $, better availability, more elasticity • Some problems, but we worked through them
  3. 3. Overview • "The Trouble with Redis" • A whole new category of functionality • Useful consumer advice
  4. 4. "THE TROUBLE WITH REDIS"
  5. 5. REDIS: THE GÜD PARTS
  6. 6. The Network Part It's a heap on the network with a great text-based protocol.
  7. 7. Extremely Efficient It does a lot with every ounce of CPU
  8. 8. Stable Very much so. Sets the bar for the vast NoSQL space.
  9. 9. Useful, Rich Types You won't find anywhere else.
  10. 10. REDIS: ZDARKSIDE
  11. 11. What's in there? Hello?
  12. 12. Snapshots (BGSAVE)
  13. 13. Single-Threaded (!)
  14. 14. Memory Allocator
  15. 15. (Lack of) Durability
  16. 16. Unsafe Replication
  17. 17. THE PHILOSOPHY
  18. 18. CACHE OR DATABASE? Neither.
  19. 19. Memory Cliff
  20. 20. Unlikely to Change :(
  21. 21. ZOOKEEPER???
  22. 22. ZOZO: *VERY* few writes, extremely high volume, no-latency reads without hotspots.
  23. 23. zozo agent /var/cache/zozo Consumer Process Consumer Process Consumer Process WATCH /zozo/files Writer UPDATE /zozo/files/knobs ZooKeeper Ensemble
  24. 24. Examples • "Knobs" key/integer live configuration • Small (<250K) User Lists • Security Data Sets (i.e. Blocked IPs)
  25. 25. JSON sucks.
  26. 26. Stay tuned.
  27. 27. zozo postmortem • Agent is dead simple, never needs updating • ZooKeeper resists both machine & human errors • Multi-region strategy coming
  28. 28. REDIS2CASSANDRA (omg, finally)
  29. 29. Some things we moved • Ad Serving Metadata • Account Suggestion Metadata • Churn Tracking • Follow Requests
  30. 30. Not a drop-in.
  31. 31. source target A B "User A has requested to follow User B" Outbound Requests Key Column Column A B ... Inbound Requests Key Column Column B A ...
  32. 32. INSTAGRAM DIRECT
  33. 33. Y U CASSANDRA? • Elasticity + Scalability • Write Availability • Storage Efficiency • Good Data Model Fit
  34. 34. WE CHOSE THRIFT.
  35. 35. fan-out-on-write • Caching doesn't work • Copy of full data for each recipient • Atomic batches super useful
  36. 36. #nocache
  37. 37. postid CompositeType(IntegerType(reversed=true), IntegerType, IntegerType) typecode entryid
  38. 38. POST_HEADER => creator_id: 12938 media_type: 1 asset_path: "/objects/23904801..." recipients: [28910, 208192] (SEEN, 28910) (SEEN, 208192) (LIKE, 208192) (COMMENT, 9302...) => [19238, "George wishes he ha..."] (COMMENT, 3499...) => [28910, "Weird"]
  39. 39. IMMUTABLE (mostly) LOG
  40. 40. FILTER-ON-READ
  41. 41. CQL3, please. no? ok. :(
  42. 42. THE PENDING QUEUE
  43. 43. sender B A C recipients D Key Column Column B [A's PostID] ... C [A's PostID] ... D [A's PostID] ... D First read, delete pending
  44. 44. 2000000 (POINTER, 1000000) 1500000 (POST_HEADER, 0) => ... (SEEN, 902394) (SEEN, 94083) (LIKE, 902394) 1000000 (POST_HEADER, 0) => ... (BUMPED, 0) => 2000000 (SEEN, 1298333) (SEEN, 848183) ...
  45. 45. Justin's Pending 25000 24000 23000 22000 21000 20000 19000 ... 1234 Justin's Inbox ... (1234, POST..) (1234, LIKE..) (1234, SEEN..) (1234, SEEN..) (1010, POST..) (1010, SEEN..) (1010, COMM..) ... Inbox ... (1234, POST..) (1234, LIKE..) (1234, SEEN..) (1234, SEEN..) (1010, POST..) (1010, SEEN..) (1010, COMM..) ... Inbox ... (1234, POST..) (1234, LIKE..) (1234, SEEN..) (1234, SEEN..) (1010, POST..) (1010, SEEN..) (1010, COMM..) ... Inbox ... (1234, POST..) (1234, LIKE..) (1234, SEEN..) (1234, SEEN..) (1010, POST..) (1010, SEEN..) (1010, COMM..) ... (1234, EXCLUDE, <JUSTIN>)
  46. 46. Rick's Unseen 849300 738939 649019 641192 629012 610392 483201 348931 Rick's Unseen 849300 738939 649019 641192 629012 610392 483201 348931 Rick's Unseen 849300 738939 649019 641192 629012 610392 483201 348931 8
  47. 47. DEPLOYMENT • 60 x hi1.4xlarge EC2 instances • 2X our highest estimates • Shrunk cluster after we knew load • Peace of mind • Data models held together
  48. 48. We ran C* in production. We learned some things.
  49. 49. young gc fun!
  50. 50. Yung GC has problems • Reads = young gen garbage • Double-collection bug in JDK 1.7+ • Best practices were for nodes with more writes
  51. 51. <blink>DEAR GOD ALMIGHTY!!!</blink> now we run: 10G Young Gen 20G Heap
  52. 52. This won't work out of the box.
  53. 53. -XX:+CMSScavengeBeforeRemark
  54. 54. -XX:CMSMaxAbortablePrecleanTime=60000
  55. 55. -XX:CMSWaitDuration=30000
  56. 56. -XX:+CMSEdenChunksRecordAlways -XX:+CMSParallelInitialMarkEnabled
  57. 57. 20-core Ivy Bridge 144GB RAM PCIe Flash 15,000 reads/sec 2,000 writes/sec
  58. 58. TWO ISSUES • Counters. Don't use them. • Hundreds of thousands / millions of tombstones.
  59. 59. GET 12349081 Async Storage Port Handler READ 12349081 Client RPC Thread Pool Queue ReadStage Thread Pool Queue
  60. 60. THANK YOU.

×