Professional Cassandra support and services




Tuesday, August 10, 2010
Cassandra: Present & Future
                           Jonathan Ellis
                             @spyced




Tuesday, Au...
Cassandra 0.6 & 0.7
                                 Jonathan Ellis
                                   @spyced




Tuesday...
Quiet change of policy

                    • 0.5.1 was bug fixes only
                    • Too early to be strict about b...
1500
                                                                                      mails sent

          1125



 ...
Lots of bug fixes


                    • 85 issues marked Resolved/Fixed in 0.6
                           branch after 0....
Runtime configuration

                    • concurrent reads, writes (0.6.2)
                           •   making it easi...
Performance

                    • JVM GC defaults (0.6.2)
                    • Faster commitlog (0.6.2)
                ...
Bulletproofing
                    •      HH disable (0.6.2)

                    •      compaction priority (0.6.3)

     ...
Hinted Handoff
                    •      0.6.0: send hints to natural replicas

                    •      0.6.0: fix row-...
Why keep HH around?




                           https://www.cloudkick.com/blog/2010/jan/12/visual-ec2-latency/



Tuesd...
Compaction priority


                    -XX:+UseThreadPriorities 
                    -XX:ThreadPriorityPolicy=42 
     ...
http://www.javamex.com/tutorials/threads/priority_what.shtml

Tuesday, August 10, 2010
JMX for bloom filters


                    • o.a.c.db:ColumnFamilyStores
                           •   getBloomFilterFals...
Flow control in 0.5


                    • Why backpressure doesn’t fit Cassandra



Tuesday, August 10, 2010
Flow Control in 0.6.4
                    • Replica nodes drop hopeless requests on
                           the floor
  ...
Flow Control, 0.6.4
                              IncomingTcpConnection


                               Message Deseriali...
IncomingTcpConnection


                                  Message Deserializer


                           Read         G...
Flow Control, 0.6.5
                               IncomingTcpConnection


                            Read      Gossip   ...
Dynamic snitch


                    • sortByProximity



Tuesday, August 10, 2010
Open problems

                    • Linux/mmap/swap unholy trio (0.6.5)
                    • Memory fragmentation (0.6.5...
mmap and swap
                    • The problem
                    • Mitigations
                           •   mmap_inde...
GC Fragmentation


                    • Culprit of infamous CASSANDRA-1014?
                    • Mitigation: tune with m...
Compaction and caches


                    • Compactions wrecks the OS fs cache
                    • Wrecks Cassandra ke...
0.7

Tuesday, August 10, 2010
New in 0.7

                    • live schema changes
                    • large rows
                    • secondary ind...
Live schema changes


                    • Details: http://www.riptano.com/blog/live-
                           schema-u...
Large rows


                    • 0.6: smaller of {2GB, memory limit}
                    • 0.7: in_memory_compaction_lim...
Secondary indexes




Tuesday, August 10, 2010
Streaming in 0.6
                               W           A




                                               F
       ...
W           A




                                                F
                                   (A-F]


           ...
W            A




                                                F


                                   Data
           ...
Streaming in 0.7
                               W            A




                                                F



  ...
DatacenterStrategy

                    • RackAwareStrategy is tuned for 3 replicas
                           and 2 data ...
Minor features in 0.7

                    • read_repair_chance
                    • per-keyspace request scheduling
    ...
0.7 API changes

                    • String keys become byte[]
                    • Thrift keyspace argument moved to
 ...
0.7 performance
                    • Reads roughly 100% faster, thanks largely to
                           removing Str...
Thrift

                    • OOMs on malformed packets
                    • Python Unicode string issues
               ...
After 0.7.0
                    • IndexOperator.GT
                    • Triggers / plugins
                    • Avro?
  ...
Questions




Tuesday, August 10, 2010
Upcoming SlideShare
Loading in …5
×

State of Cassandra, August 2010

7,208 views
7,121 views

Published on

Keynote from the 2010 Cassandra Summit

Published in: Technology
0 Comments
15 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
7,208
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
221
Comments
0
Likes
15
Embeds 0
No embeds

No notes for slide

State of Cassandra, August 2010

  1. 1. Professional Cassandra support and services Tuesday, August 10, 2010
  2. 2. Cassandra: Present & Future Jonathan Ellis @spyced Tuesday, August 10, 2010
  3. 3. Cassandra 0.6 & 0.7 Jonathan Ellis @spyced Tuesday, August 10, 2010
  4. 4. Quiet change of policy • 0.5.1 was bug fixes only • Too early to be strict about bugfix-only policy in stable branch, especially w/ 0.7 being longer/more break-y • Maybe after 1.0? Tuesday, August 10, 2010
  5. 5. 1500 mails sent 1125 750 375 0 Jan Feb Apr May Jun Jul (0.5) (0.5.1) Mar (0.6, 0.6.1) (0.6.2) (0.6.3) (0.6.4) Tuesday, August 10, 2010
  6. 6. Lots of bug fixes • 85 issues marked Resolved/Fixed in 0.6 branch after 0.6 released Tuesday, August 10, 2010
  7. 7. Runtime configuration • concurrent reads, writes (0.6.2) • making it easier to bandage your foot after you shoot it • PhiConvictThreshold (0.6.2) Tuesday, August 10, 2010
  8. 8. Performance • JVM GC defaults (0.6.2) • Faster commitlog (0.6.2) • Faster range slice, Hadoop jobs (0.6.1, 2) • Better parallelization of multiget (0.6.4) • UTF8Type, UUIDType optimizations (0.6.5) Tuesday, August 10, 2010
  9. 9. Bulletproofing • HH disable (0.6.2) • compaction priority (0.6.3) • HH hourly scan (0.6.3) • JMX metrics for row-level bloom filters (0.6.3) • Flow control (0.6.4, 5) • HH paging (0.6.5) • Dynamic snitch (0.6.5) Tuesday, August 10, 2010
  10. 10. Hinted Handoff • 0.6.0: send hints to natural replicas • 0.6.0: fix row-level concurrency bottleneck • 0.6.2: option to disable entirely • 0.6.3: remove hourly scan • 0.6.4: lower priority • 0.6.5: paging of large hinted rows • 0.7.0: large rows Tuesday, August 10, 2010
  11. 11. Why keep HH around? https://www.cloudkick.com/blog/2010/jan/12/visual-ec2-latency/ Tuesday, August 10, 2010
  12. 12. Compaction priority -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Dcassandra.compaction.priority=1 Extended to HH in 0.6.4 Tuesday, August 10, 2010
  13. 13. http://www.javamex.com/tutorials/threads/priority_what.shtml Tuesday, August 10, 2010
  14. 14. JMX for bloom filters • o.a.c.db:ColumnFamilyStores • getBloomFilterFalsePositives • [not in nodetool yet] Tuesday, August 10, 2010
  15. 15. Flow control in 0.5 • Why backpressure doesn’t fit Cassandra Tuesday, August 10, 2010
  16. 16. Flow Control in 0.6.4 • Replica nodes drop hopeless requests on the floor • Coordinator node is unaffected • TimedOutException signals client to back off • Requires enough memory to buffer RPCTimeout’s worth of requests • (In the short term, you’re still screwed) Tuesday, August 10, 2010
  17. 17. Flow Control, 0.6.4 IncomingTcpConnection Message Deserializer Uncapped Read Mutation Capped at 4096 Tuesday, August 10, 2010
  18. 18. IncomingTcpConnection Message Deserializer Read Gossip Mutation Tuesday, August 10, 2010
  19. 19. Flow Control, 0.6.5 IncomingTcpConnection Read Gossip Mutation Uncapped Tuesday, August 10, 2010
  20. 20. Dynamic snitch • sortByProximity Tuesday, August 10, 2010
  21. 21. Open problems • Linux/mmap/swap unholy trio (0.6.5) • Memory fragmentation (0.6.5?) • Compaction effect on caches (0.7.1?) Tuesday, August 10, 2010
  22. 22. mmap and swap • The problem • Mitigations • mmap_index_only • swappiness=0 • turn off swap • mlockall at startup (Xms=Xmx) Tuesday, August 10, 2010
  23. 23. GC Fragmentation • Culprit of infamous CASSANDRA-1014? • Mitigation: tune with much larger new generation / tenuring threshold? Tuesday, August 10, 2010
  24. 24. Compaction and caches • Compactions wrecks the OS fs cache • Wrecks Cassandra key cache, too • (but not row cache) Tuesday, August 10, 2010
  25. 25. 0.7 Tuesday, August 10, 2010
  26. 26. New in 0.7 • live schema changes • large rows • secondary indexes • efficient Streaming • DatacenterStrategy Tuesday, August 10, 2010
  27. 27. Live schema changes • Details: http://www.riptano.com/blog/live- schema-updates-cassandra-07 Tuesday, August 10, 2010
  28. 28. Large rows • 0.6: smaller of {2GB, memory limit} • 0.7: in_memory_compaction_limit_in_mb Tuesday, August 10, 2010
  29. 29. Secondary indexes Tuesday, August 10, 2010
  30. 30. Streaming in 0.6 W A F (A-L] T L Tuesday, August 10, 2010
  31. 31. W A F (A-F] (A-F] T (F-L] L Tuesday, August 10, 2010
  32. 32. W A F Data T L Index Filter Tuesday, August 10, 2010
  33. 33. Streaming in 0.7 W A F T L Index Filter Tuesday, August 10, 2010
  34. 34. DatacenterStrategy • RackAwareStrategy is tuned for 3 replicas and 2 data centers • DS allows configuring replicas per data center, per Keyspace Tuesday, August 10, 2010
  35. 35. Minor features in 0.7 • read_repair_chance • per-keyspace request scheduling • Hadoop OutputFormat • Per CF what used to be global (gc_grace_seconds, memtable thresholds) Tuesday, August 10, 2010
  36. 36. 0.7 API changes • String keys become byte[] • Thrift keyspace argument moved to set_keyspace • i64 timestamp becomes Clock • SlicePredicate for _count methods Tuesday, August 10, 2010
  37. 37. 0.7 performance • Reads roughly 100% faster, thanks largely to removing String creation • Row-cached reads up to 8x faster after optimizations by tjake and jbellis • Optimizations for reads of large rows • 0.7.1? ~20% improvement everywhere from Thrift optimizations Tuesday, August 10, 2010
  38. 38. Thrift • OOMs on malformed packets • Python Unicode string issues • PHP support is buggy and maintainerless Tuesday, August 10, 2010
  39. 39. After 0.7.0 • IndexOperator.GT • Triggers / plugins • Avro? • On-disk data format improvements (Compression, heirarchical data?) • Auth Tuesday, August 10, 2010
  40. 40. Questions Tuesday, August 10, 2010

×