Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A glimpse of cassandra 4.0 features netflix

1,586 views

Published on

These slides are from the recent meetup @ Uber - Apache Cassandra at Uber and Netflix on new features in 4.0.
Abstract:

A glimpse of Cassandra 4.0 features:

There are a lot of exciting features coming in 4.0, but this talk covers some of the features that we at Netflix are particularly excited about and looking forward to. In this talk, we present an overview of just some of the many improvements shipping soon in 4.0.

Published in: Software
  • Be the first to comment

A glimpse of cassandra 4.0 features netflix

  1. 1. A glimpse of Cassandra 4.0 Vinay Chella Cloud Database Engineering
  2. 2. Vinay Chella Cloud Data Architect Cloud Database Engineering @ Netflix
  3. 3. State of C* @ Netflix Notable 4.0+ Changes ● Reliability ● Features ○ Audit Logging ● Correctness ○ Scheduled Repair Overview
  4. 4. There are lot of exciting features coming in 4.0, but this talk covers some of the features that we at Netflix are particularly excited about and looking forward to. There are thousands of improvements shipping soon in 4.0, and some in later 4.x. This is just a sample of the goodness. Disclaimer
  5. 5. ● In process of migrating to Apache Cassandra 3.0 ● Majority on Cassandra 2.1 ● Cassandra is the source truth for 99%+ streaming persistent data State of C* @ Netflix
  6. 6. Cassandra 4.0 for Netflix Reliability Correctness New Features Why 4.0?
  7. 7. Reliability Your database should be available (aka fast)
  8. 8. Internode Networking CASSANDRA-8457!! ● No more thread per peer, fully async server-server communication ● Streaming 20% faster (12229) ● Access to critical OS networking features
  9. 9. ● Gossip slows down (8457, 12966) ● Restarted nodes coordinate before they have functional connections (13993, 14297) ● Non-restarted nodes will continue sending on dead connections for a while (14358) ● DynamicEndpointSnitch sends to latent nodes after restart (14459) Restarting Cassandra
  10. 10. Restarting Cassandra in 4.x
  11. 11. ● Meet SLOs with Hybrid Speculation (14293) ○ MIN(99PERCENTILE,10MS) ~= “only speculate if I am slower than P99 SLO” ○ MAX(99PERCENTILE,100MS) ~= “stop speculating if the cluster is hosed” ● Reduce default number of vnodes (13701) ○ vnodes reduce availability, better to have fewer. (mailing list) ○ Context: https://github.com/jolynch/python_performance_toolkit/tree/master/notebooks/cassandra_availability ● Which queries are slow/huge? (13001, 14347) ● Circuit break queries of death (12106) Some Other Improvements
  12. 12. Features Your database should help you build great apps
  13. 13. CDC improvements Materialized views in other datastores CREATE TABLE foo (a int, b text, PRIMARY KEY(a)) WITH cdc=true; ALTER TABLE foo WITH cdc=true; ALTER TABLE foo WITH cdc=false;
  14. 14. Pluggable Storage Engine CASSANDRA-13474 ● Unlocking great performance improvements ○ Rocksandra (13476) ○ Persistent memory (13981)
  15. 15. Audit Logging - #12151
  16. 16. Where should it go in the first place? ● Why isn’t it the client’s job? ● Why not via dynamic tracing (e.g. cqltrace)? Design.
  17. 17. Once we’re in the database ● Why not log to database itself? ● Why is it in files? ○ Should it be restricted to one type of implementation? ● Why not log everything? Design. (continued)
  18. 18. ● Audits everything ● Yaml based configuration ● Highly performant ● Pluggable ● Supports FQL ● BinLog ● Default implementations ○ BinAuditLogger ○ FileAuditLogger ● Droppable jars for custom loggers Audit Logging.
  19. 19. ● User ● Host ● Source ip address ● Source port ● Timestamp ● Type ○ SELECT, INSERT, etc., What does it log. ● Category: ○ DDL, DML, etc., ● Keyspace ● Scope ○ Table name, Function name etc., ● Operation ○ Select * from tbl1 limit 2;
  20. 20. How to configure.
  21. 21. user:anonymous|host:localhost/X.X.X.X|source:/X.X.X.X|port:60878|timestamp:1521158923615|type:USE_KS |category:DDL|ks:dev1|operation:USE "dev1" user:anonymous|host:null|source:/X.X.X.X|port:57229|timestamp:1524127488487|type:USE_KEYSPACE|catego ry:OTHER|ks:dev1|operation:use dev1 ; user:anonymous|host:null|source:/X.X.X.X|port:57229|timestamp:1524127494445|type:SELECT|category:QUE RY|ks:dev1|scope:emp|operation:SELECT * from emp limit 2; user:anonymous|host:null|source:/X.X.X.X|port:57229|timestamp:1524127536425|type:CREATE_TABLE|catego ry:DDL|ks:dev1|scope:emp2|operation:create TABLE emp2 (id int, c1 text, c2 text, PRIMARY KEY (id )); user:anonymous|host:null|source:/X.X.X.X|port:57229|timestamp:1528147672512|type:DROP_TABLE|category :DDL|ks:dev1|scope:emp2|operation:DROP TABLE emp2; Sample logs.
  22. 22. $ ./nodetool enableauditlog $ ./nodetool disableauditlog StorageService.java:5420 - AuditLog is enabled with logger: [BinAuditLogger], included_keyspaces: [movies, thumbs_ratings], excluded_keyspaces: [star_ratings], included_categories: [DDL, DML, DCL], excluded_categories: [QUERY],included_users: [prod_user1, prod_user2], excluded_users: [admin_user, ops_user] Ease of use with NodeTool
  23. 23. ● Per table write metrics (14232) ○ Very important for multi-tenant clusters ● Virtual tables (7622) ○ Ask Cassandra for system state, config via CQL ● NetworkTopologyStrategy takes a default replication count (14303) ○ CREATE KEYSPACE test WITH replication = {'class': 'NetworkTopologyStrategy', default_datacenter_replication: 3 } ● Operate your database with HTTP (14395) Some Other New Features
  24. 24. Correctness Your database should give you the correct responses
  25. 25. Incremental Repair Works! CASSANDRA-9143 et al. !! ● Incremental repair running super quickly on petabyte datasets ● Preview data inconsistency without streaming (13257) ● Without repair Cassandra is hopefully consistent. Probably not what you expect from a database.
  26. 26. Repair Scheduling* CASSANDRA-14346 ● Decentralized ● Fault Tolerant ● “Just works” ● Rough design consensus ○ Hopefully will be merged in 4.x
  27. 27. ● Ideal Consistency Level (13289) ○ Having metrics on inconsistency is useful ● Cassandra should be correct when nodes fail (5901) ○ Do you run repair after node failures? ● Continuous repair (13924) ○ Repair only inconsistent data, no wasted work. Some Other Improvements
  28. 28. Easy Repair Operations # Get repair configuration or status curl localhost:7007/v1/repair/config curl localhost:7007/v1/repair/status # Mutate configuration live, or start or pause repair curl -XPOST localhost:7007/v1/repair/config curl -XPOST localhost:7007/v1/repair/start curl -XPOST localhost:7007/v1/repair/stop If 14346 can get finished and reviewed for 4.x we can: ● Make repair easy to schedule and run ● Allow non java interactions via HTTP ● Ship repair out of the box, not in addons.
  29. 29. Thank you.

×