Download as PDF, PPTX
![@doanduyhai
The tokens
20
Random hash of #partition à token = hash(#p)
Hash: ] –x, x ]
hash range: 264 values
x = 264/2
C*
C*
C*
C*
C* C*
C* C*](https://image.slidesharecdn.com/cassandraintroduction-160324222223/85/Cassandra-introduction-2016-20-320.jpg)
![@doanduyhai
WHERE clause restrictions
66
What if I want to perform "arbitrary" WHERE clause ?
• search form scenario, dynamic search fields
Or Datastax Enterprise Search (battle field proven)
• ☞ Apache Solr (Lucene) integration (Datastax Enterprise Search)
• ☞ Same JVM, 1-cluster-2-products (Solr & Cassandra)
SELECT * FROM users WHERE solr_query = 'age:[33 TO *] AND gender:male';
SELECT * FROM users WHERE solr_query = 'lastname:*schwei?er';](https://image.slidesharecdn.com/cassandraintroduction-160324222223/85/Cassandra-introduction-2016-66-320.jpg)
![@doanduyhai
JSON syntax for SELECT
74
> SELECT JSON * FROM users WHERE id = 'me';
[json]
----------------------------------------
{"id": "me", "age": 25, "state": "CA”}
> SELECT JSON age,state FROM users WHERE id = 'me';
[json]
----------------------------------------
{"age": 25, "state": "CA"}
> SELECT age, toJson(state) FROM users WHERE id = 'me';
age | system.tojson(state)
-----+----------------------
25 | "CA"](https://image.slidesharecdn.com/cassandraintroduction-160324222223/85/Cassandra-introduction-2016-74-320.jpg)
![@doanduyhai
User Defined Functions (UDF)
77
CREATE [OR REPLACE] FUNCTION [IF NOT EXISTS]
maxOf (col1 int, col2 int)
CALL ON NULL INPUT | RETURNS NULL ON NULL INPUT
RETURN int
LANGUAGE java
AS $$
return Math.max(col1, col2);
$$;
SELECT maxOf(col1, col2) FROM table WHERE id = xxx;](https://image.slidesharecdn.com/cassandraintroduction-160324222223/85/Cassandra-introduction-2016-77-320.jpg)
![@doanduyhai
User Defined Aggregates (UDA)
78
CREATE [OR REPLACE] AGGREGATE [IF NOT EXISTS]
sum(bigint)
SFUNC accumulatorFunction
STYPE bigint
[FINALFUNC finalFunction]
INITCOND 0;
CREATE FUNCTION accumulatorFunction(accu bigint, column bigint)
RETURNS NULL ON NULL INPUT RETURN bigint LANGUAGE java
AS $$ return accu + colum; $$;](https://image.slidesharecdn.com/cassandraintroduction-160324222223/85/Cassandra-introduction-2016-78-320.jpg)
Uploaded byDuyhai Doan
Cassandra introduction 2016
There are a few options for performing more complex queries in Cassandra beyond the restrictions of the WHERE clause: 1. Denormalize/duplicate data across tables to allow querying on different columns. For example, have one table keyed on user ID and another keyed on message date to allow filtering by date. 2. Offload complex queries to an external search index like Solr or Elasticsearch that can handle full-text and complex queries, and keep Cassandra as the system of record. 3. Use Spark/Hive on Cassandra to run queries across the cluster and leverage their more powerful query engines. 4. Consider a different database if your queries require joins, complex where clauses, or don't map well to
More Related Content
Similar to Cassandra introduction 2016
Cassandra introduction 2016
- 1. Introduction to Cassandra DuyHaiDOAN Apache Cassandra Evangelist
- 2. @doanduyhai Datastax • Founded inApril 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 450+ employees • Headquarter in San Francisco Bay area • EU headquarter in London, offices in France and Germany • Datastax Enterprise = OSS Cassandra + extra features 2
- 3. @doanduyhai Cassandra history • createdat Facebook • open-sourced since 2008 • current version: 3.4 • column-oriented ☞ distributed table 3
- 4. 5 Cassandra keypoints • Linear scalability • Continuous availability • Multi Data-center native • Operational simplicity • Spark integration
- 5. @doanduyhai 1) Linear scalability 5 C* C* C* NetcoSports 3nodes, ≈3GB 1k+ nodes, PB+ YOU
- 6. @doanduyhai 2) Continuous availability 6 • thanks to the Dynamo architecture
- 7. @doanduyhai 3) Multi Data-centers 7 • out-of-the-box (config only) • AWS config for multi-regions DCs • GCE support • Microsoft Azure support • CloudStack support
- 8. @doanduyhai Multi DC usages Datalocality, disaster recovery 8 C* C* C* C* C* C* C* C* C* C* C* C* C* New York (DC1) London (DC2) Async replication
- 9. @doanduyhai Multi DC usages VirtualDC for workload segregation 9 C* C* C* C* C* C* C* C* C* C* C* C* C* Production (LIVE) Analytics (Spark) Async replication Same room
- 10. @doanduyhai Multi DC usages Proddata copy for back-up/benchmark 10 C* C* C* C* C* C* C* C* C* C* C* C* C* Use LOCAL_XXX Consistency Levels My tiny test DC READ-ONLY!!! Async replication
- 11. @doanduyhai 4) Operational simplicity 11 • 1 node = 1 process + 2 config files (cassandra.yaml + cassandra-rackdc.properties) • No role between nodes, perfect symmetry • deployment automation • OpsCenter* for • monitoring • provisioning • services (repair, performance, …) * only with Datastax Enterprise from Cassandra 3.x
- 12.
- 13. @doanduyhai 5) Eco System 13 • Apache Spark – Apache Cassandra integration • analytics • joins, aggregation • SparkSQL/Dataframe integration with CQL (predicates push down) • Apache Zeppelin – Apache Cassandra integration • web-based notebook • tabular/graph display
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20. @doanduyhai The tokens 20 Random hashof #partition à token = hash(#p) Hash: ] –x, x ] hash range: 264 values x = 264/2 C* C* C* C* C* C* C* C*
- 21. @doanduyhai Token ranges 21 A: −x,− 3x 4 ⎤ ⎦ ⎥ ⎥ ⎤ ⎦ ⎥ ⎥ B:− 3x 4 ,− 2x 4 ⎤ ⎦ ⎥ ⎥ ⎤ ⎦ ⎥ ⎥ C: − 2x 4 ,− x 4 ⎤ ⎦ ⎥ ⎥ ⎤ ⎦ ⎥ ⎥ D: − x 4 ,0 ⎤ ⎦ ⎥ ⎥ ⎤ ⎦ ⎥ ⎥ E: 0, x 4 ⎤ ⎦ ⎥ ⎥ ⎤ ⎦ ⎥ ⎥ F: x 4 , 2x 4 ⎤ ⎦ ⎥ ⎥ ⎤ ⎦ ⎥ ⎥ G: 2x 4 , 3x 4 ⎤ ⎦ ⎥ ⎥ ⎤ ⎦ ⎥ ⎥ H : 3x 4 ,x ⎤ ⎦ ⎥ ⎥ ⎤ ⎦ ⎥ ⎥ C* C* C* C* C* C* C* C*
- 22. @doanduyhai Distributed tables 22 H A E D B C GF user_id1 user_id2 user_id3 user_id4 user_id5 CREATE TABLE users( user_id int, …, PRIMARY KEY(user_id) ),
- 23. @doanduyhai Distributed tables 23 H A E D B C GF user_id1 user_id2 user_id3 user_id4 user_id5
- 24. @doanduyhai Linear scalability 24 H A E D B C GF Today = high load • disk occupation 80% • CPU 70% • saturated memory
- 25. @doanduyhai Scaling out 25 H A E D B C G F I J +2 nodes • disk occupation 50% • CPU 50% • memory ✌︎ Automatic data rebalancing • each node gives up some tokens • flag to throttle network bandwidth • streamingthroughput
- 26. @doanduyhai Automatic data re-balancingwith virtual nodes 26 A: B: C: D: E: F: G: H: A: B: C: D: E: F: G: H: I: J: +2 nodes
- 27.
- 28. Replication Model &Consistency
- 29. @doanduyhai Failure tolerance 29 Replication factor(RF) = 3 H A E D B C G F 1 2 3 {A, H, G} {B, A, H} {C, B, A}
- 30. @doanduyhai Coordinator node 30 Responsible forhandling requests (read/write) Every node can be coordinator • masterless • no SPOF • proxy role H A E D B C G F coordinator request 1 2 3
- 31. @doanduyhai Consistency level 31 Tunable atruntime • ONE • QUORUM (strict majority w.r.t RF) • ALL Applicable to any request (read/write)
- 32. @doanduyhai Consistency in action 32 BA A B A A Read ONE: A data replication in progress … Write ONE: B ack RF = 3, Write ONE, Read ONE
- 33. @doanduyhai Consistency in action 33 BA A B A A Read QUORUM: A data replication in progress … Write ONE: B ack RF = 3, Write ONE, Read QUORUM
- 34. @doanduyhai Consistency in action 34 BA A B A A Read ALL: B data replication in progress … Write ONE: B ack RF = 3, Write ONE, Read ALL
- 35. @doanduyhai Last Write Win 35 H A E D BC G F coordinator Read the value back 1 2 3 B (t2) A (t1) A (t1)
- 36. @doanduyhai Consistency in action 36 BB A B B A Read ONE: A data replication in progress … Write QUORUM: B ack RF = 3, Write QUORUM, Read ONE
- 37. @doanduyhai Consistency in action 37 BB A B B A Read QUORUM: A data replication in progress … Write QUORUM: B ack RF = 3, Write QUORUM, Read QUORUM
- 38. @doanduyhai Consistency level =trade-off 38
- 39. @doanduyhai Consistency level 39 ONE Fast, maynot read latest written value
- 40. @doanduyhai Consistency level 40 QUORUM Strict majorityw.r.t. Replication Factor Good balance
- 41. @doanduyhai Consistency level 41 ALL Paranoid Slow, lostof high availability
- 42. @doanduyhai Consistency level commonpatterns 42 ONERead + ONEWrite ☞ available for read/write even (N-1) replicas down QUORUMRead + QUORUMWrite ☞ available for read/write even if (RF - 1) replica (s) down
- 43.
- 44. Last Write Win& Compaction
- 45. @doanduyhai Last Write Win(LWW) 45 jdoe age name 33 John DOE INSERT INTO users(login, name, age) VALUES('jdoe', 'John DOE', 33); #partition
- 46. @doanduyhai Last Write Win(LWW) 46 INSERT INTO users(login, name, age) VALUES('jdoe', 'John DOE', 33); jdoe age (t1) name (t1) 33 John DOE auto-generated timestamp (μs) .
- 47. @doanduyhai Last Write Win(LWW) 47 UPDATE users SET age = 34 WHERE login = 'jdoe'; jdoe age (t1) name (t1) 33 John DOE jdoe age (t2) 34 SSTable1 SSTable2
- 48. @doanduyhai Last Write Win(LWW) 48 DELETE age FROM users WHERE login = 'jdoe'; jdoe age (t1) name (t1) 33 John DOE jdoe age (t2) 34 SSTable1 SSTable2 tombstone SSTable3 jdoe age (t3) ý
- 49. @doanduyhai Last Write Win(LWW) 49 SELECT age FROM users WHERE login = 'jdoe'; jdoe age (t1) name (t1) 33 John DOE jdoe age (t2) 34 SSTable1 SSTable2 SSTable3 jdoe age (t3) ý ???
- 50. @doanduyhai Last Write Win(LWW) 50 SELECT age FROM users WHERE login = 'jdoe'; jdoe age (t1) name (t1) 33 John DOE jdoe age (t2) 34 SSTable1 SSTable2 SSTable3 jdoe age (t3) ý ✓✕✕
- 51. @doanduyhai Compaction 51 SSTable1 SSTable2 SSTable3 jdoe age(t3) ý jdoe age (t1) name (t1) 33 John DOE jdoe age (t2) 34 New SSTable jdoe age (t3) name (t1) ý John DOE
- 52.
- 53. @doanduyhai Table creation 53 CREATE TABLEusers ( login text, name text, age int, … PRIMARY KEY(login)); partition key (#partition)
- 54. @doanduyhai DML statements 54 INSERT INTOusers(login, name, age) VALUES('jdoe', 'John DOE', 33); UPDATE users SET age = 34 WHERE login = 'jdoe'; DELETE age FROM users WHERE login = 'jdoe'; SELECT age FROM users WHERE login = 'jdoe';
- 55. @doanduyhai What’s about joins? 55 How can I join data between tables ? How can I model 1 – N relationships ? How to model a mailbox ? EmailsUser 1 n
- 56. @doanduyhai Compound primary key 56 CREATETABLE mailbox ( login text, message_id timeuuid, interlocutor text, message text, PRIMARY KEY((login), message_id)); partition key clustering column unicity
- 57. @doanduyhai Compound primary key 57 rsmith 2014-11-2116:00:00 ‘bobm’, ‘It’s really…’ 2014-11-21 17:32:12 ‘bobm’, ‘It depends..’ 2014-11-21 21:21:09 ‘bobm’, ‘Don’t do…’ … hsue 2014-11-21 11:04:43 ‘jdoe’, ‘Hi, …’ 2014-11-21 11:22:43 ‘rsmith’, ‘Hello,…’ jdoe 2014-11-21 11:00:00 ‘hsue’, ‘Hi there!’ 2014-11-21 11:22:43 ‘rsmith’, ‘Hello,…’ 2014-11-21 13:06:19 ‘bobm’, ‘Do you…’ ordered by clustering column (date) Not ordered
- 58. @doanduyhai Queries 58 Get message byuser and message_id (date) Get message by user and date interval SELECT * FROM mailbox WHERE login = 'jdoe' and message_id = ‘2014-11-21 16:00:00’; SELECT * FROM mailbox WHERE login = 'jdoe' and message_id <= ‘2014-11-25 23:59:59’ and message_id >= ‘2014-11-20 00:00:00’;
- 59. @doanduyhai Queries 59 Get message bymessage_id only Get message by date interval SELECT * FROM mailbox WHERE message_id = ‘2014-11-21 16:00:00’; ??? SELECT * FROM mailbox WHERE and message_id <= ‘2014-11-25 23:59:59’ ??? and message_id >= ‘2014-11-20 00:00:00’;
- 60. @doanduyhai Queries 60 Get message bymessage_id only (#partition not provided) Get message by date interval (#partition not provided) SELECT * FROM mailbox WHERE message_id = ‘2014-11-21 16:00:00’; SELECT * FROM mailbox WHERE and message_id <= ‘2014-11-25 23:59:59’ and message_id >= ‘2014-11-20 00:00:00’;
- 61. @doanduyhai Without #partition 61 No #partition ☞no token ☞ where are my data ? C* C* C* C* C* C* C* C* ❓ ❓ ❓ ❓ ❓ ❓ ❓ ❓
- 62. @doanduyhai Queries 62 Get message byuser range (range query on #partition) Get message by user pattern (non exact match on #partition) SELECT * FROM mailbox WHERE login >= 'hsue' and login <= 'jdoe'; SELECT * FROM mailbox WHERE login like ‘%doe%‘;
- 63. @doanduyhai WHERE clause restrictions 63 AllDML queries must provide #partition Only exact match (=) on #partition, range queries (<, ≤, >, ≥) not allowed • ☞ full cluster scan On clustering columns, only range queries (<, ≤, >, ≥) and exact match (=) WHERE clause only possible • on columns defined in PRIMARY KEY • on indexed columns ( )
- 64. @doanduyhai WHERE clause restrictions 64 Whatif I want to perform "arbitrary" WHERE clause ? • search form scenario, dynamic search fields
- 65. @doanduyhai WHERE clause restrictions 65 Whatif I want to perform "arbitrary" WHERE clause ? • search form scenario, dynamic search fields NEW SASI secondary index (contribution from Apple engineers) • ☞ https://github.com/apache/cassandra/blob/trunk/doc/SASI.md • ☞ integrated Query Planner (for multi-criteria searches) • ☞ Cassandra version ≥ 3.4 • ☞ still buggy (CASSANDRA-11383, CASSANDRA-11399, …) SELECT * FROM users WHERE firstname LIKE '%John%'; SELECT * FROM users WHERE age >= 20 AND age <=30;
- 66. @doanduyhai WHERE clause restrictions 66 Whatif I want to perform "arbitrary" WHERE clause ? • search form scenario, dynamic search fields Or Datastax Enterprise Search (battle field proven) • ☞ Apache Solr (Lucene) integration (Datastax Enterprise Search) • ☞ Same JVM, 1-cluster-2-products (Solr & Cassandra) SELECT * FROM users WHERE solr_query = 'age:[33 TO *] AND gender:male'; SELECT * FROM users WHERE solr_query = 'lastname:*schwei?er';
- 67.
- 68.
- 69. @doanduyhai Collection types 69 CREATE TABLEusers ( login text, name text, age int, friends set<text>, hobbies list<text>, languages map<int, text>, … PRIMARY KEY(login));
- 70. @doanduyhai User Defined Type(UDT) 70 Instead of CREATE TABLE users ( login text, … street_number int, street_name text, postcode int, country text, … PRIMARY KEY(login));
- 71. @doanduyhai User Defined Type(UDT) 71 CREATE TYPE address ( street_number int, street_name text, postcode int, country text); CREATE TABLE users ( login text, … location frozen <address>, … PRIMARY KEY(login));
- 72. @doanduyhai UDT Insert 72 INSERT INTOusers(login,name, location) VALUES ( 'jdoe', 'John DOE', { 'street_number': 124, 'street_name': 'Congress Avenue', 'postcode': 95054, 'country': ‘USA’ });
- 73. @doanduyhai JSON syntax forINSERT/UPDATE/DELETE 73 CREATE TABLE users ( id text PRIMARY KEY, age int, state text ); INSERT INTO users JSON '{"id": "user123", "age": 42, "state": "TX"}’; INSERT INTO users(id, age, state) VALUES('me', fromJson('20'), 'CA'); UPDATE users SET age = fromJson('25’) WHERE id = fromJson('"me"'); DELETE FROM users WHERE id = fromJson('"me"');
- 74. @doanduyhai JSON syntax forSELECT 74 > SELECT JSON * FROM users WHERE id = 'me'; [json] ---------------------------------------- {"id": "me", "age": 25, "state": "CA”} > SELECT JSON age,state FROM users WHERE id = 'me'; [json] ---------------------------------------- {"age": 25, "state": "CA"} > SELECT age, toJson(state) FROM users WHERE id = 'me'; age | system.tojson(state) -----+---------------------- 25 | "CA"
- 75. @doanduyhai Why Materialized Views? Relieve the pain of manual denormalization 75 CREATE TABLE user( id int PRIMARY KEY, country text, …); CREATE TABLE user_by_country( country text, id int, …, PRIMARY KEY(country, id));
- 76. @doanduyhai Materialzed View InAction 76 CREATE MATERIALIZED VIEW user_by_country AS SELECT country, id, firstname, lastname FROM user WHERE country IS NOT NULL AND id IS NOT NULL PRIMARY KEY(country, id) CREATE TABLE user_by_country ( country text, id int, firstname text, lastname text, PRIMARY KEY(country, id));
- 77. @doanduyhai User Defined Functions(UDF) 77 CREATE [OR REPLACE] FUNCTION [IF NOT EXISTS] maxOf (col1 int, col2 int) CALL ON NULL INPUT | RETURNS NULL ON NULL INPUT RETURN int LANGUAGE java AS $$ return Math.max(col1, col2); $$; SELECT maxOf(col1, col2) FROM table WHERE id = xxx;
- 78. @doanduyhai User Defined Aggregates(UDA) 78 CREATE [OR REPLACE] AGGREGATE [IF NOT EXISTS] sum(bigint) SFUNC accumulatorFunction STYPE bigint [FINALFUNC finalFunction] INITCOND 0; CREATE FUNCTION accumulatorFunction(accu bigint, column bigint) RETURNS NULL ON NULL INPUT RETURN bigint LANGUAGE java AS $$ return accu + colum; $$;
- 79.
- 80.