2. Maurits Lawende
•
•
•
dinsdag 12 november 13
Work at Dutch Open Projects (DOP) since 2007
Development and technical design for challenging Drupal sites
Development of SaaS solutions in PHP & NodeJS
18. Operations in Cassandra 1.0
•
CREATE KEYSPACE name
•
•
•
•
dinsdag 12 november 13
USE name
CREATE COLUMN FAMILY name
DROP KEYSPACE name
DROP COLUMN FAMILY name
19. Operations in Cassandra 1.0
•
•
•
•
•
dinsdag 12 november 13
SET columnfamily[‘row’][‘column’] = ‘value’;
GET columnfamily[‘row’]
LIST columnfamily
DEL columnfamily[‘row’]
DEL columnfamily[‘row’][‘column’]
21. Operations in Cassandra 1.0
post
•
•
•
dinsdag 12 november 13
post[‘uuid’][‘title’] = ‘First post!’;
user[‘mau’][‘firstname’] = ‘Maurits’;
user[‘mau’][‘lastname’] = ‘Lawende’;
title
uuid First post!
user
firstname
mau Maurits
lastname
Lawende
22. Operations in Cassandra 1.0
sorted by rowkey, columnname (all ascending)
•
•
•
dinsdag 12 november 13
post[‘uuid’][‘title’] = ‘First post!’;
user[‘mau’][‘firstname’] = ‘Maurits’;
user[‘mau’][‘lastname’] = ‘Lawende’;
24. Operations in Cassandra 1.0
How to get a list
of blogs by “mau”?
•
•
•
dinsdag 12 november 13
post[‘uuid’][‘title’] = ‘First post!’;
post[‘uuid’][‘user’] = ‘mau’;
user[‘mau’][‘firstname’] = ‘Maurits’;
25. Operations in Cassandra 1.0
How to get a list
of blogs by “mau”?
•
•
•
dinsdag 12 november 13
post[‘uuid’][‘title’] = ‘First post!’;
post[‘uuid’][‘user’] = ‘mau’;
user[‘mau’][‘firstname’] = ‘Maurits’;
WHERE user = ‘mau’
26. Operations in Cassandra 1.0
How to get a list
of blogs by “mau”?
•
•
•
dinsdag 12 november 13
WHERE user = ‘mau’
post[‘uuid’][‘title’] = ‘First post!’;
Bad Request:
No indexed columns present in
post[‘uuid’][‘user’] = ‘mau’;
by-columns clause with
user[‘mau’][‘firstname’] = ‘Maurits’;
Equal operator
27. Operations in Cassandra 1.0
How to get a list
of blogs by “mau”?
•
•
•
WHERE user = ‘mau’
post[‘uuid’][‘title’] = ‘First post!’;
Bad Request:
No indexed columns present in
post[‘uuid’][‘user’] = ‘mau’;
by-columns clause with
user[‘mau’][‘firstname’] = ‘Maurits’;
Equal operator
sequal scans
are rejected
dinsdag 12 november 13
28. Operations in Cassandra 1.0
How to get a list
of blogs by “mau”?
WHERE user = ‘mau’
post[‘uuid’][‘title’] = ‘First post!’;
Bad Request:
No indexed columns present in
post[‘uuid’][‘user’] = ‘mau’;
by-columns clause with
user[‘mau’][‘firstname’] = ‘Maurits’;
Equal operator
Bad Request: Order by is currently only supported
on the clustered columns of the PRIMARY KEY
•
•
•
dinsdag 12 november 13
29. Operations in Cassandra 1.0
How to get a list
of blogs by “mau”?
WHERE user = ‘mau’
post[‘uuid’][‘title’] = ‘First post!’;
Bad Request:
No indexed columns present in
post[‘uuid’][‘user’] = ‘mau’;
by-columns clause with
user[‘mau’][‘firstname’] = ‘Maurits’;
Equal operator
Bad Request: Order by is currently only supported
on the clustered columns of the PRIMARY KEY
Bad Request: ORDER BY is only supported when the partition key is
restricted by an EQ or an IN.
•
•
•
dinsdag 12 november 13
30. Operations in Cassandra 1.0
How to get a list
of blogs by “mau”?
•
•
•
dinsdag 12 november 13
post[‘uuid’][‘title’] = ‘First post!’;
post[‘uuid’][‘user’] = ‘mau’;
user[‘mau’][‘firstname’] = ‘Maurits’;
WHERE user = ‘mau’
ORDER BY date DESC
LIMIT 10
31. Operations in Cassandra 1.0
How to get a list
of blogs by “mau”?
•
•
•
dinsdag 12 november 13
post[‘uuid’][‘title’] = ‘First post!’;
post[‘uuid’][‘user’] = ‘mau’;
user[‘mau’][‘firstname’] = ‘Maurits’;
WHERE user = ‘mau’
ORDER BY date DESC
LIMIT 10
only possible when user and
date is in primary key
38. Operations in Cassandra 1.0
•
•
•
•
dinsdag 12 november 13
post[‘uuid’][‘title’] = ‘First post!’;
user[‘mau’][‘firstname’] = ‘Maurits’;
only one query required
to get user profile
with latest posts
user[‘mau’][‘post001:uuid’] = ‘First post!’;
user[‘mau’][‘post002:uuid’] = ‘Second post!’;
40. Beauty?
•
•
•
•
dinsdag 12 november 13
Dirty in the SQL world, but;
It’s a best practice in Big Data
Don’t think of it as a relational database
No strict rules on how to use it, just push it to the limits
66. Read repair
•
•
•
dinsdag 12 november 13
Compares data with 2 other replica’s in the background
Fixes inconsistent and missing data
At 10% of all reads
67. Node repair
•
•
dinsdag 12 november 13
Gradually compares all data in nodes with replica’s
Required in conjunction with read repair to fix ‘forgotten deletes’
68. ACID theorem
•
•
•
•
dinsdag 12 november 13
Atomic; completed successfully or entirely rolled back
Consistent; transations never invalidates the database state
Isolated; transactions are processed sequential
Durable; completed actions are persistent
69. CAP theorem
Impossible to achieve all three:
•
•
•
dinsdag 12 november 13
Consistency
Availability
Partition tolerance
71. Eventual consistency
•
•
Best effort
•
Configurable consistency level, but no transaction support
dinsdag 12 november 13
Consistency is not always more important than speed and scalability
(doesn’t require locking)
73. Surrogate keys
Say bye to sequences
ss cluster
istent acro
not cons
dinsdag 12 november 13
74. Surrogate keys
Say bye to sequences
ss cluster
istent acro
not cons
counters a
re for cou
n
dinsdag 12 november 13
ting
75. Native support for uuid’s
f47ac10b-58cc-4372-a567-0e02b2c3d479
Surrogate keys
Say bye to sequences
ss cluster
istent acro
not cons
counters a
re for cou
n
dinsdag 12 november 13
ting
79. Lists
•
•
user[‘mau’][‘posts’] = ‘uuid’;
•
•
UPDATE user SET posts = posts + [‘uuid’]
dinsdag 12 november 13
CREATE TABLE user (
username text PRIMARY KEY,
posts list<uuid>
);
UPDATE user SET posts = [‘uuid’] + posts
80. Set
•
CREATE TABLE user (
username text PRIMARY KEY,
email set<text>
);
•
UPDATE user SET emails = emails + {‘mail@example.com’}
dinsdag 12 november 13
81. Maps
•
CREATE TABLE user (
username text PRIMARY KEY,
attending map<timestamp,text>
);
•
•
UPDATE user SET attending[‘2013-11-12’] = ‘PHPMeetup’
dinsdag 12 november 13
DELETE attending[‘2013-12-05’] FROM user
82. Limits on collections
•
•
•
dinsdag 12 november 13
64K
Whole collection loaded in memory when reading / writing
Not an alternative to wide tables!
83. Limits on collections
•
•
•
dinsdag 12 november 13
64K
No size check in CQL
SET list = list + [‘...’]
Whole collection loaded in memory when reading / writing
Not an alternative to wide tables!
85. Wide tables in CQL3
•
•
dinsdag 12 november 13
CREATE TABLE tweets (
tweet_id uuid PRIMARY KEY,
author varchar,
body varchar
);
CREATE TABLE timeline (
user_id varchar,
tweet_id uuid,
author varchar,
body varchar,
PRIMARY KEY (user_id, tweet_id)
)
user_id
mau
user_id
mike
uuid:author
anne
uuid:author
david
uuid:body
Tweet from Anne
uuid:body
Tweet from David
86. Wide tables in CQL3
For schemaless lovers:
•
•
dinsdag 12 november 13
CREATE TABLE tweets (
tweet_id uuid PRIMARY KEY,
author varchar,
body varchar
);
CREATE TABLE timeline (
user_id varchar,
tweet_id uuid,
author varchar,
body varchar,
PRIMARY KEY (user_id, tweet_id)
)
user_id
mau
user_id
mike
CREATE TABLE name (
rowkey varchar,
columnname varchar,
value blob,
PRIMARY KEY (rowkey, columnname)
);
uuid:author uuid:body
anne
Tweet from Anne
uuid:author uuid:body
david
Tweet from David
87. Secondary index
•
•
dinsdag 12 november 13
CREATE INDEX name ON table (column);
High memory usage when used with high cardinality
91. Iteration
•
•
dinsdag 12 november 13
SELECT * FROM users
SELECT token(username), username, country, age FROM user
WHERE token(username) > 23947239 LIMIT 10
112. Pig
input_lines = LOAD '/tmp/my-copy-of-all-pages-on-internet' AS (line:chararray);
words = FOREACH input_lines GENERATE FLATTEN(TOKENIZE(line)) AS word;
filtered_words = FILTER words BY word MATCHES 'w+';
word_groups = GROUP filtered_words BY word;
word_count = FOREACH word_groups GENERATE COUNT(filtered_words) AS
count, group AS word;
ordered_word_count = ORDER word_count BY count DESC;
STORE ordered_word_count INTO '/tmp/number-of-words-on-internet';
dinsdag 12 november 13
113. Hive
SELECT v['ip'], COUNT(1) AS cnt FROM www_access
GROUP BY v['ip']
ORDER BY cnt DESC LIMIT 30
dinsdag 12 november 13
114. Pig and Hive
•
•
•
dinsdag 12 november 13
Using MapReduce
No(t very) predictable performance
Good for analysis
115. Hack your own
•
•
•
•
dinsdag 12 november 13
Not too difficult
Data can be split into subsets by filtering on tokens
Application must run on all MapRed nodes
Probably better performance than Pig / Hive
118. Thrift
•
•
•
•
•
dinsdag 12 november 13
Something like SOAP in a binary format
Tool which generates libraries based on definition files
Supports many languages (incl. PHP, JS, NodeJS, c, java, python, ruby.....)
Also used by HyperTable, HBase, Accumulo and ElasticSearch
Sole interface before 1.2
120. Binary protocol
•
•
•
dinsdag 12 november 13
Recommended protocol for Cassandra 1.2
Few client libraries available
No binary connectors were available for PHP
https://github.com/mauritsl/php-cassandra
121. php-cassandra
require('lib/cassandra/Cassandra.php');
use CassandraConnection as Cassandra;
$connection = new Cassandra('localhost', 'keyspace');
$rows = $connection->query('SELECT * FROM user');
foreach ($rows as $row) {
print $row->firstname;
print $row->listfield[0];
}
$rows->count();
$rows->getColumns();
dinsdag 12 november 13
123. Rule 1:
Don’t ask for NoSQL drivers for a CMS
dinsdag 12 november 13
124. Cassandra does not fit all
(same story for every NoSQL solution)
dinsdag 12 november 13
125. Every page (or API call) should only
require a few (if not one) query
dinsdag 12 november 13
126. Static versus Dynamic data
•
Static: information that doesn’t change very often
•
•
•
I.e.: translations
May go in a RDBMS or local storage (files?)
Dynamic: many changes
•
•
dinsdag 12 november 13
Changes must be visible on all nodes
Use Cassandra
127. Local versus Global data
•
Logging
•
•
Separate logs per node
Cache
•
•
Sometimes no need to share cache between nodes
Statistics
•
dinsdag 12 november 13
Can be kept local for a limited time
128. Local versus Global data
•
Sessions
•
dinsdag 12 november 13
Dependent on session stickiness
129. Caching
•
•
Memcache is recommended for local cache
Cassandra can be used for global cache
•
dinsdag 12 november 13
Has a TTL feature
INSERT INTO ... (...) VALUES (...) USING TTL 86400
131. What about files?
•
•
dinsdag 12 november 13
Use Hadoop Distributed File System (HDFS) or GlusterFS
Or use Cassandra
132. What about files?
•
•
Split files in chunks to avoid hotspots and save the heap
Not uncommon to have files in Cassandra
•
•
dinsdag 12 november 13
github.com/Netflix/astyanax
GB’s are ok, but do not store TB’s
133. Maximum size of cluster?
•
•
No satisfactory answer
Probably more dependent on network equipment
•
•
•
dinsdag 12 november 13
Rack awareness helps here
Facebook: 150 node cluster, 50TB data (2010)
Easou: 400 node cluster, 300TB data (300 million images)
134. Minimum size of a cluster?
•
•
•
dinsdag 12 november 13
Can run on a single node
4GB RAM recommended
Runs fine on 1GB RAM
135. Minimum size of a cluster?
•
•
•
dinsdag 12 november 13
Can run on a single node
4GB RAM recommended
Runs fine on 1GB RAM
“hot data” should fit in RAM
136. Installing Cassandra
•
Install JDK
Oracle Java recommended but OpenJDK works ok
•
•
•
•
Add Cassandra repository
dinsdag 12 november 13
apt-get install cassandra
Set listen and seed address (IP address of node and seed)
(Re)start Cassandra