Apache Cassandra
By Markus Klems
(2014)
Big Data & NoSQL
Why do we need NoSQL Databases?
● Modern Internet application requirements
○
○
○
○
○
○

low-latency CRUD operations
elasti...
What is Big Data?
Velocity

realtime
near realtime

interactive

periodic
batch

structured
data

MB GB

TB

PB

Volume

u...
Scalability & High Availability
● Workload and data volume grows ->
Partition and distribute data across multiple
servers ...
NoSQL Databases categorized by
System Architecture
Architecture

Techniques

Systems

Dynamo-style
Ring (P2P)

All nodes a...
NoSQL Databases categorized by
Data Model and Storage
Wide-Column

Each row stores a flexible
number of columns. Data is
p...
Cassandra Background:
Amazon Dynamo +
Google BigTable
Dynamo: Amazon's highly available
key-value store
● Amazon Dynamo Paper
Giuseppe DeCandia, Deniz Hastorun, Madan Jampani,
...
Amazon Dynamo: Techniques (1)
Problem

Technique

Advantage

Partitioning

Consistent Hashing

Incremental
Scalability

Hi...
Amazon Dynamo: Techniques (2)
Problem

Technique

Advantage

Recovering from

Anti-entropy using

Synchronizes

permanent ...
Dynamo: Incremental scalability
● Simple key/value API
● Consistent hashing
○ Cryptographic MD5 hash of key generates a 12...
Dynamo: Incremental scalability

Source: Amazon Dynamo paper
Dynamo: Data versioning
● Optimistic replication: updates may travel
asynchronously. This can lead to
inconsistencies.
○ H...
Dynamo: Sloppy Quroum
● (N,R,W) quorum configuration
○ N replica
○ R votes for a successful READ operation
○ W votes for a...
Dynamo: Merkle Trees
● Each server node calculates one Merkle
Tree for one owned keyrange.
● Merkle Tree = A tree of hash ...
Dynamo: Anti Entropy Protocol
● Anti-entropy protocol
○
○
○
○

Exchange Merkle Trees with replica server nodes.
Lazy loadi...
Dynamo: Membership & failure
detection
● Gossip-based protocol
○ All nodes (eventually) share the same view of the
system....
Dynamo: Summary
● CAP theorem: in a widely distributed
system, strong consistency and high
availability (+ low latency) ca...
Bigtable: A Distributed Storage System for
Structured Data
● Google Bigtable Paper
Fay Chang, Jeffrey Dean, Sanjay Ghemawa...
Google Bigtable
"A Bigtable is a sparse, distributed, persistent
multi-dimensional sorted map. The map is
indexed by a row...
Google Bigtable: Data Model

Source: Bigtable paper
Google Bigtable: Structure and
query model
● A table contains a number of rows and is
broken down into tablets which each ...
Google Bigtable: Local persistence
● Tablet servers store updates in commit logs
written in so-called SSTables
● Fresh upd...
Google Bigtable: Local persistence

Source: Bigtable paper
Cassandra Architecture and
Main Features
Cassandra: a decentralized structured
storage system
● Cassandra Paper
Avinash Lakshman and Prashant Malik. 2010.
Cassandr...
Architecture Topics
● Data Partitioning & Distribution
○ Partitioners
○ Virtual Nodes

● Data Replication
● Network Toplog...
Architecture Topics
● Data Partitioning & Distribution
○ Partitioners
○ Virtual Nodes

● Data Replication
● Network Toplog...
Data Partitioning & Distribution
922

-922

Server [Keyrange]
A

461
460

0

-1

-461
-460

B

[-460,-1]

C

Keyspace

[-9...
Partitioners
● ByteOrderedPartitioner (not recommended)
○ Plus: You can scan across lexically ordered keys
○ Minus: bad lo...
Data Partitioning & Distribution
● Virtual Nodes (Vnodes)
● Since Cassandra 1.2: Virtual Nodes for
○ better load balancing...
Virtual Nodes
Example with num_tokens: 3
882

Server [Keyrange]

-854
-798
-743

675

A

[882, -854]
[-110, --26]
[372, 46...
Architecture Topics
● Data Partitioning & Distribution
○ Partitioners
○ Virtual Nodes

● Data Replication
● Network Toplog...
Data Replication
922

-922

Server [Keyrange]
A

461
460

0

-1

-461
-460

B

[-922,-1]

C

Keyspace

[461, -461]

[-460,...
Data Replication
● Replication for high availability and data
durability
○ Replication factor N: Each row is replicated at...
Multi-DC Data Replication
922

461
460

-922

Keyspac
DC1:2
e

922

-461
-460

DC2:1

0 -1

A

B

-922

0 -1

C

D

[461,4...
Architecture Topics
● Data Partitioning & Distribution
○ Partitioners
○ Virtual Nodes

● Data Replication
● Network Toplog...
Network Topology (Snitches)
● Selected Snitches
○
○
○
○

SimpleSnitch (default)
RackInferringSnitch
PropertyFileSnitch
Gos...
SimpleSnitch
● SimpleSnitch does not recognize data center
or rack information
● Only useful for small single-DC deploymen...
RackInferringSnitch
● Assumes the network topology from the
node's IP address
DC Rack Server

10.100.200.222
PropertyFileSnitch
● Uses conf/cassandra-topology.properties file
to infer data center and rack information
● Useful if cl...
GossipingPropertyFileSnitch
● Each node sets its own data center and rack
info via conf/cassandra-rackdc.rpoperties
file.
...
Dynamic Snitching
● Dynamic snitching avoids routing requests to
badly performing nodes.
● Properties in the cassandra.yam...
Architecture Topics
● Data Partitioning & Distribution
○ Partitioners
○ Virtual Nodes

● Data Replication
● Network Toplog...
Server-to-Server Communication:
Gossip
● Cassandra uses a gossip protocol to
exchange information between servers in a
clu...
Server-to-Server Communication:
Seeds
● Seeds
○ The list of seeds addresses which in the cassandra.
yaml file is only used...
Server-to-Server Communication
● Delete gossip state on a server
○ You can delete the gossip state of server by adding
the...
Server-to-Server Communication:
Failure Detection
● Cassandra implements a Gossip-based
accrual failure detector that adap...
Heartbeat Failure Detection
● Heartbeat failure detector

Naohiro Hayashibara, Xavier Defago, Rami Yared, and Takuya Katay...
Accrual Failure Detection
● Accrual failure detector

Naohiro Hayashibara, Xavier Defago, Rami Yared, and Takuya Katayama....
Architecture Topics
● Data Partitioning & Distribution
○ Partitioners
○ Virtual Nodes

● Data Replication
● Network Toplog...
Scaling a Cluster
● Bootstrapping managed via command line
admin tool
● A node starts for the first time

●

○ Chose posit...
Add Server
922

-922

A
461
460

0

-1

-461
-460

B

[-460,-1]

C

Keyspace

[-922, -461]

[0, 460]

D

[461, 922]
Add a Server
Z

-922
-712
-711

461
460

Keyspace

0

-1

-461
-460

[-922, -712]

A

[-711, -461]

B

[-460,-1]

C

922

...
Remove a Server
Z

-922
-712
-711

461
460

Keyspace

0

-1

-461
-460

[-922, -712]

A

[-711, -461]

B

[-460,460]

C

9...
Architecture Topics
● Data Partitioning & Distribution
○ Partitioners
○ Virtual Nodes

● Data Replication
● Network Toplog...
Client-Server Communication
-922

A

pace

-461 [-922, -461]
-460

B
[-460,-1]
-1

Request record with
primary key -333

C...
Client-Server Communication
-922

A

pace

-461 [-922, -461]
-460

B
[-460,-1]
-1

Request record with
primary key -333

R...
Client-Server Communication
-922

A

pace

-461 [-922, -461]
-460

B
[-460,-1]
-1

return record

return
record

Cassandra...
Architecture Topics
● Data Partitioning & Distribution
○ Partitioners
○ Virtual Nodes

● Data Replication
● Network Toplog...
Local Persistence - Write
● Write
1. Append to commit log for durability (recoverability)
2. Update of in-memory, per-colu...
Local Persistence - Write (2)
Local Persistence - Write
Example (1)
Local Persistence - Write
Example (2)
Local Persistence - Write
Example (3)
Local Persistence - CommitLog
cassandra.yaml
# commitlog_sync may be either "periodic" or "batch."
# When in batch mode, C...
Local Persistence - Read
● Read
○ Query in-memory Memtable
○ Check in-memory bloom filter
■ Used to prevent unnecessary di...
Local Persistence - Read (2)
Local Persistence - Read (3)
Local Persistence - Read (4)
Compactions in Cassandra
Compactions in Cassandra (2)
Size-Tiered Compaction Strategy
Leveled Compaction Strategy
Cassandra: Summary
● Scalability: Peer-to-Peer architecture
● High availability:
○ replication
○ quorum-based replica cont...
Upcoming SlideShare
Loading in...5
×

Cassandra background-and-architecture

2,036

Published on

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,036
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
47
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Cassandra background-and-architecture

  1. 1. Apache Cassandra By Markus Klems (2014)
  2. 2. Big Data & NoSQL
  3. 3. Why do we need NoSQL Databases? ● Modern Internet application requirements ○ ○ ○ ○ ○ ○ low-latency CRUD operations elastic scalability high availability reliable and durable storage geographic distribution flexible schema ● Less prioritized ○ Transactions, ACID guarantees ■ ... but some form of data consistency is still desirable ○ SQL support ■ … but some SQL features are still desirable
  4. 4. What is Big Data? Velocity realtime near realtime interactive periodic batch structured data MB GB TB PB Volume unstructured data Variety c.f http://www.datasciencecentral.com /forum/topics/the-3vs-that-define-big-data
  5. 5. Scalability & High Availability ● Workload and data volume grows -> Partition and distribute data across multiple servers (horizontal scaling) ○ How to determine partition boundaries? ○ Can partitions be changed dynamically? ○ How to route requests to the right server? ● High Availability -> replicate data ○ ○ ○ ○ What kind of failures can we deal with? Sync or async replication? Local or geo replication? Consistency model?
  6. 6. NoSQL Databases categorized by System Architecture Architecture Techniques Systems Dynamo-style Ring (P2P) All nodes are equal. Each node stores a data partition + replicated data. Eventually consistent. Cassandra, Riak, Voldemort, Amazon DynamoDB Master-Slave Data partitioned across slaves. Each slave stores a data partition + replicated data. Strong consistency guarantees. HBase, MongoDB, Redis, Yahoo! Sherpa, Neo4j Full replication Bi-directional, incremental replication between all nodes. Each node stores all data. Eventually consistent. CouchDB
  7. 7. NoSQL Databases categorized by Data Model and Storage Wide-Column Each row stores a flexible number of columns. Data is partitioned by row key. Cassandra, HBase, Amazon DynamoDB Document Storage and retrieval of structured data in the form of JSON, YAML, or RDF documents. CouchDB, MongoDB Key-value Row-oriented data storage of simple (key,value) pairs in a flat namespace. Riak, Redis, Voldemort, Yahoo! Sherpa Graph Storage and retrieval of data Neo4j that is stored as nodes and links of graphs in a graph-space.
  8. 8. Cassandra Background: Amazon Dynamo + Google BigTable
  9. 9. Dynamo: Amazon's highly available key-value store ● Amazon Dynamo Paper Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall, and Werner Vogels. 2007. Dynamo: Amazon's highly available key-value store. In Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles (SOSP '07). ACM, New York, NY, USA, 205220. ● Download: http://www.read.seas.harvard. edu/~kohler/class/cs239-w08/decandia07dynamo.pdf
  10. 10. Amazon Dynamo: Techniques (1) Problem Technique Advantage Partitioning Consistent Hashing Incremental Scalability High Availability Vector clocks with Version size is for writes reconciliation during decoupled from reads update rates. Handling temporary Sloppy Quorum and Provides high failures hinted handoff availability and durability guarantee when some of the replicas are not available.
  11. 11. Amazon Dynamo: Techniques (2) Problem Technique Advantage Recovering from Anti-entropy using Synchronizes permanent failures Merkle trees divergent replicas in the background. Membership and Gossip-based Preserves symmetry failure detection membership protocol and avoids having a and failure detection. centralized registry for storing membership and node liveness information.
  12. 12. Dynamo: Incremental scalability ● Simple key/value API ● Consistent hashing ○ Cryptographic MD5 hash of key generates a 128-bit identifier ○ The largest value wraps around the smallest one. The keyspace is a "ring".
  13. 13. Dynamo: Incremental scalability Source: Amazon Dynamo paper
  14. 14. Dynamo: Data versioning ● Optimistic replication: updates may travel asynchronously. This can lead to inconsistencies. ○ How to identify inconsistencies? ○ Data versions, vector clocks!
  15. 15. Dynamo: Sloppy Quroum ● (N,R,W) quorum configuration ○ N replica ○ R votes for a successful READ operation ○ W votes for a successful WRITE operation ● Quorum intersection invariants ○ N < R+W ○ (W > N/2) ● Sloppy quorum ○ If a node goes down, save the data temporarily as "Hinted Handoffs" on another node. ○ Thus avoiding unavailability. ○ Hinted Handoffs must be resolved after some time.
  16. 16. Dynamo: Merkle Trees ● Each server node calculates one Merkle Tree for one owned keyrange. ● Merkle Tree = A tree of hash values of parts of the keyrange. Hash(Keyrange) Hash (Part1) Hash (Part1-1) Hash (Part1-2) Hash (Part2) Hash (Part2-1) Hash (Part2-2)
  17. 17. Dynamo: Anti Entropy Protocol ● Anti-entropy protocol ○ ○ ○ ○ Exchange Merkle Trees with replica server nodes. Lazy loading of Merkle Trees Compare the tree and track down inconsistencies. Trigger conflict resolution (last write wins) Send Hash (Keyrange) Server A Comparison failed Send Hash(Part1) ... Server B
  18. 18. Dynamo: Membership & failure detection ● Gossip-based protocol ○ All nodes (eventually) share the same view of the system. ○ Information is exchanged via gossip between peers E B G C A D F H
  19. 19. Dynamo: Summary ● CAP theorem: in a widely distributed system, strong consistency and high availability (+ low latency) cannot be achieved at the same time. ● Optimistic replication: improve availability + latency at the cost of inconsistencies ● Requires conflict resolution: ○ when to resolve conflicts? ○ who resolves conflicts?
  20. 20. Bigtable: A Distributed Storage System for Structured Data ● Google Bigtable Paper Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Mike Burrows, Tushar Chandra, Andrew Fikes, and Robert E. Gruber. 2008. Bigtable: A Distributed Storage System for Structured Data. ACM Trans. Comput. Syst. 26, 2, Article 4 (June 2008), 26 pages. ● http://research.google.com/archive/bigtable.html
  21. 21. Google Bigtable "A Bigtable is a sparse, distributed, persistent multi-dimensional sorted map. The map is indexed by a row key, column key, and a timestamp; each value in the map is an uninterpreted array of bytes." Source: Bigtable paper
  22. 22. Google Bigtable: Data Model Source: Bigtable paper
  23. 23. Google Bigtable: Structure and query model ● A table contains a number of rows and is broken down into tablets which each contain a subset of the rows ● Tablets are the unit of distribution and load balancing ● Rows are sorted lexicographically by row key ○ Subsequent row keys are within a tablet ○ Allows efficient range queries for small numbers of rows ● Operations: Read, write and delete items + batch writes ● Supports single-row transactions
  24. 24. Google Bigtable: Local persistence ● Tablet servers store updates in commit logs written in so-called SSTables ● Fresh updates are kept in memory (memtable), old updates are stored in GFS ● Minor compactions flush memtable into new SSTable ● Major compaction merge SSTables into just one new SSTable
  25. 25. Google Bigtable: Local persistence Source: Bigtable paper
  26. 26. Cassandra Architecture and Main Features
  27. 27. Cassandra: a decentralized structured storage system ● Cassandra Paper Avinash Lakshman and Prashant Malik. 2010. Cassandra: a decentralized structured storage system. SIGOPS Oper. Syst. Rev. 44, 2 (April 2010), 35-40. ● URL: http://www.cs.cornell. edu/Projects/ladis2009/papers/Lakshman-ladis2009. PDF
  28. 28. Architecture Topics ● Data Partitioning & Distribution ○ Partitioners ○ Virtual Nodes ● Data Replication ● Network Toplogy (Snitches) ● Server-to-Server Communication (Gossip) ○ Membership ○ Failure detection ● Scaling a Cluster ● Client-Server Communication ● Local Persistence
  29. 29. Architecture Topics ● Data Partitioning & Distribution ○ Partitioners ○ Virtual Nodes ● Data Replication ● Network Toplogy (Snitches) ● Server-to-Server Communication (Gossip) ○ Membership ○ Failure detection ● Scaling a Cluster ● Client-Server Communication ● Local Persistence
  30. 30. Data Partitioning & Distribution 922 -922 Server [Keyrange] A 461 460 0 -1 -461 -460 B [-460,-1] C Keyspace [-922, -461] [0, 460] D [461, 922]
  31. 31. Partitioners ● ByteOrderedPartitioner (not recommended) ○ Plus: You can scan across lexically ordered keys ○ Minus: bad load balancing, hotspots, etc. ● RandomPartitioner (default before 1.2) ○ The RandomPartition distributes data evenly across the nodes using an MD5 hash value of the row key. The possible range of hash values is from 0 to 2127 -1. ● Murmur3Partitioner (default since 1.2) ○ The Murmur3Partitioner uses the MurmurHash function. This hashing function creates a 64-bit hash value of the row key. The possible range of hash values is from -263 to +263. Source: http://www.datastax.com/docs/1.2/cluster_architecture/partitioners
  32. 32. Data Partitioning & Distribution ● Virtual Nodes (Vnodes) ● Since Cassandra 1.2: Virtual Nodes for ○ better load balancing ○ easier scaling with differently sized servers
  33. 33. Virtual Nodes Example with num_tokens: 3 882 Server [Keyrange] -854 -798 -743 675 A [882, -854] [-110, --26] [372, 469] B [675, 882] [227, 298] [298, 372] C [-798, -743] [-364, -110] [469, -675] D [-854, -798] [-743, -364] [-26, 277] 469 Keyspace 372 298 277 -364 -110 -26
  34. 34. Architecture Topics ● Data Partitioning & Distribution ○ Partitioners ○ Virtual Nodes ● Data Replication ● Network Toplogy (Snitches) ● Server-to-Server Communication (Gossip) ○ Membership ○ Failure detection ● Scaling a Cluster ● Client-Server Communication ● Local Persistence
  35. 35. Data Replication 922 -922 Server [Keyrange] A 461 460 0 -1 -461 -460 B [-922,-1] C Keyspace [461, -461] [-460, 460] D [0, 922]
  36. 36. Data Replication ● Replication for high availability and data durability ○ Replication factor N: Each row is replicated at N nodes. ○ Each row key k is assigned to a coordinator node. ○ The coordinator is responsible for replicating the rows within its key range.
  37. 37. Multi-DC Data Replication 922 461 460 -922 Keyspac DC1:2 e 922 -461 -460 DC2:1 0 -1 A B -922 0 -1 C D [461,461] [-922,-1][-460,460] [0,922] E F [0,922] [-922,-1]
  38. 38. Architecture Topics ● Data Partitioning & Distribution ○ Partitioners ○ Virtual Nodes ● Data Replication ● Network Toplogy (Snitches) ● Server-to-Server Communication (Gossip) ○ Membership ○ Failure detection ● Scaling a Cluster ● Client-Server Communication ● Local Persistence
  39. 39. Network Topology (Snitches) ● Selected Snitches ○ ○ ○ ○ SimpleSnitch (default) RackInferringSnitch PropertyFileSnitch GossipingPropertyFileSnitch ● Set the endpoint_snitch property in cassandra.yaml
  40. 40. SimpleSnitch ● SimpleSnitch does not recognize data center or rack information ● Only useful for small single-DC deployments
  41. 41. RackInferringSnitch ● Assumes the network topology from the node's IP address DC Rack Server 10.100.200.222
  42. 42. PropertyFileSnitch ● Uses conf/cassandra-topology.properties file to infer data center and rack information ● Useful if cluster layout is not matched by IP addresses or if you have complex grouping requirements ● Example properties file: # Data Center One 175.56.12.105=DC1:RAC1 120.53.24.101=DC1:RAC2 # Data Center Two 110.56.12.120=DC2:RAC1 50.17.10.203=DC2:RAC2
  43. 43. GossipingPropertyFileSnitch ● Each node sets its own data center and rack info via conf/cassandra-rackdc.rpoperties file. ● The info is propagated to other nodes via gossip. Fits nicely the P2P style of Cassandra. ● Example properties file: dc=DC1 rack=RAC1
  44. 44. Dynamic Snitching ● Dynamic snitching avoids routing requests to badly performing nodes. ● Properties in the cassandra.yaml dynamic_snitch_update_interval_in_ms: 100 dynamic_snitch_reset_interval_in_ms: 600000 dynamic_snitch_badness_threshold: 0.1
  45. 45. Architecture Topics ● Data Partitioning & Distribution ○ Partitioners ○ Virtual Nodes ● Data Replication ● Network Toplogy (Snitches) ● Server-to-Server Communication (Gossip) ○ Membership ○ Failure detection ● Scaling a Cluster ● Client-Server Communication ● Local Persistence
  46. 46. Server-to-Server Communication: Gossip ● Cassandra uses a gossip protocol to exchange information between servers in a cluster in a peer-to-peer fashion ○ The gossip process runs every second on each Cassandra server ○ Each server sends its state in a message to other servers in the cluster ○ Each gossip message has a version. Old gossip state information on a server is overwritten.
  47. 47. Server-to-Server Communication: Seeds ● Seeds ○ The list of seeds addresses which in the cassandra. yaml file is only used during initial bootstrapping of a new server in the cluster. ○ The bootstrapping server establishes gossip communication with the servers in the seeds list. ● You should use the same seeds list on all servers to prevent split-brain partitions in gossip communication. ● In a Multi-DC setup, the seeds list must include at least one server from each DC.
  48. 48. Server-to-Server Communication ● Delete gossip state on a server ○ You can delete the gossip state of server by adding the following in your cassandra-env.sh file: -Dcassandra.load_ring_state=false ● This is necessary in certain situations when you restart one or more servers, such as ○ You restart a server after its IP address has been changed ○ You restart all servers with a new cluster_name
  49. 49. Server-to-Server Communication: Failure Detection ● Cassandra implements a Gossip-based accrual failure detector that adapts the time interval based on historic latency data. ○ Every Cassandra node maintains a sliding window of inter-arrival times of Gossip messages. ○ The Cassandra failure detector assumes an exponential distribution of inter-arrival times. ○ The failure detector can be configured with the cassandra.yaml ○ parameter phi_convict_threshold Tip: You can make the failure detector less sensitive to latency variability, for example during times of network congestion or in Multi-DC setups, by increasing the phi_convict_threshold value.
  50. 50. Heartbeat Failure Detection ● Heartbeat failure detector Naohiro Hayashibara, Xavier Defago, Rami Yared, and Takuya Katayama. 2004. The Phi Accrual Failure Detector. In Proceedings of the 23rd IEEE International Symposium on Reliable Distributed Systems (SRDS '04). IEEE Computer Society, Washington, DC, USA, 66-78.
  51. 51. Accrual Failure Detection ● Accrual failure detector Naohiro Hayashibara, Xavier Defago, Rami Yared, and Takuya Katayama. 2004. The Phi Accrual Failure Detector. In Proceedings of the 23rd IEEE International Symposium on Reliable Distributed Systems (SRDS '04). IEEE Computer Society, Washington, DC, USA, 66-78.
  52. 52. Architecture Topics ● Data Partitioning & Distribution ○ Partitioners ○ Virtual Nodes ● Data Replication ● Network Toplogy (Snitches) ● Server-to-Server Communication (Gossip) ○ Membership ○ Failure detection ● Scaling a Cluster ● Client-Server Communication ● Local Persistence
  53. 53. Scaling a Cluster ● Bootstrapping managed via command line admin tool ● A node starts for the first time ● ○ Chose position on the ring (via token) ○ Join cluster: connect with seed node(s) and start data streaming from neighboring node. ○ Bootstrapping/joining phase is completed when all data for the newly split keyrange has been streamed to the new node. Basically the same process for a cluster with Vnodes, however, a server with Vnodes has multiple positions in the ring and streams from multiple neighboring servers.
  54. 54. Add Server 922 -922 A 461 460 0 -1 -461 -460 B [-460,-1] C Keyspace [-922, -461] [0, 460] D [461, 922]
  55. 55. Add a Server Z -922 -712 -711 461 460 Keyspace 0 -1 -461 -460 [-922, -712] A [-711, -461] B [-460,-1] C 922 [0, 460] D [461, 922]
  56. 56. Remove a Server Z -922 -712 -711 461 460 Keyspace 0 -1 -461 -460 [-922, -712] A [-711, -461] B [-460,460] C 922 [0, 460] D [461, 922]
  57. 57. Architecture Topics ● Data Partitioning & Distribution ○ Partitioners ○ Virtual Nodes ● Data Replication ● Network Toplogy (Snitches) ● Server-to-Server Communication (Gossip) ○ Membership ○ Failure detection ● Scaling a Cluster ● Client-Server Communication ● Local Persistence
  58. 58. Client-Server Communication -922 A pace -461 [-922, -461] -460 B [-460,-1] -1 Request record with primary key -333 Cassandra client
  59. 59. Client-Server Communication -922 A pace -461 [-922, -461] -460 B [-460,-1] -1 Request record with primary key -333 Request record from B Cassandra client
  60. 60. Client-Server Communication -922 A pace -461 [-922, -461] -460 B [-460,-1] -1 return record return record Cassandra client
  61. 61. Architecture Topics ● Data Partitioning & Distribution ○ Partitioners ○ Virtual Nodes ● Data Replication ● Network Toplogy (Snitches) ● Server-to-Server Communication (Gossip) ○ Membership ○ Failure detection ● Scaling a Cluster ● Client-Server Communication ● Local Persistence
  62. 62. Local Persistence - Write ● Write 1. Append to commit log for durability (recoverability) 2. Update of in-memory, per-column-family Memtable ● If Memtable crosses a threshold 1. Sequential write to disk (SSTable). 2. Merge SSTables from time to time (compactions)
  63. 63. Local Persistence - Write (2)
  64. 64. Local Persistence - Write Example (1)
  65. 65. Local Persistence - Write Example (2)
  66. 66. Local Persistence - Write Example (3)
  67. 67. Local Persistence - CommitLog cassandra.yaml # commitlog_sync may be either "periodic" or "batch." # When in batch mode, Cassandra won't ack writes until the # commit log has been fsynced to disk. It will wait up to # commitlog_sync_batch_window_in_ms milliseconds for other # writes, before performing the sync. # commitlog_sync: batch # commitlog_sync_batch_window_in_ms: 50 # the other option is "periodic" where writes may be acked # immediately and the CommitLog is simply synced every # commitlog_sync_period_in_ms milliseconds. commitlog_sync: periodic commitlog_sync_period_in_ms: 10000
  68. 68. Local Persistence - Read ● Read ○ Query in-memory Memtable ○ Check in-memory bloom filter ■ Used to prevent unnecessary disk access. ■ A bloom filter summarizes the keys in a file. ■ False Positives are possible ○ Check column index to jump to the columns on disk as fast as possible. ■ Index for every 256K chunk.
  69. 69. Local Persistence - Read (2)
  70. 70. Local Persistence - Read (3)
  71. 71. Local Persistence - Read (4)
  72. 72. Compactions in Cassandra
  73. 73. Compactions in Cassandra (2)
  74. 74. Size-Tiered Compaction Strategy
  75. 75. Leveled Compaction Strategy
  76. 76. Cassandra: Summary ● Scalability: Peer-to-Peer architecture ● High availability: ○ replication ○ quorum-based replica control ○ failure detection and recovery ● High performance (particularly for WRITE operations): Bigtable-style storage engine ○ All writes to disk are sequential. Files are written once (immutable) and not updated hereafter
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×