Cassandra uses a layered architecture with an API layer, Dynamo layer, and database layer. The API layer supports Thrift, CQL, and JMX interfaces. The Dynamo layer handles ring operations and messaging. The database layer includes the memtable, SSTables, and components for data storage and access like Table and ColumnFamilyStore. Messages are processed asynchronously across stages like READ and MUTATION using handlers and callbacks.

1. CASSANDRA SF 2013
CASSANDRA
INTERNALS
Aaron Morton
@aaronmorton
www.thelastpickle.com
#Cassandra13
Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License

2. About Me
Freelance Cassandra Consultant
Based in Wellington, New Zealand
Apache Cassandra Committer
#Cassandra13

3. Architecture
Code
#Cassandra13

4. Cassandra Architecture
API's
Cluster Aware
Cluster Unaware
Clients
Disk
#Cassandra13

5. Cassandra Cluster Architecture
API's
Cluster Aware
Cluster Unaware
Clients
Disk
API's
Cluster Aware
Cluster Unaware
Disk
Node 1 Node 2
#Cassandra13

6. Dynamo Cluster Architecture
API's
Dynamo
Database
Clients
Disk
API's
Dynamo
Database
Disk
Node 1 Node 2
#Cassandra13

7. Architecture
API
Dynamo
Database
#Cassandra13

8. APITransports
Thrift
Native Binary
#Cassandra13

9. ThriftTransport
//Custom TServer implementations
o.a.c.thrift.CustomTThreadPoolServer
o.a.c.thrift.CustomTNonBlockingServer
o.a.c.thrift.CustomTHsHaServer
#Cassandra13

10. APITransports
Thrift
Native Binary
#Cassandra13

11. Native BinaryTransport
Beta in Cassandra 1.2
Uses Netty
Enabled with
start_native_transport
(Disabled by default)
#Cassandra13

12. o.a.c.transport.Server.run()
//Setup the Netty server
new ExecutionHandler()
new NioServerSocketChannelFactory()
ServerBootstrap.setPipelineFactory()
#Cassandra13

13. o.a.c.transport.Message.Dispatcher.messageReceived()
//Process message from client
ServerConnection.validateNewMessage()
Request.execute()
ServerConnection.applyStateTransition()
Channel.write()
#Cassandra13

14. Messages
Defined in the Native Binary
Protocol
$SRC/doc/native_protocol.spec
#Cassandra13

15. API Services
JMX
Thrift
CQL 3
#Cassandra13

16. JMX Management Beans
Spread around the code base.
Interfaces named *MBean
#Cassandra13

17. JMX Management Beans
Registered with names such as
org.apache.cassandra.db:
type=StorageProxy
#Cassandra13

18. API Services
JMX
Thrift
CQL 3
#Cassandra13

19. o.a.c.thrift.CassandraServer
// Implements Thrift Interface
// Access control
// Input validation
// Mapping to/from Thrift and internal types
#Cassandra13

20. Thrift Interface
Thrift IDL
$SRC/interface/cassandra.thrift
#Cassandra13

21. o.a.c.thrift.CassandraServer.get_slice()
// get columns for one row
Tracing.begin()
ClientState cState = state()
cState.hasColumnFamilyAccess()
multigetSliceInternal()
#Cassandra13

22. CassandraServer.multigetSliceInternal()
// get columns for may rows
ThriftValidation.validate*()
// Create ReadCommands
getSlice()
#Cassandra13

23. CassandraServer.getSlice()
// Process ReadCommands
// return Thrift types
readColumnFamily()
thriftifyColumnFamily()
#Cassandra13

24. CassandraServer.readColumnFamily()
// Process ReadCommands
// Return ColumnFamilies
StorageProxy.read()
#Cassandra13

25. API Services
JMX
Thrift
CQL 3
#Cassandra13

26. o.a.c.cql3.QueryProcessor
// Prepares and executes CQL3 statements
// Used by Thrift & Native transports
// Access control
// Input validation
// Returns transport.ResultMessage
#Cassandra13

27. CQL3 Grammar
ANTLR Grammar
$SRC/o.a.c.cql3/Cql.g
#Cassandra13

28. o.a.c.cql3.statements.ParsedStatement
// Subclasses generated by ANTLR
// Tracks bound term count
// Prepare CQLStatement
prepare()
#Cassandra13

29. o.a.c.cql3.statements.CQLStatement
checkAccess(ClientState state)
validate(ClientState state)
execute(ConsistencyLevel cl,
QueryState state,
List<ByteBuffer> variables)
#Cassandra13

30. statements.SelectStatement.RawStatement
// Implements ParsedStatement
// Input validation
prepare()
#Cassandra13

31. statements.SelectStatement.execute()
// Create ReadCommands
StorageProxy.read()
#Cassandra13

32. Architecture
API
Dynamo
Database
#Cassandra13

33. Dynamo Layer
o.a.c.service
o.a.c.net
o.a.c.dht
o.a.c.gms
o.a.c.locator
o.a.c.stream
#Cassandra13

34. o.a.c.service.StorageProxy
// Cluster wide storage operations
// Select endpoints & check CL available
// Send messages to Stages
// Wait for response
// Store Hints
#Cassandra13

35. o.a.c.service.StorageService
// Ring operations
// Track ring state
// Start & stop ring membership
// Node & token queries
#Cassandra13

36. o.a.c.service.IResponseResolver
preprocess(MessageIn<T> message)
resolve() throws
DigestMismatchException
RowDigestResolver
RowDataResolver
RangeSliceResponseResolver
#Cassandra13

37. Response Handlers / Callback
implements IAsyncCallback<T>
response(MessageIn<T> msg)
#Cassandra13

38. o.a.c.service.ReadCallback.get()
//Wait for blockfor & data
condition.await(timeout,
TimeUnit.MILLISECONDS)
throw ReadTimeoutException()
resolver.resolve()
#Cassandra13

39. o.a.c.service.StorageProxy.fetchRows()
getLiveSortedEndpoints()
new RowDigestResolver()
new ReadCallback()
MessagingService.sendRR()
---------------------------------------
ReadCallback.get() # blocking
catch (DigestMismatchException ex)
catch (ReadTimeoutException ex)
#Cassandra13

40. Dynamo Layer
o.a.c.service
o.a.c.net
o.a.c.dht
o.a.c.gms
o.a.c.locator
o.a.c.stream
#Cassandra13

41. o.a.c.net.MessagingService.verb<<enum>>
MUTATION
READ
REQUEST_RESPONSE
TREE_REQUEST
TREE_RESPONSE
(And more...)
#Cassandra13

42. o.a.c.net.MessagingService.verbHandlers
new EnumMap<Verb,
IVerbHandler>(Verb.class)
#Cassandra13

43. o.a.c.net.IVerbHandler<T>
doVerb(MessageIn<T> message,
String id);
#Cassandra13

44. o.a.c.net.MessagingService.verbStages
new EnumMap<MessagingService.Verb,
Stage>(MessagingService.Verb.class)
#Cassandra13

45. o.a.c.net.MessagingService.receive()
runnable = new MessageDeliveryTask(
message, id, timestamp);
StageManager.getStage(
message.getMessageType());
stage.execute(runnable);
#Cassandra13

46. o.a.c.net.MessageDeliveryTask.run()
// If dropable and rpc_timeout
MessagingService.incrementDroppedMessag
es(verb);
MessagingService.getVerbHandler(verb)
verbHandler.doVerb(message, id)
#Cassandra13

47. Architecture
API Layer
Dynamo Layer
Database Layer
#Cassandra13

48. Database Layer
o.a.c.concurrent
o.a.c.db
o.a.c.cache
o.a.c.io
o.a.c.trace
#Cassandra13

49. o.a.c.concurrent.StageManager
stages = new EnumMap<Stage,
ThreadPoolExecutor>(Stage.class);
getStage(Stage stage)
#Cassandra13

50. o.a.c.concurrent.Stage
READ
MUTATION
GOSSIP
REQUEST_RESPONSE
ANTI_ENTROPY
(And more...)
#Cassandra13

51. Database Layer
o.a.c.concurrent
o.a.c.db
o.a.c.cache
o.a.c.io
o.a.c.trace
#Cassandra13

52. o.a.c.db.Table
// Keyspace
open(String table)
getColumnFamilyStore(String cfName)
getRow(QueryFilter filter)
apply(RowMutation mutation,
boolean writeCommitLog)
#Cassandra13

53. o.a.c.db.ColumnFamilyStore
// Column Family
getColumnFamily(QueryFilter filter)
getTopLevelColumns(...)
apply(DecoratedKey key,
ColumnFamily columnFamily,
SecondaryIndexManager.Updater
indexer)
#Cassandra13

54. o.a.c.db.IColumnContainer
addColumn(IColumn column)
remove(ByteBuffer columnName)
ColumnFamily
SuperColumn
#Cassandra13

55. o.a.c.db.ISortedColumns
addColumn(IColumn column,
Allocator allocator)
removeColumn(ByteBuffer name)
ArrayBackedSortedColumns
AtomicSortedColumns
TreeMapBackedSortedColumns
#Cassandra13

56. o.a.c.db.Memtable
put(DecoratedKey key,
ColumnFamily columnFamily,
SecondaryIndexManager.Updater
indexer)
flushAndSignal(CountDownLatch latch,
Future<ReplayPosition>
context)
#Cassandra13

57. o.a.c.db.ReadCommand
getRow(Table table)
SliceByNamesReadCommand
SliceFromReadCommand
#Cassandra13

58. o.a.c.db.IDiskAtomFilter
getMemtableColumnIterator(...)
getSSTableColumnIterator(...)
IdentityQueryFilter
NamesQueryFilter
SliceQueryFilter
#Cassandra13

59. Summary
CustomTThreadPoolServer Message.Dispatcher
CassandraServer QueryProcessor
ReadCommand
StorageProxy
IResponseResolver
IAsyncCallback
MessagingService
IVerbHandler
Table ColumnFamilyStore IDiskAtomFilter
API
Dynamo
Database
#Cassandra13

60. Thanks.
#Cassandra13

61. Aaron Morton
@aaronmorton
www.thelastpickle.com
Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License