Cassandra SF 2013 Conference - Cassandra Internals

1. CASSANDRA SF 2013
CASSANDRA
INTERNALS
Aaron Morton
@aaronmorton
www.thelastpickle.com
#Cassandra13
Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License

2. About Me
Freelance Cassandra Consultant
Based in Wellington, New Zealand
Apache Cassandra Committer
#Cassandra13

3. Architecture
Code
#Cassandra13

4. Cassandra Architecture
API's
Cluster Aware
Cluster Unaware
Clients
Disk
#Cassandra13

5. Cassandra Cluster Architecture
API's
Cluster Aware
Cluster Unaware
Clients
Disk
API's
Cluster Aware
Cluster Unaware
Disk
Node 1 Node 2
#Cassandra13

6. Dynamo Cluster Architecture
API's
Dynamo
Database
Clients
Disk
API's
Dynamo
Database
Disk
Node 1 Node 2
#Cassandra13

7. Architecture
API
Dynamo
Database
#Cassandra13

8. APITransports
Thrift
Native Binary
#Cassandra13

9. ThriftTransport
//Custom TServer implementations
o.a.c.thrift.CustomTThreadPoolServer
o.a.c.thrift.CustomTNonBlockingServer
o.a.c.thrift.CustomTHsHaServer
#Cassandra13

10. APITransports
Thrift
Native Binary
#Cassandra13

11. Native BinaryTransport
Beta in Cassandra 1.2
Uses Netty
Enabled with
start_native_transport
(Disabled by default)
#Cassandra13

12. o.a.c.transport.Server.run()
//Setup the Netty server
new ExecutionHandler()
new NioServerSocketChannelFactory()
ServerBootstrap.setPipelineFactory()
#Cassandra13

13. o.a.c.transport.Message.Dispatcher.messageReceived()
//Process message from client
ServerConnection.validateNewMessage()
Request.execute()
ServerConnection.applyStateTransition()
Channel.write()
#Cassandra13

14. Messages
Defined in the Native Binary
Protocol
$SRC/doc/native_protocol.spec
#Cassandra13

15. API Services
JMX
Thrift
CQL 3
#Cassandra13

16. JMX Management Beans
Spread around the code base.
Interfaces named *MBean
#Cassandra13

17. JMX Management Beans
Registered with names such as
org.apache.cassandra.db:
type=StorageProxy
#Cassandra13

18. API Services
JMX
Thrift
CQL 3
#Cassandra13

19. o.a.c.thrift.CassandraServer
// Implements Thrift Interface
// Access control
// Input validation
// Mapping to/from Thrift and internal types
#Cassandra13

20. Thrift Interface
Thrift IDL
$SRC/interface/cassandra.thrift
#Cassandra13

21. o.a.c.thrift.CassandraServer.get_slice()
// get columns for one row
Tracing.begin()
ClientState cState = state()
cState.hasColumnFamilyAccess()
multigetSliceInternal()
#Cassandra13

22. CassandraServer.multigetSliceInternal()
// get columns for may rows
ThriftValidation.validate*()
// Create ReadCommands
getSlice()
#Cassandra13

23. CassandraServer.getSlice()
// Process ReadCommands
// return Thrift types
readColumnFamily()
thriftifyColumnFamily()
#Cassandra13

24. CassandraServer.readColumnFamily()
// Process ReadCommands
// Return ColumnFamilies
StorageProxy.read()
#Cassandra13

25. API Services
JMX
Thrift
CQL 3
#Cassandra13

26. o.a.c.cql3.QueryProcessor
// Prepares and executes CQL3 statements
// Used by Thrift & Native transports
// Access control
// Input validation
// Returns transport.ResultMessage
#Cassandra13

27. CQL3 Grammar
ANTLR Grammar
$SRC/o.a.c.cql3/Cql.g
#Cassandra13

28. o.a.c.cql3.statements.ParsedStatement
// Subclasses generated by ANTLR
// Tracks bound term count
// Prepare CQLStatement
prepare()
#Cassandra13

29. o.a.c.cql3.statements.CQLStatement
checkAccess(ClientState state)
validate(ClientState state)
execute(ConsistencyLevel cl,
QueryState state,
List<ByteBuffer> variables)
#Cassandra13

30. statements.SelectStatement.RawStatement
// Implements ParsedStatement
// Input validation
prepare()
#Cassandra13

31. statements.SelectStatement.execute()
// Create ReadCommands
StorageProxy.read()
#Cassandra13

32. Architecture
API
Dynamo
Database
#Cassandra13

33. Dynamo Layer
o.a.c.service
o.a.c.net
o.a.c.dht
o.a.c.gms
o.a.c.locator
o.a.c.stream
#Cassandra13

34. o.a.c.service.StorageProxy
// Cluster wide storage operations
// Select endpoints & check CL available
// Send messages to Stages
// Wait for response
// Store Hints
#Cassandra13

35. o.a.c.service.StorageService
// Ring operations
// Track ring state
// Start & stop ring membership
// Node & token queries
#Cassandra13

36. o.a.c.service.IResponseResolver
preprocess(MessageIn<T> message)
resolve() throws
DigestMismatchException
RowDigestResolver
RowDataResolver
RangeSliceResponseResolver
#Cassandra13

37. Response Handlers / Callback
implements IAsyncCallback<T>
response(MessageIn<T> msg)
#Cassandra13

38. o.a.c.service.ReadCallback.get()
//Wait for blockfor & data
condition.await(timeout,
TimeUnit.MILLISECONDS)
throw ReadTimeoutException()
resolver.resolve()
#Cassandra13

39. o.a.c.service.StorageProxy.fetchRows()
getLiveSortedEndpoints()
new RowDigestResolver()
new ReadCallback()
MessagingService.sendRR()
---------------------------------------
ReadCallback.get() # blocking
catch (DigestMismatchException ex)
catch (ReadTimeoutException ex)
#Cassandra13

40. Dynamo Layer
o.a.c.service
o.a.c.net
o.a.c.dht
o.a.c.gms
o.a.c.locator
o.a.c.stream
#Cassandra13

41. o.a.c.net.MessagingService.verb<<enum>>
MUTATION
READ
REQUEST_RESPONSE
TREE_REQUEST
TREE_RESPONSE
(And more...)
#Cassandra13

42. o.a.c.net.MessagingService.verbHandlers
new EnumMap<Verb,
IVerbHandler>(Verb.class)
#Cassandra13

43. o.a.c.net.IVerbHandler<T>
doVerb(MessageIn<T> message,
String id);
#Cassandra13

44. o.a.c.net.MessagingService.verbStages
new EnumMap<MessagingService.Verb,
Stage>(MessagingService.Verb.class)
#Cassandra13

45. o.a.c.net.MessagingService.receive()
runnable = new MessageDeliveryTask(
message, id, timestamp);
StageManager.getStage(
message.getMessageType());
stage.execute(runnable);
#Cassandra13

46. o.a.c.net.MessageDeliveryTask.run()
// If dropable and rpc_timeout
MessagingService.incrementDroppedMessag
es(verb);
MessagingService.getVerbHandler(verb)
verbHandler.doVerb(message, id)
#Cassandra13

47. Architecture
API Layer
Dynamo Layer
Database Layer
#Cassandra13

48. Database Layer
o.a.c.concurrent
o.a.c.db
o.a.c.cache
o.a.c.io
o.a.c.trace
#Cassandra13

49. o.a.c.concurrent.StageManager
stages = new EnumMap<Stage,
ThreadPoolExecutor>(Stage.class);
getStage(Stage stage)
#Cassandra13

50. o.a.c.concurrent.Stage
READ
MUTATION
GOSSIP
REQUEST_RESPONSE
ANTI_ENTROPY
(And more...)
#Cassandra13

51. Database Layer
o.a.c.concurrent
o.a.c.db
o.a.c.cache
o.a.c.io
o.a.c.trace
#Cassandra13

52. o.a.c.db.Table
// Keyspace
open(String table)
getColumnFamilyStore(String cfName)
getRow(QueryFilter filter)
apply(RowMutation mutation,
boolean writeCommitLog)
#Cassandra13

53. o.a.c.db.ColumnFamilyStore
// Column Family
getColumnFamily(QueryFilter filter)
getTopLevelColumns(...)
apply(DecoratedKey key,
ColumnFamily columnFamily,
SecondaryIndexManager.Updater
indexer)
#Cassandra13

54. o.a.c.db.IColumnContainer
addColumn(IColumn column)
remove(ByteBuffer columnName)
ColumnFamily
SuperColumn
#Cassandra13

55. o.a.c.db.ISortedColumns
addColumn(IColumn column,
Allocator allocator)
removeColumn(ByteBuffer name)
ArrayBackedSortedColumns
AtomicSortedColumns
TreeMapBackedSortedColumns
#Cassandra13

56. o.a.c.db.Memtable
put(DecoratedKey key,
ColumnFamily columnFamily,
SecondaryIndexManager.Updater
indexer)
flushAndSignal(CountDownLatch latch,
Future<ReplayPosition>
context)
#Cassandra13

57. o.a.c.db.ReadCommand
getRow(Table table)
SliceByNamesReadCommand
SliceFromReadCommand
#Cassandra13

58. o.a.c.db.IDiskAtomFilter
getMemtableColumnIterator(...)
getSSTableColumnIterator(...)
IdentityQueryFilter
NamesQueryFilter
SliceQueryFilter
#Cassandra13

59. Summary
CustomTThreadPoolServer Message.Dispatcher
CassandraServer QueryProcessor
ReadCommand
StorageProxy
IResponseResolver
IAsyncCallback
MessagingService
IVerbHandler
Table ColumnFamilyStore IDiskAtomFilter
API
Dynamo
Database
#Cassandra13

60. Thanks.
#Cassandra13

61. Aaron Morton
@aaronmorton
www.thelastpickle.com
Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License