Your SlideShare is downloading. ×
Cassandra SF 2013 - Cassandra Internals
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Cassandra SF 2013 - Cassandra Internals

2,104
views

Published on

Cassandra SF 2013 Conference - Cassandra Internals

Cassandra SF 2013 Conference - Cassandra Internals

Published in: Technology

0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,104
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
7
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. CASSANDRA SF 2013CASSANDRAINTERNALSAaron Morton@aaronmortonwww.thelastpickle.com#Cassandra13Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License
  • 2. About MeFreelance Cassandra ConsultantBased in Wellington, New ZealandApache Cassandra Committer#Cassandra13
  • 3. ArchitectureCode#Cassandra13
  • 4. Cassandra ArchitectureAPIsCluster AwareCluster UnawareClientsDisk#Cassandra13
  • 5. Cassandra Cluster ArchitectureAPIsCluster AwareCluster UnawareClientsDiskAPIsCluster AwareCluster UnawareDiskNode 1 Node 2#Cassandra13
  • 6. Dynamo Cluster ArchitectureAPIsDynamoDatabaseClientsDiskAPIsDynamoDatabaseDiskNode 1 Node 2#Cassandra13
  • 7. ArchitectureAPIDynamoDatabase#Cassandra13
  • 8. APITransportsThriftNative Binary#Cassandra13
  • 9. ThriftTransport//Custom TServer implementationso.a.c.thrift.CustomTThreadPoolServero.a.c.thrift.CustomTNonBlockingServero.a.c.thrift.CustomTHsHaServer#Cassandra13
  • 10. APITransportsThriftNative Binary#Cassandra13
  • 11. Native BinaryTransportBeta in Cassandra 1.2Uses NettyEnabled withstart_native_transport(Disabled by default)#Cassandra13
  • 12. o.a.c.transport.Server.run()//Setup the Netty servernew ExecutionHandler()new NioServerSocketChannelFactory()ServerBootstrap.setPipelineFactory()#Cassandra13
  • 13. o.a.c.transport.Message.Dispatcher.messageReceived()//Process message from clientServerConnection.validateNewMessage()Request.execute()ServerConnection.applyStateTransition()Channel.write()#Cassandra13
  • 14. MessagesDefined in the Native BinaryProtocol$SRC/doc/native_protocol.spec#Cassandra13
  • 15. API ServicesJMXThriftCQL 3#Cassandra13
  • 16. JMX Management BeansSpread around the code base.Interfaces named *MBean#Cassandra13
  • 17. JMX Management BeansRegistered with names such asorg.apache.cassandra.db:type=StorageProxy#Cassandra13
  • 18. API ServicesJMXThriftCQL 3#Cassandra13
  • 19. o.a.c.thrift.CassandraServer// Implements Thrift Interface// Access control// Input validation// Mapping to/from Thrift and internal types#Cassandra13
  • 20. Thrift InterfaceThrift IDL$SRC/interface/cassandra.thrift#Cassandra13
  • 21. o.a.c.thrift.CassandraServer.get_slice()// get columns for one rowTracing.begin()ClientState cState = state()cState.hasColumnFamilyAccess()multigetSliceInternal()#Cassandra13
  • 22. CassandraServer.multigetSliceInternal()// get columns for may rowsThriftValidation.validate*()// Create ReadCommandsgetSlice()#Cassandra13
  • 23. CassandraServer.getSlice()// Process ReadCommands// return Thrift typesreadColumnFamily()thriftifyColumnFamily()#Cassandra13
  • 24. CassandraServer.readColumnFamily()// Process ReadCommands// Return ColumnFamiliesStorageProxy.read()#Cassandra13
  • 25. API ServicesJMXThriftCQL 3#Cassandra13
  • 26. o.a.c.cql3.QueryProcessor// Prepares and executes CQL3 statements// Used by Thrift & Native transports// Access control// Input validation// Returns transport.ResultMessage#Cassandra13
  • 27. CQL3 GrammarANTLR Grammar$SRC/o.a.c.cql3/Cql.g#Cassandra13
  • 28. o.a.c.cql3.statements.ParsedStatement// Subclasses generated by ANTLR// Tracks bound term count// Prepare CQLStatementprepare()#Cassandra13
  • 29. o.a.c.cql3.statements.CQLStatementcheckAccess(ClientState state)validate(ClientState state)execute(ConsistencyLevel cl,QueryState state,List<ByteBuffer> variables)#Cassandra13
  • 30. statements.SelectStatement.RawStatement// Implements ParsedStatement// Input validationprepare()#Cassandra13
  • 31. statements.SelectStatement.execute()// Create ReadCommandsStorageProxy.read()#Cassandra13
  • 32. ArchitectureAPIDynamoDatabase#Cassandra13
  • 33. Dynamo Layero.a.c.serviceo.a.c.neto.a.c.dhto.a.c.gmso.a.c.locatoro.a.c.stream#Cassandra13
  • 34. o.a.c.service.StorageProxy// Cluster wide storage operations// Select endpoints & check CL available// Send messages to Stages// Wait for response// Store Hints#Cassandra13
  • 35. o.a.c.service.StorageService// Ring operations// Track ring state// Start & stop ring membership// Node & token queries#Cassandra13
  • 36. o.a.c.service.IResponseResolverpreprocess(MessageIn<T> message)resolve() throwsDigestMismatchExceptionRowDigestResolverRowDataResolverRangeSliceResponseResolver#Cassandra13
  • 37. Response Handlers / Callbackimplements IAsyncCallback<T>response(MessageIn<T> msg)#Cassandra13
  • 38. o.a.c.service.ReadCallback.get()//Wait for blockfor & datacondition.await(timeout,TimeUnit.MILLISECONDS)throw ReadTimeoutException()resolver.resolve()#Cassandra13
  • 39. o.a.c.service.StorageProxy.fetchRows()getLiveSortedEndpoints()new RowDigestResolver()new ReadCallback()MessagingService.sendRR()---------------------------------------ReadCallback.get() # blockingcatch (DigestMismatchException ex)catch (ReadTimeoutException ex)#Cassandra13
  • 40. Dynamo Layero.a.c.serviceo.a.c.neto.a.c.dhto.a.c.gmso.a.c.locatoro.a.c.stream#Cassandra13
  • 41. o.a.c.net.MessagingService.verb<<enum>>MUTATIONREADREQUEST_RESPONSETREE_REQUESTTREE_RESPONSE(And more...)#Cassandra13
  • 42. o.a.c.net.MessagingService.verbHandlersnew EnumMap<Verb,IVerbHandler>(Verb.class)#Cassandra13
  • 43. o.a.c.net.IVerbHandler<T>doVerb(MessageIn<T> message,String id);#Cassandra13
  • 44. o.a.c.net.MessagingService.verbStagesnew EnumMap<MessagingService.Verb,Stage>(MessagingService.Verb.class)#Cassandra13
  • 45. o.a.c.net.MessagingService.receive()runnable = new MessageDeliveryTask(message, id, timestamp);StageManager.getStage(message.getMessageType());stage.execute(runnable);#Cassandra13
  • 46. o.a.c.net.MessageDeliveryTask.run()// If dropable and rpc_timeoutMessagingService.incrementDroppedMessages(verb);MessagingService.getVerbHandler(verb)verbHandler.doVerb(message, id)#Cassandra13
  • 47. ArchitectureAPI LayerDynamo LayerDatabase Layer#Cassandra13
  • 48. Database Layero.a.c.concurrento.a.c.dbo.a.c.cacheo.a.c.ioo.a.c.trace#Cassandra13
  • 49. o.a.c.concurrent.StageManagerstages = new EnumMap<Stage,ThreadPoolExecutor>(Stage.class);getStage(Stage stage)#Cassandra13
  • 50. o.a.c.concurrent.StageREADMUTATIONGOSSIPREQUEST_RESPONSEANTI_ENTROPY(And more...)#Cassandra13
  • 51. Database Layero.a.c.concurrento.a.c.dbo.a.c.cacheo.a.c.ioo.a.c.trace#Cassandra13
  • 52. o.a.c.db.Table// Keyspaceopen(String table)getColumnFamilyStore(String cfName)getRow(QueryFilter filter)apply(RowMutation mutation,boolean writeCommitLog)#Cassandra13
  • 53. o.a.c.db.ColumnFamilyStore// Column FamilygetColumnFamily(QueryFilter filter)getTopLevelColumns(...)apply(DecoratedKey key,ColumnFamily columnFamily,SecondaryIndexManager.Updaterindexer)#Cassandra13
  • 54. o.a.c.db.IColumnContaineraddColumn(IColumn column)remove(ByteBuffer columnName)ColumnFamilySuperColumn#Cassandra13
  • 55. o.a.c.db.ISortedColumnsaddColumn(IColumn column,Allocator allocator)removeColumn(ByteBuffer name)ArrayBackedSortedColumnsAtomicSortedColumnsTreeMapBackedSortedColumns#Cassandra13
  • 56. o.a.c.db.Memtableput(DecoratedKey key,ColumnFamily columnFamily,SecondaryIndexManager.Updaterindexer)flushAndSignal(CountDownLatch latch,Future<ReplayPosition>context)#Cassandra13
  • 57. o.a.c.db.ReadCommandgetRow(Table table)SliceByNamesReadCommandSliceFromReadCommand#Cassandra13
  • 58. o.a.c.db.IDiskAtomFiltergetMemtableColumnIterator(...)getSSTableColumnIterator(...)IdentityQueryFilterNamesQueryFilterSliceQueryFilter#Cassandra13
  • 59. SummaryCustomTThreadPoolServer Message.DispatcherCassandraServer QueryProcessorReadCommandStorageProxyIResponseResolverIAsyncCallbackMessagingServiceIVerbHandlerTable ColumnFamilyStore IDiskAtomFilterAPIDynamoDatabase#Cassandra13
  • 60. Thanks.#Cassandra13
  • 61. Aaron Morton@aaronmortonwww.thelastpickle.comLicensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License