Getting to Know the Cassandra Codebase

Cozy with Cassandra
Getting to know the Cassandra Codebase
Gary Dusbabek • Rackspace
@gdusbabek
Cassandra Summit • Mission Bay Conference Center • San Francisco • 10 August 2010

Outline
Code Themes
Startup Sequence
Key Classes
Read Path
Write Path
Stages & Threading
Bootstrap & Streaming
Tests & IDE considerations
Adding API methods
Questions

Themes and Patterns
Layers

Themes and Patterns
Services

Themes and Patterns
Singletons
and
statics

Themes and Patterns
Stages &
Thread pools

StartupProcess

CassandraDaemon
Loads configuration
Transport initialization
Storage (Keyspace initialization)
CommitLogrecovery
StorageService.initServer()
Initializes CassandraServer
Passes it off to transport

CassandraServer
Implements IDL interface methods (cassandra.thrift, cassandra.genavro)
Good place to start diving when adding or troubleshooting API methods

Configuration
DatabaseDescriptor
Via CassandraDaemon.setup()
Looks for config path, loads yaml
Doesn’t spin anything up
Defines system tables
KS and CF described by CFMetaData and KSMetaData

Code
Some
Classes

Main Controllers
End with *Service or *Manager
StorageService, MessagingService
CompactionManager, HintedHandoffManager, StageManager, StreamInManager

StorageProxy
Put & Get methods
Collection of static methods
Merges local and distributed operations
Tracks latency
Exposed via StorageProxyMBean

StorageService
initServer()—Starts services
Registers verb handlers (in MessagingService)
Main event responders
Repository of replication strategies and TokenMetadata
Ring topology & token information

MessagingService
Verb handlers reside here
Sets up socket listeners
Gateway for outbound messages
MS.sendRR()
MS.sendOneWay()
Inbound too
MS.receive()

Table & ColumnFamilyStore
Also RowMutation
Low-level storage operations
o.a.c.db.*
SSTable
Local operations

Read+Write Paths

Reading
Socket->CassandraServer
Permissions
Request validation
Marshalling

Reading
StorageProxy
Ranges
Collectors
Local & remote branches

Reading
StorageProxy local
Table, ColumnFamilyStore
CFS
Make QueryFilter
Query Memtables
Query SSTables
Coalesce in iterators
o.a.c.db package
o.a.c.db.filter

Reading
StorageProxy remote
read command
Response handler
Send to remote nodes

Writing
Socket->CassandraServer
Validation
Convert to Mutation (IDL object)
Penalties!

Writing
StorageProxy
blocking/non-blocking mutate
local/remote branch
RowMutation
one ColumnFamily per
ColumnFamily
Collection of column modifications

Writing
RM.apply->Table.apply
Write to CL
Iterate over RM CFs
CFS.apply()
Overwrites results on pre-existing column families

Writing
RM is serialized into a Message and sent to other nodes
Waits for ACKs depending on CL

Stages & Threading

Stages
SEDA
http://www.eecs.harvard.edu/~mdw/papers/seda-sosp01.pdf
o.a.c.concurrent.StageManager
Read, mutation, stream, gossip, response, anti entropy, load balance, migration
Thread pools that consume tasks

Threads
MessagingService.listen() spawns thread.
Each incoming connection spawns a new short-lived thread (IncomingTcpConnection)
Non-stream ops go to MS.messageDeserializerExecutor_
Stream ops handled there.
Anti-entropy repair

Bootstraping!
Any interest?
//FIXMETODOCRAP

<=0.6 Bootstraping
A wants data, B has data.
StreamingRequestMessage A->B
Handled on B by StreamRequestVerbHandler
For each range StreamOut.transferRanges()
Flush, anticompaction
StreamInitiateMessage B->A for each range transfer
Meanwhile, back on A…
StreamInitiateVerbHandler gets the SIM from B, does some nesting.
StreamInitiateDone A->B
Back to B…
StreamInitiateDoneHandler gets the SID from A
Calls StreamOutManager.startNext() which sends a single file to A
MessagingService on A picks this up and the file is streamed.
Sstable is created
STREAM_FINISHED A->B
B gets rid of the file, calls SOM.startNext()

0.7 Bootstrapping
A wants data, B has data
StreamRequestMessage A->B
On B, StreamRequestVerbHandler
If single file, sends it.
If range, StreamOut.transferRangesForRequest()
Send next file (first will contain meta data about all files)
On A, IncomingStreamReader.read()
Data is received, sstable created
Ack, request next file

Tests
Testable & Untestable
Unit tests
ant clean build test
System tests
ant gen-thrift-py
nosetests test/system/test_thrift_server.py

IDE
Configuration file must be in the classpath
Treat as source
lib vs build/lib
Log at debug

IDE
-ea -Xms128M –Xmx2G
-Dcom.sun.management.jmxremote.port=8081
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcassandra-foreground=yes
-Dlog4j.configuration=log4j-server.properties
-Dmx4jport=9081

Adding API methods
Same goes for modifying
Define method and structures in IDL
interface/cassandra.thrift
Regenerate files
ant gen-thrift-java gen-thrift-py
Implement methods in o.a.c.thrift.CassandraServer
Create a system test (tests/system/test_thrift_server.py)

Questions?
gdusbabek@gmail.com
@gdusbabek

Getting to Know the Cassandra Codebase

by gdusbabek on Aug 10, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

2 Embeds 3

Statistics

Getting to Know the Cassandra Codebase — Presentation Transcript

http://coral.corp.apple.com	2
http://www.slideshare.net	1