More Related Content
What's hot
What's hot (20)
Similar to Cassandra Internals Overview
Similar to Cassandra Internals Overview (20)
Recently uploaded
Recently uploaded (20)
Cassandra Internals Overview
- 1. CASSANDRA INTERNALS OVERVIEW DATASTAX BOOTCAMP 2015 Sam Tunnicliffe sam@datastax.com / @beobal
- 3. STARTUP org.apache.cassandra.service.CassandraDaemon protected void setup() Load config Run preflight checks Load schema Clean up local temporary state Recover CommitLog Schedule background compactions Initialize storage service
- 4. PREFLIGHT CHECKS Sane clock JNI JVM & Instrumentation Filesystem permissions System keyspace status Upgrades (#8049) Incompatible SSTables (#8049)
- 5. STARTUP org.apache.cassandra.service.CassandraDaemon protected void setup() Load config Run pre-flight checks Load schema Clean up local temporary state Recover CommitLog Schedule background compactions Initialize storage service
- 6. CLEAN UP LOCAL STATE Truncate compactions_in_progress Scrub data directories
- 7. STARTUP org.apache.cassandra.db.commitlog.CommitLog public int recover() throws IOException Load config Run pre-flight checks Load schema Clean up local temporary state Recover CommitLog Schedule background compactions Initialize storage service
- 8. INITIALIZE STORAGE SERVICE org.apache.cassandra.service.StorageService public synchronized void initServer() throws ConfigurationException Load ring state (unless don't) Start gossip & get initial ring info Set tokens
- 9. BOOTSTRAP Abort if other range movements happening Fetch bootstrap data Build secondary indexes
- 10. INITIALIZE STORAGE SERVICE Load ring state (unless don't) Start gossip & get initial ring info Set tokens Setup auth resources Ensure gossip stabilized
- 11. STARTUP Load config Run preflight checks Load schema Clean up local temporary state Recover CommitLog Schedule background compactions Initialize storage service
- 12. -- it is done -- STARTUP
- 13. MESSAGINGSERVICE org.apache.cassandra.net.MessagingService Low level one-way messaging public void sendOneWay(MessageOut message, InetAddress to) Async Request/Response public int sendRR(MessageOut message, InetAddress to, IAsyncCallback cb)
- 15. MESSAGINGSERVICE Pre-emptively drops messages when overwhelmed Dropped if time at execution > send time + timeout Timeout value dependant on message type Most client-initated requests can be dropped (see MessagingService.DROPPABLE_VERBS)
- 16. GOSSIP What it does do: Disseminates members' state around the cluster Versioned: generation (per JVM) & version (per value) Heartbeats: incremented every gossip round Application state: Status Tokens Release & schema version DC & Rack Addresses Data size Health
- 17. GOSSIP What doesn't it do: Notify about up or down nodes Propagate schema Transmit data files Distribute mutations
- 19. GOSSIP org.apache.cassandra.gms.Gossiper private class GossipTask implements Runnable { public void run() {... Each round (1 second) gossip to: 1 live endpoint maybe 1 unreachable endpoint maybe 1 seed - if neither of the above
- 20. SCHEMA MIGRATION Another custom protocol Also uses MessagingService Target schema objects serialized as Mutations diff/merge schema representations
- 23. Client request arrives at coordinator: COORDINATION Transformed into actionable command(s): IReadCommand IMutation Coordinator distributes execution around the cluster Replicas perform commands and respond to coordinator Gather responses and determine client response
- 25. https://wiki.apache.org/cassandra/ArchitectureInternals COORDINATING WRITES org.apache.cassandra.service.StorageProxy public static void mutate(Collection<? extends IMutation> mutations, ConsistencyLevel consistency_level) Get endpoints using replication strategy Get pending endpoints from ring metadata Deliver mutations to both sets of endpoints Collate responses & determine client response Maybe store local hints for unreachable replicas
- 28. https://wiki.apache.org/cassandra/ArchitectureInternals COORDINATING WRITES org.apache.cassandra.service.StorageProxy public static void mutate(Collection<? extends IMutation> mutations, ConsistencyLevel consistency_level) Get endpoints using replication strategy Get pending endpoints from ring metadata Deliver mutations to both sets of endpoints Collate responses & determine client response Maybe store local hints for unreachable replicas
- 29. DELIVERING MUTATIONS org.apache.cassandra.service.StorageProxy public static void sendToHintedEndpoints(final Mutation mutation, Iterable<InetAddress> targets, AbstractWriteResponseHandler responseHandler, String localDataCenter) Mutations sent to replicas using MessagingService ResponseHandler registered as callback Callback registry triggers an event on expiry Sent directly within local datacenter Forwarded via single node in each remote DC
- 30. COORDINATING WRITES org.apache.cassandra.service.StorageProxy public static void mutate(Collection<? extends IMutation> mutations, ConsistencyLevel consistency_level) Get endpoints using replication strategy Get pending endpoints from ring metadata Deliver mutations to both sets of endpoints Collate responses & determine client response Maybe store local hints for unreachable replicas
- 31. HINTS Nodes can be down Writes may timeout In which case we may hint Enabled/disabled globally or enabled per-DC Writing a hint counts towards ConsistencyLevel.ANY Deliver hints when a node comes back up & periodically Too many hints in progress for a replica means we bail early
- 32. Determine point of failure by WriteType LOGGED BATCHES org.apache.cassandra.service.StorageProxy public static void mutateAtomically(Collection<Mutation> mutations, ConsistencyLevel consistency_level) CommitLog for batches Guarantee eventual success of batched statements Strives to distribute to across racks in local DC On success, cleanup log entries asynchronously Failed batches replayed by the nodes holding the logs WriteType.BATCH_LOG WriteType.BATCH
- 33. COORDINATING READS org.apache.cassandra.service.StorageProxy public static List<Row> read(List<ReadCommand> commands, ConsistencyLevel consistencyLevel, ClientState state) Partition based reads Read Repair & Data vs Digest Requests Rapid Read Protection & (non)speculating executors Distribution is more slightly complex than for writes
- 34. IDENTIFY TARGET ENDPOINTS org.apache.cassandra.service.AbstractReadExecutor public static AbstractReadExecutor getReadExecutor(ReadCommand command, ConsistencyLevel consistencyLevel) Use replication strategy to get live endpoints Snitch sorts by proximity & health of replicas Consult table metadata for Read Repair Decision
- 35. READ REPAIR DECISION Apply filter to sorted list of all live replicas NONE: closest n replicas required by CL GLOBAL: all live replicas DC_LOCAL: all local replicas Add closest n remotes needed to satisfy CL Default Global Chance: 0.0 Default Local Chance: 0.1 Give us a list of replicas to send read requests
- 36. RAPID READ PROTECTION Never Always Fixed timeout Table latency percentile
- 37. LIGHTS, CAMERA, EXECUTION Fire off each command using read executor Requests are sent via MessagingService Closest replica(s) sent full data requests Others get digest requests
- 38. RESOLUTION Resolution can have two outcomes:
- 39. RESOLUTION DigestMismatchException Trigger a foreground read repair Of all targetted replicas
- 40. FOREGROUND READ REPAIR All data requests, no digests Includes replicas contacted initially Effectively ConsistencyLevel.ALL Specialized resolver: RowDataResolver Retry any short reads May also perform background Read Repair
- 41. OVERVIEW OVER