• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Fault Tolerance in Cassandra
 

Fault Tolerance in Cassandra

on

  • 4,709 views

A short talk on how Cassandra deals with various failure modes. Discussion of replication and consistency levels and how they can be used to survive many kinds of failure. Ends with explanation of ...

A short talk on how Cassandra deals with various failure modes. Discussion of replication and consistency levels and how they can be used to survive many kinds of failure. Ends with explanation of recovery methods - repair, hinted handoff and read repair.

Statistics

Views

Total Views
4,709
Views on SlideShare
4,489
Embed Views
220

Actions

Likes
5
Downloads
37
Comments
0

12 Embeds 220

http://feeds.feedburner.com 93
http://www.acunu.com 87
http://acunu.com 27
http://twitter.com 2
http://tweets.acunu.com 2
http://www.twylah.com 2
http://www.slashdocs.com 2
http://us-w1.rockmelt.com 1
http://notacunu.com 1
http://www.weebly.com 1
https://twitter.com 1
http://wave.webaim.org 1
More...

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Fault Tolerance in Cassandra Fault Tolerance in Cassandra Presentation Transcript

    • Fault tolerance in Cassandra Richard Low richard@acunu.com @acunu @richardalowCassandra London Meetup, 5 Sept 2011Tuesday, 6 September 2011
    • Menu • Failure modes • Maintaining availability • RecoveryTuesday, 6 September 2011
    • Failure modesTuesday, 6 September 2011
    • Failures are the norm • With more than a few nodes, something goes wrong all the time • Don’t want to be down all the timeTuesday, 6 September 2011
    • Failure causes • Hardware failure • Bug • Power • Natural disasterTuesday, 6 September 2011
    • Failure modes • Data centre failure • Node failure • Disk failureTuesday, 6 September 2011
    • Failure modes • Data centre failure • Node failure • Disk failure • Temporary • PermanentTuesday, 6 September 2011
    • Failure modes • Network failure • One node • Network partition • Whole data centreTuesday, 6 September 2011
    • Failure modes • Operator failure • Delete files • Delete entire database • Incorrect configurationTuesday, 6 September 2011
    • Failure modes • Want a system that can tolerate all the above failures • Make assumptions about probabilities of multiple events • Be careful when assuming independenceTuesday, 6 September 2011
    • Solutions • Do nothing • Make boxes bullet proof • ReplicationTuesday, 6 September 2011
    • AvailabilityTuesday, 6 September 2011
    • How maintain availability in the presence of failure?Tuesday, 6 September 2011
    • Replication • Buy cheap nodes and cheap disks • Store multiple copies of the data • Don’t care if some disappearTuesday, 6 September 2011
    • Replication • What about consistency? • What if I can’t tolerate out-of-date reads? • How restore a replica?Tuesday, 6 September 2011
    • RF and CL • Replication factor • How many copies • How much failure can tolerate • Consistency Level • How many nodes must be contactable for operation to succeedTuesday, 6 September 2011
    • Simple example • Replication factor 3 • Uniform network topology • Read and write at CL.QUORUM • Strong consistency • Available if any one node is down • Can recover if any two nodes failTuesday, 6 September 2011
    • In general • RF N, reads and writes at CL.QUORUM • Available if ceil(N/2)-1 nodes fail • Can recover if N-1 nodes failTuesday, 6 September 2011
    • Multi data centre • Cassandra knows location of hosts • Through the snitch • Can ensure replicas in each DC • NetworkTopologyStrategy • => can cope with whole DC failureTuesday, 6 September 2011
    • RecoveryTuesday, 6 September 2011
    • Recovery • Want to maintain replication factor • Ensures recovery guarantees • Methods: • Automatic • ManualTuesday, 6 September 2011
    • AutomaticTuesday, 6 September 2011
    • Automatic processes • Eventually moves replicas towards consistency • The ‘eventual’ in ‘eventual consistency’Tuesday, 6 September 2011
    • Hinted Handoff • Hints • Stored on any node • When a node is temporarily unavailable • Delivered when the node comes back • Can use CL.ANY • Writes not immediately readableTuesday, 6 September 2011
    • Read Repair • Since done a read, might as well repair any old copies • Compare values, update any out of syncTuesday, 6 September 2011
    • ManualTuesday, 6 September 2011
    • Repair: method • Ensures a node is up to date • Run ‘nodetool -h <node> repair’ • Reads through entire data on the node • Builds a Merkel tree • Compares with replicas • Streams differencesTuesday, 6 September 2011
    • Repair: when • After node has been down a long time • After increasing replication factor • Every 10 days to ensure tombstones are propagated • Can be used to restore a failed nodeTuesday, 6 September 2011
    • Replace a node: method • Bootstrap new node with <old_token>-1 • Tell existing nodes old node is dead • nodetool removeTuesday, 6 September 2011
    • Replace a node: when • Complete node failure • Cannot replace failed disk • CorruptionTuesday, 6 September 2011
    • Restore from backup: method • Stop Cassandra on the node • Copy SSTables from backup • Restart Cassandra • Make take a while reading indexesTuesday, 6 September 2011
    • Restore from backup: when • Disk failure • with no RAID rebuild available • Operator error • Corruption • HackerTuesday, 6 September 2011
    • Thanks :) www.acunu.com richard@acunu.com @acunu @richardalowTuesday, 6 September 2011