Continuous Deployment with Cassandra

1,733 views
1,600 views

Published on

Michael Kjellman, Software Engineer at Barracuda Networks, has offered to present on his experiences with Apache Cassandra.

Come learn about:

• Continuous Deployments with Cassandra
• Upgrading Cassandra
• When Upgrades Go Wrong
• Coding Complexity Moved to Operations (How to Prepare and Plan)
• Why 'Apt-get/Yum Install Cassandra' is a bad idea
• Why You Should Treat Cassandra’s Code like it's Your Own

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,733
On SlideShare
0
From Embeds
0
Number of Embeds
857
Actions
Shares
0
Downloads
6
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Continuous Deployment with Cassandra

  1. 1. Continuous Deployment with C*: Treating C* as First-Class Code Michael Kjellman @mkjellman Software Engineer, Barracuda Networks
  2. 2. C* At Barracuda • Powers 100% of our Spam and Webfilter Backend • 48 Node Cluster • 2 Datacenters • Requests: 20k writes/sec 30k reads/sec • Latency: 1 ms/write 1.6 ms/read • > 30TB of Data • Almost entirely native protocol/CQL3
  3. 3. Hardware Configuration • 32GB of RAM • 1x SSD • 2x Spinning Disks • 2x 6 Core AMD
  4. 4. Key Configuration Options • key_cache_size_in_mb: 1024 • row_cache_size_in_mb: 0 • memtable_total_space_in_mb: 2048 • HEAP_NEWSIZE = “1200M” (-Xmn) • MAX_HEAP_SIZE = “8G” (-Xmx) • -XX:SurvivorRatio=6 • Sidenote: Java 7u40 is out!
  5. 5. How do I keep my graphs pretty during a C* upgrade? September 18th 2013
  6. 6. Make a C* Build $> git clone http://git-wip- us.apache.org/repos/asf/cassandra.git $> git checkout –t origin/cassandra-1.2 $> git log $> vim build.xml (change version number every time you make a build!) $> ant clean release
  7. 7. Deployment • Make release • Test release with CCM • Push release to Puppet (deals with config, etc) • Run controlled and scripted rolling restart one datacenter at a time – flush – stop – start – validate node
  8. 8. Automate, Automate, Automate
  9. 9. So, why not just apt-get install cassandra? • Makes running a custom release in the future a complete nightmare • Lost visibility into changes in the release • WHY are you upgrading • Treat a C* build just as if it was a release of your code. What commits did you put into your own release?
  10. 10. MY CODE DOESN’T WORK WITHOUT A STABLE C* CLUSTER Simply Put:
  11. 11. When things go wrong • Every commit (those by C* committers or my own) come with potential bugs and regressions • Gossip Bugs Can Bite Hard: – CASSANDRA-5665: Gossiper.handleMajorStateChange can lose existing node ApplicationState • At 48 nodes, even small mistakes are massive
  12. 12. Writing your code to deal with node failure • Upgrading a C* cluster means constant node failures for the duration of the rolling restart • How does your code deal with read latency and retries – CASSANDRA-4705: Eager Retries for reads for 2.0+ • The mythical “constantly failing” code != stability. – Handle exceptions (and node/read failures) gracefully!
  13. 13. Why treat C* like your own code • Using C* will move much of your own application logic to C* • The bugs have to go somewhere! • Data replication at database layer or at application layer
  14. 14. QUESTIONS? Thanks for Listening!

×