This document discusses best practices for running Cassandra on Amazon EC2. It recommends instance sizes like m1.xlarge for most use cases. It emphasizes configuring data and commit logs on ephemeral drives for better performance than EBS volumes. It also stresses the importance of distributing nodes across availability zones and regions for high availability. Overall, the document provides guidance on optimizing Cassandra deployments on EC2 through choices of hardware, data storage, networking and operational practices.

1. Cassandra On EC2
Matthew F. Dennis // @mdennis
@mdennis

2. Instance Sizes
● m1.xlarge is by far the most common size
● m1.large is ok for many use cases
● m2.4xlarge in some cases
● keep the entire dataset in memory
● c1.xlarge / cc1.4xlarge
● Smallish but very hot set of data
– regardless of how much data is on disk
● Extremely high request rate
● Encrypted node-node communications and high traffic
● Usually better off with many m1.xlarge instances because of
the extra memory, but not always
@mdennis

3. Configuration
● Stripe All Ephemeral Drives
● data directory and commit log on same volume
● Only applies to EC2 and SSDs, not physical HW
● Why?
● 6-8 GB heap on m1.xlarge
● 3-4 GB heap on m1.large
● Phi Convict Threshold? Maybe ...
@mdennis

4. EBS versus Ephemeral
● Ephemeral drives are:
● Generally faster for C*
● More stable (no pauses/freezes; outages?)
● Cheaper
● Easier to initially configure
● Striped EBS?
● yeah, about that …
● TL;DL don't use EBS for C* on EC2
@mdennis

5. Multi-Zone
● Alternate zones in your token topology
● No really, this is important, alternate zones
– We should probably fix this ...
● “complicated, but possible” to add new zones
after initial deployment
● Never move a *token* to a different region or
zone
● If you think that is what you want to do, really you
want to bootstrap new one at token-1 in the new
region/zone and then decom the old one
@mdennis

6. Multi-Region C* on EC2
● Connectivity is the complicated part
● Ec2MultiRegionSnitch is not the entire answer
– https://issues.apache.org/jira/browse/CASSANDRA-2452
● Don't try to make a “fail over” DC, just go with active-active
● If you insist, then do the fail over in your application and configure C* the
same as you would active-active
● Generally requires a lot more storage
● Doesn't matter though because you're using ephemeral drives (right?)
and don't want a TB of data on each node anyway
@mdennis

7. Multi-Region Connectivity Options
● VPN
● Encrypted node-node communication
● CPU utilization is often a downside
● VPNCubed / VPCPlus
● I've never deployed it, heard good things about it though
● Amazon VPC
● anyone know if a single VPC can span regions yet?
● SSH Tunnels
● EC2 security groups
● IPTables
● Encrypted node-node + public IP binding + AWS security groups +
IPTables (EIPs may simplify this, never actually tried it)
@mdennis

8. Recovery From Failures
● Don't “fix” EC2 nodes, replace them
● boostrap at token-1, remove old token
– bootstrap can be slow, but will get better
● Other than that it's the same in EC2 as not ...
@mdennis

9. Node Maintenance
● “Maintenance” On EC2?
● Usually not required (just replace the node)
● If it is, just stop C*, CL+HH/repair/RR will fix it
● Same as physical HW
● https://issues.apache.org/jira/browse/CASSANDRA-2034
● Stop Trying To Decom Nodes Just To Replace a Disk !!!
@mdennis

10. Backups
● C* snapshots and push to S3
● Directory Watcher that pushes new files to S3
● SimpleGeo: https://github.com/simplegeo/tablesnap
● Netflix: http://slidesha.re/NFOnCassBkup
● Keep a log of all incoming writes
● Not specific to S3
● Can be coupled with snapshots / S3
● Useful for other reasons as well
● Compression in transit to S3 (or where ever) can be done on
a separate EC2 instance to avoid burning CPU
● Usually not worth the extra complexity / cost
@mdennis

11. Changing Node Sizes
● Start a new instance
● rsync data from from original node to new node
● Shutdown C* on original node
● rsync data from from original node to new node
● Start C* on new node
● Shutdown original instance
● NB: Assumes same token, region, zone, etc
@mdennis

12. Elastic Load Balancers
● They're awesome, use them
● Could be more awesome (e.g. better integration with Route 53)
● What I really want is TCP anycast for ELB across regions (AWS could
make it work)
● Balance across regions with GeoIP / GeoDNS
● Zerigo, TZOHA, Neustar, “homegrown”, etc
● Route 53? You wish (though Route 53 itself is run over anycast)
– “in the future we plan for Route 53 to also give you greater control over
… the route your users take to reach an endpoint” --Werner Vogels
● Put them in front of your app servers, not your C* instances
● Keep your app servers stateless or at least “weakly” stateless (e.g. no sticky
sessions required)
@mdennis

13. AMIs versus Scripted Setup
● DataStax publishes C* AMIs
● Chef Recipes as well
● Or roll your own …
● Whatever you do, just make sure it's automated
and repeatable
● *personally* I prefer scripting the setup
remotely, but this is … “less than ideal”
● PSSH is, in general, awesome
@mdennis

14. WTF?!
● Your zone X is not the same as my zone X
● Consistent within an EC2 account
● Problematic across accounts
● Does not apply to regions (i.e. your region X is my region X)
● EIPs resolve to private IPs from within AWS
● EBS volumes sometimes just “freeze”
● AWS: “yeah, that happens sometimes under load”
● steal% sometimes 20% or more (1%-3% is “normal”)
● This is AWS literally stealing your money
● Thankfully not all that common, but watch out for it
@mdennis

15. Missing AWS Features
● ELB over anycast
● Probably doable by AWS, but not others ...
● GeoDNS from Route53
● No really, WTF Doesn't Route53 Do GeoDNS ?!?!
● Multi-Region VPC
● Local SSDs
@mdennis

16. We're Hiring !
● Developers
● QA
● Community Manager
● Sales / SE
● Interns
– Dev
– Support
– QA
● Smart People Interested In Cassandra
@mdennis

17. Cassandra On EC2
Q?
(yes, I'll post the slides on slideshare)
@mdennis