Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Running Cassandra on
Amazon’s ECS



Anirvan Chakraborty 

@anirvan_c 

© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Agenda
• Motivation
• Docker
• Cassandra
• Cassandra on Docker best practices
•...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Motivation
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Motivation
• Ease of development
• Support polyglot languages, frameworks and
c...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Docker
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Docker history
• Came out of dotCloud, a PaaS company
• Was originally written ...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Docker tag line
Build, ship and run any app, anywhere
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Docker tag line
• Build: package your application in a container
• Ship: move i...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Why Docker?
• Deploy reliably & consistently
• Execution is fast and light weig...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Apache Cassandra
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
What is Apache Cassandra?
• Fast distributed database
• High Availability
• Lin...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Hash ring
• Data is partitioned around the ring
• Location of data in ring is d...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
CAP Tradeoff
• During network partition it is impossible to be
both consistent ...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Replication
• Choose “replication factor” or RF
• Data is always replicated to ...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Consistency level
source: https://academy.datastax.com/courses/ds101-introducti...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Cassandra on Docker
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Dockerize C* Dev Environment
• Make it run as slow, but as stable as possible!
...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Dockerize C* Production
• Use host networking (—net=host) for better
network pe...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Amazon EC2
Container Service
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
What is ECS?
…is a highly scalable, fast, container
management service that mak...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
What is ECS?
Amazon Docker as a Service
https://www.expeditedssl.com/aws-in-pla...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
How does ECS work?
http://www.allthingsdistributed.com/2015/07/under-the-hood-o...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Cluster
http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazo...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Container Instance
http://www.allthingsdistributed.com/2015/07/under-the-hood-o...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
ECS Agent
http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-ama...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Task
http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-e...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Task
http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-e...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Service
http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazo...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
ECS Service
http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-a...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Typical ECS workflow
• Build Docker image using whatever you want
• Push image ...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Cassandra on ECS
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Dockerize C* on ECS
• Simple service discovery using ECS API
• Custom Dockerfil...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Service Discovery
Using
• AWS EC2 API
• AWS ECS API
• AWS Metadata URL
1. Find ...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Custom Dockerfile
• Expose Cassandra configuration parameters as
Docker environ...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Node Startup
1. Find the IPs of all running seed nodes
2.Check and identify any...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Work in progress
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Where to next?
• Open source C* Dockerfile & entry-point script
• Use Consul fo...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Thanks!
Twitter: @cakesolutions

Tel: 0845 617 1200
Email: enquiries@cakesoluti...
© 2016 Anirvan Chakraborty CC BY-NC-SA 4.0
Resources
https://academy.datastax.com/courses/ds101-introduction-cassandra
htt...
Upcoming SlideShare
Loading in …5
×

Atmosphere 2016 - Anirvan Chakraborty - Running Cassandra on Amazon's ECS

142 views

Published on

Containers have become a key component of modern distributed application design. Similarly, Apache Cassandra has emerged as a clear choice for a scalable, high available and performant database in a highly distributed cloud computing architecture.
In this session, we'll walk you through a number of patterns that enabled us to run Cassandra successfully with Docker in production. We will also share the lessons learned in running Cassandra with a very small set of resources that are applicable to both your local development environment and larger, less constrained production deployments.
We'll then introduce you to Amazon ECS, a highly scalable, high-performance service to run and manage distributed applications using Docker. Finally, we will discuss a few best practices used by our customers for running their Cassandra clusters on ECS.

Published in: Technology
  • Be the first to comment

Atmosphere 2016 - Anirvan Chakraborty - Running Cassandra on Amazon's ECS

  1. 1. Running Cassandra on Amazon’s ECS
 
 Anirvan Chakraborty 
 @anirvan_c 

  2. 2. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Agenda • Motivation • Docker • Cassandra • Cassandra on Docker best practices • EC2 Container Service ECS • Cassandra on ECS
  3. 3. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Motivation
  4. 4. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Motivation • Ease of development • Support polyglot languages, frameworks and components • Operational simplicity • Quick feedback loop
  5. 5. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Docker
  6. 6. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Docker history • Came out of dotCloud, a PaaS company • Was originally written in Python • Got re-written in Golang in Feb, 2013 • Docker 0.1 was released on Mar, 2013 • Docker 1.11.1 is the latest release
  7. 7. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Docker tag line Build, ship and run any app, anywhere
  8. 8. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Docker tag line • Build: package your application in a container • Ship: move it between machines • Run: execute that container with your application • Any application: as long as it runs on Linux • Anywhere: local VM, bare metal, cloud instances
  9. 9. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Why Docker? • Deploy reliably & consistently • Execution is fast and light weight • Simplicity • Developer friendly workflow • Fantastic community
  10. 10. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Apache Cassandra
  11. 11. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 What is Apache Cassandra? • Fast distributed database • High Availability • Linear Scalability • Predictable performance • No single point of failure • Multi-DC • Easy to manage • Can use commodity hardware • Not a drop in replacement for RDBMS
  12. 12. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Hash ring • Data is partitioned around the ring • Location of data in ring is determined by partition key • Data is replicated to N servers based on RF • All nodes hold data and can answer read or write queries source: https://academy.datastax.com/courses/ds101-introduction-cassandra/introduction-cassandra-overview
  13. 13. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 CAP Tradeoff • During network partition it is impossible to be both consistent and highly available • Latency between data centres also makes consistency impractical • Cassandra chooses Availability & Partition tolerance over Consistency
  14. 14. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Replication • Choose “replication factor” or RF • Data is always replicated to each replica • If node is down, missing data is replayed via hinted handoff source: https://academy.datastax.com/courses/ds101-introduction-cassandra/introduction-cassandra-overview
  15. 15. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Consistency level source: https://academy.datastax.com/courses/ds101-introduction-cassandra/introduction-cassandra-overview • Per query consistency • ALL, QUORUM, ONE • How many replicas to respond OK for query to succeed
  16. 16. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Cassandra on Docker
  17. 17. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Dockerize C* Dev Environment • Make it run as slow, but as stable as possible! • Super low memory settings in cassandra-env.sh • MAX_HEAP_SIZE=“128M” • HEAP_NEWSIZE=“24M” • Remove caches in dev mode in cassandra.yml • key_cache_size_in_mb: 0 • reduce_cache_sizes_at: 0 • reduce_cache_capacity_to: 0
  18. 18. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Dockerize C* Production • Use host networking (—net=host) for better network performance • Put data, commitlog and saved_caches in volume mount folders to the underlying host • Run cassandra on the foreground using (-f) • Tune JVM heap for optimal size • Tune JVM garbage collector for your workload
  19. 19. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Amazon EC2 Container Service
  20. 20. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 What is ECS? …is a highly scalable, fast, container management service that makes it easy to run, stop, and manage Docker containers on a cluster of Amazon EC2 instances. http://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html
  21. 21. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 What is ECS? Amazon Docker as a Service https://www.expeditedssl.com/aws-in-plain-english
  22. 22. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 How does ECS work? http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html
  23. 23. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Cluster http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html
  24. 24. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Container Instance http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html
  25. 25. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 ECS Agent http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html
  26. 26. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Task http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html
  27. 27. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Task http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html { "containerDefinitions": [ { "name": "wordpress", "links": [ "mysql" ], "image": "wordpress", "essential": true, "portMappings": [ { "containerPort": 80, "hostPort": 80 } ], "memory": 500, "cpu": 10 }, { "name": "mysql", "image": "mysql", "cpu": 10, "memory": 500, "essential": true } ], "family": "hello_world" }Task definition Docker Container Docker Container
  28. 28. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Service http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html
  29. 29. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 ECS Service http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html
  30. 30. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Typical ECS workflow • Build Docker image using whatever you want • Push image to registry • Create JSON file describing your task definition • Register this task definition with ECS • Make sure that your cluster has enough resources • Start a new task from the task definition
  31. 31. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Cassandra on ECS
  32. 32. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Dockerize C* on ECS • Simple service discovery using ECS API • Custom Dockerfile and entry-point script to control Cassandra configuration • Cleanup downed node and repair cluster on node startup
  33. 33. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Service Discovery Using • AWS EC2 API • AWS ECS API • AWS Metadata URL 1. Find all running tasks for a cluster 2.Find EC2 instance ids for the tasks 3.Find private IP addresses for the container instances
  34. 34. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Custom Dockerfile • Expose Cassandra configuration parameters as Docker environment variables • Update entry-point script to use Docker environment variables to configure Cassandra
  35. 35. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Node Startup 1. Find the IPs of all running seed nodes 2.Check and identify any downed nodes in the cluster nodetool status 3.Remove the downed nodes nodetool remove 4.Repair the cluster nodetool repair 5.Start a new Cassandra node
  36. 36. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Work in progress
  37. 37. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Where to next? • Open source C* Dockerfile & entry-point script • Use Consul for Service Discovery in ECS • Consider Flocker for managing volumes with Dockerized C* • Consider Weave for forming C* cluster across network domains
  38. 38. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Thanks! Twitter: @cakesolutions
 Tel: 0845 617 1200 Email: enquiries@cakesolutions.net Jobs: http://www.cakesolutions.net/ careers
  39. 39. © 2016 Anirvan Chakraborty CC BY-NC-SA 4.0 Resources https://academy.datastax.com/courses/ds101-introduction-cassandra http://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2- container-service.html http://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html https://www.expeditedssl.com/aws-in-plain-english https://github.com/aws/amazon-ecs-agent https://blog.docker.com/2015/03/why-i-love-docker-and-why-youll-love-it-too/ http://www.datastax.com/resources/whitepapers/best-practices-running-datastax- enterprise-within-docker

×