Going public in 2014 presented Zalando's engineering team with some interesting new challenges, especially in the realms of auditing and compliance. To ensure that we remain in good standing with both our auditors and our customers, Zalando's cloud engineering team built STUPS: an open source set of tools and components that provides a convenient and audit-compliant Platform-as-a-Service (PaaS) for multiple autonomous teams on top of AWS. For those of us at Zalando who work with Cassandra, the creation and implementation of STUPS has added another set of challenges: How to orchestrate Cassandra in such an environment? Zalando's teams use Cassandra because of its incredible performance, linear scalability and multi-datacenter support. In a dynamic cloud environment, you can't just bootstrap a Cassandra cluster in the usual way. In this talk, I will show you how my team and I got the job done--not only while working with AWS, but while facing the auditing and compliance standards that our STUPS PaaS aims to address. I'll describe our solution to orchestrate Cassandra on AWS's EC2 platform using STUPS, Docker, and etcd. I'll also summarize how we've been able to extend our Cassandra clusters from our datacenters to AWS without any downtime. By the end, we hope you'll have some useful takeaways for deploying and managing Cassandra on AWS if you have similar requirements.

1. Bootstrapping Cassandra
in a Dynamic Cloud Environment
Luis Mineiro - @voidmaze
Thorbjörn Gruda - @ThorbyG

2. Agenda
Who We Are and What We Do
How Do We Run a Tech Company
Cloud Infrastructure for Autonomous Teams
Cassandra in a Dynamic Cloud Environment

3. 15 countries
3 fulfilment centers
15+ million active customers
2.2+ billion € revenue 2014
130+ million visits per month
8.000+ employees
ONE OF EUROPE’S LARGEST ONLINE FASHION RETAILERS
Visit us: tech.zalando.com
Tech hubs in Berlin, Dublin, Dortmund
and Helsinki

4. #CassandraSummit
Zalando
Technology

5. #CassandraSummit
Environment
Credits to our colleague Kolja Wilcke

6. #CassandraSummit
Environment
Credits to our colleague Kolja Wilcke

7. #CassandraSummit
Fashion Store Website

8. #CassandraSummit
Fashion Store Apps
iOSAndroid Windows Mobile

9. #CassandraSummit
Shoes in Space
http://thespaceshoe.com/

10. #CassandraSummit
Zalando Infrastructure
Based on https://flic.kr/p/bX5E4c
2 datacenters
thousands of production instances
serving 15 countries

11. #CassandraSummit
Radical
Agility

12. #CassandraSummit
What’s That?
Radical Agility is Zalando Tech's approach
to running a technology organization -
an approach based on the three pillars of
autonomy, mastery and purpose
http://zln.do/ra-video

13. Compliance Innovation
#CassandraSummit

14. #CassandraSummit
STUPS
A Cloud Infrastructure for
Autonomous Teams

15. #CassandraSummit
What Is It?
“The STUPS platform is a set of tools
and components to provide a convenient
and audit-compliant Platform-as-a-Service
(PaaS) for multiple autonomous teams
on top of Amazon Web Services (AWS)”
Learn more at https://stups.io

16. #CassandraSummit
A Platform on Top of AWS
Learn why at https://stups.io/why

17. #CassandraSummit
Relevant Requirements
All STUPS applications are Docker containers
We can only use audit-compliant AMIs
STUPS is dynamic. VPC is DHCP only

18. #CassandraSummit
Challenges Ahead

19. #CassandraSummit
Docker Container
Plenty of options out there
None was created specifically for STUPS
They should be immutable

20. #CassandraSummit
Audit-Compliant AMI
We can’t use DataStax’s AMIs
Cloud Engineering team publishes
audit-compliant AMIs - Taupage
Taupage has its own bootstrapping process

21. #CassandraSummit
Dynamic Environment
Clusters require special nodes - Seeds
EC2 instances get a random IP address
EC2 instances come and go

22. #CassandraSummit
How can I help you?

23. #CassandraSummit
I’d like to bootstrap Cassandra

24. #CassandraSummit
YADC
Yet Another Docker Cassandra
• Kept the best options from the other
recipes out there
• Added STUPS specific options

25. #CassandraSummit
and then…?

26. #CassandraSummit
Run Container in Taupage
With this Cassandra container
we should be ready to run in our
audit-compliant AMI - Taupage

27. #CassandraSummit
You wish!

28. #CassandraSummit
I’d like to discover seed nodes dynamically

29. #CassandraSummit
Discovery
STUPS makes use of etcd to register
EC2 instances
Applications can use etcd for discovery
and distributed locking

30. #CassandraSummit
and then…?

31. #CassandraSummit
Custom SeedProvider
We created the EtcdSeedProvider
seed_provider:
- class_name: org.zalando.cassandra.locator.EtcdSeedProvider
parameters:
- url: http://example.org/v2/keys/cassandra/my-cluster-name/seeds
Dynamically updates list of seed nodes
from etcd

32. #CassandraSummit
and then…?

33. #CassandraSummit
Senza Template
We use the STUPS senza tool
to instrument AWS’ CloudFormation
Bootstraps a Cassandra Cluster
with one simple command
$ senza create https://goo.gl/qyTy7b
my-new-cluster
etcd.example.org

34. #CassandraSummit
STUPS Appliance
CloudFormation
Cluster Details
Name: my-new-cluster
Size: 3
etcd.example.org

35. #CassandraSummit
STUPS Appliance
CloudFormation
Cluster Details
Name: my-new-cluster
Size: 3
etcd.example.org
Auto Scaling Group
Min, Max, Desired = 3

36. #CassandraSummit
STUPS Appliance
CloudFormation
Cluster Details
Name: my-new-cluster
Size: 3
etcd.example.org
Auto Scaling Group
Min, Max, Desired = 3
Seed
Node #1

37. #CassandraSummit
STUPS Appliance
CloudFormation
Cluster Details
Name: my-new-cluster
Size: 3
etcd.example.org
Auto Scaling Group
Min, Max, Desired = 3
Seed
Node #1
Node #2

38. #CassandraSummit
STUPS Appliance
CloudFormation
Cluster Details
Name: my-new-cluster
Size: 3
etcd.example.org
Auto Scaling Group
Min, Max, Desired = 3
Seed
Node #1
Node #2
Node #3

39. #CassandraSummit
STUPS OpsCenter
Cluster
discovers and
registers into
OpsCenter
while
bootstrapping

40. #CassandraSummit
What about operations?

41. #CassandraSummit
Appliance Operations
First release focused only
on initial bootstrapping
Some things Just Work™

42. #CassandraSummit
Adding Nodes
Just adjust your Auto Scaling Group
Or trigger CloudWatch ScaleUp actions

43. #CassandraSummit
Useless piece of sh*t

44. #CassandraSummit
Upcoming Features
Better distribution of seed nodes per AZ
Scheduled backups
HTTP Health check
JMX metrics and instrumentation over HTTP
Replacement of dead nodes

45. #CassandraSummit
All of this is open source

46. #CassandraSummit
We Want to Hear From You
https://github.com/zalando/stups-cassandra
https://github.com/zalando/stups-opscenter
https://github.com/zalando/cassandra-etcd-seed-provider
Try it out
Help us find issues
Send us pull requests

47. Thank you for listening
Check out our blog https://tech.zalando.com
Our many open source products https://github.com/zalando
The STUPS stack https://stups.io
Got more questions? You can reach us on twitter
@ZalandoTech
We’re hiring!
Special thanks to Jessie Dude.
No Continuum Transfunctioners were harmed during the production of these slides.