Video URL: https://youtu.be/GjI6MUz7AyE This is the slide deck of the Percona Live Online 2020 talk given by me in May 2020: https://www.percona.com/resources/videos/orchestrating-cassandra-kubernetes-operator-and-yelp-paasta-percona-live-online The talk delves into the architecture of our Cassandra Kubernetes Operator and the multi-region multi-AZ clusters it manages, and strategies we have in place for safe rollouts and zero-downtime migration.

1. Orchestrating Cassandra
with Kubernetes Operator &
PaaSTA
Percona Live Online
May 19, 2020

2. About
● Me: Raghavendra D Prabhu
○ rprabhu@yelp.com / t: @randomsurfer / me@rdprabhu.com
○ Snr Software Engineer, Yelp
● Team: Database Reliability Engineering (DRE)
● Where: London, UK & San Francisco, CA
● What: Databases at Yelp: MySQL, Cassandra, ZooKeeper

3. Yelp’s Mission
Connecting people with great
local businesses.

4. Overview
● Introduction
● Cassandra at Yelp
● Orchestration
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

5. Overview
● Introduction
● Cassandra at Yelp
● Orchestration
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

6. “ A distributed system is one in which the failure of a computer
you didn't even know existed can render your own computer
unusable. ”
- Leslie Lamport

7. Desired Traits of Distributed
Systems
Reliability
Scalability
Maintainability

8. Distributed Systems Fallacies
● The network is reliable
● Latency is zero
● Bandwidth is infinite
● The network is secure
● Topology doesn't change
● One administrator
● Zero Transport cost
● Homogeneous network

9. Overview
● Introduction
● Cassandra at Yelp
● Orchestration
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

11. ● Distributed wide-column NoSQL datastore
● Leaderless / Multi-master
● Multi-region support
● Tunable consistency
● Write optimized
● Cloud-aware: gossip, failure detection, topology-aware, handoffs

12. C* at Yelp
● Both primary and derived data
● Use cases
● Deployed on Amazon Web Services (AWS)
○ EBS for Storage
● Smartstack for service discovery
● Automated schema management
● ZooKeeper-based cluster coordination

13. us-west-2 us-east-1
Yelp (EC2) Cassandra @ 100000 ft
Multi-region Cluster

14. Yelp (EC2) Cassandra @ 10000 ft

15. Overview
● Introduction
● Cassandra at Yelp
● Orchestration
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

16. Orchestration

17. Velocity
● Churn
● Feature
● Scale
Safety
● Reliable
● Available
● Maintainable

18. Orchestration
● What is Orchestration
● Why Orchestrate
○ Reliability, Scalability, Maintainability
○ Sustainable management of constantly growing infrastructure
○ Automation of best practices and learnings
● Orchestration + Control Plane = FTW
○ = Cassandra Operator + PaaS + Kubernetes

19. Orchestration
Stack

20. Kubernetes / k8s
● Popular Open Source Container-based orchestration
● Actively developed
● Stateful and stateless applications
● Well-defined building blocks for distributed systems
● Integrates into our PaaS
○ k8s: generic but extensible
● Organizes containers into pods

21. ● Yelp PaaSTA: Stateless and Stateful Microservices on Kubernetes
● Few thousand microservices deployed and growing
● Hundreds of deployments every day
● Handles compute, storage and network abstractions
● Why PaaSTA
○ Uniform interface - deployment, restarts, rollbacks ...
○ Clusterman
○ Spot and statically-reserved fleet
PaaSTA: Kubernetes at Yelp

22. Overview
● Introduction
● Cassandra at Yelp
● Orchestration
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

23. Cassandra
Operator

24. Cassandra Operator
● Introduction
● Specifications
● Reconciliation
● Putting it together

25. C* Operator: Intro
● Developed by DRE team
● Controller Loop for Reconciliation
● Defines a custom resource for k8s
○ Statefulset, Container spec, Storage, Secrets and more
● “Big Red Button”
○ Stop for human takeover

26. C* Operator: Responsibilities
● Creating cluster from specs
● Scaling the cluster up and down
● Safe and Reliable Change Deployments
● Lifecycle Management
● Multi-region coordination
● Credential management
● Balance resource utilization

27. Specifications

28. The Recipe aka the Cluster Spec

29. Cassandra Pod
● What is a pod
● Cassandra container + Sidecars
● Sidecar containers
○ HAcheck for Smartstack
○ Cron
○ Sensu
○ Change Data Capture (CDC) publisher
○ Metrics exporters

30. Storage aka State
● EBS for Cassandra
○ Clear separation of stateful and stateless
○ Quick healing upon underlying node failure
○ Dynamic Provisioning
○ “Compute follows Data”
○ Stripe cluster across AZs

31. Reconciliation

37. Putting it Together

38. Big
Picture

39. us-west-2
us-east-1
Yelp Cassandra @ 100000 ft + Operator
Multi-region Cluster

42. Overview
● Introduction
● Cassandra at Yelp
● Orchestration
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

43. Opportunities & Challenges

44. Opportunities

45. Cost Accounting

46. Autoscaling
0
1x
2x
3x
Wed Thu Fri Sat Sun Mon Tue Wed Thu Fri Sat Sun
Holiday
Requests

47. Security

48. Challenges

49. Sidecars

50. Isolation

51. Deployment Strategies
Dealing with
Immutability

52. Overview
● Introduction
● Cassandra at Yelp
● Orchestration
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

53. Distributed Systems Fallacies
The network is reliable
Latency is zero
Bandwidth is infinite
The network is secure
Topology doesn't change
One administrator
Zero Transport cost
Homogeneous network

54. Orchestration:
Building A Reliable System
Out of Unreliable Components.

55. Q & A

56. @YelpEngineering
fb.com/YelpEngineers
engineeringblog.yelp.com
github.com/yelp

57. Credits
● Apache cassandra logo
● https://puppet.com/
● https://www.terraform.io/
● https://kubernetes.io/
● https://etcd.io/
● https://aws.amazon.com/architecture/icons/
● https://dataintensive.net/
● https://www.yelp.com/brand
● https://thenounproject.com/
● https://upload.wikimedia.org/wikipedia/commons/2/2e/Chicago_Symphony
_Orchestra_2005.jpg
● https://upload.wikimedia.org/wikipedia/commons/thumb/e/e0/Blank_US_m
ap_borders_labels.svg/1000px-Blank_US_map_borders_labels.svg.png