This is a talk about orchestration of Cassandra with cassandra operator, kubernetes and Yelp PaaSTA (https://github.com/Yelp/paasta). The talk was presented at Computer Laboratory, University of Cambridge as part of the Engineering, Science and Technology Event (https://www.careers.cam.ac.uk/recruiting/event2Tech.asp) in November 2019.

1. Orchestrating Cassandra
with Kubernetes
Challenges and Opportunities
Raghavendra Prabhu, Software Engineer, Yelp
University of Cambridge Engineering Event

2. About
● Me: Raghavendra D Prabhu
○ rprabhu@yelp.com / @randomsurfer / me@rdprabhu.com
○ Senior Software Engineer, Yelp
● Team: Database and Reliability Engineering (DRE)
● Where: London, UK & San Francisco, CA
● What: Databases at Yelp: MySQL, Cassandra, ZooKeeper

3. Yelp’s Mission
Connecting people with great
local businesses.

4. Overview
● Introduction
● Cassandra and Cassandra at Yelp
● Orchestration
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

5. Overview
● Introduction
● Cassandra and Cassandra at Yelp
● Orchestration and Abstractions
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

6. “ A distributed system is one in which the failure of
a computer you didn't even know existed can
render your own computer unusable. ”
- Leslie Lamport

7. Desired Traits of
Distributed Systems
● Reliability
● Scalability
● Maintainability

8. Distributed systems fallacies
● The network is reliable
● Latency is zero
● Bandwidth is infinite
● The network is secure
● Topology doesn't change
● One administrator
● Zero Transport cost
● Homogeneous network

9. Overview
● Introduction
● Cassandra and Cassandra at Yelp
● Orchestration and Abstractions
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

11. Powered by Cassandra
Your Reviews
1
2
1
Waitlist for diners2

12. Cassandra (C*)
● Distributed wide-column NoSQL datastore
● Leaderless / Multi-master
● Written in Java
● Multi-region
● Tunable consistency
● Write optimized
● Cloud-aware: gossip, failure detection, topology-aware, handoffs

13. C* at Yelp
● Both primary and derived data
● Use cases
● Deployed on Amazon Web Services (AWS)
● Network-attached storage
● Automated schema management
● Backups into S3
● ZooKeeper-based cluster coordination

14. us-west-2
us-west-1
us-east-1
Yelp Cassandra @ 100000 ft
Multi-region Cluster

15. Yelp (EC2) Cassandra @ 10000 ft

16. Overview
● Introduction
● Cassandra and Cassandra at Yelp
● Orchestration
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

17. Orchestration

18. Pets vs Cattle

19. Velocity
● Churn
● Feature
● Scale
Safety
● Reliable
● Consistent
● Available

20. Orchestration
● What is Orchestration
● Why Orchestrate
○ Reliability, Scalability, Maintainability
● Clean abstractions
● Extended Control Plane
○ Helps us sleep well ;)
● C* Operator + PaaS + Kubernetes

22. Orchestrator

23. Orchestrator

24. Kubernetes / k8s
● Popular Open Source Container-based orchestration
● Actively developed
● Organizes containers into pods
● Stateful and stateless applications
● Well-defined building blocks for distributed systems
● Integrates into our PaaS
○ k8s: generic but extensible

25. Orchestrator

26. ● Yelp PaaSTA: Stateless Microservices on Mesos
○ Clusters on Kubernetes
● Few thousand microservices deployed and growing
● Hundreds of deployments every day
● Handles compute, storage and network abstractions
● Why PaaSTA
○ Clusterman
○ Spot and statically-reserved fleet
PaaSTA: Kubernetes/Mesos at Yelp

27. Overview
● Introduction
● Cassandra and Cassandra at Yelp
● Orchestration and Abstractions
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

28. Orchestrator

29. Cassandra
Operator

30. C* Operator: Intro
● Developed by DRE team
● Defines a custom resource for k8s
○ Statefulset, Container spec, Storage and more
● operator-sdk and controller-runtime
○ Ideal for non-trivial clustered services
● Extended control plane of our C* clusters
● Written in Golang and deployed in k8s itself

31. C* Operator: Responsibilities
● Creating cluster from specs
● Scaling the cluster up and down
● Safe and Reliable Change Deployments
● Lifecycle Management
● Multi-region coordination
● Credential management
● Optimizing resource usage

32. us-west-2
us-west-1
us-east-1
Yelp Cassandra @ 100000 ft + Operator
Multi-region Cluster

33. us-west-2
us-west-1
us-east-1
Exclusive Leases

34. Where we
are heading

35. Overview
● Introduction
● Cassandra and Cassandra at Yelp
● Orchestration and Abstractions
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

36. Opportunities &
Challenges

37. Opportunities

38. Toil Avoidance

39. Lifecycle Management

40. Agile Deployment

41. Autoscaling
0
1x
2x
3x
Wed Thu Fri Sat Sun Mon Tue Wed Thu Fri Sat Sun
Thanksgiving /
Holiday
Requests

42. Security

43. Challenges

44. Isolation

45. Performance

46. Config Management

47. Observability

48. Deployment Strategies

49. Overview
● Introduction
● Cassandra and Cassandra at Yelp
● Orchestration and Abstractions
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

50. Distributed systems fallacies
● The network is reliable
● Latency is zero
● Bandwidth is infinite
● The network is secure
● Topology doesn't change
● One administrator
● Zero Transport cost
● Homogeneous network

51. Distributed Systems:
Building Reliable Systems
Out of Unreliable Components

52. We're Hiring!
www.yelp.com/careers/
Software Engineering Interns:
London & Hamburg

53. Questions?

54. Credits
● Apache cassandra logo
● https://puppet.com/
● https://www.terraform.io/
● https://kubernetes.io/
● https://etcd.io/
● https://xkcd.com/
● https://aws.amazon.com/architecture/icons/
● https://dataintensive.net/
● https://www.yelp.com/brand
● https://thenounproject.com/
● https://upload.wikimedia.org/wikipedia/commons/2/2e/Chicago_Symphony_O
rchestra_2005.jpg
● https://upload.wikimedia.org/wikipedia/commons/thumb/e/e0/Blank_US_map
_borders_labels.svg/1000px-Blank_US_map_borders_labels.svg.png

55. @YelpEngineering
fb.com/YelpEngineers
engineeringblog.yelp.com
github.com/yelp