The document discusses the orchestration of Cassandra with Kubernetes, highlighting challenges and opportunities faced by Yelp's database and reliability engineering team. It covers various aspects such as the desired traits of distributed systems, the use of Cassandra at Yelp, and the development of a Cassandra operator for Kubernetes to manage and optimize clusters. Overall, it emphasizes the importance of reliability, scalability, maintainability, and addresses lifecycle management and deployment strategies in distributed systems.

1. Orchestrating Cassandra
with Kubernetes
Challenges and Opportunities
Raghavendra Prabhu, Software Engineer, Yelp
University of Cambridge Engineering Event

2. About
● Me: Raghavendra D Prabhu
○ rprabhu@yelp.com / @randomsurfer / me@rdprabhu.com
○ Senior Software Engineer, Yelp
● Team: Database and Reliability Engineering (DRE)
● Where: London, UK & San Francisco, CA
● What: Databases at Yelp: MySQL, Cassandra, ZooKeeper

3. Yelp’s Mission
Connecting people with great
local businesses.

4. Overview
● Introduction
● Cassandra and Cassandra at Yelp
● Orchestration
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

5. Overview
● Introduction
● Cassandra and Cassandra at Yelp
● Orchestration and Abstractions
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

6. “ A distributed system is one in which the failure of
a computer you didn't even know existed can
render your own computer unusable. ”
- Leslie Lamport

7. Desired Traits of
Distributed Systems
● Reliability
● Scalability
● Maintainability

8. Distributed systems fallacies
● The network is reliable
● Latency is zero
● Bandwidth is infinite
● The network is secure
● Topology doesn't change
● One administrator
● Zero Transport cost
● Homogeneous network

9. Overview
● Introduction
● Cassandra and Cassandra at Yelp
● Orchestration and Abstractions
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

11. Powered by Cassandra
Your Reviews
1
2
1
Waitlist for diners2

12. Cassandra (C*)
● Distributed wide-column NoSQL datastore
● Leaderless / Multi-master
● Written in Java
● Multi-region
● Tunable consistency
● Write optimized
● Cloud-aware: gossip, failure detection, topology-aware, handoffs

13. C* at Yelp
● Both primary and derived data
● Use cases
● Deployed on Amazon Web Services (AWS)
● Network-attached storage
● Automated schema management
● Backups into S3
● ZooKeeper-based cluster coordination

14. us-west-2
us-west-1
us-east-1
Yelp Cassandra @ 100000 ft
Multi-region Cluster

15. Yelp (EC2) Cassandra @ 10000 ft

16. Overview
● Introduction
● Cassandra and Cassandra at Yelp
● Orchestration
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

17. Orchestration

18. Pets vs Cattle

19. Velocity
● Churn
● Feature
● Scale
Safety
● Reliable
● Consistent
● Available

20. Orchestration
● What is Orchestration
● Why Orchestrate
○ Reliability, Scalability, Maintainability
● Clean abstractions
● Extended Control Plane
○ Helps us sleep well ;)
● C* Operator + PaaS + Kubernetes

22. Orchestrator

23. Orchestrator

24. Kubernetes / k8s
● Popular Open Source Container-based orchestration
● Actively developed
● Organizes containers into pods
● Stateful and stateless applications
● Well-defined building blocks for distributed systems
● Integrates into our PaaS
○ k8s: generic but extensible

25. Orchestrator

26. ● Yelp PaaSTA: Stateless Microservices on Mesos
○ Clusters on Kubernetes
● Few thousand microservices deployed and growing
● Hundreds of deployments every day
● Handles compute, storage and network abstractions
● Why PaaSTA
○ Clusterman
○ Spot and statically-reserved fleet
PaaSTA: Kubernetes/Mesos at Yelp

27. Overview
● Introduction
● Cassandra and Cassandra at Yelp
● Orchestration and Abstractions
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

28. Orchestrator

29. Cassandra
Operator

30. C* Operator: Intro
● Developed by DRE team
● Defines a custom resource for k8s
○ Statefulset, Container spec, Storage and more
● operator-sdk and controller-runtime
○ Ideal for non-trivial clustered services
● Extended control plane of our C* clusters
● Written in Golang and deployed in k8s itself

31. C* Operator: Responsibilities
● Creating cluster from specs
● Scaling the cluster up and down
● Safe and Reliable Change Deployments
● Lifecycle Management
● Multi-region coordination
● Credential management
● Optimizing resource usage

32. us-west-2
us-west-1
us-east-1
Yelp Cassandra @ 100000 ft + Operator
Multi-region Cluster

33. us-west-2
us-west-1
us-east-1
Exclusive Leases

34. Where we
are heading

35. Overview
● Introduction
● Cassandra and Cassandra at Yelp
● Orchestration and Abstractions
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

36. Opportunities &
Challenges

37. Opportunities

38. Toil Avoidance

39. Lifecycle Management

40. Agile Deployment

41. Autoscaling
0
1x
2x
3x
Wed Thu Fri Sat Sun Mon Tue Wed Thu Fri Sat Sun
Thanksgiving /
Holiday
Requests

42. Security

43. Challenges

44. Isolation

45. Performance

46. Config Management

47. Observability

48. Deployment Strategies

49. Overview
● Introduction
● Cassandra and Cassandra at Yelp
● Orchestration and Abstractions
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

50. Distributed systems fallacies
● The network is reliable
● Latency is zero
● Bandwidth is infinite
● The network is secure
● Topology doesn't change
● One administrator
● Zero Transport cost
● Homogeneous network

51. Distributed Systems:
Building Reliable Systems
Out of Unreliable Components

52. We're Hiring!
www.yelp.com/careers/
Software Engineering Interns:
London & Hamburg

53. Questions?

54. Credits
● Apache cassandra logo
● https://puppet.com/
● https://www.terraform.io/
● https://kubernetes.io/
● https://etcd.io/
● https://xkcd.com/
● https://aws.amazon.com/architecture/icons/
● https://dataintensive.net/
● https://www.yelp.com/brand
● https://thenounproject.com/
● https://upload.wikimedia.org/wikipedia/commons/2/2e/Chicago_Symphony_O
rchestra_2005.jpg
● https://upload.wikimedia.org/wikipedia/commons/thumb/e/e0/Blank_US_map
_borders_labels.svg/1000px-Blank_US_map_borders_labels.svg.png

55. @YelpEngineering
fb.com/YelpEngineers
engineeringblog.yelp.com
github.com/yelp