This talk was given at Cassandra London meetup: https://www.meetup.com/Cassandra-London/events/267271963/ . The talk is about orchestration of Cassandra with our Kubernetes Operator and Yelp PaaSTA. We also outline some of the opportunities and challenges associated with this architecture. Youtube link: https://www.youtube.com/watch?v=JqAILFkkibA

1. Orchestrating Cassandra
with Kubernetes
Challenges and Opportunities
Raghavendra Prabhu, Software Engineer, Yelp
Cassandra Meetup, Jan 2020

2. About
● Me: Raghavendra D Prabhu
○ rprabhu@yelp.com / @randomsurfer / me@rdprabhu.com
○ Snr Software Engineer, Yelp
● Team: Database and Reliability Engineering (DRE)
● Where: London, UK & San Francisco, CA
● What: Databases at Yelp: MySQL, Cassandra, ZooKeeper

3. Yelp’s Mission
Connecting people with great
local businesses.

4. Overview
● Introduction
● Cassandra and Cassandra at Yelp
● Orchestration
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

5. Overview
● Introduction
● Cassandra and Cassandra at Yelp
● Orchestration and Abstractions
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

6. Desired Traits of
Distributed Systems
● Reliability
● Scalability
● Maintainability

7. Distributed systems fallacies
● The network is reliable
● Latency is zero
● Bandwidth is infinite
● The network is secure
● Topology doesn't change
● One administrator
● Zero Transport cost
● Homogeneous network

8. Overview
● Introduction
● Cassandra and Cassandra at Yelp
● Orchestration and Abstractions
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

10. Powered by Cassandra
Your Reviews
1
2
1
Waitlist for diners2

11. Cassandra (C*)
● Distributed wide-column NoSQL datastore
● Leaderless / Multi-master
● Written in Java
● Multi-region
● Tunable consistency
● Write optimized
● Cloud-aware: gossip, failure detection, topology-aware, handoffs

12. C* at Yelp
● Both primary and derived data
● Use cases
● Deployed on Amazon Web Services (AWS)
● Network-attached storage
● Smartstack-based seed provider
● Backups into S3
● ZooKeeper-based cluster coordination

13. us-west-2
us-west-1
us-east-1
Yelp Cassandra @ 100000 ft
Multi-region Cluster

14. Yelp (EC2) Cassandra @ 10000 ft

15. Overview
● Introduction
● Cassandra and Cassandra at Yelp
● Orchestration
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

16. Orchestration

17. Velocity
● Churn
● Feature
● Scale
Safety
● Reliable
● Consistent
● Available

18. Orchestration
● What is Orchestration
● Why Orchestrate
○ Reliability, Scalability, Maintainability
● Clean abstractions
● Extended Control Plane
○ Helps us sleep well ;)
● C* Operator + PaaS + Kubernetes

20. Orchestrator

21. Orchestrator

22. Kubernetes / k8s
● Popular Open Source Container-based orchestration
● Actively developed
● Organizes containers into pods
● Stateful and stateless applications
● Well-defined building blocks for distributed systems
● Integrates into our PaaS
○ k8s: generic but extensible

23. Orchestrator

24. ● Yelp PaaSTA: Microservices and Stateful Clusters
● Few thousand microservices deployed and growing
● Hundreds of deployments every day
● Handles compute, storage and network abstractions
● Why PaaSTA
○ Clusterman
○ Spot and statically-reserved fleet
○ Consistent interface
PaaSTA: Kubernetes/Mesos at Yelp

25. Overview
● Introduction
● Cassandra and Cassandra at Yelp
● Orchestration and Abstractions
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

26. Orchestrator

27. Cassandra
Operator

28. C* Operator: Intro
● Developed by DRE team
● Defines a custom resource for k8s
○ Statefulset, Container spec, Storage and more
● operator-sdk and controller-runtime
○ Ideal for non-trivial clustered services
● Extended control plane of our C* clusters
● Written in Golang and deployed in k8s itself

29. C* Operator: Responsibilities
● Reconciling cluster from specs
● Scaling the cluster up and down
● Safe and Reliable Change Deployments
● Lifecycle Management
● Multi-region coordination
● Credential management
● Optimizing resource usage

30. The Recipe aka the Cluster Spec

31. us-west-2
us-west-1
us-east-1
Yelp Cassandra @ 100000 ft + Operator
Multi-region Cluster

32. us-west-2
us-west-1
us-east-1
Exclusive Leases

33. Where we
are heading

34. Overview
● Introduction
● Cassandra and Cassandra at Yelp
● Orchestration and Abstractions
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

35. Opportunities &
Challenges

36. Opportunities

37. Toil Avoidance

38. Lifecycle Management

39. Agile Deployment

40. Autoscaling
0
1x
2x
3x
Wed Thu Fri Sat Sun Mon Tue Wed Thu Fri Sat Sun
Thanksgiving /
Holiday
Requests

41. Security

42. Challenges

43. Isolation

44. Performance

45. Config Management

46. Observability

47. Deployment Strategies

48. Overview
● Introduction
● Cassandra and Cassandra at Yelp
● Orchestration and Abstractions
● Cassandra Operator
● Opportunities and Challenges
● Conclusion

49. We're Hiring!
www.yelp.com/careers/
Hiring for multiple full time positions in
the London office

50. Questions?

51. Credits
● Apache cassandra logo
● https://puppet.com/
● https://www.terraform.io/
● https://kubernetes.io/
● https://etcd.io/
● https://xkcd.com/
● https://aws.amazon.com/architecture/icons/
● https://dataintensive.net/
● https://www.yelp.com/brand
● https://thenounproject.com/
● https://upload.wikimedia.org/wikipedia/commons/2/2e/Chicago_Symphony_O
rchestra_2005.jpg
● https://upload.wikimedia.org/wikipedia/commons/thumb/e/e0/Blank_US_map
_borders_labels.svg/1000px-Blank_US_map_borders_labels.svg.png

52. @YelpEngineering
fb.com/YelpEngineers
engineeringblog.yelp.com
github.com/yelp