Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

PuppetConf 2016: Automating Datastore Fleets with Puppet – Joseph Lynch, Yelp

127 views

Published on

Here are the slides from Joseph Lynch's PuppetConf 2016 presentation called Automating Datastore Fleets with Puppet. Watch the videos at https://www.youtube.com/playlist?list=PLV86BgbREluVjwwt-9UL8u2Uy8xnzpIqa

Published in: Technology
  • Be the first to comment

  • Be the first to like this

PuppetConf 2016: Automating Datastore Fleets with Puppet – Joseph Lynch, Yelp

  1. 1. Joey Lynch jlynch@yelp.com Data Shepherd Automating Datastore Fleets with Puppet
  2. 2. Yelp’s Mission Connecting people with great local businesses.
  3. 3. Yelp Stats As of Q2 2016 92M 3272%108M
  4. 4. jlynch Stats As of Q2 2016 Yelp 4+ years Datastore Whisperer Distributed Systems Addict
  5. 5. Yelp Polyglot Stack Presentation Logic Data
  6. 6. But Why?
  7. 7. Logic Web Search Ads API Logging Svc 5 Platform Data MySQL Lucene Cass ... Kafka DB 5 Zookeeper ... ...
  8. 8. Logic Web Search Ads API Logging Svc 5 Platform Data MySQL Lucene Cass ... Kafka DB 5 Zookeeper ... ... It’s Complicated
  9. 9. MUST support ● Different datastores ● Different configurations ● Cloud and datacenter ready ● Ready for production ● Cope with Data Gravity Yelp’s Datastore Problem
  10. 10. Yelp’s Datastore Solution
  11. 11. Different Datastores
  12. 12. Cluster Modules
  13. 13. Wrap the vendor modules Manage cluster wide config
  14. 14. Different Configurations
  15. 15. Configure ● Functions help ● Hierarchical config is even better
  16. 16. Configure - Hierarchy
  17. 17. Configure - Hierarchy
  18. 18. Configure - Hierarchy
  19. 19. Configure - Adding a Cluster
  20. 20. Configure - Adding a Cluster
  21. 21. Configure - Flexible
  22. 22. Configure - Flexible
  23. 23. Configure - Flexible
  24. 24. Functions ● No hard coding! ● Changes with infra ● Sometimes as easy as $::processorcount
  25. 25. Cloud and Datacenter Ready
  26. 26. Datacenter
  27. 27. Datacenter
  28. 28. Cloud
  29. 29. Cloud
  30. 30. ENC is very nice
  31. 31. ENC is very nice
  32. 32. Production Ready
  33. 33. Using Puppet to Manage Datastores ● About 100 clusters ● Over 500 datastore nodes ● Puppet runs take ~5 minutes ● Run Puppet every hour, CI/CD ○ Automation runs via separate scheduler
  34. 34. Monitor
  35. 35. Monitor Clusters
  36. 36. Monitor Clusters https://engineeringblog.yelp.com/2016/06/monitoring-cassandra-at-scale.html
  37. 37. Monitor Clusters https://engineeringblog.yelp.com/2016/06/monitoring-cassandra-at-scale.html Totally fine maybe BAD
  38. 38. cron
  39. 39. KISS
  40. 40. DatabaseClient configure_nerveconfigure_synapse JSON files hacheck ZookeeperSynapse Nerve Healthchecks ConfigurationConfiguration Configuration Healthchecks Registration Service Discovery
  41. 41. Service Discovery
  42. 42. Service Discovery DatabaseClient configure_nerveconfigure_synapse JSON files hacheck ZookeeperSynapse Nerve Healthchecks ConfigurationConfiguration Configuration Healthchecks Registration
  43. 43. Service Discovery
  44. 44. Data Gravity
  45. 45. Autoscale!
  46. 46. Autoscale! ● Debian package ● Cron
  47. 47. Backup! ● Another Debian package ● Cron
  48. 48. Recap
  49. 49. Using Puppet to Manage Datastores
  50. 50. Using Puppet to Manage Datastores cron MonitorDiscoverAutomate Puppet enables automation
  51. 51. @YelpEngineering fb.com/YelpEngineers engineeringblog.yelp.com github.com/yelp
  52. 52. ● Server rack, unknown author http://www.clipartkid.com/server-rack-cabinet-clip-art-at-clker-com-vector-clip-art-online-IPTtWc-clipart/ ● Devops venn diagram, Kharnagy https://commons.wikimedia.org/wiki/File:Devops.svg ● Steve Jobs, waldryno https://pixabay.com/en/steve-jobs-technology-illustration-1249665/ Image Citations

×