![Static: humans decide
puppet/chef: role X gets service Y
static mappings: boxes [A,B,C,...] get service Y
simple, reliable
slow to react to failure, resource changes
Scheduling:
Decide where to run the code](https://image.slidesharecdn.com/untitled-151005102027-lva1-app6891/85/Paasta-Application-Delivery-at-Yelp-24-320.jpg)
Uploaded byC4Media
Paasta: Application Delivery at Yelp
The document discusses Yelp's PAAS (Platform as a Service) tool, PAAS-TA, created to improve application delivery through service-oriented architecture, which facilitates faster and more manageable deployments. It highlights the challenges of managing numerous microservices, the processes of scheduling, delivery, discovery, and alerting, and the use of tools like Mesos and Docker for effective orchestration and containerization. The document emphasizes the goals of enhancing reliability, developer satisfaction, and operational efficiency within distributed systems.
More Related Content
![Modern Monitoring [ with Prometheus ]](https://cdn.slidesharecdn.com/ss_thumbnails/modernmonitoringv1-171124143735-thumbnail.jpg?width=640&height=640&fit=bounds)
![[개방형 클라우드 플랫폼 오픈세미나 오픈클라우드 Pub] 4. 종합분석[1]](https://cdn.slidesharecdn.com/ss_thumbnails/pub4-140818080528-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Paasta: Application Delivery at Yelp
Paasta: Application Delivery at Yelp
- 1.
- 2. InfoQ.com: News &Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /paasta-yelp
- 3. Presented at QConNew York www.qconnewyork.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
- 4. Evan Krall SRE =development + sysadmin 4+ years at Yelp Who is this person?
- 6.
- 7.
- 8. History Yelp started outas monolithic python app Builds/pushes take a long time Messing up is painful So we build process to avoid messing up Process makes pushes even slower
- 9. Service Oriented Architecture Pullfeatures out of monolith Split into different applications Smaller services -> faster pushes fewer issues per push total # of issues increases, but we can fix issues faster. Smaller parts -> easier to reason about Bonus: can scale parts independently
- 10. SOA comes withchallenges Lots of services means lots of dependencies Now your code is a ~distributed system~ If you thought running 1 app was hard, try 100
- 11. Standalone application Stateless Separate gitrepo Usually, at Yelp: HTTP API Python, Pyramid, uWSGI virtualenv What is a service?
- 12. services responsible forproviding init script often not idempotent central list of which hosts run which services pull-model file transfers reasonably reliable push-model control (for host in hosts: ssh host ...) What if hosts are down? What if transfer hasn't completed yet? Yelp SOA before Paasta
- 13.
- 14. What is Paasta? Buildsservices Deploys services Interconnects services Monitors services Internal PaaS
- 15. What is Paasta? Deploying servicesshould be better! Continuous integration is awesome! Servers are not snowflakes! Declarative control++ Monitor your services!
- 16.
- 17. Make ops happy Faulttolerance no single points of failure recover from failures Efficient use of resources Simplify adding/removing resources
- 18. Make devs happy Weneed adoption, but can't impose on devs Must be possible to seamlessly port services Must work in both datacenter and AWS Must be compelling to developers Features Documentation Flexibility
- 19. Make ourselves happy Pickgood abstractions Avoid hacks Write good tests Don't reinvent the wheel: use open-source tools Enforce opinions when necessary for scale (paasta devs)
- 20.
- 21. What runs in production? (orstage, or dev, or ...)
- 22. Scheduling: Decide whereto run the code Delivery: Get the code + dependencies onto boxes Discovery: Tell clients where to connect Alerting: Tell humans when things are wrong What parts do we need?
- 23. Scheduling: Decide whereto run the code Delivery: Get the code + dependencies onto boxes Discovery: Tell clients where to connect Alerting: Tell humans when things are wrong What parts do we need?
- 24. Static: humans decide puppet/chef:role X gets service Y static mappings: boxes [A,B,C,...] get service Y simple, reliable slow to react to failure, resource changes Scheduling: Decide where to run the code
- 25. Dynamic: computers decide Mesos,Kubernetes, Fleet, Swarm IaaS: dedicated VMs for service, let Amazon figure it out. Automates around failure, resource changes Makes discovery/delivery/monitoring harder Scheduling: Decide where to run the code
- 26. Scheduling in Paasta: Mesos+ Marathon Mesos is an "SDK for distributed systems", not batteries-included. Requires a framework Marathon (ASGs for Mesos) Can run many frameworks on same cluster Supports Docker as task executor mesosphere.io mesos.apache.org
- 27.
- 28.
- 29.
- 30. Scheduling: Decide whereto run the code Delivery: Get the code + dependencies onto boxes Discovery: Tell clients where to connect Alerting: Tell humans when things are wrong What parts do we need?
- 31. Push-based: • for boxin $boxes; do rsync code $box:/code Simple, easy to tell when finished what about failures? retry, but how long? how do we make sure new boxes get code? cron deploys Delivery: Get the code + dependencies onto boxes
- 32. Delivery: Pull-based: cron job onevery box downloads code built-in retries new boxes download soon after boot have to wait for cron job baked VM/container images container/VM can't start on failure ASG, Marathon will retry Get the code + dependencies onto boxes
- 33. Shared sudo {gem,pip,apt-get} install lotsof tooling exists already shared = space/bandwidth savings what if two services need different versions? how to update a library that 20 services need? Delivery: Get the code + dependencies onto boxes
- 34. Isolated virtualenv / rbenv/ VM-per-service / Docker more freedom for dev services don't step on each others' toes more disk/bandwidth harder to audit for vulnerabilities Delivery: Get the code + dependencies onto boxes
- 35. Delivery in Paasta: Docker Containers:like lightweight VMs Provides a language (Dockerfile) for describing container image Reproducible builds (mostly) Provides software flexibility docker.com
- 36. Scheduling: Decide whereto run the code Delivery: Get the code + dependencies onto boxes Discovery: Tell clients where to connect Alerting: Tell humans when things are wrong What parts do we need?
- 37. Static: Constants in code Configfiles Static records in DNS Simple, reliable Slow reaction time Discovery: Tell clients where to connect
- 38. Dynamic: Dynamic DNS zone ELB Zookeeper,Etcd, Consul Store IPs in database, not text files Reacts to change faster, allows better scheduling Complex, can be fragile Recursive: How do you know where ZK is? Discovery: Tell clients where to connect
- 39. in-process DNS Everyone supports DNS TTLsare rarely respected, limit update rate Lookups add critical-path latency Talking to ZK, Etcd, Consul in service Tricky. Risk of worker lockup if ZK hangs Delegate to library Few external dependencies Discovery: Tell clients where to connect
- 41. external SmartStack, consul-template, vulcand Reverseproxy on local box Simpler client code (just hit localhost:$port) Avoids library headaches Easy cross-language support Must be load-balanceable Discovery: Tell clients where to connect
- 42. Nerve registers servicesin ZooKeeper Synapse discovers from ZK + writes HAProxy config Registration, discovery, load balancing Hard problems! Let's solve them once. Provides migration path: port legacy version to SmartStack have Paasta version register in same pool Discovery in Paasta: Smartstack nerds.airbnb.com/smartstack-service-discovery-cloud
- 43.
- 44. why bother withregistration? why not ask your scheduler? Scheduler portability!
- 45.
- 46. Every box runsHAProxy Paper over network issues with retries Load balancing scales with # of clients Downside: lots of healthchecks hacheck caches to avoid hammering services Downside: many LBs means LB algorithms don't work as well There's no place like 127.0.0.1* *We actually use 169.254.255.254, because every container has its own 127.0.0.1
- 47. Scheduling: Decide whereto run the code Delivery: Get the code + dependencies onto boxes Discovery: Tell clients where to connect Alerting: Tell humans when things are wrong What parts do we need?
- 48. Alerting: Static E.g. nagios, icinga File-basedconfiguration Simple, familiar Often not highly available Hard to dynamically generate checks/alerts Tell humans when things are wrong
- 49. Dynamic e.g. Sensu, Consul Allowsyou to add hosts & checks on the fly Flexible Generally newer, less battle-tested Newer software is often built for high availability Alerting: Tell humans when things are wrong
- 50. Based around eventbus Replication monitoring how many instances are up in HAProxy? Marathon app monitoring is service failing to start? Cron jobs on master boxes do checks, emit results. Alerting in Paasta: Sensu sensuapp.org
- 51. Runtime Components Scheduling: Mesos+Marathon Delivery:Docker Discovery: SmartStack Alerting: Sensu
- 52. How do we controlthis thing?
- 53. Primary control plane Convenientaccess controls (via gitolite, etc) deploys, stop/start/restart indicated by tags less-frequently changed metadata stored in a repo
- 54. Declarative control Describe endgoal, not path Helps us achieve fault tolerance. "Deploy 12abcd34 to prod" vs. "Commit 12abcd34 should be running in prod" Gas pedal vs. Cruise Control
- 55. editable by serviceauthors marathon-$cluster.yaml how many instances to run? canary secondary tasks smartstack.yaml deploy.yaml list of deploy steps Boilerplate can be generated with paasta fsm metadata repo
- 56. python 2.7 package,dh-virtualenv CLI for users control + visibility cron jobs: Collect information from git Configure Marathon Configure Nerve Resilient to failure This is how we build higher-order systems paasta_tools
- 57. Bounce strategies up-then-down: wait fornew version to start; kill old down-then-up wait for old version to die; start new crossover as instances of new version start, kill old instances
- 58. Builds Docker images Pushesto Docker registry Marks image for deployment GUI configuration is a Bad Idea, so we automate it (deploy.yaml) Most build steps call paasta command Jenkins
- 59. Multi- Environment Region Datacenter habitat AZ orcage 0.3ms region AWS region, nearby cages 1ms superregion nearby regions <5ms ecosystem copy of site (dev/stage/prod) Mesos cluster per superregion Services choose at which level SmartStack works
- 60. Superregion Superregion Superregion Datacenter Datacenter AWS Region AWSRegion AWS Region (region) (region) (region) (region) (region)
- 61. Walkthrough Let's deploy aservice
- 70.
- 74.
- 75. Watch the videowith slide synchronization on InfoQ.com! http://www.infoq.com/presentations/paasta- yelp