Uploaded byAriel Tseitlin
Accelerating Innovation and Time-to-Market @ Camp Devops Houston 2015
This document discusses how DevOps practices can help organizations accelerate innovation through software delivery. It outlines the core components of DevOps as self-service, automation, and collaboration. It then walks through the software development lifecycle from developing to operating software. Throughout, it emphasizes practices like continuous delivery, automated testing and configuration, zero-downtime deployments, monitoring, and designing systems to withstand failures using techniques like Netflix's "Chaos Monkey" which intentionally causes failures to test resiliency. The overall message is that DevOps can help organizations innovate faster by breaking down silos and automating the process of delivering high quality software.
More Related Content
Similar to Accelerating Innovation and Time-to-Market @ Camp Devops Houston 2015
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
Accelerating Innovation and Time-to-Market @ Camp Devops Houston 2015
- 1.
- 2.
- 3.
- 4. 4@atseitlin Everything-as-a-Service Alphabet soup • IaaS •PaaS • DaaS • mBaaS • SBS • … All started with SaaS With great power comes great responsibility (Consumer web companies figured this out long ago)
- 5. 5@atseitlin Competition Software is aat the core of every business, not just software companies Ability to deliver software to market quickly is a competitive advantage
- 6. 6@atseitlin DevOps Accelerating Pace ofInnovation Rate of Innovation Quality Cloud
- 7. 7@atseitlin DevOps Core Components 1.Self-service 2. Automation 3. Collaboration
- 8.
- 9. 9@atseitlin Develop Self-provisioning & self-service Operationalplatform that provides - Load balancing - Service discovery - Logging, metrics, alerting, and notifications - Resiliency design patterns (bulk heads & fallbacks) - Security - Data access (caching, persistence)
- 10. 10@atseitlin Test Unit / functional/ regression / integration Chaos (failure injection / simulation) Performance / Squeeze Security Automate as much as possible (all!)
- 11. 11@atseitlin Deploy Continuous Delivery Automated configuration Zero-downtime(red/black or rolling) Canary Analysis Redundancy
- 12. 12@atseitlin Deploy: Continuous Delivery Replacemanual release process with automation Assembly line
- 13. 13@atseitlin Netflix API ContinuousDelivery pipeline * Sangeeta Narayanan, Move Fast;Stay Safe:Developing & Deploying the Netflix API http://www.slideshare.net/SangeetaNarayanan/move-faststay-safedeveloping-deploying- the-netflix-api
- 14. 14@atseitlin Deploy: Automated Configuration “ServersAren’t Pets; They’re Cattle” • The old: curated, hand-created infrastructure • The new: disposable, elastic, self-configuring
- 15. 15@atseitlin Deploy: Automated Configuration Twoapproaches 1. Bake virtual images or containers 2. Bootstrap at launch
- 16. 16@atseitlin Deploy: Zero downtime Twooptions 1. Rolling upgrade 2. Red/black deployment Must be able to rollback quickly!
- 17. 17@atseitlin Deploy: Canary Analysis Always,always stage deployments and compare them to baseline before fully deploying Launch 2 instances, one with new code one with old. Compare them. If good, continue deploying new Lots (183 slides!) of detailed tips & tricks from QConNY at: http://www.slideshare.net/royrapoport/20140612-q-con- canary-analysis
- 18. 18@atseitlin Deploy: Redundancy Think in3s Multiple data centers (AZs) Multiple regions Active-active
- 19.
- 20. 20@atseitlin Operate: Monitoring Collect everything Preferstructured time series over log data Build base health metrics into the platform (e.g. 2xx, 3xx, 4xx, 5xx counts, latency percentiles)
- 21. 21@atseitlin Operate: Alerting Self-service, basedon meaningful service metrics Error rates often a better metric than success
- 22. 22@atseitlin Operate: Notifications Don’t relyon people to watch screens. Alerts should trigger notifications On-call rotations should include engineers that built the service Alerts should be granular down to service
- 23. 23@atseitlin Operate: Dynamic auto-scaling Savecosts Increase availability
- 24.
- 25. 25@atseitlin Failure is allaround us • Disks fail • Power goes out. And your generator fails • Software bugs introduced • People make mistakes Failure is unavoidable
- 26. 26@atseitlin We design aroundfailure • Exception handling • Clusters • Redundancy • Fault tolerance • Fall-back or degraded experience • All to insulate our users from failure Is that enough?
- 27. 27@atseitlin It’s not enough •How do we know if we’ve succeeded? • Does the system work as designed? • Is it as resilient as we believe? • How do we prevent drifting into failure? The typical answer is…
- 28. 28@atseitlin More testing! • Unittesting • Integration testing • Stress testing • Exhaustive test suites to simulate and test all failure mode Can we effectively simulate a large-scale distributed system?
- 29. 29@atseitlin Building distributed systemsis hard Testing them exhaustively is even harder • Massive data sets and changing shape • Internet-scale traffic • Complex interaction and information flow • Asynchronous nature • 3rd party services • All while innovating and building features Prohibitively expensive, if not impossible, for most large-scale systems
- 30. 30@atseitlin What if wecould reduce variability of failures?
- 31. 31@atseitlin There is anotherway • Cause failure to validate resiliency • Test design assumption by stressing them • Don’t wait for random failure. Remove its uncertainty by forcing it periodically
- 32. 32@atseitlin And that’s exactlywhat we did
- 33.
- 34.
- 35. 35@atseitlin Chaos Monkey taughtus… • State is bad • Clusters are good • Surviving single instance failure is not enough
- 36.
- 37.
- 38. 38@atseitlin Chaos Gorilla taughtus… • Hidden assumptions on deployment topology • Infrastructure control plane can be a bottleneck • Large scale events are hard to simulate • Rapidly shifting traffic is error prone • Smooth recovery is a challenge • Cassandra works as expected
- 39. 39@atseitlin What about largercatastrophes? Anyone remember Sandy?
- 40.
- 41. 41@atseitlin The Sick andWounded
- 42.
- 43.
- 44.
- 45. 45@atseitlin Latency Monkey taughtus • Startup resiliency is often missed • An ongoing unified approach to runtime dependency management is important (visibility & transparency gets missed otherwise) • Know thy neighbor (unknown dependencies) • Fallbacks can fail too • It’s hard to be ready for Latency Monkey, especially with no customer impact
- 46.
- 47.
- 48. 48@atseitlin Clutter accumulates • Complexity •Cruft • Vulnerabilities • Cost
- 49.
- 50. 50@atseitlin Janitor Monkey taughtus… • Label everything • Clutter builds up
- 51. 51@atseitlin Ranks of theSimian Army • Chaos Monkey • Chaos Gorilla • Latency Monkey • Janitor Monkey • Conformity Monkey • Circus Monkey • Doctor Monkey • Howler Monkey • Security Monkey • Chaos Kong • Efficiency Monkey
- 52. 52@atseitlin Observability is key •Don’t exacerbate real customer issues with failure exercises • Deep system visibility is key to root-cause failures and understand the system
- 53. 53@atseitlin Organizational elements • Everyengineer is an operator of the service • Each failure is an opportunity to learn • Blameless culture Goal is to create a learning organization
- 54. 54@atseitlin Summary Software is becomingubiquitous, is delivered as an always-on service, and is at the heart of enabling business innovation. Every organizations is looking for ways to accelerate innovation. DevOps accelerates innovation.
- 55. 55@atseitlin Q & A Questions? Youcan reach me at ariel@scalevp.com www.linkedin.com/in/atseitlin @atseitlin