More Related Content
Similar to Software MTTR: Continuous Delivery Reduces Repair Time
Similar to Software MTTR: Continuous Delivery Reduces Repair Time (20)
Software MTTR: Continuous Delivery Reduces Repair Time
- 1. +
Software MTTR: the Path
from Continuous Integration
to Continuous Delivery
Jeff Sussna, Ingineering.IT
- 2. Software Mean Time to Repair
Bug-Hours: MTTR for software
Enhancement requests are bug reports
Gap between actual and desired value
All software development is “repair”
Continuous Delivery = Continuous Repair = Minimize MTTR
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 3. Change is Good/Change is Bad
Change is good: how we deliver value
Change is bad: introduces uncertainty and often failure
Continuous Delivery answers its own question
Reduces uncertainty
Increases rate of value delivery
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 4. Continuous Delivery is Lean
Small batch sizes
Just-in-time production
Andon cord
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 5. Batch Size
Batch size: unit of work passing between process stages
Small batch sizes:
Reduce complexity and risk
Increase efficiency
Lower overhead
Reduce cycle time
Speed up feedback loops
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 6. Just-in-time Production
Only make what’s needed, when and how it’s needed
Avoid wasteful inventory
Applies to:
Features
Designs
Code
Tests
Infrastructure
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 7. Andon Cord
Any worker can stop the production line
Treat the next step in the pipeline as your customer
Improve quality and efficiency by fixing defects now not later
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 8. Agile is Lean
Batch size: one year => one month
JIT production: groom the backlog each iteration
Andon cord: continuous integration
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 9. Sources of Software Failures
Unit errors
Requirements misunderstanding
Logic errors
Design errors
Integration errors
Hard due to inherent complexity
Delayed/batched due to fear
Irony: large integration batch sizes increase level of difficulty
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 10. Continuous Integration
Check into trunk often
Build on every commit
Test on every build
Notify on every error
Fix every broken build
Necessitates build/deploy/test/report automation
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 11. Value of Continuous Integration
Reduces integration batch size and latency
Enables Andon cord
Pulls QA into development
Removes waste/variation from manual build/deploy/test/report
Minimizes “merge-hell” due to long-lived branches
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 12. Redefining “Done”
Agile says nothing about getting product in customer’s hands
Software product era: cost of delivery is high for all
Vendor: burn and mail CD
Customer: upgrade to new version
Cloud changes the development/delivery equation
Customer’s delivery cost shrinks
Development org IS the delivery org
Done: “the customer has the change and is satisfied with it”
Kind of like any other service industry
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 13. Continuous Delivery
Finishes what Continuous Integration starts
Minimizes friction in the flow from integration to satisfaction
Forces end-to-end batch size reduction
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 14. Doing Continuous Delivery
Small batch sizes all the way to prod
Automation all the way to prod
Decoupled technical and non-technical deployment
Deployed in prod != visible to customer
Shift your focus from iterations to changes
Think in terms of bug-hours
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 15. Small Batch Sizes
Don’t batch testing ‘til the end of the iteration
Pipeline testing stages
Acceptance
Security
Performance/Scalability/Fault-tolerance
UAT
Fail as far upfront as possible
Don’t batch release‘til the end of iteration
Hotfixes go away!
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 16. Automation
Human gates are OK
When/why to move forward
Manual labor is not OK
Acceptance testing
Non-functional testing
Deployment throughout the pipeline
Deal in atomic releases
DB + Code + Infrastructure
Deployment + rollback
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 17. Decoupled Deployment
Techniques:
Feature toggles
Feature branches
Application componentization
Business-defined release: the ultimate goal
Canary
A-B
Demographic/Geographical
Multi-tenancy for code
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 18. Code Multi-Tenancy as Strategy
Empower the business: release at 12:01 Christmas morning
Free the developers: don’t deploy at 12:01 Christmas morning
1st-class “non-functional” requirement
Job is to deliver service
Biz ops needs control over features as part of customer interaction
Should be fundamental part of application architecture
Test it!
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 19. Continuous Delivery Discipline
Don’t check it in if you don’t want it to go to prod
Check-in is the middle of the delivery lifecycle, not the beginning
Don’t write code you don’t want to go to prod
Don’t assume technical release equals business release
Strive for release boredom
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 20. Boring Releases
Just like integration we avoid release because it’s scary
Do scary things more, not less often
Practice makes perfect
Find ways to make it less scary
Making it easy to back out makes it less scary
Holy grail: prod releases are boring
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 21. The Importance of Integrated
Quality
Continuously delivering lousy software is not getting to done
QA must be deeply embedded to avoid friction
Deep QA integration can make things faster AND better
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 22. Quality Integration Techniques
Testing as software engineering (SET)
Testing pyramid
Test the service not just the software
In XaaS’the ‘S’ stands for ‘service’ not ‘software’
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 23. Software Engineer in Test
Approach test development as software engineering
Object-oriented design
TDD
Performance, maintainability, extensibility
Use productivity enablers (Groovy, Spock, Geb…)
Do test design spikes, not just code design spikes
Be the advocate for the new definition of done
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 24. The Testing Pyramid
Automated tests’ value depends on how often/fast they can run
System tests: few
Acceptance tests: some
Integration tests: many
Unit tests: lots
The pyramid spans traditional dev/test boundaries
Developers need to think about testing
Testers mind developers’ business at the lower pyramid levels
Test design spikes should involve everyone
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 25. Test the Service not the Software
The delivery pipeline is part of the service, so test it
Inefficiencies are bugs too
What about “non-functional” requirements?
Performance, resiliency, DB, InfoSec
Incorporate them into upfront processes
Test them using a deployment pipeline
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 26. The Importance of Configuration
Automation
Beware the sandbox
Variation creates waste and errors
Manual developer configuration has opportunity cost
Configuration automation is a solved problem
Environments: cloud, Vagrant
Automation: Puppet, Chef, CFEngine
Deployment: Jenkins, Capistrano, Liquibase, flyway
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 27. Keys to Configuration Automation
Do everything to all the things
Code, database, system, infrastructure
Version control
Packaging
Be able to go from any Point A to and Point B
Including when A = nothing
Testers: don’t be codependent with developers
Remember, you’re minding their business
Remember, waste/variation degrades service quality
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 28. Solving the Change Conundrum
Smaller changes are easier
Automated processes are less error-prone
More frequent changes require practice => perfection
Change is no longer the enemy
Change is your business
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 29. Getting from Here to There
Don’t do “continuous delivery”, do “more continuous delivery”
CD holy grail requires tremendous tool and process maturity
Start with something small and simple
Apply Kaizen
Question assumptions
Especially about anything that feels “clumsy” or “we have to
because…”
Slow down to go faster
Find opportunities to reduce batch size
Think in terms of one-piece flow/new definition of “done”
Find ways to move quality inward and forward
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.
- 30. Software MTTR
Repeat after me: our job is to deliver service, not software
Continual value repair is part of the service
MTTR is a (the?) key metric
If you grok the logic of CD, the tools/process will flow
If not, they either frustrate or else represent ‘cargo cult’
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.