What is the best way to measure DevOps performance? There are many ways that people have tried to measure productivity in software delivery in the past -- what works and what doesn’t? In this webinar, Dr. Nicole Forsgren and Robert Reeves will present some lessons learned about measuring software delivery and why it’s important. The webinar will also highlight the key factors driving DevOps performance and offer a preview to some of the challenges on the horizon. Tune into this webinar to learn about: Flaws in previous attempts to measure performance What really matters (hint: focus on outcomes) The four measures that are key to delivery performance The *big* difference between high performers and the rest Why maturity models don’t work What high performers can do for their organizations What challenges are up next in technology transformations… things like data, serverless, and security.

1. Measuring Performance:
See the Science of DevOps
Measurement in Action

2. Your Hosts
Dr. Nicole
Forsgren
@nicolefv
Robert Reeves
@robertreeves

3. The Highlights
• Measuring Performance
• Most important things to remember
• Common mistakes
• An example of a good measure: Software Delivery Performance
• Where are you?
• Maturity Models
• Broken
• Where they work
• Learning theory
• You can help

4. Most important!

5. Common Mistakes
• Outcomes > outputs
• Global or team > Local or individual

6. Common Mistakes

7. Common Mistakes
• Some common examples:
• Lines of code
• Velocity
• Utilization

8. Common Mistakes: Lines of Code
• More is better?
• Bloated software
• Higher maintenance costs
• Higher cost of change
• Less is better?
• Cryptic code that no one can read
• Ideal: solve business problems with most efficient code

9. Common Mistakes: Velocity
• Agile: problems are broken down into stories, which are
assigned “points” of estimated effort to complete
• At end of sprint, total points signed off by customer is recorded
= velocity
• Velocity is a capacity planning tool. NOT a productivity tool.
• Why doesn’t this work for productivity?
• Velocity is a relative measure, not absolute. So: bad for comparing
teams
• Gaming by inflating estimates
• Focus on team completion at the expense of collaboration (a global
goal)

10. Common Mistakes: Utilization
• Utilization is only good up to a point
• Higher utilization is better?
• High utilization doesn’t allow slack for unplanned work
• Queue theory: as utilization approaches 100%, lead times approach infinity
• Once you hit higher and higher levels of utilization (a poor goal of
productivity), teams will take longer and longer to get work done

11. An example:
Software Delivery
Performance

12. Measuring Software Delivery Performance
• Focus on both Outcomes and Global measures:
- Deploy frequency (when business demands)
- Lead Time for Changes
- Mean Time to Recover (MTTR)
- Change Fail Rate

13. We see
More throughput
More stability
In tandem. Without the tradeoffs that some suggest are
necessary.

14. High Performing DevOps Teams
More agile
More frequent
Code deployments
46x
That’s the difference between multiple
times per day and once a week or less.
Faster lead time from commit to
deploy
440x
That’s the difference between less than
an hour and more than a week.

15. High Performing DevOps Teams
More reliable
Faster mean time to
recover from downtime
96x
That means high performers recover in
less than an hour instead of several days
As likely that changes will
fail
1/5x
That means high performers changes fail 0-15% of
the time, compared to 31-45% of the time.

17. Maturity Models
(generally bad news)

18. Maturity models are for CHUMPS

19. Maturity models are for CHUMPS

20. Maturity Models
(one exception)

21. Additional Resources

22. You Can Help!

23. Your Role in this
• Start by measuring a few things
• Focus on outcomes and global measures
• Think about what things you control that can drive improvements
in those measures – both with tech and with not tech
• Share your stories! Leverage community

24. Questions?

25. Thank You