Metrics that Matter
•
4 likes•1,092 views
“Every line is the perfect length if you don't measure it.” - Marty Rubin So your organization has embarked upon a transformation to be more nimble and responsive by employing the latest tools and thinking in the Agile and DevOps arena. In this transformational context, how do you know that your initiatives are effective? Empirical measurements should provide insights on business value flow and delivery efficiency, allowing teams and organizations to see how they are progressing toward achieving their goals, but all too often we find ourselves mired in measurement traps that don't quite provide the right guidance in steering our efforts. Rooted in contemporary thinking and tested in practice, this talk explores the principles of good measurement, what to measure, what not to measure, and enumerates some key metrics to help guide and inform our Agile and DevOps efforts. If done right, metrics can present a true picture of performance, and any progression, digression of these metrics can drive learning and improvement. It is our hope that this session inspires organizations and teams to start or take a fresh look at implementing a valuable measurement program.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Metrics that Matter
Similar to Metrics that Matter (20)
Recently uploaded
Recently uploaded (20)
Metrics that Matter
- 1. What’s Measured Improves Metrics That Matter ! Raj Indugula raj.indugula@lithespeed.com Rob Brown robert.david.brown@gmail.com
- 2. Agenda • Why and What of Metrics • Principles of Measurement • Key Drivers & Metrics • Key Takeaways • Q&A
- 3. What does mean in today’s Lean-Agile-DevOps world? 3
- 4. What is Measurement? Measures • I have 5 apples • # incidents Metrics • I have 5 more apples than yesterday • % of Sev1 incidents since feature rollout Observation that reduces uncertainty… 4
- 6. 6
- 7. What Makes A Good Metric? http://www.flickr.com/photos/circasassy/7858155676/ Understandable Comparative Behavior changing 7
- 8. Measure movement towards your business goals & outcomes Where are you going? 8 Picture source New Relic
- 9. KEY DRIVERS & METRICS Development ReleaseQA OperateDeploy Customer Success Business Success
- 10. 1. Business Success • Starts with Measurable Objectives • Indicators for success • Market share • New business service enablement 10
- 11. #ofStories 0 40 80 120 160 Sprints 78 (6/8-6/21) 80 (7/6-7/19) 82 (8/3-8/16) 84 (8/31-9/13) 86 (9/28-10/11) 88 (10/26-11/8) Scope (Total Stories Planned) Total Completed (Cumulative) Stories Completed Per Sprint Projected (Optimistic) Projected (Pessimistic) Projected (Median) 26 19 483 2019 1116 108106 100 81 77 6966 46 27 16 138138138138 134 128 121118 109 99 94 88 16 27 46 66 69 77 81 100 106 108 88 94 99 109 118 121 128 134 138 138 138 138 Empirical Data Drives Enquiry and Adjustment 11Business success Q: When are all planned stories estimated to be completed? A: End of Sprint 90 (12/06/16) Q: What is the total number of stories estimated to be completed by 11/08/2016? A: 119 Stories based on average velocity Q: Can we deliver all current planned stories by 10/16/2016? A: Not Likely, given the current trend Fixed Scope Fixed Date Fixed Scope & Date
- 12. Business Success • Average Revenue per User • Customer Lifetime Value • Daily/Monthly Active Users • Average Session Time • % Change in Customer Volume • Feature Metrics Business Success
- 13. 2. Customer/User Success • Frequency of key transactions • Amount of time spent in the application • User satisfaction survey results • A/B test results • Customer ticket volume www.glasbergen.com 13
- 14. Product 1 0 23 45 68 90 7/26/2020 8/2/20208/9/2020 8/16/2020 8/23/2020 8/30/2020 9/6/2020 9/13/2020 9/20/2020 9/27/2020 10/4/2020 10/11/2020 4 11 10 6 11 4 4 8 1 14 6 0 90 79 69 63 52 48 44 36 35 21 15 15 Pending Resolved Product 2 0 75 150 225 300 7/26/2020 8/2/20208/9/2020 8/16/2020 8/23/2020 8/30/2020 9/6/2020 9/13/2020 9/20/2020 9/27/2020 10/4/2020 10/11/2020 18 58 9 2 6 10 20 25 28 19 22 3 229 171 162 160 154 144 124 99 71 52 30 27 Pending Resolved Customer Satisfaction “You Build it, You Own It” Improving Responsiveness to Issues
- 15. Customer Success Customer Success • Customer Ticket Volume • Net Promoter Score • Net Value Score • Customer Anxiety • Reliability
- 16. 3. Operations Success • Availability • Performance • Efficiency • Support 16 http://landing.google.com/sre/book/ Deploy Release Operate
- 17. Improving Deployment Success Rate, Not Just Frequency 17Operations success Build Daily Success Rate Date Build name 9/12/2016 9/13/2016 9/14/2016 9/15/2016 9/16/2016 Test Suite 1 Test Suite 2 Test Suite 3
- 18. Operate Deploy Release Operate • Deployment Lead Time • MTTR • % of Failed Deployments • Deployment frequency, duration • Release Frequency • Time/Cost per Release • Predictability • MTTR • Performance - Response Time • Availability • Security Pass Rate
- 19. 4. Development Success • Lead time for changes (from development to deployment) • Deployment frequency • % Failed deployments • Incident severity • Outstanding defects Develop QA Deploy
- 20. 0% 25% 50% 75% 100% Sprint83 Sprint84 Sprint85 Sprint86 Other New Capability Major Interface Product Enhancements Production Incidents Technical Debt Arch. Improvement Insight into Team Capacity and Work Items 20Development success Balancing New Feature Work with Maintenance and Improvement
- 21. Development • Development Lead Time • Idle Time • Cycle Time • Work in Progress- Technical Debt • Rework • Idle Time • MTTD • Defects • Deployment Lead Time • MTTR • % of Failed Deployments • Deployment frequency, duration DeployQADevelopment
- 22. Outcome Title: Provide a common platform for CI/CD - PaaS/CaaS Outcome Description: Provide teams a ubiquitous, self service platform for build, deploy and test. Measure: Team Adoption rate Baseline: 1 team/month Budget: 6 million Dependancies: platform & security engineers, IaaS provider Adoption Rate Measures: Features Frequency Form Factor New Service USER.id Date Event Form Count billy.bob 2/10/17 login IDE 3 suzy.q 2/11/17 image stream GUI 4
- 23. • Average Revenue per User • Customer Lifetime Value • Daily/Monthly Active Users • Average Session Time • % Change in Customer Volume • Feature Metrics • Development Lead Time • Idle Time • Cycle Time • Work in Progress- Technical Debt • Rework • Idle Time • MTTD • Defects • Deployment Lead Time • MTTR • % of Failed Deployments • Deployment frequency, duration • MTTR • Performance - Response Time • Availability • Security Pass Rate • Customer Ticket Volume • Net Promoter Score • Net Value Score • Customer Anxiety • Reliability • Release Frequency • Time/Cost per Release • Predictability Accelerate Value Delivery Balance speed, risk, quality & cost Reduce time to obtain/respond to customer Now, All Together 23 Operations SuccessDevelopment Success
- 24. Monitoring Framework • Aggregate metrics from all success contexts • See the entire system • Enables visualization, anomaly detection, trending, alerting… Source Adaptation: Turnbull, The Art of Monitoring, Kindle edition, chap. 2. Business Success Customer Success Development Success Operations Success 24 Event Aggregator
- 26. KEY TAKEAWAYS
- 27. • Keep inventory of metrics small • Measure outcomes, not individuals • Monitor balanced set of metrics • Monitor trends, not data points • Share Widely – Interaction encourages exploration “If you measure me in an illogical way…don’t complain about illogical behavior” - Goldratt 27
- 28. Next Steps Clear Objective Well formed metrics Measure & Improve “Metrics illuminate, not indicate” – George Dinwiddie
- 29. REFERENCES
- 30. • http://onstartups.com/tabid/3339/bid/96738/Measuring-What- Matters-How-To-Pick-A-Good-Metric.aspx • http://www.slideshare.net/GoAtlassian/understanding-metrics- what-to-measure-and-why-john-custy • http://www.slideshare.net/TeamQualityPro/the-good-the-bad- and-the-metrics • http://devopsenterprise.io/media/ DOES_forum_metrics_102015.pdf • http://devops.com/2014/11/10/devops-scorecard/ • http://www.datical.com/blog/9-metrics-devops-teams-tracking/ • http://devops.com/2015/01/26/metrics-devops/ • https://blog.appdynamics.com/devops/quantified-devops/ • http://www.slideshare.net/jedi4ever/devops-metrics • http://www.slideshare.net/ITRevolution/does15-troy-magennis- and-julia-wester-metrics-and-modeling-helping-teams-see-how- to-improve • https://thenewstack.io/tag/ebook5-monitoring-and- management/ • https://www.artofmonitoring.com/