Successfully reported this slideshow.
Upcoming SlideShare
×

# DevOps Metrics - Lies, Damned Lies and Statistics

6,439 views

Published on

my presentation at DevOps Day 2015 Italy

Published in: Data & Analytics
• Full Name
Comment goes here.

Are you sure you want to Yes No
• My personal experience with research paper writing services was highly positive. I sent a request to ⇒ www.HelpWriting.net ⇐ and found a writer within a few minutes. Because I had to move house and I literally didn’t have any time to sit on a computer for many hours every evening. Thankfully, the writer I chose followed my instructions to the letter. I know we can all write essays ourselves. For those in the same situation I was in, I recommend ⇒ www.HelpWriting.net ⇐.

Are you sure you want to  Yes  No

Are you sure you want to  Yes  No
• accessibility Books Library allowing access to top content, including thousands of title from favorite author, plus the ability to read or download a huge selection of books for your pc or smartphone within minutes.........ACCESS WEBSITE Over for All Ebooks ..... (Unlimited) ......................................................................................................................... Download FULL PDF EBOOK here { https://urlzs.com/UABbn } .........................................................................................................................

Are you sure you want to  Yes  No

Are you sure you want to  Yes  No

Are you sure you want to  Yes  No

### DevOps Metrics - Lies, Damned Lies and Statistics

1. 1. DevOps Metrics lies, damned lies and statistics Gaetano Mazzanti @mgaewsj agile42
2. 2. why do we need metrics?
3. 3. why do we need metrics? key reason: to improve
4. 4. why do we need metrics? decisions
5. 5. why do we need metrics? predictability
6. 6. beware of cheating averages fallacies
7. 7. beware metrics can be gamed
8. 8. metrics & statistics still require reasoning and visual examination beware
9. 9. same mean, variance & correlation (7.50 3.75 0.816)
10. 10. use your eyes (same mean, variance & correlation) median = 7.58 median = 8.14 median = 7.11 median = 7.04
11. 11. ecological fallacy average math score 50/100 60/100 group A 70% of people in A have better score than B! group B 70%
12. 12. ecological fallacy 50/100 60/100 70% Group A Group B # score # score 50 45 70 43 50 55 30 100 group A group B average math score
13. 13. exception fallacy
14. 14. Simpson’s paradox Global Natural Treat Live 108 153 Die 123 120 Natural 47% live Treat 56% live Women Natural Treat Live 57 32 Die 100 57 Natural 36,3% live Treat 36,0% live Men Natural Treat Live 51 121 Die 23 63 Natural 69% live Treat 45% live
15. 15. Simpson’s paradox Global Natural Treat Live 108 153 Die 123 120 Natural 47% live Treat 56% live Women Natural Treat Live 57 32 Die 100 57 Natural 36,3% live Treat 36,0% live Men Natural Treat Live 51 121 Die 23 63 Natural 69% live Treat 45% live
16. 16. which metrics?
17. 17. deployment frequency lead time for changes mean time to recover change fail rate how IT performance was measured !?!?!?
18. 18. ITIL KPIs “ITIL Key Performance Indicators (ITIL KPIs) are used to assess if the processes of an IT organization are running according to expectations” and if not… just kidding
19. 19. a few ITIL KPIs… example (1/2)
20. 20. a few ITIL KPIs… example (2/2)
21. 21. it’s easy to get lost in a maze of (not relevant) data
23. 23. what do you want to learn? key question about your metrics
24. 24. loops
25. 25. improvement loops build/ change measurelearn experiment actionable metric hypothesis
26. 26. which metrics matter to customers no yes end-to-end (global) functional (local) typical ideal
27. 27. service oriented mindset DevOps as a service provided to deliver value to the business
28. 28. pizza delivery fast delivery accuracy and quality predictability what matters to customers
29. 29. let work ﬂow ﬂow is the movement and delivery of customer value through a process
30. 30. derive from poor ﬂow slow delivery low quality unpredictability
31. 31. poor ﬂow => queues
32. 32. just 3 metrics? Work In Progress Lead Time Throughput
33. 33. Little’s Law Items In Queue = Arrival Rate * Waiting Time Lead Time = Work In Progress / Throughput
34. 34. focus on lead time
35. 35. 0 1 2 3 4 5 6 7 8 1-Feb 3-Feb 5-Feb 7-Feb 9-Feb 11-Feb 13-Feb 15-Feb 17-Feb 19-Feb 21-Feb scatterplot 54% 71% 88% 96% lead time (days) average
36. 36. scatterplot source ActionableMetrics book Lead
37. 37. scatterplot (only bugs) source ActionableMetrics book Lead
38. 38. frequency distribution source ActionableMetrics book Lead
39. 39. Weibull distribution
40. 40. what to aim for
41. 41. aging source ActionableMetrics book
42. 42. ___ efﬁciency process efﬁciency = total time active time________ 20 5 = 25% !1!!2!!3!!4!!5!!6!!7!!8!!9!10!11!12!13!14!15!16!17!18!19!20! elaborate do validate deliver waiting active
43. 43. SLAs Service Level Agreements agreementexpectation a SLA is a contract between a service provider and the user/ customer that deﬁnes the level of service expected from the service provider i.e. we expect an item to ﬂow through the process and exit in 5 days or less with an 85% probability of success
44. 44. SLAs – some hints do not set a SLA without analyzing Lead Time data do not allow a SLA to be set by someone external to your group do not set a SLA without collaborating with customers and/or other stakeholders use different SLAs for different Work Item Types
45. 45. SLA
46. 46. slack – avoid full utilization absorb variations % capacity utilization queuesize queue size grows exponentially at high capacity0 5 10 15 20 25 0 10 20 30 40 50 60 70 80 90 100