Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DevOps Metrics - Lies, Damned Lies and Statistics

6,150 views

Published on

my presentation at DevOps Day 2015 Italy

Published in: Data & Analytics
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • accessibility Books Library allowing access to top content, including thousands of title from favorite author, plus the ability to read or download a huge selection of books for your pc or smartphone within minutes.........ACCESS WEBSITE Over for All Ebooks ..... (Unlimited) ......................................................................................................................... Download FULL PDF EBOOK here { https://urlzs.com/UABbn } .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Download or read that Ebooks here ... ......................................................................................................................... DOWNLOAD FULL PDF EBOOK here { http://bit.ly/2m77EgH } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... Download Doc Ebook here { http://bit.ly/2m77EgH } ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

DevOps Metrics - Lies, Damned Lies and Statistics

  1. 1. DevOps Metrics lies, damned lies and statistics Gaetano Mazzanti @mgaewsj agile42
  2. 2. why do we need metrics?
  3. 3. why do we need metrics? key reason: to improve
  4. 4. why do we need metrics? decisions
  5. 5. why do we need metrics? predictability
  6. 6. beware of cheating averages fallacies
  7. 7. beware metrics can be gamed
  8. 8. metrics & statistics still require reasoning and visual examination beware
  9. 9. same mean, variance & correlation (7.50 3.75 0.816)
  10. 10. use your eyes (same mean, variance & correlation) median = 7.58 median = 8.14 median = 7.11 median = 7.04
  11. 11. ecological fallacy average math score 50/100 60/100 group A 70% of people in A have better score than B! group B 70%
  12. 12. ecological fallacy 50/100 60/100 70% Group A Group B # score # score 50 45 70 43 50 55 30 100 group A group B average math score
  13. 13. exception fallacy
  14. 14. Simpson’s paradox Global Natural Treat Live 108 153 Die 123 120 Natural 47% live Treat 56% live Women Natural Treat Live 57 32 Die 100 57 Natural 36,3% live Treat 36,0% live Men Natural Treat Live 51 121 Die 23 63 Natural 69% live Treat 45% live
  15. 15. Simpson’s paradox Global Natural Treat Live 108 153 Die 123 120 Natural 47% live Treat 56% live Women Natural Treat Live 57 32 Die 100 57 Natural 36,3% live Treat 36,0% live Men Natural Treat Live 51 121 Die 23 63 Natural 69% live Treat 45% live
  16. 16. which metrics?
  17. 17. deployment frequency lead time for changes mean time to recover change fail rate how IT performance was measured !?!?!?
  18. 18. ITIL KPIs “ITIL Key Performance Indicators (ITIL KPIs) are used to assess if the processes of an IT organization are running according to expectations” and if not… just kidding
  19. 19. a few ITIL KPIs… example (1/2)
  20. 20. a few ITIL KPIs… example (2/2)
  21. 21. it’s easy to get lost in a maze of (not relevant) data
  22. 22. why? key question about your metrics
  23. 23. what do you want to learn? key question about your metrics
  24. 24. loops
  25. 25. improvement loops build/ change measurelearn experiment actionable metric hypothesis
  26. 26. which metrics matter to customers no yes end-to-end (global) functional (local) typical ideal
  27. 27. service oriented mindset DevOps as a service provided to deliver value to the business
  28. 28. pizza delivery fast delivery accuracy and quality predictability what matters to customers
  29. 29. let work flow flow is the movement and delivery of customer value through a process
  30. 30. derive from poor flow slow delivery low quality unpredictability
  31. 31. poor flow => queues
  32. 32. just 3 metrics? Work In Progress Lead Time Throughput
  33. 33. Little’s Law Items In Queue = Arrival Rate * Waiting Time Lead Time = Work In Progress / Throughput
  34. 34. focus on lead time
  35. 35. 0 1 2 3 4 5 6 7 8 1-Feb 3-Feb 5-Feb 7-Feb 9-Feb 11-Feb 13-Feb 15-Feb 17-Feb 19-Feb 21-Feb scatterplot 54% 71% 88% 96% lead time (days) average
  36. 36. scatterplot source ActionableMetrics book Lead
  37. 37. scatterplot (only bugs) source ActionableMetrics book Lead
  38. 38. frequency distribution source ActionableMetrics book Lead
  39. 39. Weibull distribution
  40. 40. what to aim for
  41. 41. aging source ActionableMetrics book
  42. 42. ___ efficiency process efficiency = total time active time________ 20 5 = 25% !1!!2!!3!!4!!5!!6!!7!!8!!9!10!11!12!13!14!15!16!17!18!19!20! elaborate do validate deliver waiting active
  43. 43. SLAs Service Level Agreements agreementexpectation a SLA is a contract between a service provider and the user/ customer that defines the level of service expected from the service provider i.e. we expect an item to flow through the process and exit in 5 days or less with an 85% probability of success
  44. 44. SLAs – some hints do not set a SLA without analyzing Lead Time data do not allow a SLA to be set by someone external to your group do not set a SLA without collaborating with customers and/or other stakeholders use different SLAs for different Work Item Types
  45. 45. SLA
  46. 46. slack – avoid full utilization absorb variations % capacity utilization queuesize queue size grows exponentially at high capacity0 5 10 15 20 25 0 10 20 30 40 50 60 70 80 90 100
  47. 47. your policies shape your data your data shape your policies
  48. 48. where to start from?
  49. 49. a possible approach understand sources of dissatisfaction analyze demand & capacity discover work item types measure flow set SLAs setup metric based improvement experiments (similar to STATIK…)
  50. 50. obsessions you should have improve process continuously remove problems/impediments asap get help from metrics
  51. 51. Gaetano Mazzanti @mgaewsj gaetano.mazzanti@agile42.com agile42

×