0
DEVOPS METRICS             ANDREW SHAFER - PATRICK DEBOIS - ISRAEL GATThursday 16 June 2011
Patrick Debois                                                                         Jedi                               ...
Israel Gat                            Cutter Consortium                                @agileexec                        h...
Andrew Shafer - Rabble Rouser - Cloudscaling        @littleidea - http://stochasticresonance.wordpress.comThursday 16 June...
http://www.flickr.com/photos/quinnanya/3588989219Thursday 16 June 2011
http://www.flickr.com/photos/zen/3233023066/Thursday 16 June 2011
To Compare                        http://www.flickr.com/photos/thebusybrain/2492945625/Thursday 16 June 2011
http://www.flickr.com/photos/topshampatti/219953012/Thursday 16 June 2011
To        UnderstandThursday 16 June 2011
To ProtectThursday 16 June 2011
To Improve                        http://www.flickr.com/photos/vizzzual-dot-com/2655969483/Thursday 16 June 2011
Describing a                  Metric                               http://www.flickr.com/photos/chandramarsono/4324373384/T...
Describing a                  Metric                •basis of measurement                                        http://ww...
Describing a                  Metric                •basis of measurement                •assumptions                     ...
Describing a                  Metric                •basis of measurement                •assumptions                •leve...
Describing a                  Metric                •basis of measurement                •assumptions                •leve...
Describing a                  Metric                •basis of measurement                •assumptions                •leve...
Describing a                  Metric                •basis of measurement                •assumptions                •leve...
Describing a                  Metric                •basis of measurement                •assumptions                •leve...
Describing a                  Metric                •basis of measurement                •assumptions                •leve...
Beware of           Vanity Metrics                        UNACTIONABLE                        INACCESSIBLE                ...
Red is good ,                        Green is worthless                                 •   http://www.reliableplant.com/R...
Thursday 16 June 2011
“In truth, a good case could be made                        that if your knowledge is meagre and                        un...
What we can learn from monitoring                        http://www.experimentalistsanonymous.com/stuff/paralleluniverse/i...
Level 0 - Individual ResourceThursday 16 June 2011
Availability  Processes, Port,SSH Login   Network, Memory, CPUThursday 16 June 2011
Availability  Processes, Port,SSH Login   Network, Memory, CPUThursday 16 June 2011
Capacity  Processes, Port,SSH Login   Network, Memory, CPUThursday 16 June 2011
Capacity  Processes, Port,SSH Login   Network, Memory, CPUThursday 16 June 2011
Health                        Thresholds                                     http://www.smiley-faces.org/wallpaper/smiley-...
Health                        Thresholds                                     http://www.smiley-faces.org/wallpaper/smiley-...
Metric versus Diagnostic                                           Does it                                         contrib...
Level 1 - Services LevelThursday 16 June 2011
Performance                SSH, HTTP, NTPThursday 16 June 2011
Performance                SSH, HTTP, NTP                                 # Servers/AdminThursday 16 June 2011
Performance                SSH, HTTP, NTP                                 # Servers/Admin                                 ...
“Gameable”                        http://kithicor.org/wp/wp-content/uploads/2010/01/gamerEvolution.jpgThursday 16 June 2011
Level 2 - Team LevelThursday 16 June 2011
Aggregation                              LB-WEB                        WEB    WEB     WEB                         1      2...
Aggregation                                  DEV                              Project Team                        DEV     ...
Burn up chartThursday 16 June 2011
Team UtilizationThursday 16 June 2011
“Velocity also has                        a direction component”                                        http://www.randoms...
What about efficiency?                                   http://www.flickr.com/photos/jdiggans/92981578/Thursday 16 June 2011
http://sievertschreiber.files.wordpress.com/2009/12/good-code-is-measured-in-wtf-per-minute.jpgThursday 16 June 2011
Incentives & Behaviors                                  http://www.flickr.com/photos/turtlemadness/2439050717/Thursday 16 J...
Development                   Operations               Level 3 - Integration Levelhttp://www.xgn.nl/images/upload/20081020...
Integration                        WEB       APP       DBThursday 16 June 2011
Collaboration                        DEV                                                QA                   OPS          ...
Counting Collaboration                           More interaction doesn’t                            mean a better party  ...
Measuring Collaboration?                        “Play Nice Study”   http:///www.healthylarimer.org/documents/PlayNice.pptT...
Levels Collaboration                               Collaboration                                 Coalition                ...
Cooperation                             provide information                          somewhat defined roles                ...
•share information & resources   •defined roles   •frequent communication          Coordination   •some shared decision mak...
•share ideas                                •share resources                    Coalition   •frequent & prioritized       ...
Collaboration   •members belong to one system   •frequent communication   characterized by mutual trust   •consensus on al...
Knowledge sharing                                   http://www.au.af.mil/au/awc/awcgate/navy/onr_metrics_on_collab.pdfThur...
Team Resilience                        10th Floor Test                                                                    ...
Black Hole Theory                                        Absorbs light,                                    to see the effe...
Quest for proof                                                               •        Innovating and/or                  ...
More vs Better Widgets             Add more servers    Tune existing servers        Process Improvement      Quality Impro...
Collaboration Challenge                        “Maximize                        Team Utility”Thursday 16 June 2011
Level 4 - Business LevelThursday 16 June 2011
AHA              KA-CHING!Thursday 16 June 2011
Return on Investment                                         http://www.flickr.com/photos/hanekomu/4481674104/Thursday 16 J...
Inventory (WIP)Thursday 16 June 2011
Time to Market                               http://stefanroock.wordpress.com/2010/03/02/kanban-definition-of-lead-time-and...
Mean Time Between Failure                                            http://museumdosgames.blogspot.com/2010/04/game-over....
Mean Time Between ResolutionThursday 16 June 2011
Technical DebtThursday 16 June 2011
LeaksThursday 16 June 2011
Debt Metaphor             not all debt is created equal                                             http://www.flickr.com/p...
Customer Responsiveness                        Jim HighsmithThursday 16 June 2011
Types of Tech debt                        “oops”            “yeah I know”               unintentional                inten...
Gartner           •      http://www.castsoftware.com/resources/materials/analyst/Gartner-How_to_Monetize_Application_Techn...
Vicious Cycle of         The Vicious Cycle of TechnicalDebt                   Technical Debt                              ...
Symptoms - Measurable                                   • Poor Customer Responsiveness                                   •...
Secret Sauce                                http://radar.oreilly.com/archives/2007/10/Ops-ROI-jesserobbins.htmlThursday 16...
The Team Knows                               http://kingsinsight.com/2010/07/31/ask-the-team-about-technical-debt-the-cree...
Failure                                                 Value     Demand                %                                D...
Level 5 - End User LevelThursday 16 June 2011
End User Perspective                          • END to END monitoring                          • User Experience (browser ...
Participatory Design                                     http://www.informaworld.com/smpp/content~db=all~content=a79078665...
“Crowd” - Monitoring                    Sudden                Change in User                Activity Pattern              ...
Spend more time                        Automation   =>    creatively , early in                                           ...
Design Strategies                                                                                  To Design is Human     ...
Open Innovation             Wikipedia        Open Source     SharedConfig Management          Shared      Infrastructures  ...
Novice vs Expert        •        depth first                                                      • breath first , depth div...
Continuous Evolution                                                                             “Problem space           ...
Cultural TaxonomyThursday 16 June 2011
Cultural TaxonomyThursday 16 June 2011
tl;dr                        • avoid vanity metrics                        • insight > data                        • don`t...
Questions?Thursday 16 June 2011
Thursday 16 June 2011
Thank you!Thursday 16 June 2011
Upcoming SlideShare
Loading in...5
×

Devops Metrics

14,786

Published on

Published in: Technology
2 Comments
20 Likes
Statistics
Notes
No Downloads
Views
Total Views
14,786
On Slideshare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
301
Comments
2
Likes
20
Embeds 0
No embeds

No notes for slide

Transcript of "Devops Metrics"

  1. 1. DEVOPS METRICS ANDREW SHAFER - PATRICK DEBOIS - ISRAEL GATThursday 16 June 2011
  2. 2. Patrick Debois Jedi @patrickdebois http://jedi.be/blog http://www.vectronicsappleworld.com/collection/articlepics/joysticks/snap23.jpgThursday 16 June 2011
  3. 3. Israel Gat Cutter Consortium @agileexec http://theagileexecutive.comThursday 16 June 2011
  4. 4. Andrew Shafer - Rabble Rouser - Cloudscaling @littleidea - http://stochasticresonance.wordpress.comThursday 16 June 2011
  5. 5. http://www.flickr.com/photos/quinnanya/3588989219Thursday 16 June 2011
  6. 6. http://www.flickr.com/photos/zen/3233023066/Thursday 16 June 2011
  7. 7. To Compare http://www.flickr.com/photos/thebusybrain/2492945625/Thursday 16 June 2011
  8. 8. http://www.flickr.com/photos/topshampatti/219953012/Thursday 16 June 2011
  9. 9. To UnderstandThursday 16 June 2011
  10. 10. To ProtectThursday 16 June 2011
  11. 11. To Improve http://www.flickr.com/photos/vizzzual-dot-com/2655969483/Thursday 16 June 2011
  12. 12. Describing a Metric http://www.flickr.com/photos/chandramarsono/4324373384/Thursday 16 June 2011
  13. 13. Describing a Metric •basis of measurement http://www.flickr.com/photos/chandramarsono/4324373384/Thursday 16 June 2011
  14. 14. Describing a Metric •basis of measurement •assumptions http://www.flickr.com/photos/chandramarsono/4324373384/Thursday 16 June 2011
  15. 15. Describing a Metric •basis of measurement •assumptions •level and usage http://www.flickr.com/photos/chandramarsono/4324373384/Thursday 16 June 2011
  16. 16. Describing a Metric •basis of measurement •assumptions •level and usage •desired trend http://www.flickr.com/photos/chandramarsono/4324373384/Thursday 16 June 2011
  17. 17. Describing a Metric •basis of measurement •assumptions •level and usage •desired trend •when to use http://www.flickr.com/photos/chandramarsono/4324373384/Thursday 16 June 2011
  18. 18. Describing a Metric •basis of measurement •assumptions •level and usage •desired trend •when to use •when to stop http://www.flickr.com/photos/chandramarsono/4324373384/Thursday 16 June 2011
  19. 19. Describing a Metric •basis of measurement •assumptions •level and usage •desired trend •when to use •when to stop •how to game http://www.flickr.com/photos/chandramarsono/4324373384/Thursday 16 June 2011
  20. 20. Describing a Metric •basis of measurement •assumptions •level and usage •desired trend •when to use •when to stop •how to game •warnings http://www.flickr.com/photos/chandramarsono/4324373384/Thursday 16 June 2011
  21. 21. Beware of Vanity Metrics UNACTIONABLE INACCESSIBLE UNAUDITABLE http://www.flickr.com/photos/centralasian/5829549813/Thursday 16 June 2011
  22. 22. Red is good , Green is worthless • http://www.reliableplant.com/Read/21305/lean-metrics-red-is-good,-green-is-worthlessThursday 16 June 2011
  23. 23. Thursday 16 June 2011
  24. 24. “In truth, a good case could be made that if your knowledge is meagre and unsatisfactory, the last thing in the world you should do is make measurements; the chance is negligible that you will measure the right things accidentally.” George Miller http://stochasticresonance.wordpress.com/2008/08/03/miles-per-gallon/Thursday 16 June 2011
  25. 25. What we can learn from monitoring http://www.experimentalistsanonymous.com/stuff/paralleluniverse/images/2edmoneyshot.jpgThursday 16 June 2011
  26. 26. Level 0 - Individual ResourceThursday 16 June 2011
  27. 27. Availability Processes, Port,SSH Login Network, Memory, CPUThursday 16 June 2011
  28. 28. Availability Processes, Port,SSH Login Network, Memory, CPUThursday 16 June 2011
  29. 29. Capacity Processes, Port,SSH Login Network, Memory, CPUThursday 16 June 2011
  30. 30. Capacity Processes, Port,SSH Login Network, Memory, CPUThursday 16 June 2011
  31. 31. Health Thresholds http://www.smiley-faces.org/wallpaper/smiley-face-wallpaper-001.jpg http://www.geocities.jp/nikonikocalendar/index_en.htmlThursday 16 June 2011
  32. 32. Health Thresholds http://www.smiley-faces.org/wallpaper/smiley-face-wallpaper-001.jpg http://www.geocities.jp/nikonikocalendar/index_en.htmlThursday 16 June 2011
  33. 33. Metric versus Diagnostic Does it contribute to Yes Measurement the Metric bottom line? No Diagnostic Pete Behrens,Trail Ridge Consulting http://www.slideshare.net/krivitsky/agile-metrics-presentation-65425Thursday 16 June 2011
  34. 34. Level 1 - Services LevelThursday 16 June 2011
  35. 35. Performance SSH, HTTP, NTPThursday 16 June 2011
  36. 36. Performance SSH, HTTP, NTP # Servers/AdminThursday 16 June 2011
  37. 37. Performance SSH, HTTP, NTP # Servers/Admin # Line of Code/DeveloperThursday 16 June 2011
  38. 38. “Gameable” http://kithicor.org/wp/wp-content/uploads/2010/01/gamerEvolution.jpgThursday 16 June 2011
  39. 39. Level 2 - Team LevelThursday 16 June 2011
  40. 40. Aggregation LB-WEB WEB WEB WEB 1 2 3Thursday 16 June 2011
  41. 41. Aggregation DEV Project Team DEV DEV DEVThursday 16 June 2011
  42. 42. Burn up chartThursday 16 June 2011
  43. 43. Team UtilizationThursday 16 June 2011
  44. 44. “Velocity also has a direction component” http://www.randomstew.com/wp-content/uploads/mooneye.jpgThursday 16 June 2011
  45. 45. What about efficiency? http://www.flickr.com/photos/jdiggans/92981578/Thursday 16 June 2011
  46. 46. http://sievertschreiber.files.wordpress.com/2009/12/good-code-is-measured-in-wtf-per-minute.jpgThursday 16 June 2011
  47. 47. Incentives & Behaviors http://www.flickr.com/photos/turtlemadness/2439050717/Thursday 16 June 2011
  48. 48. Development Operations Level 3 - Integration Levelhttp://www.xgn.nl/images/upload/20081020225036.jpg http://www.kawaiikakkoiisugoi.com/wp-content/uploads/2011/03/pacman-screen.jpgThursday 16 June 2011
  49. 49. Integration WEB APP DBThursday 16 June 2011
  50. 50. Collaboration DEV QA OPS # of deploys / dayhttp://www.sage.org/lists/sage-members-archive/2001/msg01208.htmlhttp://www.datacenterknowledge.com/archives/2009/12/30/how-many-servers-can-one-admin-manage/http://fc04.deviantart.net/fs70/f/2010/108/2/3/Pac_Man_Widescreen_by_molotov_arts.jpgThursday 16 June 2011
  51. 51. Counting Collaboration More interaction doesn’t mean a better party • http://www.nature.com/news/2010/101013/full/news.2010.538.htmlThursday 16 June 2011
  52. 52. Measuring Collaboration? “Play Nice Study” http:///www.healthylarimer.org/documents/PlayNice.pptThursday 16 June 2011
  53. 53. Levels Collaboration Collaboration Coalition Coordination CooperationThursday 16 June 2011
  54. 54. Cooperation provide information somewhat defined roles formal communication decisions made independentlyThursday 16 June 2011
  55. 55. •share information & resources •defined roles •frequent communication Coordination •some shared decision makingThursday 16 June 2011
  56. 56. •share ideas •share resources Coalition •frequent & prioritized communication •all members have a vote on decisionsThursday 16 June 2011
  57. 57. Collaboration •members belong to one system •frequent communication characterized by mutual trust •consensus on all decisionsThursday 16 June 2011
  58. 58. Knowledge sharing http://www.au.af.mil/au/awc/awcgate/navy/onr_metrics_on_collab.pdfThursday 16 June 2011
  59. 59. Team Resilience 10th Floor Test Truck Factor to take a random machine in your infrastructure ,drop it from the 10th floor, and be able to restore your infrastructure to working within 5-10 minute The number of people on your team who have to be hit with a truck before the project is in serious trouble Resilience should be a 4th management objective, alongside Better/Faster/Cheaper John Allspaw http://www.kitchensoap.com/ Thursday 16 June 2011
  60. 60. Black Hole Theory Absorbs light, to see the effects you need to look at nearby objects http://www.collaborate.com/blog/2010/01/28/musing-metrics-collaborationThursday 16 June 2011
  61. 61. Quest for proof • Innovating and/or providing decision support • Saving time or Money (tangible but less than (tangible) quality) • Increase Quality • Easing access to and (tangible... but less so) interactions with subject- matter experts (intangible) http://communityandbusiness.groupsite.com/group/Collaboration20/post/measuring-the-value-of-collaboationThursday 16 June 2011
  62. 62. More vs Better Widgets Add more servers Tune existing servers Process Improvement Quality Improvement Easier More Difficult!! http://socialconsulting.wordpress.com/2009/02/24/measuring-collaboration/Thursday 16 June 2011
  63. 63. Collaboration Challenge “Maximize Team Utility”Thursday 16 June 2011
  64. 64. Level 4 - Business LevelThursday 16 June 2011
  65. 65. AHA KA-CHING!Thursday 16 June 2011
  66. 66. Return on Investment http://www.flickr.com/photos/hanekomu/4481674104/Thursday 16 June 2011
  67. 67. Inventory (WIP)Thursday 16 June 2011
  68. 68. Time to Market http://stefanroock.wordpress.com/2010/03/02/kanban-definition-of-lead-time-and-cycle-time/Thursday 16 June 2011
  69. 69. Mean Time Between Failure http://museumdosgames.blogspot.com/2010/04/game-over.htmlThursday 16 June 2011
  70. 70. Mean Time Between ResolutionThursday 16 June 2011
  71. 71. Technical DebtThursday 16 June 2011
  72. 72. LeaksThursday 16 June 2011
  73. 73. Debt Metaphor not all debt is created equal http://www.flickr.com/photos/shoot-art/4536509717/Thursday 16 June 2011
  74. 74. Customer Responsiveness Jim HighsmithThursday 16 June 2011
  75. 75. Types of Tech debt “oops” “yeah I know” unintentional intentional “we’ll fix it the next minor “we’ll fix it the next major revision” revision” short term (tactical) long term (strategical) http://forums.construx.com/blogs/stevemcc/archive/2007/11/01/technical-debt-2.aspxThursday 16 June 2011
  76. 76. Gartner • http://www.castsoftware.com/resources/materials/analyst/Gartner-How_to_Monetize_Application_Technical_Debt.pdfThursday 16 June 2011
  77. 77. Vicious Cycle of The Vicious Cycle of TechnicalDebt Technical Debt (More) Relentless Pressure Take Diminished Technical Dev Velocity Debt Technical Fail to Pay Debt Debt Back Accrues Neglect Maintenance http://theagileexecutive.com/2010/09/20/how-to-break-the-vicious-cycle-of-technical-debt/ Israel Gat 1Thursday 16 June 2011
  78. 78. Symptoms - Measurable • Poor Customer Responsiveness • Long Delivery Times • Late deliveries • Lots of defects (Toxic Code) • Rising Development costs • Frustrated & Poor performing Teams http://www.theagileengineer.com/public/Home/Home_files/TechnicalDebt_published.pdfThursday 16 June 2011
  79. 79. Secret Sauce http://radar.oreilly.com/archives/2007/10/Ops-ROI-jesserobbins.htmlThursday 16 June 2011
  80. 80. The Team Knows http://kingsinsight.com/2010/07/31/ask-the-team-about-technical-debt-the-creeping-doom-graph/Thursday 16 June 2011
  81. 81. Failure Value Demand % Demand As a Team http://farm4.static.flickr.com/3167/2769773425_16c235dc28.jpgThursday 16 June 2011
  82. 82. Level 5 - End User LevelThursday 16 June 2011
  83. 83. End User Perspective • END to END monitoring • User Experience (browser testing) • External parties (Email, ISP, Plugins)Thursday 16 June 2011
  84. 84. Participatory Design http://www.informaworld.com/smpp/content~db=all~content=a790786652Thursday 16 June 2011
  85. 85. “Crowd” - Monitoring Sudden Change in User Activity Pattern Detection of problem? A/B TestingThursday 16 June 2011
  86. 86. Spend more time Automation => creatively , early in the process Traditional Engineering http://www.informaworld.com/smpp/content~db=all~content=a790786652Thursday 16 June 2011
  87. 87. Design Strategies To Design is Human To Design does not require super powers Design used to be a collective or shared ability http://www.amazon.com/Research-Design-Thinking-Nigel-Cross/dp/9062757960 p 4 http://www.psych.lancs.ac.uk/people/lindenball.htmlThursday 16 June 2011
  88. 88. Open Innovation Wikipedia Open Source SharedConfig Management Shared Infrastructures http://en.wikipedia.org/wiki/Open_innovation http://www.clickadvisor.com/downloads/Bughin_Next_Step_in_Open_Innovation.pdfThursday 16 June 2011
  89. 89. Novice vs Expert • depth first • breath first , depth dives • vertical oriented • horizontal oriented • can not deal with • uncertainty = options uncertainty Linden Ballhttp://www.amazon.com/Research-Design-Thinking-Nigel-Cross/dp/9062757960 p 147 http://www.psych.lancs.ac.uk/people/lindenball.htmlThursday 16 June 2011
  90. 90. Continuous Evolution “Problem space evolves with the solution” http://www.amazon.com/Research-Design-Thinking-Nigel-Cross/dp/9062757960 http://www.psych.lancs.ac.uk/people/lindenball.htmlThursday 16 June 2011
  91. 91. Cultural TaxonomyThursday 16 June 2011
  92. 92. Cultural TaxonomyThursday 16 June 2011
  93. 93. tl;dr • avoid vanity metrics • insight > data • don`t hate the player • hate the game • business value good • beware of technical debtThursday 16 June 2011
  94. 94. Questions?Thursday 16 June 2011
  95. 95. Thursday 16 June 2011
  96. 96. Thank you!Thursday 16 June 2011
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×