Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

IT Metrics in Real Life

81 views

Published on

It's all about measuring, and that's just the easy part. The hard part is focusing on what really matters.
I'll show you a bunch of metrics at different levels and from different perspectives, trying to make some sense out of them. I'll try to expand the current vision of what could be a metric, how it affects you and how you can choose your next ones.
The main goal is being able to understand what you need to monitor and improve in order to get better at what you're doing.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

IT Metrics in Real Life

  1. 1. IT Metrics in Real Life IMPROVING YOURSELF BY MEASURING RIGHT
  2. 2. Hello! I AM FABIO CICERCHIA Chief Technology Officer @ Skuola.net You can find me at: @fabiocicerchia
  3. 3. Let’s Clarify ‘em First
  4. 4. What’s a Metric? Metrics are parameters or measures of quantitative assessment used for measurement, comparison, or to track performance or production. https://www.investopedia.com/terms/m/metrics.asp
  5. 5. Are they useful? ● Provide Transparency ● Highlight Improvements, Regressions or Mistakes ● Lead to Self Improvement ● Drive Decisions ● Reduce Costs ● Allow you to be Objective
  6. 6. What’s a KPI? A KPI is a measurable value that demonstrates how effectively a company is achieving key business objectives. Organisations use KPIs at multiple levels to track performance measures. https://www.klipfolio.com/resources/articles/what-is-a-key-performance-indicator
  7. 7. Are they useful? ● I’ll leave it to you
  8. 8. Recommendation #1 Measure what matters while it matters https://www.oreilly.com/ideas/ask-the-cto-measuring-team-performance
  9. 9. Recommendation #2 The focus should be on the goal not the metric
  10. 10. Recommendation #3 Don't use raw metrics as performance evaluation
  11. 11. An Example By The Book
  12. 12. McCalls’ Software Quality Factors https://www.researchgate.net/profile/Caroline_Jay/publication/264239621/figure/fig1/AS:614318999625766@1523476471560/McCalls-Software-Quality-Factors.png
  13. 13. - Auditability – The ease with which conformance to standards can be checked. - Accuracy – The precision of computation and control. - Communication Commonality – The degree to which standard interfaces, protocols and bandwidth are used. - Completeness – The degree to which full implementation of required function has been achieved. - Complexity – The degree to which the program is complex. - Conciseness – The compactness of the program in terms of lines of codes. - Consistency – The use of uniform design and documentation technique throughout the software development project. - Data Commonality – The use of standard data structures and types throughout the program. McCalls’ Software Quality Factors http://education.dewsoftoverseas.com/QE/QUickReference/Software%20Enginering/2.6.asp
  14. 14. - Error Tolerance – The damage that occurs when the program encounters an error - Execution Efficiency – The run time performance of a program. - Expandability – The degree to which architectural, data or procedural data can be extended. - Generality – The breadth of potential application of program components. - Hardware Independence – The degree to which the software is decoupled from the hardware on which it operates. - Instrumentation – The degree to which the program monitors it’s own operation and identifies errors that do occur. - Modularity – The functional independence of program components. McCalls’ Software Quality Factors (cont’d) http://education.dewsoftoverseas.com/QE/QUickReference/Software%20Enginering/2.6.asp
  15. 15. McCalls’ Software Quality Factors (cont’d) - Operability – The ease of operation of a program. - Security – The availability of mechanisms that control or protect programs and data. - Self-Documentation – The degree to which the source code provides meaningful documentation. - Simplicity – The degree to which a program can be understood without difficulty. - Software System Independence – The degree to which the program is independent of nonstandard programming language features, operating system characteristics, and other environmental constraints. - Traceability – The ability to trace a design representation or actual program component back to requirements. - Training – The degree to which the software assists in enabling new users to apply the system. http://education.dewsoftoverseas.com/QE/QUickReference/Software%20Enginering/2.6.asp
  16. 16. Metrics by Perspectives
  17. 17. Dev’s Perspective
  18. 18. LOC - Lines Of Code - Commented Lines - Logic Lines - New Lines vs Deleted Lines - Impact (new block vs lots of small changes) - Bugs Rate
  19. 19. Commit Rate
  20. 20. Cyclomatic Complexity https://www.researchgate.net/profile/Vladimir_Riabov/publication/260038796/figure/fig2/AS:297180960182275@1447864870689/Three-methods-of-evaluating-the-cyclomatic-complexity-of-the-graph.png
  21. 21. C.R.A.P Index (Change Risk Analysis and Prediction) https://i.stack.imgur.com/x3XAY.png
  22. 22. Code Duplication % https://i.pinimg.com/originals/8a/39/28/8a3928c74f0e472624a48b741f8a21bc.png
  23. 23. Code Coverage % https://i2.wp.com/blog.mallow-tech.com/wp-content/uploads/2018/05/code_coverage_example.png
  24. 24. Broken Tests % - Marked as Incomplete - Marked as Skipped - Risky - Warnings
  25. 25. % New Tests - Per Sprint - Per Project
  26. 26. Coding Style Violations https://www.integer-net.com/wp-content/uploads/sites/3/2018/07/grumphp_no.png
  27. 27. SonarQube Scores
  28. 28. Technical Debt https://blog.insight.symfony.com/img/screenshots/technical_debt_php.png
  29. 29. # 3rd Party Vendors http://knowledge.wharton.upenn.edu/wp-content/uploads/2016/12/legos.jpg
  30. 30. TTFC (Time To First Commit) ● How well-organised is the code base? ● How supportive is the team? ● How much confidence do we have in our tests? ● How mature is the CI pipeline? ● How is the dev environment?
  31. 31. # Pending PR
  32. 32. Cache Hit Ratio https://support.stackpath.com/hc/article_attachments/360002767066/Cache_Hit_Ratio.PNG
  33. 33. DevOp’s Perspective
  34. 34. Broken Builds https://d3r49iyjzglexf.cloudfront.net/blog/content/insights-1.0-large@2x-f9d4748edfb82ee810b499a39e0671bda4c9e99b849bde044e4ccdee2c5b8e08.gif
  35. 35. Build Time https://www.automatetheplanet.com/wp-content/uploads/2018/04/16-hours-bellatrix-build.png
  36. 36. Deploys of Unreviewed PR https://media-cdn.tripadvisor.com/media/photo-s/0e/21/22/31/the-three-monkeys-cafe.jpg
  37. 37. Deploys of Partially Fixed PR https://media-cdn.tripadvisor.com/media/photo-s/0e/21/22/31/the-three-monkeys-cafe.jpg
  38. 38. Deployment Frequency https://blog.newrelic.com/wp-content/uploads/EOYBlog_Graph_Opt1.jpg
  39. 39. Deployment Time
  40. 40. Change Fail % - Rollback - Manual Patches
  41. 41. # Application Errors https://blog.sentry.io/img/post-images/epsagon/monitored-backend.png
  42. 42. Sysadmin’s Perspective
  43. 43. Uptime https://www.pingdom.com/wp-content/themes/pingdom-2018/public/assets/images/uptime-monitoring/uptime-specific-still.png
  44. 44. MTTR (Mean Time To Recover) https://archive.sap.com/image/456241
  45. 45. MTBF (Mean Time Between Failures) https://archive.sap.com/image/456241
  46. 46. Size Error Logs https://preview.ibb.co/eaygbH/Capture_d_e_cran_2018_02_08_11_16_54.png
  47. 47. Free Disk Space % https://i.stack.imgur.com/cRr4H.jpg
  48. 48. Disk I/O https://haydenjames.io/linux-server-performance-disk-io-slowing-application/
  49. 49. Bandwidth Saturation %
  50. 50. # Infrastructure Alerts - Disaster - High - Average - All the others
  51. 51. Customer Support’s Perspective
  52. 52. Unsolved Issues/Tickets
  53. 53. % Returning Problems https://marketplace-cdn.atlassian.com/files/images/f16e86e7-c5ec-41ae-8c0a-8b0cbec978d4.png
  54. 54. FRT (First Response Time) https://www.geckoboard.com/assets/1-first-response-time-visualizations-customer-support-metric.png
  55. 55. Defect Escape Rate https://image.slidesharecdn.com/webcastpresentation-6-140822160616-phpapp01/95/webcast-presentation-accelerate-continuous-delivery-with-development-testing-and-rational-team-concert-6-638.j pg
  56. 56. Project Manager’s Perspective
  57. 57. Lead Time https://image.slidesharecdn.com/leanleadershipwebinar-part3of3slides-161029141925/95/lean-leadership-part-3-of-3-30-638.jpg
  58. 58. Process Time https://image.slidesharecdn.com/leanleadershipwebinar-part3of3slides-161029141925/95/lean-leadership-part-3-of-3-30-638.jpg
  59. 59. Total vs Open https://www.glostone.com/wp-content/uploads/2016/10/piles-of-paperwork.png
  60. 60. Closed Issues/Tickets - Per Person - Per Project - Per Team - Per Urgency
  61. 61. WIP Issues/Tickets - Per Person - Per Project - Per Team - Per Urgency
  62. 62. Estimates Accuracy https://manifesto.co.uk/wp-content/uploads/2014/08/estimating-in-agile-and-scrum.png
  63. 63. Velocity http://www.softwaretestingstudio.com/wp-content/uploads/2017/05/Initial-Agile-Velocity.png
  64. 64. Burndown Variance https://upload.wikimedia.org/wikipedia/commons/0/05/SampleBurndownChart.png
  65. 65. APF (Augmented Pony Factor) The lowest number of committers whose total contribution constitutes the majority of the codebase https://ke4qqq.wordpress.com/2015/02/08/pony-factor-math/
  66. 66. HR’s Perspective
  67. 67. Tech Test Passing Rate https://i.kinja-img.com/gawker-media/image/upload/s--EWLoqHzx--/c_scale,fl_progressive,q_80,w_800/18kz3r7mcex5bgif.jpg
  68. 68. Team Morale https://cdn-images-1.medium.com/max/1600/1*3LM47GMbaG9CvSCIHjjPYQ.png
  69. 69. Manager’s Perspective
  70. 70. PSV (Project Schedule Variance) http://agilenucleus.com/wp-content/uploads/2015/04/evmchart.png
  71. 71. PCV (Project Cost Variance) http://agilenucleus.com/wp-content/uploads/2015/04/evmchart.png
  72. 72. EV (Earned Value) https://www.pmi.org/kasimage/f58c4893-c369-409a-acf3-227ce7f0074e/p_2.jpg
  73. 73. Overall Budget - Infrastructure (Cloud / Bare Metal) - Equipment (PC, Devices, Dongles, …) - Software (Licences, …) - SaaS - Personnel - Consulting - Outsourcing - Training (Workshops, Conferences, Books, …) - Recreational (Pastries, Drinks, Snacks, …)
  74. 74. EOL Running Stuff https://zdnet4.cbsistatic.com/hub/i/2018/10/14/7b07994d-008a-4f9b-82f4-01af045f1646/7f62145966cfb80731a8e9676cd59c13/php-eols.png https://symfony.com/doc/4.0/_images/release-process.jpg https://www.dev-metal.com/wp-content/uploads/2014/12/eol-centos.png
  75. 75. New Tech Deployed - ThoughtWorks Radar - Zalando Radar - Codemotion Community Radar (WIP)
  76. 76. Employee Turnover Rate https://www.officevibe.com/blog/how-to-calculate-employee-turnover
  77. 77. Bus Factor https://cdn-images-1.medium.com/max/1200/0*A7I4FMXuGkcmJ95A.
  78. 78. How Can You Get The Data?
  79. 79. - Custom Metrics - APM (Application Performance Monitoring) - RUM (Real User Monitoring) Application Telemetry
  80. 80. Internal Tools - GitHub/GitLab/BitBucket - Sentry - SonarQube - SiteSpeed.io - StatsD - Ticket System - Zabbix/Nagios/Cacti/New Relic/… - ...
  81. 81. What Else?!
  82. 82. Graphs
  83. 83. Trends
  84. 84. Forecasts
  85. 85. Aggregates
  86. 86. KPIs
  87. 87. Pick the Metrics that have the Most Sense For You
  88. 88. Now go, and do your homeworks
  89. 89. In-House Tools
  90. 90. Scripting
  91. 91. Excel
  92. 92. InfluxDB & Grafana
  93. 93. ELK
  94. 94. References ● https://blog.usenotion.com/13-essential-software-development-metrics-to-ensure-quality-219cfc264ed1 ● https://www.frontrowagile.com/blog/posts/69-30-metrics-for-agile-software-development-teams ● https://ardalis.com/static-code-analysis-and-quality-metrics ● https://github.com/mre/awesome-static-analysis ● https://www.slideshare.net/JerryTan2/metrics-driven-dev-ops-2017 ● https://www.slideshare.net/mattiabattiston/kanban-metrics-in-practice-for-leading-continuous-improvement ● https://github.com/chaoss/metrics ● http://shop.oreilly.com/product/0636920020134.do
  95. 95. Thanks! ANY QUESTIONS? You can find me here: https://fabiocicerchia.it

×