IT Metrics in Real Life
IMPROVING YOURSELF BY MEASURING RIGHT

Hello!
I AM FABIO CICERCHIA
Chief Technology Officer @ Skuola.net
You can find me at: @fabiocicerchia

Let’s Clarify ‘em First

What’s a Metric?
Metrics are parameters or measures of quantitative assessment used
for measurement, comparison, or to track performance or production.
https://www.investopedia.com/terms/m/metrics.asp

Are they useful?
● Provide Transparency
● Highlight Improvements, Regressions or Mistakes
● Lead to Self Improvement
● Drive Decisions
● Reduce Costs
● Allow you to be Objective

What’s a KPI?
A KPI is a measurable value that demonstrates how effectively a
company is achieving key business objectives.
Organisations use KPIs at multiple levels to track performance measures.
https://www.klipfolio.com/resources/articles/what-is-a-key-performance-indicator

Are they useful?
● I’ll leave it to you

Recommendation #1
Measure what matters while it matters
https://www.oreilly.com/ideas/ask-the-cto-measuring-team-performance

Recommendation #2
The focus should be on the goal not the metric

Recommendation #3
Don't use raw metrics as performance evaluation

An Example By The Book

McCalls’ Software Quality Factors
https://www.researchgate.net/profile/Caroline_Jay/publication/264239621/figure/fig1/AS:614318999625766@1523476471560/McCalls-Software-Quality-Factors.png

- Auditability – The ease with which conformance to standards can be checked.
- Accuracy – The precision of computation and control.
- Communication Commonality – The degree to which standard interfaces, protocols and bandwidth
are used.
- Completeness – The degree to which full implementation of required function has been achieved.
- Complexity – The degree to which the program is complex.
- Conciseness – The compactness of the program in terms of lines of codes.
- Consistency – The use of uniform design and documentation technique throughout the software
development project.
- Data Commonality – The use of standard data structures and types throughout the program.
McCalls’ Software Quality Factors
http://education.dewsoftoverseas.com/QE/QUickReference/Software%20Enginering/2.6.asp

- Error Tolerance – The damage that occurs when the program encounters an error
- Execution Efficiency – The run time performance of a program.
- Expandability – The degree to which architectural, data or procedural data can be extended.
- Generality – The breadth of potential application of program components.
- Hardware Independence – The degree to which the software is decoupled from the hardware on
which it operates.
- Instrumentation – The degree to which the program monitors it’s own operation and identifies
errors that do occur.
- Modularity – The functional independence of program components.
McCalls’ Software Quality Factors (cont’d)
http://education.dewsoftoverseas.com/QE/QUickReference/Software%20Enginering/2.6.asp

McCalls’ Software Quality Factors (cont’d)
- Operability – The ease of operation of a program.
- Security – The availability of mechanisms that control or protect programs and data.
- Self-Documentation – The degree to which the source code provides meaningful documentation.
- Simplicity – The degree to which a program can be understood without difficulty.
- Software System Independence – The degree to which the program is independent of nonstandard
programming language features, operating system characteristics, and other environmental
constraints.
- Traceability – The ability to trace a design representation or actual program component back to
requirements.
- Training – The degree to which the software assists in enabling new users to apply the system.
http://education.dewsoftoverseas.com/QE/QUickReference/Software%20Enginering/2.6.asp

Metrics by Perspectives

Dev’s Perspective

LOC
- Lines Of Code
- Commented Lines
- Logic Lines
- New Lines vs Deleted Lines
- Impact (new block vs lots of small changes)
- Bugs Rate

Commit Rate

Cyclomatic Complexity
https://www.researchgate.net/profile/Vladimir_Riabov/publication/260038796/figure/fig2/AS:297180960182275@1447864870689/Three-methods-of-evaluating-the-cyclomatic-complexity-of-the-graph.png

C.R.A.P Index (Change Risk Analysis and Prediction)
https://i.stack.imgur.com/x3XAY.png

Code Duplication %
https://i.pinimg.com/originals/8a/39/28/8a3928c74f0e472624a48b741f8a21bc.png

Code Coverage %
https://i2.wp.com/blog.mallow-tech.com/wp-content/uploads/2018/05/code_coverage_example.png

Broken Tests %
- Marked as Incomplete
- Marked as Skipped
- Risky
- Warnings

% New Tests
- Per Sprint
- Per Project

Coding Style Violations
https://www.integer-net.com/wp-content/uploads/sites/3/2018/07/grumphp_no.png

SonarQube Scores

Technical Debt
https://blog.insight.symfony.com/img/screenshots/technical_debt_php.png

# 3rd Party Vendors
http://knowledge.wharton.upenn.edu/wp-content/uploads/2016/12/legos.jpg

TTFC (Time To First Commit)
● How well-organised is the code base?
● How supportive is the team?
● How much confidence do we have in our tests?
● How mature is the CI pipeline?
● How is the dev environment?

# Pending PR

Cache Hit Ratio
https://support.stackpath.com/hc/article_attachments/360002767066/Cache_Hit_Ratio.PNG

DevOp’s Perspective

Broken Builds
https://d3r49iyjzglexf.cloudfront.net/blog/content/insights-1.0-large@2x-f9d4748edfb82ee810b499a39e0671bda4c9e99b849bde044e4ccdee2c5b8e08.gif

Build Time
https://www.automatetheplanet.com/wp-content/uploads/2018/04/16-hours-bellatrix-build.png

Deploys of Unreviewed PR
https://media-cdn.tripadvisor.com/media/photo-s/0e/21/22/31/the-three-monkeys-cafe.jpg

Deploys of Partially Fixed PR
https://media-cdn.tripadvisor.com/media/photo-s/0e/21/22/31/the-three-monkeys-cafe.jpg

Deployment Frequency
https://blog.newrelic.com/wp-content/uploads/EOYBlog_Graph_Opt1.jpg

Deployment Time

Change Fail %
- Rollback
- Manual Patches

# Application Errors
https://blog.sentry.io/img/post-images/epsagon/monitored-backend.png

Sysadmin’s Perspective

Uptime
https://www.pingdom.com/wp-content/themes/pingdom-2018/public/assets/images/uptime-monitoring/uptime-specific-still.png

MTTR (Mean Time To Recover)
https://archive.sap.com/image/456241

MTBF (Mean Time Between Failures)
https://archive.sap.com/image/456241

Size Error Logs
https://preview.ibb.co/eaygbH/Capture_d_e_cran_2018_02_08_11_16_54.png

Free Disk Space %
https://i.stack.imgur.com/cRr4H.jpg

Disk I/O
https://haydenjames.io/linux-server-performance-disk-io-slowing-application/

Bandwidth Saturation %

# Infrastructure Alerts
- Disaster
- High
- Average
- All the others

Customer Support’s
Perspective

Unsolved Issues/Tickets

% Returning Problems
https://marketplace-cdn.atlassian.com/files/images/f16e86e7-c5ec-41ae-8c0a-8b0cbec978d4.png

FRT (First Response Time)
https://www.geckoboard.com/assets/1-first-response-time-visualizations-customer-support-metric.png

Defect Escape Rate
https://image.slidesharecdn.com/webcastpresentation-6-140822160616-phpapp01/95/webcast-presentation-accelerate-continuous-delivery-with-development-testing-and-rational-team-concert-6-638.j
pg

Project Manager’s Perspective

Lead Time
https://image.slidesharecdn.com/leanleadershipwebinar-part3of3slides-161029141925/95/lean-leadership-part-3-of-3-30-638.jpg

Process Time
https://image.slidesharecdn.com/leanleadershipwebinar-part3of3slides-161029141925/95/lean-leadership-part-3-of-3-30-638.jpg

Total vs Open
https://www.glostone.com/wp-content/uploads/2016/10/piles-of-paperwork.png

Closed Issues/Tickets
- Per Person
- Per Project
- Per Team
- Per Urgency

WIP Issues/Tickets
- Per Person
- Per Project
- Per Team
- Per Urgency

Estimates Accuracy
https://manifesto.co.uk/wp-content/uploads/2014/08/estimating-in-agile-and-scrum.png

Velocity
http://www.softwaretestingstudio.com/wp-content/uploads/2017/05/Initial-Agile-Velocity.png

Burndown Variance
https://upload.wikimedia.org/wikipedia/commons/0/05/SampleBurndownChart.png

APF (Augmented Pony Factor)
The lowest number of committers whose total contribution
constitutes the majority of the codebase
https://ke4qqq.wordpress.com/2015/02/08/pony-factor-math/

HR’s Perspective

Tech Test Passing Rate
https://i.kinja-img.com/gawker-media/image/upload/s--EWLoqHzx--/c_scale,fl_progressive,q_80,w_800/18kz3r7mcex5bgif.jpg

Team Morale
https://cdn-images-1.medium.com/max/1600/1*3LM47GMbaG9CvSCIHjjPYQ.png

Manager’s Perspective

PSV (Project Schedule Variance)
http://agilenucleus.com/wp-content/uploads/2015/04/evmchart.png

PCV (Project Cost Variance)
http://agilenucleus.com/wp-content/uploads/2015/04/evmchart.png

EV (Earned Value)
https://www.pmi.org/kasimage/f58c4893-c369-409a-acf3-227ce7f0074e/p_2.jpg

Overall Budget
- Infrastructure (Cloud / Bare Metal)
- Equipment (PC, Devices, Dongles, …)
- Software (Licences, …)
- SaaS
- Personnel
- Consulting
- Outsourcing
- Training (Workshops, Conferences, Books, …)
- Recreational (Pastries, Drinks, Snacks, …)

EOL Running Stuff
https://zdnet4.cbsistatic.com/hub/i/2018/10/14/7b07994d-008a-4f9b-82f4-01af045f1646/7f62145966cfb80731a8e9676cd59c13/php-eols.png
https://symfony.com/doc/4.0/_images/release-process.jpg
https://www.dev-metal.com/wp-content/uploads/2014/12/eol-centos.png

New Tech Deployed
- ThoughtWorks Radar
- Zalando Radar
- Codemotion Community Radar (WIP)

Employee Turnover Rate
https://www.officevibe.com/blog/how-to-calculate-employee-turnover

Bus Factor
https://cdn-images-1.medium.com/max/1200/0*A7I4FMXuGkcmJ95A.

How Can You Get The Data?

- Custom Metrics
- APM (Application Performance Monitoring)
- RUM (Real User Monitoring)
Application Telemetry

Internal Tools
- GitHub/GitLab/BitBucket
- Sentry
- SonarQube
- SiteSpeed.io
- StatsD
- Ticket System
- Zabbix/Nagios/Cacti/New Relic/…
- ...

What Else?!

Graphs

Trends

Forecasts

Aggregates

KPIs

Pick the Metrics that have the
Most Sense For You

Now go, and do your homeworks

In-House Tools

Scripting

Excel

InfluxDB & Grafana

ELK

References
● https://blog.usenotion.com/13-essential-software-development-metrics-to-ensure-quality-219cfc264ed1
● https://www.frontrowagile.com/blog/posts/69-30-metrics-for-agile-software-development-teams
● https://ardalis.com/static-code-analysis-and-quality-metrics
● https://github.com/mre/awesome-static-analysis
● https://www.slideshare.net/JerryTan2/metrics-driven-dev-ops-2017
● https://www.slideshare.net/mattiabattiston/kanban-metrics-in-practice-for-leading-continuous-improvement
● https://github.com/chaoss/metrics
● http://shop.oreilly.com/product/0636920020134.do

Thanks!
ANY QUESTIONS?
You can find me here: https://fabiocicerchia.it