This document discusses metrics and how to use them effectively. It covers what metrics are, why they are important, and provides examples of metrics to measure things like acquisition, activation, retention, and revenue for products. It also discusses visualizing metrics, potential tools to use, and some pitfalls to avoid like vanity metrics, useless metrics, and weaponized metrics that are used for harm rather than good. The overall message is that metrics should be used to guide improvements, identify constraints, and measure outcomes, not just for their own sake or in ways that don't reflect the true performance of a business.

1. Metrics
You are what you measure
@robdcrowley
robdcrowley

2. Empower you with new ideas to bring your team together!
Metrics. What are they? What to measure? How to visualise?
Tooling. Some secret sauce to make our lives easier
Metrics as a catalyst for cultural change
Goals for Session

3. Metrics 101
What metrics are and why we measure

4. To inform
To compare
To motivate
To understand
To improve
To protect
Not: Just because successful companies do!
Why do we measure

5. A quantifiable measure of any component or process
used to gauge the performance of your business
– Application
– Business
– People
– Process
– System
What is a metric

6. Basis of measurement
Assumptions
Level and usage
Desired trend
When to use
When to stop
How to game
Warnings
Facets of a metric

7. Red is good,
Green is worthless

8. Metrics 201
What should you measure

9. “Lack of direction, not lack of time, is the
problem. We all have twenty-four hour days.”
- Zig Zigler

10. Pivot: Base conversations on metrics instead of
non actionable opinions
Use stakeholder input as the basis for your initial
set of metrics
– “What do you want out of this?”
– “How quickly do you want this?”
Metrics can help guide us

11. Awesome book, poor title
Core Principles
– Minimum Viable Product (MVP)
– Continuous Deployment
– Split Testing (aka A/B Testing)
– Actionable Metrics
– Pivot
– Innovation Accounting
– Build-Measure-Learn Loop

13. Acquisition – finding new users
Activation – getting users to give your product a try
Retention – making sure those users stick around
Referral – have your loyal users invite others
Revenue – making some money from all this
Note: These steps are not strictly sequential
Pirate Funnel (AARRR!)

14. Do not focus solely on metrics
that focus on reducing errors
Focus on achieving excellence
and positive outcomes will
ensue

15. DevOps State of the Union focuses on these four
core metrics
– Deployment Frequency
– Lead Time for Changes
– Mean Time to Recover (MTTR)
– Failed Change Rate
Example Metrics

16. Fact: We only realise business value from the
work we do when it’s released into production
The bottleneck or constraint is the limiting factor
on our ability to deliver more quickly
Strengthening any link other than the weakest is
a waste of time and effort
Use metrics to identify current constraint
Theory of Constraints

17. Metrics 301
How to visualise and share metrics

18. Display all metrics on a dashboard
– Business (i.e. Application dropouts)
– Dev (i.e. Performance metrics)
– Ops (i.e. Web Server CPU Usage)
Single URL for all data
Make it easy for colleagues to access dashboard
Visualisation

19. Use a tool that can handle different kinds of metrics
A few standouts are:
– StatsD/Graphite/Grafana
– Heka
– Logstash/Elastic Seach/Kibana
Tools

20. StatsD WhisperCarbon
Graphite
DashboardApps
UDP
TCP
HTTP
Graphite and StatsD

21. Metrics 401
Potential pitfalls when using metrics

22. Case Study: US Health Service
In the 1990s the US Health Service decided to
make hospital mortality rates public. What better
metric for hospital performance?
Observed Behaviour: Best way to improve rating
was not to admit critically ill patients in the first
place
You are what you measure

23. “The actual company values, as opposed to
the nice-sounding values, are shown by who
gets rewarded, promoted or let go”
- Patty McCord

24. Case Study: Enron
Enron which went bankrupt from fraud and whose
leaders went to jail, had the following company
values:
Integrity, Communication, Respect, Excellence
Where these really the core values at Enron?
You are what you measure

25. “In the absence of clarity around an
objective, any measure will do”
- Bob Paladino
Too Many Metrics

26. Depict rosiest picture possible but do not
accurately reflect the key drivers of your business
– Inaccessible
– Not auditable
– Not actionable
Vanity Metrics

27. Metrics based on output rather than outcomes
Aim to maximise outcomes and minimise output
– Total lines of code
– Number of bugs fixed
– Total hours worked
– Lines of code per developer
Useless Metrics

28. “We’re not doing this data analysis thing
just because. We’re gathering data points
because we’ve seen it work to great
effect in other organisations and industries.”
- Anonymous
Cargo Cult Metrics

29. Metrics being used for evil rather than for good
Confronting this organisational anti-pattern can
be a good first step
Information sharing is the best defense
Break down knowledge silos in the organisation
Weaponized Metrics

30. Questions