Slay the Dragons of Agile Measurement

AW4
Session
6/8/16 11:30 AM

Slay the Dragons of Agile Measurement

Presented by:

Larry Maccherone
AgileCraft

Brought to you by:

350 Corporate Way, Suite 400, Orange Park, FL 32073
888---268---8770 ·· 904---278---0524 - info@techwell.com - http://www.techwell.com/

Larry Maccherone
AgileCraft

An industry-recognized leader in agile, metrics, and visualization, Larry
Maccherone currently helps a number of companies with the design of their
analytics products including AgileCraft and Pendo.io. Previously, Larry led the
insights product line at Rally Software which enabled better decisions with data,
leveraged big data techniques to conduct groundbreaking research, and offered
the first-ever agile performance benchmarking capability. Before Rally, Larry
worked at the Software Engineering Institute for seven years conducting research
on software engineering metrics with a particular focus on reintroducing
quantitative insight back into the agile communities.

https://LinkedIn.com/in/LarryMaccherone
The Eight
Dragons of
Agile
Measurement
Topic presented by:
Larry Maccherone
LinkedIn.com/in/LarryMaccherone

Jean
Tabaka
She taught us how
to collaborate. She
made us all better.
She made us want
to be better. The
agile community
has lost a most
insightful, caring,
inspiring person.

Larry Maccherone
LinkedIn.com/in/LarryMaccherone

Using measurement in
an agile environment
can be dangerous

The Dragons of Agile Measurement
If you do metrics wrong, you will harm your agile transformation
1. Dragon: Measurement as a lever
Slayer: Measurement as feedback
2. Dragon: Unbalanced metrics
Slayer: 1 each for Do it
fast/right/on-time, and Keep doing it
3. Dragon: Metrics can replace
thinking
Slayer: Metrics compliment thinking
4. Dragon: Expensive metrics
Slayer: 1st work with the data you
are already passively gathering
5. Dragon: Using a convenient
metric
Slayer: Outcomes  Decisions 
Insight  Metric (ODIM)
6. Dragon: Bad analysis
Slayer: Simple stats and
simulation
7. Dragon: Single outcome forecasts
Slayer: Forecasts w/ probability
8. Dragon: Human emotion and bias
Slayer: Tricks to avoid your own
biases and overcome those of
others

Dragon #1
Using measurement as a
lever to directly drive
behavior

Manipulating
Others
Dragon #1
Using metrics as a
lever to drive
someone else’s
behavior

Self
Improvement
Dragon slayer #1
Using metrics to
reflect on your own
performance

Dragon #2
Unbalanced Metrics

Do it
fast
Do it
on
time
Do it
right
Keep
doing
it
Balanced scorecard of metrics
at least one from each quadrant

Dragon #3
Metrics can replace thinking

Dragon #4
Expensive metrics

Favor metrics derived from
existing records

Dragon #5
Using a convenient metric
aka “Lamp post metrics”

Good players?
Monta Ellis
9th highest scorer
(8th last season)
Carmelo Anthony (Melo)
8th highest scorer
(3rd last season)

Not everything
that counts
can be counted,
and
not everything
that can be
counted,
counts.

Dragon slayer #5
ODIM
O U T C O M E
D E C I S I O N
I N S I G H T
M E A S U R E
THINK
EFFECT
like Vic Basili’s
Goal-Question-Metric (GQM)
but without
ISO/IEC 15939 baggage

Dragon #6
Bad analysis and
visualization

Top 10 criteria for great visualization
1. Answers the question,
"Compared with what?”
(SO What?)
2. Shows causality, or is at least
informed by it.
(NOW WHAT?)
3. Tells a story with whatever it
takes.
4. Is credible.
5. Has business value or impact in
its social context.
6. Shows
differences
easily.
7. Allows you to see the forest
AND the trees.
8. Informs along multiple
dimensions.
9. Leaves in the numbers where
possible.
10. Leaves out glitter.
Credits:
• Edward Tufte
• Stephen Few
• Gestalt
(School of Psychology)
11. Uses good visual grammar

Time check?
Skip the next 8 slides if it’s
now past half the allotted time.
They go over the elements of
good visual grammar.

11. Good visual grammar (1)
When to use a straight line series?
1. Cumulative data
(later data points include
items from earlier data points)
example: Burn or Scope
series
2. Benchmark for column series
3. Connected by “ordinal inertia”
(ordinal usually = temporal)
However, spline is usually
what you want in this
instance.

When to use a dashed line?
1. “Ideal” or reference
2. Forecast
3. Regression or fit
line

When to use a column (vertical bar)
series?
1. Counts/sums independent of
neighbors
example: Throughput or
velocity chart

When to use a bar (horizontal)
series?
(Rarely)
1. The x-axis variable is causal?
2. In a table
3. Stylistic reasons
• Length of labels
• Two-variable display

When to use stacked area?
1. Multiple cumulative series
(the ordinal sum means
something)
example: a cumulative
flow diagram (CFD)

When to use a single area series or
multiple (non-cumulative) area
series?
NEVER?

When to use a single spline series?
1. Filling in for missing granularity
2. Rolling average

When to use a single scatter series?
1. Two variables that are
imperfectly
correlated

Dragon #7
Single outcome forecasts

Monte Carlo Forecasting
What it looks like
Live demo: http://lumenize.com (use Chrome)

Seek to
change the nature of
the conversation

Every decision is a
forecast!

You are forecasting that
your choice will have better
outcomes than the other
alternatives

So…
quality of decision depends upon:
1. alternatives considered, and
2. models used to forecast the
outcome of those alternatives.
Probabilistic models are superior

For a given alternative, let:
Pg = Probability of good thing happening
Vg = “Value” of good thing happening
Then:
Value of the alternative = Pg × Vg

An
lean/agile product
management example

$8M
Best case
(25%)
$1M
Likely case
(50%)
$1M
Worst case
(25%)
1
$2M$2M$1M2
Which strategy is best…
…for your company?
PW × VW = .25 × -$1.00M = -$0.25M
PL × VL = .50 × $1.00M = $0.50M
PB × VB = .25 × $8.00M = $2.00M
-----------
$2.25M
…for your career?
PW × VW = .25 × $1.00M = $0.25M
PL × VL = .50 × $2.00M = $1.00M
PB × VB = .25 × $2.00M = $0.50M
-----------
$1.75M

If you get only 1 project then
strategy 2 is better
75% of the time
If you get ∞ projects then
strategy 1 is better
100% of the time
How many projects do you need for
strategy 1 to be better
more often than not?

Play with it yourself at:
http://jsfiddle.net/lmaccherone/j3wh61r7/

Dragon #8
Human emotion and bias

https://LinkedIn.com/in/LarryMaccherone @LMaccherone @TheAgileCraft
We don't see things
the way they are.
We see things
the way we are.
~The Talmud
Men's Airo Outdoor Shoes

Getting
probability
input you
can trust

Trained/Calibrated
Untrained/Uncalibrated
Statistical Error
“Ideal” Confidence
30%
40%
50%
60%
70%
80%
90%
100%
50% 60% 80% 90% 100%
25
75 71 65 58
21
17
68 152
65
45
21
70%
Assessed Chance Of Being Correct
PercentCorrect
99 # of Responses
We are inaccurate when assessing probabilities
Copyright HDR 2007
dwhubbard@hubbardresearch.com
But, training can “calibrate” people so that of all the times they
say they are X% confident, they will be right X% of the time

Equivalent Bet calibration
What year did Newton published the Universal Laws of
Gravitation?
Pick year range that you are 90% certain it would fall within.
Win $1,000:
1. It is within your range; or
2. You spin this wheel and it lands green
Adjust your range until 1 and 2 seem equal.
Even pretending to bet money works.
90%
10%

Now what?
• Questions?
• Day-long seminar on agile metrics
• Workshop to design your own
metrics regimen
• AgileCraft Demo
• Make contact with me on LinkedIn
https://linkedin.com/in/larrymaccherone
Now what?
• Meet me at the AgileCraft booth to
answer questions

“They” say…
Nobody knows what’s gonna happen
next: not on a freeway, not in an
airplane, not inside our own bodies
and certainly not on a racetrack with
40 other infantile egomaniacs.
– Days of Thunder
Trying to predict the future is like
trying to drive down a country road
at night with no lights while looking
out the back window.
– Peter Drucker
Never make predictions, especially
about the future.
– Casey Stengel

When you come to a
fork in the road…
take it!
~Yogi Berra

Now what?
• Questions?
• Day-long seminar on agile metrics
• Workshop to design your own
metrics regimen
• AgileCraft Demo
https://linkedin.com/in/larrymaccherone
Now what?
• Meet me at the AgileCraft booth to
answer questions
• Share webinar recording of this
content to co-workers who couldn’t
be here
https://attendee.gotowebinar.com/register
/5432450926944227330

Slay the Dragons of Agile Measurement

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Similar to Slay the Dragons of Agile Measurement

Similar to Slay the Dragons of Agile Measurement (20)

More from Josiah Renaudin

More from Josiah Renaudin (20)

Recently uploaded

Recently uploaded (20)

Slay the Dragons of Agile Measurement