Agile Metrics: Make Better Decisions with Data

K4
Keynote
11/17/2016 4:15:00 PM
Agile Metrics: Make Better Decisions
with Data
Presented by:
Larry Maccherone
AgileCraft
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888--‐268--‐8770 ·∙ 904--‐278--‐0524 - info@techwell.com - http://www.stareast.techwell.com/

Larry Maccherone
AgileCraft
An industry-recognized leader in agile, metrics, and visualization, Larry
Maccherone currently helps a number of companies with the design of their
analytics products including AgileCraft and Pendo.io. Previously, Larry led the
insights product line at Rally Software which enabled better decisions with data,
leveraged big data techniques to conduct groundbreaking research, and offered
the first-ever agile performance benchmarking capability. Before Rally, Larry
worked at the Software Engineering Institute for seven years conducting research
on software engineering metrics with a particular focus on reintroducing
quantitative insight back into the agile communities.

11/3/2016
1
Make BetterMake Better
Decisions
with Data
Larry Maccherone
LinkedIn.com/in/LarryMaccherone

11/3/2016
2
What?What?
So what?
NOW WHAT?
What?

11/3/2016
3
So what?

11/3/2016
4
Visualization is like photography.
Impact is a function of focus,
illumination, and perspective.
What?
Credit: Edward Tufte
NOW WHAT?

11/3/2016
5
D ’tDon’t
Launch!
Prevent your own
disastrous decisions

11/3/2016
6
Larry Maccherone
Cognitive bias
works against good decisions

11/3/2016
7
We don't see things
the way they are.
We see things
the way we are.
LinkedIn.com/in/LarryMaccherone LinkedIn.com/in/LarryMaccherone
~The Talmud
Next slide is a movie
click to play

11/3/2016
8
Denying the Evidence
Truths about cognitive bias
1. Very few people are immune to it.
2 We all think that we are part of that2. We all think that we are part of that
small group.
3. You can be trained to get much, much better.
Douglass Hubbard – How to Measure Anything
4. We do a first-fit pattern match. Not a best-fit pattern
match And we only use about 5% of the information
match. And we only use about 5% of the information
to do the matching.
5. We evolved to be this way (survival trait).

11/3/2016
9

11/3/2016
10
An example of overcoming
cognitive bias
We are overconfident when assessing our own uncertainty
But, training can “calibrate” people so that of all the times they
say they are X% confident, they will be right X% of the time
Trained/Calibrated
Untrained/Uncalibrated
Statistical Error
“Ideal” Confidence
50%
60%
70%
80%
90%
100%
75 71 65 58
21
17
68 152
65
45
21
PercentCorrect
# f R
30%
40%
50% 60% 80% 90% 100%
25
70%
Assessed Chance Of Being Correct
P
99 # of Responses
Copyright HDR 2007
dwhubbard@hubbardresearch.com

11/3/2016
11
Equivalent Bet calibration
What year did Newton published the Universal Laws of
Gravitation?
Pick year range that you are 90% certain it would fall within.
Win $1,000:
1. It is within your range; or
2. You spin this wheel and it lands green
10%
Adjust your range until 1 and 2 seem equal.
Even pretending to bet money works.
90%
Agile Teams Programs andAgile Teams, Programs, and
Portfolios
benefit from similar
calibration exercises

11/3/2016
12
Every decision is a
forecast!
You are forecasting thatg
your choice will have better
outcomes than the other
alternatives
alternatives

11/3/2016
13
How to avoid cognitive bias in decision making
 Don't focus on consensus.
Ritual dissent is a much more successful approach.
 “But that doesn’t explain _______”.
An FBI agent knew that some folks were being trained to fly
but not take off and land.
 Assign someone the role of devil’s advocate
 Assign someone the role of devil s advocate.
Israel’s 10th man.
In other words… Really consider the other ALTERNATIVES
Types of bias
http://srconstantin.wordpress.com/2014/06/09/do-rationalists-exist/

11/3/2016
14
For a given alternative, let:
Pg = Probability of good thing happening
V = “Value” of good thing happening
Vg = “Value” of good thing happening
Then:
Value of the alternative = Pg × Vg

11/3/2016
15
AA
lean/agile product
management example
$8M
Best case
(25%)
Likely case
(50%)
Worst case
(25%)
1
PW × VW = .25 × -$1.00M = -$0.25M
PL × VL = .50 × $1.00M = $0.50M
P × V 25 × $8 00M $2 00M
$1M
$1M
1
$2M$2M$1M2
PB × VB = .25 × $8.00M = $2.00M
-----------
$2.25M
PW × VW = .25 × $1.00M = $0.25M
PL × VL = .50 × $2.00M = $1.00M
PB × VB = .25 × $2.00M = $0.50M
-----------
Which strategy is best…
…for your company?
…for your career?
$1.75M

11/3/2016
16
If you get only 1 project then
strategy 2 is better
75% of the time
If you get ∞ projects then
strategy 1 is better
100% of the time
How many projects do you need for
strategy 1 to be better
more often than not?

11/3/2016
17
Play with it yourself at:
http://jsfiddle.net/lmaccherone/j3wh61r7/

11/3/2016
18
Emotion and bias plays a part
Did any of you get emotional
about the $1M loss?
Did any of you want to
question the $8M number?
It’s critical to
It s critical to…
…eliminate fear from the equation
…change the nature of the conversation

11/3/2016
19
Argument is about who is right.
Decision making is about what is right.
AAn
agile delivery date forecast
example

11/3/2016
20
Monte Carlo Forecasting
What it looks like
Live demo: http://lumenize.com (use Chrome)
Seek toSeek to
change the nature of
the conversation
the conversation

11/3/2016
21
Criteria for
great visualizationg
Credits:
Edward Tufte (mostly)
Edward Tufte (mostly)
Stephen Few
Gestalt School of Psychology
1. Answers the question, "Compared with what?”
(So what?)
 Trends
 Benchmarks

11/3/2016
22
2. Shows causality, or is at least informed by it
The primary chart used by the
NASA scientists showed O-ring
failure indicators by launch datefailure indicators by launch date.
Tufte's alternative shows the same
data by the critical factor,
temperature.
The fateful shuttle launch occurred
at 31 degree Tufte's visualization
at 31 degree. Tufte s visualization
makes it obvious that there is great
risk for any launch at temperatures
below 66 degrees.
3. Tells a story with whatever it takes
 Still
 Moving
 Numbers
 Graphics
And …
 Maybe some fun

11/3/2016
23
4. Is credible
 Calculations explained
 Sources
 Assumptions
 Who (name drop?)
 Drill-down
 How?
 Etc.
5. Has business value
(or value in it’s social context)
like Vic Basili’s
Goal-Question-Metric (GQM)
but without
ISO/IEC 15939 baggage
The ODIM framework
D
I N S I G H T
M E A S U R E
THINK
EFFECT
O U T C O M E
D E C I S I O N
THINK

11/3/2016
24
6. Shows comparisons easily (1)
aka:
Save the “pie” for dessert Credit:
• Stephen Few (Perceptual Edge)
• http://www.perceptualedge.com/ar
ticles/08-21-07.pdf
Can you compare the market share from one year to the next?
Q i kl Whi h t i i h th f t t?Quickly: Which two companies are growing share the fastest?
One pie chart is bad. Multiple pie charts are worse!!!

11/3/2016
25
How about now?
Can you compare the
market share from one
year to the next?
7. Allows you to
see the forest
ANDAND
the trees

11/3/2016
26
8. Informs along multiple dimensions (1)
9. Leaves in the
numbers
wherewhere
possible

11/3/2016
27
10. Leaves out glitter
Examples of how NOT to do it.
Top 10 criteria for great visualization
1. Answers the question,
"Compared with what?”
6. Shows
differences
Credits:
• Edward Tufte
• Stephen Few
• Gestalt
(School of Psychology)
(SO What?)
2. Shows causality, or is at least
informed by it.
(NOW WHAT?)
3. Tells a story with whatever it
takes.
easily.
7. Allows you to see the forest
AND the trees.
8. Informs along multiple
dimensions.
9 Leaves in the numbers where
takes.
4. Is credible.
5. Has business value or impact in
its social context.
9. Leaves in the numbers where
possible.
10. Leaves out glitter.
11. Uses good visual grammar

11/3/2016
28
“They” say…
Nobody knows what’s gonna happen
next: not on a freeway, not in an
airplane, not inside our own bodies
d t i l t t k ithand certainly not on a racetrack with
40 other infantile egomaniacs.
– Days of Thunder
Trying to predict the future is like
trying to drive down a country road
at night with no lights while looking
t th b k i d
out the back window.
– Peter Drucker
Never make predictions, especially
about the future.
– Casey Stengel
When you come to a
fork in the road…
take it!
~Yogi Berra
What?
the metrics and analysis
S h t?
So what?
how it compares/trends
what it means
NOW WHAT?
every decision is a forecast

11/3/2016
29
Questions?

Agile Metrics: Make Better Decisions with Data

Recommended

Recommended

More Related Content

Similar to Agile Metrics: Make Better Decisions with Data

Similar to Agile Metrics: Make Better Decisions with Data (20)

More from TechWell

More from TechWell (20)

Recently uploaded

Recently uploaded (20)

Agile Metrics: Make Better Decisions with Data