Evidenced based management - Presentation at Scrum Australia 24 oct 2018

Evidenced Based Management (EBM)
for Business Agility
Scrum Australia 24 October 2018
Mia Horrigan
Partner ZXM and Enterprise Agile Coach
page
01
Mia Horrigan@zenexmachina.com
Blog – www.zenexmachina.wordpress.com
Twitter - @miahorri
www.zenexmachina.com

Has our investment in Agile been
worth it?
2
Organisations invest in
agile processes, tools,
training, and coaching,
but how much are they
getting back?
Has product delivery
improved?
How much happier are
users and the business
customers?
Are employees
empowered and
enabled?

Successful Project ? Success =
• All requirements
delivered…
• By agreed-upon
date…
• For an agreed-
upon cost
“On track” = project
follows plan, hits
milestones

Traditional MeasuresActivity: reflect actions
taken e.g. No. of meetings,
No. of projects, No. people
in team, or cost per period
Output: reflect things that
were produced, No. of
releases, No. of defects
fixed, No. of features
delivered, or even velocity
Using progress as main
guide can be misleading
and misalign business and
delivery organisations
Activity Output

Waterfall Triangle Traditional Iron
Triangle assumes
Minimal change
“Plan — Do”
Problem: focus on
activity and output, not
outcomes and value
Measuring activity just
tells where time was
spent, not whether
value was produced

Agile TriangleAgile
Triangle enables
inevitable changes
“Envision — evolve”
Value is the reason
why we do things;
activity and output do
not tell you whether
what you are doing is
worthwhile or
successful

How is Value Measured?Success = value is
maximised
Quality and capability
are sustainable
Focus on outcomes,
not activity and output
Focus on Business
Value, improvements
and helps
accountability
Only 20% Useful 64% Rarely or Never Used
Chaos Report 2015 Standish Group

Agile Measures
Outcome: measures
that reflect the change
to customers or users.
Impact: Affect on the
company itself
Be value maximisers
What value did we
deliver and what was
the behaviour change
or increased outcome?
Outcome Impact

Evidenced Based Management
(EBM) Key Value Areas (KVAs)
Market Value
(Customer Value)
Organisational
Capability
EBM is an empirical
approach that measures
value delivered as evidence
of organisational agility
Gives organisations the
ability to measure the value
they deliver to customers
and the means by which
they deliver that value,
and to use those measures
to guide improvements in
both
Unrealised
Value
(UV)
Current
Value
(CV)
Ability to
Innovate
(A2I)
Time to
Market
(T2M)
Agility
Business Value
Source Evidenced Based Management Guide
https://www.scrum.org/resources/evidence-based-management

Use of a practice’s effectiveness
does mean we are delivering value
10
Organisations adopting agile product
delivery practices can easily lose
sight of their real goal of improving
the value, by focusing on improving
activities and outputs instead of
business outcomes
Agile is a means to an end, not the
end itself
The whole point of adopting agile
practices is to improve business
performance
When organisations lose sight of
this, managers ask questions that
seem sensible, but might create
unintended and undesirable
consequences
Is build automation
present?
Are code standards
being met?
Are test first practices
being used?
Is the team velocity
increasing? Are developers integrating
code frequently? How
frequently?What is the quality of
the code?

What we Measure is Important
What you measure
translates into what we
think is important
Typically teams are used
to metrics being used
against them
Velocity is data to help aid
team’s own forecasting
not a measure to compare
teams
Counting lines of code is
easy but doesn’t tell
anything about quality, the
functionality provided or
even the effectiveness
Simply
observing a
situation
necessarily
changes
that
situation
Human
tendency to
look for
answers
where it’s
easy to look

Measurement is Strategic
Without measuring
value, the success of any
agile initiative is based
on nothing more than
intuition and assumption
Helps us know if we are
moving closer to the
goals of the organisation
Understand what
actually will be valuable
that we’re not currently
delivering

EBM Approach to Improving Value
13
Evidence Based
Decision-Making is a
process for making
decisions about a
program, practice, or policy
based on Evidence and
informed by experiential
evidence
EBM approach suggests
four phases that enable
organisations to constantly
learn and improve the
value derived from software
investments
1. Measure KVMs
(Key Value Metrics)
2. Select KVAs
(Key Value Areas)
to Improve
3. Conduct
Experiments to
Improve Value
4. Evaluate
Outcome Results

If you had one wish:
What is one measure/s you would
want to improve?
page
014

Current Value
page
015

Reveals the value that the
organisation delivers to
customers, today
Goal: to maximise the value
an organisation delivers at
the present time; it considers
only what exists right now,
not what it might do in the
future
Reveals organisation’s
actual value in the
marketplace but has no
relevance on an
organisation’s ability to
sustain value in the future
Current Value (CV)

How happy are our stakeholders?• How happy are
customers today? Is
their happiness
improving or declining?
•
• How happy are
employees? Is their
happiness improving or
declining?
• How happy are
investors and other
stakeholders? Is their
happiness improving or
declining?
Leading indicators Lagging indicators
• Employee
Satisfaction
• Customer
Satisfaction
• Usage Index
• Revenue per
Employee
• Product Cost
Ratio

Personas
• Who are our
customers?
• What is their story?
• What are their goals,
motivations and pain
points?
• Why do they want to
work with us?
• What outcomes do
they want from our
products?
How would we measure
that!
Understand our Customers

Customers ValueWhat is the
Organisations Value
Proposition?
Why do our customers
work with us?
What outcomes do they
want from our
products?
How would we measure
that!

Current Value and Goal Driven
Prioritisation
Use Impact Mapping to:
• Be goal driven when
coming up with and
prioritising features
• Prioritise features
based on impacts
• Show progress towards
impacts. E.g. how
many features are
finished relate to each
impact? Guides
measurements of
impacts being realised
• Validate an existing list
of features back to a
goal to see if they are
relevant

Visualise goal driven
prioritisation for your
customer
Think Outcome rather
than Deliverable
Helps User
Stories/PBIs to be
outcome focused
Modified Impact Map
Goal – Persona – Impact – Outcome – PBI

Hypothesis Driven Delivery - Express Hypotheses
Adapted from Gothelf and Seiden: Lean UX
We believe [doing this] for [these people] will
achieve [this outcome] We will know that this
is true when we see [this measurement]
changed
Feature
Persona
Outcome
Measure

Current Value – Key Value Measures
KVM Measuring:
Revenue per Employee The ratio (gross revenue / # of employees) is a key competitive indicatorwithin anindustry.
Thisvaries significantlybyindustry.
Product Cost Ratio Total expenses and costs for the product(s)/system(s) being measured, including operational
costs compared to revenue.
Employee Satisfaction Some form of sentiment analysis to help gauge employee engagement, energy, and
enthusiasm.
Customer Satisfaction Some form of sentiment analysis to help gauge customer engagement and happiness with the
product.
Usage Index
Measurement of usage, by feature, to help infer the degree to which customers find the product
useful and whether actual usage meets expectations on how long users should be taking with a
feature.

Unrealised Value
page
024

The potential future value
that could be realised if
the organization was able
to perfectly meet the
needs of all potential
customers.
Goal: to maximise the
value that the organization
realises from a product
over time
Unrealised Value (UV)

What potential is to be gained?Can any additional money
be made in this market?
Is it worth the effort and
risk to pursue further
returns in this market?
Should further investments
be made to capture
additional Unrealised
Value?
• Competitor
strength/weakness
• Customer
acquisition or
defection
• Market share trends
• Overall market
growth/decline
relative to market
share trends

Balance of Current and Future
Benefits
Decision to invest in
one product means not
investing in others
Considering both CV
and UV provides a way
to balance present and
possible future benefits
Current Value
UnrealisedValue
Early version used to test
the market, but there is
great market potential so
investment warranted
High
HighLow
Low
Current “Cash
Cow” with limited
competitors may
not warrant new
investment unless
reinventing

When you test a hypothesis, what is success?
How long would it take for you to know whether
you’ve actually achieved..
• the outcome you wanted to achieve, and
• the impact you wanted to create?

Unrealised Value – Key Value Measures
KVM Measuring:
Market Share The relative percentage of the market controlled by the product.
Customer or user satisfaction gap The difference between a customer or user’s desired experience and their current experience.

Time to Market
(T2M)
page
030

Time to Market Organisation’s ability to
quickly deliver new
capabilities, services, or
products
Goal: to minimise the
amount of time it takes for
the organisation to deliver
value
Without actively
managing T2M, the ability
to sustain delivering value
in the future remains
uncertain
• How fast can the organisation learn from
new experiments?
• How fast can we learn from new
information and adapt?
• How fast can we deliver new value to
customers?

What does it take to go from idea
to value and improvement
opportunities?
How can you improve your
time to market and reduce
the time it takes you to get
feedback?
What are the steps needed
from forming an idea to
delivering value to
customers?

Visualise the delivery process
Create a value stream
map that represents your
delivery process
Start with the idea and
end with when the
customer or user realises
value
• What is the current time to
market?
• What bottlenecks are there?
• Identify areas to improve
delivery pipeline
automation, improve
maintainability or remove
technical debt to reduce
waste

T2M MeasuresHow long would it take
for to know whether
we’ve actually
achieved the outcome
we wanted to achieve
and the impact we
wanted to achieve?
• Frequency of Build
Success
• Build pass/fail
trends
• Release
Stabilisation trends
• MTTR – mean time
to Repair
• Cycle Time
• Release
Frequency
• Lead Time
• Time to Learn

T2M – Key Value Measures
KVM Measuring:
Build and integration frequency The number of integrated and tested builds per time period. For a team that is releasing
frequently or continuously, this measure is superseded by actual release measures.
Release Frequency
The number of releases per time period, e.g. continuously, daily, weekly, monthly, quarterly,
etc. This helps reflect the time needed to satisfy the customer with new and competitive
products.
Release Stabilization Period
The time spent correcting product problems between the point the developers say it is ready to
release and the point where it is actually released to customers. This helps represent the
impact of poor development practices and underlying design and code base.
Mean Time to Repair
The average amount of time it takes from when an error is detected and when it is fixed. This
helps reveal the efficiency of an organization to fix an error.
Cycle Time
The amount of time from when work starts on a release until the point where it is actually
released. This measure helps reflect an organization’s ability to reach its customer.
Lead Time
The amount of time from when an idea is proposed or a hypothesis is formed until a
customer can benefit from that idea. This measure may vary based on customer and
product. It is a contributing factor for
customer satisfaction.
Time-to-Learn The total time needed to sketch an idea or improvement, build it, deliver it to users, and learn
from their usage.

Ability to innovate
(T2I)
page
036

Ability to Innovate (A2I)Ability of a organisation
to deliver new
capabilities that might
better meet customer
needs
Goal: to maximise ability
to deliver new products
and innovative solutions
Ability to Innovate helps
avoid software that is
overloaded by low value
features
Continually Re-evaluate A2I by asking:
• What prevents the organisation from
delivering new value?
• What prevents customers or users from
benefiting from that innovation?

Impediments to delivering value
Reduces A2I
As low-value features
accumulate, more of the
budget and time is
consumed maintaining the
product, not increasing
capacity to innovate
Anything that prevents
users from benefitting from
innovation, such as hard to
install software or lack of
capabilities, will also
reduce A2I
•Maintaining
multiple code
branches or
product
versions
•Complex or
monolithic
application
architecture
•Insufficient
product-like
environments
to test on
•Lack of
operational
excellence
•Lack of
decentralised
decision-
making
•Spending too
much time
fixing defects
or reducing
technical debt
•Inability to hire
and inspire
talented,
passionate
team-members

A2I MeasuresHow long would it take
for to know whether
we’ve actually
achieved the outcome
we wanted to achieve
and the impact we
wanted to achieve?
• Technical Debt trends
• Architectural Coupling
• Defect trends
• Production incident
trends
• Downtime trends
• Number of active
branches, time spent
merging
• Time spent context
switching
• Velocity trends
• Innovation Rate
• Installed Version
Index
• Usage Index

Defect Reduction 12 months into Agile
Transition – Large scale
32 delivery areas and 45
teams across the
Organisation
• 18Q2, defects significantly
down from 17Q2, a release
of comparable scope and
volume
• Previously 6 teams would be
taken offline to fix defects for
3 months pre and 1 month
post deployment
• Robust Function testing
meant defects discovered
earlier by agile teams
• Integration of components
became part of DoD
17Q2
18Q2

Innovation RateWhat percentage of
your product budget is
spent on:
• Innovation and
building new
functionality
vs.
• Incremental
business change to
expand capacity?
vs.
• Maintaining
business
operations?
Source: Forrester, October 2010, 2011 IT Budget Planning Guide For CIOs Source: 2016-2017 Global CIO Survey N=1081
29%
Incremental business change
Business innovation
18%
58%
2010 2016-17

Innovation rate – Which is Better?
52%
Business innovation
10%
38%
Business Operations
29%
Business innovation
53%
18%
Business Operations
What is good depends
on:
• What stage of
Product Development
Life Cycle
• What type of Industry
• Pace of change and
competition for users
• Risk appetite vs
Opportunity
Enablement value
29% is the cross-industry average
across products and systems, roughly
the same for SMB and Enterprise

What can we do to Improve? Support your Scrum Team
in creating great products,
with no defects, that
people actually use
The lower the cost of
supporting your products,
the more resources there
will be for satisfying your
customer’s needs
Building software is largely
a one time cost, support
and maintenance goes on
forever

A2I – Key Value Measures
KVM Measuring:
Usage Index Measurement of features in the product that are frequently used. This helps capture features
that are rarely or never used.
Innovation Rate The percentage of effort or cost spent on new product capabilities, divided by total product
effort or cost. This provides insight into the capacity of the organization to deliver new product
capabilities.
Defect trends Measurement of change in defects since last measurement. A defect is anything that reduces
the value of the product to a customer, user, or to the organization itself. Defects are
generally things that don’t work as intended.
On-Product Index The percentage of the total user base that is using the current version of the product.
Installed Version Index The number of versions of a product that are currently being supported. This reflects the effort
the organization spends supporting and maintaining older versions of software.
Technical Debt A concept in programming that reflects the extra development and testing work that arises
when “quick and dirty” solutions result in later remediation.
Production Incident Trends The number of times the Development Team was interrupted to fix a problem in an installed
product. The number and frequency of Production Incidents can help indicate the stability of
the product.
Active code branches, time spent
merging code between branches
These measures are similar to the Installed Version Index, since different deployed versions
usually have separate code branches.
Time spent context- switching Number of meetings per day per person, and the number of times a day team members are
interrupted to help people outside the team can give simple insight into the magnitude of the
problem.

Deciding what to Measure is critical

EBM Key Value Measures
Current
Value
Revenue per Employee
Product Cost Ratio
Employee Satisfaction
Customer Satisfaction
Usage Index
Time to Market
T2M
Build and integration
frequency
Release Frequency
Release Stabilization Period
Mean Time to Repair
Cycle Time
Lead Time
Time-to-Learn
Ability to Innovate
A2I
Usage Index
Innovation Rate
Defect trends
On-Product Index
Installed Version Index
Technical Debt
Production Incident Trends
Active code branches, time
spent merging code
between branches
Time spent context- switching
Unrealised
Value
Market Share
Customer or user
satisfaction gap

Inspect and AdaptMeasurement helps us
inspect so that we can
adapt and improve our
products and also the way
we work
The spirit of improvement
helps creates innovation
Understanding the changes
of the KVMs prepares the
organization for its next
learning loop and track
changes across time learn
from patterns that emerge Incremental changes performed in small learning loops are
the most effective method for increasing an organisation’s
overall agility

Experiment and Constantly Learn to
Improve Value Derived
Visualise relative
strengths and
weaknesses
Observe the impact of
these practices to
overall organisational
value
Assess the results and
impact of an
experiment to monitor
the trend of value over
time

EBM provides a wholistic view of
Value to enhance Business Agility
Measure success in terms
of value and outcomes, not
output and activity
Celebrate success and
learn what helped create it
Focus on improving results
Run experiments to
understand what is valued,
to build innovative products
users will love
Maximise learning to build
knowledge and capability

51
“If you can't measure it, you can't improve it.” - Peter Drucker
• Current Value is most important - a product that offers no value to its users won’t last long
• Customer/user experience is only part of the picture; sustaining and improving value to
customers requires engaged employees and happy investors
• Improving value requires frequent delivery of new value, i.e improving the Time-to-Market
• Understanding and removing impediments to faster delivery is essential
• Faster Time-to-Market is not the whole story - Fast release cycles delivering only very
small improvements do little to rapidly improve the value delivered by a product
• Ability to innovate is determined by ability to deliver significant innovation in each release
• Measuring this ability gives organisations insights needed to remove barriers
• Improving organisational performance is a cyclic, iterative process
• Measure current conditions, set performance goals, form small improvement
experiments, and measure again to gauge the effect, then repeating, continuously

Fin!
Mia Horrigan@zenexmachina.com
Blog – www.zenexmachina.wordpress.com
Twitter - @miahorri
www.zenexmachina.com
Source and Thanks to Patricia Kong- Co Author of Evidenced Based Management https://www.scrum.org/resources/evidence-
based-management

Evidenced based management - Presentation at Scrum Australia 24 oct 2018

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Evidenced based management - Presentation at Scrum Australia 24 oct 2018

Similar to Evidenced based management - Presentation at Scrum Australia 24 oct 2018 (20)

More from Mia Horrigan

More from Mia Horrigan (20)

Recently uploaded

Recently uploaded (20)

Evidenced based management - Presentation at Scrum Australia 24 oct 2018

Editor's Notes