1. @vzanini
WWW.5DVISION.COM
Deliver great products and
the art of measuring outcomes
Valerio Zanini

2. @vzanini 2
Rome, Italy

3. @vzanini 3
Washington, DC

4. @vzanini
Valerio Zanini
Product innovation & Agile trainer
Author of:
“Deliver Great Products that Customers Love”

5. @vzanini 5
@vzanini
Share 3 thoughts or learning points:
#AgileDC
Win a copy of the book!

6. @vzanini
STORY
6
Rome, Italy
1546 A.D.
My story begins…

7. @vzanini 7
The construction of Saint Peter’s cathedral

8. @vzanini
It had taken 40 years to find the perfect design
8
…7 different architects had come and gone…

9. @vzanini
STORY
9
7 years to
build a
wooden
model with all
the details
Antonio Da
Sangallo “The
Young”

10. @vzanini
STORY
10
No more Waterfall !

11. @vzanini
STORY
11
Michelangelo

12. @vzanini 12
“The Lean Startup” – Eric Ries

13. @vzanini 13
Build clay
prototypes
Get
feedback
from Pope
Pivot and
continue
7 days

14. @vzanini 14
He made no overall plans for Saint Peter’s
Michelangelo created plans only for what could be built
shortly, and then iterated

15. @vzanini 15
Michelangelo:
The first “Agilist”
500 years ago?

16. @vzanini 16
Have you seen a
product or project fail?
What was the key
reason?

17. @vzanini 17
Let’s look at some Agile big fails…

18. @vzanini 18

19. @vzanini 19
“We just finished our product,
we need help with marketing”
Failure #1:
Shortcutting Discovery

20. @vzanini
The team was focused on execution
20
DISCOVER DESIGN DEVELOP DEPLOY DELIVER
Source: book “Deliver great products that customers love”, V. Zanini 2018
?

21. @vzanini
Understanding your customers helps you understand how to
deliver value
21
DISCOVER DESIGN DEVELOP DEPLOY DELIVER
Source: book “Deliver great products that customers love”, V. Zanini 2018

22. @vzanini 22
“Of course we are Agile, we
use Scrum in development”
Failure #2: Big Waterfall
using Scrum

23. @vzanini
Agile is used in Development but everything else is set upfront
23
DISCOVER DESIGN
6+ months to define all
the requirements

24. @vzanini
Agile is used in Development but everything else is set upfront
DISCOVER DESIGN
DEVELOP
SCRUM
1 year to build it
using “Agile”
24

25. @vzanini
Agile is used in Development but everything else is set upfront
25
DISCOVER DESIGN
DEVELOP
DEPLOY DELIVER
SCRUM
?

26. @vzanini 26
Download:
Source: book “Deliver great products that customers love”, V. Zanini 2018

27. @vzanini
The 5 Dimensions of great products
27
DISCOVER DESIGN DEVELOP DEPLOY DELIVER
SolutionsProblems, needs,
opportunities
Products Market launch Customer
adoption and
feedback
Measure value to customers all the way

28. @vzanini
Work across the 5 Dimensions in rapid iterations and deliver
value along the way
28
MVP
V.2.0
V.2.1

29. @vzanini 29
In your project:
Which of the 5
Dimensions are you
least prepared for?
Which needs most
attention?

30. @vzanini 30
The tiramisu’ challenge

31. @vzanini 31
Coffee
(Espresso)

32. @vzanini 32
Coffee
(Espresso)
Ladyfingers
(Savoiardi)

33. @vzanini 33
Coffee
(Espresso)
Ladyfingers
(Savoiardi)
Mascarpone
cheese

34. @vzanini 34

35. @vzanini
Three primary responsibilities of a Product Manager
Understand
the
customer
Drive product
development
Measure
the
outcomes
This is also well
represented by the
Learn, Build,
Measure cycle

36. @vzanini
In reality, it often feels
this way
Understand
the
customer
Drive product
development
Measure
the
outcomes
Too often we focus on
just getting the
product out the door,
instead of making sure
we build the right
product

37. @vzanini
It gets even worse
Understand
the
customer
Drive product
development
Measure
the
outcomes We do
“Agile”

38. @vzanini 38
When we build products, we try to go from
idea to launch in the shortest time possible
Image: Created by Alekksall - Freepik.com
Image: Created by Cornecoba - Freepik.com
From Idea To Launch
Agile DevOps
Build, Measure, Learn
MVP
Prototypes
Design Thinking

39. @vzanini 39
Accelerate software delivery 74%
VersionOne State of Agile Survey 2019

40. @vzanini 40
Where is the
customer ?

41. @vzanini
Customer focus
41
Can you articulate?
Who the customer is
What they need
Why they buy your product
What value they get from it
Too often teams shortcut the
Discovery dimension jumping
straight to the Solution

42. @vzanini 42
The goal should be to go from Idea to Customer
Image: Created by Alekksall - Freepik.com
Image: Created by Vectorpocket - Freepik.com
From Idea To Customer

43. @vzanini 43
DEPLOYMENT DELIVERY
Focus is on getting
product out
(outputs)
Focus is on customers,
needs, CX, value
(outcomes)
DELIVERY
GAP
Deployment is not the real goal Delivery is the goal

44. @vzanini
Employ a healthy balance of activities to build the right product
Understand
the
customer
Drive product
development
Measure
the
outcomes
Build
Measure
Learn

45. @vzanini 45

46. @vzanini 46
How do we
measure value to
our customers

47. @vzanini
Hypotheses driven
development
MVP (to learn fast)
Voice of the Customer
NPS
Value metrics
Are we delivering outcomes?
Measure
the
outcomes

48. @vzanini 48
Is it meaningful?
You can influence it
You can measure it
Value
metrics
If you improve it, your
outcomes improve
You can do something
about it
Otherwise, how do you
know?
Select a metric that:

49. @vzanini
Measure value delivered
49
Measuring delivery on time, on budget, scope, quality, etc.: all OK
But, do these matter if people don’t want your product?
Focus should be on value delivered.
Establish
baseline
metric
Establish
target /
desired
outcome
Build an MVP
or MVF
Measure
progress
towards target
/ desired
outcome
Pivot
Success
Failure
Innovation accounting, adapted from “The Lean Startup”, Eric Ries

50. @vzanini 50
Backlog of requirements
Measure outputs
Feedback is confirmatory
What’s in your backlog?

51. @vzanini 51
Backlog of experimentsBacklog of requirements
Measure outputs
Feedback is confirmatory
Measure outcomes
Feedback is a learning
opportunity
What’s in your backlog?

52. @vzanini 52
Most people use analytics the
way a drunk uses a lamppost, for
support rather than illumination
“
”
– David Ogilvy

53. @vzanini
EPS
Revenue
User acquisition
Pageviews
Unique users
Returning users
Are we delivering business results?
Business
metrics

54. @vzanini
Pirate metrics (AARRR)
54
Acquisition
Activation
Retention
Revenue
Referral
Are users finding you?
Do users sign up for your product?
Do users come back?
Are users willing to pay?
Do users invite others?
Based on Pirate metrics (AARRR) by Dave-McClures

55. @vzanini
Pirate metrics (AARRR)
55
Acquisition
Activation
Retention
Revenue
Referral
Based on Pirate metrics (AARRR) by Dave-McClures
Why are users not
willing to pay?
Example:
Linkedin/Twitter

56. @vzanini
Pirate metrics (AARRR)
56
Acquisition
Activation
Retention
Revenue
Referral
Based on Pirate metrics (AARRR) by Dave-McClures
Why are users not
signing up to use
your product?
Example: e-commerce
website

57. @vzanini
STORY
57
We focused on registered users and bounce rate,
instead of metrics that drove real value
Online e-commerce company

58. @vzanini
The HEART framework from Google
58
H E A R T
Happiness Engagement Adoption Retention Task success
Level of user
involvement, typically
measured via
behavioral proxies
such as frequency,
intensity, or depth of
interaction over some
time period. Examples
might include the
number of visits per
user per week or the
number of photos
uploaded per user
per day.
New users of a
product or feature.
For example: the
number of accounts
created in the last
seven days or the
percentage of Gmail
users who use labels.
The rate at which
existing users are
returning. For
example: how many
of the active users
from a given time
period are still present
in some later time
period? You may be
more interested in
failure to retain,
commonly known as
“churn.”
This includes
traditional behavioral
metrics of user
experience, such as
efficiency (e.g. time
to complete a task),
effectiveness (e.g.
percent of tasks
completed), and
error rate. This
category is most
applicable to areas
of your product that
are very task-focused,
such as search or an
upload flow.
Measures of user
attitudes, often
collected via survey.
For example:
satisfaction,
perceived ease of
use, and net-
promoter score.

59. @vzanini
Would you launch
your product without
any way to measure
its performance?

60. @vzanini
System and security logs
Real-time telemetry
Test and re-test
Flow break-points
Team delivery charts
Is this thing working?
Problems can arise at anytime, whether as
united consequences of your actions or
due to externalities
Telemetry
metrics

61. @vzanini
Telemetry helps you monitor the health of your app
61
Errors in production
HTTP 5xx
Performance
Events
Broken links
Server failures
Dedicated tools and system logs
Image courtesy of Splunk
E.g. Capital
One Mobile
Deposit

62. @vzanini
Telemetry helps you identify your flow break points
62
Customer journey maps help you
understand the journey users take
with your product/service and
identify breaking points
Telemetry provides real-use
information of where users stumble

63. @vzanini 63
Burndown charts can offer useful
insights into team impediments and
actions product owners can take
Look for unusual paths and changes in
scope. Overcommitment and
changing priorities are usual culprits.

64. @vzanini
What will you measure tomorrow?
Are we
delivering
business
results?
Are we
delivering
outcomes?
Is this thing
working?
Business
metrics
Customer
metrics
Telemetry
metrics

65. @vzanini
Thank you
Valerio Zanini
vzanini@5dvision.com
@vzanini
Deliver great
products that
customers love
Available:
Amazon, Kindle,
Barns and Nobles,
Leanpub