Product Management by Numbers: Using Metrics To Optimize Your Product by Dan Olsen

Product Management by Numbers:
Using Metrics to Optimize your Product
Dan Olsen
CEO & Founder, YourVersion
Dec 17, 2008 1 Copyright © 2008 Olsen Solutions LLC
Copyright

Using Metrics to Optimize your Product
Product Management’s Role
Integrating metrics & optimization
Using metrics to optimize value creation
Business Value
Customer Value
UI Design
Illustrate with real world examples

Copyright © 2008 Olsen Solutions LLC

My Background
Education
BS, Electrical Engineering, Northwestern
MS, Industrial Engineering, Virginia Tech
MBA, Stanford
PHP, MySQL, JavaScript, XHTML, CSS, UI design
17 years of Product Management Experience
Managed submarine design for 5 years
5 years at Intuit, led Quicken Product Management
Led Product Management at Friendster
Olsen Solutions LLC, PM consultant for start‐ups
CEO & Founder of YourVersion, Social Discovery start‐up

Product Management’s
Role


A Product Manager by any Other
Name Would Smell as Sweet
Product managers are sometimes called
Product marketing manager
Program manager
Project manager
Label and definition of role can vary
Based on industry or company
Based on B2C (consumer) vs. B2B (enterprise)
Based on stage of company
Can be area of responsibility vs. actual position


Product Management is
Critical Link in Value Creation

Market Product Development
• Current Management Team
customers
• Prospective
customers
• Competitors


A Process View of Product Management
“Inbound”
Product “Outbound”
Management Product
Long Business Product Management
Term Strategy Strategy
Market/
Sell
Short Business Product Product
Term Objectives Objectives Development
Service/
Support


PM’s Job: A Successful Product
Know market and customer better than anyone else
Translate business objectives and customer needs
into product requirements
Be the clearinghouse for all product ideas
Identify, plan & prioritize product ideas to maximize
ROI on engineering resources
Work with team to design and build a great product
Evangelize product within and outside the company
Keep the product team motivated


Integrating Metrics and
Optimization into your
Product Process


Adding Metrics and Optimization to
your Product Process
Site Level
Business Product Prioritized
Plan Objectives Objectives Feature List

Scoping Feature
Level

Requirements
Design & Design

Code Test Launch
Develop

Metrics & User
Optimize Feedback

Optimization through Iteration:
Continuous Improvement
Measure
the metric

Analyze
Learning the metric
Gaining knowledge:
• Market Identify top
• Customer opportunities
to improve
• Domain
• Usability Design & develop
the enhancement

Launch the
enhancement

Optimizing
Business Value Creation
with Metrics


Approaching Business as an
Optimization Exercise

Given reality as it exists today,
optimize our business results
subject to our resource constraints.


Define the Equation of your Business
“Peeling the Onion”
Profit = Revenue ‐ Cost

Unique Visitors x  Ad Revenue per Visitor

Impressions/Visitor x  Effective CPM / 1000

Visits/Visitor  x  Pageviews/Visit  x  Impressions/PV

New Visitors + Returning Visitors

Invited Visitors + Uninvited Visitors

# of Users Sending Invites  x  Invites Sent/User  x  Invite Click‐through Rate


How to Track Your Metrics
Track each metric as daily time series
Unique Page Ad New User
…
Date Visitors views Revenue Sign‐ups
4/24/08 10,100 29,600 25 490
4/25/08 10,500 27,100 24 480
…

Create ratios from primary metrics: X / Y
Example: How good is your registration page?
Okay: # of registered users per day
Better: registration conversion rate =
# registered users / # uniques to reg page

Sample Signup Page Yield Data
Daily Signup Page Yield vs. Time
New Registered Users divided by Unique Visitors to Signup Page

100%

90%

80%
Daily Signup Page Yield

70%

60%

50%

40%

30% Started requiring
registration

20%
Changed Added questions
messaging to signup page
10%

0%
1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/1
0


Using Metrics to Optimize the
Equation of your Business
What are the metrics for your business?
Where is current value for each metric?
How many resources to “move” each metric?
Developer‐hours, time, money
Which metrics have highest ROI opportunities?

Metric A Metric B Metric C
Good ROI Bad ROI Great ROI

Return
Return
Return

Investment Investment Investment


Case Study:
Optimizing Viral Loop Metrics
% of users
sending = 15% Invites per Invite
invites sender = 2.3 click-through rate

Active Invite Prospective Click Registration Fail
Users Users Process
% of users
who are Succeed
Don’t
active Click Conversion
rate = 85%
Users

• Multiplied together, these metrics determine your viral ratio
• Which metric has highest ROI opportunity?


Maximum Upside Potential of a Metric
?

100% 100%
85%

15% 2.3
0 0 0
Registration % of users sending Avg # of invites
conversion rate invitations sent per sender

Max possible 0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?%
improvement


Doubling Number of Invitations Sent per Sender by
Adding Address Book Importer


Optimizing
Customer Value Creation
with Metrics


The Fuzzy Math of Customer Value
You create customer value with a product that
Satisfies customers’ needs
Is easy to use
Has a good price
Is better than other alternatives
Not fuzzy: # of users and frequency of use
Applying metrics approach to customer value
Need framework of user needs & satisfaction
Quantitative: site analytics, usage metrics, surveys
Qualitative: 1‐on‐1 user feedback sessions are best


Olsen’s Hierarchy of Web User Needs
(adapted from Maslow)
Customer’s Perspective What does it mean to us?

How easy to use is it? Usability & Design

Satisfaction
Increasing
Does the functionality
Feature Set
meet my needs?

Does the functionality work?
Absence of Bugs

Dissatisfaction
Decreasing
Is the site fast enough?
Page Load Time

Is the site up when I want to use it?
Uptime


Kano Model: User Needs & Satisfaction
User Satisfaction
Delighter (wow)

Performance
(more is better)

Need Need
not met fully met
Must Have

Needs & features
migrate over time

User Dissatisfaction Copyright © 2008 Olsen Solutions LLC

Importance vs. Satisfaction
Ask Users to Rate for Each Feature
100 98
Great
95
84 87
90
Bad 86
85 79 84
55 70
80
Importance

80

75 72
80
70
75
65

60

55
41
50
40 50 60 70 80 90 100
Satisfaction

Recommended reading: “What
Customers Want” by Anthony Ulwick Copyright © 2008 Olsen Solutions LLC

What is Your Value Proposition?
Which user benefits are most important?
How well does your product deliver?
Importance Current User Upside
User Benefit to User Satisfaction Potential
Low High – +
Benefit 1 High

Benefit 2 High
Benefit 3 Low ?
opportunity

Benefit 4 Med

Benefit 5 Low ?
opportunity

Benefit 6 Low
Doesn’t matter

Return (Value Created) Analyzing Product Ideas by ROI

?
4
Idea D
3
Idea A Idea B
2
Idea C
1
Idea F

1 2 3 4
Investment (developer-weeks)

Return (Value Created) Prioritizing Product Ideas by ROI

Idea C
8
7 Idea B
6
5
4 Idea A
3
2
1

1 2 3 4 5
Investment (developer-weeks)

The UI Design Iceberg
What most
people see
and react to Visual
Design What good
PMs and
Designers
Interaction think about
Design

Information
Architecture

Conceptual
Design

Recommended reading: Jesse James Garrett’s
“Elements of User Experience” chart, free at www.jjg.net Copyright © 2008 Olsen Solutions LLC

Approaching UI Design Analytically
Typical UI design question:
“When using web pages, do users scroll down?”
‐ Yes
‐ No

UI questions are never yes/no! (not binary)
Should ask: “What percentage of users …?”
UI changes impact your metrics
Impact can be positive, negative, small, large
Seek high‐ROI UI changes

Put Key Conversion Actions Above The Fold
Landing Page A Landing Page B

The Fold
Key conversion action
is above the fold
Key conversion action
is below the fold


The Fold: “Chrome” Takes Away Pixels

Only
768
600 pixels
pixels

1024 pixels

Analyze User Screen Height Distribution
to Select Design Height for Key Pages
100%
600, 100% 768, 92%
90%
% of Users with Given (or Higher) Screen Height

80%

70%
Going above
60%
768 px = big drop
50%

40%
772, 38%
864, 29%
30%
807, 29% 1024, 20%
866, 25%
20%
1000, 20%
10%
1030, 5%
0%
600 650 700 750 800 850 900 950 1000 1050 1100
Page Design Height (pixels)
Monitor Screen Height (pixels)

Case Study: Account Signup Process Redesign

100%
100%
Biggest
80%
drop
% of Users

62.3%
58.8%
60%
50.9%

40% 34.4% 32.7%

20%

0%

Sign in / Account Cash vs. 5 Partner 3 Partner
Registration Type Margin Pages Pages

Analysis of Sign In/Registration Flow
Open
Account
55%
44% Register Registration (24% of Total)
Process

45% drop off
64%
(20% of total)
of Total Account
36% overall Selection
83% 30% drop off for
56% (46% of Total) (14% of Total) this step
Sign in

Forget 70% Change 80%
Password (32% of Total) Password (26% of Total)

17% drop off 20% drop off
(10% of total) (6% of total)


Redesigned User Flow Improved
Registration Conversion Rate
Abandonment Rate (7 Day Moving Average)
Steps 1-2
80%

70%
Abandonment Rate (7 Day Moving Average)

60%

50%

40%

30% Released 37% improvement
New Design in conversion rate
20%

10%

0%
10/14/02

10/21/02

10/28/02

11/11/02

11/18/02

11/25/02

12/16/02

12/23/02

12/30/02
10/7/02

11/4/02

12/2/02

12/9/02

1/13/03

1/20/03
1/6/03

Product Management by Numbers
Cheat Sheet
Define what success means
Customer value proposition
Equation of your business
Instrument your site and track key metrics
Identify opportunities and prioritize by ROI
Launch, learn, and iterate


YourVersion proactively discovers relevant
web content for you based on your interests.

Discover Bookmark Share

Product Management by Numbers: Using Metrics To Optimize Your Product by Dan Olsen

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Product Management by Numbers: Using Metrics To Optimize Your Product by Dan Olsen

Similar to Product Management by Numbers: Using Metrics To Optimize Your Product by Dan Olsen (20)

More from Dan Olsen

More from Dan Olsen (20)

Recently uploaded

Recently uploaded (20)

Product Management by Numbers: Using Metrics To Optimize Your Product by Dan Olsen