World-Class Web Metrics by Dan Olsen

World‐Class Web Metrics:
How to Optimize Your Product & Business
Dan Olsen
CEO, YourVersion
June 17, 2009
Copyright © 2009 YourVersion

World‐Class Metrics:
Amazon.com
$37 B Market Cap
$19 B Revenue ‘08
•Automated testing
of page elements &
suggested products
•High degree of
user personalization
•Real‐time
recommendations


World‐Class Metrics:
Google
$137 B Market Cap
$22 B Revenue ‘08
•Auto‐testing of
UI elements &
search results
•Automatically
learns from user
mistakes
•Optimize ads to
maximize revenue

Overview
What is being World‐Class in Metrics?
Optimizing Business Value
Optimizing the Customer Value of
Your Product


My Web Metrics Epiphany
Quicken (1998 – 2001)
New version every year: shrink‐wrapped CD‐ROM
Pre‐launch
Before development: Validated product ideas with users
During development: Usability testing & user feedback
Post‐launch
No “real‐time” metrics data
Sales data: 1‐2 month lag, not actionable
User surveys: actionable, but not until next product cycle
Quicken Brokerage (2001 – 2003)
Online brokerage launched in September 2002
The day after launch: LOTS of metrics data → sought more
Much faster iteration: metrics → hypothesis → improvement

Metrics Maturity Model
Unaware Aspiring World‐Class
Metrics Comprehensive
None Sporadic
Tracking breadth & depth
Analysis None Ad hoc Recurring
Automated
Testing & Some A/B
None multivariate
Optimization tests
testing
Lack of Partial Clear, dedicated
Organization
ownership ownership owners
Intuition‐ Data‐ Continuous
Culture
based driven improvement

Optimizing Business Value


Approaching Business as an
Optimization Exercise

Given reality as it exists today,
optimize our business results
subject to our resource constraints.


Define the Equation of your Business
“Peeling the Onion”
Profit = Revenue ‐ Cost

Paying Users x  Revenue per Paying User

New Paying Users +  Repeat Paying Users

Trial Users x  Conv Rate Previous Paying Users  x  ( 1 – Cancellation Rate )

( SEO Visitors + SEM Visitors + Viral Visitors )  x  Trial Conversion Rate


How to Track Your Metrics
Track each metric as daily time series
Unique Page Ad New User
…
Date Visitors views Revenue Sign‐ups
4/24/08 10,100 29,600 25 490
4/25/08 10,500 27,100 24 480
…

Create ratios from primary metrics: X / Y
Example: How good is your registration page?
Okay: # of registered users per day
Better: registration conversion rate =
# registered users / # uniques to reg page

Sample Signup Page Yield Data
Daily Signup Page Yield vs. Time
New Registered Users divided by Unique Visitors to Signup Page

100%

90%

80%
Daily Signup Page Yield

70%

60%

50%

40%

30% Started requiring
registration

20%
Changed Added questions
messaging to signup page
10%

0%
1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/1
0


Identifying the “Critical Few” Metrics
What are the metrics for your business?
Where is current value for each metric?
How many resources to “move” each metric?
Developer‐hours, time, money
Which metrics have highest ROI opportunities?

Metric A Metric B Metric C
Good ROI Bad ROI Great ROI

Return
Return
Return

Investment Investment Investment


Turning Metrics into Meaning & Action
• Clear owner
for each key
metric
•Focus on trend
& changes
•Discuss
business &
customer impact
•Propose ideas
to improve From Avinash Kaushik’s Occam’s Razor at www.kaushik.net

Optimizing the Customer
Value of Your Product


Using Analytics to Clarify The
Fuzzy Math of Customer Value
You create customer value with a product that
Satisfies customers’ needs
Is easy to use
Has a good price
Is better than other alternatives
Offline: hard to measure
Online: can measure everything
Measuring value: # of users, repeat use, revenue
Can achieve clarity with metrics and analytics


Once You’re Measuring Everything,
Where Do You Focus?
How do you decide which user benefits
and features you should focus on?
Need framework to make decisions
“Problem space” vs. “Solution space”
Importance vs. Satisfaction
Importance of user need (problem space)
Satisfaction with how well your product meets
the user’s need (solution space)


Importance vs. Satisfaction
Ask Users to Rate for Each Feature
100 98
Great
95
84 87
90
Bad 86
85 79 84
55 70
80
Importance

80

75 72
80
70
75
65

60

55
41
50
40 50 60 70 80 90 100
Satisfaction

Recommended reading: “What
Customers Want” by Anthony Ulwick Copyright © 2009 YourVersion

Kano Model: User Needs & Satisfaction
User Satisfaction
Delighter (wow)

Performance
(more is better)

Need Need
not met fully met
Must Have

Needs & features
migrate over time

User Dissatisfaction Copyright © 2009 YourVersion

Olsen’s Hierarchy of Web User Needs
(adapted from Maslow)
Customer’s Perspective What does it mean to us?

How easy to use is it? Usability & Design

Satisfaction
Increasing
Does the functionality
Feature Set
meet my needs?

Does the functionality work?
Absence of Bugs

Dissatisfaction
Decreasing
Is the site fast enough?
Page Load Time

Is the site up when I want to use it?
Uptime


Using Web Analytics Tools to
Understand Your Users
Tracking visitors and traffic
Seeing where users are clicking
Measuring key conversions
Monitoring user feedback


Basic Tracking of Traffic:
Google Analytics
•Unique
visitors
•New vs.
returning
•Pageviews
•Time on site
•Top referrers
•Top geos


Seeing Where Users are Clicking:
CrazyEgg Heatmap
•Shows Click
Density
•Color indicates
% of clicks
•See which links
perform best
•See impact of
UI changes


Measuring Key Conversions:
Conversion Funnel
•Tie user actions to
business goals
•Instrument key steps in
user flow
•See where users are
dropping off
•Quantify improvement
from changes


Monitoring User Feedback
Quantitatively with Kampyle

Unobtrusive
Solicitation Average Grade over time
Simple popup

Average Grade
for each page
on your site


The UI Design Iceberg
What most
people see
and react to Visual
Design What good
PMs and
Designers
Interaction think about
Design

Information
Architecture

Conceptual
Design

Recommended reading: Jesse James Garrett’s
“Elements of User Experience” chart, free at www.jjg.net Copyright © 2009 YourVersion

Approaching UI Design Analytically
Typical UI design question:
“When using web pages, do users scroll down?”
‐ Yes
‐ No

UI questions are never yes/no! (not binary)
Should ask: “What percentage of users …?”
UI changes impact your metrics
Impact can be positive, negative, small, large
Seek high‐ROI UI changes

Put Key Conversion Actions Above The Fold
Landing Page A Landing Page B

The Fold
Key conversion action
is above the fold
Key conversion action
is below the fold

Copyright © 2009 Olsen Solutions LLC

Marrying UI Design with Metrics
Think of “Ease of use”
as distinct from the
functionality
User Interface
elements matter
Position
Layout
Size
Color
Font
Text copy
Interaction design
Navigation

Levels of Optimization Maturity
A/B testing vs. Multivariate testing
A/B: test 2 alternatives (ideally same time)
Multivariate: test multiple variables at once
Degree of automation & frequency
Manual → Automated → Continuous real‐time
Level of personalization
Universal: same for all users
Personalized: varying based on user, user
attributes, or user segment


Automated Multivariate Testing:
Google Website Optimizer


Organization and Culture
Impact Product Success
Roles and Skills required
Product Management (PM)
UI Design
Development
Quality Assurance (QA)
Operations
Cultural traits required
Customer‐centric
Data‐driven decision‐making with clear ownership
Rapid, iterative development
Continuous improvement mindset


The Product Manager’s Job:
A Successful Product
Be the expert on the market and the customer
Translate business objectives and customer
needs into product requirements
Be the clearinghouse for all product ideas
Work with team to design & build great product
Define and track key metrics
Use metrics to drive continuous improvement
Identify, plan & prioritize product ideas to
maximize ROI on engineering resources

Prioritizing Product Ideas by ROI

?
Return (Value Created)

4
Idea D
3
Idea A Idea B
2
Idea C
1
Idea F

1 2 3 4
Investment (developer‐weeks)

Case Study 1: Quicken Brokerage
Optimizing Sign In/Registration Flow
Open
Account
55%
44% Register Registration (24% of Total)
Process

45% drop off
64%
(20% of total)
of Total Account
36% overall Selection
83% 30% drop off for
56% (46% of Total) (14% of Total) this step
Sign in

Forget 70% Change 80%
Password (32% of Total) Password (26% of Total)

17% drop off 20% drop off
(10% of total) (6% of total)


Redesigned User Flow Improved
Registration Conversion Rate 37%

Released 37% improvement
New Design in conversion rate


Case Study 2: Friendster
Optimizing Viral Growth
% of users
sending = 15% Invites per Invite
= 20%
invites sender = 2.3 click-through rate

Active Invite Prospective Click Registration Fail
Users Users Process
% of
users who = 50% Succeed
Don’t
are active Click Conversion
rate = 85%
Users

• Multiplied together, these metrics determine your viral ratio
• Which metric offers the highest ROI opportunity?


Case Study 2: Friendster
Doubled Number of Invitations Sent per Sender


Continuous
Improvement


Adding Metrics and Optimization to
your Product Process
Site Level
Business Product Prioritized
Plan Objectives Objectives Feature List

Scoping Feature
Level

Requirements
Design & Design

Code Test Launch
Develop

Metrics & User
Optimize Feedback

Optimization through Iteration:
Continuous Improvement
Measure
the metric

Analyze
Learning the metric
Gaining knowledge:
• Market Identify top
• Customer opportunities
to improve
• Domain
• Usability Design & develop
the enhancement

Launch the
enhancement

Summary:
How to be World‐Class in Web Metrics
Define the equation of your business
Identify and track your key metrics
Use analytics to gain customer insights
Establish clear roles and data‐driven culture
Identify opportunities and prioritize by ROI
Launch, learn, and iterate


Questions? www.yourversion.com
dan@yourversion.com

World-Class Web Metrics by Dan Olsen

World-Class Web Metrics by Dan Olsen

Recommended

More Related Content

What's hot

What's hot(20)

Viewers also liked

Viewers also liked(13)

Similar to World-Class Web Metrics by Dan Olsen

Similar to World-Class Web Metrics by Dan Olsen(20)

More from Dan Olsen

More from Dan Olsen(20)

Recently uploaded

Recently uploaded(20)

World-Class Web Metrics by Dan Olsen