More Related Content Similar to World-Class Web Metrics by Dan Olsen (20) World-Class Web Metrics by Dan Olsen2. WorldâClass Metrics:
Amazon.com
$37 B Market Cap
$19 B Revenue â08Â
â˘Automated testingÂ
of page elements &Â
suggested products
â˘High degree ofÂ
user personalization
â˘RealâtimeÂ
recommendations
Copyright Š 2009 YourVersion
3. WorldâClass Metrics:
Google
$137 B Market Cap
$22 B Revenue â08Â
â˘Autoâtesting of
UI elements &Â
search results
â˘AutomaticallyÂ
learns from userÂ
mistakes
â˘Optimize ads toÂ
maximize revenue
Copyright Š 2009 YourVersion
5. My Web Metrics Epiphany
Quicken (1998 â 2001)
New version every year: shrinkâwrapped CDâROM
Preâlaunch
Before development: Validated product ideas with users
During development: Usability testing & user feedback
Postâlaunch
No ârealâtimeâ metrics data
Sales data: 1â2 month lag, not actionable
User surveys: actionable, but not until next product cycle
Quicken Brokerage (2001 â 2003)
Online brokerage launched in September 2002
The day after launch: LOTS of metrics data â sought more
Much faster iteration: metrics â hypothesis â improvement
Copyright Š 2009 YourVersion
6. Metrics Maturity Model
Unaware Aspiring WorldâClass
Metrics ComprehensiveÂ
None Sporadic
Tracking breadth & depth
Analysis None Ad hoc Recurring
AutomatedÂ
Testing & Some A/BÂ
None multivariateÂ
Optimization tests
testing
Lack of Partial Clear, dedicatedÂ
Organization
ownership ownership owners
Intuitionâ Dataâ ContinuousÂ
Culture
based driven improvement
Copyright Š 2009 YourVersion
8. Approaching Business as anÂ
Optimization Exercise
Given reality as it exists today,
optimize our business results
subject to our resource constraints.
Copyright Š 2009 YourVersion
9. Define the Equation of your Business
âPeeling the Onionâ
Profit = Revenue â Cost
Paying Users x  Revenue per Paying User
New Paying Users +  Repeat Paying Users
Trial Users x  Conv Rate Previous Paying Users  x  ( 1 â Cancellation Rate )
( SEO Visitors + SEM Visitors + Viral Visitors )  x  Trial Conversion Rate
Copyright Š 2009 YourVersion
10. How to Track Your Metrics
Track each metric as daily time series
Unique Page Ad New UserÂ
âŚ
Date Visitors views Revenue Signâups
4/24/08 10,100 29,600 25 490
4/25/08 10,500 27,100 24 480
âŚ
Create ratios from primary metrics:  X / Y
Example: How good is your registration page?
Okay: # of registered users per day
Better: registration conversion rate =
# registered users / # uniques to reg page
Copyright Š 2009 YourVersion
11. Sample Signup Page Yield Data
Daily Signup Page Yield vs. Time
New Registered Users divided by Unique Visitors to Signup Page
100%
90%
80%
Daily Signup Page Yield
70%
60%
50%
40%
30% Started requiring
registration
20%
Changed Added questions
messaging to signup page
10%
0%
1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/1
0
Copyright Š 2009 YourVersion
12. Identifying the  âCritical Fewâ Metrics
What are the metrics for your business?
Where is current value for each metric?Â
How many resources to âmoveâ each metric?
Developerâhours, time, money
Which metrics have highest ROI opportunities?
Metric A Metric B Metric C
Good ROI Bad ROI Great ROI
Return
Return
Return
Investment Investment Investment
Copyright Š 2009 YourVersion
15. Using Analytics to Clarify The
Fuzzy Math of Customer Value
You create customer value with a product that
Satisfies customersâ needs
Is easy to use
Has a good price
Is better than other alternatives
Offline: hard to measure
Online: can measure everything
Measuring value: # of users, repeat use, revenue
Can achieve clarity with metrics and analytics
Copyright Š 2009 YourVersion
16. Once Youâre Measuring Everything,Â
Where Do You Focus?
How do you decide which user benefitsÂ
and features you should focus on?
Need framework to make decisions
âProblem spaceâ vs. âSolution spaceâ
Importance vs. Satisfaction
Importance of user need (problem space)
Satisfaction with how well your product meetsÂ
the userâs need (solution space)
Copyright Š 2009 YourVersion
17. Importance vs. Satisfaction
Ask Users to Rate for Each Feature
100 98
Great
95
84 87
90
Bad 86
85 79 84
55 70
80
Importance
80
75 72
80
70
75
65
60
55
41
50
40 50 60 70 80 90 100
Satisfaction
Recommended reading: âWhatÂ
Customers Wantâ by Anthony Ulwick Copyright Š 2009 YourVersion
18. Kano Model: User Needs & Satisfaction
User Satisfaction
Delighter (wow)
PerformanceÂ
(more is better)
Need Need
not met fully met
Must Have
Needs & featuresÂ
migrate over time
User Dissatisfaction Copyright Š 2009 YourVersion
19. Olsenâs Hierarchy of Web User Needs
(adapted from Maslow)
Customerâs Perspective What does it mean to us?
How easy to use is it? Usability & Design
Satisfaction
Increasing
Does the functionality
Feature Set
meet my needs?
Does the functionality work?
Absence of Bugs
Dissatisfaction
Decreasing
Is the site fast enough?
Page Load Time
Is the site up when I want to use it?
Uptime
Copyright Š 2009 YourVersion
20. Using Web Analytics Tools toÂ
Understand Your Users
Tracking visitors and traffic
Seeing where users are clicking
Measuring key conversions
Monitoring user feedback
Copyright Š 2009 YourVersion
21. Basic Tracking of Traffic:
Google Analytics
â˘UniqueÂ
visitors
â˘New vs.Â
returning
â˘Pageviews
â˘Time on site
â˘Top referrers
â˘Top geos
Copyright Š 2009 YourVersion
22. Seeing Where Users are Clicking:Â
CrazyEgg Heatmap
â˘Shows ClickÂ
Density
â˘Color indicatesÂ
% of clicks
â˘See which linksÂ
perform best
â˘See impact ofÂ
UIÂ changes
Copyright Š 2009 YourVersion
23. Measuring Key Conversions:
Conversion Funnel
â˘Tie user actions toÂ
business goals
â˘Instrument key steps inÂ
user flow
â˘See where users areÂ
dropping off
â˘Quantify improvementÂ
from changes
Copyright Š 2009 YourVersion
24. Monitoring User FeedbackÂ
Quantitatively with Kampyle
Unobtrusive
Solicitation Average Grade over time
Simple popup
Average GradeÂ
for each pageÂ
on your site
Copyright Š 2009 YourVersion
25. The UI Design Iceberg
What most
people see
and react to Visual
Design What good
PMs and
Designers
Interaction think about
Design
Information
Architecture
Conceptual
Design
Recommended reading: Jesse James Garrettâs
âElements of User Experienceâ chart, free at www.jjg.net Copyright Š 2009 YourVersion
27. Put Key Conversion Actions Above The Fold
Landing Page A Landing Page B
The Fold
Key conversion action
is above the fold
Key conversion action
is below the fold
Copyright Š 2009 Olsen Solutions LLC
Copyright Š 2009 YourVersion
31. Organization and Culture
Impact Product Success
Roles and Skills required
Product Management (PM)
UIÂ Design
Development
Quality Assurance (QA)
Operations
Cultural traits required
Customerâcentric
Dataâdriven decisionâmaking with clear ownership
Rapid, iterative development
Continuous improvement mindset
Copyright Š 2009 YourVersion
32. The Product Managerâs Job:
A Successful Product
Be the expert on the market and the customer
Translate business objectives and customerÂ
needs into product requirements
Be the clearinghouse for all product ideasÂ
Work with team to design & build great product
Define and track key metrics
Use metrics to drive continuous improvement
Identify, plan & prioritize product ideas toÂ
maximize ROI on engineering resources
Copyright Š 2009 YourVersion
34. Case Study 1: Quicken Brokerage
Optimizing Sign In/Registration Flow
Open
Account
55%
44% Register Registration (24% of Total)
Process
45% drop off
64%
(20% of total)
of Total AccountÂ
36% overall Selection
83% 30% drop off forÂ
56% (46% of Total) (14% of Total) this step
Sign in
Forget 70% Change 80%
Password (32% of Total) Password (26% of Total)
17% drop off 20% drop off
(10% of total) (6% of total)
Copyright Š 2009 YourVersion
36. Case Study 2: Friendster
Optimizing Viral Growth
% of users
sending = 15% Invites per Invite
= 20%
invites sender = 2.3 click-through rate
Active Invite Prospective Click Registration Fail
Users Users Process
% of
users who = 50% Succeed
Donât
are active Click Conversion
rate = 85%
Users
⢠Multiplied together, these metrics determine your viral ratio
⢠Which metric offers the highest ROI opportunity?
Copyright Š 2009 YourVersion
39. Adding Metrics and Optimization toÂ
your Product Process
Site Level
Business Product PrioritizedÂ
Plan Objectives Objectives Feature List
Scoping FeatureÂ
Level
RequirementsÂ
Design &Â Design
Code Test Launch
Develop
Metrics & UserÂ
Optimize Feedback
Copyright Š 2009 YourVersion
40. Optimization through Iteration:
Continuous Improvement
Measure
the metric
Analyze
Learning the metric
Gaining knowledge:
⢠Market Identify topÂ
⢠Customer opportunities
to improve
⢠Domain
⢠Usability Design & develop Â
the enhancement
Launch the
enhancement
Copyright Š 2009 YourVersion
42. Questions? www.yourversion.com
dan@yourversion.com
Copyright Š 2009 YourVersion