0
World‐Class Web Metrics:
How to Optimize Your Product & Business
Dan Olsen
CEO, YourVersion
June 17, 2009
                ...
World‐Class Metrics:
  Amazon.com
                $37 B Market Cap
                $19 B Revenue ‘08 
                •Aut...
World‐Class Metrics:
      Google
                 $137 B Market Cap
                 $22 B Revenue ‘08 
                 ...
Overview
What is being World‐Class in Metrics?
Optimizing Business Value
Optimizing the Customer Value of
Your Product



...
My Web Metrics Epiphany
Quicken (1998 – 2001)
  New version every year: shrink‐wrapped CD‐ROM
  Pre‐launch
     Before dev...
Metrics Maturity Model
              Unaware Aspiring       World‐Class
Metrics                             Comprehensive ...
Optimizing Business Value




                  Copyright © 2009 YourVersion
Approaching Business as an 
    Optimization Exercise


Given reality as it exists today,
optimize our business results
su...
Define the Equation of your Business
          “Peeling the Onion”
Profit = Revenue ‐ Cost

   Paying Users x  Revenue per...
How to Track Your Metrics
Track each metric as daily time series
           Unique     Page       Ad      New User 
      ...
Sample Signup Page Yield Data
                                                Daily Signup Page Yield vs. Time
           ...
Identifying the  “Critical Few” Metrics
  What are the metrics for your business?
  Where is current value for each metric...
Turning Metrics into Meaning & Action
• Clear owner 
for each key 
metric
•Focus on trend 
& changes
•Discuss 
business & ...
Optimizing the Customer 
 Value of Your Product



                 Copyright © 2009 YourVersion
Using Analytics to Clarify The
   Fuzzy Math of Customer Value
You create customer value with a product that
  Satisfies c...
Once You’re Measuring Everything, 
      Where Do You Focus?
How do you decide which user benefits 
and features you shoul...
Importance vs. Satisfaction
       Ask Users to Rate for Each Feature
              100                                   ...
Kano Model: User Needs & Satisfaction
              User Satisfaction
                                  Delighter (wow)


...
Olsen’s Hierarchy of Web User Needs
       (adapted from Maslow)
    Customer’s Perspective                               ...
Using Web Analytics Tools to 
      Understand Your Users
Tracking visitors and traffic
Seeing where users are clicking
Me...
Basic Tracking of Traffic:
                Google Analytics
•Unique 
visitors
•New vs. 
returning
•Pageviews
•Time on site...
Seeing Where Users are Clicking: 
           CrazyEgg Heatmap
•Shows Click 
Density
•Color indicates 
% of clicks
•See whi...
Measuring Key Conversions:
           Conversion Funnel
•Tie user actions to 
business goals
•Instrument key steps in 
use...
Monitoring User Feedback 
            Quantitatively with Kampyle


    Unobtrusive
    Solicitation                  Aver...
The UI Design Iceberg
  What most
  people see
  and react to                       Visual
                               ...
Approaching UI Design Analytically
 Typical UI design question:
 “When using web pages, do users scroll down?”
   ‐ Yes
  ...
Put Key Conversion Actions Above The Fold
          Landing Page A                 Landing Page B




                    ...
Marrying UI Design with Metrics
Think of “Ease of use”
as distinct from the 
functionality
User Interface 
elements matter...
Levels of Optimization Maturity
A/B testing vs. Multivariate testing
  A/B: test 2 alternatives (ideally same time)
  Mult...
Automated Multivariate Testing:
  Google Website Optimizer




                       Copyright © 2009 YourVersion
Organization and Culture
      Impact Product Success
Roles and Skills required
  Product Management (PM)
  UI Design
  De...
The Product Manager’s Job:
       A Successful Product
Be the expert on the market and the customer
Translate business obj...
Prioritizing Product Ideas by ROI

                                         ?
Return (Value Created)




                 ...
Case Study 1: Quicken Brokerage
     Optimizing Sign In/Registration Flow
 Open
Account
                                  ...
Redesigned User Flow Improved 
Registration Conversion Rate 37%




             Released    37% improvement
            N...
Case Study 2: Friendster
                      Optimizing Viral Growth
        % of users
         sending = 15%   Invites...
Case Study 2: Friendster
Doubled Number of Invitations Sent per Sender




                                Copyright © 200...
Continuous
Improvement



          Copyright © 2009 YourVersion
Adding Metrics and Optimization to 
             your Product Process
                                                    ...
Optimization through Iteration:
  Continuous Improvement
                   Measure
                  the metric


       ...
Summary:
How to be World‐Class in Web Metrics
 Define the equation of your business
 Identify and track your key metrics
 ...
Questions?   www.yourversion.com
             dan@yourversion.com
                      Copyright © 2009 YourVersion
Upcoming SlideShare
Loading in...5
×

World-Class Web Metrics by Dan Olsen

7,656

Published on

Best practices in web metrics and product management

Published in: Business, Technology
5 Comments
39 Likes
Statistics
Notes
No Downloads
Views
Total Views
7,656
On Slideshare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
475
Comments
5
Likes
39
Embeds 0
No embeds

No notes for slide

Transcript of "World-Class Web Metrics by Dan Olsen"

  1. 1. World‐Class Web Metrics: How to Optimize Your Product & Business Dan Olsen CEO, YourVersion June 17, 2009 Copyright © 2009 YourVersion
  2. 2. World‐Class Metrics: Amazon.com $37 B Market Cap $19 B Revenue ‘08  •Automated testing  of page elements &  suggested products •High degree of  user personalization •Real‐time  recommendations Copyright © 2009 YourVersion
  3. 3. World‐Class Metrics: Google $137 B Market Cap $22 B Revenue ‘08  •Auto‐testing of UI elements &  search results •Automatically  learns from user  mistakes •Optimize ads to  maximize revenue Copyright © 2009 YourVersion
  4. 4. Overview What is being World‐Class in Metrics? Optimizing Business Value Optimizing the Customer Value of Your Product Copyright © 2009 YourVersion
  5. 5. My Web Metrics Epiphany Quicken (1998 – 2001) New version every year: shrink‐wrapped CD‐ROM Pre‐launch Before development: Validated product ideas with users During development: Usability testing & user feedback Post‐launch No “real‐time” metrics data Sales data: 1‐2 month lag, not actionable User surveys: actionable, but not until next product cycle Quicken Brokerage (2001 – 2003) Online brokerage launched in September 2002 The day after launch: LOTS of metrics data → sought more Much faster iteration: metrics → hypothesis → improvement Copyright © 2009 YourVersion
  6. 6. Metrics Maturity Model Unaware Aspiring World‐Class Metrics Comprehensive  None Sporadic Tracking breadth & depth Analysis None Ad hoc Recurring Automated  Testing &  Some A/B  None multivariate  Optimization tests testing Lack of  Partial  Clear, dedicated  Organization ownership ownership owners Intuition‐ Data‐ Continuous  Culture based driven improvement Copyright © 2009 YourVersion
  7. 7. Optimizing Business Value Copyright © 2009 YourVersion
  8. 8. Approaching Business as an  Optimization Exercise Given reality as it exists today, optimize our business results subject to our resource constraints. Copyright © 2009 YourVersion
  9. 9. Define the Equation of your Business “Peeling the Onion” Profit = Revenue ‐ Cost Paying Users x  Revenue per Paying User New Paying Users +  Repeat Paying Users Trial Users x  Conv Rate Previous Paying Users  x  ( 1 – Cancellation Rate ) ( SEO Visitors + SEM Visitors + Viral Visitors )  x  Trial Conversion Rate Copyright © 2009 YourVersion
  10. 10. How to Track Your Metrics Track each metric as daily time series Unique  Page  Ad  New User  … Date Visitors views Revenue Sign‐ups 4/24/08 10,100 29,600 25 490 4/25/08 10,500 27,100 24 480 … Create ratios from primary metrics:  X / Y Example: How good is your registration page? Okay: # of registered users per day Better: registration conversion rate = # registered users / # uniques to reg page Copyright © 2009 YourVersion
  11. 11. Sample Signup Page Yield Data Daily Signup Page Yield vs. Time New Registered Users divided by Unique Visitors to Signup Page 100% 90% 80% Daily Signup Page Yield 70% 60% 50% 40% 30% Started requiring registration 20% Changed Added questions messaging to signup page 10% 0% 1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/1 0 Copyright © 2009 YourVersion
  12. 12. Identifying the  “Critical Few” Metrics What are the metrics for your business? Where is current value for each metric?  How many resources to “move” each metric? Developer‐hours, time, money Which metrics have highest ROI opportunities? Metric A Metric B Metric C Good ROI Bad ROI Great ROI Return Return Return Investment Investment Investment Copyright © 2009 YourVersion
  13. 13. Turning Metrics into Meaning & Action • Clear owner  for each key  metric •Focus on trend  & changes •Discuss  business &  customer impact •Propose ideas  to improve From Avinash Kaushik’s Occam’s Razor at www.kaushik.net Copyright © 2009 YourVersion
  14. 14. Optimizing the Customer  Value of Your Product Copyright © 2009 YourVersion
  15. 15. Using Analytics to Clarify The Fuzzy Math of Customer Value You create customer value with a product that Satisfies customers’ needs Is easy to use Has a good price Is better than other alternatives Offline: hard to measure Online: can measure everything Measuring value: # of users, repeat use, revenue Can achieve clarity with metrics and analytics Copyright © 2009 YourVersion
  16. 16. Once You’re Measuring Everything,  Where Do You Focus? How do you decide which user benefits  and features you should focus on? Need framework to make decisions “Problem space” vs. “Solution space” Importance vs. Satisfaction Importance of user need (problem space) Satisfaction with how well your product meets  the user’s need (solution space) Copyright © 2009 YourVersion
  17. 17. Importance vs. Satisfaction Ask Users to Rate for Each Feature 100 98 Great 95 84 87 90 Bad 86 85 79 84 55 70 80 Importance 80 75 72 80 70 75 65 60 55 41 50 40 50 60 70 80 90 100 Satisfaction Recommended reading: “What  Customers Want” by Anthony Ulwick Copyright © 2009 YourVersion
  18. 18. Kano Model: User Needs & Satisfaction User Satisfaction Delighter (wow) Performance  (more is better) Need Need not met fully met Must Have Needs & features  migrate over time User Dissatisfaction Copyright © 2009 YourVersion
  19. 19. Olsen’s Hierarchy of Web User Needs (adapted from Maslow) Customer’s Perspective What does it mean to us? How easy to use is it? Usability & Design Satisfaction Increasing Does the functionality Feature Set meet my needs? Does the functionality work? Absence of Bugs Dissatisfaction Decreasing Is the site fast enough? Page Load Time Is the site up when I want to use it? Uptime Copyright © 2009 YourVersion
  20. 20. Using Web Analytics Tools to  Understand Your Users Tracking visitors and traffic Seeing where users are clicking Measuring key conversions Monitoring user feedback Copyright © 2009 YourVersion
  21. 21. Basic Tracking of Traffic: Google Analytics •Unique  visitors •New vs.  returning •Pageviews •Time on site •Top referrers •Top geos Copyright © 2009 YourVersion
  22. 22. Seeing Where Users are Clicking:  CrazyEgg Heatmap •Shows Click  Density •Color indicates  % of clicks •See which links  perform best •See impact of  UI changes Copyright © 2009 YourVersion
  23. 23. Measuring Key Conversions: Conversion Funnel •Tie user actions to  business goals •Instrument key steps in  user flow •See where users are  dropping off •Quantify improvement  from changes Copyright © 2009 YourVersion
  24. 24. Monitoring User Feedback  Quantitatively with Kampyle Unobtrusive Solicitation Average Grade over time Simple popup Average Grade  for each page  on your site Copyright © 2009 YourVersion
  25. 25. The UI Design Iceberg What most people see and react to Visual Design What good PMs and Designers Interaction think about Design Information Architecture Conceptual Design Recommended reading: Jesse James Garrett’s “Elements of User Experience” chart, free at www.jjg.net Copyright © 2009 YourVersion
  26. 26. Approaching UI Design Analytically Typical UI design question: “When using web pages, do users scroll down?” ‐ Yes ‐ No UI questions are never yes/no! (not binary) Should ask: “What percentage of users …?” UI changes impact your metrics Impact can be positive, negative, small, large Seek high‐ROI UI changes Copyright © 2009 YourVersion
  27. 27. Put Key Conversion Actions Above The Fold Landing Page A Landing Page B The Fold Key conversion action is above the fold Key conversion action is below the fold Copyright © 2009 Olsen Solutions LLC Copyright © 2009 YourVersion
  28. 28. Marrying UI Design with Metrics Think of “Ease of use” as distinct from the  functionality User Interface  elements matter Position Layout Size Color Font Text copy Interaction design Navigation Copyright © 2009 YourVersion
  29. 29. Levels of Optimization Maturity A/B testing vs. Multivariate testing A/B: test 2 alternatives (ideally same time) Multivariate: test multiple variables at once Degree of automation & frequency Manual → Automated → Continuous real‐time Level of personalization Universal: same for all users Personalized: varying based on user, user  attributes, or user segment Copyright © 2009 YourVersion
  30. 30. Automated Multivariate Testing: Google Website Optimizer Copyright © 2009 YourVersion
  31. 31. Organization and Culture Impact Product Success Roles and Skills required Product Management (PM) UI Design Development Quality Assurance (QA) Operations Cultural traits required Customer‐centric Data‐driven decision‐making with clear ownership Rapid, iterative development Continuous improvement mindset Copyright © 2009 YourVersion
  32. 32. The Product Manager’s Job: A Successful Product Be the expert on the market and the customer Translate business objectives and customer  needs into product requirements Be the clearinghouse for all product ideas  Work with team to design & build great product Define and track key metrics Use metrics to drive continuous improvement Identify, plan & prioritize product ideas to  maximize ROI on engineering resources Copyright © 2009 YourVersion
  33. 33. Prioritizing Product Ideas by ROI ? Return (Value Created) 4 Idea D 3 Idea A Idea B 2 Idea C 1 Idea F 1 2 3 4 Investment (developer‐weeks) Copyright © 2009 YourVersion
  34. 34. Case Study 1: Quicken Brokerage Optimizing Sign In/Registration Flow Open Account 55% 44% Register Registration  (24% of Total) Process 45% drop off 64% (20% of total) of Total Account  36% overall  Selection 83% 30% drop off for  56% (46% of Total) (14% of Total) this step Sign in Forget  70% Change  80% Password (32% of Total) Password (26% of Total) 17% drop off  20% drop off (10% of total) (6% of total) Copyright © 2009 YourVersion
  35. 35. Redesigned User Flow Improved  Registration Conversion Rate 37% Released 37% improvement New Design in conversion rate Copyright © 2009 YourVersion
  36. 36. Case Study 2: Friendster Optimizing Viral Growth % of users sending = 15% Invites per Invite = 20% invites sender = 2.3 click-through rate Active Invite Prospective Click Registration Fail Users Users Process % of users who = 50% Succeed Don’t are active Click Conversion rate = 85% Users • Multiplied together, these metrics determine your viral ratio • Which metric offers the highest ROI opportunity? Copyright © 2009 YourVersion
  37. 37. Case Study 2: Friendster Doubled Number of Invitations Sent per Sender Copyright © 2009 YourVersion
  38. 38. Continuous Improvement Copyright © 2009 YourVersion
  39. 39. Adding Metrics and Optimization to  your Product Process Site Level Business Product Prioritized  Plan Objectives Objectives Feature List Scoping Feature  Level Requirements  Design & Design Code Test Launch Develop Metrics & User  Optimize Feedback Copyright © 2009 YourVersion
  40. 40. Optimization through Iteration: Continuous Improvement Measure the metric Analyze Learning the metric Gaining knowledge: • Market Identify top  • Customer opportunities to improve • Domain • Usability Design & develop   the enhancement Launch the enhancement Copyright © 2009 YourVersion
  41. 41. Summary: How to be World‐Class in Web Metrics Define the equation of your business Identify and track your key metrics Use analytics to gain customer insights Establish clear roles and data‐driven culture Identify opportunities and prioritize by ROI Launch, learn, and iterate Copyright © 2009 YourVersion
  42. 42. Questions? www.yourversion.com dan@yourversion.com Copyright © 2009 YourVersion
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×