World-Class Web Metrics by Dan Olsen


Published on

Best practices in web metrics and product management

Published in: Business, Technology
  • If you like my presentation, I'm excited to tell you about an all-day workshop I'm hosting on Jun 12th in Palo Alto. It's called 'Toolbox', and we'll be covering best practices in product management, UI design, and marketing with an emphasis on startups.

    Learn more at Please use promo code 'slideshare' for a big discount! Thanks!
    Are you sure you want to  Yes  No
    Your message goes here
  • m3
    Are you sure you want to  Yes  No
    Your message goes here
  • Hi everyone,

    My startup YourVersion launched to the public on Sep 14 at the TechCrunch50 Conference where we won the People's Choice Award. I've developed YourVersion using the product management principles that I talk about in my SlideShares.

    YourVerson is a real-time discovery engine that continuously discovers new, relevant tailored to your interests and makes it easy to bookmark and share that content. In addition to the website, we have a free iPhone app and Firefox toolbar.

    Please check it out at


    Dan Olsen
    CEO & Founder, YourVersion
    Are you sure you want to  Yes  No
    Your message goes here
  • Hi Jon,
    Thanks for letting me know about the Meetup, seems very cool. I can't make the 21st but hope to make the next one.
    Are you sure you want to  Yes  No
    Your message goes here
  • Nice one Dan! (Found this via twitter). You coming to the lean startup thing on the 21st?
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

World-Class Web Metrics by Dan Olsen

  1. World‐Class Web Metrics: How to Optimize Your Product & Business Dan Olsen CEO, YourVersion June 17, 2009 Copyright © 2009 YourVersion
  2. World‐Class Metrics: $37 B Market Cap $19 B Revenue ‘08  •Automated testing  of page elements &  suggested products •High degree of  user personalization •Real‐time  recommendations Copyright © 2009 YourVersion
  3. World‐Class Metrics: Google $137 B Market Cap $22 B Revenue ‘08  •Auto‐testing of UI elements &  search results •Automatically  learns from user  mistakes •Optimize ads to  maximize revenue Copyright © 2009 YourVersion
  4. Overview What is being World‐Class in Metrics? Optimizing Business Value Optimizing the Customer Value of Your Product Copyright © 2009 YourVersion
  5. My Web Metrics Epiphany Quicken (1998 – 2001) New version every year: shrink‐wrapped CD‐ROM Pre‐launch Before development: Validated product ideas with users During development: Usability testing & user feedback Post‐launch No “real‐time” metrics data Sales data: 1‐2 month lag, not actionable User surveys: actionable, but not until next product cycle Quicken Brokerage (2001 – 2003) Online brokerage launched in September 2002 The day after launch: LOTS of metrics data → sought more Much faster iteration: metrics → hypothesis → improvement Copyright © 2009 YourVersion
  6. Metrics Maturity Model Unaware Aspiring World‐Class Metrics Comprehensive  None Sporadic Tracking breadth & depth Analysis None Ad hoc Recurring Automated  Testing &  Some A/B  None multivariate  Optimization tests testing Lack of  Partial  Clear, dedicated  Organization ownership ownership owners Intuition‐ Data‐ Continuous  Culture based driven improvement Copyright © 2009 YourVersion
  7. Optimizing Business Value Copyright © 2009 YourVersion
  8. Approaching Business as an  Optimization Exercise Given reality as it exists today, optimize our business results subject to our resource constraints. Copyright © 2009 YourVersion
  9. Define the Equation of your Business “Peeling the Onion” Profit = Revenue ‐ Cost Paying Users x  Revenue per Paying User New Paying Users +  Repeat Paying Users Trial Users x  Conv Rate Previous Paying Users  x  ( 1 – Cancellation Rate ) ( SEO Visitors + SEM Visitors + Viral Visitors )  x  Trial Conversion Rate Copyright © 2009 YourVersion
  10. How to Track Your Metrics Track each metric as daily time series Unique  Page  Ad  New User  … Date Visitors views Revenue Sign‐ups 4/24/08 10,100 29,600 25 490 4/25/08 10,500 27,100 24 480 … Create ratios from primary metrics:  X / Y Example: How good is your registration page? Okay: # of registered users per day Better: registration conversion rate = # registered users / # uniques to reg page Copyright © 2009 YourVersion
  11. Sample Signup Page Yield Data Daily Signup Page Yield vs. Time New Registered Users divided by Unique Visitors to Signup Page 100% 90% 80% Daily Signup Page Yield 70% 60% 50% 40% 30% Started requiring registration 20% Changed Added questions messaging to signup page 10% 0% 1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/1 0 Copyright © 2009 YourVersion
  12. Identifying the  “Critical Few” Metrics What are the metrics for your business? Where is current value for each metric?  How many resources to “move” each metric? Developer‐hours, time, money Which metrics have highest ROI opportunities? Metric A Metric B Metric C Good ROI Bad ROI Great ROI Return Return Return Investment Investment Investment Copyright © 2009 YourVersion
  13. Turning Metrics into Meaning & Action • Clear owner  for each key  metric •Focus on trend  & changes •Discuss  business &  customer impact •Propose ideas  to improve From Avinash Kaushik’s Occam’s Razor at Copyright © 2009 YourVersion
  14. Optimizing the Customer  Value of Your Product Copyright © 2009 YourVersion
  15. Using Analytics to Clarify The Fuzzy Math of Customer Value You create customer value with a product that Satisfies customers’ needs Is easy to use Has a good price Is better than other alternatives Offline: hard to measure Online: can measure everything Measuring value: # of users, repeat use, revenue Can achieve clarity with metrics and analytics Copyright © 2009 YourVersion
  16. Once You’re Measuring Everything,  Where Do You Focus? How do you decide which user benefits  and features you should focus on? Need framework to make decisions “Problem space” vs. “Solution space” Importance vs. Satisfaction Importance of user need (problem space) Satisfaction with how well your product meets  the user’s need (solution space) Copyright © 2009 YourVersion
  17. Importance vs. Satisfaction Ask Users to Rate for Each Feature 100 98 Great 95 84 87 90 Bad 86 85 79 84 55 70 80 Importance 80 75 72 80 70 75 65 60 55 41 50 40 50 60 70 80 90 100 Satisfaction Recommended reading: “What  Customers Want” by Anthony Ulwick Copyright © 2009 YourVersion
  18. Kano Model: User Needs & Satisfaction User Satisfaction Delighter (wow) Performance  (more is better) Need Need not met fully met Must Have Needs & features  migrate over time User Dissatisfaction Copyright © 2009 YourVersion
  19. Olsen’s Hierarchy of Web User Needs (adapted from Maslow) Customer’s Perspective What does it mean to us? How easy to use is it? Usability & Design Satisfaction Increasing Does the functionality Feature Set meet my needs? Does the functionality work? Absence of Bugs Dissatisfaction Decreasing Is the site fast enough? Page Load Time Is the site up when I want to use it? Uptime Copyright © 2009 YourVersion
  20. Using Web Analytics Tools to  Understand Your Users Tracking visitors and traffic Seeing where users are clicking Measuring key conversions Monitoring user feedback Copyright © 2009 YourVersion
  21. Basic Tracking of Traffic: Google Analytics •Unique  visitors •New vs.  returning •Pageviews •Time on site •Top referrers •Top geos Copyright © 2009 YourVersion
  22. Seeing Where Users are Clicking:  CrazyEgg Heatmap •Shows Click  Density •Color indicates  % of clicks •See which links  perform best •See impact of  UI changes Copyright © 2009 YourVersion
  23. Measuring Key Conversions: Conversion Funnel •Tie user actions to  business goals •Instrument key steps in  user flow •See where users are  dropping off •Quantify improvement  from changes Copyright © 2009 YourVersion
  24. Monitoring User Feedback  Quantitatively with Kampyle Unobtrusive Solicitation Average Grade over time Simple popup Average Grade  for each page  on your site Copyright © 2009 YourVersion
  25. The UI Design Iceberg What most people see and react to Visual Design What good PMs and Designers Interaction think about Design Information Architecture Conceptual Design Recommended reading: Jesse James Garrett’s “Elements of User Experience” chart, free at Copyright © 2009 YourVersion
  26. Approaching UI Design Analytically Typical UI design question: “When using web pages, do users scroll down?” ‐ Yes ‐ No UI questions are never yes/no! (not binary) Should ask: “What percentage of users …?” UI changes impact your metrics Impact can be positive, negative, small, large Seek high‐ROI UI changes Copyright © 2009 YourVersion
  27. Put Key Conversion Actions Above The Fold Landing Page A Landing Page B The Fold Key conversion action is above the fold Key conversion action is below the fold Copyright © 2009 Olsen Solutions LLC Copyright © 2009 YourVersion
  28. Marrying UI Design with Metrics Think of “Ease of use” as distinct from the  functionality User Interface  elements matter Position Layout Size Color Font Text copy Interaction design Navigation Copyright © 2009 YourVersion
  29. Levels of Optimization Maturity A/B testing vs. Multivariate testing A/B: test 2 alternatives (ideally same time) Multivariate: test multiple variables at once Degree of automation & frequency Manual → Automated → Continuous real‐time Level of personalization Universal: same for all users Personalized: varying based on user, user  attributes, or user segment Copyright © 2009 YourVersion
  30. Automated Multivariate Testing: Google Website Optimizer Copyright © 2009 YourVersion
  31. Organization and Culture Impact Product Success Roles and Skills required Product Management (PM) UI Design Development Quality Assurance (QA) Operations Cultural traits required Customer‐centric Data‐driven decision‐making with clear ownership Rapid, iterative development Continuous improvement mindset Copyright © 2009 YourVersion
  32. The Product Manager’s Job: A Successful Product Be the expert on the market and the customer Translate business objectives and customer  needs into product requirements Be the clearinghouse for all product ideas  Work with team to design & build great product Define and track key metrics Use metrics to drive continuous improvement Identify, plan & prioritize product ideas to  maximize ROI on engineering resources Copyright © 2009 YourVersion
  33. Prioritizing Product Ideas by ROI ? Return (Value Created) 4 Idea D 3 Idea A Idea B 2 Idea C 1 Idea F 1 2 3 4 Investment (developer‐weeks) Copyright © 2009 YourVersion
  34. Case Study 1: Quicken Brokerage Optimizing Sign In/Registration Flow Open Account 55% 44% Register Registration  (24% of Total) Process 45% drop off 64% (20% of total) of Total Account  36% overall  Selection 83% 30% drop off for  56% (46% of Total) (14% of Total) this step Sign in Forget  70% Change  80% Password (32% of Total) Password (26% of Total) 17% drop off  20% drop off (10% of total) (6% of total) Copyright © 2009 YourVersion
  35. Redesigned User Flow Improved  Registration Conversion Rate 37% Released 37% improvement New Design in conversion rate Copyright © 2009 YourVersion
  36. Case Study 2: Friendster Optimizing Viral Growth % of users sending = 15% Invites per Invite = 20% invites sender = 2.3 click-through rate Active Invite Prospective Click Registration Fail Users Users Process % of users who = 50% Succeed Don’t are active Click Conversion rate = 85% Users • Multiplied together, these metrics determine your viral ratio • Which metric offers the highest ROI opportunity? Copyright © 2009 YourVersion
  37. Case Study 2: Friendster Doubled Number of Invitations Sent per Sender Copyright © 2009 YourVersion
  38. Continuous Improvement Copyright © 2009 YourVersion
  39. Adding Metrics and Optimization to  your Product Process Site Level Business Product Prioritized  Plan Objectives Objectives Feature List Scoping Feature  Level Requirements  Design & Design Code Test Launch Develop Metrics & User  Optimize Feedback Copyright © 2009 YourVersion
  40. Optimization through Iteration: Continuous Improvement Measure the metric Analyze Learning the metric Gaining knowledge: • Market Identify top  • Customer opportunities to improve • Domain • Usability Design & develop   the enhancement Launch the enhancement Copyright © 2009 YourVersion
  41. Summary: How to be World‐Class in Web Metrics Define the equation of your business Identify and track your key metrics Use analytics to gain customer insights Establish clear roles and data‐driven culture Identify opportunities and prioritize by ROI Launch, learn, and iterate Copyright © 2009 YourVersion
  42. Questions? Copyright © 2009 YourVersion