World-Class Web Metrics by Dan Olsen

World‐Class Metrics: Amazon.com $37 B Market Cap $19 B Revenue ‘08 •Automated testing of page elements & suggested products •High degree of user personalization •Real‐time recommendations Copyright © 2009 YourVersion

World‐Class Metrics: Google $137 B Market Cap $22 B Revenue ‘08 •Auto‐testing of UI elements & search results •Automatically learns from user mistakes •Optimize ads to maximize revenue Copyright © 2009 YourVersion

My Web Metrics Epiphany Quicken (1998 – 2001) New version every year: shrink‐wrapped CD‐ROM Pre‐launch Before development: Validated product ideas with users During development: Usability testing & user feedback Post‐launch No “real‐time” metrics data Sales data: 1‐2 month lag, not actionable User surveys: actionable, but not until next product cycle Quicken Brokerage (2001 – 2003) Online brokerage launched in September 2002 The day after launch: LOTS of metrics data → sought more Much faster iteration: metrics → hypothesis → improvement Copyright © 2009 YourVersion

Metrics Maturity Model Unaware Aspiring World‐Class Metrics Comprehensive None Sporadic Tracking breadth & depth Analysis None Ad hoc Recurring Automated Testing & Some A/B None multivariate Optimization tests testing Lack of Partial Clear, dedicated Organization ownership ownership owners Intuition‐ Data‐ Continuous Culture based driven improvement Copyright © 2009 YourVersion

Define the Equation of your Business “Peeling the Onion” Profit = Revenue ‐ Cost Paying Users x Revenue per Paying User New Paying Users + Repeat Paying Users Trial Users x Conv Rate Previous Paying Users x ( 1 – Cancellation Rate ) ( SEO Visitors + SEM Visitors + Viral Visitors ) x Trial Conversion Rate Copyright © 2009 YourVersion

How to Track Your Metrics Track each metric as daily time series Unique Page Ad New User … Date Visitors views Revenue Sign‐ups 4/24/08 10,100 29,600 25 490 4/25/08 10,500 27,100 24 480 … Create ratios from primary metrics: X / Y Example: How good is your registration page? Okay: # of registered users per day Better: registration conversion rate = # registered users / # uniques to reg page Copyright © 2009 YourVersion

Sample Signup Page Yield Data Daily Signup Page Yield vs. Time New Registered Users divided by Unique Visitors to Signup Page 100% 90% 80% Daily Signup Page Yield 70% 60% 50% 40% 30% Started requiring registration 20% Changed Added questions messaging to signup page 10% 0% 1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/1 0 Copyright © 2009 YourVersion

Identifying the “Critical Few” Metrics What are the metrics for your business? Where is current value for each metric? How many resources to “move” each metric? Developer‐hours, time, money Which metrics have highest ROI opportunities? Metric A Metric B Metric C Good ROI Bad ROI Great ROI Return Return Return Investment Investment Investment Copyright © 2009 YourVersion

Turning Metrics into Meaning & Action • Clear owner for each key metric •Focus on trend & changes •Discuss business & customer impact •Propose ideas to improve From Avinash Kaushik’s Occam’s Razor at www.kaushik.net Copyright © 2009 YourVersion

Using Analytics to Clarify The Fuzzy Math of Customer Value You create customer value with a product that Satisfies customers’ needs Is easy to use Has a good price Is better than other alternatives Offline: hard to measure Online: can measure everything Measuring value: # of users, repeat use, revenue Can achieve clarity with metrics and analytics Copyright © 2009 YourVersion

Once You’re Measuring Everything, Where Do You Focus? How do you decide which user benefits and features you should focus on? Need framework to make decisions “Problem space” vs. “Solution space” Importance vs. Satisfaction Importance of user need (problem space) Satisfaction with how well your product meets the user’s need (solution space) Copyright © 2009 YourVersion

Importance vs. Satisfaction Ask Users to Rate for Each Feature 100 98 Great 95 84 87 90 Bad 86 85 79 84 55 70 80 Importance 80 75 72 80 70 75 65 60 55 41 50 40 50 60 70 80 90 100 Satisfaction Recommended reading: “What Customers Want” by Anthony Ulwick Copyright © 2009 YourVersion

Kano Model: User Needs & Satisfaction User Satisfaction Delighter (wow) Performance (more is better) Need Need not met fully met Must Have Needs & features migrate over time User Dissatisfaction Copyright © 2009 YourVersion

Olsen’s Hierarchy of Web User Needs (adapted from Maslow) Customer’s Perspective What does it mean to us? How easy to use is it? Usability & Design Satisfaction Increasing Does the functionality Feature Set meet my needs? Does the functionality work? Absence of Bugs Dissatisfaction Decreasing Is the site fast enough? Page Load Time Is the site up when I want to use it? Uptime Copyright © 2009 YourVersion

Measuring Key Conversions: Conversion Funnel •Tie user actions to business goals •Instrument key steps in user flow •See where users are dropping off •Quantify improvement from changes Copyright © 2009 YourVersion

The UI Design Iceberg What most people see and react to Visual Design What good PMs and Designers Interaction think about Design Information Architecture Conceptual Design Recommended reading: Jesse James Garrett’s “Elements of User Experience” chart, free at www.jjg.net Copyright © 2009 YourVersion

Approaching UI Design Analytically Typical UI design question: “When using web pages, do users scroll down?” ‐ Yes ‐ No UI questions are never yes/no! (not binary) Should ask: “What percentage of users …?” UI changes impact your metrics Impact can be positive, negative, small, large Seek high‐ROI UI changes Copyright © 2009 YourVersion

Put Key Conversion Actions Above The Fold Landing Page A Landing Page B The Fold Key conversion action is above the fold Key conversion action is below the fold Copyright © 2009 Olsen Solutions LLC Copyright © 2009 YourVersion

Marrying UI Design with Metrics Think of “Ease of use” as distinct from the functionality User Interface elements matter Position Layout Size Color Font Text copy Interaction design Navigation Copyright © 2009 YourVersion

Levels of Optimization Maturity A/B testing vs. Multivariate testing A/B: test 2 alternatives (ideally same time) Multivariate: test multiple variables at once Degree of automation & frequency Manual → Automated → Continuous real‐time Level of personalization Universal: same for all users Personalized: varying based on user, user attributes, or user segment Copyright © 2009 YourVersion

Organization and Culture Impact Product Success Roles and Skills required Product Management (PM) UI Design Development Quality Assurance (QA) Operations Cultural traits required Customer‐centric Data‐driven decision‐making with clear ownership Rapid, iterative development Continuous improvement mindset Copyright © 2009 YourVersion

The Product Manager’s Job: A Successful Product Be the expert on the market and the customer Translate business objectives and customer needs into product requirements Be the clearinghouse for all product ideas Work with team to design & build great product Define and track key metrics Use metrics to drive continuous improvement Identify, plan & prioritize product ideas to maximize ROI on engineering resources Copyright © 2009 YourVersion

Case Study 1: Quicken Brokerage Optimizing Sign In/Registration Flow Open Account 55% 44% Register Registration (24% of Total) Process 45% drop off 64% (20% of total) of Total Account 36% overall Selection 83% 30% drop off for 56% (46% of Total) (14% of Total) this step Sign in Forget 70% Change 80% Password (32% of Total) Password (26% of Total) 17% drop off 20% drop off (10% of total) (6% of total) Copyright © 2009 YourVersion

Case Study 2: Friendster Optimizing Viral Growth % of users sending = 15% Invites per Invite = 20% invites sender = 2.3 click-through rate Active Invite Prospective Click Registration Fail Users Users Process % of users who = 50% Succeed Don’t are active Click Conversion rate = 85% Users • Multiplied together, these metrics determine your viral ratio • Which metric offers the highest ROI opportunity? Copyright © 2009 YourVersion

Adding Metrics and Optimization to your Product Process Site Level Business Product Prioritized Plan Objectives Objectives Feature List Scoping Feature Level Requirements Design & Design Code Test Launch Develop Metrics & User Optimize Feedback Copyright © 2009 YourVersion

Optimization through Iteration: Continuous Improvement Measure the metric Analyze Learning the metric Gaining knowledge: • Market Identify top • Customer opportunities to improve • Domain • Usability Design & develop the enhancement Launch the enhancement Copyright © 2009 YourVersion

Summary: How to be World‐Class in Web Metrics Define the equation of your business Identify and track your key metrics Use analytics to gain customer insights Establish clear roles and data‐driven culture Identify opportunities and prioritize by ROI Launch, learn, and iterate Copyright © 2009 YourVersion

World-Class Web Metrics by Dan Olsen

3 comments

Notes on slide 1

24 Favorites