Product Management by Numbers: Using Metrics To Optimize Your Product by Dan Olsen

Using Metrics to Optimize your Product Product Management’s Role Integrating metrics & optimization Using metrics to optimize value creation Business Value Customer Value UI Design Illustrate with real world examples Copyright © 2008 Olsen Solutions LLC

My Background Education BS, Electrical Engineering, Northwestern MS, Industrial Engineering, Virginia Tech MBA, Stanford PHP, MySQL, JavaScript, XHTML, CSS, UI design 17 years of Product Management Experience Managed submarine design for 5 years 5 years at Intuit, led Quicken Product Management Led Product Management at Friendster Olsen Solutions LLC, PM consultant for start‐ups CEO & Founder of YourVersion, Social Discovery start‐up

A Product Manager by any Other Name Would Smell as Sweet Product managers are sometimes called Product marketing manager Program manager Project manager Label and definition of role can vary Based on industry or company Based on B2C (consumer) vs. B2B (enterprise) Based on stage of company Can be area of responsibility vs. actual position Copyright © 2008 Olsen Solutions LLC

A Process View of Product Management “Inbound” Product “Outbound” Management Product Long Business Product Management Term Strategy Strategy Market/ Sell Short Business Product Product Term Objectives Objectives Development Service/ Support Copyright © 2008 Olsen Solutions LLC

PM’s Job: A Successful Product Know market and customer better than anyone else Translate business objectives and customer needs into product requirements Be the clearinghouse for all product ideas Identify, plan & prioritize product ideas to maximize ROI on engineering resources Work with team to design and build a great product Evangelize product within and outside the company Keep the product team motivated Copyright © 2008 Olsen Solutions LLC

Adding Metrics and Optimization to your Product Process Site Level Business Product Prioritized Plan Objectives Objectives Feature List Scoping Feature Level Requirements Design & Design Code Test Launch Develop Metrics & User Optimize Feedback Copyright © 2008 Olsen Solutions LLC

Optimization through Iteration: Continuous Improvement Measure the metric Analyze Learning the metric Gaining knowledge: • Market Identify top • Customer opportunities to improve • Domain • Usability Design & develop the enhancement Launch the enhancement Copyright © 2008 Olsen Solutions LLC

Define the Equation of your Business “Peeling the Onion” Profit = Revenue ‐ Cost Unique Visitors x Ad Revenue per Visitor Impressions/Visitor x Effective CPM / 1000 Visits/Visitor x Pageviews/Visit x Impressions/PV New Visitors + Returning Visitors Invited Visitors + Uninvited Visitors # of Users Sending Invites x Invites Sent/User x Invite Click‐through Rate Copyright © 2008 Olsen Solutions LLC

How to Track Your Metrics Track each metric as daily time series Unique Page Ad New User … Date Visitors views Revenue Sign‐ups 4/24/08 10,100 29,600 25 490 4/25/08 10,500 27,100 24 480 … Create ratios from primary metrics: X / Y Example: How good is your registration page? Okay: # of registered users per day Better: registration conversion rate = # registered users / # uniques to reg page Copyright © 2008 Olsen Solutions LLC

Sample Signup Page Yield Data Daily Signup Page Yield vs. Time New Registered Users divided by Unique Visitors to Signup Page 100% 90% 80% Daily Signup Page Yield 70% 60% 50% 40% 30% Started requiring registration 20% Changed Added questions messaging to signup page 10% 0% 1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/1 0 Copyright © 2008 Olsen Solutions LLC

Using Metrics to Optimize the Equation of your Business What are the metrics for your business? Where is current value for each metric? How many resources to “move” each metric? Developer‐hours, time, money Which metrics have highest ROI opportunities? Metric A Metric B Metric C Good ROI Bad ROI Great ROI Return Return Return Investment Investment Investment Copyright © 2008 Olsen Solutions LLC

Case Study: Optimizing Viral Loop Metrics % of users sending = 15% Invites per Invite invites sender = 2.3 click-through rate Active Invite Prospective Click Registration Fail Users Users Process % of users who are Succeed Don’t active Click Conversion rate = 85% Users • Multiplied together, these metrics determine your viral ratio • Which metric has highest ROI opportunity? Copyright © 2008 Olsen Solutions LLC

Maximum Upside Potential of a Metric ? 100% 100% 85% 15% 2.3 0 0 0 Registration % of users sending Avg # of invites conversion rate invitations sent per sender Max possible 0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?% improvement Copyright © 2008 Olsen Solutions LLC

The Fuzzy Math of Customer Value You create customer value with a product that Satisfies customers’ needs Is easy to use Has a good price Is better than other alternatives Not fuzzy: # of users and frequency of use Applying metrics approach to customer value Need framework of user needs & satisfaction Quantitative: site analytics, usage metrics, surveys Qualitative: 1‐on‐1 user feedback sessions are best Copyright © 2008 Olsen Solutions LLC

Olsen’s Hierarchy of Web User Needs (adapted from Maslow) Customer’s Perspective What does it mean to us? How easy to use is it? Usability & Design Satisfaction Increasing Does the functionality Feature Set meet my needs? Does the functionality work? Absence of Bugs Dissatisfaction Decreasing Is the site fast enough? Page Load Time Is the site up when I want to use it? Uptime Copyright © 2008 Olsen Solutions LLC

Kano Model: User Needs & Satisfaction User Satisfaction Delighter (wow) Performance (more is better) Need Need not met fully met Must Have Needs & features migrate over time User Dissatisfaction Copyright © 2008 Olsen Solutions LLC

Importance vs. Satisfaction Ask Users to Rate for Each Feature 100 98 Great 95 84 87 90 Bad 86 85 79 84 55 70 80 Importance 80 75 72 80 70 75 65 60 55 41 50 40 50 60 70 80 90 100 Satisfaction Recommended reading: “What Customers Want” by Anthony Ulwick Copyright © 2008 Olsen Solutions LLC

What is Your Value Proposition? Which user benefits are most important? How well does your product deliver? Importance Current User Upside User Benefit to User Satisfaction Potential Low High – + Benefit 1 High Benefit 2 High Benefit 3 Low ? opportunity Benefit 4 Med Benefit 5 Low ? opportunity Benefit 6 Low Doesn’t matter Copyright © 2008 Olsen Solutions LLC

The UI Design Iceberg What most people see and react to Visual Design What good PMs and Designers Interaction think about Design Information Architecture Conceptual Design Recommended reading: Jesse James Garrett’s “Elements of User Experience” chart, free at www.jjg.net Copyright © 2008 Olsen Solutions LLC

Approaching UI Design Analytically Typical UI design question: “When using web pages, do users scroll down?” ‐ Yes ‐ No UI questions are never yes/no! (not binary) Should ask: “What percentage of users …?” UI changes impact your metrics Impact can be positive, negative, small, large Seek high‐ROI UI changes Copyright © 2008 Olsen Solutions LLC

The Fold: “Chrome” Takes Away Pixels Only 768 600 pixels pixels 1024 pixels

Analyze User Screen Height Distribution to Select Design Height for Key Pages 100% 600, 100% 768, 92% 90% % of Users with Given (or Higher) Screen Height 80% 70% Going above 60% 768 px = big drop 50% 40% 772, 38% 864, 29% 30% 807, 29% 1024, 20% 866, 25% 20% 1000, 20% 10% 1030, 5% 0% 600 650 700 750 800 850 900 950 1000 1050 1100 Page Design Height (pixels) Monitor Screen Height (pixels)

Case Study: Account Signup Process Redesign 100% 100% Biggest 80% drop % of Users 62.3% 58.8% 60% 50.9% 40% 34.4% 32.7% 20% 0% Sign in / Account Cash vs. 5 Partner 3 Partner Registration Type Margin Pages Pages Copyright © 2008 Olsen Solutions LLC

Analysis of Sign In/Registration Flow Open Account 55% 44% Register Registration (24% of Total) Process 45% drop off 64% (20% of total) of Total Account 36% overall Selection 83% 30% drop off for 56% (46% of Total) (14% of Total) this step Sign in Forget 70% Change 80% Password (32% of Total) Password (26% of Total) 17% drop off 20% drop off (10% of total) (6% of total) Copyright © 2008 Olsen Solutions LLC

Redesigned User Flow Improved Registration Conversion Rate Abandonment Rate (7 Day Moving Average) Steps 1-2 80% 70% Abandonment Rate (7 Day Moving Average) 60% 50% 40% 30% Released 37% improvement New Design in conversion rate 20% 10% 0% 10/14/02 10/21/02 10/28/02 11/11/02 11/18/02 11/25/02 12/16/02 12/23/02 12/30/02 10/7/02 11/4/02 12/2/02 12/9/02 1/13/03 1/20/03 1/6/03 Copyright © 2008 Olsen Solutions LLC

Product Management by Numbers Cheat Sheet Define what success means Customer value proposition Equation of your business Instrument your site and track key metrics Identify opportunities and prioritize by ROI Launch, learn, and iterate Copyright © 2008 Olsen Solutions LLC

YourVersion proactively discovers relevant web content for you based on your interests. Discover Bookmark Share

Product Management by Numbers: Using Metrics To Optimize Your Product by Dan Olsen

1 comments

Notes on slide 1

35 Favorites