Early Stage Web Product Management by Dan Olsen

What You Wanted to Hear About 1 2 3 4 5 6 7 8 9 10 Usability Understanding user needs Prioritizing features Translating learnings from metrics into action Maximizing ROI on development resources Setting up and tracking the right metrics UI design Copyright © 2009 YourVersion

Agenda Understanding customer needs Prioritizing, scoping, and maximizing ROI on engineering resources UI design Usability testing Using metrics for optimization Will post slides to slideshare.net/dan_o Copyright © 2009 YourVersion

My Background Education BS, Electrical Engineering, Northwestern MS, Industrial Engineering, Virginia Tech MBA, Stanford PHP, MySQL, JavaScript, XHTML, CSS, UI design 18 years of Product Management Experience Managed submarine design for 5 years 5 years at Intuit, led Quicken Product Management Led Product Management at Friendster Olsen Solutions LLC, PM consultant for startups CEO & Founder of YourVersion, real‐time discovery startup Copyright © 2009 YourVersion

How is developing Early Stage Products Different from Later Stage Products? With an early stage product, you have WAY MORE uncertainty about: Who your target customers really are Which customer needs you should address How to best meet those needs What design works best These are qualitative learnings/decisions Quantitative is also valuable (later) Copyright © 2009 YourVersion

The Product Manager’s Job: A Successful Product Be the expert on the market and the customer Translate business objectives and customer needs into product requirements Be the clearinghouse for all product ideas Work with team to design & build great product Define and track key metrics Identify, plan & prioritize product ideas to maximize ROI on engineering resources Copyright © 2009 YourVersion

Problem Space vs. Solution Space Problem Space Solution Space A customer problem, A specific need, or benefit that the implementation to product should address address the need or A product requirement product requirement Example: Ability to write in space NASA: space pen (zero gravity) ($1 M R&D cost) Russians: pencil Copyright © 2009 YourVersion

Problem Space vs. Solution Space Feature Level Problem Space Solution Space (user benefit) (feature) Gmail Make it easy importer to share a link with my friends Design Design Design #1 #2 #3 Preview with User can edit Allow me to Design checkboxes before import reuse my email #1 No No contacts #2 Yes No #3 Yes Yes Copyright © 2009 YourVersion

Types of Customer Benefits Functional Benefits Allow the user to do something they couldn’t do before Deliver a benefit more quickly, conveniently, or cheaply Booking travel more easily (travel sites) Talking for less money (VOIP software) Emotional Benefits Control (Quicken) Feeling informed (news sites) Confidence / security (anti‐virus software) Enjoyment / entertainment (YouTube) Self‐expression (MySpace, Facebook) Copyright © 2009 YourVersion

The Customer Benefits “Ladder” …which means one less thing to Higher‐level worry about in my hectic life benefit (more abstract) …which makes me feel more in control of my finances …which gives me a clear picture of how much money I have Lower‐level Quicken makes it easy for me benefit to balance my checkbook (more specific) Copyright © 2009 YourVersion

How Do You Prioritize User Benefits and Product Features? Need a framework for prioritization Which user benefits should you address? Which product features to build (or improve)? Importance vs. Satisfaction Importance of user need (problem space) Satisfaction with how well a product meets the user’s need (solution space) Opportunity = High Importance need with low Satisfaction Copyright © 2009 YourVersion

What is Your Value Proposition? Which user benefits are you going to provide? How will your product be better than rest? Importance Current User Upside User Benefit to User Satisfaction Potential Low High – + Benefit 1 High Benefit 2 High Benefit 3 Low ? opportunity Benefit 4 Med Benefit 5 Low ? opportunity Benefit 6 Low Doesn’t matter Copyright © 2009 YourVersion

Importance vs. Satisfaction Ask Users to Rate for Each Feature 100 98 Great 95 84 87 90 Bad 86 85 79 84 55 70 80 Importance 80 75 72 80 70 75 65 60 55 41 50 40 50 60 70 80 90 100 Satisfaction Recommended reading: “What Customers Want” by Anthony Ulwick Copyright © 2009 YourVersion

Kano Model: User Needs & Satisfaction User Satisfaction Delighter (wow) Performance (more is better) Need Need not met fully met Must Have Needs & features migrate over time User Dissatisfaction Copyright © 2009 YourVersion

Olsen’s Hierarchy of Web User Needs (adapted from Maslow) Customer’s Perspective What does it mean to us? How easy to use is it? Usability & Design Satisfaction Increasing Does the functionality Feature Set meet my needs? Does the functionality work? Absence of Bugs Dissatisfaction Decreasing Is the site fast enough? Page Load Time Is the site up when I want to use it? Uptime Copyright © 2009 YourVersion

Prioritization and Scope Customer value is only half the equation How much engineering effort will it take? Need to consider value and effort (ROI) Ruthlessly prioritize: rank order Be deliberate about scope & keep it small It’s easy to try to do too much Strategy = deciding what you’re NOT doing Break features down into smaller chunks Smaller scope → faster iterations → better Copyright © 2009 YourVersion

User Benefits vs. Ease of Use Q: If two products equally deliver the exact same user benefits, which product is better? A: The product that’s easier to use ‘Ease of use’ provides benefits Saves time Reduces cognitive load Reduces frustration Not many companies excel at UI design ‘Ease of use’ can be differentiator Copyright © 2009 YourVersion

The UI Design Iceberg What most people see and react to Visual Design What good PMs and Designers Interaction think about Design Information Architecture Conceptual Design Recommended reading: Jesse James Garrett’s “Elements of User Experience” chart, free at www.jjg.net Copyright © 2009 YourVersion

Elements of User Interface Design Consists of Three Distinct Elements: Information Architecture Structure and layout at both site and page level How site is structured (sitemap) How site information is organized (site layout) How each page is organized (page layout) Interaction Design How user and product interact with one another User flows (e.g., navigation across multiple pages) User input (e.g., controls and form design) Visual Design “How it looks” vs. “What it is”, often called “chrome” Fonts, colors, graphical elements Copyright © 2009 YourVersion

Information Architecture Documents used Sitemap Show how sections of website are organized Show major navigation patterns Wireframes Show the layout of components on a page Does NOT focus on visual design Black & White No graphics Templates for overall website and individual pages Tools: Visio, OmniGraffle, Axure, Powerpoint, Word, Excel, Photoshop, Balsamiq, WriteMaps, whiteboard Copyright © 2009 YourVersion

Interaction Design Documents used Flowchart Combination of Wireframes & Flowcharts Tools: Visio, OmniGraffle, Powerpoint, Photoshop, whiteboard May build prototype using HTML, jQuery, Ruby on Rails, Flash, or paper Usability testing can help find problems Copyright © 2009 YourVersion

The Value of Usability Testing Critical to talk with customers 1‐on‐1 Gain better understanding of Usability issues with your product Customer needs and problems What alternatives customers are using, pros & cons of each, customer preferences QA: use cases & bugs you haven’t seen Really a “user learning” session Make test as real for user as possible Copyright © 2009 YourVersion

“Ramen” Usability for Startups Anyone can do it! Ingredients: 1 user with their laptop 1 desk 1 person to conduct usability Pen and paper N optional observers N+2 chairs Copyright © 2009 YourVersion

Typical Format for Customer Session 5 ‐ 10 min: Ask questions to understand user needs and solutions they currently use 30 ‐ 50 min: Usability test Non‐directed as much as possible When necessary, direct user to attempt to perform a specific task 5 ‐ 10 min: Wrap‐up Answer any user questions that came up Point out/explain features you want to highlight Copyright © 2009 YourVersion

Dos & Don’ts of Conducting Usability Do Explain to the user: Their usability test will help improve the product Not to worry about hurting your feelings “Think Aloud Protocol” Ask user to attempt the task, then be a fly on the wall Ask non‐leading, open‐ended questions Take notes and review them afterwards for take‐aways Don’t Ask leading questions “Help” the user or explain the UI (e.g., “click over here”) Respond to user frustration or questions (until test is over) Get defensive Blame the user Copyright © 2009 YourVersion

Usability Case Study: Travel Sites Consider 3 major travel sites Expedia, Travelocity, Orbitz All 3 try to provide same user benefits Easily find flights that match your schedule Easily find the lowest price flights UI Design differences make some better Case study: round trip from San Fran to NYC Want to find best combo of price and airports 3 possible airports for each city 9 possible one‐way airport combinations 81 possible round‐trip airport combinations Copyright © 2009 YourVersion

Expedia: Only One Airport Combo at a Time • Have to manually check all 9 combos •3 clicks to change airport •Then wait for new results •24 clicks + 8 page reloads to see all 9 combos Copyright © 2009 YourVersion

Travelocity: Can only view results 1 combo at a time • Clicking button brings up results for this combo •Problem: browser Back button loses other airports! •Have to go through ‘Change Search’ process = 9 clicks + 4 page reloads for each combo •72 clicks + 32 page loads to see other 8 combos Copyright © 2009 YourVersion

Orbitz: Binary choices to include nearby airports • Good news: all combos appear in results •80‐mile radius might include airports I don’t want (3 for SFO, 5 for NYC) but turned out not to be an issue Copyright © 2009 YourVersion

Summary Comparison of Travel Sites User Benefit Expedia Travelocity Orbitz Ability to include other Yes Yes Yes nearby airports Ability to pick specific High High Low nearby airports (by changing) (can pre‐select) Ease of seeing results Med Low High for multiple airports Ease of trading off Low High Med airport combos vs. price Airline vs. Number of Yes Yes Yes Connections Price Grid Flight Results Sorting Med High Low Options Overall ability to easily Med Low High find best airport combo Copyright © 2009 YourVersion

Define the Equation of your Business “Peeling the Onion” Advertising Business Model: Profit = Revenue ‐ Cost Unique Visitors x Ad Revenue per Visitor Impressions/Visitor x Effective CPM / 1000 Visits/Visitor x Pageviews/Visit x Impressions/PV New Visitors + Returning Visitors Invited Visitors + Uninvited Visitors # of Users Sending Invites x Invites Sent/User x Invite Conversion Rate Copyright © 2009 YourVersion

Equation of your Business Subscription Business Model Profit = Revenue ‐ Cost Paying Users x Revenue per Paying User New Paying Users + Repeat Paying Users Trial Users x Conv Rate Previous Paying Users x ( 1 – Cancellation Rate ) ( SEO Visitors + SEM Visitors + Viral Visitors ) x Trial Conversion Rate Copyright © 2009 YourVersion

How to Track Your Metrics Track each metric as daily time series Unique Page Ad New User … Date Visitors views Revenue Sign‐ups 4/24/08 10,100 29,600 25 490 4/25/08 10,500 27,100 24 480 … Create ratios from primary metrics: X / Y Example: How good is your registration page? Okay: # of registered users per day Better: registration conversion rate = # registered users / # uniques to reg page Copyright © 2009 YourVersion

Sample Signup Page Yield Data Daily Signup Page Yield vs. Time New Registered Users divided by Unique Visitors to Signup Page 100% 90% 80% Daily Signup Page Yield 70% 60% 50% 40% 30% Started requiring registration 20% Changed Added questions messaging to signup page 10% 0% 1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/1 0 Copyright © 2009 YourVersion

Identifying the “Critical Few” Metrics What are the metrics for your business? Where is current value for each metric? How many resources to “move” each metric? Developer‐hours, time, money Which metrics have highest ROI opportunities? Metric A Metric B Metric C Good ROI Bad ROI Great ROI Return Return Return Investment Investment Investment Copyright © 2009 YourVersion

Measuring Key Conversions: Conversion Funnel •Tie user actions to business goals •Instrument key steps in user flow •See where users are dropping off •Quantify improvement from changes Copyright © 2009 YourVersion

Approaching UI Design Analytically Typical UI design question: “When using web pages, do users scroll down?” ‐ Yes ‐ No UI questions are never yes/no! (not binary) Should ask: “What percentage of users …?” UI changes impact your metrics Impact can be positive, negative, small, large Seek high‐ROI UI changes Copyright © 2009 YourVersion

Put Key Conversion Actions Above The Fold Landing Page A Landing Page B The Fold Key conversion action is above the fold Key conversion action is below the fold Copyright © 2009 Olsen Solutions LLC Copyright © 2009 YourVersion

Analyze User Screen Height Distribution to Select Design Height for Key Pages 100% 600, 100% 768, 92% 90% % of Users with Given (or Higher) Screen Height 80% 70% Going above 60% 768 px = big drop 50% 40% 772, 38% 864, 29% 30% 807, 29% 1024, 20% 866, 25% 20% 1000, 20% 10% 1030, 5% 0% 600 650 700 750 800 850 900 950 1000 1050 1100 Page Design Height (pixels) Monitor Screen Height (pixels) Copyright © 2009 YourVersion

Marrying UI Design with Metrics Think of “Ease of use” as distinct from the functionality User Interface elements matter Position Layout Size Color Font Text copy Interaction design Navigation Copyright © 2009 YourVersion

Case Study 1: Quicken Brokerage Optimizing Sign In/Registration Flow 100% 100% Biggest 80% drop % of Users 62.3% 58.8% 60% 50.9% 40% 34.4% 32.7% 20% 0% Sign in / Account Cash vs. 5 Partner 3 Partner Registration Type Margin Pages Pages Copyright © 2009 YourVersion

Mapping the Flow to See Where Users Were Dropping Off Open Account 55% 44% Register Registration (24% of Total) Process 45% drop off 64% (20% of total) of Total Account 36% overall Selection 83% 30% drop off for 56% (46% of Total) (14% of Total) this step Sign in Forget 70% Change 80% Password (32% of Total) Password (26% of Total) 17% drop off 20% drop off (10% of total) (6% of total) Copyright © 2009 YourVersion

Case Study 2: Friendster Optimizing Viral Growth % of users sending = 15% Invites per Invite invites sender = 2.3 click-through rate Active Invite Prospective Click Registration Fail Users Users Process % of users who are Succeed Don’t active Click Conversion rate = 85% Users • Multiplied together, these metrics determine your viral ratio • Which metric has highest ROI opportunity? Copyright © 2009 YourVersion

The Upside Potential of a Metric ? 100% 100% 85% 15% 2.3 0 0 0 Registration % of users sending Avg # of invites Process Yield invitations sent per sender Max possible 0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?% improvement Copyright © 2009 YourVersion

Okay, so how can we improve the metric? How do we increase the average number of invites being sent out per sender? For each idea: What’s the expected benefit? (how much will it improve the metric?) What’s the expected cost? (how many engineer‐ hours will it take?) You want to identify highest benefit/cost idea Copyright © 2009 YourVersion

Adding Metrics and Optimization to your Product Process Site Level Business Product Prioritized Plan Objectives Objectives Feature List Scoping Feature Level Requirements Design & Design Code Test Launch Develop Metrics & User Optimize Feedback Copyright © 2009 YourVersion

Optimization through Iteration: Continuous Improvement Measure the metric Analyze Learning the metric Gaining knowledge: • Market Identify top • Customer opportunities to improve • Domain • Usability Design & develop the enhancement Launch the enhancement Copyright © 2009 YourVersion

Early Stage Product Management Cheat Sheet Truly understand your customer needs Ruthlessly prioritize and launch v1 Talk with users 1‐on‐1 and get feedback Define equation of your business Identify and track key metrics Identify opportunities & prioritize by ROI Launch, learn, and iterate Copyright © 2009 YourVersion