How to Make Sure Your
Product Rocks
Dan Olsen
CEO, YourVersion
July 21, 2010

                   Copyright © 2010 YourV...
What I’m Covering
What is product management?
Understanding customer needs
Prioritization and maximizing ROI on 
My Background
  BS, Electrical Engineering, Northwestern
  MS, Industrial Engineering, Virginia Tech
  MBA, Stan...
Quick Poll of Audience
Product stage
  Product live to public
  Private beta
  Powerpoint or napkin stage
What is Product Management?

               6    Copyright © 2010 YourVersion
Product Management is
    Critical Link in Value Creation

Market            Product    Development 
• Current       Ma...
A Process View of Product Management
Team Roles & Interactions
                           Prospective                        Existing
Product Management’s Job:
       A Successful Product
Be the expert on the market and the customer
Translate business obje...
Lean Product Management for 
Web 2.0 Products
Dan Olsen, CEO, YourVersion
O’Reilly Web 2.0 Expo SF
May 6, 2010
What’s So Great about “Lean”?
What’s wrong with    Startups are at risk until 
being not‐so‐lean?   they’re profitable
What’s the Formula
       for Product‐Market Fit?
A product that:
  Meets customers’ needs
  Is better than other alternat...
Understanding Customer Needs

               14   Copyright © 2010 YourVersion
Problem Space vs. Solution Space
 Problem Space                  Solution Space
   A customer problem,            A specif...
Problem Space vs. Solution Space
         Product Level
Problem Space       Solution Space
 (user benefit)       (product)...
Problem Space vs. Solution Space
           Feature Level
Problem Space              Solution Space
 (user benefit)       ...
How Do You Prioritize User Benefits 
     and Product Features?
Need a framework for prioritization
  Which user benefits ...
High Importance + Low Satisfaction =
Importance of User Need
Importance vs. Satisfaction
       Ask Users to Rate for Each Feature
              100                                   ...
Kano Model: User Needs & Satisfaction
              User Satisfaction
                                  Delighter (wow)

Olsen’s Hierarchy of Web User Needs
       (adapted from Maslow)
    Customer’s Perspective                               ...
What is Your Value Proposition?
 Which user benefits are you providing?
 How are you better than competitors?
Prioritization and Scope
Customer value is only half the equation
How much engineering effort will it take?
Need to consid...
Prioritizing Product Ideas by ROI

Return (Value Created)

Have to Prioritize Across Multiple 
                  Dimensions At The Same Time

UI Design & Ease of Use

             27    Copyright © 2010 YourVersion
User Benefits vs. Ease of Use
Q: If two products equally deliver the exact 
same user benefits, which product is better?
The Design Gap at Many Startups
Level      Define      Design      Code

 1                              Engineering

 2  ...
The UI Design Iceberg
  What most
  people see
  and react to                       Visual
Elements of User Interface Design
Consists of Three Distinct Elements:
  Information Architecture
     Structure and layou...
Information Architecture
Documents used
     Show how sections of website are organized
     Show major navigati...
Information Architecture

                    Copyright © 2010 YourVersion

            Copyright © 2010 YourVersion
Interaction Design
Documents used
  Combination of Wireframes & Flowcharts
Tools:  Visio, OmniGraffle, Powerpo...
Flowchart showing conditional logic

                          Copyright © 2010 YourVersion
Visual Design
Documents used
  Mockups (aka comps)
Tools used:  Photoshop, Illustrator, 
Designer may also deliv...
Visual Design
   UI Spec

                Copyright © 2010 YourVersion
Visual Design
Specifying Color Palette

                    Copyright © 2010 YourVersion
Learning from Customer Feedback

                 40   Copyright © 2010 YourVersion
Iterating Your Product Vector Based on 
    User Feedback in Solution Space
      Problem Space                   Solution...
What Are You Getting Feedback & 
       Learnings About?
 Problem Space            Solution Space
(your mental model)   (w...
What Can Solution Space Objects
  Can You Test with Customers?
Range of solution space options:
  Wireframe: low‐fidelity ...
Ramen Usability

              Copyright © 2010 YourVersion
The Value of User Feedback Sessions
Critical to talk with customers 1‐on‐1
Gain better understanding of
  Customer needs a...
“Ramen” User Feedback for Startups
Anyone can do it!
  Solution‐space product/mockup to test
  1 customer (wi...
Typical Format for Customer Session
5 ‐ 10 min: Ask questions to understand user 
needs and solutions they currently use
Dos & Don’ts of Conducting Usability
  Explain to the user:
     Their usability test will help improve the product
Usability Case Study: Travel Sites

                         Copyright © 2010 YourVersion
Usability Case Study: Travel Sites
Consider 3 major travel sites
  Expedia, Travelocity, Orbitz
All 3 try to provide same ...
Only One Airport Combo at a Time

                                     • Have to 
Can Select which Airports to Include

                          Copyright © 2010 YourVersion
Can only view results 1 combo at a time

                           • Clicking button brings up 
Binary choices to include nearby airports

                            • Good news: all combos 
All 3 Travel Sites have Identical
   “Airline vs. Number of Stops” Price Grids




Options for Sorting Flight Results

Can Sort By      Expedia Travelocity      Orbitz
Summary Comparison of Travel Sites
User Benefit                  Expedia         Travelocity               Orbitz
Ability ...
Getting Quantitative:
Optimization Using Metrics

              58    Copyright © 2010 YourVersion
Approaching Your Business as an 
     Optimization Exercise

Given reality as it exists today,
optimize our business resu...
Define the Equation of your Business
          “Peeling the Onion”
Advertising Business Model:
Profit = Revenue ‐ Cost

Equation of your Business
          Subscription Business Model
Profit = Revenue ‐ Cost

   Paying Users x  Revenue per Pa...
How to Track Your Metrics
Track each metric as daily time series
           Unique     Page       Ad      New User 
Sample Signup Page Yield Data
                                                Daily Signup Page Yield vs. Time
Identifying the  “Critical Few” Metrics
  What are the metrics for your business?
  Where is current value for each metric...
Google Analytics
•New vs. 
•Time on site
•Top referrers
•Top geos

Measuring Key Conversions:
           Conversion Funnel
•Tie user actions to 
business goals
•Instrument key steps in 
Metrics to Validate Product‐Market Fit
 Survey results
   Importance & Satisfaction
   Net Promoter Score
Approaching UI Design Analytically
 Typical UI design question:
 “When using web pages, do users scroll down?”
   ‐ Yes
Case Studies:
Translating Metrics Into Action

                       Copyright © 2010 YourVersion
Case Study 1: Quicken Brokerage
             Optimizing Sign In/Registration Flow
Mapping the Flow to See Where
            Users Were Dropping Off
Redesigned User Flow Improved 
Registration Conversion Rate 37%

             Released    37% improvement
Case Study 2: Friendster
                      Optimizing Viral Growth
   % of users
    sending = 15%       Invites per  ...
The Upside Potential of a Metric

Okay, so how can we improve the metric?
 How do we increase the average number of 
 invites being sent out per sender?
Case Study 2: Friendster
Doubled Number of Invitations Sent per Sender

                                Copyright © 201...

      77   Copyright © 2010 YourVersion
Adding Metrics and Optimization to 
             your Product Process
Optimization through Iteration:
  Continuous Improvement
                  the metric

How to Make Sure Your Product Rocks
           Cheat Sheet
Clarify problem space by iterating in the 
solution space & get...
Great way to stay on top of your interests
Real‐time discovery engine
  Discovers new, relevant content tailored to your 
How to Make Sure Your
Product Rocks

How to Make Sure Your Product Rocks
Upcoming SlideShare
Loading in...5

How to Make Sure Your Product Rocks


Published on

Talk to SSE Labs on best practices in product management, UI design, and metrics.

Published in: Business
  • what do you mean by inbound and outbound PM in slide 8?
    Are you sure you want to  Yes  No
    Your message goes here
  • Hi. If you like my presentations, I encourage you to check out my startup YourVersion where I've been applying the product management principles that I talk about in my SlideShares.
    YourVersion is 'Pandora for your real-time web content': a personalized discovery engine that brings you the latest relevant news, blogs, tweets, and videos tailored to your specific interests.
    We launched to the public at the 2009 TechCrunch50 Conference where we won the People’s Choice Award.

    In addition to the website, we also offer a free iPad app, a free iPhone app, and a free Android app so you can discover on the go. We also offer free browser extensions for Firefox, Chrome, and Safari browsers. Learn more at

    I'd love to hear any feedback or suggestions you have on our products.


    Dan Olsen
    CEO & Founder, YourVersion
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

How to Make Sure Your Product Rocks

  1. 1. How to Make Sure Your Product Rocks Dan Olsen CEO, YourVersion July 21, 2010 Copyright © 2010 YourVersion
  2. 2. What I’m Covering What is product management? Understanding customer needs Prioritization and maximizing ROI on  engineering resources UI Design & Ease of Use Using metrics to optimize your product Will post slides to Copyright © 2010 YourVersion
  3. 3. My Background Education BS, Electrical Engineering, Northwestern MS, Industrial Engineering, Virginia Tech MBA, Stanford Web development and UI design 19 years of Product Management Experience Managed submarine design for 5 years 5 years at Intuit, led Quicken Product Management Led Product Management at Friendster PM consultant to startups:, YouSendIt, Epocrates CEO & Cofounder of YourVersion, startup building  “Pandora for your real‐time web content” Copyright © 2010 YourVersion
  4. 4. Quick Poll of Audience Product stage Product live to public Private beta Alpha Powerpoint or napkin stage Consumer vs. B2B Engineering, Marketing, PM, Designer Copyright © 2010 YourVersion
  5. 5. What is Product Management? 6 Copyright © 2010 YourVersion
  6. 6. Product Management is Critical Link in Value Creation Market Product  Development  • Current  Management Team customers • Prospective customers • Competitors Copyright © 2010 YourVersion
  7. 7. A Process View of Product Management “Inbound” Product “Outbound” Management Product Long Business Product Management Term Strategy Strategy Market/ Sell Short Business Product Product Term Objectives Objectives Development Service/ Support Copyright © 2010 YourVersion
  8. 8. Team Roles & Interactions Prospective Existing Customers Customers External Marketing/ Sales Listening to customers Product  UI Interface Management Design Support Engineering QA Internal Copyright © 2010 YourVersion
  9. 9. Product Management’s Job: A Successful Product Be the expert on the market and the customer Translate business objectives and customer  needs into product requirements Be the clearinghouse for all product ideas  Work with team to design & build great product Define and track key metrics Identify, plan & prioritize product ideas to  maximize ROI on engineering resources Copyright © 2010 YourVersion
  10. 10. Lean Product Management for  Web 2.0 Products Dan Olsen, CEO, YourVersion O’Reilly Web 2.0 Expo SF May 6, 2010 Copyright © 2010 YourVersion
  11. 11. What’s So Great about “Lean”? What’s wrong with  Startups are at risk until  being not‐so‐lean? they’re profitable Funding cocoon only lasts  so long Limited resources Tech markets move fast Time is the real enemy “Time is the scarcest  resource and unless it is  managed nothing else can  be managed.” ‐ Peter Drucker Copyright © 2010 YourVersion
  12. 12. What’s the Formula for Product‐Market Fit? A product that: Meets customers’ needs Is better than other alternatives Is easy to use Has a good value/price Simple, right? It’s easy to understand conceptually  what we want to achieve HOW to achieve it is the hard part Copyright © 2010 YourVersion
  13. 13. Understanding Customer Needs 14 Copyright © 2010 YourVersion
  14. 14. Problem Space vs. Solution Space Problem Space Solution Space A customer problem,  A specific  need, or benefit that the  implementation to  product should address address the need or  A product requirement product requirement Example: Ability to write in space  NASA: space pen (zero gravity) ($1 M R&D cost) Russians: pencil Copyright © 2010 YourVersion
  15. 15. Problem Space vs. Solution Space Product Level Problem Space Solution Space (user benefit) (product) Pen and Prepare paper my taxes TurboTax File my taxes TaxCut Copyright © 2010 YourVersion
  16. 16. Problem Space vs. Solution Space Feature Level Problem Space Solution Space (user benefit) (feature) Gmail Make it easy importer to share a link with my friends Design Design Design #1 #2 #3 Preview with  User can edit  Allow me to Design checkboxes before import reuse my email #1 No No contacts #2 Yes No #3 Yes Yes Copyright © 2010 YourVersion
  17. 17. How Do You Prioritize User Benefits  and Product Features? Need a framework for prioritization Which user benefits should you address? Which product features to build (or improve)? Importance vs. Satisfaction Importance of user need (problem space) Satisfaction with how well a product meets the  user’s need (solution space) Opportunity = High Importance need with low Satisfaction Copyright © 2010 YourVersion
  18. 18. High Importance + Low Satisfaction = Importance of User Need Opportunity High Competitive Opportunity Market Not Worth Going After Low Low High User Satisfaction with Current Alternatives Copyright © 2010 YourVersion
  19. 19. Importance vs. Satisfaction Ask Users to Rate for Each Feature 100 98 Great 95 84 87 90 Bad 86 85 79 84 55 70 80 Importance 80 75 72 80 70 75 65 60 55 41 50 40 50 60 70 80 90 100 Satisfaction Recommended reading: “What  Customers Want” by Anthony Ulwick Copyright © 2010 YourVersion
  20. 20. Kano Model: User Needs & Satisfaction User Satisfaction Delighter (wow) Performance  (more is better) Need Need not met fully met Must Have Needs & features  migrate over time User Dissatisfaction Copyright © 2010 YourVersion
  21. 21. Olsen’s Hierarchy of Web User Needs (adapted from Maslow) Customer’s Perspective What does it mean to us? How easy to use is it? Usability & Design Satisfaction Increasing Does the functionality Feature Set meet my needs? Does the functionality work? Absence of Bugs Dissatisfaction Decreasing Is the site fast enough? Page Load Time Is the site up when I want to use it? Uptime Copyright © 2010 YourVersion
  22. 22. What is Your Value Proposition? Which user benefits are you providing? How are you better than competitors? Competitor A Competitor B You Must Have Benefit 1 Y Y Y Performance Benefit 1 High Low Med Performance Benefit 2 Low High Low Performance Benefit 3 Med Med High Delighter Benefit 1 Y ‐ ‐ Delighter Benefit 2 ‐ ‐ Y Copyright © 2010 YourVersion
  23. 23. Prioritization and Scope Customer value is only half the equation How much engineering effort will it take? Need to consider value and effort (ROI) Ruthlessly prioritize: rank order Be deliberate about scope & keep it small It’s easy to try to do too much Strategy = deciding what you’re NOT doing Break features down into smaller chunks Smaller scope → faster iterations → better Copyright © 2010 YourVersion
  24. 24. Prioritizing Product Ideas by ROI ? Return (Value Created) 4 Idea D 3 Idea A Idea B 2 Idea C 1 Idea F 1 2 3 4 Investment (developer‐weeks) Copyright © 2010 YourVersion
  25. 25. Have to Prioritize Across Multiple  Dimensions At The Same Time Ease of Use Customer Value Quality Functionality Customer  Understanding Time Copyright © 2010 YourVersion
  26. 26. UI Design & Ease of Use 27 Copyright © 2010 YourVersion
  27. 27. User Benefits vs. Ease of Use Q: If two products equally deliver the exact  same user benefits, which product is better? A: The product that’s easier to use “Ease of use” provides benefits Saves time Reduces cognitive load Reduces frustration UI Design can be differentiator Olsen’s Law: “The less user effort required, the  higher the percentage of users who will do it” Copyright © 2010 YourVersion
  28. 28. The Design Gap at Many Startups Level Define Design Code 1 Engineering 2 Product Mgmt Engineering 3 Product Mgmt Engineering Product Mgmt Engineering 4 PM Eng UI 5 PM Eng Copyright © 2010 YourVersion
  29. 29. The UI Design Iceberg What most people see and react to Visual Design What good product people Interaction think about Design Information Architecture Conceptual Design Recommended reading: Jesse James Garrett’s “Elements of User Experience” chart, free at Copyright © 2010 YourVersion
  30. 30. Elements of User Interface Design Consists of Three Distinct Elements: Information Architecture Structure and layout at both site and page level How site is structured (sitemap) How site information is organized (site layout) How each page is organized (page layout) Interaction Design How user and product interact with one another User flows (e.g., navigation across multiple pages) User input (e.g., controls and form design) Visual Design “How it looks” vs. “What it is”, often called “chrome” Fonts, colors, graphical elements Copyright © 2010 YourVersion
  31. 31. Information Architecture Documents used Sitemap Show how sections of website are organized Show major navigation patterns Wireframes Show the layout of components on a page Does NOT focus on visual design Black & White No graphics Templates for overall website and individual pages Tools:  Visio, OmniGraffle, Axure, Powerpoint, Word,  Excel, Photoshop, Balsamiq, WriteMaps, whiteboard Copyright © 2010 YourVersion
  32. 32. Information Architecture Sitemap Copyright © 2010 YourVersion
  33. 33. Wireframe Copyright © 2010 YourVersion
  34. 34. Interaction Design Documents used Flowchart Combination of Wireframes & Flowcharts Tools:  Visio, OmniGraffle, Powerpoint,  Photoshop, whiteboard May build prototype using HTML, jQuery,  Ruby on Rails, Flash, or paper Usability testing can help find problems Copyright © 2010 YourVersion
  35. 35. Flowchart showing conditional logic Copyright © 2010 YourVersion
  36. 36. Visual Design Documents used Mockups (aka comps) Tools used:  Photoshop, Illustrator,  Fireworks Designer may also deliver HTML & CSS  version (with no back‐end code) Copyright © 2010 YourVersion
  37. 37. Visual Design UI Spec Copyright © 2010 YourVersion
  38. 38. Visual Design Specifying Color Palette Copyright © 2010 YourVersion
  39. 39. Learning from Customer Feedback 40 Copyright © 2010 YourVersion
  40. 40. Iterating Your Product Vector Based on  User Feedback in Solution Space Problem Space Solution Space (your mental model) (what users can react to) Help user Help user book travel plan travel Mockups / Code Customer Feedback Copyright © 2010 YourVersion
  41. 41. What Are You Getting Feedback &  Learnings About? Problem Space Solution Space (your mental model) (what users can react to) Feature Set Customer Understanding  (needs &  preferences) UI Design Messaging  Copyright © 2010 YourVersion
  42. 42. What Can Solution Space Objects Can You Test with Customers? Range of solution space options: Wireframe: low‐fidelity graphics Mockup: high‐fidelity graphics Prototype: interactive graphics or code Alpha product: production code Graphics usually quicker/cheaper to  change than code Goal is to gain learnings quickly Copyright © 2010 YourVersion
  43. 43. Ramen Usability Copyright © 2010 YourVersion
  44. 44. The Value of User Feedback Sessions Critical to talk with customers 1‐on‐1 Gain better understanding of Customer needs and problems Issues with your feature set, UI, messaging What alternatives customers are using, pros & cons of each, customer preferences QA: use cases & bugs you haven’t seen Really a “user learning” session Make test as real for user as possible Copyright © 2010 YourVersion
  45. 45. “Ramen” User Feedback for Startups Anyone can do it! Ingredients: Solution‐space product/mockup to test 1 customer (with laptop if testing code) 1 desk 1 person to conduct the session Pen and paper Optional note‐taker and observers Copyright © 2010 YourVersion
  46. 46. Typical Format for Customer Session 5 ‐ 10 min: Ask questions to understand user  needs and solutions they currently use 30 ‐ 50 min: User feedback Show user product/mockup Non‐directed as much as possible When necessary, direct user to attempt to  perform a specific task 5 ‐ 10 min: Wrap‐up Answer any user questions that came up Point out/explain features you want to highlight Ask them if they would use the product Copyright © 2010 YourVersion
  47. 47. Dos & Don’ts of Conducting Usability Do Explain to the user: Their usability test will help improve the product Not to worry about hurting your feelings “Think Aloud Protocol” Ask user to attempt the task, then be a fly on the wall Ask non‐leading, open‐ended questions Take notes and review them afterwards for take‐aways Don’t Ask leading questions “Help” the user or explain the UI (e.g., “click over here”) Respond to user frustration or questions (until test is over) Get defensive Blame the user Copyright © 2010 YourVersion
  48. 48. Usability Case Study: Travel Sites Copyright © 2010 YourVersion
  49. 49. Usability Case Study: Travel Sites Consider 3 major travel sites Expedia, Travelocity, Orbitz All 3 try to provide same user benefits Easily find flights that match your schedule Easily find the lowest price flights UI Design differences make some better Case study: round trip from San Fran to NYC Want to find best combo of price and airports 3 possible airports for each city 9 possible one‐way airport combinations 81 possible round‐trip airport combinations Copyright © 2010 YourVersion
  50. 50. Expedia: Only One Airport Combo at a Time • Have to  manually  check all 9  combos •3 clicks to  change airport •Then wait for  new results •24 clicks + 8 page reloads  to see all 9  combos Copyright © 2010 YourVersion
  51. 51. Travelocity: Can Select which Airports to Include Copyright © 2010 YourVersion
  52. 52. Travelocity: Can only view results 1 combo at a time • Clicking button brings up  results for this combo •Problem: browser Back  button loses other  airports! •Have to go through  ‘Change Search’ process = 9 clicks + 4 page reloads  for each combo •72 clicks + 32 page loads  to see other 8 combos Copyright © 2010 YourVersion
  53. 53. Orbitz: Binary choices to include nearby airports • Good news: all combos  appear in results •80‐mile radius might  include airports I don’t  want (3 for SFO, 5 for NYC)  but turned out not to be  an issue Copyright © 2010 YourVersion
  54. 54. All 3 Travel Sites have Identical “Airline vs. Number of Stops” Price Grids Expedia Orbitz Travelocity Copyright © 2010 YourVersion
  55. 55. Options for Sorting Flight Results Website Can Sort By Expedia Travelocity Orbitz Airline Y Departure Time Y Y Y Arrival Time Y Y Travel Time Y Y Y Price Y Y Y Copyright © 2010 YourVersion
  56. 56. Summary Comparison of Travel Sites User Benefit Expedia Travelocity Orbitz Ability to include other  Yes Yes Yes nearby airports Ability to pick specific  High High Low nearby airports (by changing) (can pre‐select) Ease of seeing results  Med Low High for multiple airports Ease of trading off  Low High Med airport combos vs. price Airline vs. Number of  Yes Yes Yes Connections Price Grid Flight Results Sorting  Med High Low Options Overall ability to easily  Med Low High find best airport combo Copyright © 2010 YourVersion
  57. 57. Getting Quantitative: Optimization Using Metrics 58 Copyright © 2010 YourVersion
  58. 58. Approaching Your Business as an  Optimization Exercise Given reality as it exists today, optimize our business results subject to our resource constraints. Copyright © 2010 YourVersion
  59. 59. Define the Equation of your Business “Peeling the Onion” Advertising Business Model: Profit = Revenue ‐ Cost Unique Visitors x  Ad Revenue per Visitor Impressions/Visitor x  Effective CPM / 1000 Visits/Visitor  x  Pageviews/Visit  x  Impressions/PV New Visitors + Returning Visitors Invited Visitors + Uninvited Visitors # of Users Sending Invites  x  Invites Sent/User  x  Invite Conversion Rate Copyright © 2010 YourVersion
  60. 60. Equation of your Business Subscription Business Model Profit = Revenue ‐ Cost Paying Users x  Revenue per Paying User New Paying Users +  Repeat Paying Users Trial Users x  Conv Rate Previous Paying Users  x  ( 1 – Cancellation Rate ) ( SEO Visitors + SEM Visitors + Viral Visitors )  x  Trial Conversion Rate Copyright © 2010 YourVersion
  61. 61. How to Track Your Metrics Track each metric as daily time series Unique  Page  Ad  New User  … Date Visitors views Revenue Sign‐ups 4/24/08 10,100 29,600 25 490 4/25/08 10,500 27,100 24 480 … Create ratios from primary metrics:  X / Y Example: How good is your registration page? Okay: # of registered users per day Better: registration conversion rate = # registered users / # uniques to reg page Copyright © 2010 YourVersion
  62. 62. Sample Signup Page Yield Data Daily Signup Page Yield vs. Time New Registered Users divided by Unique Visitors to Signup Page 100% 90% 80% Daily Signup Page Yield 70% 60% 50% 40% 30% Started requiring registration 20% Changed Added questions messaging to signup page 10% 0% 1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/1 0 Copyright © 2010 YourVersion
  63. 63. Identifying the  “Critical Few” Metrics What are the metrics for your business? Where is current value for each metric?  How many resources to “move” each metric? Developer‐hours, time, money Which metrics have highest ROI opportunities? Metric A Metric B Metric C Good ROI Bad ROI Great ROI Return Return Return Investment Investment Investment Copyright © 2010 YourVersion
  64. 64. Google Analytics •Unique  visitors •New vs.  returning •Pageviews •Time on site •Top referrers •Top geos Copyright © 2010 YourVersion
  65. 65. Measuring Key Conversions: Conversion Funnel •Tie user actions to  business goals •Instrument key steps in  user flow •See where users are  dropping off •Quantify improvement  from changes Copyright © 2010 YourVersion
  66. 66. Metrics to Validate Product‐Market Fit Survey results Importance & Satisfaction Net Promoter Score “How would you feel if you could no longer use Product X?” Very disappointed, Somewhat disappointed, Not disappointed User behavior Prospects sign up (high conversion rate) They keep using it (high retention rate) They use it often (high frequency of use) Copyright © 2010 YourVersion
  67. 67. Approaching UI Design Analytically Typical UI design question: “When using web pages, do users scroll down?” ‐ Yes ‐ No UI questions are never yes/no! (not binary) Should ask: “What percentage of users …?” UI changes impact your metrics Impact can be positive, negative, small, large Seek high‐ROI UI changes Copyright © 2010 YourVersion
  68. 68. Case Studies: Translating Metrics Into Action Copyright © 2010 YourVersion
  69. 69. Case Study 1: Quicken Brokerage Optimizing Sign In/Registration Flow 100% 100% Biggest  80% drop % of Users 62.3% 58.8% 60% 50.9% 40% 34.4% 32.7% 20% 0% Sign in / Account  Cash vs. 5 Partner 3 Partner Registration Type Margin Pages Pages Copyright © 2010 YourVersion
  70. 70. Mapping the Flow to See Where Users Were Dropping Off Open Account 55% 44% Register Registration  (24% of Total) Process 45% drop off 64% (20% of total) of Total Account  36% overall  Selection 83% 30% drop off for  56% (46% of Total) (14% of Total) this step Sign in Forget  70% Change  80% Password (32% of Total) Password (26% of Total) 17% drop off  20% drop off (10% of total) (6% of total) Copyright © 2010 YourVersion
  71. 71. Redesigned User Flow Improved  Registration Conversion Rate 37% Released 37% improvement New Design in conversion rate Copyright © 2010 YourVersion
  72. 72. Case Study 2: Friendster Optimizing Viral Growth % of users sending = 15% Invites per Invite invites sender = 2.3 click-through rate Active Invite Prospective Click Registration Fail Users Users Process % of users who are Succeed Don’t active Click Conversion rate = 85% Users • Multiplied together, these metrics determine your viral ratio • Which metric has highest ROI opportunity? Copyright © 2010 YourVersion
  73. 73. The Upside Potential of a Metric ? 100% 100% 85% 15% 2.3 0 0 0 Registration  % of users sending  Avg # of invites  Process Yield invitations sent per sender Max possible  0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?% improvement Copyright © 2010 YourVersion
  74. 74. Okay, so how can we improve the metric? How do we increase the average number of  invites being sent out per sender? For each idea: What’s the expected benefit? (how much will it  improve the metric?) What’s the expected cost? (how many engineer‐ hours will it take?) You want to identify highest benefit/cost idea Copyright © 2010 YourVersion
  75. 75. Case Study 2: Friendster Doubled Number of Invitations Sent per Sender Copyright © 2010 YourVersion
  76. 76. Continuous Improvement 77 Copyright © 2010 YourVersion
  77. 77. Adding Metrics and Optimization to  your Product Process Site Level Business Product Prioritized  Plan Objectives Objectives Feature List Scoping Feature  Level Requirements  Design & Design Code Test Launch Develop Metrics & User  Optimize Feedback Copyright © 2010 YourVersion
  78. 78. Optimization through Iteration: Continuous Improvement Measure the metric Analyze Learning the metric Gaining knowledge: • Market Identify top  • Customer opportunities to improve • Domain • Usability Design & develop   the enhancement Launch the enhancement Copyright © 2010 YourVersion
  79. 79. How to Make Sure Your Product Rocks Cheat Sheet Clarify problem space by iterating in the  solution space & getting user feedback Revise feature set, UI design, and  messaging to improve product‐market fit Ruthlessly prioritize based on ROI Define equation of your business Identify and track key metrics Launch, learn, and iterate Copyright © 2010 YourVersion
  80. 80. Great way to stay on top of your interests Real‐time discovery engine Discovers new, relevant content tailored to your  specific interests News, Blogs, Tweets, Webpages, Videos Bookmark and share via email, Twitter, Facebook Weekly personalized email digest Free iPhone app Extensions for Firefox, Chrome, Safari & bookmarklet Launched at TechCrunch50, won People’s Choice Check it out at Copyright © 2010 YourVersion
  81. 81. How to Make Sure Your Product Rocks Questions? @danolsen Copyright © 2010 YourVersion
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.