Designing and Optimizing the DNA of a Killer App by Dan Olsen - Startonomics

2. Building the DNA of a Killer App with Metrics and Optimization How to integrate metrics & optimization Using metrics to optimize value creation Business Value Customer Value UI Design Illustrate with real world examples Copyright © 2008 Olsen Solutions LLC

3. Adding Metrics and Optimization to your Product Process Site Level Business Product Prioritized Plan Objectives Objectives Feature List Scoping Feature Level Requirements Design & Design Code Test Launch Develop Metrics & User Optimize Feedback Copyright © 2008 Olsen Solutions LLC

4. Optimization through Iteration: Continuous Improvement Measure the metric Analyze Learning the metric Gaining knowledge: • Market Identify top • Customer opportunities to improve • Domain • Usability Design & develop the enhancement Launch the enhancement Copyright © 2008 Olsen Solutions LLC

7. Define the Equation of your Business “Peeling the Onion” Profit = Revenue ‐ Cost Unique Visitors x Ad Revenue per Visitor Impressions/Visitor x Effective CPM / 1000 Visits/Visitor x Pageviews/Visit x Impressions/PV New Visitors + Returning Visitors Invited Visitors + Uninvited Visitors # of Users Sending Invites x Invites Sent/User x Invite Click‐through Rate Copyright © 2008 Olsen Solutions LLC

8. How to Track Your Metrics Track each metric as daily time series Unique Page Ad New User … Date Visitors views Revenue Sign‐ups 4/24/08 10,100 29,600 25 490 4/27/08 10,500 27,100 24 480 … Create ratios from primary metrics: X / Y Example: How good is your registration page? Okay: # of registered users per day Better: registration conversion rate = # registered users / # uniques to reg page Copyright © 2008 Olsen Solutions LLC

9. Sample Signup Page Yield Data Daily Signup Page Yield vs. Time New Registered Users divided by Unique Visitors to Signup Page 100% 90% 80% Daily Signup Page Yield 70% 60% 50% 40% 30% Started requiring registration 20% Changed Added questions messaging to signup page 10% 0% 1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/1 0 Copyright © 2008 Olsen Solutions LLC

10. Using Metrics to Optimize the Equation of your Business What are the metrics for your business? Where is current value for each metric? How many resources to “move” each metric? Developer‐hours, time, money Which metrics have highest ROI opportunities? Metric A Metric B Metric C Good ROI Bad ROI Great ROI Return Return Return Investment Investment Investment Copyright © 2008 Olsen Solutions LLC

11. Case Study: Optimizing Viral Loop Metrics % of users sending = 15% Invites per Invite invites sender = 2.3 click-through rate Active Invite Prospective Click Registration Fail Users Users Process % of users who are Succeed Don’t active Click Conversion rate = 85% Users • Multiplied together, these metrics determine your viral ratio • Which metric has highest ROI opportunity? Copyright © 2008 Olsen Solutions LLC

12. Maximum Upside Potential of a Metric ? 100% 100% 85% 15% 2.3 0 0 0 Registration % of users sending Avg # of invites conversion rate invitations sent per sender Max possible 0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?% improvement Copyright © 2008 Olsen Solutions LLC

15. The Fuzzy Math of Customer Value You create customer value with a product that Satisfies customers’ needs Is easy to use Has a good price Is better than other alternatives Customer value = customer satisfaction Not fuzzy: # of users and frequency of use Understanding user needs and satisfaction Quantitative: site analytics, usage metrics, surveys Qualitative: support emails, usability sessions (best) Copyright © 2008 Olsen Solutions LLC

16. What is Your Value Proposition? Which user benefits are most important? How well does your product deliver? Importance Current User Upside User Benefit to User Satisfaction Potential Low High – + Benefit 1 High Benefit 2 High Benefit 3 Low ? opportunity Benefit 4 Med Benefit 5 Low ? opportunity Benefit 6 Low Doesn’t matter Copyright © 2008 Olsen Solutions LLC

19. The UI Design Iceberg What most people see and react to Visual Design What good PMs and Designers Interaction think about Design Information Architecture Conceptual Design Recommended reading: Jesse James Garrett’s “Elements of User Experience” chart, free at www.jjg.net Copyright © 2008 Olsen Solutions LLC

20. Approaching UI Design Analytically Typical UI design question: “When using web pages, do users scroll down?” ‐ Yes ‐ No UI questions are never yes/no! (not binary) Should ask: “What percentage of users …?” UI changes impact your metrics Impact can be positive, negative, small, large Seek high‐ROI UI changes Copyright © 2008 Olsen Solutions LLC

21. Put Key Conversion Actions Above The Fold Landing Page A Landing Page B The Fold Key conversion action is above the fold Key conversion action is below the fold Copyright © 2008 Olsen Solutions LLC

23. Case Study: Account Signup Process Redesign 100% 100% Biggest 80% drop % of Users 62.3% 58.8% 60% 50.9% 40% 34.4% 32.7% 20% 0% Sign in / Account Cash vs. 5 Partner 3 Partner Registration Type Margin Pages Pages Copyright © 2008 Olsen Solutions LLC

24. Analysis of Sign In/Registration Flow Open Account 55% 44% Register Registration (24% of Total) Process 45% drop off 64% (20% of total) of Total Account 36% overall Selection 83% 30% drop off for 56% (46% of Total) (14% of Total) this step Sign in Forget 70% Change 80% Password (32% of Total) Password (26% of Total) 17% drop off 20% drop off (10% of total) (6% of total) Copyright © 2008 Olsen Solutions LLC

25. Redesigned User Flow Improved Registration Conversion Rate Abandonment Rate (7 Day Moving Average) Steps 1-2 80% 70% Abandonment Rate (7 Day Moving Average) 60% 50% 40% 30% Released 37% improvement New Design in conversion rate 20% 10% 0% 10/14/02 10/21/02 10/28/02 11/11/02 11/18/02 11/25/02 12/16/02 12/23/02 12/30/02 10/7/02 11/4/02 12/2/02 12/9/02 1/13/03 1/20/03 1/6/03 Copyright © 2008 Olsen Solutions LLC

26. Summary: Building the DNA of a Killer App Define what success means Customer value proposition Equation of your business Instrument your site and track key metrics Identify opportunities and prioritize by ROI Launch, learn, and iterate Copyright © 2008 Olsen Solutions LLC