Designing & Optimizing the DNA of a Killer App (Dan Olsen, Startonomics SF 2008)

Designing & Optimizing the DNA of a Killer App Dan Olsen, CEO & Founder, YourVersion Startonomics · San Francisco · Oct 2, 2008

Building the DNA of a Killer App with Metrics and Optimization

How to integrate metrics & optimization

Using metrics to optimize value creation

Business Value

Customer Value

UI Design

Illustrate with real world examples

Adding Metrics and Optimization to your Product Process Copyright © 2008 Olsen Solutions LLC Plan Design Develop Business Objectives Product Objectives Prioritized Feature List Scoping Requirements & Design Code Test Launch Site Level Feature Level Optimize Metrics & User Feedback

Optimization through Iteration: Continuous Improvement Copyright © 2008 Olsen Solutions LLC Measure the metric Analyze the metric Identify top opportunities to improve Design & develop the enhancement Launch the enhancement

Learning

Gaining knowledge:

Market

Customer

Domain

Usability

Approaching Business as an Optimization Exercise

Given reality as it exists today,

optimize our business results

subject to our resource constraints.

Profit = Revenue - Cost

Define the Equation of your Business “Peeling the Onion” Unique Visitors x Ad Revenue per Visitor Impressions/Visitor x Effective CPM / 1000 Visits/Visitor x Pageviews/Visit x Impressions/PV New Visitors + Returning Visitors Invited Visitors + Uninvited Visitors # of Users Sending Invites x Invites Sent/User x Invite Click-through Rate Copyright © 2008 Olsen Solutions LLC

How to Track Your Metrics

Track each metric as daily time series

Create ratios from primary metrics: X / Y

Example: How good is your registration page?

Okay: # of registered users per day

Better: registration conversion rate =

# registered users / # uniques to reg page

Using Metrics to Optimize the Equation of your Business

What are the metrics for your business?

Where is current value for each metric?

How many resources to “move” each metric?

Developer-hours, time, money

Which metrics have highest ROI opportunities?

Case Study: Optimizing Viral Loop Metrics

Which metric has highest ROI opportunity?

Multiplied together, these metrics determine your viral ratio

Users % of users who are active = 15% = 2.3 = 85% Registration Process

Maximum Upside Potential of a Metric Copyright © 2008 Olsen Solutions LLC 0 100% 0 100% 0 ? Registration conversion rate % of users sending invitations Avg # of invites sent per sender 2.3 85% 15% Max possible improvement 0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?%

The Fuzzy Math of Customer Value

You create customer value with a product that

Satisfies customers’ needs

Is easy to use

Has a good price

Is better than other alternatives

Customer value = customer satisfaction

Not fuzzy: # of users and frequency of use

Understanding user needs and satisfaction

Quantitative: site analytics, usage metrics, surveys

Qualitative: support emails, usability sessions (best)

What is Your Value Proposition?

Which user benefits are most important?

How well does your product deliver?

Copyright © 2008 Olsen Solutions LLC Benefit 1 Importance to User Current User Satisfaction Benefit 2 User Benefit Upside Potential Benefit 4 Benefit 5 Benefit 6 High High Med Low Low Low High – + Doesn’t matter Benefit 3 Low opportunity opportunity ? ?

The UI Design Iceberg Copyright © 2008 Olsen Solutions LLC Visual Design Interaction Design Information Architecture Conceptual Design Recommended reading: Jesse James Garrett’s “Elements of User Experience” chart, free at www.jjg.net What most people see and react to What good PMs and Designers think about

Approaching UI Design Analytically

Typical UI design question:

“ When using web pages, do users scroll down?”

- Yes

- No

UI questions are never yes/no! (not binary)

Should ask: “What percentage of users …?”

UI changes impact your metrics

Impact can be positive, negative, small, large

Seek high-ROI UI changes

Analysis of Sign In/Registration Flow Copyright © 2008 Olsen Solutions LLC Open Account Sign in Account Selection Register 56% 44% Forget Password Registration Process 45% drop off (20% of total) 36% overall drop off for this step 70% (32% of Total) 17% drop off (10% of total) 20% drop off (6% of total) 30% (14% of Total) 80% (26% of Total) 55% (24% of Total) 64% of Total Change Password 83% (46% of Total)

Summary: Building the DNA of a Killer App

Define what success means

Customer value proposition

Equation of your business

Instrument your site and track key metrics

Identify opportunities and prioritize by ROI

Launch, learn, and iterate

[email_address] Questions? www.yourversion.com

Taking Optimization to The Next Level

Multivariate testing, aka A/B testing

New challenges emerge

Are we at local vs. global optimum?

How do we identify top new ideas to test? (creativity and innovation)

Are there assumptions we should challenge?

Are we missing the forest for the trees?

Olsen’s Hierarchy of Web User Needs (adapted from Maslow) Copyright © 2008 Olsen Solutions LLC Is the site up when I want to use it? Is the site fast enough? Does the functionality work? How compelling and easy to use is the functionality? Customer’s Perspective What does it mean to us? Uptime Page Load Time Absence of Bugs Feature Set Usability & Design Decreasing Dissatisfaction Increasing Satisfaction

Kano Model: User Needs & Satisfaction Copyright © 2008 Olsen Solutions LLC User Satisfaction User Dissatisfaction Performance (more is better) Delighter (wow) Need not met Need fully met Must Have Needs & features migrate over time