Early Stage Web Product Management
Dan Olsen
CEO, YourVersion
July 24, 2009
Copyright © 2009 YourVersion

What You Wanted to Hear About
1 2 3 4 5 6 7 8 9 10
Usability
Understanding user
needs
Prioritizing features
Translating learnings
from metrics into
action
Maximizing ROI on
development
resources
Setting up and
tracking the right
metrics
UI design
Copyright © 2009 YourVersion

Agenda
Understanding customer needs
Prioritizing, scoping, and maximizing ROI
on engineering resources
UI design
Usability testing
Using metrics for optimization
Will post slides to slideshare.net/dan_o
Copyright © 2009 YourVersion

My Background
Education
BS, Electrical Engineering, Northwestern
MS, Industrial Engineering, Virginia Tech
MBA, Stanford
PHP, MySQL, JavaScript, XHTML, CSS, UI design
18 years of Product Management Experience
Managed submarine design for 5 years
5 years at Intuit, led Quicken Product Management
Led Product Management at Friendster
Olsen Solutions LLC, PM consultant for startups
CEO & Founder of YourVersion, real‐time discovery startup
Copyright © 2009 YourVersion

For an Early Stage Startup,
Which is More Important?
or
Qualitative Quantitative
Information? Information?
“When you’re Small,
start with Qual!”
Copyright © 2009 YourVersion

How is developing Early Stage Products
Different from Later Stage Products?
With an early stage product, you have
WAY MORE uncertainty about:
Who your target customers really are
Which customer needs you should address
How to best meet those needs
What design works best
These are qualitative learnings/decisions
Quantitative is also valuable (later)
Copyright © 2009 YourVersion

What is Product Management?
Copyright © 2009 YourVersion

Product Management is
Critical Link in Value Creation
Market Product Development
• Current Management Team
customers
• Prospective
customers
• Competitors
Copyright © 2009 YourVersion

The Product Manager’s Job:
A Successful Product
Be the expert on the market and the customer
Translate business objectives and customer
needs into product requirements
Be the clearinghouse for all product ideas
Work with team to design & build great product
Define and track key metrics
Identify, plan & prioritize product ideas to
maximize ROI on engineering resources
Copyright © 2009 YourVersion

What’s the Formula
for a Winning Product?
A product that:
Meets customers’ needs
Is better than other alternatives
Is easy to use
Has a good value/price
Copyright © 2009 YourVersion

Problem Space vs. Solution Space
Problem Space Solution Space
A customer problem, A specific
need, or benefit that the implementation to
product should address address the need or
A product requirement product requirement
Example:
Ability to write in space NASA: space pen
(zero gravity) ($1 M R&D cost)
Russians: pencil
Copyright © 2009 YourVersion

Problem Space vs. Solution Space
Product Level
Problem Space Solution Space
(user benefit) (product)
Pen and
Prepare paper
my taxes
TurboTax
File my
taxes
TaxCut
Copyright © 2009 YourVersion

Problem Space vs. Solution Space
Feature Level
Problem Space Solution Space
(user benefit) (feature)
Gmail
Make it easy importer
to share a
link with my
friends
Design Design Design
#1 #2 #3
Preview with User can edit
Allow me to Design checkboxes before import
reuse my
email #1 No No
contacts #2 Yes No
#3 Yes Yes
Copyright © 2009 YourVersion

Types of Customer Benefits
Functional Benefits
Allow the user to do something they couldn’t do before
Deliver a benefit more quickly, conveniently, or cheaply
Booking travel more easily (travel sites)
Talking for less money (VOIP software)
Emotional Benefits
Control (Quicken)
Feeling informed (news sites)
Confidence / security (anti‐virus software)
Enjoyment / entertainment (YouTube)
Self‐expression (MySpace, Facebook)
Copyright © 2009 YourVersion

The Customer Benefits “Ladder”
…which means one less thing to
Higher‐level
worry about in my hectic life
benefit
(more abstract)
…which makes me feel more in
control of my finances
…which gives me a clear picture
of how much money I have
Lower‐level Quicken makes it easy for me
benefit to balance my checkbook
(more specific)
Copyright © 2009 YourVersion

How Do You Prioritize User Benefits
and Product Features?
Need a framework for prioritization
Which user benefits should you address?
Which product features to build (or improve)?
Importance vs. Satisfaction
Importance of user need (problem space)
Satisfaction with how well a product meets the
user’s need (solution space)
Opportunity =
High Importance need with low Satisfaction
Copyright © 2009 YourVersion

What is Your Value Proposition?
Which user benefits are you going to provide?
How will your product be better than rest?
Importance Current User Upside
User Benefit to User Satisfaction Potential
Low High – +
Benefit 1 High
Benefit 2 High
Benefit 3 Low ?
opportunity
Benefit 4 Med
Benefit 5 Low ?
opportunity
Benefit 6 Low
Doesn’t matter
Copyright © 2009 YourVersion

Importance vs. Satisfaction
Ask Users to Rate for Each Feature
100 98
Great
95
84 87
90
Bad 86
85 79 84
55 70
80
Importance
80
75 72
80
70
75
65
60
55
41
50
40 50 60 70 80 90 100
Satisfaction
Recommended reading: “What
Customers Want” by Anthony Ulwick Copyright © 2009 YourVersion

Kano Model: User Needs & Satisfaction
User Satisfaction
Delighter (wow)
Performance
(more is better)
Need Need
not met fully met
Must Have
Needs & features
migrate over time
User Dissatisfaction Copyright © 2009 YourVersion

Olsen’s Hierarchy of Web User Needs
(adapted from Maslow)
Customer’s Perspective What does it mean to us?
How easy to use is it? Usability & Design
Satisfaction
Increasing
Does the functionality
Feature Set
meet my needs?
Does the functionality work?
Absence of Bugs
Dissatisfaction
Decreasing
Is the site fast enough?
Page Load Time
Is the site up when I want to use it?
Uptime
Copyright © 2009 YourVersion

Prioritization and Scope
Customer value is only half the equation
How much engineering effort will it take?
Need to consider value and effort (ROI)
Ruthlessly prioritize: rank order
Be deliberate about scope & keep it small
It’s easy to try to do too much
Strategy = deciding what you’re NOT doing
Break features down into smaller chunks
Smaller scope → faster iterations → better
Copyright © 2009 YourVersion

Have to Prioritize Across Multiple
Dimensions At The Same Time
Ease of Use
Customer Value
Quality
Functionality
Customer
Understanding
Time
Copyright © 2009 YourVersion

Prioritizing Product Ideas by ROI
?
Return (Value Created)
4
Idea D
3
Idea A Idea B
2
Idea C
1
Idea F
1 2 3 4
Investment (developer‐weeks)
Copyright © 2009 YourVersion

User Benefits vs. Ease of Use
Q: If two products equally deliver the exact
same user benefits, which product is better?
A: The product that’s easier to use
‘Ease of use’ provides benefits
Saves time
Reduces cognitive load
Reduces frustration
Not many companies excel at UI design
‘Ease of use’ can be differentiator
Copyright © 2009 YourVersion

The Design Gap at Most Start‐ups
Level Define Design Code
1 Product Mgmt Engineering
2 Product Mgmt Engineering
Product Mgmt Engineering
3 PM Eng
UI
4 PM Eng
Copyright © 2009 YourVersion

The UI Design Iceberg
What most
people see
and react to Visual
Design What good
PMs and
Designers
Interaction think about
Design
Information
Architecture
Conceptual
Design
Recommended reading: Jesse James Garrett’s
“Elements of User Experience” chart, free at www.jjg.net Copyright © 2009 YourVersion

Elements of User Interface Design
Consists of Three Distinct Elements:
Information Architecture
Structure and layout at both site and page level
How site is structured (sitemap)
How site information is organized (site layout)
How each page is organized (page layout)
Interaction Design
How user and product interact with one another
User flows (e.g., navigation across multiple pages)
User input (e.g., controls and form design)
Visual Design
“How it looks” vs. “What it is”, often called “chrome”
Fonts, colors, graphical elements
Copyright © 2009 YourVersion

Information Architecture
Documents used
Sitemap
Show how sections of website are organized
Show major navigation patterns
Wireframes
Show the layout of components on a page
Does NOT focus on visual design
Black & White
No graphics
Templates for overall website and individual pages
Tools: Visio, OmniGraffle, Axure, Powerpoint, Word,
Excel, Photoshop, Balsamiq, WriteMaps, whiteboard
Copyright © 2009 YourVersion

Information Architecture
Sitemap
Copyright © 2009 YourVersion

Information Architecture
Sitemap
Copyright © 2009 YourVersion

Wireframe
Copyright © 2009 YourVersion

Napkin Wireframe
Copyright © 2009 YourVersion

Interaction Design
Documents used
Flowchart
Combination of Wireframes & Flowcharts
Tools: Visio, OmniGraffle, Powerpoint,
Photoshop, whiteboard
May build prototype using HTML, jQuery,
Ruby on Rails, Flash, or paper
Usability testing can help find problems
Copyright © 2009 YourVersion

Flowchart showing conditional logic
Copyright © 2009 YourVersion

Napkin Flowchart
Copyright © 2009 YourVersion

Combo Wireframe & Flowchart
Copyright © 2009 YourVersion

Visual Design
Documents used
Mockups (aka comps)
Tools used: Photoshop, Illustrator,
Fireworks
Designer may also deliver HTML & CSS
version (with no back‐end code)
Copyright © 2009 YourVersion

Visual Design
Mockup
Copyright © 2009 YourVersion

Visual Design
UI Spec
Copyright © 2009 YourVersion

Visual Design
Specifying Color Palette
Copyright © 2009 YourVersion

The Value of Usability Testing
Critical to talk with customers 1‐on‐1
Gain better understanding of
Usability issues with your product
Customer needs and problems
What alternatives customers are using,
pros & cons of each, customer preferences
QA: use cases & bugs you haven’t seen
Really a “user learning” session
Make test as real for user as possible
Copyright © 2009 YourVersion

“Ramen” Usability for Startups
Anyone can do it!
Ingredients:
1 user with their laptop
1 desk
1 person to conduct usability
Pen and paper
N optional observers
N+2 chairs
Copyright © 2009 YourVersion

Typical Format for Customer Session
5 ‐ 10 min: Ask questions to understand user
needs and solutions they currently use
30 ‐ 50 min: Usability test
Non‐directed as much as possible
When necessary, direct user to attempt to
perform a specific task
5 ‐ 10 min: Wrap‐up
Answer any user questions that came up
Point out/explain features you want to highlight
Copyright © 2009 YourVersion

Dos & Don’ts of Conducting Usability
Do
Explain to the user:
Their usability test will help improve the product
Not to worry about hurting your feelings
“Think Aloud Protocol”
Ask user to attempt the task, then be a fly on the wall
Ask non‐leading, open‐ended questions
Take notes and review them afterwards for take‐aways
Don’t
Ask leading questions
“Help” the user or explain the UI (e.g., “click over here”)
Respond to user frustration or questions (until test is over)
Get defensive
Blame the user
Copyright © 2009 YourVersion

Usability Case Study: Travel Sites
Consider 3 major travel sites
Expedia, Travelocity, Orbitz
All 3 try to provide same user benefits
Easily find flights that match your schedule
Easily find the lowest price flights
UI Design differences make some better
Case study: round trip from San Fran to NYC
Want to find best combo of price and airports
3 possible airports for each city
9 possible one‐way airport combinations
81 possible round‐trip airport combinations
Copyright © 2009 YourVersion

Expedia:
Only One Airport Combo at a Time
• Have to
manually
check all 9
combos
•3 clicks to
change airport
•Then wait for
new results
•24 clicks +
8 page reloads
to see all 9
combos
Copyright © 2009 YourVersion

Travelocity:
Can Select which Airports to Include
Copyright © 2009 YourVersion

Travelocity:
Can only view results 1 combo at a time
• Clicking button brings up
results for this combo
•Problem: browser Back
button loses other
airports!
•Have to go through
‘Change Search’ process =
9 clicks + 4 page reloads
for each combo
•72 clicks + 32 page loads
to see other 8 combos
Copyright © 2009 YourVersion

Orbitz:
Binary choices to include nearby airports
• Good news: all combos
appear in results
•80‐mile radius might
include airports I don’t
want (3 for SFO, 5 for NYC)
but turned out not to be
an issue
Copyright © 2009 YourVersion

All 3 Travel Sites have Identical
“Airline vs. Number of Stops” Price Grids
Expedia
Orbitz
Travelocity
Copyright © 2009 YourVersion

Options for Sorting Flight Results
Website
Can Sort By Expedia Travelocity Orbitz
Airline Y
Departure Time Y Y Y
Arrival Time Y Y
Travel Time Y Y Y
Price Y Y Y
Copyright © 2009 YourVersion

Summary Comparison of Travel Sites
User Benefit Expedia Travelocity Orbitz
Ability to include other
Yes Yes Yes
nearby airports
Ability to pick specific High High
Low
nearby airports (by changing) (can pre‐select)
Ease of seeing results
Med Low High
for multiple airports
Ease of trading off
Low High Med
airport combos vs. price
Airline vs. Number of
Yes Yes Yes
Connections Price Grid
Flight Results Sorting
Med High Low
Options
Overall ability to easily
Med Low High
find best airport combo
Copyright © 2009 YourVersion

And now,
for something completely
different…
Quantitative!
(finally)
Copyright © 2009 YourVersion

Approaching Business as an
Optimization Exercise
Given reality as it exists today,
optimize our business results
subject to our resource constraints.
Copyright © 2009 YourVersion

Define the Equation of your Business
“Peeling the Onion”
Advertising Business Model:
Profit = Revenue ‐ Cost
Unique Visitors x Ad Revenue per Visitor
Impressions/Visitor x Effective CPM / 1000
Visits/Visitor x Pageviews/Visit x Impressions/PV
New Visitors + Returning Visitors
Invited Visitors + Uninvited Visitors
# of Users Sending Invites x Invites Sent/User x Invite Conversion Rate
Copyright © 2009 YourVersion

Equation of your Business
Subscription Business Model
Profit = Revenue ‐ Cost
Paying Users x Revenue per Paying User
New Paying Users + Repeat Paying Users
Trial Users x Conv Rate Previous Paying Users x ( 1 – Cancellation Rate )
( SEO Visitors + SEM Visitors + Viral Visitors ) x Trial Conversion Rate
Copyright © 2009 YourVersion

How to Track Your Metrics
Track each metric as daily time series
Unique Page Ad New User
…
Date Visitors views Revenue Sign‐ups
4/24/08 10,100 29,600 25 490
4/25/08 10,500 27,100 24 480
…
Create ratios from primary metrics: X / Y
Example: How good is your registration page?
Okay: # of registered users per day
Better: registration conversion rate =
# registered users / # uniques to reg page
Copyright © 2009 YourVersion

Sample Signup Page Yield Data
Daily Signup Page Yield vs. Time
New Registered Users divided by Unique Visitors to Signup Page
100%
90%
80%
Daily Signup Page Yield
70%
60%
50%
40%
30% Started requiring
registration
20%
Changed Added questions
messaging to signup page
10%
0%
1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/1
0
Copyright © 2009 YourVersion

Identifying the “Critical Few” Metrics
What are the metrics for your business?
Where is current value for each metric?
How many resources to “move” each metric?
Developer‐hours, time, money
Which metrics have highest ROI opportunities?
Metric A Metric B Metric C
Good ROI Bad ROI Great ROI
Return
Return
Return
Investment Investment Investment
Copyright © 2009 YourVersion

Using Web Analytics Tools to
Understand Your Users
Tracking visitors and traffic
Seeing where users are clicking
Measuring key conversions
Monitoring user feedback
Copyright © 2009 YourVersion

Basic Tracking of Traffic:
Google Analytics
•Unique
visitors
•New vs.
returning
•Pageviews
•Time on site
•Top referrers
•Top geos
Copyright © 2009 YourVersion

Seeing Where Users are Clicking:
CrazyEgg Heatmap
•Shows Click
Density
•Color indicates
% of clicks
•See which links
perform best
•See impact of
UI changes
Copyright © 2009 YourVersion

Measuring Key Conversions:
Conversion Funnel
•Tie user actions to
business goals
•Instrument key steps in
user flow
•See where users are
dropping off
•Quantify improvement
from changes
Copyright © 2009 YourVersion

Monitoring User Feedback
Quantitatively with Kampyle
Unobtrusive
Solicitation Average Grade over time
Simple popup
Average Grade
for each page
on your site
Copyright © 2009 YourVersion

Approaching UI Design Analytically
Typical UI design question:
“When using web pages, do users scroll down?”
‐ Yes
‐ No
UI questions are never yes/no! (not binary)
Should ask: “What percentage of users …?”
UI changes impact your metrics
Impact can be positive, negative, small, large
Seek high‐ROI UI changes
Copyright © 2009 YourVersion

The Fold: “Chrome” Takes Away Pixels
Only
768
600 pixels
pixels
1024 pixels Copyright © 2009 YourVersion

Put Key Conversion Actions Above The Fold
Landing Page A Landing Page B
The Fold
Key conversion action
is above the fold
Key conversion action
is below the fold
Copyright © 2009 Olsen Solutions LLC
Copyright © 2009 YourVersion

The Fold Isn’t Binary Either
768 px
The chrome steals
% of Users
about 170 pixels
600 px 1024 px
Data courtesy of ClickTale
Free trial at www.clicktale.com Copyright © 2009 YourVersion

Analyze User Screen Height Distribution
to Select Design Height for Key Pages
100%
600, 100% 768, 92%
90%
% of Users with Given (or Higher) Screen Height
80%
70%
Going above
60%
768 px = big drop
50%
40%
772, 38%
864, 29%
30%
807, 29% 1024, 20%
866, 25%
20%
1000, 20%
10%
1030, 5%
0%
600 650 700 750 800 850 900 950 1000 1050 1100
Page Design Height (pixels)
Monitor Screen Height (pixels)
Copyright © 2009 YourVersion

Marrying UI Design with Metrics
Think of “Ease of use”
as distinct from the
functionality
User Interface
elements matter
Position
Layout
Size
Color
Font
Text copy
Interaction design
Navigation
Copyright © 2009 YourVersion

Automated Multivariate Testing:
Google Website Optimizer
Copyright © 2009 YourVersion

Case Studies:
Translating Metrics Into Action
Copyright © 2009 YourVersion

Case Study 1: Quicken Brokerage
Optimizing Sign In/Registration Flow
100%
100%
Biggest
80%
drop
% of Users
62.3%
58.8%
60%
50.9%
40% 34.4% 32.7%
20%
0%
Sign in / Account Cash vs. 5 Partner 3 Partner
Registration Type Margin Pages Pages
Copyright © 2009 YourVersion

Mapping the Flow to See Where
Users Were Dropping Off
Open
Account
55%
44% Register Registration (24% of Total)
Process
45% drop off
64%
(20% of total)
of Total Account
36% overall Selection
83% 30% drop off for
56% (46% of Total) (14% of Total) this step
Sign in
Forget 70% Change 80%
Password (32% of Total) Password (26% of Total)
17% drop off 20% drop off
(10% of total) (6% of total)
Copyright © 2009 YourVersion

Redesigned User Flow Improved
Registration Conversion Rate 37%
Released 37% improvement
New Design in conversion rate
Copyright © 2009 YourVersion

Case Study 2: Friendster
Optimizing Viral Growth
% of users
sending = 15% Invites per Invite
invites sender = 2.3 click-through rate
Active Invite Prospective Click Registration Fail
Users Users Process
% of users
who are Succeed
Don’t
active Click Conversion
rate = 85%
Users
• Multiplied together, these metrics determine your viral ratio
• Which metric has highest ROI opportunity?
Copyright © 2009 YourVersion

The Upside Potential of a Metric
?
100% 100%
85%
15% 2.3
0 0 0
Registration % of users sending Avg # of invites
Process Yield invitations sent per sender
Max possible
0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?%
improvement
Copyright © 2009 YourVersion

Okay, so how can we improve the metric?
How do we increase the average number of
invites being sent out per sender?
For each idea:
What’s the expected benefit? (how much will it
improve the metric?)
What’s the expected cost? (how many engineer‐
hours will it take?)
You want to identify highest benefit/cost idea
Copyright © 2009 YourVersion

Case Study 2: Friendster
Doubled Number of Invitations Sent per Sender
Copyright © 2009 YourVersion

Continuous
Improvement
Copyright © 2009 YourVersion

Adding Metrics and Optimization to
your Product Process
Site Level
Business Product Prioritized
Plan Objectives Objectives Feature List
Scoping Feature
Level
Requirements
Design & Design
Code Test Launch
Develop
Metrics & User
Optimize Feedback
Copyright © 2009 YourVersion

Optimization through Iteration:
Continuous Improvement
Measure
the metric
Analyze
Learning the metric
Gaining knowledge:
• Market Identify top
• Customer opportunities
to improve
• Domain
• Usability Design & develop
the enhancement
Launch the
enhancement
Copyright © 2009 YourVersion

Early Stage Product Management
Cheat Sheet
Truly understand your customer needs
Ruthlessly prioritize and launch v1
Talk with users 1‐on‐1 and get feedback
Define equation of your business
Identify and track key metrics
Identify opportunities & prioritize by ROI
Launch, learn, and iterate
Copyright © 2009 YourVersion

Questions?
@danolsen
dan@yourversion.com www.yourversion.com
Copyright © 2009 YourVersion