Design for Strangers: Effective User Experience
Design When Your Users are on Another
Continent
Rashmi Sinha
Jonathan Boutelle
Uzanto Consulting
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Structure of workshop
Introduction

Evaluating Systems (Morning session)

Overview of evaluation

Heuristic Evaluation

Usability Testing

GOMS

Understanding users (Afternoon session)

Personas and Scenarios

Mental Models and Information Architecture

Business of Usability (time permitting)

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Evaluating systems: Available data
streams
Different data streams yield different types of metrics
 Heuristic Evaluation
 Usability Testing
Remote Usability Testing

Server Logs or Transaction Logs

Satisfaction Data

Page Level Ratings

G O MS

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Heuristic Evaluation
Using heuristics (or rules of thumb) for evaluating

systems.
Expert analyze degree to which system complies with rules

Heuristics such as

Keep user informed of system status

Speak the user’s language

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Usability Tests
Test with users

Very useful for design purposes

But software must be built before it can be tested

Difficult to use to convince management

Often conducted in artificial scenarios

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Remote Usability Testing
Advantages

Large Sample Size

Disadvantages

Cost

Most of the usual disadvantages of usability testing

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Server and Transaction Logs
Can give an accurate view of site activity

Can give detailed view of site activity – possible to

drill down
Hard to relate to user experience and user goals

Hard to understand – massive reams of data

Often used by corporations to roughly track user

experience
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Satisfaction Ratings
Give an overall view of the site

Such ratings often have business buy-in

Very difficult to move such numbers

Might not relate to specific aspects of the site

Make effort not to let the satisfaction levels fall

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

GOMS
Can help track the complexity of an interface

How much work it will take to complete a task

Might not tell you what real users will do

Very helpful in comparing interfaces

Can be used with interfaces that have not been

implemented yet
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

What Data Streams to Use
What does it measure

User Behavior (navigation paths, errors) or User Attitudes

(user loyalty, satisfaction)?
Gap between reported and actual behavior.

Recommendation: Have at least one data stream of each.

How comprehensive is the coverage?

how much of the site is covered

the frequency of measurement

Sensitivity of measurement:

How sensitive is data stream to changes in the user

experience
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

What Data Streams to Use continued
Sampling Bias: Every data stream comes with its own set of
•
sampling biases.
The economics of measurement will determine what types of
•
data are practical to collect.
Initial cost
•
Ongoing cost
•
Cost of increasing sample size
•
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Structure of workshop
Introduction

Evaluating Systems (Morning session)

Overview of evaluation

Heuristic Evaluation

Usability Testing

GOMS

Understanding users (Afternoon session)

Personas and Scenarios

Mental Models and Information Architecture

Business of Usability (time permitting)

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Heuristic Evaluation
Developed by Jakob Nielsen

Helps find usability problems in a UI design

Small set (3-5) of evaluators examine UI

independently check for compliance with usability principles

(“heuristics”)
different evaluators will find different problems

evaluators only communicate afterwards

findings are then aggregated

Can perform on working UI or on prototypes or

designs

What are heuristics?
Simple easy rules of thumbs for enhancing usability

For example:

Have simple and natural dialog

Speak the users’ language


Heuristic Evaluation Process
Evaluators go through UI several times

inspect various dialogue elements

compare with list of usability principles

consider other principles/results that come to mind

Usability principles

Nielsen’s “heuristics”

supplementary list of category-specific heuristics

competitive analysis & user testing of existing products

Use violations to redesign/fix problems

From Jakob Neilsen

Heuristic 1: Visibility of system status
sear hi g at base or
cnda f
m at hes
c
The system should always keep users informed
about what is going on, through appropriate
feedback within reasonable time.

Visibility of system status (cont)
Response Time parameters

0.1 sec: no special indicators needed, why?

1.0 sec: user tends to lose track of data

10 sec: max. duration if user to stay focused on action

for longer delays, use percent-done progress bars


Heuristic 2: Match between system &
real world
• The system should speak the users' language,
with words, phrases and concepts familiar to the
user, rather than system-oriented terms.
• Follow real-world conventions, making
information appear in a natural and logical order.
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Use User’s language,
not developer’s
language
There should be a match between system
& real world
follow real world conventions


Heuristic 3: User Control and Freedom
Provide ways for users to backtrack when they

make mistakes.
Have clearly labeled exits allowing users to

backtrack without an extended interaction.
Support undo and redo.


User Freedom Heuristics (cont.)
H2-3: User

control & freedom
“exits” for

mistaken choices,
undo, redo
don’t force down

fixed paths
Wizards

must respond to Q before going to next

Should be easy to good for beginners

have 2 versions (WinZip)


Heuristic 4: Consistency and Standards
Use a consistent look and feel.

Do not confuse users by changing platform

conventions.

Consistency (cont.)
Is this confusing?

Heuristic 5: Error Prevention
Even better than good error messages is a careful

design which prevents a problem from occurring in
the first place.
Example:
If user is asked to spell something, e.g. file names, it
might be easier to give them a menu from which they
can choose the files.
Example: Modes
When the same action leads to different consequences
in different states. For example in older word processors,
there was an insert and edit modes. The same key press
in the different modes would lead to different outcomes.
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Heuristic 6: Recognition rather than recall
Make objects, actions, and options visible.

The user should not have to remember information

from one part of the dialogue to another.
Instructions for use of the system should be visible

or easily retrievable whenever appropriate.
Computers good at remembering things, human

beings are not.
Computer should display dialog elements to the

user, and have them make a choice.
During web navigation, remind users where they are

currently.

Heuristic 7: Flexibility & efficiency of use
Accelerators -- unseen by the novice user -- may

often speed up the interaction for the expert user
such that the system can cater to both
inexperienced and experienced users.
Allow users to tailor frequent actions.


Flexibility (cont.)
Edit
Ctrl-X
Cut
Ctrl-C
Copy OR
Ctrl-V
Paste
accelerators for experts (e.g., gestures, kb shortcuts)

allow users to tailor frequent actions (e.g., macros)


Heuristic 8: Aesthetic and minimalist
design
Dialogues should not contain information which is irrelevant or

rarely needed.
Every extra unit of information in a dialogue competes with the

relevant units of information and diminishes their relative
visibility.

Heuristic 9: Help users recognize, diagnose,
and recover from errors
Error messages should be expressed in plain

language (no codes), precisely indicate the problem,
and constructively suggest a solution.

Heuristic 10: Help and documentation
•It is better if the system can be used without
documentation, but it may be necessary to
provide help and documentation.
•Any such information should be easy to
search, focused on the user's task, list concrete
steps to be carried out, and not be too large.

Phases of Heuristic Evaluation
Pre-evaluation training

give evaluators needed domain knowledge and information

on the scenario
Evaluation

individuals evaluate and then aggregate results

Severity rating

determine how severe each problem is (priority)

can do this first individually and then as a group

Debriefing

discuss the outcome with design team


How to Perform Evaluation
At least two passes for each evaluator

first to get feel for flow and scope of system

second to focus on specific elements

If system is walk-up-and-use or evaluators are

domain experts, no assistance needed
otherwise might supply evaluators with scenarios

Each evaluator produces list of problems

explain why with reference to heuristic or other information

be specific and list each problem separately


Examples
Can’t copy info from one window to another

violates “Minimize the users’ memory load” (H1-3)

fix: allow copying

Typography uses mix of upper/lower case formats

and fonts
violates “Consistency and standards” (H2-4)

slows users down

probably wouldn’t be found by user testing

fix: pick a single format for entire interface


Severity Rating
Used to allocate resources to fix problems

Estimates of need for more usability efforts

Combination of

frequency

impact

persistence (one time or repeating)

Should be calculated after all evals. are in

Should be done independently by all judges

Severity Ratings

0 - don’t agree that this is a usability problem

1 - cosmetic problem

2 - minor usability problem

3 - major usability problem; important to fix

4 - usability catastrophe; imperative to fix


Debriefing
Conduct with evaluators, observers, and

development team members
Discuss general characteristics of UI

Suggest potential improvements to address major

usability problems
Dev. team rates how hard things are to fix

Make it a brainstorming session

little criticism until end of session


Results of Using HE
Single evaluator achieves poor results

only finds 35% of usability problems

5 evaluators find ~ 75% of usability problems

why not more evaluators???? 10? 20?

adding evaluators costs more

many evaluators won’t find many more problems


Summary
Heuristic evaluation is a discount method

Have evaluators go through the UI twice

Ask them to see if it complies with heuristics

note where it doesn’t and say why

Combine the findings from 3 to 5 evaluators

Have evaluators independently rate severity

Discuss problems with design team

Alternate with user testing


Heuristic Evaluation Exercise
Split into two groups

Conduct Heuristic Evaluation as a group (Create list

of heuristic violation)
Each person within group provides a severity rating

for each heuristic violation (eliminate redundancies)
Average severity for each group

Present back to larger group

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Structure of workshop
Introduction

Evaluating Systems (Morning session)

Overview of evaluation

Heuristic Evaluation

Usability Testing

GOMS

Understanding users (Afternoon session)

Personas and Scenarios

Mental Models and Information Architecture

Business of Usability (time permitting)

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Overview of user testing
Why do user testing?

Choosing participants

Designing the test

Collecting data

Analyzing the data

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Why do User Testing?
Can’t tell how good or bad UI is until

people use it!

Other methods are based on evaluators who?

may know too much

may not know enough (about tasks, etc.)

Summary:

Hard to predict what real users will do
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Choosing Participants
Representative of eventual users in terms of

job-specific vocabulary / knowledge

tasks

If you can’t get real users, get approximation

system intended for doctors

get medical students

system intended for electrical engineers

get engineering students

Use incentives to get participants

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Ethical Considerations
Sometimes tests can be distressing

users have left in tears

users can be embarrassed by mistakes

You have a responsibility to alleviate this

make voluntary with informed consent

avoid pressure to participate

let them know they can stop at any time [Gomoll]

stress that you are testing the system, not them

make collected data as anonymous as possible

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

User Test Proposal
A report that contains

objective

description of system being testing

task environment & materials

participants

methodology

tasks

test measures

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Selecting Tasks
Should reflect what real tasks will be like

Tasks from analysis & design can be used

may need to shorten if

they take too long

require background that test user won’t have

Avoid bending tasks in direction of what your design

best supports
Don’t choose tasks that are too fragmented

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Deciding on Data to Collect
Two types of data

process data

observations of what users are doing & thinking

bottom-line data

summary of what happened (time, errors, success…)

i.e., the dependent variables

Focus on process data first

gives good overview of where problems are

Bottom-line doesn’t tell you where to fix

just says: “too slow”, “too many errors”, etc.

Hard to get reliable bottom-line results

need many users for statistical significance (don’t bother unless

needed)
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

The “Thinking Aloud” Method
Need to know what users are thinking, not just what

they are doing
Ask users to talk while performing tasks

tell us what they are thinking

tell us what they are trying to do

tell us questions that arise as they work

tell us things they read

Make a recording or take good notes

make sure you can tell what they were doing

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Thinking Aloud (cont.)
Prompt the user to keep talking

“tell me what you are thinking”

Only help on things you have pre-decided

keep track of anything you do give help on

Re c o r d in g

use a digital watch/clock

take notes, plus if possible

record audio and video (or even event logs)

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Using the Test Results
Summarize the data

make a list of all critical incidents (CI)

positive: something they liked or worked well

negative: difficulties with the UI

include references back to original data

try to judge why each difficulty occurred

What does data tell you?

UI work the way you thought it would?

consistent with heuristic evaluation

users take approaches you expected?

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Using the Results (cont.)
Update task analysis and rethink design

rate severity & ease of fixing CI’s

fix both severe problems & make the easy fixes

Will thinking aloud give the right answers?

not always

if you ask a question, people will always give an answer,

even it is has nothing to do with the facts
try to avoid specific questions

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Measuring Bottom-Line Usability
Situations in which numbers are useful

time requirements for task completion

successful task completion

compare two designs on speed or # of errors

Do not combine with thinking-aloud

talking can affect speed and accuracy (neg. & pos.)

Time is easy to record

Error or successful completion is harder

define in advance what these mean

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Analyzing the Numbers
Example: trying to get task time <=30 min.

test gives: 20, 15, 40, 90, 10, 5

mean (average) = 30

median (middle) = 17.5

looks good!

wrong answer, not certain of anything

Factors contributing to our uncertainty

small number of test users (n = 6)

results are very variable (standard deviation = 32)

std. dev. measures dispersal from the mean

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Measuring User Preference
How much users like or dislike the system

can ask them to rate on a scale of 1 to 10

or have them choose among statements

“best UI I’ve ever…”, “better than average”…

hard to be sure what data will mean

novelty of UI, feelings, not realistic setting, etc.

If many give you low ratings, you are in trouble

Can get some useful data by asking

what they liked, disliked, where they had trouble, best part,

worst part, etc. (redundant questions)
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

User Testing: Cultural Issues
Are users the same all over

Obviously not

Getting users that are as similar as possible to your real

users is important
Can you test on users from another country?

Probably not for things that are culturally specific

Entertainment

marketing-ware

Generic business software

Yes for applications targeted at specialists with strong

international work cultures
Doctors

Software engineers

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Testing Details
Order of tasks

choose one simple order (simple -> complex)

Training

depends on how real system will be used

What if someone doesn’t finish

assign very large time & large # of errors

Pilot study

helps you fix problems with the study

do twice, first with colleagues, then with real users

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Instructions to Participants
Describe the purpose of the evaluation

“I’m testing the product; I’m not testing you”

Tell them they can quit at any time

Demonstrate the equipment

Explain how to think aloud

Explain that you will not provide help

Describe the task

give written instructions

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Details (cont.)
Keeping variability down

recruit test users with similar background

brief users to bring them to common level

perform the test the same way every time

don’t help some more than others (plan in advance)

make instructions clear

Debriefing test users

often don’t remember, so show video segments

ask for comments on specific features

show them screen (online or on paper)

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Summary
User testing is important, but takes time & effort

Early testing can be done on a mock-ups (low-fi)

Use real tasks & representative participants

Be ethical & treat your participants well

Want to know what people are doing & why

i.e., collect process data

Using bottom line data requires more users to get

statistically reliable results
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

User Testing Exercise
Divide into groups

Each group devise a test plan

2 tasks, where to get users from, who to test

Test someone from the other group

Note findings

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Structure of workshop
Introduction

Evaluating Systems (Morning session)

Overview of evaluation

Heuristic Evaluation

Usability Testing

G O MS

Understanding users (Afternoon session)

Personas and Scenarios

Mental Models and Information Architecture

Business of Usability (time permitting)

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

GOMS
Can help track the complexity of an interface

How much work it will take to complete a task

Might not tell you what real users will do

Very helpful in comparing interfaces

Can be used with interfaces that have not been

implemented yet
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

GOMS Overview
Goals, Objects, Methods, Selection Rules

A way of measuring how much work it takes to do

something using a given information system
System doesn’t have to exist yet

Many GOMS variants: most are quite complex and

difficult to implement
A simplified version of Keystroke-Level GOMS will

be presented today
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

GOMS Keystroke Actions
The actions

K (Click, Keying): .2 Seconds

M (mentally preparing): 1.35 Seconds

P (pointing): 1.1 Seconds

H (homing) (move hand between keyboard and pointing device) .4

Second
R (system responding): varies by system / action

Very approximate estimates of time to do task

Useless for predicting how much time a task will take

 Thinking doesn’t always take 1.35 second
 Pointing time varies with size of target and distance from current
location (Fitt’s law)
Yet valid on a comparative basis if two designs / systems are analyzed

using the same technique
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

EZ-GOMS Calculation
Explicitly specify a task

Typically many potential paths through a given design, optional fields

etc: get explicit
Consider using ranges (minimum, maximum, typical) to get a better

sense of best / worst case scenarios
Calculate all the actions that will be taken to perform that task

Add M (mental preparation) in using this rules

In front of all clicking

In front of all pointing

Remove “M”s using these rules (you’ll do this automatically after a

little practice)
Remove anticipated “M”s (M P M K-> M P K)

Remove “M”s within cognitive units (“fred”-> MKMKMKMK->MKKKK)

Remove overlapping “M”s (adjacent to Rs)

Remove “M”s before consecutive terminators }}

Remove “M”s that are terminators of commands

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

EZ-GOMS Example
Sign in to Yahoo!
Yahoo! ID:
Yahoo!
Password:
Remember my ID & Password
Sign in
Need help signing in?
H M P K H (select name text box)

M K K K K K K (enter name)

H M P K H (select password text box)

M K K K K K K (enter password)

H M P K (click “sign in” button)

R (waiting for the server to respond)

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Understanding User Needs
Afternoon Session
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Structure of workshop
Introduction

Evaluating Systems (Morning session)

Overview of evaluation

Heuristic Evaluation

Usability Testing

GOMS

Understanding users (Afternoon session)

 Personas and Scenarios
Mental Models and Information Architecture

Business of Usability (time permitting)

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Problem with traditional user research
methods
Long sessions of observing users or interviewing

them or participatory design.
Appropriate in face to face interaction situations.

Methods work well in designing for easy to access

audiences. Difficult to use for remote users.
Difficult to use when designing for global audiences.

Also difficult to use such methods to make business case

since numbers are small and data is qualitative.
So what is the answer?

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Semi-structured user research methods
Using mostly phone and online surveys

Complementary with, rather than an alternative to

open-ended methods
Can work for information-rich domains

Help understand information representations in user’s

minds. e.g. design of navigation for cell phone.
Work well in remote situations

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Two types of user research methods
Part 1: User information needs

What user needs are important?

Can users be differentiated into groups on

the basis of such needs? Can this grouping
be used to form personas?
Part 2: User Categorizations

Scope & boundaries of information domain

Structure of information domain

Differences between groups of people

(different user groups, different cultures,
stakeholders)
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Part 1: Understanding user needs,
creating scenarios & personas remotely
Why persona based design

One of the problems in design is

that it is very hard to visualize an
abstract “USER” and what he / she
might want
Many potential users
• Develop one or two persona of the
typical “user” from interviews with
many users
• Persona is made up person, your
so called “typical user”.
• Should be based on your
experiences with actual users in
the interview stage.
One Persona
esio orS tanger or shop
Fr nm Alan Cooper Uzanto Consulting
g f r sW k

Persona based Design Process
Persona:

The archetypical user

G o a ls

Goals of the persona in using the software

Tasks

Specific steps needed to accomplish goal.

Scenario

The usage scenario, the whole incident of software usage

From Alan Cooper
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Characteristics of Personas (from Cooper)
“Hypothetical Archetypes”

Archetype:

An original model after which other similar things are patterned; a

prototype
A precise description of a user and what they want

to accomplish
Imaginary, but precise

Specific, but stereotyped

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Targeted Design with Personas
Describe a person in terms of their

Goals in life (especially relating to this project)

Capabilities, inclinations, and background

People have a “visceral” ability to generalize about real and

fictional people
They won’t be 100% accurate, but it feels natural to think about

people this way
Why use personas

 If you try to satisfy everyone, you end up satisfying no one. A
compromise design pleases no-one
 From all your interviews etc.,
decide what is your typical user / users,

create a specific persona

then try to please that that persona 100% of the time.

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Advantages of Personas
Targeted Design Works Better

Example: Roller suitcases

Was designed specifically for airline employees, pilots, airhostesses

etc.
Has become popular with all classes of people

In order to do good design you need to have a specific person in mind,

and think in terms of that person every time a design decision needs to
be made
Puts an end to feature debates

Makes hypothetical arguments less hypothetical

Q: “What if the user wants to print this out?”

Typical discussion “The user will / wiil not want to print often.”

“Given her tasks, and Emilee won’t want to print often.”

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Case Study using Personas
Primary Persona

Joe, the executive

Make him happy 100% of the time

Secondary Persona

Dan, the traveler

Try to take care of his needs as well

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Developing Personas cont.
Joe: The busy traveling executive from a

multinational company. He is on the road about 10
days a month.
He is very fond of food but is afraid to explore

in strange cities, and prefers restaurants
which serve good, but not exotic food.
He is also fond of a beer with his meal.

He does not like to travel far for food, prefers

to walk or hop into a cab for a short ride
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Developing Personas cont.
Dan: Driving his car across the country after

graduating. Gets to a different city every
night and finds a hotel and a restaurant.
He wants to explore the town, find the local

hangouts, understand the town’s culture.
He likes to try different kinds of food.

He prefers restaurant in the middle of the town.

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Goals and Tasks of Users
Goals are larger functions that the user is hoping to

satisfy
Get acquainted with the city, discover its special cuisine

Not have to travel too much for food

Relax after a hard day’s work / driving

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Tasks of users
Tasks are the specific steps that the user has to go through in

order to accomplish his goals. Asks include the usage of the
software.
Find information about various restaurants

Decide on the one based on factors such as price, cuisine,

serves alcohol or not/ distance from location
Get to the restaurant

Eat

Pay for meal

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Development of Scenarios
Primary Persona: Joe, the executive
Make him happy 100% of the time
•Scenario: Joe’s company has tied up with some Delhi IT
company, and he is visiting Delhi for the first time.
•He is staying somewhere near South Ex.
•He needs to find a restaurant to eat at.
•He is not feeling adventerous, so not Dosa! Just
some safe Burger and Fries.
•So Joe turns to his trusted Palm
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Development of Scenarios
Joe needs to input his location into his palm.
Input what kind of food he wants or the
program can use defaults
The information returned: list of possible restaurants
along with their relevant details, kinds of food etc.
More details about each on request: details such as the
availability of beer, if they take credit cards, links to
reviews etc.
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Development of Scenarios
The information returned to Joe
needs to be broad (offer a
number of options) and deep
(offer more details upon request)
Location Information is another concern of Joe’s.
Ideally he wants exact distance & directions to
restaurant.
Not possible, not live website
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Development of Scenarios
Compromise: Tag restaurants in terms
of neighborhoods.
Joe can give current neighborhood.
Can be shown map with neighborhoods
marked out & approximate distances.
What else does Joe need?
To mark restaurants that he liked.
Lets think more…
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Our secondary Persona
Does this design make Dan happy?
Designing for one specific user often makes other
users happy as well.
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Aspects of Scenarios
Daily Use


Fast to learn

Shortcuts and customization after more use

Necessary Use

Infrequent but required

Nothing fancy needed

Edge Cases

Ignore or save for version 2

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Personas and Market Segmentation
Uses of Market Segmentation

Used to identify clusters of people product can

appeal to.
Using demographics or using

attitudinal/psychological/psychographic variables.
Questions focus on like / dislike of product concept

what do you think of vanilla coke or green Heinz ketchup?

Forecasts marketplace acceptance of products.

Helps convince executives to build product.

Not helpful for defining and designing product.

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Reconciling personas and market
segments
Build personas on top of segments

Ground the personas in reality.

Define a persona for each main segment

Focus on goals and behaviors of users.

Advantages:

Easy to get buy-in for personas from management,

engineering etc.
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Persona building method
Method

Conduct secondary research

Examine existing market segments

Conduct interviews with various stakeholders, including multiple

users
Conduct online survey if users are remote.

Find patterns.

Pick nugget and interesting tidbit and build persona around it.

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Conduct secondary research
Examine existing market segments

What type of user population is product/site targeting

How should you identify current segments?

Easier for demographic segments

More difficult for attitudinal segments

What type of population characteristics are useful for

design purposes?
Example: Segments for Palm based restaurant finder

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Stakeholder and user interviews
Can be in person or on phone

Semi-structured interviews:

Decide on few questions before-hand leaving room for

change.
Ask about scenarios of usage: e.g., last time

they used product.
Go through steps of usage, exact context, motivations

etc.
Tape interview if possible or keep a phone log.

Interview people from each user segment.

Ask for a few ratings on a five-point scale.

Aggregate rating information for sake of comparison.

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Online survey of user needs
(optional)
Important for remote users or if there are many types of

users
Example

Conduct online survey on factors used in finding

restaurants for travelers.
Identified factors important in choosing restaurants. e.g., Food

quality, décor, wine selection, cuisine, service.
Ask for importance ratings (on 5-point scale) of factors.

Tie response to behavior: Asked respondents to recall a

specific incident of choosing a restaurant, rather than
answer questions in an abstract fashion.
Option: Ask about several scenarios of usage from same

person. e.g., One restaurant visit with business
colleagues, another with friends.
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Personas Exercise
Divide into groups

Craft a primary and secondary persona for your product

Think of all that you know about your users

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Structure of workshop
Introduction

Evaluating Systems (Morning session)

Overview of evaluation

Heuristic Evaluation

Usability Testing

GOMS

Understanding users (Afternoon session)

Personas and Scenarios

Mental Models and Information Architecture

Business of Usability (time permitting)

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Understanding User categorizations
Overview

Why people categorize?

The structure of semantic memory

Is understanding user categorization important for design?

Methods

Free-listing.

Types of Card Sorting.

Testing information architecture.

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Is understanding categorization
useful for design?
Direct use: when user categorization informs design, such as

that of menus or of navigation design. Often referred to as
information architecture (IA).
Indirect use: good to have broad understanding how users

think about product even when user categorization does not
directly inform IA.
Important to remember:

Categorization is not static. People are good at learning new

categories. If you provide the context and the right examples,
they can learn new categories or alter boundaries of old
categories.
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Should interfaces always reflect user
categories faithfully?
No.

Categorization is far too important to depend only on what user

thinks.
Should also be influenced by business proposition, strategy,

brand etc.
Different user groups might differ in their perception of

domain. No one scheme can serve them all perfectly.
User research can provide several alternative

categorization schemes, allowing designers the freedom
to make choices.
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Do categorizations work across culture
Research shows

the structure of categories can be similar across cultures,

though content of categories might not be.
Enough similarity for successful design.

The net generation shares a lot of culture

Cross-cultural design has been happening anyway.

Japanese cars

Italian fashion

Swiss chocolates

Indian ???

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Free-listing methods for
understanding scope and boundary of
domain
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Free-listing to explore domain scope and
boundaries
Goals

Explore boundaries and scope of domain across a group of

people.
Gain familiarity with user vocabulary for the domain.

Use as a precursor to card-sorting, to define and limit the

domain, and frame card items in the user’s language.
Method

Can be conducted as part of interview, or as written exercise

Ask respondent, “Name all the x's you know.” Give sufficient

time to do so.
How many respondents?

Depends on how much agreement there is about the domain. more

agreement > fewer respondents.
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Free-listing menu for Mc Donald’s
User No 4
User No 1 User No 2
Chicken Mcnuggets
French fries French fries
Cheese burger
Chicken
Cheese burger
Bacon cheese burger
Cheese burger
Shake
French fries
Shake
Hamburger
French fries
User No 5
Chicken sandwich User No 3 Hamburger
Chicken Mcnuggets Hamburger Quarter pounder
Fish sandwich Cheese burger Big mac
French fries
Shake Chicken fajita
Mc rib
Hamburger French fries
Chicken Apple pie
sandwich
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Analyzing free-listing data
Create a list of all items, sorted by their average rank (of being listed

by a respondent). Examine how that rank order changes with the
addition of each new respondent. If the ranks are relatively stable,
then you can stop adding new respondents.
Items Listed by %
participants
Items Listed by %
participants
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Concept structure
• i i e t m s nt
D v d ie i o3
–P l tiem s ccor i g o
o t a dn t
concent i i cl s use our
rc c r e ( y
f equency fm enton
r o i
ow n r ak oi t )
b e p ns:
Periphery
% of times items were mentioned
Middle
40
35
30
25
Core
20
15
10
5
0
Ite m s
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Other uses for free-listing
Comparing cultural or other group differences

How do two groups perceive the same domain?

Comparing two domains

How does perception of McDonald’s menu compare with

Wendy’s?
Segment respondents into types based on

familiarity:
Find respondents with greater domain familiarity or those

who perceive domain in idiosyncratic fashion?
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Card-sorting and other methods for
designing information architecture
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Case Study: Design of online travel guide
Example: Designing an online travel guide

to help users plan trips.
Purpose of card sort:

to structure the website for helping users find

travel information, and create personalized
travel guides.
Items include

lodging, entertainment, local information, When

to Go, Travel by Car/Air/Bus, Music Events,
Hiking, Day Trips, Skiing, Diving, Golf,
Emergency Info.
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

ca rdso rt
Open card-sorting
Goal: to understand the overall

categorization scheme
Method: Open card sort

Users given items. Asked to create categories

Options:

Provide total number of categories to be

created (avoid problems with splitters and
lumpers)
Successive card sorts to create taxonomies

It is ok to put one card in multiple groups

Ask for labels for each grouping

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Cluster Analysis for card-sorting data
H ot l
es
Cluster Analysis

B ed nd
a
Suggests a structural

B r akf st
ea
solution. Easy to
R est ur nt
aas
translate into design.
H ost l
es
Challenge: How to

E m er ency nf
g Io
reconcile multiple
C ur ency
r
schemes?
C am pi g
n
Hi i g
kn
D ay r ps
Ti
S king
i
Di i g
vn
S ur i g
fn
M ount i
an
C lm bi g
in
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Closed card-sorting to design an IA
Goal: to understand goodness of existing information architecture

and labels
Method: Closed card sort

Users given items and category labels. Asked to place each item in a

category.
Do not allow creation of a miscellaneous category.

Useful for:

Understanding user categorizations when category labels are a given

Refining existing categorization scheme.

Options:

Allowing items to belong to multiple categories.

Providing category descriptions rather than category labels.

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Doing closed card-sorting online
User works with given categories

Each item (card) occupies a row

Each category is represented by a column

An “Other” category catches items that do not fit in

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Comparing card-sorts for different user
types
Very useful for understanding differences in mental maps of

various groups
 Can help understand differences between user groups,
different cultures etc.
 Try to create consensus maps to reconcile differences
between different groups.
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Practical exercise
Using the RUMM (Rapid User Mental Modeling)

method.
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Structure of workshop
Introduction

Evaluating Systems (Morning session)

Overview of evaluation

Heuristic Evaluation

Usability Testing

GOMS

Understanding users (Afternoon session)

Personas and Scenarios

Mental Models and Information Architecture

Business of Usability (time permitting)

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Swimming with Sharks: The Business
of Usability
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

What we’ll cover
Stakeholder analysis for fun and profit

Making a business case for a User Experience

project
Test out the ideas with a sample project

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Stakeholder Analysis
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Who are stakeholders and why
should we analyze them?
Stakeholder: Anyone who is affected by, or

can affect, your project
Goals of understanding stakeholders

Make your design better, by getting important

information about the business context
Identify potential obstacles ahead of time so you

can deal with them
Change design to address the issues raised by

stakeholders
Marshal evidence to counter their objections

Neutralize resistance by making stakeholders feel

heard
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Putting Stakeholders into context
It does not matter how good the design is if it is not approved

by management and actually put into operation
A given project isn’t necessarily in everybody’s best interest

This isn’t about playing politics: this is about the institutional

decision making process.
People represent different organizations within an enterprise

If a project is seen as a big negative by various organizations, it

should either address the concerns raised or justify itself
strongly in order to be approved
Stakeholders as another class of users who design should

satisfy
A real person you can talk to

Goals are typically very concrete and business-metrics oriented.

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Understanding Who’s Who in an
Organization
Org charts don’t tell the whole story

Detective work needed to sort out

Motive

Influence

How to do?

Indirect

Watch for “Influence Tells”

Direct

“What are the organizational challenges?”

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

The Interview
Ask semi-structured questions about the product in

general
What group of users is least well-served?

What one change would impact profits the most?

Where do you see <<product>> in 5 years?

Find out what their conception of your project is

What might happen if this project went well?

What are some risks associated with this project?

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Remote Interviews
Online Survey

Ask same questions as in face-to-face interview

Limit to 5 minutes of work

Phone Interviews

Follow-up on survey answers: clarify answers, try to get a

sense of a concerns
Compared to face-to-face interview

Less emotional connection

Even more necessary (remoteness means you know even

less about stakeholders and their concerns)
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Recording your understanding
Table 1: Stakeholder Perspectives
Stakeholders Pos ition Influe nce Inte r e s t in Goals Obje ctions to Project
Project
A ndr e Agassi CEO 10 High Es timates quarterly Seems like iSeems like it
estimates for next w on’t pay off in the time
4 quarters frame he’s most
concerned about
Chris Evert Produc t 6 Medium Inc reas e % of Will it reduce number of
Manager company revenue sales?
generated by this
product Get
noticed by Andre
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Prioritizing Stakeholders
A ndre
S andeep
H i h nfuence
g Il
C hrs
i
Low nfuence
Il A nu
Low nt r st H i h nt r st
I ee g I ee
• H i h nfuence i h nt r st ngage
g Il /H g I e e :E
• Low nfuence i h nt r st se s nf r aton our e
Il /H g I e e :U a I o m i S c
• H i h nfuence ow nt r st r adl atsf
g Il /L I e e :B o y S i y
• Low nfuence ow nt r st voi
Il /L I e e :A d
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

An organizational dilemma
Usability often an Independent Business Unit

IBUs provide “accountability”, make measurement easier

Engineering is responsible for paying for usability services

Engineering measured on the basis of

Schedule

Feature checklists

# bugs

Marketing/Sales measured on the basis of

 Sales
Engineering invests in usability

Money, Time

but Marketing / Sales reap the benefits!

Solution: tie engineering compensation to usability metrics

Good luck

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Building a Business Case for Usability
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

ROI of Usability: Previous work
Cost – Justifying Usability (Bias & Mayhew)

Cost (employees,subjects,equipment)

Benefit (task speed, user errors, late design

changes, increased sales)
Internal vs. external

Internal benefits increase with # users and frequency of use

External benefits increase with development budget, large base of sales

Usability Return on Investment (Nielson Norman Group)

“Usability Projects have an ROI of 150 %”

Measured by

sales conversions

Traffic / Visitor Count

User performance / productivity

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Myths of Usability ROI*
Generalizing ROI estimates

Assuming improvements are due to usability

Benefits to customer booked as benefits to software company

Support, training are profit centers in enterprise software!

How does usability increase revenue?

Win/loss reports for enterprise software sales

User research to determine buying reasons for shrink-wrap software

registration / shopping cart behavior for ecommerce

Ignores competitive landscape

Being the “overall best choice” in your niche wins you the sale

Usability may play a greater or lesser role in determining this

Ignores potential negative business impact of changes that enhance

usability
Marketing vs. User Experience in ecommerce

Ignoring opportunity costs

“Should the project be approved? Yes, because NPV is positive.”

*Rosenberg, BayCHI 2003
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Building a Business Case *
Understand your business,

The financial levers for the company

The competitive environment that company operates in

Understand Project Approval Process

Who has say, what are the stages of project approval

What metrics the enterprise cares about

Understand threats and opportunities from UX perspective

User and Stakeholder Research

Find areas where user and business interests are in tandem

Try to frame UX projects such that

Risk low, payoff high (it is all about risk)

Chances of success are high

Estimate ROI

Estimate Costs: Development,Negative Revenue Impact, Opportunity Cost

Estimate Benefit (be conservative)

After the project

Follow up: track successes and failures. Be accountable.

* ef r nce:H er an, .C H I2004
r ee m J
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Key Points
Not every project will be justifiable

ROI for some projects will be huge

Ultimate proof is in “moving the needle”

Different companies care about different “financial levers”

(business metrics)
Make your case on the basis of those numbers

For example, # Registrations, % successful registrations, support

calls per customer, average sale size
Management doesn’t care about methodology

Don’t justify methodology

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Key Points (cont.)
UX practitioners should understand business levers

and incorporate them into design at a core level
Post-hoc justification is not enough

Project selection and design should be informed by

business metrics
Some UX practitioners should learn about business

analysis
Take a process oriented approach

Evolve a process that takes into account the various

interests and goals within an organization
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Example Situations: ROI in an ecommerce
Context
Context: Online book seller is planning to improve

the checkout process
Metrics:

Number of shopping cart bailouts

Performance on usability test

It is easy to justify ROI of shopping cart

improvement since fewer bailouts means more
sales.
Design should focus on reducing bailouts

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Example Situations: ROI in a Customer
Service Context
Context: Bank is planning to two projects to reduce call volume

(a) let users look at their account balance, and (b) let users
update their contact information.
Metrics

Call volume metrics (overall # of calls, per task # of calls)

# Online Transactions (that plausibly replaced calls)

Performance on usability test

It is easier to justify ROI of updating contact information than of

looking at their account balance
Updating of contact information plausibly replaces a phone call

Looking at account balance does NOT plausibly replace a phone call.

Did they even care, or are they just browsing?

Even if they did care, benefit is more diffuse (customer convenience ->

loyalty)
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Crossing the Chasm
Where in the technology adoption life cycle does

usability matter?
Ea
n o E ar
In Lat Lag
ry
lM
l e
vat y A gar
Ma
aj
dap
or ds
jr
ort oi
s iy
tr t
os y
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Revised technology life-cycle
bow lng ley ai t eet
i al m nsr
t r ado
on
chasm
n o E ar
In Ea Lat Lag
ry
l e
lM
vat y A d gar
Ma
or ds
aj
apt jr
oi
s ort
or t
iy y
s
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

ROI of UX in an Outsourcing context
Software Services -> Software Products

Product development requires understanding users on a

deeper level
Good times ahead?

For Services

It depends on the situation of your customer

Your ROI of designing systems that satisfy your customer is

huge (duh)
But your customer is hardly ever the user

So it depends on the business situation of your client

What kind of clients would care about usability?

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

What kind of clients care about usability?
Clients who’s customers have low switching costs

Money

Time

Expertise

Clients where the buyer=the user

Business success comes from making the buyer happy: if the buyer is the user,

usability plays a bigger role
Clients operating in a fiercely competitive landscape

The better your competition is, the better you have to be to win a sale

Usability is one dimension by which products can be better

Clients making very high quality products

Trying to cross the chasm?

Four types of contexts

C ont ent

Ecommerce

D es k t op

Enterprise

Uzanto Consulting
esi n orS tanger or shop
g f r sW k

What’s Next
Where do we go from here?

Can engineers do usability work on their own products?

Are usability specialists needed?

What kind of processes / corporate structures will facilitate

usability work in software companies?
Uzanto Consulting
esi n orS tanger or shop
g f r sW k

Thank you
jon@uzanto.com
rashmi@uzanto.com
slides and other material will be posted at
www.uzanto.com/papers/indiamar04
Uzanto Consulting
esi n orS tanger or shop
g f r sW k