Usability Testing: Options to Fit Any Budget Kathy McShea 2008

Usability is: “the effectiveness and efficiency of a system to meet the user’s needs, resulting in user satisfaction and productivity .” Usability is an essential part of all great Web sites Usability Technology Content Usability Engineering is: “ tools , techniques and user centered processes applied to achieve usability.”

Satisfaction = Loyalty Rational Loyalty Are you satisfied? Would you recommend? Would you come back? Emotional Loyalty Confidence This {brand} delivers what they promise This {brand} is a name I can always trust Integrity {brand} always treats me fairly If a problem arises, I can count on {brand} to reach a fair and satisfactory resolution Pride I feel proud to be a {brand} customer {brand} always treats me with respect Passion {brand} is the perfect company for people like me I can’t imagine a world without {brand}

Rational Loyalty

Are you satisfied?

Would you recommend?

Would you come back?

Emotional Loyalty

Confidence

This {brand} delivers what they promise

This {brand} is a name I can always trust

Integrity

{brand} always treats me fairly

If a problem arises, I can count on {brand} to reach a fair and satisfactory resolution

Pride

I feel proud to be a {brand} customer

{brand} always treats me with respect

Passion

{brand} is the perfect company for people like me

I can’t imagine a world without {brand}

The elephant in the room... VISITORS 85% abandon a site due to poor design 83% leave a site if they have to make too many clicks to find what they are looking for 62% gave up looking for an item while shopping online 40% don’t return due to a negative experience like hard to use content ORGANIZATIONS 50% of sales are lost because visitors can’t find content 51% of sites comply with simple usability principles

VISITORS

85% abandon a site due to poor design

83% leave a site if they have to make too many clicks to find what they are looking for

62% gave up looking for an item while shopping online

40% don’t return due to a negative experience like hard to use content

ORGANIZATIONS

50% of sales are lost because visitors can’t find content

51% of sites comply with simple usability principles

Why usability works... Ensures your Web site is: Easy to learn and use Useful to the intended audience Satisfying to use Users can: Easily find information on your site Perform tasks on your site with confidence Understand information (jargon, acronyms, wall of words) Use the information to accomplish their goal Leave satisfied with the experience, recommend and decide to return

Ensures your Web site is:

Easy to learn and use

Useful to the intended audience

Satisfying to use

Users can:

Easily find information on your site

Perform tasks on your site with confidence

Understand information (jargon, acronyms, wall of words)

Use the information to accomplish their goal

Leave satisfied with the experience, recommend and decide to return

Top 3 Benefits Get actionable results to measure progress: are you succeeding or failing? Align with industry best practices – don’t reinvent the wheel Lead with facts not opinions; it is a research-based orientation

Get actionable results to measure progress: are you succeeding or failing?

Align with industry best practices – don’t reinvent the wheel

Lead with facts not opinions; it is a research-based orientation

Who is your user? User characteristics: Who is the site for? What are the users like? Environmental characteristics: When/where will they access the site Goal and Task Characteristics: Why will they come to the site? What will they do on the site?

User characteristics:

Who is the site for?

What are the users like?

Environmental characteristics:

When/where will they access the site

Goal and Task Characteristics:

Why will they come to the site?

What will they do on the site?

What are the right tasks? Assess audience goals using... Surveys Call record data Interviews Search log analysis Field studies Assess business and organization goals... Collect data from leadership Surveys Interviews Review relevant business documents Conduct competitive analysis

Assess audience goals using...

Surveys

Call record data

Interviews

Search log analysis

Field studies

Assess business and organization goals...

Collect data from leadership

Surveys

Interviews

Review relevant business documents

Conduct competitive analysis

What’s the right test? Inspection evaluations Expert Review with scorecard and optional report (1 week) Human performance testing Top Task Tune-up (3 to 4 weeks) Participant recruitment can be done by client Software inspection We can run the LIFT MACHINE test in conjunction with expert review to evaluate accessibility and usability issues Qualitative research Web blueprint package ; facilitating stakeholder and/or audience feedback with competitive benchmarking (4 to 6 weeks)

Inspection evaluations

Expert Review with scorecard and optional report (1 week)

Human performance testing

Top Task Tune-up (3 to 4 weeks)

Participant recruitment can be done by client

Software inspection

We can run the LIFT MACHINE test in conjunction with expert review to evaluate accessibility and usability issues

Qualitative research

Web blueprint package ; facilitating stakeholder and/or audience feedback with competitive benchmarking (4 to 6 weeks)

Expert Review A scorecard maps the user experience to a “value” ladder Prioritize: take it a step at a time , to rise above “discouraged”

A scorecard maps the user experience to a “value” ladder

Prioritize: take it a step at a time , to rise above “discouraged”

Climb the ladder… Awareness : Access to content Tactics : Speed and reliability of the framework must be high Information must be accessible to qualified users Adhere to guidelines for accessibility Marketing your site Satisfaction : Deliver an error-free experience to users – Does it work? Can I recover quickly? Tactics : Contact info for the Web team/Help should be visible in the expected location Broken links are fixed quickly Content is searchable Indexes work as expected

Awareness : Access to content

Tactics :

Speed and reliability of the framework must be high

Information must be accessible to qualified users

Adhere to guidelines for accessibility

Marketing your site

Satisfaction : Deliver an error-free experience to users – Does it work? Can I recover quickly?

Tactics :

Contact info for the Web team/Help should be visible in the expected location

Broken links are fixed quickly

Content is searchable

Indexes work as expected

Higher steps… Confidence : Is the site intuitive and learnable? Can I find it? Tactics : Navigation Link behavior Ease of use Readability Packaging Forms Trust: Do you deliver what you promise? Are you transparent? Tactics : Use a privacy policy Populate the “about us” programs on their pages Show content is current by adopting a review schedule Be authoritative, avoid duplication, use related links Graphic display guidelines

Confidence : Is the site intuitive and learnable? Can I find it?

Tactics :

Navigation

Link behavior

Ease of use

Readability

Packaging

Forms

Trust: Do you deliver what you promise? Are you transparent?

Tactics :

Use a privacy policy

Populate the “about us” programs on their pages

Show content is current by adopting a review schedule

Be authoritative, avoid duplication, use related links

Graphic display guidelines

We’ve completed over 200 Web Site evaluations using our custom scorecard to measure, usability, 508 compliance, content and policy rules.

Current ES Scorecard Custom Scorecard reviews 36 factors on usability against best practice standards Some items weighted: total score of 100 Providing valuable content is worth 46 points Helping people find stuff is worth 31 points Leveraging feedback is worth 23 points feedback finding content NOTE: The 36 factors could form the basis for future revisions to your editorial guidelines or publishing rules

Custom Scorecard reviews 36 factors on usability against best practice standards

Some items weighted: total score of 100

Providing valuable content is worth 46 points

Helping people find stuff is worth 31 points

Leveraging feedback is worth 23 points

Publishing Rules We’ve created documentation for standards and publishing rules to govern Web sites with the House Democratic Caucus and HouseNet at the U.S. House of Representatives and the AF Portal Intranet at the USAF.

Expert Review pros and cons Advantages The yardstick you use is best practice standards and evidence-based research Can pave the way for embracing standards across your organization “ Grades” are familiar Actionable report focused on problem spots You can track improvement over time You can compare score with competition Disadvantages Uniform understanding of terms and definitions needed Expert reviewer track record is critical to success You must avoid single point of failure: For two-sets of eyes use software inspection OR second reviewer Binary report means no shades of success Does not involve users Not possible to review features, tasks, functionality

Advantages

The yardstick you use is best practice standards and evidence-based research

Can pave the way for embracing standards across your organization

“ Grades” are familiar

Actionable report focused on problem spots

You can track improvement over time

You can compare score with competition

Disadvantages

Uniform understanding of terms and definitions needed

Expert reviewer track record is critical to success

You must avoid single point of failure: For two-sets of eyes use software inspection OR second reviewer

Binary report means no shades of success

Does not involve users

Not possible to review features, tasks, functionality

Why More Can be Less.... People judge their own satisfaction based on the peak moments and the end points... SOURCE: Barry Schwartz, author The Paradox of Choice, Why More is Less (2004) Too many choices = paralysis Imagined alternatives = regret Choice = escalation of expectations Today: no excuse for failure

Usability testing process Recruit participants (NLT 8) Create scenarios Conduct the test Bailey’s Usability Testing Environment Remote testing with Go to Meeting Analyze the results Prepare the test report

Recruit participants (NLT 8)

Create scenarios

Conduct the test

Bailey’s Usability Testing Environment

Remote testing with Go to Meeting

Analyze the results

Prepare the test report

Our favorite tool: UTE

Usability testing process Create scenarios (user tasks) Usually each person attempts to complete several tasks (up to 7) The scenarios are presented in random order Secure participants Representative of typical users Large enough number to represent typical users

Create scenarios (user tasks)

Usually each person attempts to complete several tasks (up to 7)

The scenarios are presented in random order

Secure participants

Representative of typical users

Large enough number to represent typical users

First time At least 95% of typical users will complete a specific task (“purchase a camera”) in less than three minutes At least 90% of users will find information on a specific topic (“risks related to taking aspirin”) within 30 seconds At least 95% of users will be able to find a particular report (“cancer mortality in the U.S.”) within 45 seconds Objectives

At least 95% of typical users will complete a specific task (“purchase a camera”) in less than three minutes

At least 90% of users will find information on a specific topic (“risks related to taking aspirin”) within 30 seconds

At least 95% of users will be able to find a particular report (“cancer mortality in the U.S.”) within 45 seconds

Compared with a Baseline The average time to complete a particular task (“make an airline reservation”) will be 10% faster when using the revised Web site than when using the original Web site At least 95% of typical users will complete a specific task (“purchase a camera”) in 10% less time than the previous test At least 95% of users will be able to find a particular report (“cancer mortality in the U.S.”) in 10% less time than the previous test Objectives

The average time to complete a particular task (“make an airline reservation”) will be 10% faster when using the revised Web site than when using the original Web site

At least 95% of typical users will complete a specific task (“purchase a camera”) in 10% less time than the previous test

At least 95% of users will be able to find a particular report (“cancer mortality in the U.S.”) in 10% less time than the previous test

Compared to the Competition The average time to complete a particular task (“make an airline reservation”) will be reliably faster when using our Web site than when using a competitors Web site At least 95% of typical users of our website will complete a specific task (“purchase a camera”) reliably faster than a competitors Web site At least 90% of users will find information on a specific topic (“risks related to taking aspirin”) in reliably faster time than a competitors Web site Objectives

The average time to complete a particular task (“make an airline reservation”) will be reliably faster when using our Web site than when using a competitors Web site

At least 95% of typical users of our website will complete a specific task (“purchase a camera”) reliably faster than a competitors Web site

At least 90% of users will find information on a specific topic (“risks related to taking aspirin”) in reliably faster time than a competitors Web site

Usability Test Participants Participants 40:05 10 years or more Advanced degree 60-75 Female P16 37:33 5 to 9 years High school 18-39 Male P15 21:25 10 years or more Advanced degree 40-59 Female P14 37:43 10 years or more Advanced degree 60-75 Male P13 54:29 10 years or more Some college 18-39 Male P12 18:55 10 years or more Bachelors degree 40-59 Male P11 45:27 10 years or more Some college 40-59 Female P10 40:26 5 to 9 years Bachelors degree 40-59 Female P9 36:28 10 years or more Bachelors degree 18-39 Male P8 49:51 10 years or more Bachelors degree 40-59 Female P7 38:18 5 to 9 years Bachelors degree 18-39 Male P6 36:47 5 to 9 years High school 40-59 Female P5 29:43 5 to 9 years Some college 40-59 Female P4 36:29 1 to 4 years Bachelors degree 60-75 Male P3 40:16 5 to 9 years Some college 40-59 Female P2 53:50 10 years or more Advanced degree 40-59 Male P1 Test Time Web Experience Education Age Gender Participant

Age and Education Gender Female: 50% Male: 50% Age 18 - 39: 25% 40 - 59: 56% 60 - 74: 19% Education High school: 13% Some college: 25% Bachelor’s degree: 37% Advanced degree: 25% Participants KEY QUESTION: Does the participant profile match your target audience?

Gender

Female: 50%

Male: 50%

Age

18 - 39: 25%

40 - 59: 56%

60 - 74: 19%

Education

High school: 13%

Some college: 25%

Bachelor’s degree: 37%

Advanced degree: 25%

We help you target common tasks users may find difficult and use task analysis studies to recommend improvements in usability.

Performance Results Between the old HHS site and the Redesign What Indicated Success 92% 41% Average success rate 88% 8% Your cousin is considering a career in medical research and asked you if HHS offers financial aid to undergraduate students. 94% 71% You want to know what the Fiscal Year 2001 budget for HHS was. 94% 13% You want to know what housing organizations are available to help assist the homeless in your area. 94% 73% You want to know what diabetes is and how you can prevent it. 88% 38% You want to find a nursing home for a relative. Final Prototype Baseline Test Success Rate Scenario Text

Sample Reporting Format Scenario: You want to know what diabetes is and how you can prevent it? Where would you get this information? ____Successful ____Unsuccessful ____Time -Diseases and Conditions -CDC Health Topics Notes/observations Task Completion and Time Pathways

Scenario: You want to know what diabetes is and how you can prevent it? Where would you get this information?

Task Analysis pros and cons Advantages Software automates measures of time, clicks and success Think-aloud method reveals cues on user preferences Good results possible with as few as 8 testers Results helps you focus on what matters most Can be repeated over time Disadvantages Must set up software test correctly to collect data Remote testing format hides visual cues Selecting no less than 8 testers critical Selection bias possible without good screening Deeper review of content and navigation missing One shot test limits impact

Advantages

Software automates measures of time, clicks and success

Think-aloud method reveals cues on user preferences

Good results possible with as few as 8 testers

Results helps you focus on what matters most

Can be repeated over time

Disadvantages

Must set up software test correctly to collect data

Remote testing format hides visual cues

Selecting no less than 8 testers critical

Selection bias possible without good screening

Deeper review of content and navigation missing

One shot test limits impact

3 Phases of Web Management

Draw the blueprint Endorsement Get it in writing from the organization's top brass: the visionary executive Set goals Baseline current activities and define the "problem" your Web site wants to tackle. Analyze audiences Talk to internal and external audiences and define your market. Benchmark Examine the competition and surface best practices, new ideas. Document requirements Develop list of high-level needs; validate with stakeholder team. Select navigation Lead stakeholder team to decide on primary and secondary navigation with meaningful labels. Brand it Pick a tag line and theme to match your value proposition and fit your market and objectives.

Endorsement

Get it in writing from the organization's top brass: the visionary executive

Set goals

Baseline current activities and define the "problem" your Web site wants to tackle.

Analyze audiences

Talk to internal and external audiences and define your market.

Benchmark

Examine the competition and surface best practices, new ideas.

Document requirements

Develop list of high-level needs; validate with stakeholder team.

Select navigation

Lead stakeholder team to decide on primary and secondary navigation with meaningful labels.

Brand it

Pick a tag line and theme to match your value proposition and fit your market and objectives.

Web blueprint package When? If you are at the start of a site redesign project If you want to avoid an expensive redesign and simply improve navigation If you need a competitive analysis What? Lead a card-sorting exercise to validate navigation Facilitate a session for stakeholder feedback Benchmark capabilities against top competitors

When?

If you are at the start of a site redesign project

If you want to avoid an expensive redesign and simply improve navigation

If you need a competitive analysis

What?

Lead a card-sorting exercise to validate navigation

Facilitate a session for stakeholder feedback

Benchmark capabilities against top competitors

Card-sorting definitions ...a user-centered design method for increasing a system’s findability. The process involves sorting a series of cards, each labeled with a piece of content or functionality into groups that make sense to users or participants Source: Card-sorting: a Definitive Guide by Donna Maurer and Todd Warfel “ ...can provide insight into users’ mental models, illuminating the way that they often tacitly group sort and label tasks and content within their own heads” Source: Information Architecture for the World Wide Web

...a user-centered design method for increasing a system’s findability. The process involves sorting a series of cards, each labeled with a piece of content or functionality into groups that make sense to users or participants

Source: Card-sorting: a Definitive Guide by Donna Maurer and Todd Warfel

“ ...can provide insight into users’ mental models, illuminating the way that they often tacitly group sort and label tasks and content within their own heads”

Source: Information Architecture for the World Wide Web

Card-sorting What is it? Method that generates an overall structure for your information, as well as suggestions for navigation, menus and possible taxonomies Benefits? Explore how users group items – low cost + low tech Develop structures that are logical to users Maximize probability of users finding information

What is it?

Method that generates an overall structure for your information, as well as suggestions for navigation, menus and possible taxonomies

Benefits?

Explore how users group items – low cost + low tech

Develop structures that are logical to users

Maximize probability of users finding information

How do you do it? Item names to be categorized printed on note cards, 1 item/ card Participants are asked to group cards in a way that makes sense to them; Participants may be asked to... assign cards to groups and label groups by giving each category names (open sort) sort cards into pre-defined categories (closed sort) Options: Use larger cards and ask whether they understood the term (yes, no or unsure) or how frequently the user uses info. Ask participants to rename cards or think of synonyms clarify taxonomy unmask jargon

Item names to be categorized printed on note cards, 1 item/ card

Participants are asked to group cards in a way that makes sense to them; Participants may be asked to...

assign cards to groups and label groups by giving each category names (open sort)

sort cards into pre-defined categories (closed sort)

Options:

Use larger cards and ask whether they understood the term (yes, no or unsure) or how frequently the user uses info.

Ask participants to rename cards or think of synonyms

clarify taxonomy

unmask jargon

Do users want to see information grouped by subject, process, business group or information type? How many potential main categories are there? (typically relates to navigation) What should these groups be called? Which terms/labels are logical and intuitive to users? How similar (or different) are the needs of user groups? Card-sorting answers questions

Do users want to see information grouped by subject, process, business group or information type?

How many potential main categories are there? (typically relates to navigation)

What should these groups be called?

Which terms/labels are logical and intuitive to users?

How similar (or different) are the needs of user groups?

Card-sorting pros and cons Advantages Easy and inexpensive Reliable User-centric Helps to develop categories that are logical to users Helps to identify items that need to be renamed Helps with terminology Disadvantages Does not consider users’ tasks Not an evaluation technique (like usability testing) Can be time-consuming Sometimes difficult to analyze – tools have limitations Limited number of participants

Advantages

Easy and inexpensive

Reliable

User-centric

Helps to develop categories that are logical to users

Helps to identify items that need to be renamed

Helps with terminology

Disadvantages

Does not consider users’ tasks

Not an evaluation technique (like usability testing)

Can be time-consuming

Sometimes difficult to analyze – tools have limitations

Limited number of participants

Facilitated feedback Organization feedback Where is the organization today? (Customers? Suppliers? Competitors? Investors and shareholders?) What are our distinct competencies? What are our vulnerabilities? Where could we easily be attacked? What could we improve easily? What do we feel proud and sorry (rant and rave) about? What are the issues, problems and complaints that we have? Stakeholder feedback How is our world changing? What are the new market conditions and forces? What are the new ways to win? What are others doing? What is threatening us or what are the most urgent things for us to do? Who do we want to be like? Sample

Organization feedback

Where is the organization today? (Customers? Suppliers? Competitors? Investors and shareholders?)

What are our distinct competencies?

What are our vulnerabilities? Where could we easily be attacked?

What could we improve easily?

What do we feel proud and sorry (rant and rave) about?

What are the issues, problems and complaints that we have?

Stakeholder feedback

How is our world changing?

What are the new market conditions and forces?

What are the new ways to win?

What are others doing?

What is threatening us or what are the most urgent things for us to do?

Who do we want to be like?

What are YOU ready for? Expert Review with scorecard with scorecard and report Top Task Tune-up with client recruiting participants with participant recruitment Web Blueprint Package

Expert Review

with scorecard

with scorecard and report

Top Task Tune-up

with client recruiting participants

with participant recruitment

Web Blueprint Package

Contact us Kathy McShea Owner & Founder Emerald Strategies, Inc. 202.543.2112 Web Management & Marketing ~ http:// www.emeraldstrategies.net Media Training ~ http:// www.prclinic.com Join my professional network ~ http:// www.linkedin.com/in/kathymcshea

Kathy McShea Owner & Founder Emerald Strategies, Inc. 202.543.2112 Web Management & Marketing ~ http:// www.emeraldstrategies.net Media Training ~ http:// www.prclinic.com Join my professional network ~ http:// www.linkedin.com/in/kathymcshea