Email
Favorite
Favorited ×
Download
Embed
Private Content
Copy and paste this code into your blog or website Copy Customize Close
We have emailed the verification/download link to "".
Login to your email and click the link to download the file directly.

To request the link at a different email address, update it here. Close
Validation messages. Success message. Fail message.

Check your bulk/spam folders if you can't find our mail.

Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

+ Follow

Beyond Usability Testing: Assessing the Usefulness of Your Design

by Dan Berlin on May 25, 2011

695 views

More…

Less

Statistics

Favorites: 3
Downloads: 0
Comments: 0
Embed Views: 0
Views on SlideShare: 695
Total Views: 695

Goal of usability testing is to find usability problems.
Transactional Applications
Transactional Applications
Transactional Applications
Content / Informational Sites
Content / Informational Sites
Content / Informational Sites
Gaming
Gaming
Gaming
Social Sites
Transactional Applications
Transactional Applications
Marketing / Persuasive
Marketing / Persuasive
Marketing / Persuasive
So when a client wants discount research to guide their UX strategy and wants to learn about the value and usefulness of a prototype or existing interface… what do we do? We turn to what we know – usability testing.
What strengths can we take from usability testing?How should the methodology evolve?What questions should we pose to elicit this information from participants?What is an acceptable prototype fidelity level?
Moderator and participant interact one-on-one (except for unmoderated) Three primary components:Pre-task questions: moderator interviews participant to clarify demographics, assess level of expertise, and gauge prior experienceTasks: participant proceeds through defined set of tasks with design artifact while “thinking aloud”. Moderator minimizes intrusion, identifies usability problems, and tracks metrics such as time on task and completion percentagePost-task questions: moderator asks participant about their experience with the tasks, level of satisfaction with the design, and points of confusion or ambiguityNo longer just determining demographics and level of experienceRather, need to set participant’s mindset for an evaluation of the design from a usefulness perspectiveBorrow from Ethnography, Contextual Inquiry, Interviewing and Laddering Daily task flow, current mechanisms Pain points Expectations, desires Ideal scenarios Other systemsEmphasis changes from contrived tasks to participant-directed tasks Tasks determined from pre-task and probing questions“Tell me how you would use this application in your daily routine. Okay, now please go do some of those on this prototype.”“You mentioned that you would need to determine your size – how would you do that?”Gather expectations at outset of the task “What would you expect to happen? What information would expect the page to contain?”Sets the basis for participant commentary (not that you should design exactly what they say, but it can inform ideas)This may not be a natural way to interact with the design, but we’re not testing usabilityFollow up questions emphasize expectations and value “How did completing this task compare with your expectations? Was it better or worse and why? What’s missing? What’s superfluous?” “How does it compare with other applications or systems?” “What other features or functions would you need when completing this task?”Participant directed to walk back through the task, responding to questions at key interaction pointsStrategic probing questions on salient components that interrupt or inform the task“What would make this more useful? How would you use this in your work/life?”“Does this table contain the columns that you need? Did the categories make sense?” Probe on areas that were not covered by the participant task“How would you get the detailed information about this product?"
Moderator and participant interact one-on-one (except for unmoderated) Three primary components:Pre-task questions: moderator interviews participant to clarify demographics, assess level of expertise, and gauge prior experienceTasks: participant proceeds through defined set of tasks with design artifact while “thinking aloud”. Moderator minimizes intrusion, identifies usability problems, and tracks metrics such as time on task and completion percentagePost-task questions: moderator asks participant about their experience with the tasks, level of satisfaction with the design, and points of confusion or ambiguityNo longer just determining demographics and level of experienceRather, need to set participant’s mindset for an evaluation of the design from a usefulness perspectiveBorrow from Ethnography, Contextual Inquiry, Interviewing and Laddering Daily task flow, current mechanisms Pain points Expectations, desires Ideal scenarios Other systemsEmphasis changes from contrived tasks to participant-directed tasks Tasks determined from pre-task and probing questions“Tell me how you would use this application in your daily routine. Okay, now please go do some of those on this prototype.”“You mentioned that you would need to determine your size – how would you do that?”Gather expectations at outset of the task “What would you expect to happen? What information would expect the page to contain?”Sets the basis for participant commentary (not that you should design exactly what they say, but it can inform ideas)This may not be a natural way to interact with the design, but we’re not testing usabilityFollow up questions emphasize expectations and value “How did completing this task compare with your expectations? Was it better or worse and why? What’s missing? What’s superfluous?” “How does it compare with other applications or systems?” “What other features or functions would you need when completing this task?”Participant directed to walk back through the task, responding to questions at key interaction pointsStrategic probing questions on salient components that interrupt or inform the task“What would make this more useful? How would you use this in your work/life?”“Does this table contain the columns that you need? Did the categories make sense?” Probe on areas that were not covered by the participant task“How would you get the detailed information about this product?"
Moderator and participant interact one-on-one (except for unmoderated) Three primary components:Pre-task questions: moderator interviews participant to clarify demographics, assess level of expertise, and gauge prior experienceTasks: participant proceeds through defined set of tasks with design artifact while “thinking aloud”. Moderator minimizes intrusion, identifies usability problems, and tracks metrics such as time on task and completion percentagePost-task questions: moderator asks participant about their experience with the tasks, level of satisfaction with the design, and points of confusion or ambiguityNo longer just determining demographics and level of experienceRather, need to set participant’s mindset for an evaluation of the design from a usefulness perspectiveBorrow from Ethnography, Contextual Inquiry, Interviewing and Laddering Daily task flow, current mechanisms Pain points Expectations, desires Ideal scenarios Other systemsEmphasis changes from contrived tasks to participant-directed tasks Tasks determined from pre-task and probing questions“Tell me how you would use this application in your daily routine. Okay, now please go do some of those on this prototype.”“You mentioned that you would need to determine your size – how would you do that?”Gather expectations at outset of the task “What would you expect to happen? What information would expect the page to contain?”Sets the basis for participant commentary (not that you should design exactly what they say, but it can inform ideas)This may not be a natural way to interact with the design, but we’re not testing usabilityFollow up questions emphasize expectations and value “How did completing this task compare with your expectations? Was it better or worse and why? What’s missing? What’s superfluous?” “How does it compare with other applications or systems?” “What other features or functions would you need when completing this task?”Participant directed to walk back through the task, responding to questions at key interaction pointsStrategic probing questions on salient components that interrupt or inform the task“What would make this more useful? How would you use this in your work/life?”“Does this table contain the columns that you need? Did the categories make sense?” Probe on areas that were not covered by the participant task“How would you get the detailed information about this product?"
Moderator and participant interact one-on-one (except for unmoderated) Three primary components:Pre-task questions: moderator interviews participant to clarify demographics, assess level of expertise, and gauge prior experienceTasks: participant proceeds through defined set of tasks with design artifact while “thinking aloud”. Moderator minimizes intrusion, identifies usability problems, and tracks metrics such as time on task and completion percentagePost-task questions: moderator asks participant about their experience with the tasks, level of satisfaction with the design, and points of confusion or ambiguityNo longer just determining demographics and level of experienceRather, need to set participant’s mindset for an evaluation of the design from a usefulness perspectiveBorrow from Ethnography, Contextual Inquiry, Interviewing and Laddering Daily task flow, current mechanisms Pain points Expectations, desires Ideal scenarios Other systemsEmphasis changes from contrived tasks to participant-directed tasks Tasks determined from pre-task and probing questions“Tell me how you would use this application in your daily routine. Okay, now please go do some of those on this prototype.”“You mentioned that you would need to determine your size – how would you do that?”Gather expectations at outset of the task “What would you expect to happen? What information would expect the page to contain?”Sets the basis for participant commentary (not that you should design exactly what they say, but it can inform ideas)This may not be a natural way to interact with the design, but we’re not testing usabilityFollow up questions emphasize expectations and value “How did completing this task compare with your expectations? Was it better or worse and why? What’s missing? What’s superfluous?” “How does it compare with other applications or systems?” “What other features or functions would you need when completing this task?”Participant directed to walk back through the task, responding to questions at key interaction pointsStrategic probing questions on salient components that interrupt or inform the task“What would make this more useful? How would you use this in your work/life?”“Does this table contain the columns that you need? Did the categories make sense?” Probe on areas that were not covered by the participant task“How would you get the detailed information about this product?"
Usability is still important!But, recognize when you have different goals and adjust accordinglyBe aware of nuances and differences between usability and usefulnessSet clear expectations and goals with project teamName it “usefulness study” rather than a “usability test”?Drawbacks: may need deeper prototypes, not getting at usability problems, Danger: asking the participant to help design it is easy to fall into, but not good

3 Favorites

Iron at Iron 8 months ago
yuhsiu 8 months ago
EBI Interfaces at EBI Interfaces Tags differentiation usefulness usability 11 months ago

Beyond Usability Testing: Assessing the Usefulness of Your Design — Presentation Transcript

Beyond Usability Testing: Assessing the Usefulness of Your Design
UPA Boston Mini-Conference 2011
Prepared by:
Michael Hawley – Chief Design Officer
Daniel Berlin – Experience Research Director
May 25, 2011
Have You Run Usability Tests?
Usability Testing
Participants attempt to complete a set of defined tasks.
Researchers learn what to improve by observing and interpreting think-aloud.
Virzi, R.A., Refining the Test Phase of Usability Evaluation: How Many Subjects is Enough? Human Factors, 1992. 34(4): p. 457-468.
Trend
Business sponsors turn to us as UX professionals with questions that are not about usability problems.
Rather, their questions are about overall user experience strategy, value and usefulness.
Usability:
Find interruptions in workflow that prevent users from performing tasks quickly and efficiently.
Usefulness:
Inform a re-structure of the application to best align with workflow.
Determine where to position productivity tips and help buttons within the application for best utilization.
Find optimal level of personalization and customization that users would take advantage of.
Usability:
Assess effectiveness of navigation system in guiding users to desired pages.
Evaluate descriptiveness and clarity of links.
Gauge ability of page layouts to orient users to relevant content.
Usefulness:
Identify content that is missing which will help overcome objections or answer critical questions?
Understand how branded labels and content themes contribute to the overall experience or detract from it.
Determine level/types of promotions and interstitials that are acceptable to users.
Understand how different audience personas prefer to consume information for the particular domain.
Usability:
Determine optimal level of difficulty to encourage advancement to multiple levels of the game.
Assess discoverability of game features and controls.
Usefulness:
Find the optimal rate of point accumulation and alignment with prize levels.
Understand best use of social media within or around the game.
Determine the threshold for ads, interstitials and registration for game play.
Usability:
Assess if users can figure out how to add a comment, share content, or use a tagging mechanism to find what they are looking for.
Usefulness:
Determine the most compelling and appealing topics or categories for conversation.
Understand the level of involvement the sponsoring company should have in the social experience, if any.
Balance branded or non-branded experience for optimal trust of the site.
Determine the elements or attributes that should allow comment and review.
Usability:
Find any confusion points or interruptions that prevent users from registering.
Find misleading or ambiguous terminology.
Usefulness:
Determine the most persuasive elements that will compel the target audience to register.
Understand a design’s impact a user’s perception of the brand.
Position the offering and messaging against the company’s competitors.
Determine missing content that can help target audience make an informed decision about the product.
Sound Familiar?
Natural Reaction
Turn to what we know: Usability Testing
(one-on-one interviews, design artifact, and tasks)
Are You Forgetting Contextual Inquiry and Foundational Research?
Discovery research and needs analysis is valid, but:
- Time and budget for separate research is not always an option
- Many participants need design artifacts to elicit appropriate reaction and commentary
Our Goal
Leverage the strengths of usability testing but adjust our approach when objectives differ from finding usability problems.
Three Components
Three Components
Three Components
Three Components
Example: Usability
Goal:
Find usability problems
Pre-Task Questions:
Audience segmentation
Tasks:
Defined
Post-Task Questions:
Baseline satisfaction
Example: Usefulness
Goal:
Role of personalization
Pre-Task Questions:
Common tasks, pain points
Tasks:
Emphasis on participant direction
Post-Task Questions:
Comparison of expectation
Summary
Foundational research is still important.
Usability testing is still important.
However, recognize when you have different goals and adapt the research method as necessary.
Additional Information
Complete Presentation Slides
- http://www.slideshare.net/hawleymichael
- http://www.slideshare.net/banderlin
Contact Information
Michael Hawley
mhawley@madpow.net
@hawleymichael
Dan Berlin
dberlin@madpow.net
@banderlin

Beyond Usability Testing: Assessing the Usefulness of Your Design

by Dan Berlin on May 25, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

Beyond Usability Testing: Assessing the Usefulness of Your Design — Presentation Transcript