SlideShare a Scribd company logo
1 of 32
Beyond Usability Testing: Assessing the Usefulness
of Your Design
UPA Boston Mini-Conference 2011




Prepared by:
Michael Hawley – Chief Design Officer
Daniel Berlin – Experience Research Director


May 25, 2011
Have You Run Usability Tests?
Usability Testing

Participants attempt to
complete a set of defined tasks.


Researchers learn what to
improve by observing and
interpreting think-aloud.
Virzi, R.A., Refining the Test Phase of Usability Evaluation: How Many Subjects is Enough? Human Factors, 1992. 34(4): p. 457-468.
Trend

Business sponsors turn to us as UX professionals with
questions that are not about usability problems.


Rather, their questions are about overall user experience
strategy, value and usefulness.
Usability:
Find interruptions in workflow that prevent users from
performing tasks quickly and efficiently.
Usefulness:

Inform a re-structure of the application to best align with
workflow.

Determine where to position productivity tips and help
buttons within the application for best utilization.

Find optimal level of personalization and customization
that users would take advantage of.
Usability:

Assess effectiveness of navigation system in guiding users
to desired pages.

Evaluate descriptiveness and clarity of links.

Gauge ability of page layouts to orient users to relevant
content.
Usefulness:

Identify content that is missing which will help overcome
objections or answer critical questions?

Understand how branded labels and content themes
contribute to the overall experience or detract from it.

Determine level/types of promotions and interstitials that
are acceptable to users.

Understand how different audience personas prefer to
consume information for the particular domain.
Usability:

Determine optimal level of difficulty to encourage
advancement to multiple levels of the game.

Assess discoverability of game features and controls.
Usefulness:

Find the optimal rate of point accumulation and alignment
with prize levels.

Understand best use of social media within or around the
game.

Determine the threshold for ads, interstitials and
registration for game play.
Usability:
Assess if users can figure out how to add a comment,
share content, or use a tagging mechanism to find what
they are looking for.
Usefulness:

Determine the most compelling and appealing topics or
categories for conversation.

Understand the level of involvement the sponsoring
company should have in the social experience, if any.

Balance branded or non-branded experience for optimal
trust of the site.

Determine the elements or attributes that should allow
comment and review.
Usability:

Find any confusion points or interruptions that prevent
users from registering.

Find misleading or ambiguous terminology.
Usefulness:

Determine the most persuasive elements that will compel
the target audience to register.

Understand a design’s impact a user’s perception of the
brand.

Position the offering and messaging against the company’s
competitors.

Determine missing content that can help target audience
make an informed decision about the product.
Sound Familiar?
Natural Reaction

Turn to what we know:
Usability Testing
(one-on-one interviews, design
artifact, and tasks)
Are You Forgetting Contextual Inquiry and Foundational Research?

Discovery research and needs analysis is valid, but:
•  Time and budget for separate research is not always an
   option
•  Many participants need design artifacts to elicit
   appropriate reaction and commentary
Our Goal

Leverage the strengths of usability testing but adjust our
approach when objectives differ from finding usability
problems.
Three Components

Phase
Pre-Task
Questions




Tasks




Post-Task
Questions
Three Components

Phase        Usability                      Usefulness
Pre-Task     Demographics, level of         Daily task flow, pain points,
Questions    expertise, prior experience.   expectations, desires,
             Goal: validating groups        scenarios.
             and classify results.          Goal: set mindset for
                                            usefulness evaluation of the
                                            design

Tasks




Post-Task
Questions
Three Components

Phase        Usability                      Usefulness
Pre-Task     Demographics, level of         Daily task flow, pain points,
Questions    expertise, prior experience.   expectations, desires,
             Goal: validating groups        scenarios.
             and classify results.          Goal: set mindset for
                                            usefulness evaluation of the
                                            design

Tasks        Pre-defined tasks, minimal     Emphasis on participant-
             moderator intrusion,           directed tasks
             Goal: find usability           Goal: understand how proposed
             problems, measure time         design aligns with user needs
             on task and completion
             percentage.

Post-Task
Questions
Three Components

Phase        Usability                      Usefulness
Pre-Task     Demographics, level of         Daily task flow, pain points,
Questions    expertise, prior experience.   expectations, desires,
             Goal: validating groups        scenarios.
             and classify results.          Goal: set mindset for
                                            usefulness evaluation of the
                                            design

Tasks        Pre-defined tasks, minimal     Emphasis on participant-
             moderator intrusion,           directed tasks
             Goal: find usability           Goal: understand how proposed
             problems, measure time         design aligns with user needs
             on task and completion
             percentage.

Post-Task    Level of satisfaction with     Comparison with expectations
Questions    the design, and points of      and value. Task retrospectives.
             confusion or ambiguity.        Goal: discuss opportunities for
             Goal: measure usability        improvement.
Example: Usability

                     Goal:
                     Find usability problems

                     Pre-Task Questions:
                     Audience segmentation

                     Tasks:
                     Defined

                     Post-Task Questions:
                     Baseline satisfaction
Example: Usefulness

                      Goal:
                      Role of personalization

                      Pre-Task Questions:
                      Common tasks, pain
                      points

                      Tasks:
                      Emphasis on participant
                      direction

                      Post-Task Questions:
                      Comparison of
                      expectation
Summary

Foundational research is still important.
Usability testing is still important.


However, recognize when you have different goals and
adapt the research method as necessary.
Additional Information

Complete Presentation Slides
• http://www.slideshare.net/hawleymichael
• http://www.slideshare.net/banderlin


Contact Information

                 Michael Hawley             Dan Berlin
                 mhawley@madpow.net         dberlin@madpow.net
                 @hawleymichael             @banderlin

More Related Content

What's hot

What's hot (20)

User Experience Design + Agile: The Good, The Bad, and the Ugly
User Experience Design + Agile: The Good, The Bad, and the UglyUser Experience Design + Agile: The Good, The Bad, and the Ugly
User Experience Design + Agile: The Good, The Bad, and the Ugly
 
Research Ready to Build: Compelling Artefacts that Speak Your Agile Team's La...
Research Ready to Build: Compelling Artefacts that Speak Your Agile Team's La...Research Ready to Build: Compelling Artefacts that Speak Your Agile Team's La...
Research Ready to Build: Compelling Artefacts that Speak Your Agile Team's La...
 
UXPA 2021: How do you know your users feel satisfied
UXPA 2021: How do you know your users feel satisfied   UXPA 2021: How do you know your users feel satisfied
UXPA 2021: How do you know your users feel satisfied
 
Tackle the Problem with Design Thinking - GDSC UAD
Tackle the Problem with Design Thinking - GDSC UADTackle the Problem with Design Thinking - GDSC UAD
Tackle the Problem with Design Thinking - GDSC UAD
 
UXPA DC Redux 2013 Notetaker Perspective 10-25-2013.ppt
UXPA DC Redux 2013 Notetaker Perspective 10-25-2013.pptUXPA DC Redux 2013 Notetaker Perspective 10-25-2013.ppt
UXPA DC Redux 2013 Notetaker Perspective 10-25-2013.ppt
 
Sample - Design Portfolio Walkthrough
Sample - Design Portfolio WalkthroughSample - Design Portfolio Walkthrough
Sample - Design Portfolio Walkthrough
 
The Secret Sauce for Effective Usability Testing
The Secret Sauce for Effective Usability Testing The Secret Sauce for Effective Usability Testing
The Secret Sauce for Effective Usability Testing
 
Website Usability
Website UsabilityWebsite Usability
Website Usability
 
UX Research - Overview
UX Research - OverviewUX Research - Overview
UX Research - Overview
 
Effective ui
Effective uiEffective ui
Effective ui
 
Embracing OOUX for Better Projects and Happier Teams
Embracing OOUX for Better Projects and Happier TeamsEmbracing OOUX for Better Projects and Happier Teams
Embracing OOUX for Better Projects and Happier Teams
 
Preference and Desirability Testing: Measuring Emotional Response to Guide De...
Preference and Desirability Testing: Measuring Emotional Response to Guide De...Preference and Desirability Testing: Measuring Emotional Response to Guide De...
Preference and Desirability Testing: Measuring Emotional Response to Guide De...
 
Proposal Template To Increase Traffic To A Website PowerPoint Presentation Sl...
Proposal Template To Increase Traffic To A Website PowerPoint Presentation Sl...Proposal Template To Increase Traffic To A Website PowerPoint Presentation Sl...
Proposal Template To Increase Traffic To A Website PowerPoint Presentation Sl...
 
UX Resaecrh
UX ResaecrhUX Resaecrh
UX Resaecrh
 
UXPA International 2013 The Note-Taker's Perspective
UXPA International 2013 The Note-Taker's Perspective UXPA International 2013 The Note-Taker's Perspective
UXPA International 2013 The Note-Taker's Perspective
 
Lean User Research - UXPA 2013 Workshop
Lean User Research - UXPA 2013 WorkshopLean User Research - UXPA 2013 Workshop
Lean User Research - UXPA 2013 Workshop
 
Richard Marsh, Enterprising User Experience - Flex and the city
Richard Marsh, Enterprising User Experience - Flex and the cityRichard Marsh, Enterprising User Experience - Flex and the city
Richard Marsh, Enterprising User Experience - Flex and the city
 
A Systematic Approach to Design Critique
A Systematic Approach to Design CritiqueA Systematic Approach to Design Critique
A Systematic Approach to Design Critique
 
How to effectively implement different online research methods - UXPA 2015 - ...
How to effectively implement different online research methods - UXPA 2015 - ...How to effectively implement different online research methods - UXPA 2015 - ...
How to effectively implement different online research methods - UXPA 2015 - ...
 
Great Agile in a UX World
Great Agile in a UX WorldGreat Agile in a UX World
Great Agile in a UX World
 

Viewers also liked

Beyond Just Usability: Desirability and Usefulness Testing
Beyond Just Usability: Desirability and Usefulness TestingBeyond Just Usability: Desirability and Usefulness Testing
Beyond Just Usability: Desirability and Usefulness Testing
Susan Mercer
 
Desirability Testing: Analyzing Emotional Response to a Design
Desirability Testing: Analyzing Emotional Response to a DesignDesirability Testing: Analyzing Emotional Response to a Design
Desirability Testing: Analyzing Emotional Response to a Design
Megan Grocki
 
Validity, reliability & practicality
Validity, reliability & practicalityValidity, reliability & practicality
Validity, reliability & practicality
Samcruz5
 
Presentation Validity & Reliability
Presentation Validity & ReliabilityPresentation Validity & Reliability
Presentation Validity & Reliability
songoten77
 

Viewers also liked (12)

Beyond Just Usability: Desirability and Usefulness Testing
Beyond Just Usability: Desirability and Usefulness TestingBeyond Just Usability: Desirability and Usefulness Testing
Beyond Just Usability: Desirability and Usefulness Testing
 
Teaching and testing
Teaching and testingTeaching and testing
Teaching and testing
 
Diving Deep: Uncovering Hidden Insights Through User Interviews
Diving Deep: Uncovering Hidden Insights Through User InterviewsDiving Deep: Uncovering Hidden Insights Through User Interviews
Diving Deep: Uncovering Hidden Insights Through User Interviews
 
Validity, reliablility, washback
Validity, reliablility, washbackValidity, reliablility, washback
Validity, reliablility, washback
 
Test Usefulness
Test UsefulnessTest Usefulness
Test Usefulness
 
Using the Livescribe Pen for Usability Studies
Using the Livescribe Pen for Usability StudiesUsing the Livescribe Pen for Usability Studies
Using the Livescribe Pen for Usability Studies
 
Desirability Testing: Analyzing Emotional Response to a Design
Desirability Testing: Analyzing Emotional Response to a DesignDesirability Testing: Analyzing Emotional Response to a Design
Desirability Testing: Analyzing Emotional Response to a Design
 
How emotions work: Preference and action
How emotions work: Preference and actionHow emotions work: Preference and action
How emotions work: Preference and action
 
Principles of Software testing
Principles of Software testingPrinciples of Software testing
Principles of Software testing
 
Principles of Language Assessment
Principles of Language AssessmentPrinciples of Language Assessment
Principles of Language Assessment
 
Validity, reliability & practicality
Validity, reliability & practicalityValidity, reliability & practicality
Validity, reliability & practicality
 
Presentation Validity & Reliability
Presentation Validity & ReliabilityPresentation Validity & Reliability
Presentation Validity & Reliability
 

Similar to Beyond Usability Testing: Assessing the Usefulness of Your Design

Tale of two customers: Addressing the struggle between flexibility and usabil...
Tale of two customers: Addressing the struggle between flexibility and usabil...Tale of two customers: Addressing the struggle between flexibility and usabil...
Tale of two customers: Addressing the struggle between flexibility and usabil...
Eileen O'Brien
 
LavaCon 2011 UCD Presentation
LavaCon 2011 UCD PresentationLavaCon 2011 UCD Presentation
LavaCon 2011 UCD Presentation
jtkachuk
 
Usability Testing - Sivaprasath Selvaraj
Usability Testing - Sivaprasath SelvarajUsability Testing - Sivaprasath Selvaraj
Usability Testing - Sivaprasath Selvaraj
Sivaprasath Selvaraj
 

Similar to Beyond Usability Testing: Assessing the Usefulness of Your Design (20)

U Xmagic Agile Presentation
U Xmagic Agile PresentationU Xmagic Agile Presentation
U Xmagic Agile Presentation
 
Understanding Goals and Requirements
Understanding Goals and RequirementsUnderstanding Goals and Requirements
Understanding Goals and Requirements
 
Usability Testing for Qualitative Researchers - QRCA NYC Chapter event
Usability Testing for Qualitative Researchers - QRCA NYC Chapter eventUsability Testing for Qualitative Researchers - QRCA NYC Chapter event
Usability Testing for Qualitative Researchers - QRCA NYC Chapter event
 
Usability methods to improve EMRs
Usability methods to improve EMRsUsability methods to improve EMRs
Usability methods to improve EMRs
 
Tale of two customers: Addressing the struggle between flexibility and usabil...
Tale of two customers: Addressing the struggle between flexibility and usabil...Tale of two customers: Addressing the struggle between flexibility and usabil...
Tale of two customers: Addressing the struggle between flexibility and usabil...
 
Formative Evaluation for Educational Product Development
Formative Evaluation for Educational Product DevelopmentFormative Evaluation for Educational Product Development
Formative Evaluation for Educational Product Development
 
11 - Evaluating Framework in Interaction Design_new.pptx
11 - Evaluating Framework in Interaction Design_new.pptx11 - Evaluating Framework in Interaction Design_new.pptx
11 - Evaluating Framework in Interaction Design_new.pptx
 
Dual Track Agile & Data Driven Design
Dual Track Agile & Data Driven DesignDual Track Agile & Data Driven Design
Dual Track Agile & Data Driven Design
 
Choosing the Right UX Method
Choosing the Right UX MethodChoosing the Right UX Method
Choosing the Right UX Method
 
User experience & design user centered analysis
User experience & design user centered analysisUser experience & design user centered analysis
User experience & design user centered analysis
 
LavaCon 2011 UCD Presentation
LavaCon 2011 UCD PresentationLavaCon 2011 UCD Presentation
LavaCon 2011 UCD Presentation
 
Handout #5 - QIAMay4
Handout #5 - QIAMay4 Handout #5 - QIAMay4
Handout #5 - QIAMay4
 
Using Your Logic Model to Focus Evaluation
Using Your Logic Model to Focus EvaluationUsing Your Logic Model to Focus Evaluation
Using Your Logic Model to Focus Evaluation
 
Usability Testing - Sivaprasath Selvaraj
Usability Testing - Sivaprasath SelvarajUsability Testing - Sivaprasath Selvaraj
Usability Testing - Sivaprasath Selvaraj
 
Jan Moons at WUD16
Jan Moons at WUD16Jan Moons at WUD16
Jan Moons at WUD16
 
Things you should know before you build your site
Things you should know before you build your siteThings you should know before you build your site
Things you should know before you build your site
 
Design process in English
Design process in English Design process in English
Design process in English
 
MSF (Microsoft Solution Framework)
MSF (Microsoft Solution Framework)MSF (Microsoft Solution Framework)
MSF (Microsoft Solution Framework)
 
Lean UX in the Enterprise: A Government Case Study
Lean UX in the Enterprise: A Government Case StudyLean UX in the Enterprise: A Government Case Study
Lean UX in the Enterprise: A Government Case Study
 
UX Design Process | Sample Proposal
UX Design Process | Sample Proposal UX Design Process | Sample Proposal
UX Design Process | Sample Proposal
 

More from Dan Berlin

Beyond Eye Tracking: Bringing Biometrics to Usability Research
Beyond Eye Tracking: Bringing Biometrics to Usability ResearchBeyond Eye Tracking: Bringing Biometrics to Usability Research
Beyond Eye Tracking: Bringing Biometrics to Usability Research
Dan Berlin
 
Usability Testing Medical Devices
Usability Testing Medical DevicesUsability Testing Medical Devices
Usability Testing Medical Devices
Dan Berlin
 
Social Media Content Strategy
Social Media Content StrategySocial Media Content Strategy
Social Media Content Strategy
Dan Berlin
 
Applying Design Psychology
Applying Design PsychologyApplying Design Psychology
Applying Design Psychology
Dan Berlin
 
Eye Tracking for Usability
Eye Tracking for UsabilityEye Tracking for Usability
Eye Tracking for Usability
Dan Berlin
 

More from Dan Berlin (16)

UXPA Boston 2024 Maximize the Client Consultant Relationship.pdf
UXPA Boston 2024 Maximize the Client Consultant Relationship.pdfUXPA Boston 2024 Maximize the Client Consultant Relationship.pdf
UXPA Boston 2024 Maximize the Client Consultant Relationship.pdf
 
Your "Psychologist Voice": Leveraging Voice Mindfulness for UX Research
Your "Psychologist Voice": Leveraging Voice Mindfulness for UX ResearchYour "Psychologist Voice": Leveraging Voice Mindfulness for UX Research
Your "Psychologist Voice": Leveraging Voice Mindfulness for UX Research
 
Your "Psychologist Voice": Leveraging Voice Mindfulness for UX Research
Your "Psychologist Voice": Leveraging Voice Mindfulness for UX ResearchYour "Psychologist Voice": Leveraging Voice Mindfulness for UX Research
Your "Psychologist Voice": Leveraging Voice Mindfulness for UX Research
 
Biometrics in UX Research: The Next Big Step
Biometrics in UX Research: The Next Big StepBiometrics in UX Research: The Next Big Step
Biometrics in UX Research: The Next Big Step
 
User Experience (UX) Research in Healthcare
User Experience (UX) Research in HealthcareUser Experience (UX) Research in Healthcare
User Experience (UX) Research in Healthcare
 
Experience Research Best Practices
Experience Research Best PracticesExperience Research Best Practices
Experience Research Best Practices
 
Focus Groups are a Joke
Focus Groups are a JokeFocus Groups are a Joke
Focus Groups are a Joke
 
Beyond Eye Tracking: Bringing Biometrics to Usability Research
Beyond Eye Tracking: Bringing Biometrics to Usability ResearchBeyond Eye Tracking: Bringing Biometrics to Usability Research
Beyond Eye Tracking: Bringing Biometrics to Usability Research
 
Visual Principles of Experience Design: Blending Art and Science
Visual Principles of Experience Design: Blending Art and ScienceVisual Principles of Experience Design: Blending Art and Science
Visual Principles of Experience Design: Blending Art and Science
 
Psychophysiology and Eyetracking in User Experience
Psychophysiology and Eyetracking in User ExperiencePsychophysiology and Eyetracking in User Experience
Psychophysiology and Eyetracking in User Experience
 
Usability Testing Medical Devices
Usability Testing Medical DevicesUsability Testing Medical Devices
Usability Testing Medical Devices
 
Social Media Content Strategy
Social Media Content StrategySocial Media Content Strategy
Social Media Content Strategy
 
Applying Design Psychology
Applying Design PsychologyApplying Design Psychology
Applying Design Psychology
 
Intro to UX
Intro to UXIntro to UX
Intro to UX
 
Eye tracking and neuromarketing primer
Eye tracking and neuromarketing primerEye tracking and neuromarketing primer
Eye tracking and neuromarketing primer
 
Eye Tracking for Usability
Eye Tracking for UsabilityEye Tracking for Usability
Eye Tracking for Usability
 

Recently uploaded

Presentation on 3D Printing.pptx presentation
Presentation on 3D Printing.pptx presentationPresentation on 3D Printing.pptx presentation
Presentation on 3D Printing.pptx presentation
ajroy0196
 
Knowing, Understanding and Planning Cities- Role and Relevance Physical Plan...
Knowing, Understanding and Planning Cities- Role and Relevance  Physical Plan...Knowing, Understanding and Planning Cities- Role and Relevance  Physical Plan...
Knowing, Understanding and Planning Cities- Role and Relevance Physical Plan...
JIT KUMAR GUPTA
 
Heuristic Evaluation of System & Application
Heuristic Evaluation of System & ApplicationHeuristic Evaluation of System & Application
Heuristic Evaluation of System & Application
Jaime Brown
 
一比一原版格林威治大学毕业证成绩单如何办理
一比一原版格林威治大学毕业证成绩单如何办理一比一原版格林威治大学毕业证成绩单如何办理
一比一原版格林威治大学毕业证成绩单如何办理
cyebo
 

Recently uploaded (20)

spColumn-Manual design column by spcolumn software.pdf
spColumn-Manual design column by spcolumn software.pdfspColumn-Manual design column by spcolumn software.pdf
spColumn-Manual design column by spcolumn software.pdf
 
iF_Design_Trend_Report_twentytwenrythree
iF_Design_Trend_Report_twentytwenrythreeiF_Design_Trend_Report_twentytwenrythree
iF_Design_Trend_Report_twentytwenrythree
 
Latest Trends in Home and Building Design
Latest Trends in Home and Building DesignLatest Trends in Home and Building Design
Latest Trends in Home and Building Design
 
Recycled Modular Low Cost Construction .pdf
Recycled Modular Low Cost Construction .pdfRecycled Modular Low Cost Construction .pdf
Recycled Modular Low Cost Construction .pdf
 
Presentation on 3D Printing.pptx presentation
Presentation on 3D Printing.pptx presentationPresentation on 3D Printing.pptx presentation
Presentation on 3D Printing.pptx presentation
 
Heidi Livengood's Professional CADD Portfolio
Heidi Livengood's Professional CADD PortfolioHeidi Livengood's Professional CADD Portfolio
Heidi Livengood's Professional CADD Portfolio
 
CADD 141 - Puzzle Cube Project - Product Photos
CADD 141 - Puzzle Cube Project - Product PhotosCADD 141 - Puzzle Cube Project - Product Photos
CADD 141 - Puzzle Cube Project - Product Photos
 
Bit Dhrumi shah Graphic Designer portfolio
Bit Dhrumi shah Graphic Designer portfolioBit Dhrumi shah Graphic Designer portfolio
Bit Dhrumi shah Graphic Designer portfolio
 
Week of Action 2022_EIT Climate-KIC_Headers
Week of Action 2022_EIT Climate-KIC_HeadersWeek of Action 2022_EIT Climate-KIC_Headers
Week of Action 2022_EIT Climate-KIC_Headers
 
Spring 2024 wkrm_Enhancing Campus Mobility.pdf
Spring 2024 wkrm_Enhancing Campus Mobility.pdfSpring 2024 wkrm_Enhancing Campus Mobility.pdf
Spring 2024 wkrm_Enhancing Campus Mobility.pdf
 
BIT Khushi gandhi project.pdf graphic design
BIT Khushi gandhi project.pdf graphic designBIT Khushi gandhi project.pdf graphic design
BIT Khushi gandhi project.pdf graphic design
 
Knowing, Understanding and Planning Cities- Role and Relevance Physical Plan...
Knowing, Understanding and Planning Cities- Role and Relevance  Physical Plan...Knowing, Understanding and Planning Cities- Role and Relevance  Physical Plan...
Knowing, Understanding and Planning Cities- Role and Relevance Physical Plan...
 
Heuristic Evaluation of System & Application
Heuristic Evaluation of System & ApplicationHeuristic Evaluation of System & Application
Heuristic Evaluation of System & Application
 
Claire's designing portfolio presentation
Claire's designing portfolio presentationClaire's designing portfolio presentation
Claire's designing portfolio presentation
 
Webhost NVME Cloud VPS Hosting1234455678
Webhost NVME Cloud VPS Hosting1234455678Webhost NVME Cloud VPS Hosting1234455678
Webhost NVME Cloud VPS Hosting1234455678
 
一比一原版格林威治大学毕业证成绩单如何办理
一比一原版格林威治大学毕业证成绩单如何办理一比一原版格林威治大学毕业证成绩单如何办理
一比一原版格林威治大学毕业证成绩单如何办理
 
The Impact of Artificial Intelligence on Modern Healthcare.pptx
The Impact of Artificial Intelligence on Modern Healthcare.pptxThe Impact of Artificial Intelligence on Modern Healthcare.pptx
The Impact of Artificial Intelligence on Modern Healthcare.pptx
 
Abdulaziz Tariq Abdulaziz Mustafa CV 2024
Abdulaziz Tariq Abdulaziz Mustafa CV 2024Abdulaziz Tariq Abdulaziz Mustafa CV 2024
Abdulaziz Tariq Abdulaziz Mustafa CV 2024
 
NO1 Popular kala jadu karne wale ka contact number kala jadu karne wale baba ...
NO1 Popular kala jadu karne wale ka contact number kala jadu karne wale baba ...NO1 Popular kala jadu karne wale ka contact number kala jadu karne wale baba ...
NO1 Popular kala jadu karne wale ka contact number kala jadu karne wale baba ...
 
Dos And Dont's Of Logo Design For 2024..
Dos And Dont's Of Logo Design For 2024..Dos And Dont's Of Logo Design For 2024..
Dos And Dont's Of Logo Design For 2024..
 

Beyond Usability Testing: Assessing the Usefulness of Your Design

  • 1. Beyond Usability Testing: Assessing the Usefulness of Your Design UPA Boston Mini-Conference 2011 Prepared by: Michael Hawley – Chief Design Officer Daniel Berlin – Experience Research Director May 25, 2011
  • 2. Have You Run Usability Tests?
  • 3. Usability Testing Participants attempt to complete a set of defined tasks. Researchers learn what to improve by observing and interpreting think-aloud.
  • 4. Virzi, R.A., Refining the Test Phase of Usability Evaluation: How Many Subjects is Enough? Human Factors, 1992. 34(4): p. 457-468.
  • 5. Trend Business sponsors turn to us as UX professionals with questions that are not about usability problems. Rather, their questions are about overall user experience strategy, value and usefulness.
  • 6.
  • 7. Usability: Find interruptions in workflow that prevent users from performing tasks quickly and efficiently.
  • 8. Usefulness: Inform a re-structure of the application to best align with workflow. Determine where to position productivity tips and help buttons within the application for best utilization. Find optimal level of personalization and customization that users would take advantage of.
  • 9.
  • 10. Usability: Assess effectiveness of navigation system in guiding users to desired pages. Evaluate descriptiveness and clarity of links. Gauge ability of page layouts to orient users to relevant content.
  • 11. Usefulness: Identify content that is missing which will help overcome objections or answer critical questions? Understand how branded labels and content themes contribute to the overall experience or detract from it. Determine level/types of promotions and interstitials that are acceptable to users. Understand how different audience personas prefer to consume information for the particular domain.
  • 12.
  • 13. Usability: Determine optimal level of difficulty to encourage advancement to multiple levels of the game. Assess discoverability of game features and controls.
  • 14. Usefulness: Find the optimal rate of point accumulation and alignment with prize levels. Understand best use of social media within or around the game. Determine the threshold for ads, interstitials and registration for game play.
  • 15.
  • 16. Usability: Assess if users can figure out how to add a comment, share content, or use a tagging mechanism to find what they are looking for.
  • 17. Usefulness: Determine the most compelling and appealing topics or categories for conversation. Understand the level of involvement the sponsoring company should have in the social experience, if any. Balance branded or non-branded experience for optimal trust of the site. Determine the elements or attributes that should allow comment and review.
  • 18.
  • 19. Usability: Find any confusion points or interruptions that prevent users from registering. Find misleading or ambiguous terminology.
  • 20. Usefulness: Determine the most persuasive elements that will compel the target audience to register. Understand a design’s impact a user’s perception of the brand. Position the offering and messaging against the company’s competitors. Determine missing content that can help target audience make an informed decision about the product.
  • 22. Natural Reaction Turn to what we know: Usability Testing (one-on-one interviews, design artifact, and tasks)
  • 23. Are You Forgetting Contextual Inquiry and Foundational Research? Discovery research and needs analysis is valid, but: •  Time and budget for separate research is not always an option •  Many participants need design artifacts to elicit appropriate reaction and commentary
  • 24. Our Goal Leverage the strengths of usability testing but adjust our approach when objectives differ from finding usability problems.
  • 26. Three Components Phase Usability Usefulness Pre-Task Demographics, level of Daily task flow, pain points, Questions expertise, prior experience. expectations, desires, Goal: validating groups scenarios. and classify results. Goal: set mindset for usefulness evaluation of the design Tasks Post-Task Questions
  • 27. Three Components Phase Usability Usefulness Pre-Task Demographics, level of Daily task flow, pain points, Questions expertise, prior experience. expectations, desires, Goal: validating groups scenarios. and classify results. Goal: set mindset for usefulness evaluation of the design Tasks Pre-defined tasks, minimal Emphasis on participant- moderator intrusion, directed tasks Goal: find usability Goal: understand how proposed problems, measure time design aligns with user needs on task and completion percentage. Post-Task Questions
  • 28. Three Components Phase Usability Usefulness Pre-Task Demographics, level of Daily task flow, pain points, Questions expertise, prior experience. expectations, desires, Goal: validating groups scenarios. and classify results. Goal: set mindset for usefulness evaluation of the design Tasks Pre-defined tasks, minimal Emphasis on participant- moderator intrusion, directed tasks Goal: find usability Goal: understand how proposed problems, measure time design aligns with user needs on task and completion percentage. Post-Task Level of satisfaction with Comparison with expectations Questions the design, and points of and value. Task retrospectives. confusion or ambiguity. Goal: discuss opportunities for Goal: measure usability improvement.
  • 29. Example: Usability Goal: Find usability problems Pre-Task Questions: Audience segmentation Tasks: Defined Post-Task Questions: Baseline satisfaction
  • 30. Example: Usefulness Goal: Role of personalization Pre-Task Questions: Common tasks, pain points Tasks: Emphasis on participant direction Post-Task Questions: Comparison of expectation
  • 31. Summary Foundational research is still important. Usability testing is still important. However, recognize when you have different goals and adapt the research method as necessary.
  • 32. Additional Information Complete Presentation Slides • http://www.slideshare.net/hawleymichael • http://www.slideshare.net/banderlin Contact Information Michael Hawley Dan Berlin mhawley@madpow.net dberlin@madpow.net @hawleymichael @banderlin

Editor's Notes

  1. Goal of usability testing is to find usability problems.
  2. Transactional Applications
  3. Transactional Applications
  4. Transactional Applications
  5. Content / Informational Sites
  6. Content / Informational Sites
  7. Content / Informational Sites
  8. Gaming
  9. Gaming
  10. Gaming
  11. Social Sites
  12. Transactional Applications
  13. Transactional Applications
  14. Marketing / Persuasive
  15. Marketing / Persuasive
  16. Marketing / Persuasive
  17. So when a client wants discount research to guide their UX strategy and wants to learn about the value and usefulness of a prototype or existing interface… what do we do? We turn to what we know – usability testing.
  18. What strengths can we take from usability testing?How should the methodology evolve?What questions should we pose to elicit this information from participants?What is an acceptable prototype fidelity level?
  19. Moderator and participant interact one-on-one (except for unmoderated) Three primary components:Pre-task questions: moderator interviews participant to clarify demographics, assess level of expertise, and gauge prior experienceTasks: participant proceeds through defined set of tasks with design artifact while “thinking aloud”. Moderator minimizes intrusion, identifies usability problems, and tracks metrics such as time on task and completion percentagePost-task questions: moderator asks participant about their experience with the tasks, level of satisfaction with the design, and points of confusion or ambiguityNo longer just determining demographics and level of experienceRather, need to set participant’s mindset for an evaluation of the design from a usefulness perspectiveBorrow from Ethnography, Contextual Inquiry, Interviewing and Laddering Daily task flow, current mechanisms Pain points Expectations, desires Ideal scenarios Other systemsEmphasis changes from contrived tasks to participant-directed tasks Tasks determined from pre-task and probing questions“Tell me how you would use this application in your daily routine. Okay, now please go do some of those on this prototype.”“You mentioned that you would need to determine your size – how would you do that?”Gather expectations at outset of the task “What would you expect to happen? What information would expect the page to contain?”Sets the basis for participant commentary (not that you should design exactly what they say, but it can inform ideas)This may not be a natural way to interact with the design, but we’re not testing usabilityFollow up questions emphasize expectations and value “How did completing this task compare with your expectations? Was it better or worse and why? What’s missing? What’s superfluous?” “How does it compare with other applications or systems?” “What other features or functions would you need when completing this task?”Participant directed to walk back through the task, responding to questions at key interaction pointsStrategic probing questions on salient components that interrupt or inform the task“What would make this more useful? How would you use this in your work/life?”“Does this table contain the columns that you need? Did the categories make sense?” Probe on areas that were not covered by the participant task“How would you get the detailed information about this product?"
  20. Moderator and participant interact one-on-one (except for unmoderated) Three primary components:Pre-task questions: moderator interviews participant to clarify demographics, assess level of expertise, and gauge prior experienceTasks: participant proceeds through defined set of tasks with design artifact while “thinking aloud”. Moderator minimizes intrusion, identifies usability problems, and tracks metrics such as time on task and completion percentagePost-task questions: moderator asks participant about their experience with the tasks, level of satisfaction with the design, and points of confusion or ambiguityNo longer just determining demographics and level of experienceRather, need to set participant’s mindset for an evaluation of the design from a usefulness perspectiveBorrow from Ethnography, Contextual Inquiry, Interviewing and Laddering Daily task flow, current mechanisms Pain points Expectations, desires Ideal scenarios Other systemsEmphasis changes from contrived tasks to participant-directed tasks Tasks determined from pre-task and probing questions“Tell me how you would use this application in your daily routine. Okay, now please go do some of those on this prototype.”“You mentioned that you would need to determine your size – how would you do that?”Gather expectations at outset of the task “What would you expect to happen? What information would expect the page to contain?”Sets the basis for participant commentary (not that you should design exactly what they say, but it can inform ideas)This may not be a natural way to interact with the design, but we’re not testing usabilityFollow up questions emphasize expectations and value “How did completing this task compare with your expectations? Was it better or worse and why? What’s missing? What’s superfluous?” “How does it compare with other applications or systems?” “What other features or functions would you need when completing this task?”Participant directed to walk back through the task, responding to questions at key interaction pointsStrategic probing questions on salient components that interrupt or inform the task“What would make this more useful? How would you use this in your work/life?”“Does this table contain the columns that you need? Did the categories make sense?” Probe on areas that were not covered by the participant task“How would you get the detailed information about this product?"
  21. Moderator and participant interact one-on-one (except for unmoderated) Three primary components:Pre-task questions: moderator interviews participant to clarify demographics, assess level of expertise, and gauge prior experienceTasks: participant proceeds through defined set of tasks with design artifact while “thinking aloud”. Moderator minimizes intrusion, identifies usability problems, and tracks metrics such as time on task and completion percentagePost-task questions: moderator asks participant about their experience with the tasks, level of satisfaction with the design, and points of confusion or ambiguityNo longer just determining demographics and level of experienceRather, need to set participant’s mindset for an evaluation of the design from a usefulness perspectiveBorrow from Ethnography, Contextual Inquiry, Interviewing and Laddering Daily task flow, current mechanisms Pain points Expectations, desires Ideal scenarios Other systemsEmphasis changes from contrived tasks to participant-directed tasks Tasks determined from pre-task and probing questions“Tell me how you would use this application in your daily routine. Okay, now please go do some of those on this prototype.”“You mentioned that you would need to determine your size – how would you do that?”Gather expectations at outset of the task “What would you expect to happen? What information would expect the page to contain?”Sets the basis for participant commentary (not that you should design exactly what they say, but it can inform ideas)This may not be a natural way to interact with the design, but we’re not testing usabilityFollow up questions emphasize expectations and value “How did completing this task compare with your expectations? Was it better or worse and why? What’s missing? What’s superfluous?” “How does it compare with other applications or systems?” “What other features or functions would you need when completing this task?”Participant directed to walk back through the task, responding to questions at key interaction pointsStrategic probing questions on salient components that interrupt or inform the task“What would make this more useful? How would you use this in your work/life?”“Does this table contain the columns that you need? Did the categories make sense?” Probe on areas that were not covered by the participant task“How would you get the detailed information about this product?"
  22. Moderator and participant interact one-on-one (except for unmoderated) Three primary components:Pre-task questions: moderator interviews participant to clarify demographics, assess level of expertise, and gauge prior experienceTasks: participant proceeds through defined set of tasks with design artifact while “thinking aloud”. Moderator minimizes intrusion, identifies usability problems, and tracks metrics such as time on task and completion percentagePost-task questions: moderator asks participant about their experience with the tasks, level of satisfaction with the design, and points of confusion or ambiguityNo longer just determining demographics and level of experienceRather, need to set participant’s mindset for an evaluation of the design from a usefulness perspectiveBorrow from Ethnography, Contextual Inquiry, Interviewing and Laddering Daily task flow, current mechanisms Pain points Expectations, desires Ideal scenarios Other systemsEmphasis changes from contrived tasks to participant-directed tasks Tasks determined from pre-task and probing questions“Tell me how you would use this application in your daily routine. Okay, now please go do some of those on this prototype.”“You mentioned that you would need to determine your size – how would you do that?”Gather expectations at outset of the task “What would you expect to happen? What information would expect the page to contain?”Sets the basis for participant commentary (not that you should design exactly what they say, but it can inform ideas)This may not be a natural way to interact with the design, but we’re not testing usabilityFollow up questions emphasize expectations and value “How did completing this task compare with your expectations? Was it better or worse and why? What’s missing? What’s superfluous?” “How does it compare with other applications or systems?” “What other features or functions would you need when completing this task?”Participant directed to walk back through the task, responding to questions at key interaction pointsStrategic probing questions on salient components that interrupt or inform the task“What would make this more useful? How would you use this in your work/life?”“Does this table contain the columns that you need? Did the categories make sense?” Probe on areas that were not covered by the participant task“How would you get the detailed information about this product?"
  23. Usability is still important!But, recognize when you have different goals and adjust accordinglyBe aware of nuances and differences between usability and usefulnessSet clear expectations and goals with project teamName it “usefulness study” rather than a “usability test”?Drawbacks: may need deeper prototypes, not getting at usability problems, Danger: asking the participant to help design it is easy to fall into, but not good