Notes on evaluation and usability inspection, Baobab Health Trust, March 2014

379 views
233 views

Published on

Introduction to evaluation and usability inspection, presented to Baobab Health Trust, Lilongwe, Malawi, March 2014.

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
379
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Notes on evaluation and usability inspection, Baobab Health Trust, March 2014

  1. 1. Boabab Health, March 2014Harry Hochheiser, harryh@pitt.edu Evaluation Harry Hochheiser University of Pittsburgh Department of Biomedical Informatics harryh@pitt.edu +1 412 648 9300 Attribution-ShareAlike CC BY-SA
  2. 2. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu Why Evaluation?
  3. 3. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu Why Evaluation? Show that it works? Figure out what doesn’t work? Figure out how to make things better? Allocate resources effectively? Support investment of resources Justify scaling up
  4. 4. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu DECIDE Evaluation Framework Preece, Rogers, Sharpe, Interaction Design, 2011 Determine the goals. Explore the questions. Choose the evaluation methods. Identify the practical issues. Decide how to deal with the ethical issues. Evaluate, analyze, interpret and present the data.
  5. 5. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu Determine the goals You already know the goals! Decided during design Costs Quality Time/Efficiency Satisfaction Materials, Capital, Revenue Patient, Staff, MOH Capacity, Wait times, Turnaround time Safety Waste Removal, Best Practices IMPROVE Safety, Quality, Satisfaction DECREASE Costs
  6. 6. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu Tradeoffs? • Evaluation results may not be clear cut… • Increased efficiency but decreased satisfaction? • Increased quality but decreased efficiency? • Decreased cost and decreased satisfaction? • Mixed signals are not uncommon!
  7. 7. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu Evaluation Study types
  8. 8. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu Spectrum of Methods Inspection Methods Heuristic Evaluation Cognitive Walkthrough Lab Usability Studies “think-aloud” Comparative Empirical Studies in situ evaluation Quantitative and qualitative Low  Cost   Low  Fidelity High  Cost   High  Fidelity Formative Summative
  9. 9. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu No bright dividing line in process Design Fully-functional Prototype Paper Prototype Release Usability Inspections Usability Studies Empirical User Studies, Case Studies, Longitudinal Studies, Acceptance Tests Low  cost,  low  validity Higher  cost,  validity
  10. 10. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu Usability Assessments • This is where we start talking about how to assess usability! • Goals - discuss usability measurements that can • Inexpensively identify problems • both potential and real • Guide design • ideally early, before deploymen
  11. 11. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu Goals of Usability Assessments • Can users complete tasks? • Appropriateness of mental models • Comparative efficiency • Subjective satisfaction • How do we assess?
  12. 12. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu Usability Inspections • “clean-room” static examination of usability • Methodically scrutinize interfaces in search of potential problems • Pros: – Inexpensive – no users, relatively easy – Identify major issues at a relatively early stage • Cons: – May miss problems: generally find < 50% – All results are hypothetical – don’t know which problems might really lead to errors
  13. 13. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu Common Questions •Who inspects?   •Background   •How many?   • Which tasks?   • How to interpret findings?   •Problems are not “real” problems experienced by users   •How important are they?
  14. 14. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu Two broad classes of usability inspections Heuristic Evaluations: How well does an interface conform to guidelines for interface design? Walkthroughs: Analytic examination of interface and interaction requirements, usually informed by some model of the user Many variants...
  15. 15. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Heuristic inspection Usability experts Domain experts Combination? (Double experts) Users should participate as users when possible 3-5 experts? (Nielsen) Or more... Work alone, or in teams.. Walkthroughs May require more cognitive background Domain expert feedback helpful Conducted by a team? Who inspects?
  16. 16. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Heuristic inspections Set tasks Open-ended exploration Walkthroughs Generally, specific tasks Tasks
  17. 17. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Results
 Nielsen and Molich 1990 , Nielsen 1993 • Few false positives • Find < 50% of errors • “In general, we would expect aggregates of five evaluators to find about two thirds of the usability problems which is really quite good for an informal and inexpensive technique like heuristic evaluation.”
  18. 18. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Results
 Nielsen and Molich 1990 Source of the “you only need 5 evaluators” rule… Hotly contested…
  19. 19. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Which Evaluators?
 Nielsen, 1992 • “Double Experts” - Domain and Usability expertise – considered best • Not always available – use some of each, or teams
  20. 20. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu When to conduct inspections? Paper prototypes? Working system? Early and often…
  21. 21. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Evaluators work alone (except for when they work in teams) Optional observer can help explain confusing issues and to record issues. Go through interface several times overview and specific Heuristic focus or task focus Note discrepancies between interface and heuristic Individual evaluators meet to aggregate results Heuristic Evaluation Procedure
  22. 22. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Nielsen's Heuristics Nielsen, 1994 - http://www.nngroup.com/articles/ten-usability-heuristics/ • Visibility of system status • Match between system and real world • User control and freedom • Consistency and standards • Error prevention • Recognition rather than recall • Flexibility and efficiency of use • Aesthetic and minimalist design • Help users recognize, diagnose, and recover from errors • Help and documentation
  23. 23. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Nielsen's Heuristics Nielsen, 1994 - http://www.nngroup.com/articles/ten-usability-heuristics/ • Visibility of system status •“The system should always keep users informed about what is going on, through appropriate feedback within reasonable time”   • Match between system and real world “The system should speak the users’ language, with words, phrases, and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order.”   • User control and freedom “Users often choose system functions by mistake and will need a clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.”
  24. 24. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Nielsen's Heuristics Nielsen, 1994 - http://www.nngroup.com/articles/ten-usability-heuristics/ • Consistency and standards   “Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.” • Error prevention “Even better than good error messages is a careful design which prevents a problem from occurring in the first place. Either eliminate error-prone conditions or check for them and present users with a confirmation option before they commit to the action.” • Recognition rather than recall “Minimize the user's memory load by making objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.”
  25. 25. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Nielsen's Heuristics Nielsen, 1994 - http://www.nngroup.com/articles/ten-usability-heuristics/ • Flexibility and efficiency of use “Accelerators -- unseen by the novice user -- may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.” • Aesthetic and minimalist design •  “Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.”
  26. 26. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Nielsen's Heuristics Nielsen, 1994 - http://www.nngroup.com/articles/ten-usability-heuristics/ • Help users recognize, diagnose, and recover from errors “Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.” • Help and documentation “Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large.”
  27. 27. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.edu Recording Heuristic Violations • Which Heuristic • Description • Severity • Proposed revisions
  28. 28. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Use severity judgments to prioritize fixes Frequency of problem Impact of problem Persistence – will users be repeatedly bothered? Multiple independent raters increase reliability Bigger questions – does this design work at all? As with usability studies, try to generalize Don't solve lots of small problems if the design is inherently problematic Interpretation
  29. 29. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Making use of heuristic evaluation results? ● Have member of development team present during inspection? Can be done with usability studies and other inspections as well They should observe, not analyze ● Debriefing session to explore fixes Consider redesigns where appropriate.
  30. 30. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Other Inspection Methods http://www.useit.com/papers/heuristic/inspection_summary.html Heuristic estimation – compare interfaces Feature Inspection – look for long sequences of operations, unnatural sequences, those requiring extensive skill... Consistency inspection – compare designs for consistency Standards Inspection
  31. 31. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Which Heuristics? • Multiple versions of Nielsen’s list • Other versions – Special cases? • More heuristics – more focus – ..but less study • Too many heuristics – cumbersome, complicated
  32. 32. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Web Usability Heuristics
 http://www.andybudd.com/archives/2007/01/ heuristics_for_modern_web_application_development/index.htm • Design for user expectations • Clarity • Minimize Unnecessary Complexity and Cognitive Load • Efficiency and Task Completion • Provide Users with Context • Consistency and Standards • Prevent Errors • Help users, notice, understand, and recover from errors • Promote a pleasurable and positive web experience
  33. 33. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Using usability heuristics to evaluate patient safety of medical devices
 Zhang, et al. 2003 • Nielsen heuristics + Shneiderman’s 8 golden rules • Revised and clarified • New severity ratings • Demonstrations of use… – … but less study than Nielsen’s list
  34. 34. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Zhang, et al. 2003 - Heuristics • Consistency • Visibility • Match • Minimalist • Memory load • Feedback • Flexibility • Messages (errors) • Errors (prevention) • Closure • Language • Control • Document Severity Rating 0. No problem 1. Cosmetic 2. Minor 3. Major 4. Catastrophe
  35. 35. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Zhang, et al. 2003 - Heuristics • Severity rating • “As a guideline for rating the problems, we consider the proportion of users who will experience it, the impact it will have on their experience with the product, and whether the usability problem will be a problem only the first time they encounter it, or whether it will persistently bother them. A persistent problem with a major impact that most users will encounter will get the highest severity rating.” • What about rare but potentially catastrophic problems?
  36. 36. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Assessing the usability of a telemedicine-based Medication Delivery Unit for older adults through inspection methods
 Ligons, et al. 2011 ● EMMA: The Electronic Medication Management Assistant (INRangeSystems, Aloona PA) ● Electronic medication delivery + wireless communication with pharmacy ● Track adherence ● Adjust dosage ● Touch screen interface
  37. 37. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Assessing the usability of a telemedicine-based Medication Delivery Unit …
 Ligons, et al. 2011 • Goal: use MDUs to improve adherence among older patients • Concern – touch-screen might be hard to use – Calibration of touch – Low vision or blindness - difficulty seeing screen/reading? – Cognitive challenges – understanding use? • Use Heuristic evaluation for first-pass investigation of usability – Question – “Can users use the system?”, not “How can we improve the system?”
  38. 38. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Assessing the usability of a telemedicine-based Medication Delivery Unit …
 Ligons, et al. 2011 NLM guidelines for website design for older adults • User only sans-serif font, 12-14 point, medium or bold weight, double-spaced, left- justified • Avoid blue, yellow, and green, in close proximity • Use light background with dark text or dark backgrounds with light text • Modified Nielsen (1990) heuristics Simple and Natural Dialog Speak the user’s language Minimize memory load Be consistent Provide Feedback Provide clearly marked exits User good error messages Prevent errors Provide shortcuts
  39. 39. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Assessing the usability of a telemedicine-based 
 Medication Delivery Unit …
 Ligons, et al. 2011 • Explore interface with respect to each heuristic – Multiple passes
  40. 40. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Assessing the usability of a
 telemedicine-based Medication 
 Delivery Unit …
 Ligons, et al. 2011 • Minimize memory load - major issues – Confusing buttons complicate access to help system – “Drop” menu for advance indications doesn’t include displays of appropriate dates and medications • Be Consistent – minor issues – “back” button vs. “cancel” buttons – “medication drops” vs. “advance drops” on different screens • Provide Feedback – medium – No feedback when buttons are pressed – Slow response
  41. 41. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Assessing the usability of a telemedicine-based Medication Delivery Unit …
 Ligons, et al. 2011 • Prevent Errors – medium – Small buttons with low tolerance – Complicated process of loading and unloading medication packs • Typeface – Small fonts for on-screen medication and information on blister packs
  42. 42. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Usability Evaluation of an Electronic Medication Administration Record (eMAR) Application
 Guo, et al. 2011 • Medication Administration Record - record of drug delivery • eMAR tasks – Access to drug information, policies, and procedures related to meds – Context-specific patient clinical results – Facilitate communication between nurses and pharmacists – Verification of the five “rights” (patient, drug, dose, route, and time) • Heuristic Evaluation for usability • Using Zhang’s heuristics and severity ratings
  43. 43. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Electronic Medication Administration Record (eMAR) Application
 Guo, et al. 2011
  44. 44. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Heuristic evaluation of eMAR
 Guo, et al. 2011 • Tasks – Login – Order and modify medication – Verify medication orders – Access drug references – Administer medication – Edit eMAR – Generate reports and review • Procedure – Train evaluators – Conduct evaluation – Calculate agreement – Build consensus – Review with staff – conduct tasks
  45. 45. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Heuristic evaluation of eMAR
 Guo, et al. 2011 • Observations – classified by heuristic, – grouped into usability problems – Organized by task
  46. 46. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Heuristic evaluation of eMAR
 Guo, et al. 2011
  47. 47. Baobab Health, March 2014Harry Hochheiser, harryh@pitt.eduHarry Hochheiser, harryh@pitt.edu Baobab Health, March 2014 Heuristic evaluation of eMAR
 Guo, et al. 2011

×