• Save
Usability and Health IT
Upcoming SlideShare
Loading in...5
×
 

Usability and Health IT

on

  • 3,812 views

by Robert Schumacher, Ph.D. ...

by Robert Schumacher, Ph.D.
Presentation given on 21 May to the GCC HIMSS group in Chicago with ~50 people present.
www.usercentric.com
The point was to provide some background on usability (a gentle introduction to some of the science), some case studies, and introduce the measurement AND design components of user centered design.

Note because of all the animations, some pages do not view properly. Please contact me if you would like more information:
bob at usercentric.com

Statistics

Views

Total Views
3,812
Views on SlideShare
3,616
Embed Views
196

Actions

Likes
15
Downloads
0
Comments
0

4 Embeds 196

http://chilmarkresearch.com 186
http://www.slideshare.net 8
http://static.slidesharecdn.com 1
http://www.usercentric.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Thank you.NamePleasure to be here to speak to you.I am Managing Director of a user research firm HQ in OBT, with test studio @ 500N and in Beijing ChinaWant to start out with a few non-HIT examples to illustrate some points about usability- In fact these examples are set to illustrate – three areas of usability: cognition, motor performance, and attention.Some may seem a little strained, but we’ll weave them togetherIt’s my hope that you’ll walk out tonight with a greater appreciation for what Usability is, how it is used, and specifically how it can be applied in HIT
  • Cognitive
  • Driver performance when talking on a cellphone is roughly equivalent to being drunkDriver performance is no different when on handsfree and hand holding Epidemiological and forensic analysis support these lab findingsLive conversation is much less impaired than cell conversationReason: Cognitive demands of conversation impair ability to perform primary task track lanes and brakeTheory: Multiple resource theoryImplications: Accidents may not go down, and could possibly go upMeta-analysis by The Canadian Automobile Association[9] and The University of Illinois[10] found that response time while using both hands-free and hand-held phones was approximately 0.5 standard deviations higher than normal driving (i.e., an average driver, while talking on a cell phone, has response times of a driver in roughly the 40th percentile). Horrey, William; Christopher Wickens (Spring 2006). \"Examining the Impact of Cell Phone Conversations on Driving Using Meta-Analytic Techniques\" (PDF). Human Factors (Human Factors and Ergonomics Society) 38 (1): 196–205. Strayer D. L., Drews F. A. & Johnston W. A. (2003). \"Cell Phone-Induced Failures of Visual Attention During Simulated Driving.\". Journal of Experimental Psychology: Applied 1 (9): 23–32. doi:10.1037/1076-898X.9.1.23Strayer D. L. & William J. A. (2001). \"Driven to distraction: Dual-Task Studies of Simulated Driving and Conversing on a Cellular Telephone.\". Psychological Science 6 (12): 462–466. cell phone drivers exhibited greater impairment than intoxicated drivers.  POINT: Cognitive impairment\"Drivers need to keep not only their hands on the wheel; they also have to keep their brains on the road,\" passenger conversations differ from cell phoneconversations because the surrounding traffic not only becomes a topic of the conversation, helpingdriver and passenger to share situation awareness, but the driving condition also has a direct influenceon the complexity of the conversation, thereby mitigating the potential negative effects of a conversationon driving.cell phone and passengerconversation differ in their impact on a driver’s performanceand that these differences are apparent at the operational, tactical,and strategic levels of performance.
  • Motor
  • MOTOR Performance
  • A 26 x 26 matrix was constructed to further analyze the errors users made on the iPhone keyboard.The columns represent the intended key press or the targeted letter.The rows represent the actual key press made by the participant.
  • ExampleSet up
  • PerceptualSeeFeelHearSmellTouchCognitive-Remember-Recall-DecideMotor-Act
  • 5 minutes
  • Not market research, focus groups, user acceptance testing, surveys, graphic designThere is a kernel of truth in all of these, they are worthy subjects and have their place, however these are not what we’ll be talking about here
  • Main Entry:us·ableVariant(s):also use·able  \\ˈyü-zə-bəl\\Function: nounDate:14th century1 : capable of being used2 : convenient and practicable for use
  • Usability is a soft construct
  • Here are just a few examples of present and past work by human factors/ergonomics experts that has made a difference in our lives:What would driving today be like without the center high-mounted rear brake light in cars? Human factors/ergonomics professionals found that use of this safety feature helped to prevent rear-end collisions, saving millions of dollars each year. Numerous contributions by HF/E professionals have made air travel safer. These efforts focus on both cockpit safety and effectiveness as well as improved interaction between pilots and air traffic controllers.Those with diabetes are benefiting from usability studies performed on a range of blood glucose monitoring systems, which have resulted in improvements to these devices. HF/E professionals are helping to ensure that other medical devices for use in both health care institutions and the home are safer and easier to use.
  • Czerwinski, M., Horvitz, E., & Cutrell, E. (2001). Subjective Duration Assessment: An Implicit Probe for Software Usability. In Proceedings of IHM-HCI 2001 Conference, Volume 2, (September, 2001, Lille, France), p. 167–170
  • It is the foundation upon which interface design can be successfully implemented.
  • Evaluations and EHR
  • I tried to find some more information about what “harder to read” means statistically. I managed to find some numbers in Janice Redish’s book, Letting Go of the Words: Writing Web Content that Works. (This is a great writing-for-the-web book, by the way!) ALL CAPS slow reading speed by about 15 percent. Given that website visitors tend to want to obtain and read information quickly on the web, making web content harder to read by using ALL CAPS should generally be avoided.
  • Clark, H. H., & Chase, W. G. (1972). On the process of comparing sentences against pictures. Cognitive Psychology, 3, 472-517.Just, M. A., & Carpenter, P. A. (1971). Comprehension of negation with quantification. Journal of Verbal Learning and Verbal Behavior, 10, 244-253.
  • All have same result if you press “return”
  • People have goals when they use systems. We are constantly looking over our visual field Looking for a passport
  • Evaluations and EHR
  • Guideline: Do not use two (or more) different ways to highlight the same information on one page.Comments: One study found that participants were able to complete tasks faster when the interface contained either color-coding or a form of ranking, but not both. The presence of both seemed to present too much information, and reduced the performance advantage by about half.Sources: Bandos and Resnick, 2004; Resnick and Fares, 2004. Bandos, J. & Resnick, M.L. (2004). The effects of semantic and syntactic instruction on users performance and satisfaction in search user interface design. Proceedings of the Human Factors and Ergonomics Society Annual Meeting.Resnick, M.L. & Fares, C. (2004). Visualizations to facilitate online tabular presentation of product data. Proceedings of the Human Factors and Ergonomics Society Annual Meeting.
  • Like any quality activity, benefits are hard to quantify for an individual project.There is ample research that supports incorporation of usability activities as a cost-reduction mechanism.Typical figures quoted are a benefit of between 10:1 and 100:1 (that is, a saving of between $10 and $100 for every dollar spent).Decreased user errorsOne project we did was with New Zealand Surf Lifesaving, where we helped simplify the web interface to their database. SLSNZ are actually surprisingly big on measuring statistics. The season that we did the work there were 1847 rescues performed by 4000 active lifeguards around New Zealand. They need these kind of statistics to justify their funding – they make a difference by saving lives. One of the biggest advantages of having a user centred focus was fewer errors in the database. Decreased training costsFurthermore, the new interface was much simpler and resulted in less training cost. Not having their trainer fly all over the country to visit 75 surf clubs from the far north to the deep south makes a big difference for a not-for-profit.Decreased user support costsWork with did with the official e-commerce site of the Canterbury clothing company resulted in far fewer enquiries about shipping information, and where stores are located, as a result of the changes we recommended.
  • ***Will reduce these slides so fewer words and details***
  • We’ve seen that specific techniques for measuring user performance exist…A lot of current problems are solved problemsApplying usability and human factors to healthcare systems would help organizations reach their patient-centered goals (STEEP) and practitioner-centered goals.This diagram is from a study showing the impact of interface design on technology-induced error in healthcare – by applying usability principles, errors decrease.
  • ***Will adjust the diagram to better show this message:***There are often tradeoffs and many constraints in designing healthcare technology.Nobody starts out saying, “I’m going to make this really difficult to use.”HIT has come a long way, using healthcare domain expertise along with technology expertise,but to take it to the next level, usability is needed.
  • That is, [CLICK] from our analysis of 34 publicly available RFPs for EHRs, [CLICK] 26 out of 34 RFPs made no mention of usability. Seventy-five percent of the RFPs do not include usability as an important component to the proposal of developing and implementing EHRs. Of those RFPs that did make mention of usability, [CLICK] many RFPs used vague descriptions when discussing usability.From our content analysis, the importance attributed to usability is miniscule at best.

Usability and Health IT Usability and Health IT Presentation Transcript

  • Usability: Introduction and Application to HIT GCC HIMSS Meeting Robert M. Schumacher, Ph.D. Managing Director User Centric, Inc. Oakbrook Terrace, Illinois rschumacher@usercentric.com May 21, 2009
  • Who has the highest risk of accident? (1) Conversing while (2) Conversing with hands holding a phone free phone (3) Conversing with (4) Drunk person in car
  • Cognitive Impairment While Driving with Cell Phone Meta-analysis by Horrey & Wickens (2006) found: - There are significant costs to driver reactions to external hazards or events associated with cell phone use, and - Hands-free cell phones do not eliminate or substantially reduce these costs - Talking on cell phone when driving (hands-free or handed) shows performance similar to driving drunk (Strayer and colleagues) Using a cell phone impairs a driver’s cognitive function Drews, F. A., Pasupathi, M. & Strayer, D.L. (2008). Passenger and Cell Phone Conversations in Simulated Driving. Journal of Experimental Psychology: Applied. 14 (4), 392–400 Horrey, W., & Wickens, C. (Spring 2006). Examining the Impact of Cell Phone Conversations on Driving Using Meta-Analytic Techniques, Human Factors, 38 (1): 196–205. Strayer D. L. & William J. A. (2001). Driven to distraction: Dual-Task Studies of Simulated Driving and Conversing on a Cellular Telephone.quot;. Psychological Science, 6 (12): 462–466. 3
  • Which Phone Is Best for Text Entry? Speed Speed Accuracy Accuracy (1) iPhone (3) Blackberry Full QWERTY QWERTY (Touchscreen) (Hard Key) (2) Razr Numeric (Hard Key)
  • Motor Differences When Typing on Cell Phones Average Time to Enter Messages Average Number of Errors  QWERTY fast and accurate  iPhone fast but inaccurate  Numeric slow but accurate Much of performance can be characterized by the Speed Accuracy Tradeoff Lew, G. et al. (2007). User performance with iPhone. Unpublished research.
  • Intended Keypresses vs. Actual Keypresses Presented Letter (Intended Key Press) Actual Key Pressed Correct key presses > 10 incorrect > 20 incorrect > 40 incorrect 6
  • Implications for Design 7
  • Imagine You Are Shown This Picture: Hokusai's Wave 8
  • Perceptual Processes 9
  • Eye tracking Reveals Perceptual Processing Quantitative Measures Meaning Informativeness of an area / user # fixations on an area interest in the area Info clarity / info density / info Fixation length processing demands # fixations before target Layout effectiveness / search Time to 1st target fixation demands Scanpath complexity % users fixating on an area Prominence / perceived importance Order of 1st fixation of an area # visits to area Cognitive processing demands / Pupil diameter user mental workload e.g., Bojko, A. (2009). Informative or misleading? Heatmaps deconstructed. Proceedings of the 13th International Conference on Human-Computer Interaction (HCII). San Diego, CA. 10
  • Usability Derives from Behavioral Science Motor Cognitive Perceptual
  • Plan  Usability Defined  Some Examples and Implications  Usability in Healthcare Technology 12
  • Usability Defined Usability is not… So what is usability? 13
  • Usability Defined What Usability is Not User Acceptance Testing Aesthetic Functional Attitude & Opinion (Not behavior) 14
  • Usability Goes by Many Names  Human factors  Engineering psychology  Ergonomics  User experience  User research  User-centered design 15
  • Usability Defined yü-zə-ˈbí-lə-tē Usability is the effectiveness, efficiency, and satisfaction with which the intended users can achieve their tasks in the intended context of product use. Adapted from National Institute of Standards & Technology Outcome Often Get Confused Process Or © May 28, 2009 – Proprietary and Confidential Attribute 16
  • Some Context  Usability derives from applied Key Components of Usability: behavioral sciences – Measurement (Diagnosis) – Psychology – Intervention (Design) • Cognitive • Experimental • Biological – Neuroscience – Anthropology – Industrial engineering – Human-computer interaction – Computer science – Linguistics – Market research  From these we borrow: – Research base – Experimental methods 17
  • Usability Defined A Brief History of Usability 18
  • A Few that Embrace Usability and User Research Each of these organizations has more than 100 usability staff members 19
  • A Few Ways to Measure User Performance  Reaction times  Error – Stimulus onset to response – Number / type / severity  Task Completion Times  Efficiency – Complex sequence of actions – Ratio of Actual behavior to Optimal behavior – E.g., Steps, keystrokes, clicks  Attention – Eye tracking  Subjective Measures – Ratings of Satisfaction / Usability /  Subjective time on task duration Usefulness – Difficult tasks overestimate time  Learnability – Measure trials to criterion 20
  • Usability as a Process Existing Context Mental Iterative Validation interface for use model design & testing Diagnose Explore Identify key user Design, review, validate design, repeat context for use goals and approach • Navigation and • User • Goals screen flow • Evidence- research analysis based interviews • Interaction evaluation/ • Task storyboards Cognitive • Design • Field analysis reviews walkthrough observations • Interaction • Low-fidelity • User surveys • Site visits models wireframing • Validation interviews, lo • Content/ • Card sorting w-fidelity UT feature • High-fidelity mapping • Site prototyping, inc Mapping luding • Usability • Eye tracking iconography testing and visual aesthetics • A/B testing
  • User Research (Measure & Design)  Observe users in their “native” Pharmacy Visit environment  Identify features most needed and relevant to users  Identify user interface and functional areas that users see as most lacking  Determine metrics that can be used to measure return on investment Hospital Visit On-site in an airplane simulator 22
  • Lab-Based Usability Testing (Measure & Design)  Representative users perform representative tasks in a controlled setting  Collect performance and feedback data Usability Testing  Outcome: Diagnose problems, discover remedies for repair and /or collect base- line performance metrics for future comparisons Observation Using One-way Mirrors and Video 23
  • Eye Tracking (Measure)  Captures the location of the eyes while looking at a stimulus  Offers insight into the perceptual and cognitive processes involved in user interaction  Can be used with computer interfaces, physical products, and Eye tracking study for a financial Web site printed material Eye tracking quot;heat mapquot; report 24
  • Evidence-Based Evaluation & Cognitive Walkthrough (Diagnose)  A detailed assessment of the artifact: – Grounded in applied research – Tempered by experience – Guided by best practice and Evaluating a mobile application population stereotypes – Scenario-based simulation of common and critical tasks based on user personas Actionable Recommendations in a Report 25
  • Information Architecture & Interaction Design (Design) Information architecture (IA) is the Interaction design defines the user blueprint for a website or interface. interactions with the interface.  Structure  Organization  Navigation  Labeling Good interaction design makes the user’s interactions as simple and efficient as possible. 26
  • Experience Design (Design) User-Centered (Clinician-Centered) Design  Discovery – Measurement – Observation – Interview – Survey  Information architecture  Wireframe mockups Card sorting  Prototyping  Specifications  Visual design Wireframe mockups Graphic design 27
  • Some Examples and Implications Typography Language and labeling Visual search Form design 28
  • Impacts of Typography on User Performance 29
  • Evaluation: Simple example Ficticious Patient Let’s look at something as the fonts: Button labels: ALL CAPS, color coded, Button labels: ALL CAPS, color 12 pt, Arial, Bold coded, 14 pt, Arial, Bold Field labels: Mixed case, 12 pt, Arial, Field labels: Mixed case, 12 pt, Arial, regular regular regular pt, Arial, © May 28, 2009 – Proprietary and Confidential Patient info1: Mixed case, 32 pt, Arial, Patient info1: Mixed case, 32 pt, blue, bold Arial, blue, bold pt, Arial, blue, bold Patient info2: Mixed case, 9 pt, Arial, Patient info2: Mixed case, 9 pt, Arial, underlined, regular underlined, regular pt, Arial, underlined, regular 30
  • Typography for Labels – Which is best? Baseline ALL CAPS Underline Bold Highlighted Arial 12 Arial 12 Arial 12 Arial 9 Arial 14 Times 12 Arial 12 Arial 12 Common COMMON Common Common Common Common Common Common Department DEPARTMENT Department Department Department Department Department Department Meds MEDS Meds Meds Meds Meds Meds Meds OrderSets ORDERSETS OrderSets OrderSets OrderSets OrderSets OrderSets OrderSets Fluids FLUIDS Fluids Fluids Fluids Fluids Fluids Fluids Drips DRIPS Drips Drips Drips Drips Drips Drips Insulin INSULIN Insulin Insulin Insulin Insulin Insulin Insulin TPN TPN TPN TPN TPN TPN TPN TPN Unsent UNSENT Unsent Unsent Unsent Unsent Unsent Unsent Existing EXISTING Existing Existing Existing Existing Existing Existing
  • Line Length & Case It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light Line lengths of 4 (+/- 1.2) inches are easiest to read. Lines greater It was the best of times, it was the worst than 5 inches strain the eye as it back of times, it was the age of wisdom, it scans, making it easier to jump to the was the age of foolishness, it was the wrong next line. Lines less than 2.5 epoch of belief, it was the epoch of incredulity, it was the season of Light inches slow reading due to the large number of eye movements IT WAS THE BEST OF TIMES, IT WAS ALL CAPS can reduce reading THE WORST OF TIMES, IT WAS THE speeds by as much as 20% AGE OF WISDOM, IT WAS THE AGE OF FOOLISHNESS, IT WAS THE EPOCH OF BELIEF, IT WAS THE EPOCH OF INCREDULITY, IT WAS THE SEASON OF LIGHT 32
  • What Do We know?  Reading speed:12% slower on screen than paper  Typographic characteristics also negatively affect reading speed: ALL CAPS,Bold, underline, font type and fon t s ze i  Reading comprehension: Often worse on screen than paper  Reading speed and comprehension all affected by: Font color, background color, contrast, alignment, and line length Implications? So What?  Reading charts on line is slower with lower comprehension than paper  With fatigue, typography has cumulative effects  Use visual design to improve (and even beat) status quo
  • So What Should You Do? Typography Recommendations  Stay within one font family (usually sans serif online)  Keep text sizes readable (12 pt optimal, >9 pt, < 14 pt)  Use high contrast ratios darker letters on a lighter background  Use mixed case  Avoid use of italics and underlining as emphasis tools  Use left aligned and ragged right text (Not centered or right aligned)  Use only a few unique margins on page (Use indentations sparingly)  Use “Title Casequot; or bold when emphasis is needed  Avoid color coding text 34
  • Implications of Language 35
  • Language Affects Speed And Comprehension False True statement, statement, Negat Affirmative (Yes) ive (No) Response Response The circle isis above the star The circle not above the star Yes or No? 36
  • Findings Statement Statement is Response is (1) The circle is above the star. True Affirmative (2) The star is above the circle. False Affirmative (3) The star is not above the circle. True Negative (4) The circle is not above the star. False Negative  True statements evoked faster responses than false statements  Positive (affirmative) responses much faster than negative (~ 500 msec) Carpenter, P. A., and Just, M. A. (1975). Sentence comprehension: A Clark, H. H., & Chase, W. G. (1972). On the process of comparing psycholinguistic processing model of verification. Psychological Review, sentences against pictures. Cognitive Psychology, 3, 472-517. 82 (1), 45-73. 37
  • So What? Think About Dialog Boxes in CPOE… Affirmative Negative Non Destructive What just Happened? Different phrasings are used when you want different responses Statement Type Use When  True Affirmative Fast, easy, low-cost-to-user outcome, confirmation only  False Affirmative Need user to think about the response, high cost to user  True Negative Should almost never use  False Negative Never use, unless trying to deceive user e.g., Opt out response [ ] Do not send me the newsletter 38
  • Visual Search Finding needed data 39
  • Visual Design and Signal Detection  Users have goals, e.g., – Review lab reports for abnormalities – Ensure no allergies to certain medications  We use perceptual and cognitive processes to review screens – Goal is “target” or “signal” – All else is “noise”  Good design has high signal:noise ratios – Anticipates important signals – Enables location of signals • Proximity • Coding • Redundant queing  To illustrate this point…we need a little help 40
  • Signal Detection: Find Waldo
  • End of a Long Frustrating Day… Based on patient information would you prescribe Amoxicillin?
  • Ficticious Patient Expert Evaluations: CPOE example © May 28, 2009 – Proprietary and Confidential 43
  • End of a Long Frustrating Day… How did the patient’s white cell count change?
  • Improper Incorrect Alignment use of Of fields color Incorrect use of Poor use Incorrect color of coding use of data color
  • What Should You Look for in Evaluations & Walkthroughs? Use the published research, knowledge of best practices, population stereotypes, and expert opinion to analyze the user interface. For example we would go deep into the following areas: • Page Layout • Screen-Based Controls • Navigation • User experience (e.g., response • Scrolling & Paging times) • Headings, Titles & Labels • Graphics, Images and Multimedia • Links • Content Organization & Style • Text Appearance • Search • Lists • Accessibility Additionally, we simulate common and critical use cases and scenarios based on personas 46
  • Navigation & Mental Models 47
  • User Research: Defining Process Flows  Stocking Errors: Drug is placed where it is not supposed to be  Selection Errors: Wrong drug is selected from among other drugs  Verification Errors: Incorrectly selected drug is confirmed as correct.  Data entry Errors: Drug administering device is incorrectly programmed. 48
  • Examples: User Testing, Eye Tracking & Form Design May 28, 2009 49
  • What Do We Know (or Think We Know) About Form Layout? Ficticious Patient 50
  • “Left” Layout  Advantages – Because eyes are naturally drawn to the hard edge on the left: • It is easy to find the beginning of the next label. • Labels are easy to scan (in previewing an unfamiliar form or finding required fields).  Disadvantages – Large distances between labels and fields can make the label-field association difficult. – Cannot easily accommodate labels of very different lengths. 51
  • “Right” Layout  Advantage – Labels and fields are easy to associate because of their proximity.  Disadvantages – Ragged left edge makes it more difficult to: • Scan of all labels quickly in order to preview the form. • Find the beginning of the next label when completing the form. 52
  • “Top” Layout  Advantages – Can accommodate long labels and fields. – Provides more flexibility for localization.  Disadvantages – Page length increases and the form may require scrolling. – Labels may “get lost” between the fields. 53
  • “In-Field” Layout  Advantages – Requires the least amount of space. – Does not interfere with other elements on the page as much as the other layouts do.  Disadvantages – Sometimes it is not obvious that the fields are editable. – Fields must be at least as long as their labels, which can lead to unnecessarily long fields. – Interaction can be unclear, frustrating, and cumbersome: • If the label disappears on focus, users might miss it. • If the label does not disappear on focus, users have to delete it. 54
  • “Flow” Layout  Advantages – Labels and fields are easy to associate because of their proximity. – Labels are easy to scan.  Disadvantages – Form can appear cluttered and unprofessional, which may decrease the site’s credibility. – Quick scan of user inputs is difficult due to the lack of left alignment of the fields. 55
  • The Five Layouts, Again FORM 1 FORM 2 FORM 3 FORM 4 FORM 5 “left” layout “right” layout “top” layout “in-field” layout “flow” layout  We asked for familiar information that everyone should know: – Name, address, phone number(s), email address(es), and date of birth 56
  • Lab Setup Moderator’s station to view Face Tobii 1750 remote eye-tracking eye gaze in real time and camera system integrated into a 17” control eye tracking software monitor (set to 1024 x 768 px) One-way mirror Moderator Participant 57
  • Participants and Procedure  33 adults – 16 males and 17 females – Ages 22 – 61 (M = 29) – From the Chicago area  Individual, 10-minute long sessions  Within-subjects experimental design – Every participant completed all five forms. – Layouts were presented in a counterbalanced order.  At the end, participants viewed all 5 layouts and rated them in terms of: – Ease of filling out – Visual appeal 58
  • Findings Outline  Performance – Form completion time – Errors • Corrected • Uncorrected  Eye movements – Number and distribution of fixations – Average fixation duration  User satisfaction – Ratings of ease of filling out – Ratings of visual appeal 59
  • Form Completion Time Time measured from the form appearance on the screen to its submission. Form Completion Time (s) 80 60 Participants Form 1 (left) spent less than a minute per form. Form 2 (right) 40 Form 3 (top) Form 4 (in-field) Form 5 (flow) 20 54.2 55.0 58.8 59.3 54.8 0 There were no differences between the form layouts in any of the error categories All layouts were (p > .05). equally fast (p > .05) 60
  • Number and Distribution of Fixations FORM 1 FORM 2 FORM 3 FORM 4 FORM 5 “left” layout “right” layout “top” layout “in-field” layout “flow” layout 61
  • Number and Distribution of Fixations Indicates scanning efficiency. Number of Fixations 100 On average, partici pants made 77 80 – 94 fixations (min 40 s) per Form 1 (left) form. 60 Form 2 (right) Form 3 (top) 40 Form 4 (in-field) Form 5 (flow) 20 92 94 77 90 91 0 There were no Form 3 had the least fixations of differences between all forms (p < .05). Forms 1, 2, 4, and 5. 62
  • Number and Distribution of Fixations  To understand why Form 3 had fewer fixations than the other forms, we decided to focus on fixations on the fields and labels as opposed to fixations on the entire form. 63
  • Number and Distribution of Fixations Form 3 had fewer fixations on white space than the other four forms (p < .005). Number of Fixations 100 Form 1 (left) Form 2 (right) Form 3 (top) 80 Form 4 (in-field) Form 5 (flow) 60 40 20 92 94 77 90 91 48 52 51 51 50 42 41 25 36 38 0 Entire Form Only Labels & Fields White Space 64
  • Number and Distribution of Fixations …can be explained by… Number of Fixations 100 Form 1 (left) Form 2 (right) Form 3 (top) 80 Form 4 (in-field) Form 5 (flow) 60 40 20 92 94 77 90 91 48 52 51 51 50 42 41 25 36 38 0 Entire Form Only Labels & Fields White Space Form 3 had the least fixations because participants did not look at white space as much as they did on the other forms. 65
  • Number and Distribution of Fixations  Form 3 had a focused top-down scan pattern, and in terms of scannability, was the most efficient.  Fewer fixations were “wasted” on white space 66
  • Average Fixation Duration Indicates difficulty in extracting and processing information (due to higher information Average Fixation Duration (ms) density, ambiguity, or 350 complexity). Average fixation duration was ~one quarter of 300 a second. 250 Form 1 (left) Form 2 (right) 200 Form 3 (top) 150 Form 4 (in-field) Form 5 (flow) 100 50 236 263 294 298 262 0 Forms 3 & 4 had longer fixations than the other forms (p < .05). 67
  • User Ratings: Ease of Filling Out & Visual Appeal  Using a scale from 1 to 10, please rate each of the forms you filled out based on how easy they were to fill out:  Using a scale from 1 to 10, please rate each of the forms you filled out based on how nice they look / how visually pleasing they are: 1 2 3 4 5 6 7 8 9 10 Very Very Difficult Easy FORM 1 FORM 2 FORM 3 FORM 4 FORM 5 “left” layout “right” layout “top” layout “in-field” layout “flow” layout 68
  • User Ratings: Ease of Filling Out & Visual Appeal User Ratings 10 The two scales have a different loser. 7 Form 1 (left) Form 2 (right) Form 3 (top) 4 Form 4 (in-field) Form 5 (flow) 8.5 8.9 7.4 4.9 6.4 8.3 8.6 6.1 5.1 4.2 1 Form 5 was considered the least visually appealing Ease of Filling Out Visual Appeal (likely due to the lack of field alignment). Form 4 received the lowest ratings for ease of use (likely due to its label impermanence). 69
  • Summary of Findings PERFORMANCE: All EYE MOVEMENTS: Form EYE MOVEMENTS: forms were completed 3 had fewer fixations than Forms 3 & 4 had the in the same amount of the other layouts because longest fixations due to time and with similar fewer fixations were higher information accuracy. “wasted” on white space. density per fixated area. FORM 1 FORM 2 FORM 3 FORM 4 FORM 5 “left” layout “right” layout “top” layout “in-field” layout “flow” layout SATISFACTION: Forms 1 & 2 were SATISFACTION: SATISFACTION: rated as the easiest to fill out and the Form 4 was rated as Form 5 was rated as most visually pleasing. the most difficult. the least appealing. 70
  • Illustration of…  Pure user testing methods Ficticious Patient – No differential findings  Eye tracking – Teased out form 3 – “Top”  Preference / satisfaction data – Visual appeal ruled out Form 5 - Flow – Satisfaction eliminated Form 4 – Inline  Pragmatics / style allow selection of one of three styles (left, right, top) © May 28, 2009 – Proprietary and Confidential 71
  • Design Can Be Fraught With Usability Issues Bad screen Layout Wrong Mental Model Poor use of Language Inefficient Navigation 72
  • Usability in Healthcare Technology Tangible Financial Benefits of Usability User Centric repeatedly sees and the data support  Reduced development costs  Increased user productivity  Decreased user errors  Increased market acceptance  Increased satisfaction  Decreased training costs  Decreased user support costs Roger Pressman, Software Engineering: A Practitioner’s Approach, McGraw-Hill Support calls reduced by one third for canterburyofnz.com 73
  • So How Does Usability Fit in HIT? 74
  • What Can Usability Do For HIT?  Measure  Design – Where is current performance? – Design user interface architecture • Baseline key activities – Develop UI workflows • Understand errors • Measure satisfaction – Wire frame screens – Where can automation improve on current practices?  Iteratively Test – Test early with real users – Test often  Define User Needs – Redesign – Interview key users – Translate those to user interface  Commitment to Performance requirements Objectives 75
  • Representative Design / Test Cycle 76
  • STEEP and Usability Benefits  Improved Safety – Reduce errors  Timeliness – Improve comprehension and decision making  Effectiveness – better decision making, reduced errors, improved adoption  Efficiency – Faster comprehension and decision making, reduced learning time  Equitable – Improved adoption…  Patient – Centered – Better decision making, reduced errors 77
  • What Would Make a Successful Project Team Clinician-Centered Design Practice Involves:  Clinical Staff (Users) – Representatives from all impacted groups  Information Technology Team – Business analysts – Programmers  Usability Team – Researchers – Designers  Visual designer 78
  • Improving Usability in Healthcare Technology Understand the business objectives and constraints, content, and users. Business/Context  Understand the business objectives and constraints, technological constraints, resources, culture and politics IT Professionals Content  Understand the quantity and the quality of the content Users Healthcare Usability Experts  An effective information architecture Professionals must reflect the way users think. Expand the “sweet spot” 79
  • Barriers to implementing Electronic Health Records (EHRs) 81
  • Implementing EHRs: At what cost? 82
  • How does this happen? Content analysis of RFPs  Analyzed 47 RFPs and Selection Guidelines for EHRs  32 out of 47 RFPs made no mention of usability  Of the 15, where usability is mentioned, descriptions are often vague: – “Ease of use” is mentioned in the selection guideline – “Demonstrations are evaluated on intuitiveness and usability.” – “Ease of Use (minimizes typing, is intuitive, simple layout)” is given a suggested 8% weight in the vendor selection criteria – “General usability” 83
  • Evaluating the usability of EHRs during the procurement process How to evaluate the usability of EHR systems: 1. Identify the key user groups 2. Identify critical and frequent tasks 3. Benchmark key task efficiency 4. Estimate usability 5. Measure usability 84
  • Key Process Element #1: Identify the key user groups  Users who will be doing the critical tasks – Physician – Nurses – Administrative staff 85
  • Measuring usability of EHR systems User Groups: Physicians | Nurses | Administrative Measuring Usability Relative to Goals Effectiveness Efficiency Satisfaction Unassisted task Maximum user time (not Post-task ratings on 5- Task completion rate of: machine time) of: point usability scale: Task A Task B Task C 86
  • Key Process Element #2: Identify critical and frequent tasks  Critical tasks and frequent tasks will help prioritize EHR features.  ROI is maximized by: – Task efficiency – Effective training 87
  • Measuring usability of EHR systems User Groups: Physicians | Nurses | Administrative Measuring Usability Relative to Goals Effectiveness Efficiency Satisfaction Task Unassisted task Maximum user time (not Post-task ratings on 5- Create an appointment for a new patient Check patient eligibility for Medicare Enter patient vitals 88
  • Key Process Element #3: Benchmark current task efficiency  Benchmark current critical and frequent tasks in order to: – Show that the EHR system produces better task efficiency than paper methods – Show improved efficiency over an existing electronic system – Project ROI for a given period  Benchmark by: – Directly measurement – Expert estimation 89
  • Measuring usability of EHR systems User Groups: Physicians | Nurses | Administrative Measuring Usability Relative to Goals Effectiveness Efficiency Satisfaction Task Unassisted task Maximum user time (not Post-task ratings on 5- Goal: 100% Goal: 2 mins Goal: 4.00 Create an appointment for a new patient Goal: 100% Goal: 1 min Goal: 4.00 Check patient eligibility for Medicare Goal: 100% Goal: 30 secs Goal: 4.00 Enter patient vitals 90
  • Key Process Element #4: Do First Pass Assessment  Estimating usability, as a first pass, helps to narrow the field of available EHR systems.  Estimation can be obtained via: – Cognitive walkthrough – Expert testing – User Performance Modeling GOMS 91
  • Measuring usability of EHR systems User Groups: Physicians | Nurses | Administrative Measuring Usability Relative to Goals Effectiveness Efficiency Satisfaction Task Unassisted task Maximum user time (not Post-task ratings on 5- Goal: 100% Goal: 2 mins Goal: 4.00 Estimated Estimated Estimated Create an appointment EHR A: 90% EHR A: 4 mins EHR A: 3.00 for a new patient EHR B: 100% EHR B: 1.5 mins EHR B: 4.50 Goal: 100% Goal: 1 min Goal: 4.00 Estimated Estimated Estimated Check patient eligibility EHR A: 95% EHR A: 2.5 mins EHR A: 4.00 for Medicare EHR B: 100% EHR B: 1 min EHR B: 5.00 Goal: 100% Goal: 30 secs Goal: 4.00 Estimated Estimated Estimated EHR A: 90% EHR A: 45 secs EHR A: 3.25 Enter patient vitals EHR B: 100% EHR B: 30 secs EHR B: 4.75 92
  • Key Process Element #5: Measure usability  Measure the usability of the remaining EHR candidates.  Measuring usability can be obtained via: – Usability testing – Log files – Observation – Questionnaires  Experts are required to measure the usability of EHR systems. 93
  • Measuring usability of EHR systems User Groups: Physicians | Nurses | Administrative Measuring Usability Relative to Goals Effectiveness Efficiency Satisfaction Task Unassisted task Maximum user time (not Post-task ratings on 5- Goal: 100% Goal: 2 mins Goal: 4.00 Estimated Estimated Estimated Create an appointment EHR A: 90% EHR A: 4 mins EHR A: 3.00 for a new patient EHR B: 100% EHR B: 1.5 mins EHR B: 4.50 Actual Actual Actual EHR B: 100% EHR B: 2 mins EHR B: 5.00 Goal: 100% Goal: 1 min Goal: 4.00 Estimated Estimated Estimated Check patient eligibility EHR A: 95% EHR A: 2.5 mins EHR A: 4.00 for Medicare EHR B: 100% EHR B: 1 min EHR B: 5.00 Actual Actual Actual EHR B: 100% EHR B: 45 secs EHR B: 5.00 Goal: 100% Goal: 30 secs Goal: 4.00 Estimated Estimated Estimated EHR A: 90% EHR A: 45 secs EHR A: 3.25 Enter patient vitals EHR B: 100% EHR B: 30 secs EHR B: 4.75 Actual Actual Actual EHR B: 100% EHR B: 30 secs EHR B: 5.00 94
  • User Centric, Inc.  User experience consulting firm specializing in user research and user-centered design.  Experience in the healthcare industry: – Medical and pharmaceutical devices and applications – Patient and professional reference materials – Drug dispensing devices – Prescription and medical labeling – Patient-oriented and physician-oriented websites – Medical packaging – Personal Health Records – Electronic Health Records 2 Trans Am Plaza Drive, Ste 100 Oakbrook Terrace, IL 60181 +1 630.320.3900 For papers and presentations: www.UserCentric.com © May 28, 2009 – Proprietary and Confidential 95