Applying Usability and UCD Methodologies to Achieve Meaningful Use

Applying Usability and User-Centered Design
Methodologies to Achieve Meaningful Use 
June 9, 2010, HIMSS Virtual Conference 

Lisa Battle, Jasmin Phua & Duane Degler 
Design for Context

Conflict of Interest Disclosure
Lisa Battle
User-Centered Design Lead

Jasmin Phua
User Experience Researcher & Designer
HIMSS EHR Usability Taskforce member

Duane Degler
User-Centered Design Strategist & Semantic Web Expert

Consult for
and have no real or apparent conflicts of interest to report.
Copyright © 2010 Lisa Battle, Jasmin Phua & Duane Degler Slide 2

Agenda

•  Meaningful Use Overview
•  Role of Usability in EHR Meaningful Use
•  Build-in Usability through User-Centered
Processes
•  What is user-centered design
•  Evaluating product usability
•  Designing for usability


Objectives

This education session aims to help you:

•  Understand the role of usability in helping
achieve meaningful use

•  Identify methods of evaluating usability

•  Apply user-centered design methodologies to
incorporate end-user feedback


“It’s not about technology,
it’s about transforming
healthcare.”
Joshua Seidman, Acting Director, Meaningful Use
Office of Provider Adoption Support, ONC


What is all this Meaningful Use buzz?

Through the provisions of the American Recovery
and Reinvestment Act (ARRA; Recovery Act) of
2009, the Centers for Medicare & Medicaid Services
(CMS) will provide incentive payments for the
meaningful use of certified electronic health record
(EHR) technology.

Source: Medicare and Medicaid Programs; Electronic Health Record Incentive Program; Proposed Rule. January 13, 2010


Who qualifies?

Eligible professionals (EPs), eligible hospitals,
critical access hospitals (CAHs) for:

•  Efforts to adopt, implement, or upgrade certified
EHR technology,
or
•  Meaningful use in first year of participation and
for demonstrating meaningful use during each of
5 subsequent years.



How is Meaningful Use measured?

Stage 1 Stage 2 Stage 3

incremental process
2011 2016

•  Stage 1 has been defined; meaningful use
objectives and certification criteria provided by
CMS and ONC.
•  Stages 2 & 3 have not been define yet but policy
priorities have been painted in broad strokes.
•  Different measures for eligible professionals (EPs)
vs. hospitals.
Sources:
Medicare and Medicaid Programs; Electronic Health Record Incentive Program; Proposed Rule. January 13, 2010.
Health Information Technology: Initial Set of Standards, Implementation Specifications, and Certification Criteria for
Electronic Health Record Technology; Interim final Rule. January 12, 2010.


Meaningful Use: Stage One

Stage One (2011-2012) focuses on:
•  Electronically capturing health information in a coded format,
•  Using that information to track key clinical conditions,
•  Communicating that information for care coordination purposes,
•  Implementing clinical decision support tools to facilitate disease and
medication management,
•  Reporting clinical quality measures and public health information.

Defined measures for:
•  Eligible professionals (EPs): 25 measures
•  Eligible hospitals: 23 measures



Why Meaningful Use?

Anticipated health policy outcomes for
meaningful use of EHR technology are:

•  Improvements in quality, safety, efficiency, and
reductions in health disparities,
•  Engagement of patients and families in their
health care,
•  Improvements in care coordination,
•  Improvements in population and public health,
•  Adequate privacy and security protections for
personal health information.

Example: Meaningful Use Criteria

Health outcomes Improving quality, safety, efficiency, and reducing
policy priority health disparities.

Use evidence-based order sets and CPOE.
Care goals
Apply clinical decision support at the point of care.
EHR mea
ningful u
se
Implement drug-drug, drug-allergy, drug-formulary
Objectives checks. (same for eligible professionals and hospitals)

Eligible professional/hospital has enabled this
Measures
functionality.


Role of Usability in Meaningful Use

Achieving meaningful use requires successful
implementation of certified EHR technology.

Efficiency People get things done The technology does not
quickly and productively get in the way

They get the info they need,
Effectiveness complete work accurately They don’t make mistakes
and achieve their goals

They feel confident and
Satisfaction They are not frustrated
pleased

Usability Quality attribute defined in ISO 9241, Part 11

Why does usability matter?

Good usability can help...

increase reduce
Patient and staff safety Training time
Productivity and accuracy Error rates
Staff morale Staff turnover
Customer loyalty Product liability
Competitive advantage Customer support


14

Why does usability matter?

Risks of poor usability
•  Endangers patients
•  Increases adverse events
•  Information overload leading to erroneous
decisions
•  High costs of training, errors, rework
•  Increased product & practice liability
and last but not least...
Barrier to EHR implementation and adoption


Poor Usability = Adverse Events

Over a 2 year period, voluntary reporting of adverse events resulting
from health IT malfunctions to FDA found:

260 reports with potential for patient harm
44 reports of injuries
6 deaths
Usability-related adverse events:
The user documented
A sleep lab’s workstation BEST EMR SYSTEM
activities in the task list for Task List
software had confusing John Saint
one patient and used the John Smith 1. _______
user interface, which led to
“previous” or “next” arrows Janet Smith 2. _______
the overwriting and 3. _______
to select another patient L. Smite
replacement of one Paul Smote
< prev
chart, the patient’s task list next >
patient’s data with another
displayed for second
patient’s study.
patient.
How many are unreported or caught before they become serious problems?
Source: Jeffrey Shuren, Director of CDRH, FDA. Testimony to ONC Health IT Policy Committee, February 25, 2010.


Poor Usability=EHR Adoption Barrier

Administrators, clinicians, CIOs, CMOs, and policymakers listed
Top 10 barriers to EHR implementation:
# 10 Usability
Hard to use and not well engineered for clinician workflow.

# 9 Politics/naysayers
Every organization has a powerful clinician or administrator who is
convinced that EHRs will cause harm, disruption, and budget disasters.

Solvable # 8 Fear of lost productivity

with good
Concerned they will lose 25% productivity for 3 months after
implementation.

usability # 7 Computer illiteracy/training
Many clinicians are not comfortable with technology; often reluctant to
attend training sessions.

# 6 Interoperability
Applications do not seamlessly exchange data for coordination of care,
performance reporting, and public health.

Source: John Halamka, CIO, Beth Israel Deaconess Medical Center during “Leadership Strategies for Information Technology
in Health Care” class at Harvard. February 1, 2010. http://geekdoctor.blogspot.com/2010/02/top-10-barriers-to-ehr-
implementation.html

Poor Usability=EHR Adoption Barrier

Administrators, clinicians, CIOs, CMOs, and policymakers listed
Top 10 barriers to EHR implementation:
# 5 Privacy
Significant local variation in privacy policy and consent management
strategies.
# 4 Infrastructure/IT reliability
Many IT departments cannot provide reliable computing and storage
support, leading to EHR downtime.

Barriers # 3 Vendor product selection/ suitability
mitigated Hard to know what product to choose, particularly for specialists who
have unique workflow needs.

with user- # 2 Cost
the stimulus money does not flow until meaningful use is achieved. Who
centered will pay in the meantime?

processes #1 People
Hard to get sponsorship from senior leaders, find clinician champions,
and hire the trained workers to get the EHR rollout done.

Source: John Halamka, CIO, Beth Israel Deaconess Medical Center during “Leadership Strategies for Information Technology
in Health Care” class at Harvard. February 1, 2010. http://geekdoctor.blogspot.com/2010/02/top-10-barriers-to-ehr-
implementation.html

Biggest EHR Usability Problem

HIMSS EHR Usability Pain Points Workgroup conducted a
survey on the specific factors that resulted in poor usability
(catch their presentation on Thursday, June 10).

Workflow is the overwhelming problem
in almost all facets

“Must view many areas to capture the entire patient’s story”
“Too much info in too many different places, getting lost and
overwhelming”
“Doesn't match clinician thought process”

“We failed to focus on the most
important part of the decision—the
human/computer interface....we didn’t
listen to our guts on the design of the
computer screen that we would have to
look at for hours on end. It is about
functionality and workflow.”
-Joseph G. Cramer, MD
“Bought Wrong EMR” in Medical Economics Magazine
February 5, 2010, pp 28-30


Good Usability = a Good “Fit”

•  “Fit” between the object and …
•  Its purpose
•  Human mental and physical capabilities
•  Target audience
•  Environment in which it will be used
•  Tasks it will be used for


Usability by Design

•  Usability does not happen by chance
•  It can’t be “painted on” at the end
•  It can’t be achieved through testing alone...
but usability testing gives
great insights as to
improvements needed!


How to build-in usability?
Usability Test
Common problem
This isn’t
Timeline
what I
expected…

Release Date

Brainstorm
design ideas Usability
Observe the Create Revise designs Test
work designs

Timeline
Timeline

Fantastic! And
Get user Usability test we don’t even
feedback need any
Interview training
Release
users
Date

What is User-Centered Design?

Industry best practice for creating usable
products from the beginning
ISO 13407

Perspective Discipline based on
human factors
affordances
working memory
mental models predictability
feedback Fitts’ law

Toolkit of methods & guidelines Risk mitigation strategy
task analysis
usability testing

ethnographic studies walkthroughs

usage tracking
evidence-based design


How does UCD help with Meaningful Use?

Participatory process that
involves true end-users and
other key stakeholders.

When done right...
•  Meaningful use requirements are met in a way that fits
with how all staff members truly work.
•  Your EHR has features that your staff actually need,
rather than unnecessary “cool ideas”.
•  Finding clinician champions and gaining buy-in from
key stakeholders is easier because it is an inclusive
process. (helps solve the “people” barrier!)

How does UCD help with Meaningful Use?
Practice predicated on
getting into users’ heads to EHR
understand specific:
•  Characteristics
•  Tasks and goals
•  Context in which they do work

When done right...
•  Realistic, productive workflows for meaningful use
requirements are identified.
•  Scenarios where meaningful use criteria are applicable
and can be met are comprehensively identified.
•  The impact of “meaningful use” implementation on
staff duties and responsibilities is anticipated.

User-Centered Design Process

Understanding Designing a
the needs solution that works

Who are the users? Brainstorm
Design
What are their tasks and ITERATION

goals? Test
What situations bring with users
them here? Best practices for
What are their usable design
expectations? Progressive refinement

Multidisciplinary collaboration


Always start with goals

•  Business goals

•  Stakeholder goals

•  User goals

•  Usability goals

  Business goals include care goals and meaningful use
objectives as defined in the proposed rules.


UCD: Observe & Analyze

Observe
&
Analyze

Evaluate Envision
& &
Refine Design


Conduct user studies
Observe & Analyze Envision & Design Evaluate & Refine

A variety of activities that gather information about
•  Users
•  Tasks
•  Context of use

Users are the actual people who use the product.
Users are not:
•  The CEO
•  Their organizations and managers
•  You and the development team
•  Your public affairs or marketing department

User research methods

•  Interviewing •  Analysis of emails,
requests, or issues
•  Site visits/contextual inquiry
•  Conferences, training, user
•  Usability testing
group meetings
•  Surveys
•  Usage logs
•  Focus groups
•  Search logs

Not all at once—choose the techniques that fit best with your project


Different user groups have different needs


General practice Emergency room Patient Insurance
physician unit and billing

When researching user needs, gather requirements from:
•  Direct users
•  Indirect users, e.g. billing
•  Others who have contact with users, e.g. caregivers

UCD in Meaningful Use: User Needs

Meaningful use objective:
Implement drug-drug, drug-allergy, drug-formulary checks.

Certification criteria example:
Automatically and electronically generate and indicate in real-time, alerts
at the point of care for drug-drug and drug-allergy contraindications
based on medication list, medication allergy list, age, and CPOE.

Who will use these drug-drug and drug-allergy checks?
Do these user groups have the same needs?
Which user group can override alerts? Should they?


Document user needs in Personas

Personas are:
•  Personal, composite sketches
of real users
•  Discovered through user
research
•  Representative of typical
users, not edge cases

Used to:
•  Debunk false assumptions
•  Help envision users and
design what is best for them
•  Keep a focus on the user
throughout the project

Analyze user tasks

Tasks are specific activities that people need to accomplish.
•  Examples:
•  Complete all prescription refill requests
•  Discuss the MRI results with the neurologist
•  Call Ms. Nelson to discuss test results
•  Many tasks include both information and action
Task Information Action

Discuss MRI Read over patient record, Schedule discussion
results with the concentrating on problem time with neurologist
neurologist list. Consider diagnosis
possibilities.


How to analyze tasks

•  List them
High

•  Match them to user groups
Medium

•  Prioritize them based on

Frequency
Low

•  Frequency
•  Criticality of failure
•  Break them down into Low Medium High

Criticality

•  their component parts, and/or
•  the sequence of steps involved
•  Write stories that illustrate them (scenarios)

Documenting tasks in scenario form

Scenarios are realistic narrative
descriptions of activities that users
engage in, detailed enough that
design implications can be inferred
—Based on J.M. Carroll, 1995

How it works:
•  Write the story of the work from
the user’s perspective
•  Share the story with team to help
them visualize how tasks occur in
the actual work environment
•  Conduct walkthroughs of the
proposed designs using these
scenarios

Example: UCD in Physician Environment

Type of Total No. per Physician
No.
Would want to understand:
Service (day)
Telephone call 21, 796 23.7
•  Typical amount of time spent
Laboratory report 17,794 19.5
during each service interaction
Visit 16,640 18.1 •  Most beneficial patient
E-mail message 15,499 16.8 interaction
Consultation report 12,822 13.9 •  Other categories of services e.g.
Prescription refill 11,145 12.1 health plan correspondence,
Imaging report 10,229 11.1 FMLA forms
4 FTE physicians, each working 50-60 hrs/wk, 230 days/yr. •  Types of interruptions and
Frequent Infrequent frequency
(e.g. daily, weekly) (e.g. quarterly, annually)
•  Tasks that are queued up to be
One thing at a time Several at a time addressed all at once vs.
(single) (multiple)
piecemeal
Source: What’s Keeping Us So Busy in Primary Care? A Snapshot from One Practice by Richard J. Baron, M.D., New
England Journal of Medicine, 362; 17, April 29, 2010S

Example: UCD in ER or ICU

Clinician is developing Patient care at individual
individual and shared How would you
automate an ER clinical and unit levels
mental model of patient
whiteboard?
Pt 1 needs CT
scan, had blood
work-up this
morning

Pt 1 scheduled for
Pt 2 doesn’t look CTC at 1500. Needs
good. May need line changed
to be intubated. beforehand

Adapted from Representing Reality: The Human Factors of Health Care Information, C P. Nemeth,M. O'Connor, M.
Nunnally, and R I. Cook Chapter 28, Handbook of Human Factors and Ergonomics in Health Care and Patient Safety

UCD in Meaningful Use: Task & Context

In what situations will these alerts come up?
•  Acute episodes / hospitalizations
• 
• 
Chronic conditions
Primary care encounters Do not use
vancomycin
X
Drug alert!

•  Preventative care
In what context will these alerts come up?
Emergency room, general practice, pharmacy. ICU System


UCD: Envision & Design

Observe
&
Analyze

Evaluate Envision
& &
Refine Design


Bridging the gap from analysis to design

MI
Humans calculate B

are good
at some …machines
things… are good at
other things.
about
rem ind me
ions
drug interact
lookup
drug a
llergie
s

To optimize the system, let each
focus on what they are good at.

EHR

“It is all about design, which we see
every day, but mostly ignore....Design of
the computer screen and the underlying
program is how our brains see the whole
picture of the patient.”
-Joseph G. Cramer, MD
“Bought Wrong EMR” in Medical Economics Magazine
February 5, 2010, pp 28-30

Design is Hard

•  You don’t get it right the first time
•  There are always trade-offs
•  Keys to success:
•  Start with a deep knowledge of your users and
their tasks
•  Generate multiple ideas up front (e.g. through
parallel design and brainstorming)
•  Iteration – walkthroughs and user feedback
•  Progressive refinement
•  Following guidelines and patterns for usable design


“Computers are strong
medicine. Done well, they are
wonderful: done poorly they
can kill people”
- Justin Starren MD
Marshfield Clinic, Wisconsin

Source: As Doctors Shift to Electronic Health Systems, Signs of Harm Emerge, by Fred Schulte and Emma Schwartz,
April 20, 2010, Huffington Post

UCD in Meaningful Use: Design

I talked to my users and stakeholders,
now magic happens!

It can still can go very wrong. For example, CPOE systems often flood
doctors with warning alerts, leading physicians to ignore them, which is
a human factor principle known as “alert fatigue” or “pop-up fatigue”.

Source: Role of Computerized Physician Order Entry Systems in Facilitating Medication Errors, Ross Koppel; Joshua P.
Metlay; Abigail Cohen; et al. JAMA. 2005;293(10):1197-1203


Magic happens or not...

Dangerous Drug Interaction

Warfarin x Sulfa
Potential for bleeding details

Patient currently on: Coumadin, since 03/09/2001

Consider alternatives:
MyCPOE System Patient: Jane Smith
Cephalexin
Change order
Dangerous Drug Interaction: Warfarin x Sulfa Nitrofurantoin

Patient Summary Warfarin x Sulfa interaction:
Potential for bleeding Don’t fill
Clinical Notes Fill order
Pt currently on:
Problem List Coumadin (warfarin)
Medication
-----------------------
Consults
Lab Results


Visioning

Example: Designing a new house
What are we trying to
build?
What would be best for
our users?
What will help them
accomplish their tasks?
How can we meet their
expectations?
What is the best way to
meet our goals?


Visioning

•  In user-centered design,
this involves:
•  “Blue sky” brainstorming
sessions
•  Fast, informal generation of
ideas
•  Flip charts
•  Whiteboard drawings


Parallel Design

Sketch #B
“What I really
like about this Alert is helpful and lets
idea is…?” me change the order if I
need to. I don’t need to
click through 5 screens
again to do that.
Sketch #A
Serious alerts are easy Sketch #C
to spot It lets me get to
Like being able to see more details so I
the rest of the patient’s don’t need to go
record back out to look it
up

Generates a wide range of design possibilities quickly
The full group discusses pros and cons of each

Structural Level: Organizing

How should it be organized?
What are its main sections?
What will people do in each area?
What will people expect each area
to be called?
How can we streamline the path
from one section to another?


Structural Level: Organizing

this involves:
•  Abstract prototypes
•  Sitemaps
•  Flow charts
•  Card sorts
Abstract prototype

Sitemap


Abstract Prototype

Used in discussion with stakeholders to clarify content and organization

Representing and Refining

Which things should be
near each other because
they go together?
Is this the right style of
interaction?
Does this layout support
the flow of the task?


Representing and Refining

this involves:
•  Sketches
•  Low-fidelity
prototypes
or mockups
•  Wireframes
Paper prototypes

Wireframes

Types of Prototypes

Low Fidelity High Fidelity
Good for testing: Good for testing: Good for testing:
- Concepts -  Terminology -  Visual appeal
- Organization -  Headings
-  Interactions
- Screen flow -  Navigation
-  Accessibility
- User tasks
- Main ideas

Iteration

•  Design in repeated cycles
•  Results of each cycle feed into the next cycle

Brainstorm
Design
ITERATION

Test
with users
(or walkthrough
with specialists)

•  Get input and feedback early and often
•  Prototypes don’t need to be working yet
•  Less “finished” looking, more options, elicit more feedback

UCD: Evaluate & Refine

Observe
&
Analyze

Evaluate Envision
& &
Refine Design


Many Ways to Get User Feedback

•  In addition to usability testing, you can use:
•  Informal, scenario-based walkthroughs
•  Surveys
•  Web metrics and usage tracking
•  Management information on transactions
•  Help desk log
•  Emailed feedback


User Feedback Sessions


•  Set expectations
•  Ask the user to “think aloud” and interpret what they see
•  Ask the user about realistic tasks
•  Ask the user to compare alternatives

Refer to handout for details


Testing for Usability

Get real users
Ask them to perform realistic tasks using
your system, prototype, or web site
Observe, take notes, and see what works
and what doesn’t work
If they have problems, fix them before
the release!


Types of Usability Tests

•  Formal vs informal
•  Formal: In a lab with two-way mirrors, logging
software, video cameras, observers
•  Informal: In a cafeteria, senior center, at home, or
wherever, with nothing but your prototype
•  Formative vs summative
•  Formative: Identify problems and opportunities for
improvement
•  Summative: Determine whether performance
measures were met, or to set a baseline
•  In person vs remote


Typical Measures for Usability

•  Efficiency
•  Time to complete task
•  Number of clicks
•  Number of days/hours training reduced
•  Effectiveness
•  Success rate (or completion rate)
•  Number or % of errors
•  Number of attempts before successful completion
•  Cost savings from reduced errors
•  Satisfaction
•  Number of positive and negative statements or feedback
messages received from users
•  Satisfaction scores on a survey (e.g. SUS, QUIS, ACSI)
•  Number of users who rate the system as good or excellent
•  Actual usage (number of people, % increase)

Design Walkthroughs


Walkthrough Abstract Low-Fi High-Fi
prototype Prototype Prototype
1) Scenario-Based Walkthrough
  
2) Persona-Based Walkthrough
 
3) Requirements Walkthrough
  
4) Subject-Matter Expert Walkthrough
  
5) Database Walkthrough
  
6) Information Needs Walkthrough
 
7) Usable Design Principles Walkthrough
 
8) Accessibility Principles Walkthrough
 


Is there a magic number?

We have often been asked... No,
“Is there a magic number to ther
figure out if my product e
isn’
passes/fails usability?” t.
Why?
Usability is measured by:
Efficiency Effectiveness Satisfaction
and is about balancing user needs. It is not a threshold
measure.

Is there a magic number for human clinical trials? Why?

When user requirements go awry


Takeaway: Do it early

Planning Requirements Design Development Validation Roll-out

f
C ost o ons
ti
ifica
mod

Identify usability and accessibility
needs as early as possible to reduce
costs

Takeaway: Build it in from the beginning

Involve real users early and often
Observe their actual work

Work collaboratively with a
multi-disciplinary team
Follow human factors & usable
design guidelines
Design the user interface
deliberately
Iterate the design with user
feedback


Takeaway: Start now!

Start now! You have the means...

•  Look at your paper forms and their contents:
•  What you are collecting and why?
•  How do you intend to use it when it is digital?
•  Analyze patterns of work e.g. patient requests,
repetitive fixed events, tasks everyone can do
•  Look at your current workflow. What’s optimal?
•  Scrutinize problem logs. Make sure you don’t
automate the problem source!

Better design, better healthcare

Transform healthcare through better
design
reduce increase
errors ease of use
fatigue productivity
repetitive work success rate
stress & frustration human comfort
loss of time user acceptance
training needs satisfaction


Questions?

Lisa Battle: lisa@designforcontext.com
Jasmin Phua: jasmin@designforcontext.com
Duane Degler: duane@designforcontext.com

Applying Usability and UCD Methodologies to Achieve Meaningful Use

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Applying Usability and UCD Methodologies to Achieve Meaningful Use

Similar to Applying Usability and UCD Methodologies to Achieve Meaningful Use (20)

Recently uploaded

Recently uploaded (20)

Applying Usability and UCD Methodologies to Achieve Meaningful Use