Medical Device Usability: Polly Shelton presents at UK UPA (Usability Professional’s Association) and London Ergonomics Group (LEG) joint.

UKUPA and LEG, January 2012

Medical Device Usability

| Polly Shelton
| Human Factors Consultant

Page 1 COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk

pdd
Consumer

v

Industrial
Design

Medical

FMCG

Services


History

Computer Aided
Design & Manufacture
Industrial Design
& Prototyping Engineering &
Analysis

1980 1990


Innovation planning

Human factors Design Insight

2000 2010


Medical devices group
Radiation Medical
does meter oxygen

Urine meter

Needle safe
catheter IV Pump

Wireless Inhaler dose
stethoscope counter

Heart-rate
monitor

Auto
injector
device Pen injector


What is a medical device?
A product used to diagnose, treat or monitor a medical condition

Device classes (FDA)
• Class I – general controls
• Class II – general controls w/ special controls
• Class III – general controls & premarket approval
• & Combination products.

These products should all undergo a
usability engineering approach

Depending on the class, more or less HFE required

Class depends on complexity, use characteristics, risk
to patient & amount of regulatory control needed


Human Factors and Usability
International/EU FDA (USA)
ISO/IEC 62366 ANSI/AAMI HE:75

Ensuring that products are developed for safe
and effective use and that potential use-errors

have been mitigated.


History
International FDA (USA)

Usability engineering process Medical device manufacturers
must demonstrate that all
International (EU & FDA) potential use-related hazards in
6 pages (+80page annex) their devices have been
identified, tested, and mitigated
Helps “design in” usability
and “design out” use error The device must be shown to be
Used with ISO 14971 usable by the target population
in the intended environment.


History
International FDA (USA)
A virtual encyclopaedia that Provides design principles
provides human factors and practical guidelines
engineering design guidance,
25 chapters, 465 pages
case studies, and checklists
Officially recognised by FDA
in October 2010
FDA draft guidance document
June 2011


History
Human factors engineering guidelines and preferred
1988 AAMI HE: practices for the design of medical devices

Human factors engineering guidelines and preferred practices
1993 AAMI HE:48 for the design of medical devices

1999 Institute of Medicine Report : to Err is Human

2001 ANSI/AAMI HE:74 Human factors design process for medical devices FDA

Medical devices – Application of usability engineering March
2007 ISO/IEC 62366 to medical devices 2010, EU

Human factors design process for medical devices
2009 ANSI/AAMI HE:74 (reaffirmed)
FDA

November
2009 ANSI/AAMI HE:75 Human factors engineering – Design of medical devices
2010, FDA


Process flowchart: IEC/ISO 62366


COMMERCIAL IN CONFIDENCE | PDD Group Ltd 2011 ISO 9001 & ISD 134BS registered | www.pdd.co.uk

Usability & Risk management
How do they interact?

ISO 62366 has a complex flowchart
showing how.


Usability engineering process

Insight Iteration Validation


Expert
interviews
Task analysis

Iteration Validation

Insight

Desk research

Ethnographic Contextual
research enquiry


Concept
Anthropometry &
refinement
biomechanics

Insight Validation

Iteration

Interaction design
Formative Use error risk
& prototyping
usability testing analysis


Usability goals and Residual risk
acceptance criteria

Insight Iteration

Validation

Summative
usability testing
Regulatory
submission


Usability engineering process

Expert
Formative usability
interviews
testing
Anthropometry &
biomechanics
Usability goals and
Ethnographic acceptance criteria Residual risk
research

Task analysis Concept
refinement

Insight Iteration Validation
Regulatory
Summative
submission
usability testing

Desk research

Contextual Use error
enquiry Interaction design risk analysis
& prototyping


Ethnographic research
Observation, shadowing, journaling, contextual enquiry, journey mapping

• Gain a holistic view of your users, the
task and the context of use
• Identify opportunities and pain points
• Deep-dive into users needs and
experiences
• Create personas, scenarios,
journey/experience maps


Desk research
User and task characteristics, human factors design guidelines

• Understand your stakeholders
and the problem
• Review similar products
• Gather anthropometric and
strength data
• Source best practice design
guidelines

Development & refinement
Prototyping, cognitive walkthrough, heuristic review, formative usability tests

• Involve users early and often
• Test concepts
• Identify design problems early
• Understand mental models and user interactions
• Involve designers and clients where possible
• Inform the design

Task analysis and user FMEA
Cognitive walkthrough, use scenarios, journey and task mapping

• Understand the task
• Identify physical, sensory and
cognitive challenges
• Identify and understand
potential risks and foreseeable
use error
• Control and mitigate use error
risk through design

Validation & residual risk
Formal summative testing against quantitative usability goals

• Identify critical task scenarios
• Validate risk mitigation effectiveness
through testing with users
• Error analysis
• Root cause analysis
• Residual risk analysis and mitigation
• Follow-up testing
• Regulatory submission

Medical device user testing
• Why do usability testing?
• Formative vs. summative
• Recruiting participants
• Legal and ethical considerations
• Task selection
• Setting usability goals
• Test environment
• Set-up and pilot
• Moderating sessions
• Providing assistance
• Providing training
• Testing instructions
• Data collection and error logging
• Error analysis and classification


Why do usability testing?

Increase device sales,
Meet device
customer loyalty,
regulators’
reduce customer
expectations
support

Improve ease-of-use, Improve safety and
ease of learning and reduce medical
user experience device use errors


For medical devices, poor usability
is literally a matter of life or death


Some stats 100
medical
device alerts
“One third of medical device reports issued
each year involve use-error and human (MHRA,2010)
factors are inherent in virtually all and 403
device-related incidents.” FDA Manufacturer’s
10,280 Field Safety
Corrective
adverse Actions
incident Deaths and undertaken
reports in UK serious injuries
2010 (MHRA) involving
301 medical devices
An estimated15-
fatalities
up 29% 20% of all errors
last year
2,383 due to
mechanical
serious failure; 60-80%
injuries human error*

*Institute of Medicine Report 2000, & Perrow 1984)

Formative vs. summative tests
What’s the difference?

Formative Summative
• Part of the iterative design process • Determines if a finished device is vulnerable to
• Prototypes of any fidelity potential harmful use errors
• More explorative, informing the design • Production equivalent devices
• Identifies an evolving device’s strengths • Tasks must include primary operating functions
and weaknesses and all safety-related tasks
• Extensive discussion of likes and dislikes • Actual or simulated use environments
• Preliminary usability objectives • Focus is on use-safety
• Small samples (5-8 users per group) • Effectiveness, efficiency, user satisfaction & ease
• User research and marketing questions of user learning also evaluated
can easily be incorporated • Usability goals and acceptance criteria
• Modest size report including design • Record use errors, close calls and root causes
recommendations • Can be large sample sizes (25+ per user group)
• 30-90 minute sessions • Training and final IFU may be included
• 90 minute + sessions


Recruiting participants
• Reflect the actual user population
• Define inclusion and exclusion criteria within each user group
• Include users with physical, cognitive and sensory impairments
• Consider co-morbidities
• Use a medical specialist agency with BHBIA/EPhMRA
• Allow at least 4-6 weeks for recruitment
• Over-recruit by 10%
• Be prepared for difficult recruits and arranging in-home testing
where necessary
• Beware of “professional” participants
‐ ask for ID / registration numbers where possible
‐ patients to bring current medication/ a recent prescription
• Pilot your recruitment materials
• Test in different markets


Legal and ethical considerations
• Ethics committee / IRB approval
• BHBIA and EPhMRA codes of conduct
• Data protection act, 1998
• ABPI code of conduct
• Adverse events and pharmacovigilance
• Country-specific requirements
• Video recording, streaming and transfer
• Incentives (BMA guidance):
‐ minimum level
‐ proportionate to time involved
‐ appropriate to respondent type and task nature
‐ air market value of the services provided
‐ vouchers and not cash for children
• Testing with children


Task selection
• If possible, test every user task!

• If not, prioritize by:
‐ Safety Critical tasks, and
‐ Frequent tasks

• Address worst case scenarios where possible

• Link tasks to risk management


Usability goals
• Derived from:
‐ Risk analysis
‐ Primary operating functions
‐ User interface requirements
‐ Worst case scenarios
• Define acceptance criteria by criticality and risk level

But….
• Don’t base the success of your test on the achievement of
usability goals against quantitative acceptance criteria!
• Always conduct:
‐ root cause analysis
‐ follow-up risk analysis


Test environment
• Simulate the real context of use where possible
– Lighting levels
– Thermal environment
– Noise
– Furniture
– Other equipment
– Potential hazards
– Working space
– Multi-tasking and distractions

• Consider testing devices in-field
– In the home
– In clinical environments (e.g. ICU)

• For high risk products (e.g. infusion pumps)
consider validation with clinical evaluation.


Set-up and pilot
• Test logistics, protocol and timings
• Equipment checklist
• Video recording
‐ Define work area
‐ If possible, use two cameras to capture detailed
interactions and scene-view
‐ Make sure the audio is ON!
‐ Remote control
• Live streaming
• Seating arrangements:
‐ Sit behind / next to participant
‐ Observer out of field of view
• Consider recruiting “real” pilot participant
• Pilot data collection materials and analysis


Moderating sessions
• Protect participants’ rights
• Be professional and build trust
• Create rapport; but stay in charge
• Ask open questions
• Don’t be a robot! – remember the script is a guide
• Be unbiased
• Be consistent
• Let the participant speak
• Take care with “think-aloud”
• Give participants time
• Let participants struggle… but keep them safe
• Watch non-verbal cues
• Probe where appropriate to understand root causes

• Good seminar from UIE and Beth Loring
http://www.uie.com/events/virtual_seminars/good_moderating/


Providing assistance
• An assisted task is a task failure
• Observing participants struggle, commit use errors or near
misses provides insight
• Only assist as a last resort
- safety risk
- repeated pattern of unproductive behaviour
- task abandonment
- after pre-established time-limit
- in order to continue
• Provide appropriate levels of assistance
• Pre-establish prompts for consistency
• All interventions should be reported


Training

• Simulate the real-life training scenario
• Consider including a retention interval
• Simulate the worst case scenario where
appropriate
• Consider including some untrained users

Do not…
• Deliver better than normal training!
• Ask trainees to study for their test!


Testing instructions
• Simulate the anticipated situation-of-use
• Ask users to behave naturally
- refer to IFU/ quick guide as they would at home
• Follow-up to address IFU in detail
- test all risk mitigations and warnings
• Evaluate and rate specific attributes e.g.
‐ Clarity of text
‐ Terminology
‐ Pictograms
‐ Structure & flow
‐ Text size and legibility
‐ Ease of use
‐ Ease of finding information
‐ Completeness
‐ Density of information


Data collection and error logging
• Collect data concurrently
• If possible, have 2 people taking notes
• Debrief and align data post-session
• Review video where required Task log and Excel spread sheet

Data:
• Success rate
• Use errors, close calls, operational
difficulties and their root causes
• Assistance provided
• Verbatim comments
• Other observations and behaviours
• Responses to questions and ratings


Error analysis & classification
• FDA are interested in understanding the root causes of errors
• Probe to understand observed errors
• ISO 62366 includes a taxonomy (adapted from James Reason)
• Close calls and operational difficulties should also be recorded


Case Study: IV infusion set
• Pre-summative test
Changes
• 22 participants
recommend
• 3 hour sessions to design &
• 13 primary usability goals instructions
• 40 secondary usability goals

Training
needs
11 primary &
identified
14 secondary
usability
goals
10 15 Contextual
unmet major use additional insights
errors risks gathered
observed identified
to FMEA

Case Study: easypod device Improved
ease-of-use,
ease-of-dosing,
comfort,
confidence
& safety

Ethnographic
fieldwork ROI: drug
identified sales rose
Full User need
Centred 7.5% in the
Design first year to
approach €58m
User
requirements
specification


Further reading
Some good resources to get started


Get in contact
and keep updated via …


Medical Device Usability: Polly Shelton presents at UK UPA (Usability Professional’s Association) and London Ergonomics Group (LEG) joint.

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (11)

Similar to Medical Device Usability: Polly Shelton presents at UK UPA (Usability Professional’s Association) and London Ergonomics Group (LEG) joint.

Similar to Medical Device Usability: Polly Shelton presents at UK UPA (Usability Professional’s Association) and London Ergonomics Group (LEG) joint. (20)

Recently uploaded

Recently uploaded (20)

Medical Device Usability: Polly Shelton presents at UK UPA (Usability Professional’s Association) and London Ergonomics Group (LEG) joint.