This presentation will offer practical guidance in the UX engineering of medical devices with an emphasis on contrasting the objectives, methods, and documentation involved in formative versus summative usability testing. We will cite relevant FDA guidance, review some frequently used software tools, and show examples of typical deliverables. While our perspective is that of a consulting firm providing UX engineering services to medical device manufacturers, the material covered and guidance offered should be equally relevant to in-house UX teams. We will encourage the audience to raise corresponding points based on their own experience.

1. © 2019 Ipsos
Comparing formative and summative
approaches in the usability testing of
medical devices
—
Presented by : Zach Dirazonian
© 2019 Ipsos. All rights reserved. Contains Ipsos' Confidential and Proprietary information and may not be
disclosed or reproduced without the prior written consent of Ipsos.

2. © 2019 Ipsos© 2019 Ipsos
Purpose
To offer a high level overview of medical usability
testing and how it is practiced differently depending
on which phase of the product life cycle you are in
Audience Takeaways
An appreciation for how Human Factors Testing is
unique and a better idea of how to go about
conducting this research yourself
Purpose &
takeaways
—
2
© 2019 Ipsos

3. © 2019 Ipsos© 2019 Ipsos
04.
Background
08.
Submission
Package
14.
Setting up the
Study
19.
Running the
Study
29.
Reporting on
the Study
Table of contents
—
3

4. © 2019 Ipsos© 2019 Ipsos
Background

5. © 2019 Ipsos© 2019 Ipsos
Human Centered Design Lifecycle
Formative TestingStrategy & DesignDiscovery
Research
Iterative Design Validation
(Summative Testing)
Post Market
• Stakeholder
Interviews
• Discovery
Workshops
• User Analysis
• Prototyping
• Expert Review
• Heuristic Analysis
• Information
Architecture
• Visual Design
• Performance
Based
• In-Lab
• In-Field
• Remote
• Improvement
based on user
performance and
hard data
• Final validation the
entire system is
Safe & Effective
• User & Design
Specifications
• Instruction for Use
• Training Materials
• Interface
Implementation
Product Launch

6. © 2019 Ipsos© 2019 Ipsos
Food and Drug Administration
Certain types of
Medical Devices
02.
FDA requires
Usability Evaluation.
01.
6

7. © 2019 Ipsos© 2019 Ipsos
Guidance
documents
—
7
• International Electrotechnical Commission
• IEC 62366-1 -Application of usability engineering to medical devices
• IEC 62366-2
• HE75- Human factors engineering – Design of medical devices
• FDA Guidance-
• Applying Human Factors and Usability Engineering to Medical Devices
• International Organization for Standardization
• ISO 14971- Application of risk management to medical devices

8. © 2019 Ipsos© 2019 Ipsos
Submission Package

9. © 2019 Ipsos© 2019 Ipsos
Usability engineering
package
—
9
• Varies depending on submission approach
• Can include:
• Protocol
• Test Plans
• Risk Analysis
• HFE/UE findings report
• & More

10. © 2019 Ipsos© 2019 Ipsos
Formative test plans
• Could be submitted for review prior to validation
• Would increase likelihood of compliance
Summative protocols
• Scrutinized by both regulatory body (FDA, CE, etc.) AND
client
• Any deviations from the protocol must be documented
Test plans &
protocols
—
10

11. © 2019 Ipsos© 2019 Ipsos
Risk
analysis
—
11
• Essential to any submission
• Inform design at system engineering level
• Determines critical tasks – use scenarios ultimately tested
• Risk analysis:
• FMEA
• FTA

12. © 2019 Ipsos© 2019 Ipsos
Risk
analysis
—
12

13. © 2019 Ipsos© 2019 Ipsos
Risk control
measures
—
13

14. © 2019 Ipsos© 2019 Ipsos
Setting Up the Study

15. © 2019 Ipsos© 2019 Ipsos
User
groups
—
15
• “subset of intended USERS who are differentiated from other
intended USERS by factors that are likely to influence USABILITY,
such as age, culture, expertise or type of interaction with a
MEDICAL DEVICE” –IEC 62366
• Could be split by role, profession, age, dynamic etc.
• Most critical factor for determining split is tasks performed

16. © 2019 Ipsos© 2019 Ipsos
Numbers of participants
Formative
5-8
Summative
≥15 SummativeFormative
Sample sizes can be
relatively small and flexible
(n= 5-8)
Sample sizes are at least
15 participants per user
group per FDA guidance
16

17. © 2019 Ipsos© 2019 Ipsos
Recruitment
challenges
Niche populations
• Patients with specific conditions
• e.g. pediatric diabetics
• e.g. ER doctors that specialize in
thermodilution (monitoring
technique)
Expensive participants
• Doctors
• Nurses
• Techs
17

18. © 2019 Ipsos© 2019 Ipsos
Travel
• Traveling to multiple cities
• Making sure project budget
allows this
Compensation
• Adequately paying medical
professionals
• Can lead to very large
incentive payments
Potential
solutions
Networking
• Partnering with market
research firms
• Reaching out to clinics,
hospitals, support groups
• Using networking tools like
LinkedIn
18

19. © 2019 Ipsos© 2019 Ipsos
Running the Study

20. © 2019 Ipsos© 2019 Ipsos
Participant
training
—
20
• Representative of real-world training
• Requires a decay period – ensure participants are not “fresh”
• Pros:
• More accurate than nothing
• Cons:
• More expensive
• Possibility participant does not return
• Shorter than real world decay
Takeaway: Sometimes validation requires participant training
and decay periods, two things that are less commonly used in
formative studies

21. © 2019 Ipsos© 2019 Ipsos
Formative- “UX Research”
• Conversational
• Think Aloud and other methodologies
• Elicit thoughts and intentions
Summative- “Human Factors Validation”
• No small talk
• Strictly avoid biasing
• Structured; task based
Moderating
style
—
21
Takeaway: Moderating for formative is a lot more
flexible, Moderators for validation must follow the
guidance

22. © 2019 Ipsos© 2019 Ipsos
Working with
stimuli
—
22
• Stimuli can be complex and encumbering
• Balance: Go slow enough that you will not incorrectly present
stimuli, but quick enough that participant does not get bored or
uncomfortable
• Practice moderating with live stimuli even before live pilot
• Establish a stimuli reset protocol
• E.g. Is the client going to reset? is the moderator? Note-
taker? Do you know how? Contingencies?

23. © 2019 Ipsos© 2019 Ipsos
• Sometimes stimuli cannot be automatically
simulated
• “Man behind the curtain” may be required to
mimic what the device would do in real world
situations
Working with
Stimuli
—
23
Takeaway: Validation studies can be very large and
cumbersome but knowing stimuli well can help them run
smoothly

24. © 2019 Ipsos© 2019 Ipsos
Definition
This technique of delineating the origin of use errors is critical for
explaining any task performance failures that occur during summative
human factors validation testing and identifying those that warrant
some form of mitigation
Goal
Elicit participants thoughts and intentions without leading
or biasing in anyway
Root cause
analysis
—
24

25. © 2019 Ipsos© 2019 Ipsos
Bad probe example
Bad root cause probes
“Tell me about how you failed
to turn on the device.”
25

26. © 2019 Ipsos© 2019 Ipsos
Better probe example
Good root cause probes
“Let’s walk through that task
one more time.”
26

27. © 2019 Ipsos© 2019 Ipsos
Root cause
probes
Avoid
• Pointing out the user error
• Identifying what would have been
correct
• Using language not first brought up by
the participant
Do
• Start with a “Wide net”
• Gradually zone in on a root cause
• Identify a preliminary root cause with
your back room before moving on
27

28. © 2019 Ipsos© 2019 Ipsos
Reporting on the Study

29. © 2019 Ipsos© 2019 Ipsos
Formative
• Design Recommendations
Summative
• The approval of a product
Focus of
Report
—
29
Takeaway:
Formative- focus of report improving the product through design recommendations
Summative- focus of report is final evaluation of the product

30. © 2019 Ipsos© 2019 Ipsos
Questions?
30