mHealth Israel_Human Factors Engineering BOOTCAMP_Tressa J. Daniels_AAMI

Human Factors
Engineering Bootcamp
April 13, 2020
Tressa J. Daniels, M.S.
Global Director
Human Factors Engineering
T. Daniels
Do Not Copy or Distribute

Disclaimer
The opinions expressed in this presentation and on the following slides are solely
those of the presenter and do not represent those of Teleflex Inc. or AAMI.
Presentations are intended for educational purposes only and do not replace
independent professional judgement. Teleflex Inc. and AAMI do not endorse or
approve, and assumes no responsibility for the content, accuracy or completeness
of information presented.

Agenda
• Introduction (5 minutes)
• Human Factors Engineering in the Product Development Process (15 minutes)
• Regulatory Standards and Expectations regarding Human Factors (20 minutes)
• Human Factors Validation Testing (20 minutes)
• What leads to failure (5 minutes)
• What to do if failure occurs (5 minutes)
• Submitting a Human Factors Report to a Regulatory Body (5 minutes)
• Human Factors Best Practices (5 minutes)
• Concluding Remarks (5 minutes)

Definition
“The scientific discipline concerning understanding of interactions among humans and other
elements of a system, and applying theory, principles, data, and other methods to design of user
interfaces in order to optimize human well-being and overall system performance.”*
*As defined by the Human Factors and Ergonomics Society
T. Daniels

Human
Factors
Engineers
Research Research user and product problems
Study Study workflows
Observe Observe environments of use
Define Define product and user needs
Run
Moderate usability testing to verify
solutions meet user expectations
Ensure
Ensure safety and effectiveness are
designed into the product
T. Daniels

Industries that apply
Human Factors
T. Daniels

Human
Factors
Engineering
Unique to
Medical
Devices
Globally regulated industry
Balance between “Safe” &
“Usable” Systems
Risk assessment oriented
T. Daniels

The Primary
Objective
Identification and control of user
interface design problems or flaws that
could lead to…
• Harm to user or patients
• Compromised medical care
T. Daniels

Human Factors in Product Development Process:
Activities & Deliverables
T. Daniels

Human Factors
Skillsets &
Activities
• Formative Research - Identifying user’s needs and
wants
• Ethnographic research
• Usability Inspection & Heuristic Evaluations
• Formative Usability Testing
• Tasks Analysis
• Interaction Design
• Workflows, wire frames
• Online, mobile, embedded graphic design
• Industrial Design
• Product Color, shape and form
• Physical interface design
• Ergonomic design
• Human Factors Validation - Testing and
validating safety and efficiency
• HF/UE Documentation
T. Daniels

Process
T. Daniels

In the beginning: HF Focus is on Research
CONTEXTUAL
INQUIRY
BEHAVIORAL
OBSERVATION
JOB
SHADOWING
SITE VISITS
T. Daniels

In the middle: HF Focus is on Design
Feedback
Prototypes
Low fidelity: Paper
High fidelity: Clickable screens or foam
core mockups
Formative
Evaluations
T. Daniels

Feedback gathered
on all product UI
• IFU
• Packaging
• Labeling
• Hardware
• Software
T. Daniels

Design changes implemented based on
feedback
T. Daniels

At the end: HF Validates Production Units
Sample Size: 15 users per user
group
Final production equivalent
product
All aspects of the UI
Focus on critical tasks
Relies on uFMEA or Hazards Analysis
If design changes are needed
after validation – those
changes must be validated
T. Daniels

Documentation
created during
the PDP
• HF/UE Project Plan
• Exploratory Research Plan
• Exploratory Research Report
• Use Specification
• Task Analysis
• uFMEA
• User Interface Specification
• Formative Usability Test Plan (per test)
• Formative Usability Test Report (per test)
• Human Factors Validation Test Plan
• Human Factors Validation Test Report
• HF/UE Project Report
• HF/UE File
T. Daniels

Regulatory Standards
Related to Human Factors
T. Daniels

Medical
products are
regulated by
these
standards &
guidances
IEC-62366
ANSI/AAMI HE75
FDA Human Factors in MD Design
PMA Submissions
510K Submissions
T. Daniels

US and OUS The IEC report provides
guidance on how to implement
a usability engineering process.
It contains requirements for
safety and human factors.
By following the IEC, AAMI and
FDA guidelines the likelihood
of a successful PMA and/or
510k submission increases
greatly.
Code of Federal
Regulations, Part 820 Title
21.
T. Daniels

AAMI: TIR 59:
Integrating Human
Factors
into Design Controls

Regulatory
Expectations
Use Specification
Identify Intended
Users
Develop User
Profiles
Define User
Groups
Understand Known
Use Errors
Utilize Risk
Management Tools
Plan for Formatives
Execute Design
Validation
T. Daniels

Initiate Use
Specification
• Use specification is refined over time starting with
Intended use.
• Level of detail is increased through user research
• A Use specification might contain the following:
• Identify user groups which are going to be
approached for interviews
• Identify use environment which is going to be
inspected
• Identify medical indications which are needed to be
further explored
• These elements aid in identifying the known and
foreseeable hazards and hazardous situations related to
the user interface.
• Understanding these elements is necessary to develop an
adequate usability evaluation plan
T. Daniels

Methodologies
for
development of
Use Spec
• Contextual Inquiry and Observation
• Researcher observes users while performing User Tasks in intended use
environment and discusses the details with the user
• Interviews and Surveys
• Location agnostic discussion with users as to knowledge, perception and
opinions as they relate to the medical device and its use. This can be
stand-alone or supplement the above-mentioned observation
• Expert reviews
• An expert might cite strengths and weaknesses of a device or heuristic
analysis might be performed where a number of experts independently
identify and prioritize improvements
• Advisory Panel reviews
• 6-12 members who collaborate with the design team on design options.
Such an advisory panel might be leveraged continuously throughout the
design process
• Competitive/Predicate Device Usability Testing
• Performing base-line usability testing to identify strengths and weaknesses
of existing products
T. Daniels

Identify
Intended
Users
• Example Primary Users:
• Patients
• Physicians
• Nurses
• Technicians
• Therapists
• Pharmacists
• Emergency Response
Personnel
• Example Non-Primary
Users:
• Assemblers
• Installers
• Trainers
• Transporters
• Engineers
• Repair Personnel
• Recyclers
• Sterile Processors
• Admin Personnel
T. Daniels

Develop User
Profiles
• Common characteristics of Users
• Examples include:
• Occupation
• Demographics
• Knowledge and Skills (education, experience
level, language, literacy)
• Limitations (vision, hearing, cognitive,
impairments)
• Performance Shaping Factors (learning style,
preferences, tendencies)
• Work responsibilities (Tasks pertinent to medical
device)
• It may be helpful to develop a “Persona” or a
fictitious user who represents a user profile
T. Daniels

Define User
Groups
• Where users share characteristics, users can be
grouped to inform usability
• Examples include:
• Age: Child (>2 yrs to 12 yrs), adolescent (>12
yrs to 21 yrs), Adult (>21 yrs)
• Occupation: Physician, Nurse, Therapist,
Technician, Patient, Installer
• Prior experience using similar medical
devices: New user (none), inexperienced (<6
mo), experienced user (>6 mo)
• Level of training
• Education
T. Daniels

Anticipate
Environment
of Use
• Characteristics of the anticipated use
environment should be defined to inform
usability. Examples include:
• Physical environment (gloves, eye protection,
heavy clothing)
• Lighting
• Noise level
• Professional interactions and responsibilities
• Additional equipment present
• Furnishings
• Climate
• Distractions
T. Daniels

Understand
Known Use
Errors
• Complaint Data
• MAUDE
• EUDAMED
• Literature Review
• Social Media
T. Daniels

Finalize Use
Specification
• IEC 62366-1 requires the following:
• Intended Medical Indication
• Intended Patient Population
• Intended Part of the Body or Type of Tissue
Interacted With
• Intended User Profiles
• Intended Use Environment
• Operating Principle
• The following are not required but are
recommended:
• User Tasks (Use Cases, User Stories)
• User needs derived from User Tasks
T. Daniels

User
Interface
Specification
• The User Interface Specification consists of:
• The Use Specification
• The known or foreseeable Use Errors
• The selected Hazard-Related Use Scenarios
• The User Interface Specification should be developed
ahead of the User Interface, but then evolve with the
ensuing design
• User interface requirements should be developed to
encompass Instructions for Use and other
Accompanying Documentation
• IEC 62366-1 requires that the manufacturer
determine whether Accompanying Documentation or
Training are required for safe use of the medical
device
T. Daniels

Iterative
Formative
Usability
Testing
• IEC Recommends 2 to 3 Formative evaluations to be conducted at a
minimum. They further note that multiple small-scale evaluations can
be more productive than fewer larger scale evaluations.
• Formative Evaluations can vary in formality and methodology but
should begin to resemble the planned Validation as it approaches.
• Both Formative and Summative Evaluations typically involve one or
more Usability Tests.
• IEC 62366-1 requires that the Usability Test Protocol include the
following:
• Participants in the Usability Test to be representative of each
intended User Group
• Test environment and other use conditions, to be representative
of the Intended Use Environment
• The Accompanying Documentation to be provided during the
Usability Test, if any
• The Training to be provided during the Usability Test, if any, and
the minimum elapsed time between training and the beginning of
the Usability Test
T. Daniels

Human Factors Validation Test
T. Daniels

Design
Validation
Defined
Validation is the process of making sure that you
have objective evidence that user needs and
intended uses are met. It is usually done by tests,
inspections, and in some cases analysis. However,
the target of the validation is to make sure
the user needs are met in a medical device that
consistently provides the intended medical benefit
in actual-use conditions.
https://www.asme.org/topics-resources/content/validation-verification-for-medical-devices
T. Daniels

Design
Verification
Defined
• According to the FDA, design verification is “confirmation
by examination and provision of objective evidence that
specified requirements have been fulfilled.”
• Keep in mind that while it will involve testing, there are
other acceptable verification activities.
• They can include tests, inspections, and analyses (for
more on this, check out FDA Design Control Guidance
• While Validation looks at User Needs, Verification looks
at Product Requirements & Design Specs.
T. Daniels

Design
Validation
Guidance
• CFR 820.30
• Design validation. Each manufacturer shall establish and maintain
procedures for validating the device design. Design validation shall be
performed under defined operating conditions on initial production
units, lots, or batches, or their equivalents. Design validation shall
ensure that devices conform to defined user needs and intended uses
and shall include testing of production units under actual or simulated
use conditions. Design validation shall include software validation and
risk analysis, where appropriate. The results of the design validation,
including identification of the design, method(s), the date, and the
individual(s) performing the validation, shall be documented in the DHF.
• The FDA Design Controls Guidance
• VALIDATION METHODS. Many medical devices do not require clinical trials.
• However, all devices require clinical evaluation and should be tested in
the actual or simulated use environment as a part of validation. This
testing should involve devices which are manufactured using the same
methods and procedures expected to be used for ongoing production.
While testing is always a part of validation, additional validation
methods are often used in conjunction with testing, including analysis
and inspection methods, compilation of relevant scientific literature,
provision of historical evidence that similar designs and/or materials are
clinically safe, and full clinical investigations or clinical trials.
T. Daniels

Human
factors
Validation
Defined
• Human Factors Validation is a type of Design
Validation
• Demonstrates that devices can be used by the
intended users, under expected use conditions,
without serious use errors or problems.
• Test participants represent the intended (actual)
users of the device.
• All critical tasks are performed during the test.
• The device user interface represents the final
design (production equivalent).
• The test conditions are sufficiently realistic to
represent actual conditions of use.
T. Daniels

IEC Guidance Clause 5.9
• The MANUFACTURER shall perform a SUMMATIVE EVALUATION of each
HAZARD-RELATED USE SCENARIO on the final or production equivalent
USER INTERFACE.
• The data from the SUMMATIVE EVALUATION shall be analyzed to
identify the potential consequences of all USE ERRORS that occurred. If
the consequences can be linked to a HAZARDOUS SITUATION, the root
cause of each USE ERROR shall be determined.
T. Daniels

FDA Guidance section 3.7
• Testing conducted at the end of the device development process to
assess user interactions with a device user interface to identify use
errors that would or could result in serious harm to the patient or user.
• Human factors validation testing is also used to assess the effectiveness
of risk management measures.
• Human factors validation testing represents one portion of design
validation.
T. Daniels

Simulated HFVT
Simulated-use should be
sufficiently realistic so that
the results of the testing are
generalizable to actual use.
It is not acceptable to use of
the “think aloud” technique.
Mirrors participant’s real-life
interaction with labeling.
Testing carried out with final
finished combination
product. (can contain
placebo)
Simulation methods can vary
(e.g., manikin, injection
pads, placebo, etc.)
T. Daniels

Test Participants
Represent the population of
intended users.
Minimum number of
participants is 15 for each
distinct user population; (e.g.
pediatric, adult, healthcare
providers and lay users)
For FDA
Reside in the United States
Exceptions to this are considered on a
case-by-case basis.
Does not include your
employees.
T. Daniels

Task & Use
Scenarios
• Tasks that logically occur in sequence >
Use Scenarios
• Organize to represent a natural workflow.
• Prior to testing, define user performance
that represents success for each task.
Not just a percentage of completion!
T. Daniels

Select
Hazard-
Related Use
Scenarios
• Choose tasks based on severity
• Critical tasks that have a low frequency of occurrence require careful
consideration and should be included in the testing.
• the manufacturer must determine which Hazard-Related Use
Scenarios shall be evaluated
• The purpose of this determination is to ensure that the Validation
includes all Use Scenarios needed to demonstrate Safety related to
the User Interface of the Medical Device
• IEC 62366-1 provides the following three options:
• Include all Hazard-Related Use Scenarios
• Include a subset of Hazard-Related Use Scenarios based on
Severity of Harm
• Include a subset of Hazard-Related Use Scenarios based on
Severity of Harm and circumstances specific to device and
Manufacturer
• HF Validation must also show that Users are able to accomplish the
intended purpose of the Medical Device as described in the Use
Specification
T. Daniels

Instructions
for Use
• The labeling must represent the final designs.
• Validation test can indirectly assess the IFU,
but only in the context of use.
• Stating that you mitigated the risks by
“modifying the IFU” is not acceptable, unless
you provide additional test data
demonstrating that the modified elements
were effective in reducing the risks to
acceptable levels
T. Daniels

IFU
Knowledge
Tasks • Assess comprehension
• Read-then-do
• Read-then- paraphrase
T. Daniels

Participant
Training
• Should approximate the training that actual
users would receive.
• Minimum 1-hour training decay must follow
training
• Longer time may be appropriate if potential
source of use-related risk.
• Stating that you mitigated the risks by
“providing additional training” is not acceptable
unless you provide additional data that
demonstrates that it would be effective in
reducing the risks to acceptable levels.
T. Daniels

Objective
Data
Collection
• Observational data - observations of
participants’ performance of all the critical
use
• Knowledge task data - comprehension of
interface components
• DFU, quick start guide, labeling on the device
itself, and training.
T. Daniels

Subjective
Data
Collection
• Post Session Interview data from Participants
• Their feedback considering the overall
device focused on each critical task or use
scenario.
• Their identification of any use difficulties,
confusions or errors that were
experienced during the test.
• Their assessment of root cause for any
observed or participant reported use
difficulties, confusions or errors
• Do not use Likert scale ratings
T. Daniels

Actual Use
Testing • Definition: Use of final finished product in a
real (not simulated) environment of use.
• It is rare that actual-use testing is determined
to be necessary to ensure safe use of the
proposed device.
T. Daniels

Analyzing the Data
T. Daniels

Why we do Human Factors Validation Tests
T. Daniels

REGULATION
• FDA HF Device Guidance:
• “CDRH recommends that manufacturers
consider human factors testing for medical
devices as a part of a robust design control
subsystem.
• CDRH believes that for those devices where an
analysis of risk indicates that users performing
tasks incorrectly or failing to perform tasks could
result in serious harm, manufacturers should
submit human factors data in premarket
submissions (i.e., PMA, 510(k)).”
T. Daniels

We should
Responsible for
safe and effective
products
Controls risk
Reduces support
calls
Reduces
complaints
Reduces recalls
Competitive
advantage
Brand loyalty
T. Daniels

What is the difference between a Usability
Test & a Human Factors Validation Test?
T. Daniels

Formative
Methods
• Analytical
► Task Analysis
► Heuristic Evaluations
► Expert Reviews
• Empirical
► Interviews & Focus Groups
► Contextual Inquiry
► Formative Studies (Cognitive
Walkthroughs, & Usability Testing)
T. Daniels

Goals of Formative Evaluations
COLLECT
FEEDBACK
UNDERSTAND
PAIN POINTS
UPDATE DESIGN ITERATIVE
TEST/EVALUATION
T. Daniels

Formative vs.
HF Validation
• Formative – expect to iterate on the design to “form” it
• Formative studies can be relatively short and conducted
with a small number of representative users.
• Conducted early in the design cycle, as few as 8-10
participants can reveal 90 percent of existing design
flaws in the UI that can be modified to eliminate errors
and use problems.
• As the UI design matures, formative studies may be
employed as trial run tests before conducting the final
validation test, which saves significant time and cost by
preventing the need to repeat the validation study due
to residual problems in the interface or the study
methodology itself.
T. Daniels

Formative vs.
HF Validation
(cont.)
• Validation – prove safe and effective use
• is not meant to be an exploratory effort seeking inputs on
design features but should serve as a ‘final’ demonstration of
use safety for the device. Therefore, participants are not
interrupted with questions or corrected in their performance.
• Participants engage in use scenarios chosen to represent
sequences of typical interaction with the device. These
scenarios should cover all tasks that have been categorized as
containing high risk in the use risk analysis.
• Training and device familiarization should be provided and
represent realistic conditions. This could be done by conducting
a training session prior to testing with an appropriate
intervening interval to represent potential memory and learning
decay.
• Device instructions for use should be available to the
participant, but with no requirement to read the instructions
prior to performing the use scenarios, unless the participant
chooses to do so on their own.
T. Daniels

Formative vs.
HF Validation
(cont.)
• Validation – prove safe and effective use
• User performance on each scenario task (or step) should be
observed and categorized with respect to failure, success, and
success with difficulties such as hesitation, self correcting of
actions, and potential confusion. These instances, along with
failures, should be noted for further post-test analysis.
• Should include a post-test dialogue with the study team in order
to determine the root cause of any failures and difficulties
including both specific questions about unexpected or incorrect
actions, or general questions about task difficulty during the
test.
• Results of the validation test should support an overall
conclusion regarding use safety for the device. This conclusion
should not be based on meeting pre-defined quantitative goals
(i.e. 95 per cent success rate across tasks and participants), but
on whether there is a remaining pattern of use-related
problems that are directly attributable to the UI or
accompanying documentation (labelling and instructions, etc.).
T. Daniels

Order of Events
T. Daniels

Where is
HFVT within
the Process
• End
• Production Equivalent System
► Training (IFU, ILT, Computer Based)
► Product
► Labels/Labeling
► Hardware
► Software
T. Daniels

What Leads to Failure
T. Daniels

Use Error
• A situation where device use
was different than intended,
but not due to malfunction
of the device
• Frequently, use errors are
due to poorly designed
devices
• Well-designed medical
devices eliminate or
minimize the possibility of a
user using a device
incorrectly
Easy to use correctly = fewer
medical use errors = safer
device
https://www.fda.gov/medical-devices/human-factors-and-
medical-devices/postmarket-information-device-surveillance-and-
reporting-processes
T. Daniels

Common
Mistakes
• Testing an incomplete sample size (fewer than
15 users per user group)
• Lack of evidence or due diligence of developing
an appropriate HF protocol
• Not relying on your use related risk analysis to
drive test scenarios
• Lack of consideration of KUE’s for a predicate
product
• Inability to prove an effective mitigation
• Preparing to make it a formative test if
validation fails
T. Daniels

Reasons for validation
Failure
• Failure to conduct formative evaluations
• Validating untested final product and
study design
• Failure to focus on strong mitigations
• Red flags identified before study
initiation
• Timeline priority affecting decisions
T. Daniels

Responding to
Validation Failure
• Pause the study
• Redesign your protocol
• Repeat validation
• Convert validation to formative
• Identify opportunities for improvement
• Initiate redesign
• Assess study results
• Determine the validation failures
• Initiate redesign
• Focused validation
T. Daniels

Design Changes
T. Daniels

Making
Design
Changes
after HFVT
• Considerations for Design Changes after HF
Validation
• ► Clinical study reveals design flaws
• ► Post-market surveillance
• ► Respond to post-market use-related
safety reports, complaints or problems
• ► Meet needs of an expanded indication
or user population
T. Daniels

Making
Design
Changes
after HFVT
• Conduct updated use-related risk analysis
• ► Does the design change alter the user
interface in any way (e.g., audible, tactile,
color recognition, user instructions, etc.)?
• ► Does the design change alter an existing
critical task or add a new critical task?
• ► Does the design change alter the
expected users or their knowledge base?
T. Daniels

Submitting an HFVT Report
T. Daniels

Submitting
an HFVT
Report
Submissions to FDA pre-market approval
process generally require validation test
results
Typical Manufacturer Deficiencies
-Not using the preliminary use risk
analysis to justify testing protocols
-Reporting just the success rates
without making the overall use-safety
case
-Reporting preference results
-Failure to include the right user
groups
-Attributing test result failures to
human error without further
justification
-Failure to test the IFU when indication
it is a risk control
T. Daniels

Concluding Remarks
T. Daniels

Best Practices
Early research:
•Ethnography
•Customer confirmation
•Build user needs based on data
Iterative formative testing
during design and
development – with all
intended user groups
Ensure a minimum of 15
users per distinct user
group
Your employees should
not serve as test
participants
Be sure to test mitigations
that map to IFU
comprehension
Clearly list critical tasks
and the mitigation you
are testing – linked to a
user need
Clearly define
performance
success/failure
(acceptance criteria)
Clearly describe how you
will identify root cause(s)
of use errors, difficulties,
and close calls
If design changes are
made to any part of a
user facing portion of the
product you must do a
focused validation test
T. Daniels

Thank You!
Tressa J. Daniels
tressadaniels@gmail.com
https://www.linkedin.com/in/tressadaniels/
T. Daniels

mHealth Israel_Human Factors Engineering BOOTCAMP_Tressa J. Daniels_AAMI

Recommended

Recommended

More Related Content

Similar to mHealth Israel_Human Factors Engineering BOOTCAMP_Tressa J. Daniels_AAMI

Similar to mHealth Israel_Human Factors Engineering BOOTCAMP_Tressa J. Daniels_AAMI (20)

More from Levi Shapiro

More from Levi Shapiro (20)

Recently uploaded

Recently uploaded (20)

mHealth Israel_Human Factors Engineering BOOTCAMP_Tressa J. Daniels_AAMI