Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

UXPA2019 Comparing formative and summative approaches in the usability testing of medical devices

60 views

Published on

This presentation will offer practical guidance in the UX engineering of medical devices with an emphasis on contrasting the objectives, methods, and documentation involved in formative versus summative usability testing. We will cite relevant FDA guidance, review some frequently used software tools, and show examples of typical deliverables. While our perspective is that of a consulting firm providing UX engineering services to medical device manufacturers, the material covered and guidance offered should be equally relevant to in-house UX teams. We will encourage the audience to raise corresponding points based on their own experience.

  • Be the first to comment

  • Be the first to like this

UXPA2019 Comparing formative and summative approaches in the usability testing of medical devices

  1. 1. © 2019 Ipsos Comparing formative and summative approaches in the usability testing of medical devices — Presented by : Zach Dirazonian © 2019 Ipsos. All rights reserved. Contains Ipsos' Confidential and Proprietary information and may not be disclosed or reproduced without the prior written consent of Ipsos.
  2. 2. © 2019 Ipsos© 2019 Ipsos Purpose To offer a high level overview of medical usability testing and how it is practiced differently depending on which phase of the product life cycle you are in Audience Takeaways An appreciation for how Human Factors Testing is unique and a better idea of how to go about conducting this research yourself Purpose & takeaways — 2 © 2019 Ipsos
  3. 3. © 2019 Ipsos© 2019 Ipsos 04. Background 08. Submission Package 14. Setting up the Study 19. Running the Study 29. Reporting on the Study Table of contents — 3
  4. 4. © 2019 Ipsos© 2019 Ipsos Background
  5. 5. © 2019 Ipsos© 2019 Ipsos Human Centered Design Lifecycle Formative TestingStrategy & DesignDiscovery Research Iterative Design Validation (Summative Testing) Post Market • Stakeholder Interviews • Discovery Workshops • User Analysis • Prototyping • Expert Review • Heuristic Analysis • Information Architecture • Visual Design • Performance Based • In-Lab • In-Field • Remote • Improvement based on user performance and hard data • Final validation the entire system is Safe & Effective • User & Design Specifications • Instruction for Use • Training Materials • Interface Implementation Product Launch
  6. 6. © 2019 Ipsos© 2019 Ipsos Food and Drug Administration Certain types of Medical Devices 02. FDA requires Usability Evaluation. 01. 6
  7. 7. © 2019 Ipsos© 2019 Ipsos Guidance documents — 7 • International Electrotechnical Commission • IEC 62366-1 -Application of usability engineering to medical devices • IEC 62366-2 • HE75- Human factors engineering – Design of medical devices • FDA Guidance- • Applying Human Factors and Usability Engineering to Medical Devices • International Organization for Standardization • ISO 14971- Application of risk management to medical devices
  8. 8. © 2019 Ipsos© 2019 Ipsos Submission Package
  9. 9. © 2019 Ipsos© 2019 Ipsos Usability engineering package — 9 • Varies depending on submission approach • Can include: • Protocol • Test Plans • Risk Analysis • HFE/UE findings report • & More
  10. 10. © 2019 Ipsos© 2019 Ipsos Formative test plans • Could be submitted for review prior to validation • Would increase likelihood of compliance Summative protocols • Scrutinized by both regulatory body (FDA, CE, etc.) AND client • Any deviations from the protocol must be documented Test plans & protocols — 10
  11. 11. © 2019 Ipsos© 2019 Ipsos Risk analysis — 11 • Essential to any submission • Inform design at system engineering level • Determines critical tasks – use scenarios ultimately tested • Risk analysis: • FMEA • FTA
  12. 12. © 2019 Ipsos© 2019 Ipsos Risk analysis — 12
  13. 13. © 2019 Ipsos© 2019 Ipsos Risk control measures — 13
  14. 14. © 2019 Ipsos© 2019 Ipsos Setting Up the Study
  15. 15. © 2019 Ipsos© 2019 Ipsos User groups — 15 • “subset of intended USERS who are differentiated from other intended USERS by factors that are likely to influence USABILITY, such as age, culture, expertise or type of interaction with a MEDICAL DEVICE” –IEC 62366 • Could be split by role, profession, age, dynamic etc. • Most critical factor for determining split is tasks performed
  16. 16. © 2019 Ipsos© 2019 Ipsos Numbers of participants Formative 5-8 Summative ≥15 SummativeFormative Sample sizes can be relatively small and flexible (n= 5-8) Sample sizes are at least 15 participants per user group per FDA guidance 16
  17. 17. © 2019 Ipsos© 2019 Ipsos Recruitment challenges Niche populations • Patients with specific conditions • e.g. pediatric diabetics • e.g. ER doctors that specialize in thermodilution (monitoring technique) Expensive participants • Doctors • Nurses • Techs 17
  18. 18. © 2019 Ipsos© 2019 Ipsos Travel • Traveling to multiple cities • Making sure project budget allows this Compensation • Adequately paying medical professionals • Can lead to very large incentive payments Potential solutions Networking • Partnering with market research firms • Reaching out to clinics, hospitals, support groups • Using networking tools like LinkedIn 18
  19. 19. © 2019 Ipsos© 2019 Ipsos Running the Study
  20. 20. © 2019 Ipsos© 2019 Ipsos Participant training — 20 • Representative of real-world training • Requires a decay period – ensure participants are not “fresh” • Pros: • More accurate than nothing • Cons: • More expensive • Possibility participant does not return • Shorter than real world decay Takeaway: Sometimes validation requires participant training and decay periods, two things that are less commonly used in formative studies
  21. 21. © 2019 Ipsos© 2019 Ipsos Formative- “UX Research” • Conversational • Think Aloud and other methodologies • Elicit thoughts and intentions Summative- “Human Factors Validation” • No small talk • Strictly avoid biasing • Structured; task based Moderating style — 21 Takeaway: Moderating for formative is a lot more flexible, Moderators for validation must follow the guidance
  22. 22. © 2019 Ipsos© 2019 Ipsos Working with stimuli — 22 • Stimuli can be complex and encumbering • Balance: Go slow enough that you will not incorrectly present stimuli, but quick enough that participant does not get bored or uncomfortable • Practice moderating with live stimuli even before live pilot • Establish a stimuli reset protocol • E.g. Is the client going to reset? is the moderator? Note- taker? Do you know how? Contingencies?
  23. 23. © 2019 Ipsos© 2019 Ipsos • Sometimes stimuli cannot be automatically simulated • “Man behind the curtain” may be required to mimic what the device would do in real world situations Working with Stimuli — 23 Takeaway: Validation studies can be very large and cumbersome but knowing stimuli well can help them run smoothly
  24. 24. © 2019 Ipsos© 2019 Ipsos Definition This technique of delineating the origin of use errors is critical for explaining any task performance failures that occur during summative human factors validation testing and identifying those that warrant some form of mitigation Goal Elicit participants thoughts and intentions without leading or biasing in anyway Root cause analysis — 24
  25. 25. © 2019 Ipsos© 2019 Ipsos Bad probe example Bad root cause probes “Tell me about how you failed to turn on the device.” 25
  26. 26. © 2019 Ipsos© 2019 Ipsos Better probe example Good root cause probes “Let’s walk through that task one more time.” 26
  27. 27. © 2019 Ipsos© 2019 Ipsos Root cause probes Avoid • Pointing out the user error • Identifying what would have been correct • Using language not first brought up by the participant Do • Start with a “Wide net” • Gradually zone in on a root cause • Identify a preliminary root cause with your back room before moving on 27
  28. 28. © 2019 Ipsos© 2019 Ipsos Reporting on the Study
  29. 29. © 2019 Ipsos© 2019 Ipsos Formative • Design Recommendations Summative • The approval of a product Focus of Report — 29 Takeaway: Formative- focus of report improving the product through design recommendations Summative- focus of report is final evaluation of the product
  30. 30. © 2019 Ipsos© 2019 Ipsos Questions? 30

×