Download as PDF, PPTX
Uploaded byDan Berlin
Usability Testing Medical Devices
This document discusses key considerations for usability testing of medical devices. It notes that while usability activities are not clinical trials, they will still require extensive documentation and safety protocols. Researchers must understand relevant regulations and work closely with clients, medical experts, and Institutional Review Boards. Additional precautions are needed when testing with vulnerable participants or children. Moderator guides and data collection methods must be rigorously defined to meet regulatory requirements.
More Related Content
Similar to Usability Testing Medical Devices
![[BROCHURE] Italy Tour Project | @SlideON](https://cdn.slidesharecdn.com/ss_thumbnails/brochure8-251215152319-2805af68-thumbnail.jpg?width=640&height=640&fit=bounds)
Usability Testing Medical Devices
- 1. UPA 2012 Usability TestingMedical Devices Las Vegas, NV Dan Berlin Chris Hass Experience Research Director SVP of Experience Design @banderlin @chrishass dberlin@madpow.net chass@madpow.net
- 2. Usability activities areNOT clinical Trials Generally the process will be less stringent: • More loosely defined, faster • Will likely involve a relatively small number of people per activity • Initial studies stress simulation, with no medicinal agents or clinical impact. • Dovetail with interaction design, visual design, and marketing activities Usability Tes,ng Medical Devices 2
- 3. It’s going tofeel like a clinical trial • Extremely detailed “Usability Specification” documents • “Usability Validation” (Risk assessment and amelioration) • “Usability engineering file” / “Design Efficiency File” • Know Federal and International regulations: • FDA Quality System Regulation • HE: 75 2009 • ISO/IEC 62366:2007 • Knowledge of (and inspiration for) emerging standards Usability Tes,ng Medical Devices 3
- 4. Medical Device Manufacturersmay be unfamiliar with your practices and lack of regulations Usability Tes,ng Medical Devices 4
- 5. Safeguarding participants maybe more challenging • Your safety protocols must be exacting, fully defined, piloted, and effective • Even simulated product use may put participants at risk • Target audiences may be physically or cognitively vulnerable • Activity environments may be unusual • Work with client and medical care specialists Usability Tes,ng Medical Devices 5
- 6. Reporting responsibilities, guidelines,formats, and requirements will be more complex • Reporting guidelines, formats, and requirements will likely be extensive and complex • Industry reporting standards (CIF) • Specialized reporting formats • Raw data may be reviewed again and again and again. . . Usability Tes,ng Medical Devices 6
- 7. Research locales maybe more challenging Device use environments may be: • Unfamiliar • Sterile • Stressful • Germ-laden • Emotionally distressing • Gross • Secure • Rarely observable • Complex • or otherwise unusual to non-medical personnel • (And fainting can be a factor!) Usability Tes,ng Medical Devices 7
- 8. Client Relationship Source: http://robot6.comicbookresources.com/2010/07/take-a-look-at-the-avengers-assemble-painting-by-alex-ross/ • A robust set of client team members is essential – as is knowing their roles: – Legal – Medical advisor – Safety advisor – Compliance – Adverse Event Specialist – Physician is helpful, but not enough – The usual suspects • Clients may ask for proof of insurance – But too much insurance could be a bad thing Usability Tes,ng Medical Devices 8
- 9. FDA Relationship Source: http://news.petpardons.com/fda-denies-rumors-over- possible-discovery-of-chicken-jerky-toxin/ • If the study is for FDA approval – learn about the client’s relationship with the FDA: – What are the relevant regulations? – Who is the client’s liaison to the FDA? – What is the client’s relationship with the FDA? – Has the client met with the FDA? – Have they received comments from the FDA? – Does the client understand usability testing? – Are we allowed to talk to the FDA? Usability Tes,ng Medical Devices 9
- 10. Working with anIRB • IRB = Institutional Review Board Source: http://www.bioedge.org/index.php/bioethics/bioethics_article/ who_is_watching_the_irb_watchdogs_asks_nature • For FDA approval, typically work with an independent IRB • Their job: to ensure that research conducted on human subjects is carried out correctly and humanely • Not as scary as you may think – We are NOT evaluating medical efficacy or safety – But we DO have to be conscientious • Have your ducks in a row and it will be a smooth process Usability Tes,ng Medical Devices 10
- 11. Working with anIRB • Adds about $2000-$5000 to the cost of the study Source: http://www.bioedge.org/index.php/bioethics/bioethics_article/ who_is_watching_the_irb_watchdogs_asks_nature • Adds a day or two to the project for dealing with the IRB • Be cognizant of any non-disclosure agreements • Prior to the study, signature pages should be signed-off, circulated, and returned in a timely manner • Everyone on the team will take this very seriously! Usability Tes,ng Medical Devices 11
- 12. IRB Documents -Forms Source: http://blog.pennysaverusa.com/tag/tax-tips/ • Indemnification Agreement: So the IRB doesn’t get sued • Financial Disclosure: The researchers don’t have conflicts of interest • Investigator Statement: The researcher code of conduct • Satellite Site Application: May be needed if the study involves multiple locations • Shipping & Invoice: Where documents are sent • Study Application: Detailed information about the study • Curriculum Vitae for all researchers • Other forms that your IRB requires Usability Tes,ng Medical Devices 12
- 13. IRB Documents –Study and Device Info • Recruitment Screener: Be very precise Source: http://healthcareit.etisbew.com/ • Recruitment Flyer: To hang in doctor’s office to aid with recruiting • Moderator’s Guide: A second guide may be needed if there are child participants • Consent and asset forms: For adults and children, respectively • Safety Concerns: Outline potential risks • Device Background: Detailed explanation of the device including indication and new features • Device Instructions for Use: If they are also included in the study Usability Tes,ng Medical Devices 13
- 14. IRB Documents –Validation Plan • A detailed explanation of how the study will validate that the device is usable – If you’re used to winging it, you can’t do that here – Lay out every question and how it will be interpreted – Lay out a rationale for every metric – Break down the pieces of the tasks and what will be observed – A detailed explanation of how the will be analyzed and presented • Quantitative measures (asked and observed) – Rating scales – Yes/No observations Source: hDp://www.vnet5.org/pub/approach/uva.html Usability Tes,ng Medical Devices 14
- 15. Chart Color QuantitativeMeasures % of chart that is red % of chart that is black • Rating Scales – Asked – Observed • Ease of use • Usability • Instructions for • Help needed Use clarity • Number of • Design errors • Ergonomics • Yes/No observations – Did the participant do this, that, AND the other thing? Ra,ng No Help Needed LiDle Difficulty Some Help Much Help Task Failure Explana,on Completed with no help 1-‐2 small errors 3-‐5 errors Moderator gets involved Failure Small errors: Doesn't press the green buDon Doesn't remove cap properly Etc. Usability Tes,ng Medical Devices 15
- 16. Recruiting • Give yourrecruiter a heads up LONG before the study starts – And start recruiting early • Make a flyer to be hung in doctor’s offices and distributed electronically which includes: – The purpose of the study – Why they would want to participate – Compensation Source: http://www.toptenz.net/top-10-wwii-propaganda-posters.php – Recruiter contact information • During the project, any screener changes need to be re-approved by the IRB • Many institutions can’t “promote” a study that is under their auspices Usability Tes,ng Medical Devices 16
- 17. Sample Flyer Usability Tes,ng Medical Devices 17
- 18. Recruiting • Don’t beshy – makes friends with local hospitals and clinics – Nurses and admins are the gateway – Have the sanitized IRB approval in-hand • Find groups that can be a gateway to your desired audience Source: http://www.toptenz.net/top-10-wwii-propaganda-posters.php – Healthcare institutions – Local and national support groups – Federal agencies – Community support groups – Craigslist & Social Media • Plan for surprises – like extensions to the timeline Usability Tes,ng Medical Devices 18
- 19. Working with Children • From a legal standpoint, a child is someone under 18 years old Source: http://evilstaring.com/2011/06/08/amazingly-evil-child-3/ • Children cannot consent to participate in the study, they can only assent – An assent form explains the details of the study, in the first person • You will treat a 17 year old different than an 8 year old – Be cognizant of the protocol to ensure you are consistent in your data capture • The parent conundrum: should they participate? – At what point does it become co-discovery and is that Source: http://cautionmagnet.blogspot.com/2011/05/confused-kids.html okay? Usability Tes,ng Medical Devices 19
- 20. Working with Children • Put the child at ease: – Explain the lab (to a point) – Ask if s/he would like a break Source: http://lovefunnypictures.com/funny-pic-bush_funny_face.php – Give reminders that s/he is doing a great job in the study – Structure the study so that it isn’t drudgery – Don’t make them remember rating scales – Not all children like to be spoken to like a child • Be extra vigilant for signs that the participant is telling you want you want to hear – As well as signs of fatigue and confusion • You don’t need to be their friend, but don’t scare them away Usability Tes,ng Medical Devices 20
- 21. Moderator’s Guide • Beprecise with a tightly written protocol Source: http://tips.tntdental.com/?tag=website-navigation • Must be well-defined in advance and piloted – No changes to the protocol without revisiting the IRB, the client, the FDA approval minutes, etc. • Clients may be used to clinical trials and view usability methods as “thin” – We’re not running a clinical trials, but that’s no excuse for not adequately preparing • Adhere to the protocol! – Any changes require IRB approval Usability Tes,ng Medical Devices 21
- 22. Moderator’s Guide • Themoderator’s guide should capture: Source: http://tips.tntdental.com/?tag=website-navigation – The tasks – Steps in the tasks – What should be observed – The questions to be asked of the participants – Observer rating scales • Basically, a shortened version of the Validation Plan • Be precise in the metrics and methods – They cannot vary – Data reporting requirements will be significant – Gaps in the reporting will be very visible Usability Tes,ng Medical Devices 22
- 23. Notes Grid • Weare using “micro-tasks” and want to capture precise information • The notes grid should facilitate the analysis Source: http://beta.armagetronad.net/design/tron.html yet be easy to use – Every step and question on its own row – Have clear study sections (introduction, etc.) – Grey out areas that can be ignored • Team note taking: moderator concentrates on certain items, note taker does the rest Usability Tes,ng Medical Devices 23
- 24. Notes Grid Usability Tes,ng Medical Devices 24
- 25. Lab Preparation • Capture the participants from two different angles – A close-up view of the device interaction (over-the-shoulder) – Straight-on to capture the entire field • Have all device supplies at the ready – Alcohol wipes, extra needles, batteries, Source: http://www.ebaumsworld.com/pictures/view/218095/ etc. – You’ll use more supplies than you expect • Get a first aid kit and nitrile gloves – Have a first aid plan and practice Usability Tes,ng Medical Devices 25
- 26. Participant Safety • Aswith any usability, participant comfort and safety is the top priority Source: http://en.wikipedia.org/wiki/File:Sign_first_aid.svg • Must be mindful of participants with health conditions – Will they need accessibility accommodations? – Will they have an aide/caretaker? – How will the condition affect their participation? • What you can do to ensure participant comfort and safety: – Have a robust first aid kit (and possibly medical personnel) – Provide water, juice, sugary, and non-sugary snacks – Ensure they know that they will be working with a placebo – Accessible food Usability Tes,ng Medical Devices 26
- 27. Participant Safety • Warn participants of safety hazards while using the device, if it’s part of the protocol Source: http://en.wikipedia.org/wiki/File:Sign_first_aid.svg • Have a plan of action if someone is injured • Establish a path of notification if a participant is injured – Reporting may have many phases: reporting it internally, then to the client, to the FDA, and the IRB • Educate yourself about the condition • Ensure the protocol does not exacerbate a disease or condition • Be vigilant about fatigue, confusion, disorientation, dehydration, & stress Usability Tes,ng Medical Devices 27
- 28. Adverse Effects • Anadverse effect (AE) event is when a participant tells you about a side effect or other experience while using a device or drug • This is critical information for the client and must be reported, typically immediately Source: http://www.cartoonstock.com/directory/d/Din.asp • Some clients may require you to go through AE training • Before the study begins, determine to whom AEs are reported When in doubt, report! • AEs can be complex to identify, but are VERY important Usability Tes,ng Medical Devices 28
- 29. Adverse Effects • Inthe course of a usability study for a medication website, a participant mentions that she stopped taking the drug. She explains that the blister packs and pills sometimes arrived crushed. • While testing an infusion pump, the participant mentions that the tube always falls out of his previously used pump, which is the client’s product. • During a study, a participant mentions that her arms started to itch when she stopped taking the client’s medication. • A male participant mentions that his wife got pregnant two weeks after he started taking the client’s medication, even though she used the birth control pill. Usability Tes,ng Medical Devices 29
- 30. Analysis • The setupfrom the validation plan, moderator’s guide, and notes grid should make analysis easy Source: http://www.webseoanalytics.com/blog/how-to-conduct-market- research-competitor-analysis/ • Dealing with gaps in the data can be tricky – Did the question get asked? – Was the behavior observed? – Do we have to go back to the recordings? • Make sure you HAVE the recordings! • Rely on quantitative measures – What percentage of participants passed the task? – What ease of use ratings did participants give? – How many participants committed the most common errors? Usability Tes,ng Medical Devices 30
- 31. Analysis • Remember to think on a micro-task scale – every little step of using the device should be observed – It’s difficult to watch and report on every step of a complex interaction and document it precisely, Source: http://www.webseoanalytics.com/blog/how-to-conduct-market- research-competitor-analysis/ especially live – Tallying needs to be specific, yet practical • Be mindful of capturing and organizing the important qualitative data – This can be especially tough when dealing with many participants – Summarize qualitative data directly after the session – don’t wait • Know what you’re reporting – there may be data to help marketing • Sometimes the goal of these studies is to identify marketing claims, which can raise concerns Usability Tes,ng Medical Devices 31
- 32. Thank You! • Conductinga usability study for a medical device is similar to a typical study, except with much stricter protocols Questions/Comments? • Your safety and the safety & comfort of the Dan Berlin participants are paramount dberlin@madpow.net @banderlin • Use quantitative measures and report percentages of participants that did or Chris Hass didn’t perform an action chass@madpow.net @chrishass • It’s all about proper planning! Usability Tes,ng Medical Devices 32