• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Usability Testing Medical Devices

Usability Testing Medical Devices






Total Views
Views on SlideShare
Embed Views



10 Embeds 86

http://user-experience-specialist.blogspot.de 61
http://user-experience-specialist.blogspot.com 11
http://user-experience-specialist.blogspot.nl 3
http://flavors.me 2
http://user-experience-specialist.blogspot.ro 2
http://user-experience-specialist.blogspot.co.il 2
https://twitter.com 2
http://user-experience-specialist.blogspot.kr 1
http://user-experience-specialist.blogspot.cz 1
http://es.flavors.me 1


Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    Usability Testing Medical Devices Usability Testing Medical Devices Presentation Transcript

    • UPA 2012Usability Testing Medical Devices Las Vegas, NV Dan Berlin Chris Hass Experience Research Director SVP of Experience Design @banderlin @chrishass dberlin@madpow.net chass@madpow.net
    • Usability activities are NOT clinical Trials Generally the process will be less stringent: •  More loosely defined, faster •  Will likely involve a relatively small number of people per activity •  Initial studies stress simulation, with no medicinal agents or clinical impact. •  Dovetail with interaction design, visual design, and marketing activities Usability  Tes,ng  Medical  Devices   2  
    • It’s going to feel like a clinical trial •  Extremely detailed “Usability Specification” documents •  “Usability Validation” (Risk assessment and amelioration) •  “Usability engineering file” / “Design Efficiency File” •  Know Federal and International regulations: •  FDA Quality System Regulation •  HE: 75 2009 •  ISO/IEC 62366:2007 •  Knowledge of (and inspiration for) emerging standards Usability  Tes,ng  Medical  Devices   3  
    • Medical Device Manufacturers may be unfamiliar with your practices and lack of regulations Usability  Tes,ng  Medical  Devices   4  
    • Safeguarding participants may be more challenging •  Your safety protocols must be exacting, fully defined, piloted, and effective •  Even simulated product use may put participants at risk •  Target audiences may be physically or cognitively vulnerable •  Activity environments may be unusual •  Work with client and medical care specialists Usability  Tes,ng  Medical  Devices   5  
    • Reporting responsibilities, guidelines, formats, and requirements will be more complex •  Reporting guidelines, formats, and requirements will likely be extensive and complex •  Industry reporting standards (CIF) •  Specialized reporting formats •  Raw data may be reviewed again and again and again. . . Usability  Tes,ng  Medical  Devices   6  
    • Research locales may be more challenging Device use environments may be: •  Unfamiliar •  Sterile •  Stressful •  Germ-laden •  Emotionally distressing •  Gross •  Secure •  Rarely observable •  Complex •  or otherwise unusual to non-medical personnel •  (And fainting can be a factor!) Usability  Tes,ng  Medical  Devices   7  
    • Client Relationship Source: http://robot6.comicbookresources.com/2010/07/take-a-look-at-the-avengers-assemble-painting-by-alex-ross/•  A robust set of client team members is essential – as is knowing their roles: –  Legal –  Medical advisor –  Safety advisor –  Compliance –  Adverse Event Specialist –  Physician is helpful, but not enough –  The usual suspects•  Clients may ask for proof of insurance –  But too much insurance could be a bad thing Usability  Tes,ng  Medical  Devices   8  
    • FDA Relationship Source: http://news.petpardons.com/fda-denies-rumors-over- possible-discovery-of-chicken-jerky-toxin/•  If the study is for FDA approval – learn about the client’s relationship with the FDA: –  What are the relevant regulations? –  Who is the client’s liaison to the FDA? –  What is the client’s relationship with the FDA? –  Has the client met with the FDA? –  Have they received comments from the FDA? –  Does the client understand usability testing? –  Are we allowed to talk to the FDA? Usability  Tes,ng  Medical  Devices   9  
    • Working with an IRB•  IRB = Institutional Review Board Source: http://www.bioedge.org/index.php/bioethics/bioethics_article/ who_is_watching_the_irb_watchdogs_asks_nature•  For FDA approval, typically work with an independent IRB•  Their job: to ensure that research conducted on human subjects is carried out correctly and humanely•  Not as scary as you may think –  We are NOT evaluating medical efficacy or safety –  But we DO have to be conscientious•  Have your ducks in a row and it will be a smooth process Usability  Tes,ng  Medical  Devices   10  
    • Working with an IRB•  Adds about $2000-$5000 to the cost of the study Source: http://www.bioedge.org/index.php/bioethics/bioethics_article/ who_is_watching_the_irb_watchdogs_asks_nature•  Adds a day or two to the project for dealing with the IRB•  Be cognizant of any non-disclosure agreements•  Prior to the study, signature pages should be signed-off, circulated, and returned in a timely manner•  Everyone on the team will take this very seriously! Usability  Tes,ng  Medical  Devices   11  
    • IRB Documents - Forms Source: http://blog.pennysaverusa.com/tag/tax-tips/•  Indemnification Agreement: So the IRB doesn’t get sued•  Financial Disclosure: The researchers don’t have conflicts of interest•  Investigator Statement: The researcher code of conduct•  Satellite Site Application: May be needed if the study involves multiple locations•  Shipping & Invoice: Where documents are sent•  Study Application: Detailed information about the study•  Curriculum Vitae for all researchers•  Other forms that your IRB requires Usability  Tes,ng  Medical  Devices   12  
    • IRB Documents – Study and Device Info•  Recruitment Screener: Be very precise Source: http://healthcareit.etisbew.com/•  Recruitment Flyer: To hang in doctor’s office to aid with recruiting•  Moderator’s Guide: A second guide may be needed if there are child participants•  Consent and asset forms: For adults and children, respectively•  Safety Concerns: Outline potential risks•  Device Background: Detailed explanation of the device including indication and new features•  Device Instructions for Use: If they are also included in the study Usability  Tes,ng  Medical  Devices   13  
    • IRB Documents – Validation Plan•  A detailed explanation of how the study will validate that the device is usable –  If you’re used to winging it, you can’t do that here –  Lay out every question and how it will be interpreted –  Lay out a rationale for every metric –  Break down the pieces of the tasks and what will be observed –  A detailed explanation of how the will be analyzed and presented•  Quantitative measures (asked and observed) –  Rating scales –  Yes/No observations Source:  hDp://www.vnet5.org/pub/approach/uva.html   Usability  Tes,ng  Medical  Devices   14  
    • Chart  Color  Quantitative Measures %  of  chart  that  is  red   %  of  chart  that  is  black  •  Rating Scales –  Asked –  Observed •  Ease of use •  Usability •  Instructions for •  Help needed Use clarity •  Number of •  Design errors •  Ergonomics•  Yes/No observations –  Did the participant do this, that, AND the other thing? Ra,ng   No  Help  Needed   LiDle  Difficulty   Some  Help     Much  Help   Task  Failure   Explana,on   Completed  with  no  help   1-­‐2  small  errors   3-­‐5  errors   Moderator  gets  involved   Failure   Small  errors:   Doesnt  press  the  green  buDon   Doesnt  remove  cap  properly   Etc.   Usability  Tes,ng  Medical  Devices   15  
    • Recruiting•  Give your recruiter a heads up LONG before the study starts –  And start recruiting early•  Make a flyer to be hung in doctor’s offices and distributed electronically which includes: –  The purpose of the study –  Why they would want to participate –  Compensation Source: http://www.toptenz.net/top-10-wwii-propaganda-posters.php –  Recruiter contact information•  During the project, any screener changes need to be re-approved by the IRB•  Many institutions can’t “promote” a study that is under their auspices Usability  Tes,ng  Medical  Devices   16  
    • Sample Flyer Usability  Tes,ng  Medical  Devices   17  
    • Recruiting•  Don’t be shy – makes friends with local hospitals and clinics –  Nurses and admins are the gateway –  Have the sanitized IRB approval in-hand•  Find groups that can be a gateway to your desired audience Source: http://www.toptenz.net/top-10-wwii-propaganda-posters.php –  Healthcare institutions –  Local and national support groups –  Federal agencies –  Community support groups –  Craigslist & Social Media•  Plan for surprises – like extensions to the timeline Usability  Tes,ng  Medical  Devices   18  
    • Working with Children•  From a legal standpoint, a child is someone under 18 years old Source: http://evilstaring.com/2011/06/08/amazingly-evil-child-3/•  Children cannot consent to participate in the study, they can only assent –  An assent form explains the details of the study, in the first person•  You will treat a 17 year old different than an 8 year old –  Be cognizant of the protocol to ensure you are consistent in your data capture•  The parent conundrum: should they participate? –  At what point does it become co-discovery and is that Source: http://cautionmagnet.blogspot.com/2011/05/confused-kids.html okay? Usability  Tes,ng  Medical  Devices   19  
    • Working with Children•  Put the child at ease: –  Explain the lab (to a point) –  Ask if s/he would like a break Source: http://lovefunnypictures.com/funny-pic-bush_funny_face.php –  Give reminders that s/he is doing a great job in the study –  Structure the study so that it isn’t drudgery –  Don’t make them remember rating scales –  Not all children like to be spoken to like a child•  Be extra vigilant for signs that the participant is telling you want you want to hear –  As well as signs of fatigue and confusion•  You don’t need to be their friend, but don’t scare them away Usability  Tes,ng  Medical  Devices   20  
    • Moderator’s Guide•  Be precise with a tightly written protocol Source: http://tips.tntdental.com/?tag=website-navigation•  Must be well-defined in advance and piloted –  No changes to the protocol without revisiting the IRB, the client, the FDA approval minutes, etc.•  Clients may be used to clinical trials and view usability methods as “thin” –  We’re not running a clinical trials, but that’s no excuse for not adequately preparing•  Adhere to the protocol! –  Any changes require IRB approval Usability  Tes,ng  Medical  Devices   21  
    • Moderator’s Guide•  The moderator’s guide should capture: Source: http://tips.tntdental.com/?tag=website-navigation –  The tasks –  Steps in the tasks –  What should be observed –  The questions to be asked of the participants –  Observer rating scales•  Basically, a shortened version of the Validation Plan•  Be precise in the metrics and methods –  They cannot vary –  Data reporting requirements will be significant –  Gaps in the reporting will be very visible Usability  Tes,ng  Medical  Devices   22  
    • Notes Grid•  We are using “micro-tasks” and want to capture precise information•  The notes grid should facilitate the analysis Source: http://beta.armagetronad.net/design/tron.html yet be easy to use –  Every step and question on its own row –  Have clear study sections (introduction, etc.) –  Grey out areas that can be ignored•  Team note taking: moderator concentrates on certain items, note taker does the rest Usability  Tes,ng  Medical  Devices   23  
    • Notes Grid Usability  Tes,ng  Medical  Devices   24  
    • Lab Preparation•  Capture the participants from two different angles –  A close-up view of the device interaction (over-the-shoulder) –  Straight-on to capture the entire field•  Have all device supplies at the ready –  Alcohol wipes, extra needles, batteries, Source: http://www.ebaumsworld.com/pictures/view/218095/ etc. –  You’ll use more supplies than you expect•  Get a first aid kit and nitrile gloves –  Have a first aid plan and practice Usability  Tes,ng  Medical  Devices   25  
    • Participant Safety•  As with any usability, participant comfort and safety is the top priority Source: http://en.wikipedia.org/wiki/File:Sign_first_aid.svg•  Must be mindful of participants with health conditions –  Will they need accessibility accommodations? –  Will they have an aide/caretaker? –  How will the condition affect their participation?•  What you can do to ensure participant comfort and safety: –  Have a robust first aid kit (and possibly medical personnel) –  Provide water, juice, sugary, and non-sugary snacks –  Ensure they know that they will be working with a placebo –  Accessible food Usability  Tes,ng  Medical  Devices   26  
    • Participant Safety•  Warn participants of safety hazards while using the device, if it’s part of the protocol Source: http://en.wikipedia.org/wiki/File:Sign_first_aid.svg•  Have a plan of action if someone is injured•  Establish a path of notification if a participant is injured –  Reporting may have many phases: reporting it internally, then to the client, to the FDA, and the IRB•  Educate yourself about the condition•  Ensure the protocol does not exacerbate a disease or condition•  Be vigilant about fatigue, confusion, disorientation, dehydration, & stress Usability  Tes,ng  Medical  Devices   27  
    • Adverse Effects•  An adverse effect (AE) event is when a participant tells you about a side effect or other experience while using a device or drug•  This is critical information for the client and must be reported, typically immediately Source: http://www.cartoonstock.com/directory/d/Din.asp•  Some clients may require you to go through AE training•  Before the study begins, determine to whom AEs are reported When in doubt, report!•  AEs can be complex to identify, but are VERY important Usability  Tes,ng  Medical  Devices   28  
    • Adverse Effects•  In the course of a usability study for a medication website, a participant mentions that she stopped taking the drug. She explains that the blister packs and pills sometimes arrived crushed.•  While testing an infusion pump, the participant mentions that the tube always falls out of his previously used pump, which is the client’s product.•  During a study, a participant mentions that her arms started to itch when she stopped taking the client’s medication.•  A male participant mentions that his wife got pregnant two weeks after he started taking the client’s medication, even though she used the birth control pill. Usability  Tes,ng  Medical  Devices   29  
    • Analysis•  The setup from the validation plan, moderator’s guide, and notes grid should make analysis easy Source: http://www.webseoanalytics.com/blog/how-to-conduct-market- research-competitor-analysis/•  Dealing with gaps in the data can be tricky –  Did the question get asked? –  Was the behavior observed? –  Do we have to go back to the recordings? •  Make sure you HAVE the recordings!•  Rely on quantitative measures –  What percentage of participants passed the task? –  What ease of use ratings did participants give? –  How many participants committed the most common errors? Usability  Tes,ng  Medical  Devices   30  
    • Analysis•  Remember to think on a micro-task scale – every little step of using the device should be observed –  It’s difficult to watch and report on every step of a complex interaction and document it precisely, Source: http://www.webseoanalytics.com/blog/how-to-conduct-market- research-competitor-analysis/ especially live –  Tallying needs to be specific, yet practical•  Be mindful of capturing and organizing the important qualitative data –  This can be especially tough when dealing with many participants –  Summarize qualitative data directly after the session – don’t wait•  Know what you’re reporting – there may be data to help marketing•  Sometimes the goal of these studies is to identify marketing claims, which can raise concerns Usability  Tes,ng  Medical  Devices   31  
    • Thank You!•  Conducting a usability study for a medical device is similar to a typical study, except with much stricter protocols Questions/Comments?•  Your safety and the safety & comfort of the Dan Berlin participants are paramount dberlin@madpow.net @banderlin•  Use quantitative measures and report percentages of participants that did or Chris Hass didn’t perform an action chass@madpow.net @chrishass•  It’s all about proper planning! Usability  Tes,ng  Medical  Devices   32