Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Userwise FDA Training Decay Research


Published on

Introduction to the FDA research project to investigate Training Decay.

Published in: Design
  • Be the first to comment

  • Be the first to like this

Userwise FDA Training Decay Research

  1. 1. Copyright © 2018, UserWise, Inc. Training Decay Shannon E. Clark, P.E. 1 Dan Nathan-Roberts, Ph.D., CHFP Intro to the Industry Consortium 1 October 24, 2018
  2. 2. Copyright © 2018, UserWise, Inc. 1 2 4 Statistical Plan Background Consortium Research Team Outline 5 2 3 Medical Product Selection
  3. 3. Copyright © 2018, UserWise, Inc. Background 3
  4. 4. Copyright © 2018, UserWise, Inc. What is a Training Decay for Usability Validation? “On the Form of Forgetting,” Wixted (1991) 4
  5. 5. Copyright © 2018, UserWise, Inc. GoodBad 5 Decay Curves
  6. 6. Copyright © 2018, UserWise, Inc. Injection Device Training Decays 6
  7. 7. Copyright © 2018, UserWise, Inc. Drawbacks of Real-Time Training Decay • Specialty surgeons from around the country • Training Decay may reduce the sponsor’s ability to recruit from diverse institutions • For products with unique user populations • Training Decay may reduce the sponsor’s ability to recruit as many users • Reduced number of participants who actually return for the usability validation study. • Training Decay introduce self-selection bias • Evaluating a team of participants at once (e.g. An Operating Room Team), and ensuring that the same participants return • Training Decay, and needing to mix teams, may introduce more study artifacts 7
  8. 8. Copyright © 2018, UserWise, Inc. Benefits of this study • The time required for training decay can increase the cost of conducting a usability study by 10-30% • A 30% increase in cost conservatively equates to a $101.25 Million dollar toll on manufacturers per year1 1. Based on a preliminary analysis performed by UserWise, Inc. 8 This study will provide objective evidence to either justify or decrease the “toll” on manufacturer/sponsor resources.
  9. 9. Copyright © 2018, UserWise, Inc. Objectives of this Study 9 Secondary Objectives: ● Identify differences in decay between device types ● Explore effects of “accelerated decay” ● Test the hypothesis that the difficulty of the task influences memory retention ● Determine if different types of tasks have different training decay curve profiles Primary Objective: Identify generalizable detailed training decay curves
  10. 10. Copyright © 2018, UserWise, Inc. Study Approach 10 Intended Users General population will be recruited to represent lay users Use Environment The study environment will represent an at-home setting Training Decay Participants will be subject to a specified training decay period Medical Devices Participants will be asked to use 1 of 3* medical devices *(Device type and number to be defined) Training The training will represent expected training in the field for each selected medical device No Decay 1 hour 24 hours 1 week 2 days
  11. 11. Copyright © 2018, UserWise, Inc. Latest Plan for Participant Cohort Sizes (still TBD) Training Decay Medical Device 1* Medical Device 2* Total No Training Decay N = 54 to 60 N = 54 to 60 N = 108 to 120 Training Decay = 1 hour N = 13 to 15 N = 13 to 15 N = 27 to 30 Training Decay = 1 day N = 13 to 15 N = 13 to 15 N = 27 to 30 Training Decay = 2 days N = 13 to 15 N = 13 to 15 N = 27 to 30 Training Decay = 7 days (FDA comments that there are a lot of drop-outs in more than 1 week) N = 13 to 15 N = 13 to 15 N = 27 to 30 11 *Sample sizes will change depending on the final number of devices and further assessment of appropriate effect size.
  12. 12. Copyright © 2018, UserWise, Inc. Consortium 12
  13. 13. Copyright © 2018, UserWise, Inc. Recruitment of the Consortium 13 Roles/Job TitlesSize of Company Recruit stakeholders from companies of all sizes, reflecting the makeup of the medical device and pharmaceutical industries Type of Products Recruit stakeholders who are developing a diverse set of products (e.g. at least 20 different products) Recruit a variety of roles, including: • R&D Engineering (in-house) • Human Factors Engineering (in-house) • Human Factors Consultants • Quality Engineering • Individuals with expertise in validation testing • Project Managers • Academia • Qualitative Researchers
  14. 14. Copyright © 2018, UserWise, Inc. Consortium Purpose (Phase 1) To provide input on the following questions: • Do you have access to research that could inform our effect size for sample size selection? • What medical product(s) or prototoype products are accessible and good candidates? Is any company open to sharing a design for a product intended for lay users that was “shelved” due to usability concerns? • For the selected devices, what use issues should we look out for? • What is “representative” training for the selected device(s)? • Do you agree with simulating training decay lengths of 1 hour, 1 day, 2 days, and 1 week? 14
  15. 15. Copyright © 2018, UserWise, Inc. Medical Product Selection 15
  16. 16. Copyright © 2018, UserWise, Inc. 16 Usability Criteria: ● Device has a high number of complaints or was recalled ● The IFU is difficult to follow (e.g. minimal pictures, confusing) ● Many critical use errors are possible (high severity) Basic Requirements: ● Device is intended for Lay Users ● Training is provided to users in the field ● The device is common within the industry ● Devices are within project budget Type of Device: ● Mechanical - mostly motor skills (e.g. autoinjectors, pens) ● Electro-Mechanical - mostly psychomotor skills (e.g. infusion pumps, AEDs) ● Software - mostly fact-memory skills (e.g. App-based monitoring wearables) Preliminary Selection Process Selecting the Medical Devices
  17. 17. Copyright © 2018, UserWise, Inc. Medical Device Selection 17 The device should be difficult to use to allow for: ● Differentiation between cohorts and training decay effects ● Prevent ceiling effect (tested for in pilot study) Tasks containing the following skill types will be included in the study: ● Motor skills (e.g. mechanical use of an autoinjector) ● Psychomotor skills (e.g. administering CPR compressions) ● Fact-memory skills (e.g. navigating a touch-screen) Two (2) devices will be tested instead of three (3) to: ● Increase the number of data points for each task ● Strengthen statistical significance of results
  18. 18. Copyright © 2018, UserWise, Inc. 18 Autoinjectors Multiple Use Pen Delivery System Prescription Nebulizers Infusion Pump Any other devices? Reconstitution Products Examples of medical devices for consideration: Medical Device Selection How are we identifying the devices to optimize meaningful results? ● Researching scholarly articles comparing usability of multiple products ● Identify products with a recall due to use errors or high number of reportable use-related complaints ● Review IFU (e.g. minimal pictures, confusing wording, etc.) to identify difficulty ● Select devices with where we can collect continuous data (e.g. Measure the dosage delivered in mL) ● Perform a detailed task analysis ● Conduct pilot study to detect ceiling effects
  19. 19. Copyright © 2018, UserWise, Inc. Candidate Devices: NovoSeven RT 19
  20. 20. Copyright © 2018, UserWise, Inc. Candidate Devices: Mix2Vial 20
  21. 21. Copyright © 2018, UserWise, Inc. Candidate Devices: Twinject Research article compared 4 autoinjectors Out of 48 participants, 0% correctly followed the device instructions, and only 1 preferred it over the others Source: 21
  22. 22. Copyright © 2018, UserWise, Inc. Candidate Devices: Colleague Infusion Pump The Colleague® IV infusion pump could shut down while delivering critical medication to patients. The reason for this was that the “on/off” key was so close to the “start” key that nurses would often inadvertently turn the pump off when they intended to start delivery. Over 206,000 Colleague infusion pumps, used mostly in hospitals, were recalled by the FDA.3,4 (Can build many tasks into 1 scenario) 22
  23. 23. Copyright © 2018, UserWise, Inc. Candidate Devices: MiniMed Paradigm Real-Time Revel 723 System There was a statistically significant difference in training times and error rates between the t:slim and Revel groups. The training time difference represented a 27% reduction in time to train on the t:slim versus the Revel pump. There was a 65% reduction in participants’ use error rates between the t:slim and the Revel group. The t:slim Pump had statistically significant training and usability advantages over the Revel pump. Conclusions: The reduction in training time may have been a result of an optimized information architecture, an intuitive navigational layout, and an easy-to-read screen. The reduction in use errors with the t:slim may have been a result of dynamic error handling and active confirmation screens, which may have prevented programming errors Usability and Training Differences Between Two Personal Insulin Pumps Noel E. Schaeffer, PhD, Linda J. Parks, MS, RN, CDE, Erik T. Verhoef, MBA, ... First Published October 14, 2014 Research Article . 23
  24. 24. Copyright © 2018, UserWise, Inc. Statistical Plan 24
  25. 25. Copyright © 2018, UserWise, Inc. Statistical Sample Size Plan 25 An effect size of 0.407 was used in power calculations to determine a minimum sample size of 111 participants was needed for the main phase of the study. This initial estimate was based on preliminary data available, however the effect size and corresponding power calculations need to be supported with additional data by means of one of the following options: ● 1. Obtain effect size from de-identified data from companies that have conducted training decay studies, 2. Obtain study results from companies that have conducted training decay studies, or 3. Conduct a literature review of relevant published studies
  26. 26. Copyright © 2018, UserWise, Inc. Statistical Analysis Plan 26 1. The success is measured as proportion of task success. ● Test the difference between task types ● Test the difference between timepoints ● Build a decay curve per task & task types 2. The success is measured as “success” or “failure” ● Build a model considering the interaction with the device ● Use the variable difficulty in the model ● Use confounding variables in the model 3. Possible advanced statistics (considering the time continuous) ● Fit the decay curve using a generalized linear model or a non-parametric approach ● Compare the different curves
  27. 27. Copyright © 2018, UserWise, Inc. Research Team 27
  28. 28. Copyright © 2018, UserWise, Inc. Principal Investigators Shannon has over 8 years of experience in medical device human factors engineering, design, risk management, V&V, CAD modeling, training development, marketing, compliance, and auditing quality systems. Certified Professional Industrial Engineer, California (2014) BS in Mechanical Engineering Certifications/Trainings: ● ISO 13485 Lead Auditor (2012) ● AAMI Quality System (2012) ● AAMI Risk Management (2012) ● AAMI Human Factors (2011) Shannon Clark – UserWise, Founder & CEO Dr. Dan NR. is an assistant professor of Industrial and Systems Engineering at San José State University. Dan has worked for Intel’s digital health division on telemedicine, the US Food and Drug Administration’s Center for Devices and Radiological Health (CDRH) on non-clinical medical device human factors, and the University of Michigan Stephen M. Ross School of Business as an innovation author and consultant. Dan’s research interests center on: ● Human Factors; physical, cognitive, and macro ergonomics ● Design Thinking and Innovation ● Wearable and Quantified Self technology ● User Centered Design research ● Patient-centered healthcare Dan Nathan-Roberts, PhD - San Jose State University 28
  29. 29. Copyright © 2018, UserWise, Inc. Study Team Members Kelli Sum Graduate Student SJSU Over 2 years of experience in medical device human factors research and engineering. BS in Industrial & Systems Engineering Jenni Quan Human Factors Engineer UserWise Over 2 years of experience in medical device human factors engineering. MS in Human Factors Engineering Nandini Gurunathan Human Factors Engineer UserWise Over 5 years of experience in medical device human factors and quality engineering. MS in Biomedical Devices Joined UserWise in 2015 Lana Sneath Human Factors Engineer UserWise Over 3 years of experience in medical device product development and process engineering. BS in Biomedical Engineering, MBA Denise Forkey Human Factors Specialist UserWise Over 20 years of experience in medical device human factors engineering, pre-clinical research, quality engineering, testing, and R&D. MS in Biomedical Engineering Joined UserWise in 2016 Kelli 29
  30. 30. Copyright © 2018, UserWise, Inc. Dr. Cristina Tortora is an assistant professor in Statistics at San José State University. She completed her PhD in Statistics at the University of Naples Federico II in 2012. She has previously held postdoctoral positions at McMaster University, the University of Guelph, and Stazione Zoologica Anton Dohrn of Naples. Her research focuses on data analysis, specifically on the development of advanced clustering techniques. She collaborated with experts in different applied fields, including environmental science, and industrial engineering. Dr. Cristina Tortora Samir is a Master of Information and Data Science candidate at University of California, Berkeley and a Research Data Analyst at the UCSF Memory & Aging Center. He completed his B.A. in Cognitive Science from Northwestern University in 2014, during which he held positions as a Research Assistant in various laboratories developing programs for performing data collection and data analysis. Samir Datta 30 Statistics Resources on Study Team
  31. 31. Copyright © 2018, UserWise, Inc. Questions? Shannon E. Clark, P.E. (650) 996-7480 31 Dan Nathan-Roberts, Ph.D., CHFP (408) 924-7501
  32. 32. Copyright © 2018, UserWise, Inc. Backup slides 32
  33. 33. Copyright © 2018, UserWise, Inc. How do we test for decay and bottoming out? • Test devices early in pilot study • Test group differences between time points • Example: • Psychomotor tasks (in blue) show insignificant change in success rate after 1 hour Decay overtime 33 Hypothetical Decay Curves
  34. 34. Copyright © 2018, UserWise, Inc. How do we test differences between task types? • Test groups differences within time points • Example: • Psychomotor skills have a higher success rate than fact-memory at 1 day, while the opposite is true at 1 hour Difference between task types 34 Hypothetical Decay Curves
  35. 35. Copyright © 2018, UserWise, Inc. 35 Additional Q & A Q: What if one group happens to be better than the other? A: If randomized, this should not happen. We can verify no difference by assessing cognitive scores and other demographics before time point 0. Q: What if one task happens to be harder than the other? A: By doing an analysis of success rate at time point 0, one should hope to find no differences (i.e. the tasks are equally difficult right after training, so any future differences truly have to do with decay). We are also collecting subjective ratings from participants on task difficulty.
  36. 36. Copyright © 2018, UserWise, Inc. 36
  37. 37. Relevant Qualities of Different Categories of Medical Devices Type of Device Complexity Risk Level Field training Cost Reusable Prescription Autoinjector High Medium/High Yes $1000 and up Automatic Pancreas System Very High High Yes $7000 new Negative Pressure Wound Treatment System Moderate High Yes $3500 new Infusion Pump High High Yes $600 and up Prescription Nebulizer Moderate High Yes $35 and up Home Dialysis Machines Very High High Yes $10,000 new. $2000 refurbished 37
  38. 38. Copyright © 2018, UserWise, Inc. Questions for the next phase Do the consortium members agree with: • Inclusion/Exclusion criteria for population? • Overall methodology? 38
  39. 39. Copyright © 2018, UserWise, Inc. Questions? Shannon E. Clark, P.E. (650) 996-7480 39 Dan Nathan-Roberts, Ph.D., CHFP (408) 924-7501
  40. 40. Copyright © 2018, UserWise, Inc. Kim, J. W., Ritter, F. E., & Koubek, R. J. (2013). An integrated theory for improved skill acquisition and retention in the three stages of learning. Theoretical Issues in Ergonomics Science, 14(1), 22-37. Anderson, G. S., Gaetz, M., & Statz, C. (2012). CPR Skill Retention of First Aid Attendants within the Workplace. Prehospital and Disaster Medicine Prehosp. Disaster Med., 27(04), 312-318. MacKinnon, D. J. (2007). How Individual Skill Growth And Decay Affect The Performance Of Project Organizations. Bjerrum, F., Maagaard, M., Sorensen, J. L., Larsen, C. R., Ringsted, C., Winkel, P., . . . Strandbygaard, J. (2015). Effect of Instructor Feedback on Skills Retention After Laparoscopic Simulator Training: Follow-Up of a Randomized Trial. Journal of Surgical Education, 72(1), 53-60. Kim, J. W. (n.d.), et. al. Investigation of Procedural Skills Degradation from Different Modalities. Palmer, R. (1990). Single-Channel Ground and Airborne Radio System (SINCGARS) Operator Training Evaluation. Woollard, M., Whitfield, R., Smith, A., Colquhoun, M., Newcombe, R. G., Vetter, N., & Chamberlain, D. (2004). Skill acquisition and retention in automated external defibrillator (AED) use and CPR by lay responders: A prospective study. Resuscitation, 60(1), 17-28. Sikström, S., & Jaber, M. Y. (2002). The power integration diffusion model for production breaks. Journal of Experimental Psychology: Applied, 8(2), 118-126. Jr., W. A., Jr., W. B., Stanush, P. L., & Mcnelly, T. L. (1998). Factors That Influence Skill Decay and Retention: A Quantitative Review and Analysis. Human Performance, 11(1), 57-101. Jr, S. M., & Segal, B. S. (2009). Critical action procedures testing: A novel method for test-enhanced learning. Medical Education, 43(12), 1182-1187. Lynch, B., Einspruch, E. L., Nichol, G., Becker, L. B., Aufderheide, T. P., & Idris, A. (2005). Effectiveness of a 30-min CPR self-instruction program for lay responders: A controlled randomized study. Resuscitation, 67(1), 31-43. Isbye, D. L., Meyhoff, C. S., Lippert, F. K., & Rasmussen, L. S. (2007). Skill retention in adults and in children 3 months after basic life support training using a simple personal resuscitation manikin. Resuscitation, 74(2), 296-302. Nielsen, A. M., Henriksen, M. J., Isbye, D. L., Lippert, F. K., & Rasmussen, L. S. (2010). Acquisition and retention of basic life support skills in an untrained population using a personal resuscitation manikin and video self-instruction (VSI). Resuscitation, 81(9), 1156-1160. Smith, A., Colquhoun, M., Woollard, M., Handley, A. J., Kern, K. B., & Chamberlain, D. (2004). Trials of teaching methods in basic life support (4): Comparison of simulated CPR performance at unannounced home testing after conventional or staged training. Resuscitation, 61(1), 41-47. Gronlund, S. Det, et. al.(2012). Remembering and Forgetting: From the Laboratory Looking Out. Choudhry, R. M., Hinze, J. W., Arshad, M., & Gabriel, H. F. (2012). Subcontracting Practices in the Construction Industry of Pakistan. Journal of Construction Engineering and Management J. Constr. Eng. Manage., 138(12), 1353-1359. Jaber, M. Y., & Bonney, M. (1996). Production breaks and the learning curve: The forgetting phenomenon. Applied Mathematical Modelling, 20(2), 162-169. Nembhard, D. A., & Uzumeri, M. V. (2000). Experiential learning and forgetting for manual and cognitive tasks. International Journal of Industrial Ergonomics, 25(4), 315-326. Sims, C. R, et. Al. (n.d.). Episodic versus Semantic Memory: An Exploration of Models of Memory Decay in the Serial Attention Paradigm Skinner, A. D. (2013). Retention and Retraining of Independent and Integrated Cognitive and Psychomotor Skills Related to Laparoscopic Surgery. Rohrer, D., & Pashler, H. (2007). Increasing Retention Without Increasing Study Time. Current Directions in Psychological Science Current Directions in Psychol Sci, 16(4), 183-186. Wickelgren, W. A. (1974). Single-trace fragility theory of memory dynamics. Memory & Cognition, 2(4), 775-780. Bolton, M. L., & Bass, E. J. (n.d.). Method for the formal verification of human-interactive systems. PsycEXTRA Dataset. Wixted, J. T. (2003). The Psychology And Neuroscience Of Forgetting. Sources 40