There is no such thing as                                User ErrorPRESENTER:                Gavin Lew            Korey Jo...
Company Overview: Snapshot Profile• Founded in 1999; privately held firm with over 50 full time UX  research and design co...
Introduction: In Search of Use Errors                     © July 31, 2012 – Proprietary and Confidential   3
IntroductionNo One Wants to Make a Device that Causes Errors                    © July 31, 2012 – Proprietary and Confiden...
IntroductionUsual Suspects When Use Errors Are Committed Designers cannot ever think that being “highly                  ...
IntroductionIncreased Vulnerability While it is true that HCPs are highly educated, often the tasks  performed puts them ...
IntroductionDon’t Forget Other Users… Originally, pharmacists were not considered to be a relevant  user group for use er...
IntroductionWhat About Patients? Especially when a medical device used by HCPs is  “repurposed” for use with patients…!  ...
IntroductionPatients… Are incredibly diverse   – Literacy, education, experience Receive a wide range of training   – Sk...
IntroductionA Truly Diverse User Group                    © July 31, 2012 – Proprietary and Confidential                  10
IntroductionMust Users Really Shoulder the Blame?                    © July 31, 2012 – Proprietary and Confidential       ...
IntroductionDefinition of a Use Error Use errors are made by humans and can often be  attributed to the design itself “U...
IntroductionFocus Towards Improved Risk Management In an effort to improve  patient safety…   – The healthcare industry  ...
What is FMEA?                © July 31, 2012 – Proprietary and Confidential   14
What is FMEA?FMEA: Failure Modes and Effects Analysis Answers the simple question:       “If a system component fails, wh...
What is FMEA?Engineers: Two types of FMEA Risk Analyses1. DESIGN FMEA. A design FMEA is used to examine the   components o...
What is FMEA?Wait — Where is the User in the FMEA?                    © July 31, 2012 – Proprietary and Confidential      ...
How to…Need to Focus on the User in the FMEA   “If a system component fails, what is the effect           on system perfor...
How to Conduct an FMEA                  © July 31, 2012 – Proprietary and Confidential   19
How to…Towards a More “Use–Error Focused-–FMEA” Expanding upon a chapter entitled “Risk management  in medical products” ...
How to…But First, Here is an Example FMEA   FMEA appears in many different styles   Ultimately, it is really a worksheet...
How to…12-Step Process (Israelski & Muto, 2012)1.    Form a Team2.    Perform a Task Analysis3.    Start a Worksheet4.    ...
How to…Step 1: Form a Team Team composition includes representatives from:   –   Product team   –   Regulatory   –   Huma...
How to…Step 2: Conduct a Task Analysis                                                              Back to               ...
How to…Step 3: Start a Worksheet Settle on the format and columns Consider using large sheets of paper (11” x 17” or lar...
How to…Step 4: Brainstorm Potential Use Errors (Failure Modes) Follow typical brainstorming rules (i.e., facilitate  idea...
How to…Step 5: List Potential Effects of Each Failure Mode For each use error, describe the potential harm  resulting fro...
How to…Step 6: Assign Severity Ratings (S) Assign a quantitative level of the Severity of the Harm Various levels of (S)...
MethodologyExample of a Five-level Severity Scale (S) Three-level Severity Scale   – “Severe, Moderate, Negligible”      ...
How to…Step 7: Estimate the Probability of Occurrence Ratings (P) Assign a quantitative level of the Probability of the u...
MethodologyStep 7: Estimate the Probability of Occurrence Ratings (P) (P) levels can be 3 or 10 point scales derived from...
MethodologyStep 7: Estimate the Probability of Occurrence Ratings (P) Consider that this (P) encompass situations and  ci...
How to…Step 8: Derive a Risk Index (S x P) Risk Level = Severity x Probability Forms a Risk Priority Number (RPN) Compa...
MethodologyRisk Evaluation Matrix Regulators require special attention be paid to all  catastrophic harm severities regar...
How to…Step 9: Prioritize the Risks by Risk Index Values Sort by Risk Index Level Consider Risk Control Measures   – Fai...
How to…Step 10: Take Action to Eliminate or Reduce Risk Talking point:                     © July 31, 2012 – Proprietary ...
IntroductionReally Cool Opening Scene in a Nuclear Missile Silo West Wing star was more skeptical when a message  came th...
IntroductionForcing the Reservoir Dogs Star to Suggest He Comply “Turn your key, sir!”                    © July 31, 2012...
IntroductionPotentially Dangerous Actions Should Require More Effort This is where design can play a role to mitigate use...
How to…Step 11: Estimate the Effectiveness of Actions Did the Risk Control Measure make the device safer by:   –   Elimin...
Taking FMEA a Step Further…                   © July 31, 2012 – Proprietary and Confidential   41
A Step Further…Addendum to the 12 Steps… This paper seeks to evolve the process and take it to the  next level The criti...
A Step Further…Where Does Brainstorming Occur?                    © July 31, 2012 – Proprietary and Confidential          ...
A Step Further…Review FMEA in a Simulated Environment                   © July 31, 2012 – Proprietary and Confidential    ...
A Step Further…Review FMEA in a Simulated Environment                   © July 31, 2012 – Proprietary and Confidential    ...
A Step Further…Break Out When Necessary Should a set of use errors be uncovered…   – Break off sub-team to explore the ne...
A Step Further…Takeaways An FMEA process combined with Task Analysis improves  risk management Addition of simulated env...
References   ISO 14971:2007 (with special emphasis on Annex D and E)   Flynn, E.A., Barker, K.N., & Carnahan, N.J. (2003...
Questions?PRESENTER:             Gavin Lew              Korey Johnson             Managing Director      Associate Directo...
Upcoming SlideShare
Loading in...5
×

There is No Such Thing as User Error

971

Published on

Medical device design necessitates an assessment of risk. The need is prudent given their ecosystem of use, but also mandated by regulatory agencies. Risk management uses analysis protocols to identify and predict situations where a device may fail and assess the consequences. Common techniques, like Failure Mode and Effects Analysis (FMEA) must expand to go beyond engineering and material failure to identify hazard risks due to user-device interactions or “use errors.”

During this webinar, we will discuss the practice of the FMEA with added upgrades, such as a Task Analysis and how integration with simulated use labs can help you and your design team be more successful at reducing patient risk.

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
971
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
4
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Engineers/Designers often build to a spec (set of bullets) Context can easily be lost But, no one wants to create something that is confusing
  • The usual suspects who commit Use Errors are Healthcare Professionals [NEXT] Physician Assistants, Nurses, and Physicians are highly trained individuals. They tend to commit use errors. They tend to be the user when a hazardous situation occurs… Sometimes leading to catastrophic results and possibly device recall. [NEXT]
  • Learn in UX research… Engineers are often removed from the user experience Lack of context
  • Medical Device Reports
  • Seven approaches are commonly employed to estimate probabilities: ⎯ use of relevant historical data; ⎯ prediction of probabilities using analytical or simulation techniques; ⎯ use of experimental data; ⎯ reliability estimates; ⎯ production data; ⎯ post-production information; ⎯ use of expert judgment.
  • FDA and regulators require special attention be paid to all catastrophic harm severities reqardless of likelihood estimates being low. (such as improbable) They have seen too many manufacturers low balling the likelihood estimates.
  • Lowers in acceptability as the line goes down
  • This paper seeks to evolve the process and take it to the next level The critical outcome of the FMEA with a Task Analysis is a list of potential use errors. However, these potential use errors are limited to what can be ideated in the brainstorming activity. All too often, these brainstorming activities occur in a closed office environment or even worse, sometimes conducted without the device!
  • Situations are better with more robust environments Painted mannequins and customized to expected use conditions.  The sim lab can develop indication-specific clinical scenarios that would exhibit patient outcomes that any failures may cause.
  • Connections, audio alarms, lighting,…
  • There is No Such Thing as User Error

    1. 1. There is no such thing as User ErrorPRESENTER: Gavin Lew Korey Johnson Managing Director Associate Director User Centric, Inc. User Centric, Inc.MODERATOR: #uxlunch Pamela Stoffregen-Gay Sr. Marketing Mgr. Join the discussion! User Centric, Inc. @UserCentricInc
    2. 2. Company Overview: Snapshot Profile• Founded in 1999; privately held firm with over 50 full time UX research and design consultants • Most have graduate degrees in psychology, human factors / ergonomics, ethnography, design and human computer interaction• In 2007, 2008, 2009, 2010, 2011 User Centric was included on the Inc. 5000 List of the fastest growing companies in the U.S.• UC is arguably the strongest user experience firm in the US • Healthcare (medical devices and HIT: EHRs) © 2011 – Proprietary and Confidential 2
    3. 3. Introduction: In Search of Use Errors © July 31, 2012 – Proprietary and Confidential 3
    4. 4. IntroductionNo One Wants to Make a Device that Causes Errors © July 31, 2012 – Proprietary and Confidential 4
    5. 5. IntroductionUsual Suspects When Use Errors Are Committed Designers cannot ever think that being “highly trained” solves a medical device that is poorly designed © July 31, 2012 – Proprietary and Confidential 5
    6. 6. IntroductionIncreased Vulnerability While it is true that HCPs are highly educated, often the tasks performed puts them into an “automatic mode” which makes them more vulnerable to use errors** Guenter, et al (2008). Enteral Feeding Misconnections: A Consortium PositionStatement, The Joint Commission Journal on Quality and Patient Safety, 34(5):282-292. © July 31, 2012 – Proprietary and Confidential 6
    7. 7. IntroductionDon’t Forget Other Users… Originally, pharmacists were not considered to be a relevant user group for use errors, but Flynn, et al. (2003) estimated 51.5 million errors occur per year (4 per day per pharmacy) Moreover, a human factors-related recall with pharmacists as the user as recent as September 23, 2011 © July 31, 2012 – Proprietary and Confidential 7
    8. 8. IntroductionWhat About Patients? Especially when a medical device used by HCPs is “repurposed” for use with patients…! © July 31, 2012 – Proprietary and Confidential 8
    9. 9. IntroductionPatients… Are incredibly diverse – Literacy, education, experience Receive a wide range of training – Skewed towards no training Have different perspective and context Live in varied environments – i.e., uncontrolled environments – Consider: dogs, younger children Do research long enough and patients resemble… © July 31, 2012 – Proprietary and Confidential 9
    10. 10. IntroductionA Truly Diverse User Group © July 31, 2012 – Proprietary and Confidential 10
    11. 11. IntroductionMust Users Really Shoulder the Blame? © July 31, 2012 – Proprietary and Confidential 11
    12. 12. IntroductionDefinition of a Use Error Use errors are made by humans and can often be attributed to the design itself “Use” as opposed to “human error” or “user error” removes blame from the user as the cause of the failure Targeting design deficiencies of the design itself links remedy to design changes (earlier the better) Often, we focus on the device itself, but – Interaction with other devices, connections (things) and – Instructions also contribute to use errors © July 31, 2012 – Proprietary and Confidential 12
    13. 13. IntroductionFocus Towards Improved Risk Management In an effort to improve patient safety… – The healthcare industry has focused more attention on risk management tools – Awareness of the important role that human factors plays in the design process – By driving more human factors into the design process sooner, rather than later, results in more positive change © July 31, 2012 – Proprietary and Confidential 13
    14. 14. What is FMEA? © July 31, 2012 – Proprietary and Confidential 14
    15. 15. What is FMEA?FMEA: Failure Modes and Effects Analysis Answers the simple question: “If a system component fails, what is the effect on system performance or safety?” Usage dates back to the 1940’s to evaluate military M&P to identify risks using a bottom-up assessment As technology advanced into areas, such as space exploration and nuclear power, the need to identify, prioritize and manage risk amplified  At that time, assessment of risk in the design of systems tended to involve engineering and material failure. © July 31, 2012 – Proprietary and Confidential 15
    16. 16. What is FMEA?Engineers: Two types of FMEA Risk Analyses1. DESIGN FMEA. A design FMEA is used to examine the components of a product to identify potential failures. – For example, in the automotive industry, a design FMEA is conducted on all components and subsystems of a new car during the design and manufacturing phases. – The FMEA tool is used to evaluate the correctness of the materials, accuracy of specifications, and all other elements of design required to make a safe automobile.1. PROCESS FMEA. A process FMEA is used to analyze the processes used to make a product. © July 31, 2012 – Proprietary and Confidential 16
    17. 17. What is FMEA?Wait — Where is the User in the FMEA? © July 31, 2012 – Proprietary and Confidential 17
    18. 18. How to…Need to Focus on the User in the FMEA “If a system component fails, what is the effect on system performance or safety?” Severity of Harm (S) “If a user commits an error, what is the effect Probability (P) on system performance or safety?” (S x P) © July 31, 2012 – Proprietary and Confidential 18
    19. 19. How to Conduct an FMEA © July 31, 2012 – Proprietary and Confidential 19
    20. 20. How to…Towards a More “Use–Error Focused-–FMEA” Expanding upon a chapter entitled “Risk management in medical products” by Ed Israelski and Bill Muto  In Pascale Carayon’s recently published textbook, Handbook of Human Factors and Ergonomics in Health Care and Patient Safety, Second Edition Ed and Bill describe a step-by-step process to identify hazards and integrate it into the medical product development process They advocate one major addition to the FMEA procedure which adds the user into the process– Perform a Task Analysis as this can predict user behaviors that relate to design interaction © July 31, 2012 – Proprietary and Confidential 20
    21. 21. How to…But First, Here is an Example FMEA FMEA appears in many different styles Ultimately, it is really a worksheet (table) Used by the team to document and work through issues Often has color to pull attention to specific rows © July 31, 2012 – Proprietary and Confidential 21
    22. 22. How to…12-Step Process (Israelski & Muto, 2012)1. Form a Team2. Perform a Task Analysis3. Start a Worksheet4. Brainstorm Potential Use Errors (failure modes)5. List potential Effects of each Failure Mode / operator error6. Assign Severity ratings (S)7. Estimate the Probability of occurrence ratings (P)8. Derive a Risk Index (S x P)9. Prioritize Risks by risk index values10. Take Actions to eliminate or reduce the high priority failure modes11. Estimate Effectiveness of action ratings12. Revise the Risk PrioritiesIsraelski, E., & Muto. W. (2012). Risk management in medical products. In P. Carayon (Ed.), Handbookof Human Factors and Ergonomics in Health Care and Patient Safety, Second Edition. CRC Press. © July 31, 2012 – Proprietary and Confidential 22
    23. 23. How to…Step 1: Form a Team Team composition includes representatives from: – Product team – Regulatory – Human Factors – Engineering (Development) – Medical Affairs – Quality Assurance – Training – User Group representatives Agree upon goals and procedures to conduct the FMEA © July 31, 2012 – Proprietary and Confidential 23
    24. 24. How to…Step 2: Conduct a Task Analysis Back to Entry Just-browsing... Point Seeing whats available... Just Browsing for Info by: - Act - Venue - Date - Genre User may be first-time or repeat visitor - Geographic Area User is just browsing for information on upcoming events - Upcoming Events "I want to see whats coming soon" "Whats "What would "What is "What deals playing at the be fun to do available this are available?" Metro?" with the weekend?" family?" Try to limit browse to 3-4 steps Browse by Name Browsing by Genre Browsing by Date Browse by Deals (Event / Venue) Show me whats coming up Show Genre Show me latest deals this weekend Search by Name Promo on Home Select Genre Show search results Search by Date List of Acts: Metro List of Venues: Metro Link to Promos e.g., Amazon.com Show sub-category lisitng Show Genre AlphaNumeric Jum Search by full listing? Select Promo Select Act or Select sub- Venue categories Show Conduct a formal task analysis Events Show Events Show List of Acts Show Show List of Events Events Sort by State Sort by Venue AlphaNumeric Jump Jump to Date Closest to Zip Don’t get lured into “we’ll do the Search by full listing? Search by Price Sort by State TM.com should help Show Anything No - Try another venue? Events interesting? - List headliners task analysis AND brainstorm” Yes TM.com should offer help - Email new dates Tix No Depression - Avail tickets Avail? - Next Concert - etc. Yes Select Provide team with tasks, user Performance Go to Exit Buy Ticketmaster.com profiles, use environment, and examples of use errors Save “brainstorming” for identifying use errors © July 31, 2012 – Proprietary and Confidential 24
    25. 25. How to…Step 3: Start a Worksheet Settle on the format and columns Consider using large sheets of paper (11” x 17” or larger) with gridlines Documenting electronically via MS Excel and projecting screen to a wall can facilitate group interactions and discussions © July 31, 2012 – Proprietary and Confidential 25
    26. 26. How to…Step 4: Brainstorm Potential Use Errors (Failure Modes) Follow typical brainstorming rules (i.e., facilitate ideation over debate over merit discussions) Recognize efficiencies of breaking into sub- groups to reduce “tangent takeover” Actually helps to have the device (sounds silly…) Use data from customer complaints or MDRs © July 31, 2012 – Proprietary and Confidential 26
    27. 27. How to…Step 5: List Potential Effects of Each Failure Mode For each use error, describe the potential harm resulting from the Potential Effect When multiple harms can occur – List harms and elevate worst-case scenarios to the top, so they can be addressed first © July 31, 2012 – Proprietary and Confidential 27
    28. 28. How to…Step 6: Assign Severity Ratings (S) Assign a quantitative level of the Severity of the Harm Various levels of (S) are often used © July 31, 2012 – Proprietary and Confidential 28
    29. 29. MethodologyExample of a Five-level Severity Scale (S) Three-level Severity Scale – “Severe, Moderate, Negligible” © July 31, 2012 – Proprietary and Confidential 29
    30. 30. How to…Step 7: Estimate the Probability of Occurrence Ratings (P) Assign a quantitative level of the Probability of the use error occurring Various levels of (P) are often used © July 31, 2012 – Proprietary and Confidential 30
    31. 31. MethodologyStep 7: Estimate the Probability of Occurrence Ratings (P) (P) levels can be 3 or 10 point scales derived from: – Historical data, prediction of probabilities using analytical or simulation techniques, experimental data, and even expert judgment. © July 31, 2012 – Proprietary and Confidential 31
    32. 32. MethodologyStep 7: Estimate the Probability of Occurrence Ratings (P) Consider that this (P) encompass situations and circumstances, as well as, an entire sequence of events from the occurrence of the initiating cause through to the occurrence of harm – Does the hazardous situation occur in the absence of a failure? – Does the hazardous situation occur in a fault condition? – Does the hazardous situation occur only in a multiple-fault condition? – How likely is it that a hazardous situation will lead to harm? – The likelihood that a hazardous situation will lead to harm is influenced by the life-cycle of the medical device and estimated number of devices in the market © July 31, 2012 – Proprietary and Confidential 32
    33. 33. How to…Step 8: Derive a Risk Index (S x P) Risk Level = Severity x Probability Forms a Risk Priority Number (RPN) Compare to a Risk Evaluation Matrix © July 31, 2012 – Proprietary and Confidential 33
    34. 34. MethodologyRisk Evaluation Matrix Regulators require special attention be paid to all catastrophic harm severities regardless of (P) – Historically, too many manufacturers have low balled (P) © July 31, 2012 – Proprietary and Confidential 34
    35. 35. How to…Step 9: Prioritize the Risks by Risk Index Values Sort by Risk Index Level Consider Risk Control Measures – Failure modes that can be addressed immediately – Failure modes that require no action © July 31, 2012 – Proprietary and Confidential 35
    36. 36. How to…Step 10: Take Action to Eliminate or Reduce Risk Talking point: © July 31, 2012 – Proprietary and Confidential 36
    37. 37. IntroductionReally Cool Opening Scene in a Nuclear Missile Silo West Wing star was more skeptical when a message came through and was hesitant to turn his key… © July 31, 2012 – Proprietary and Confidential 37
    38. 38. IntroductionForcing the Reservoir Dogs Star to Suggest He Comply “Turn your key, sir!” © July 31, 2012 – Proprietary and Confidential 38
    39. 39. IntroductionPotentially Dangerous Actions Should Require More Effort This is where design can play a role to mitigate use errors and improve patient safety Many medical device manufacturers mitigate risk based on the ALARP (As Low As Reasonably Practicable) – Accepts that there is a practical limit to the time, effort, and money that can be spent trying to drive a risk to zero Still, manufacturers must justify – Burden is to show that cost required to reduce the tolerable risk further would be grossly disproportionate to the benefit gained and risks are those that are improbable and marginally severe thereby requiring no further mitigation Make it hard for users to do something wrong and easier for them to do it right © July 31, 2012 – Proprietary and Confidential 39
    40. 40. How to…Step 11: Estimate the Effectiveness of Actions Did the Risk Control Measure make the device safer by: – Eliminating the particular hazard? – Reducing the Severity (S) and/or Probability (P)? – Adding protective measures, such as safeties or alarms? – Providing warning labels and instructions? Step 12: Revise Risk Priority (Risk Acceptability) ISO 14971:2007 Annex Updates © July 31, 2012 – Proprietary and Confidential 40
    41. 41. Taking FMEA a Step Further… © July 31, 2012 – Proprietary and Confidential 41
    42. 42. A Step Further…Addendum to the 12 Steps… This paper seeks to evolve the process and take it to the next level The critical outcome of the FMEA with a Task Analysis is a list of potential use errors  However, these potential use errors are limited to what can be ideated in the brainstorming activity  All too often, these brainstorming activities occur in a closed office environment or even worse, sometimes conducted without the device! © July 31, 2012 – Proprietary and Confidential 42
    43. 43. A Step Further…Where Does Brainstorming Occur? © July 31, 2012 – Proprietary and Confidential 43
    44. 44. A Step Further…Review FMEA in a Simulated Environment © July 31, 2012 – Proprietary and Confidential 44
    45. 45. A Step Further…Review FMEA in a Simulated Environment © July 31, 2012 – Proprietary and Confidential 45
    46. 46. A Step Further…Break Out When Necessary Should a set of use errors be uncovered… – Break off sub-team to explore the new line of errors while the main team will continue down the list © July 31, 2012 – Proprietary and Confidential 46
    47. 47. A Step Further…Takeaways An FMEA process combined with Task Analysis improves risk management Addition of simulated environments can increase the number of identified use errors and precision The goal is not to box the answer correctly the first time, but to get smarter all through the research process – FMEA should continue to evolve from exploratory to validation testing © July 31, 2012 – Proprietary and Confidential 47
    48. 48. References ISO 14971:2007 (with special emphasis on Annex D and E) Flynn, E.A., Barker, K.N., & Carnahan, N.J. (2003). National Observational Study of Prescription Dispensing: Methods, J Am Pharm Assoc. 43(2): 191- 200. Guenter, P., Hicks, R.W., Simmons, D., Crowley, J., Joseph, S., Croteau, R., Gosnell, C., Pratt, N.G., & Vanderveen, T.W. (2008). Enteral Feeding Misconnections: A Consortium Position Statement, The Joint Commission Journal on Quality and Patient Safety, 34(5): 282-292. Israelski, E., & Muto. W. (2012). Risk management in medical products. In P. Carayon (Ed.), Handbook of Human Factors and Ergonomics in Health Care and Patient Safety, Second Edition. CRC Press. Kaye, R,, & Crowley, J. (2000). Medical device use-safety: Incorporating human factors engineering into risk management. Food and Drug Administration Logan, M.K. (2010). A roundtable discussion: Increased focus on human factors drives device safety improvements. Human Factors Horizons. 8-15 © July 31, 2012 – Proprietary and Confidential 48
    49. 49. Questions?PRESENTER: Gavin Lew Korey Johnson Managing Director Associate Director User Centric, Inc. User Centric, Inc. glew@usercentric.com kjohnson@usercentric.com #uxlunch Up next: Conducting Longitudinal Mobile Studies Tuesday, Aug 21, 12:00 PM CDT Join the discussion! www.usercentric.com/webinars @UserCentricInc 49
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×