Your SlideShare is downloading. ×
0
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Testing in the healthcare domain
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Testing in the healthcare domain

4,718

Published on

Best practices paper on the risks, standards and challenges of Health Risk Management- Testing in the Healthcare domain by Devi.K from Siemens. Paper submitted during QAI's 12th International Software …

Best practices paper on the risks, standards and challenges of Health Risk Management- Testing in the Healthcare domain by Devi.K from Siemens. Paper submitted during QAI's 12th International Software Testing Conference

Published in: Business, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
4,718
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Testing in Healthcare Domain : Health Risk Management and Challenges Page 1 12th STC: 2012 Devi.K Oct 20 2012
  • 2. Agenda Overview What is Health Risk Mitigation Testing? –Risks , Standards Risk Management Process Flow Page 2 Risk Mitigation Tests - Unique to Healthcare Device Challenges faced during testing Healthcare Devices Conclusion
  • 3. Overview Is Testing of a Healthcare device software any different? Yes : Device software follows same principles of development and testing as any other software device Page 3 No : Includes special tests (hereon called “Health Risk Mitigation Tests” to ensure that the software is safe for use on humans for any diagnostic or therapeutic value
  • 4. What is Health Risk Mitigation Testing? Tests Performed in order to verify the mitigation of different health and safety risks with the software Increased Radiation Exposure & Electrical shock, Injuries, Extreme temperature contacts, clinical misdiagnosis, increased exposure to acoustic power Governed by different regulatory authorities & standards Page 4 Governed by different regulatory authorities & standards Regulatory Authorities: FDA, IEC, AIUM/NEMA International Safety Standards: Safety and Reg. Standards: safe for human use(FDA 21 CFR, IEC 62304 etc) Communication Standards : exchange of images and other Patient info. ( based on standards DICOM, HL7, IHE) Part of a Risk management process
  • 5. Risk Management Process Flow Hazard Identification Safety Classification Risk Estimation Implementation Verification Requirements for Risk Mitigation Page 5 Implementation of Risk Mitigation Measures Testing of Implemented Risk Mitigations Verification Reports for Risk Mitigation Measures
  • 6. Image Quality and Calibration Tests performed to verify the Image obtained to mitigate the risk of Clinical Misdiagnosis due to Poor Image Quality &/or Incorrect Image Calibration performed using a Phantom or a Simulator Data Integrity and Connectivity Tests Risk Mitigation Tests - Unique to Healthcare Device Page 6 Data Integrity and Connectivity Tests performed to verify that patient information and data are not lost when data is saved on the system memory or transferred between different network devices or to various media devices like USB, External Hard Drives etc. performed to mitigate each of the following risks. Undetected Loss of Patient Data due to irrecoverable failure or power outage or data corruption Corrupted Patient Data and Reports Mix Up between data of different Patients contd…
  • 7. Thermal and Acoustic Power Management Tests Performed to mitigate each of the following risks: Injury to Patient due to extreme temperature Injury to Patient due to Increased acoustic power Verification of Measurements and Calculations Performed to mitigate each of the following risks: Risk Mitigation Tests - Unique to Healthcare Device (contd…) Page 7 Performed to mitigate each of the following risks: Incorrect Diagnosis due to Inaccurate Analytical results and reports Tests identified based on Equivalence class Partitioning of over 1000 measurements and calculations that required to be verified Golden Dataset based tests for measurements that have a image dependency
  • 8. Challenges When Testing a HealthCare Device Common Test methodologies listed below did not provide the desired results; they had to be modified for testing of medical device software. Regression testing Stability testing Page 8 Exploratory testing
  • 9. Challenges When Testing a HealthCare Device Regression testing Problem: Regression tests did not always cover safety critical areas Solution: HRE-Health Risk Evaluation process of Impact Analysis (based on previous Experience; clinical scenarios; possible risks and hazard situations) Advantages: Clinical Interaction scenarios Improve the test coverage Stability testing Page 9 Stability testing Problem: Medical used on patients and high stability Solution: Established metric MTBF Advantages: Quantified Improvement in System Stability (MTBF goal = 150 hrs) Exploratory testing Problem: defects Missed due to Multiple possible Clinical Workflow /steps Solution: “Clinical Exploratory Testing; that are ”Typical clinical scenarios based Advantages: Defects that slipped into Clinical Validation phase reduced by 75%.
  • 10. Testing Healthcare Device software is : • Similar to any other Software Product under test • Can be tested by any test engineer with no good exposure to the domain Conclusion Page 10 domain • Major aspect that a Healthcare test engineer needs to always remember is the humanitarian aspect. That “The product that he/she is testing will be used to improve human lives and it is safe and effective for intended use.” And, any risk taken in testing is a “risk taken with a human life.”
  • 11. THANK YOU Page 11
  • 12. APPENDIX Page 12 APPENDIX
  • 13. Risks & Hazards in a Medical Device Typical Risks & Hazards in a Medical Device : Increased exposure to Radiation by Patient or Operator Electrical Shock to Patient or Operator Due to Electrical Hazards Patient or Operator Injury Due to Mechanical Hazards Injury to Patient due to extreme temperature Clinical Misdiagnosis in a patient Page 13 Clinical Misdiagnosis in a patient Patient Recall for Re-examination Injury to Patient due to Increased acoustic power
  • 14. Regulatory Authorities & International standards Regulatory Authorities : FDA- stands for Food and Drug Administration IEC - stands for International Electrotechnical Commission AIUM/NEMA – AIUM stands for American Institute of Ultrasound in Medicine and NEMA stands for National Electrical Manufacturers Association- standards for Acoustic and Thermal Power management in Ultrasound scanner Page 14 International standards : Safety and Regulatory Standards- that ensure that the product is safe for use on humans Communication Standards - Networking & Communication standards to enable exchange of medical images and other patient information between different medical devices & other 3rd party Vendor products.
  • 15. Safety and Regulatory Standards Some of the International Safety and Regulatory Standards are : • FDA 21 CFR –Standards for Design validation, Software validation and Risk analysis and Electronic Records. “FDA 510k” certification implies compliance. • IEC 62304 -ONLY specific standard for Medical Device Software. • ISO 13845:2003 –standards for Quality management system specific to medical device. • ISO 14971 -standard for “Risk management of medical device”. Page 15 • ISO 60601- standard for safety and essential performance of medical electrical equipment and medical electrical systems. • AIUM/NEMA Standard: by AIUM/NEMA are approved by FDA and IEC for Thermal and Acoustic power management and the real time display of the same. (This standard is ONLY applicable to Ultrasound Scanners) • MDD-provide regulatory requirements to be met to legally place a medical device on the European market. Devices that comply by MDD are provided “CE Marking”. CE- stands for "Conformité Européenne" which means "European Conformity".
  • 16. Communication Standards Some of the International Standards from this category are: DICOM – Standard for Digital Imaging and Communications in Medicine- relates to transfer of patient images. Health Level Seven (HL7) provides standards for the exchange, integration, sharing, and retrieval of electronic health information between medical applications. Page 16 applications. IHE-stands for Integrating the Healthcare Enterprise – This is the Integration of DICOM and HL7 standards to provide a standard for complete Healthcare workflow solutions.
  • 17. Image Quality and Calibration Tests are : performed to verify the Image that is generated by an Ultrasound scanner Need to verify each of the following characteristics: Image Quality Correctness in terms of the Image representation Calibration Accuracy Risk Mitigation Tests - Image Quality and Calibration Tests- Page 17 These tests mitigate each of the following risks: Clinical Misdiagnosis due to Poor Image Quality Clinical Misdiagnosis due to Incorrect Image Calibration performed using a Phantom or a Simulator that mimics the human tissues or blood flow (figure is of a Human Tissue Mimicking Phantom)
  • 18. Data Integrity Tests : are performed to mitigate each of the following risks. Undetected Loss of Patient Data Loss of data due to irrecoverable failure or power outage. Loss of data due to Data Corruption (probably due to a Software Crash) are performed to verify that patient information and data are not lost when data Risk Mitigation Tests - Data Integrity Tests Page 18 are performed to verify that patient information and data are not lost when data is saved on the system memory. tested by Simulating a Data Recovery /Crash Recovery mechanism implementation
  • 19. Connectivity Tests : are performed to mitigate each of the following risks. Undetected Loss of Patient Data Corrupted Patient Data and Reports Mix Up between data of different Patients are performed to verify that the patient information and data are not lost when they are transferred between different network devices or to various media devices like Risk Mitigation Tests - Connectivity Tests Page 19 are transferred between different network devices or to various media devices like USB, External Hard Drives etc. based on compliance to International Communication standards –DICOM, HL7, and IHE are performed using different simulators available that help test the systems to ensure that they are compliant with the standards contd…
  • 20. Risk Mitigation Tests - Connectivity Tests (contd.) A typical Connectivity Workflow Network that needs be simulated and tested in a lab environment is depicted in the below diagram. Page 20
  • 21. Thermal and Acoustic Power Management Tests : performed to ensure Thermal Power (surface temperature of the Transducer) and acoustic power are within safe limits. These tests mitigate each of the following risks: Injury to Patient due to extreme temperature Injury to Patient due to Increased acoustic power are System tests that involve the following tests: Risk Mitigation Tests - Thermal and Acoustic Power Management Tests Page 21 are System tests that involve the following tests: Tests to Thermal Indices (TI) and Mechanical Indices (MI) are always within the limits as specified by AIUM/NEMA standards. Tests to ensure the Transducer surface temperature is always within the limits as specified by AIUM/NEMA standards. verifies that limits are displayed to the physician as per NEMA standards as follows : Tests to ensure that the TI and MI indices and the power transmitted to the system (called Transmit Power) are displayed to the user under all conditions. Tests to ensure that the Transmit Power is system and user-controllable.
  • 22. Verification of Measurements and Calculations:: are performed to ensure that the analysis results and reports provided by the Ultrasound scanner are accurate. These tests mitigate each of the following risks: Incorrect Diagnosis due to Inaccurate Analytical results and reports tests performed based on Equivalence class Partitioning of over 1000 measurements and calculations that required to be verified Risk Mitigation Tests - Verification of Measurements and Calculations Page 22 measurements and calculations that required to be verified Golden Dataset based Measurement for measurement results that were generated only for specific anatomical images mock patient studies are created as Test datasets coined as “Golden Datasets contd….
  • 23. The Procedure followed to test based on Golden Datasets is depicted in the following flow diagram: Risk Mitigation Tests - Verification of Measurements and Calculations (contd…) Page 23
  • 24. Challenges –Regression Testing Problem: Regression tests identification was challenging since Impact analysis based on the Code/Design and requirements and based on existing functionality or based on impact analysis on defects solved was not effective. It is critical that safety critical areas are covered. Solution: Impact Analysis and Regression tests based on HRE-Health Risk Evaluation process that involves: • Impact Analysis for a feature done based on previous Experience Page 24 • Impact Analysis for a feature done based on previous Experience • Impact analysis done based on clinical scenarios that involve the affected feature • Impact Analysis considering the possible risks and hazard situations that may arise in interaction with the impacted feature • HRE process done with key stakeholders like Clinical Experts, Subject Matter Experts, Architects and Designers Advantages: Clinical Interaction scenarios identified Improve the test coverage
  • 25. Challenges –Stability Testing Problem: Medical Device will be used on patients for diagnostic data and so it is extremely important to test and ensure that the device is stable. Solution: Established metric MTBF as per below steps : • Created an automated “Clinical Workflow procedure” that consists of typical Clinical Use cases (called “A Patient Exam”) that mimic everyday workflows in hospital environments. • Randomization of each of the Steps that increased possible use cases Page 25 • Randomization of each of the Steps that increased possible use cases • MTBF (Mean Time Between Failure) Metric is used to measure the Stability. MTBF = No. of Test Hours / Total no. of Failures where Failure = incomplete “exam” • An MTBF goal set for different stages of development. • Any failures which brought the MTBF below the goal are resolved on highest priority during development so as to pull the MTBF back to the Goal. Advantages: Quantified Improvement in System Stability (goal of 150 hrs of MTBF ) contd…
  • 26. Challenges –Stability Testing(contd…) An example graph to show how MTBF is closely monitored and tracked on a daily basis. It is kept under control across all stages of development. Page 26
  • 27. Challenges –Exploratory Testing Problem: Multiple Clinical Workflow /steps possible to test a requirement. Solution: • Redefined in the Healthcare perspective to be called as “Clinical Exploratory Testing.”Typical clinical scenarios (called Clinical Workflows) identified and tested. • This is planned during all stages of a release. • Different types of exploratory testing are planned based on the phase of the Page 27 • Different types of exploratory testing are planned based on the phase of the project. Advantages: Understanding the Clinical use case and hence introducing Clinical Scenario-based exploratory testing, the defects that slipped into Clinical Validation phase of a release could be reduced by 75%. contd…..
  • 28. Challenges –Exploratory Testing(contd…) The types of Exploratory Tests across different release stages are shown in the below illustration. Page 28 ** Defects are referred to as Charms in the illustration

×