SW Validation of AI-Based Medical Devices- MedDev Soft
•Download as PPTX, PDF•
0 likes•64 views
The regulation of AI-Based Medical Devices is still unclear. How can we responsibly adopt these new technologies while remaining accountable to their suggestions?
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to SW Validation of AI-Based Medical Devices- MedDev Soft
Similar to SW Validation of AI-Based Medical Devices- MedDev Soft (20)
Recently uploaded
Recently uploaded (20)
SW Validation of AI-Based Medical Devices- MedDev Soft
- 1. SW VALIDATION OF AI BASED MEDICAL DEVICES February 3, 2020 by: Dina Sifri, CEO MedDev Soft
- 2. Terms
- 3. 3 • Artificial intelligence (AI), sometimes called Machine Intelligence (MI), is defined as the science and engineering of development an intelligence demonstrated by machines or computer programs, in contrast to the natural intelligence displayed by humans. Artificial intelligence can use different techniques, including models based on statistical analysis of data, expert systems that primarily rely on if-then statements, and machine learning. • Machine learning (ML) is the scientific study of algorithms and statistical models that computer systems use to perform a specific task without using explicit instructions, relying on patterns and inference instead. Machine Learning is an artificial intelligence technique that can be used to design and train software algorithms to learn from and act on data. • Machine Learning can be used to create an algorithm that is ‘locked’ so that its function does not change, or ‘adaptive’ so its behavior can change over time based on new data. What is Artificial Intelligence and Machine Learning?
- 4. 4 • Deep learning is part of a broader family of machine learning methods based on artificial neural networks with representation learning. Feature learning or representation learning is a set of techniques that allows a system to automatically discover the representations needed for feature detection or classification from raw data. Deep learning uses multiple layers to progressively extract higher level features from the raw input. Learning can be supervised, semi-supervised or unsupervised. • Artificial intelligence (AI) in healthcare is the use of complex algorithms and software to emulate human cognition in the analysis of complicated medical data. Specifically, AI is the ability for computer algorithms to approximate conclusions without direct human input. Deep Learning and AI in Healthcare
- 5. 5 AI vs. DL vs. CL
- 6. 6 Software SQA – IEC/EN 62304 Activities outside the scope of 62304 standardCustomer needs Customer needs satisfied System development activities (including Risk Management) 7. Software Risk Management 5.1 Software Development Planning 5.2 Software Requirements Analysis 5.3 Software Architectural Design 5.4 Software Detailed Design 5.5 Software unit Implementation & Verification 5.6 Software Integration and Integration Testing 5.7 Software System Testing 5.8 Software Release 9. Software Problem Resolution 8. Software Configuration Management
- 7. 7 • Software as a Medical Device (SaMD) - Software intended to be used for one or more medical purposes that perform these purposes without being part of a hardware medical device. • Clinical Evaluation - The assessment and analysis of clinical data pertaining to a medical device to verify the clinical safety, performance and effectiveness of the device when used as intended by the manufacturer. • Medical Device - Software, when intended to treat, diagnose, cure, mitigate, or prevent disease or other conditions, are medical devices . • Non-Medical Device: - Administrative support of a health care facility - Maintaining or encouraging a healthy lifestyle - Serve as electronic patient records - Transferring, storing, converting formats, or displaying data - Provide certain, limited clinical decision support SaMD and Clinical Evaluation
- 8. What is the Difference
- 9. 9 • Adaptive artificial intelligence and machine learning technologies differ from other software as a medical device (SaMD) in that they have the potential to adapt and optimize device performance in real-time to continuously improve health care for patients. SaMD vs. AI
- 12. 12 • Performance – Clinical and analytical performance • Inputs- Used by the algorithm and their clinical association to the SaMD output. • Intended use – The intended use of the SaMD, described through the significance of information provided by the SaMD for the state of the healthcare situation or condition. • SaMD manufacturer is expected to implement on-going lifecycle processes to thoroughly evaluate the product’s performance in its intended market. • The FDA is considering a total product lifecycle-based regulatory framework for these technologies that would allow for modifications to be made from real-world learning and adaptation, while still ensuring that the safety and effectiveness of the software as a medical device is maintained. • FDA would expect a commitment from manufacturers on transparency and real-world performance monitoring for artificial intelligence and machine learning-based software as a medical device, as well as periodic updates to the FDA on what changes were implemented as part of the approved pre- specifications and the algorithm change protocol. Types of AI/ML-based SaMD Modifications
- 13. 13 Pathway for Continuous Learning Performance Data
- 14. 14 SaMD Clinical Evaluation Examples of existing evidence: • Literature searches • Original clinical research • Professional society guidelines Examples of generating new evidence: • Secondary data analysis • Perform clinical trials
- 15. 15 SaMD Clinical Evaluation Verification – Confirmation through provision of objective evidence that specified requirements have been fulfilled. Validation – Confirmation through provision of objective evidence that the requirements for a specific intended use or application have been fulfilled.
- 16. 16 SaMD Clinical Evaluation Examples of measures of clinical validation Sensitivity Specificity Positive predictive value (PPV) Negative predictive value (NPV) Number needed to treat (NNT) Number needed to harm (NNH) Likelihood ratio negative (LR-) Likelihood ratio positive (LR+) Odds ratio (OR) Clinical usability / User Interface Confidence interval
- 17. 17 • Benefit and performance of the medical device (sensitivity and specificity) • Validated against the intended purpose and User requirements and verified against the specifications. • Software ensures repeatability, reliability and performance. • Methods were use for these verifications and validation. • Clinical evaluation is based on a comparator device, this device must be sufficiently technical equivalent, which explicitly includes the evaluation of the software algorithms. Regulatory Approach
- 18. 18 • Quality management systems and Good Machine Learning Practices (GMLP) - Appropriate separation of training, tuning and test datasets, and of an appropriate level of transparency with regard to the output and algorithms. • Planning and initial evaluation regarding safety and performance - SaMD Pre- Specifications” (SPS) explains what type of modification it anticipates. Furthermore, the modifications should be made according to an "Algorithm Change Protocol” (ACP). • Approach for modifications after the initial release - If a manufacturer did not submit an SPS or ACP for the initial approval, it must submit future modifications to the authority again • Transparency and real-world performance monitoring - Expects regular reports on the monitoring of the performance of the devices on the market in accordance with the SPS and ACP Pillars of a Best-Practice Approach
- 19. 19 Algorithm Change Protocol (ACP)
- 20. 20 Decision trees decide whether modifications to software, based on machine learning, make a re-approval necessary. Decision Trees
- 21. 21 • IMPROVING MEDICAL TREATMENT OUTCOMES WITH AI - AI-Powered Imaging - Health Monitoring - Organizing & Interpreting Large Data Sets - AI-Assisted Treatment • INCREASING ACCESS TO MEDICAL CARE WITH IMPROVED TECHNOLOGY - Medical Chatbots - Wider Availability of Diagnostic Tools - Reducing Time & Administrative Overhead - Preventing Unnecessary Testing Healthcare’s Machine Learning Revolution
- 22. 22 The Challenge R&D Regulation Bridging the gap between R&D and Regulation in the medical device and pharma industry
- 23. 23 Our Story With years of combined experience in the field of medical device & software development, we recognized a critical gap between R&D and Regulation teams, challenging the successful advancement and success of long term projects. With an aim to bridge the gap, we founded MedDev Soft. A dynamic, innovative company, bringing together hands-on knowledge and experience. We provide Medical Device and Pharma software development and regulation services.
- 24. 24 Our Solutions One Stop Shop Full Turnkey Software Development Always On Time and On Budget From Fast Prototype to Final Product Software Quality Assurance Consulting and Hands-on Solutions
- 25. 25 We strive to provide our clients with the highest standards of service, delivering time and cost efficient solutions to drive them to the forefront of their industry. By placing the critical tasks in the hands of experts from the MedDev Soft team, we become an integral part of your successful solution. Why Us? Why With Us?
- 27. 27 • “Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)- Based Software as a Medical Device (SaMD)” in April 2019 • Software as a Medical Device (SaMD): Clinical Evaluation: https://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/uc m524904.pdf • Software as a Medical Device (SaMD): http://www.imdrf.org/docs/imdrf/final/technical/imdrf-tech- 131209-samd-keydefinitions-140901.pdf • Deciding When to Submit a 510(k) for a Software Change to an Existing Device: https://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/uc m514737.pdf • Software as a Medical Device (SaMD): Possible Framework for Risk Categorization and Corresponding Considerations: http://www.imdrf.org/docs/imdrf/final/technical/imdrf-tech-140918-samd-framework- risk-categorization-141013.pdf References
Editor's Notes
- Artificial intelligence (AI) is by definition the branch of computer science concerned with making computers behave like humans.
- Supervised – input-output pairs – labeled training Semi-supervised - a small amount of labeled data with a large amount of unlabeled data during training Unsupervised learning is a type of self-organized Hebbian learning that helps find previously unknown patterns in data set without pre-existing labels.
- Artificial intelligence (AI) is by definition the branch of computer science concerned with making computers behave like humans. Artificial intelligence is based on numerous techniques, of which machine learning is only one part. Neural networks, deep learning, are part of machine learning.
- Current medical device regulations include software within their scope, depending on intended use. This can include software that is standalone or incorporated into an existing device. These new challenges include the: • level of autonomy introduced by AI technologies; • ability of continuous learning systems to change their output over time in response to new data; and • ability to explain and understand how an output has been reached. Currently there are no standards that cover the definition, development, deployment and maintenance of AI in healthcare.
- SaMD based on intended use, similar to traditional risk-based approaches used by the FDA. The IMDRF risk framework identifies the following two major factors as providing a description of the intended use of the SaMD: 1) Significance of information provided by the SaMD to the healthcare decision, which identifies the intended use of the information provided by the SaMD – i.e., to treat or diagnose; to drive clinical management; or to inform clinical management; and 2) State of healthcare situation or condition, which identifies the intended user, disease or condition, and the population for the SaMD – i.e., critical; serious; or non-serious healthcare situations or conditions. Taken together, these factors describing the intended use can be used to place the AI/ML-based SaMD into one of four categories, from lowest (I) to highest risk (IV) to reflect the risk associated with the clinical situation and device use.
- Improvements to clinical and analytical performance: These improvements may include training with additional data sets. Modification of the “input data” used by the algorithm. This can be additional laboratory data or data from another CT manufacturer. Change of intended use: The FDA gives the example of an algorithm that initially only calculated a “confidence score” intended to aid a diagnosis, but which now provides a definitive diagnosis. A change to the intended patient population would also be considered a changed to the intended use. There are many possible modifications to an AI/ML-based SaMD. Some modifications may not require a review based on guidance provided in “Deciding When to Submit a 510(k) for a Software Change to an Existing Device.”13 This paper anticipates that many modifications to AI/ML-based SaMD involve algorithm architecture modifications and re-training with new data sets, which under the software modifications guidance would be subject to premarket review.
- Typically, these have only included algorithms that are “locked ” prior to marketing, where algorithm changes likely require FDA premarket review for changes beyond the original market authorization. However, not all AI/ML-based SaMD are locked; some algorithms can adapt over time. The power of these AI/ML-based SaMD lies within the ability to continuously learn, where the adaptation or change to the algorithm is realized after the SaMD is distributed for use and has “learned” from real-world experience. Following distribution, these types of continuously learning and adaptive AI/ML algorithms may provide a different output in comparison to the output initially cleared for a given set of inputs. The traditional paradigm of medical device regulation was not designed for adaptive AI/ML technologies, which have the potential to adapt and optimize device performance in real-time to continuously improve healthcare for patients. The highly iterative, autonomous, and adaptive nature of these tools requires a new, total product lifecycle (TPLC) regulatory approach that facilitates a rapid cycle of product improvement and allows these devices to continually improve while providing effective safeguards.
- Valid Clinical Association: Is there a valid clinical association between your SaMD output, based on the inputs and algorithms selected, and your SaMD’s targeted clinical condition? Step 1: Verify that the association between the SaMD output and the targeted clinical condition is supported by evidence. Note: All SaMD should demonstrate a valid clinical association. Analytical Validation: Does your SaMD meet technical requirements? Step 1: Generate evidence that shows that the output of your SaMD is technically what you expected. Note: All SaMD should demonstrate analytical validation Clinical Validation: Does your SaMD generate clinically relevant outputs? Step 1: Generate evidence that shows your: SaMD has been tested in your target population and for your intended use; and that Users can achieve clinically meaningful outcomes through predictable and reliable use. Note: All SaMD should demonstrate clinical validation.
- Valid Clinical Association: Is there a valid clinical association between your SaMD output, based on the inputs and algorithms selected, and your SaMD’s targeted clinical condition? Step 1: Verify that the association between the SaMD output and the targeted clinical condition is supported by evidence. Note: All SaMD should demonstrate a valid clinical association. Analytical Validation: Does your SaMD meet technical requirements? Step 1: Generate evidence that shows that the output of your SaMD is technically what you expected. Note: All SaMD should demonstrate analytical validation Clinical Validation: Does your SaMD generate clinically relevant outputs? Step 1: Generate evidence that shows your: SaMD has been tested in your target population and for your intended use; and that Users can achieve clinically meaningful outcomes through predictable and reliable use. Note: All SaMD should demonstrate clinical validation.
- Valid Clinical Association: Is there a valid clinical association between your SaMD output, based on the inputs and algorithms selected, and your SaMD’s targeted clinical condition? Step 1: Verify that the association between the SaMD output and the targeted clinical condition is supported by evidence. Note: All SaMD should demonstrate a valid clinical association. Analytical Validation: Does your SaMD meet technical requirements? Step 1: Generate evidence that shows that the output of your SaMD is technically what you expected. Note: All SaMD should demonstrate analytical validation Clinical Validation: Does your SaMD generate clinically relevant outputs? Step 1: Generate evidence that shows your: SaMD has been tested in your target population and for your intended use; and that Users can achieve clinically meaningful outcomes through predictable and reliable use. Note: All SaMD should demonstrate clinical validation.
- There are currently no laws or harmonized standards that specifically regulate the use of artificial intelligence in medical devices. However, these devices must meet existing regulatory requirements, such as:
- לוגו