Using Risk Management for Validation

9,354 views

Published on

This presentation identifies a way to use Risk Management to determine the extent and scope of a validation project, including what validation documents are needed and what should be tested. One validation size does not fit all validation projects! Using the Quality/Regulatory Risk and Functionality/Distribution Risk identifies an Overall System Risk. The Overall System Risk and the Type of System Change determine the needed Validation documents. A methodology to identify the extent of validation test scripts is discussed too.

Published in: Technology

Using Risk Management for Validation

  1. 1. How Much Validation is Enough? The Breadth and the Depth Robert Sturm rmsturm@hotmail.com
  2. 2. Estimating the Dilbert Way (1 of 2)
  3. 3. Estimating the Dilbert Way (2 of 2)
  4. 4. Agenda• Brief Overview of Validation and Verification• Do we have to Validate the System?• Identify Overall System Risk• Identify Type of System Change• Identify Needed Validation Documents• Identify Extent of Test Scripts• Workshop
  5. 5. DISCLAIMER• Everything in this presentation represents my opinion and not that of my employer.• It’s all about RISK!
  6. 6. Validation and Verification• Verify each part: Test and document results• Validation = ∑ verified parts It’s the entire process! VERIFY
  7. 7. Potential Validation Documents (GAMP 5)Validation Plan Validation Summary Data Migration (DM) User Risk Analysis Requirements PQ Protocol/ Scripts Functional Specs OQ Protocol/ Scripts Design specs IQ Protocol/ Scripts Trace Matrix
  8. 8. Key Validation Deliverables• Number of validation documents – How many? vs.• Extent of the test scripts – What and how much to test? vs.
  9. 9. How Much? FDA: : Use Risk
  10. 10. How Much? QA vs. IT VALIDATION IT &QA CUSTOMER
  11. 11. Validation Estimating Workflow 1 2 3 4 Do we Determine Determine Determine have to Quality & Functionality Overallvalidate? Regulatory & Distribution System Risk Risk Risk8 7 6 5 Identify Identify Determine Determine extent of Validation Document Type oftest scripts Documents Level System Change START
  12. 12. 1 Do We Have to Validate?• Is this a GxP system - GLP, GCP or GMP? – Laboratory, Clinical or Manufacturing• Does the system affect Patient Safety, Product Quality or Data Integrity?
  13. 13. 2 What is the Quality & Regulatory Risk? • Evaluate the criticality of the data processed by the system as Low, Medium or HIGH risk • For each risk category identify examples of GMP, GCP and GLP records in the system – HIGH: GMP (Master records, batch records, etc.) – Medium: GCP (Clinical trial patient data) – Low: GLP (Training records, facilities records, etc.) • Use the highest identified risk level
  14. 14. 3 What is the Functionality Risk?• Assign values based on functions the system performs – Electronic data capture 3 – Data entry/modification/calculation 3 – Submission creation 2 – Data analysis and reporting 2 – Data/document storage 1 – Data browsing 1
  15. 15. 3 What is the Distribution Risk?• Custom system 5• Multi-industry, limited use 4• Regulated industry, limited use 3• Regulated industry, broad use 2• Multi-industry, broad use 1
  16. 16. 3 What is the Complexity?• Add Functionality and Distribution Risk• Complexity Level HIGH = 7 to 8 Medium = 4 to 6 Low = 2 to 3
  17. 17. 4 Determine Overall System RiskQuality/Regulatory Complexity System ValueHIGH + HIGH = HIGH 10HIGH + Medium = HIGH 9
  18. 18. 4 Determine Overall System Risk Quality/Regulatory Complexity System Value HIGH + HIGH = HIGH 10 HIGH + Medium = HIGH 9 HIGH + Low = HIGH 8 Medium + HIGH = Medium 7
  19. 19. 4 Determine Overall System RiskQuality/Regulatory Complexity System ValueHIGH + HIGH = HIGH 10HIGH + Medium = HIGH 9HIGH + Low = HIGH 8Medium + HIGH = Medium 7Medium + Medium = Medium 6Medium + Low = Medium 5Low + HIGH = Medium 4
  20. 20. 4 Determine Overall System RiskQuality/Regulatory Complexity System ValueHIGH + HIGH = HIGH 10HIGH + Medium = HIGH 9HIGH + Low = HIGH 8Medium + HIGH = Medium 7Medium + Medium = Medium 6Medium + Low = Medium 5Low + HIGH = Medium 4Low + Medium = Low 3Low + Low = Low 2
  21. 21. HIGH Drug Clin Safety Data Elec Data MedDRA WHO Mgt Cap Drug SAS ERP Drug Clin Stability Doc Submission Supplies Publishing Software Upgrade Mgt MEDIUM Hardware/R Drug Network Clin Trial Coding UpgradeI MgtS ExcelK Scanner Low Low MEDIUM HIGH COMPLEXITY
  22. 22. HIGH Drug Clin Safety Data Elec Data MedDRA WHO Mgt Cap Drug SAS ERP Drug Clin Stability Doc Submission Supplies Publishing Software Upgrade Mgt MEDIUM Hardware/R Drug Network Clin Trial Coding UpgradeI MgtS ExcelK Scanner Low Low MEDIUM HIGH COMPLEXITY
  23. 23. H-l H-m H-H HIGH Drug Clin Safety Data Elec Data MedDRA m-H WHO Mgt Cap Drug SAS ERP Drug m-m Clin Stability Doc Submission Supplies Publishing Software Upgrade Mgt MEDIUM m-l Hardware/R Drug Network Clin Trial Mgt Coding UpgradeI l-HS l-m ExcelK l-l Scanner Low Low MEDIUM HIGH COMPLEXITY
  24. 24. What do we know at this point?• We know the overall system risk level –H - H, H - M, H - L, M - H, M - M, …. L – M, L – L• We have a value for the overall system risk level Are we there yet?
  25. 25. 5 What Kind of Change is Happening? • Release (Major) – x.0: Adding functionality; – Change value of 1.0 • Upgrade (Minor) – x.y: Changing functionality; – Change value of 0.8 • Patch – x.y.z: Not changing functionality; – Change value of 0.5 • Hot Fix – x.y.z.w: Smell for smoke; – Change value of 0.2
  26. 26. 6 Determine Overall Document Level System Risk Value Release Upgrade Patch Hot Fix Change value: 1.0 0.8 0.5 0.2 H - H H - M H - L M - H M - M M - L L - H L - M L - L
  27. 27. 6 Determine Overall Document Level System Risk Value Release Upgrade Patch Hot Fix Change value: 1.0 0.8 0.5 0.2 H - H 10 10.0 8.0 5.0 2.0 H - M 9 9.0 7.2 4.5 1.8 H - L M - H M - M M - L L - H L - M L - L
  28. 28. 6 Determine Overall Document Level System Risk Value Release Upgrade Patch Hot Fix Change value: 1.0 0.8 0.5 0.2 H - H 10 10.0 8.0 5.0 2.0 H - M 9 9.0 7.2 4.5 1.8 H - L 8 8.0 6.4 4.0 1.6 M - H 7 7.0 5.6 3.5 1.4 M - M M - L L - H L - M L - L
  29. 29. 6 Determine Overall Document Level System Risk Value Release Upgrade Patch Hot Fix Change value: 1.0 0.8 0.5 0.2 H - H 10 10.0 8.0 5.0 2.0 H - M 9 9.0 7.2 4.5 1.8 H - L 8 8.0 6.4 4.0 1.6 M - H 7 7.0 5.6 3.5 1.4 M - M 6 6.0 4.8 3.0 1.2 M - L 5 5.0 4.0 2.5 1.0 L - H 4 4.0 3.2 2.0 0.8 L - M 3 3.0 2.4 1.5 0.6 L - L 2 2.0 1.6 1.0 0.4
  30. 30. 7 Identify Validation Documents Document Value 10 - 9 8-7 6-4 3-0 Validation Plan I I I C O User Requirements I C C C C O Risk Analysis I C C C O Functional Specs I C C O O O Design specs I C O O O Trace matrix I C C C O I = Individual, C = Combined, O = Optional IT’S A GREY AREA!
  31. 31. 7 Identify Validation DocumentsDocument Value 10 - 9 8-7 6-4 3-0IQ Protocol/Scripts I C I C C C OOQ Protocol/Scripts I C I C C C OPQ Protocol/Scripts I C I C C C ODM Protocol/Scripts I I C C C OValidation Summary I I I C OProduction Release I O I O I O OMemoI = Individual, C = Combined, O = Optional
  32. 32. Now where are we?• We identified what validation documents are needed for the project• We have QA buy-in Validation documents
  33. 33. 8 Identify Extent of Test Scripts• Determine level of testing• Apply a Functional Risk Assessment to all user requirements• Prioritize requirements as HIGH (H), Medium (M) or Low (L)• Assess requirements as regulatory/ business risk Critical (C) or Not critical (N)• High and Critical are the highest risk
  34. 34. 8 Prioritize Requirements (Reqmnt)• HIGH - H (Mandatory): Reqmnt must be present for the system to operate• Medium - M (Desirable to have): Reqmnt need not be present for the system to operate but it is a ‘nice to have” feature• Low - L (Low impact/may be useful): Reqmnt need not be present for the system to operate, may be useful and has low impact to the prime user department• Null - N (Not used): Reqmnt will not be used and will not be validated
  35. 35. 8 Prioritize Risk of the Requirement• Critical (C): If this Reqmnt is not done there is a risk of non-compliance. Reqmnt is critical for business reasons (data are correct, system is Web available, data entry and output are correct, etc.)• Not critical (N): Reqmnt has no regulatory or business risk associated with it
  36. 36. 8 To Test or Not to Test? Priority Risk Test (Y/N) (H/M/L) (C/N) H C Y M C Y L C Y -- -- -- H N Y M N N L N N N N N
  37. 37. Assessing Changes to a Validated System for Company XYZDescription Priority Risk Test (H/M/L) (C/N) (Y/N)Changes to the safety database schema H C Yand tablesAbility to capture partial dates M N NAble to print reports for FDA submissions H C Yon a representation of the revisedMedWatch formCustomer fixes for expedited reports M N NLab test name encoding from Case Form L N NElectronic Submission: Updated feature -- N Nnot used
  38. 38. Interactive Exercise• Our vendor sends us a minor upgrade to our Drug Safety system• It affects our existing requirements• What validation documents are needed?• What should be tested?
  39. 39. The Next Step Is Yours OR
  40. 40. Bonus Information: References• “General Principles of Software Validation, Final Guidance for Industry and FDA Staff”, FDA, Jan 11, 2002. www.fda.gov• Annex 11 Computerised Systems• Computer Validation: The 100 Worst Mistakes You Can Make, Tamara Follett, 2003• The Computer System Risk Management and Validation Life Cycle, R. Timothy Stein, 2006 (many references)• GAMP 5 A Risk-Based Approach to Compliant GxP Computerized Systems, ISPE, 2008• “GIP Guidance Module 1: Validation and Verification”, HIMSS, 2011, himss.learn.com/learncenter.asp?id=178409&page=184• Google “Software Compliance Science John Murray FDA” for a PDF file: www.softwarecpr/.../download.asp
  41. 41. Bonus Information: References• “RAMP: An Approach to Risk-Based Computer System Validation and Part 11 Compliance”, Drug Information Journal, Vol 41, pp 69- 79, 2007 (Use to estimate Quality, Regulatory, Functionality and Distribution Risk)• “Validation of Software for Regulated Processes”, Assoc. for Advancement of Medical Instrumentation, 13-Dec 2007• “Effective and Practical Risk Management Options for Computerised System Validation,” R. D. McDowall, Quality Assurance Journal, Vol 9, Issue 3, 2005• IT Pharma Validation Europe: http://www.ccs-inspired.com• Ask About Validation: http://www.askaboutvalidation.com• Risk Doctor Network: http://www.risk-doctor.com• Compliance Webinars: http://www.complianceonline.com/
  42. 42. Four Assessment Questions• Does the system automate a process or manage date regulated by these regulations? – GMP (21CFR 210 and 211) – GCP (21 CFR 312) – GLP (21 CFR 58) – PDMA (21 CFR 203)• If yes to any of the above, it is GxP
  43. 43. Bonus Information: Validation Life Cycle Folder Organization 1 Admin IT Computer System Validation and Document Flow: Key Validation Documents 2 Pre xQ 3 IQ ** EVALUATE ** ** PLAN ** 4 During Xq 2 7 - Configuration & Training - Major release 1 1 1 - SOPs and WIs A B K Validation/ - Added functionality Qualification Project Supplier Audit - Migration Plan 5 OQ 6 PQ 7 Post xQ 8 Post Impl ** REQUIREMENTS ** ** EXECUTION ** ** EXECUTION ** ** DEPLOYMENT ** IDENTIFY REQUIREMENTS WRITE PROTOCOL & EXECUTE TESTS SUMMARIZE TEST RESULTS TEST SCRIPTS WRITE FINAL REPORTS 3 3 3 7 Installation 2 IQ Installation Qualification Installation Validation/ 2 Qualification User 2 Protocol & Test Scripts Qualification Test Summary Report Qualification D (Validation Env.) Execution Requirement C (Val Env.) Summary Report Specifications (Val Env.) 2 7 4 4 Functional 2 4 4 4 E F G F G 5 Specifications Training Records Design Operational 5 Specifications 5 Qualification Summary Report Operational Operational 7 Qualification Protocol Qualification Test & Test Scripts Execution Support Model 6 Performance 2 6 4 Qualification 4 Performance Summary Report 7 Requirements Trace H I Matrix Qualification Test Production Release 6 Execution memo with 7 Workarounds, if any Performance Qualification Protocol 7 J & Test Scripts Pre J 3 ** MAINTENANCE ** 3 IQ 3 IQ Summary Post Implementation IQ Protocol & IQ Execution Report (Prod. Env.) (Prod Env.) 8 Test ScriptsOptional: C-Risk Analysis, D-Training Plan, D-Training Plan, Final WIs/ SOPs (Production Env.)E-Detailed Design Specs., F-Configuration Spec., F-Configuration Spec., 4 7 7 4 F F J Post JG-Training & Manuals, H-Migration Qualification Protocol,G-Training & Manuals, H-Data Migration Protocol, 8 CHANGEI-Draft WIs/ SOPs, J-Data Migration Report Report J-Migration Qualification CONTROLAdministrative: A-GxP Assessment, B-IT Statement of Work, K-MS Project Plan ** RETIREMENT **

×