Software validation do's and dont's may 2013


Published on

Software validation is often times a very misunderstood concept. For FDA regulated industries, there are clear expectations including “the least burdensome approach.” Validation alone does not guarantee software quality—many other aspects of software engineering are required.

Join software expert, John Cachat, as he discusses how to solve several software validation issues, including:
 Requirements
 Defect Prevention
 Time and Effort
 Software Life Cycle
 Plans
 Procedures
 Software Validation After a Change
 Validation Coverage
 Independence of Review
 Flexibility and Responsibility

Published in: Business, Technology
  • Be the first to comment

Software validation do's and dont's may 2013

  1. 1. © Copyright John CachatSoftware Validation:The Do’s and Don’tsJohn M.
  2. 2. © Copyright John CachatHousekeepingPhones are mutedUse the questionblock for questions Copy of presentationavailable upon request2
  3. 3. © Copyright John CachatAbout John M. Cachat• Driving Business Performance– Helping companies align their business and technology– Focus on people, process, and then the technology– Subject matter expert on business process management– On-going research into next generation of technology for enterprisesystems• 28 years experience in enterprise systems– USAF Research Project (1985)– Founder of enterprise quality software company (1988)– Chair of ASQ technical committee on computerizing quality (1992)• Trusted advisor to global organizations, government agencies,and professional groups
  4. 4. © Copyright John CachatToday’s Discussion• Discusses how to solve several softwarevalidation issues, including:– Requirements– Defect Prevention– Time and Effort– Validation Coverage– Software Life Cycle– Plans & Procedures– Software Validation After a Change– Independence of Review– Flexibility and Responsibility4
  5. 5. © Copyright John CachatWhat is Driving You?• Business Excellence– Faster Product Launch– Lower Costs (CMMI view)• Industry Requirements (i.e., FDA)– Compliance• Can you have both?5
  6. 6. © Copyright John CachatSoftware Requirements Specification(SRS)• A complete description of the behavior of a system tobe developed.• It includes a set of use cases that describe all theinteractions the users will have with the software.• Use cases are also known as functional requirements.In addition to use cases, the SRS also contains non-functional (or supplementary) requirements.• Non-functional requirements are requirements whichimpose constraints on the design or implementation(such as performance engineering requirements,standards, or design constraints6
  7. 7. © Copyright John CachatLet’s Talk Business• Do I have to validate all software?– NO• What software do I have to validate?– Based on Risk• To end users of your product• To your company7
  8. 8. © Copyright John CachatManufacturing vs. Software• Same Concepts? Yes– Prevention vs. Detection– Cannot inspect quality into the result– Total cost of quality (prevention,inspection, failure)• Same Thing? Not really, software is– Very easy to change, quickly– One change can impact a lot– Hard to inspect everything, literallyeverything8
  9. 9. © Copyright John CachatBuild, Buy, Use – Impacts Everyone!
  10. 10. © Copyright John CachatMajor Sections• Section 1. Purpose• Section 2. Scope• Section 3. Context for Software Validation• Section 4. Principles of Software Validation• Section 5. Activities and Tasks• Section 6. Validation Of Automated ProcessEquipment And Quality System Software10
  11. 11. © Copyright John CachatSection 1. Purpose• Provide general validation principles that theFDA considers to be applicable to thevalidation of medical device software or thevalidation of software used to design,develop, or manufacture medical devices.11Or pharmaceuticals, or cars, or airplanes,or anything that if it does not work,people may die, or get sick, or ……
  12. 12. © Copyright John CachatSection 2. Scope• The scope of this guidance is somewhatbroader than the scope of validation in thestrictest definition of that term.• Planning, verification, testing, traceability,configuration management, and many otheraspects of good software engineeringdiscussed in this guidance are importantactivities that together help to support a finalconclusion that software is validated.12
  13. 13. © Copyright John CachatSection 2. Scope• Based on the intended use and the safety riskassociated with the software to bedeveloped, the software developer shoulddetermine the specific approach, thecombination of techniques to be used, andthe level of effort to be applied.• Recommends an integration of software lifecycle management and risk managementactivities.13
  14. 14. © Copyright John CachatSection 2. Scope• Validate these:– Software used as a component of a medical device;– Software that is itself a medical device– Software used in the production of a device and– Software used in implementation of the devicemanufacturers quality system• What about these?– Accounting– Plant floor scheduling– Microsoft desktop software– Plant floor automation14
  15. 15. © Copyright John CachatSection 2. Scope• THE LEAST BURDENSOME APPROACH• Clarification– Software validation process should not beconfused with any other validation requirements,such as process or product validation– Does not cover any specific safety or efficacyissues with respect to product beingmanufactured15
  16. 16. © Copyright John CachatSection 3.Context for Software Validation• 3.1. Definitions and Terminology– 3.1.1 Requirements and Specifications– 3.1.2 Verification and Validation– 3.1.3 IQ/OQ/PQ• 3.2. Software Development as Part of System Design• 3.3. Software is Different from Hardware• 3.4. Benefits of Software Validation• 3.5 Design Review16
  17. 17. © Copyright John Cachat3.1. Definitions and Terminology17
  18. 18. © Copyright John Cachat3.1.1 Requirements and Specifications• A requirement can be any need or expectation for asystem or for its software.• Requirements reflect the stated or implied needs ofthe customer, and may be market-based, contractual,or statutory, as well as an organizations internalrequirements.• There can be many different kinds of requirements(e.g., design, functional, implementation, interface,performance, or physical requirements).• A specification is defined as “a document that statesrequirements.”18
  19. 19. © Copyright John Cachat3.1.2 Verification and Validation• Treats “verification” and “validation” as separateand distinct terms• Software verification provides objective evidencethat the design outputs of a particular phase ofthe software development life cycle meet all ofthe specified requirements for that phase.• Software validation to be “confirmation byexamination and provision of objective evidencethat software specifications conform to userneeds and intended uses, and that the particularrequirements implemented through softwarecan be consistently fulfilled.”19
  20. 20. © Copyright John Cachat3.1.2 Verification and Validation• Software verification - a developercannot test forever & testing doesnot guarantee no defects• Software validation - it is hard toknow how much evidence is enough.• Software validation is a matter ofdeveloping a “level of confidence”• How much “confidence?” Howmuch risk?20
  21. 21. © Copyright John Cachat3.1.3 IQ/OQ/PQ• Installation qualification (IQ) - the system hasbeen built and installed correctly and that allrequired supporting services are available andconnected correctly.• Operational qualification (OQ) - demonstratethat the newly acquired software functions asexpected; that all parts and components operatecorrectly• Performance qualification (PQ) - demonstratecompliance with all requirements given in theUser Requirements Specification document21
  22. 22. © Copyright John Cachat3.2. Software Development as Part of System Design• Software validation must be considered withinthe context of the overall design validation forthe system• End user rarely cares about the software• The users needs and intended uses fromwhich the product is developed– Correct blood pressure reading– Anti-lock brakes work– Airplane controls work22
  23. 23. © Copyright John Cachat3.3. Software is Different from Hardware• Seemingly insignificant changes in software code can createunexpected and very significant problems elsewhere in thesoftware program The vast majority of software problemsare traceable to errors made during the design anddevelopment process.• One of the most significant features of software isbranching, i.e., the ability to execute alternative series ofcommands, based on differing inputs.• Software also has the speed and ease with which it can bechanged concern regarding change management• Software failures often occur without advanced warning(no noise, vibration, etc)23
  24. 24. © Copyright John Cachat3.3. Software is Different from Hardware• Because of its complexity,the software developmentprocess should becontrolled more tightlythan hardware.24
  25. 25. © Copyright John Cachat3.4. Benefits of Software Validation• This is the business part– decreased failure rates– fewer recalls and corrective actions– less risk to patients and users, and– reduced liability to manufacturers25This Car Runs on CodeIt takes dozens of microprocessors running100 million lines of code to get a premiumcar out of the driveway.As Much Software Code as an Airbus A380
  26. 26. © Copyright John Cachat3.5 Design Review• Design reviews are documented,comprehensive, and systematicexaminations of a design to evaluate theadequacy of the design requirements,to evaluate the capability of the designto meet these requirements, and toidentify problems.• FMEA - Failure mode effect analysis• SET – Success every time26
  27. 27. © Copyright John CachatFMEA27
  28. 28. © Copyright John CachatSection 4.Principles of Software Validation4.1. Requirements4.2. Defect Prevention4.3. Time and Effort4.4. Software Life Cycle4.5. Plans4.6. Procedures4.7. Software Validation after a Change4.8. Validation Coverage4.9. Independence of Review4.10. Flexibility and Responsibility28
  29. 29. © Copyright John CachatSection 4.Software Validation Summary• Software testing is a necessary activity.• In most cases software testing by itself is not sufficient toestablish confidence that the software is fit for its intendeduse• Validation coverage should be based on the softwarescomplexity and safety risk - not on firm size or resourceconstraints.• The final conclusion that the software is validated shouldbe based on evidence collected from planned effortsconducted throughout the software lifecycle• Whenever software is changed, a validation analysisshould be conducted not just for validation of the individualchange, but also to determine the extent and impact ofthat change on the entire software system.29
  30. 30. © Copyright John CachatSection 5. Activities and Tasks5.1. Software Life Cycle Activities5.2. Typical Tasks Supporting Validation5.2.1. Quality Planning5.2.2. Requirements5.2.3. Design5.2.4. Construction or Coding5.2.5. Testing by the Software Developer5.2.6. User Site Testing5.2.7. Maintenance and Software Changes30
  31. 31. © Copyright John CachatSoftware Life Cycle• Quality Planning• System Requirements Definition• Detailed Software Requirements Specification• Software Design Specification• Construction or Coding• Testing• Installation• Operation and Support• Maintenance - Change Control, Revision Control• Retirement31
  32. 32. © Copyright John CachatQuality Planning Tasks• Risk Management Plan• Configuration Management Plan• Software Quality Assurance Plan (Software Verification & Validation Plan)• Verification and Validation Tasks, and Acceptance Criteria• Schedule and Resource Allocation• Reporting Requirements– Formal Design Review Requirements– Other Technical Review Requirements• Problem Reporting and Resolution Procedures• Other Support Activities32
  33. 33. © Copyright John Cachat5.2.2 Requirements• All software system inputs;• All software system outputs;• All functions that the software system will perform;• All performance requirements that the software will meet, (e.g.,data throughput, reliability, and timing);• Definition of all external and user interfaces, as well as anyinternal software-to-system interfaces;• How users will interact with the system;• What constitutes an error and how errors should be handled;• Required response times;• The intended operating environment for the software, if this is adesign constraint• All ranges, limits, defaults, and specific values that the softwarewill accept; and• All safety related requirements, specifications, features, orfunctions that will be implemented in software.33
  34. 34. © Copyright John CachatTesting Coverage• Statement Coverage• Decision (Branch) Coverage• Condition Coverage• Multi-Condition Coverage• Loop Coverage• Path Coverage• Data Flow Coverage34
  35. 35. © Copyright John CachatSection 6.Validation Of Automated Process EquipmentAndQuality System Software• 6.1. How Much Validation Evidence Is Needed?• 6.2. Defined User Requirements• 6.3. Validation of Off-the-Shelf Software andAutomated Equipment35
  36. 36. © Copyright John Cachat6.1. How Much Validation Evidence IsNeeded?• The level of validation effort should becommensurate with the risk posed by theautomated operation• The extent of validation evidence needed for suchsoftware depends on the device manufacturersdocumented intended use of that software• COTS - consider auditing the vendors design anddevelopment methodologies used in theconstruction of the software and assess thedevelopment and validation documentationgenerated for the COTS software36
  37. 37. © Copyright John CachatIEEE Technical Resources• SRS - Software Requirements Specification IEEE 830• SQAP - Software Quality Assurance Plan IEEE 730• SCMP- Software Configuration Management Plan IEEE 828• STD - Software Test Documentation IEEE 829• SVVP - Software Validation & Verification Plan IEEE 1012• SDD - Software Design Description IEEE 1016• SPMP - Software Project Management Plan IEEE 105837
  38. 38. © Copyright John CachatSoftware Best Practices• Develop software iteratively• Manage requirements• Use component-based architectures• Visually model software• Verify and validate• Software change control process• Document, document, document38
  39. 39. © Copyright John CachatSummary• You DO NOT have tovalidate all software• You validate based on risk• Software is not the same ashardware• Plan for the entire softwarelife cycle39
  40. 40. © Copyright John CachatAbout UsJohn Cachatjmc@peproso.comContactProven expertise in businessinformation systemsRapid SolutionDevelopment™ processServicesAssess Current StatusDevelop Short and LongTerm PlansDevelop Specific Solutionsto Your ProblemsAssist in ROI Analysis40
  41. 41. © Copyright John CachatSoftware Validation: The Do’s and Don’ts&Contact:John Cachatjmc@peproso.comCopy of Presentation&Request a DemoVisit: Webinars41