The document summarizes best practices for software validation based on a presentation by John Cachat. It discusses that software validation should be based on risk, not all software needs to be validated, and that validation requires evidence collected throughout the software development lifecycle. It also notes that software is different than hardware due to its complexity and ease of changes impacting the system. Finally, it provides an overview of typical tasks involved in software validation like requirements, design, testing, and maintaining validation when software changes.

1. © Copyright John Cachat
Software Validation:
The Do’s and Don’ts
John M. Cachat
jmc@peproso.com

2. © Copyright John Cachat
Housekeeping
Phones are muted
Use the question
block for questions Copy of presentation
available upon request
2

3. © Copyright John Cachat
About John M. Cachat
• Driving Business Performance
– Helping companies align their business and technology
– Focus on people, process, and then the technology
– Subject matter expert on business process management
– On-going research into next generation of technology for enterprise
systems
• 28 years experience in enterprise systems
– USAF Research Project (1985)
– Founder of enterprise quality software company (1988)
– Chair of ASQ technical committee on computerizing quality (1992)
• Trusted advisor to global organizations, government agencies,
and professional groups
http://www.linkedin.com/in/johncachat
3

4. © Copyright John Cachat
Today’s Discussion
• Discusses how to solve several software
validation issues, including:
– Requirements
– Defect Prevention
– Time and Effort
– Validation Coverage
– Software Life Cycle
– Plans & Procedures
– Software Validation After a Change
– Independence of Review
– Flexibility and Responsibility
4

5. © Copyright John Cachat
What is Driving You?
• Business Excellence
– Faster Product Launch
– Lower Costs (CMMI view)
• Industry Requirements (i.e., FDA)
– Compliance
• Can you have both?
5

6. © Copyright John Cachat
Software Requirements Specification
(SRS)
• A complete description of the behavior of a system to
be developed.
• It includes a set of use cases that describe all the
interactions the users will have with the software.
• Use cases are also known as functional requirements.
In addition to use cases, the SRS also contains non-
functional (or supplementary) requirements.
• Non-functional requirements are requirements which
impose constraints on the design or implementation
(such as performance engineering requirements,
standards, or design constraints
6

7. © Copyright John Cachat
Let’s Talk Business
• Do I have to validate all software?
– NO
• What software do I have to validate?
– Based on Risk
• To end users of your product
• To your company
7

8. © Copyright John Cachat
Manufacturing vs. Software
• Same Concepts? Yes
– Prevention vs. Detection
– Cannot inspect quality into the result
– Total cost of quality (prevention,
inspection, failure)
• Same Thing? Not really, software is
– Very easy to change, quickly
– One change can impact a lot
– Hard to inspect everything, literally
everything
8

9. © Copyright John Cachat
Build, Buy, Use – Impacts Everyone!
9
www.fda.gov/downloads/.../Guidances/ucm126955.pdf

10. © Copyright John Cachat
Major Sections
• Section 1. Purpose
• Section 2. Scope
• Section 3. Context for Software Validation
• Section 4. Principles of Software Validation
• Section 5. Activities and Tasks
• Section 6. Validation Of Automated Process
Equipment And Quality System Software
10

11. © Copyright John Cachat
Section 1. Purpose
• Provide general validation principles that the
FDA considers to be applicable to the
validation of medical device software or the
validation of software used to design,
develop, or manufacture medical devices.
11
Or pharmaceuticals, or cars, or airplanes,
or anything that if it does not work,
people may die, or get sick, or ……

12. © Copyright John Cachat
Section 2. Scope
• The scope of this guidance is somewhat
broader than the scope of validation in the
strictest definition of that term.
• Planning, verification, testing, traceability,
configuration management, and many other
aspects of good software engineering
discussed in this guidance are important
activities that together help to support a final
conclusion that software is validated.
12

13. © Copyright John Cachat
Section 2. Scope
• Based on the intended use and the safety risk
associated with the software to be
developed, the software developer should
determine the specific approach, the
combination of techniques to be used, and
the level of effort to be applied.
• Recommends an integration of software life
cycle management and risk management
activities.
13

14. © Copyright John Cachat
Section 2. Scope
• Validate these:
– Software used as a component of a medical device;
– Software that is itself a medical device
– Software used in the production of a device and
– Software used in implementation of the device
manufacturer's quality system
• What about these?
– Accounting
– Plant floor scheduling
– Microsoft desktop software
– Plant floor automation
14

15. © Copyright John Cachat
Section 2. Scope
• THE LEAST BURDENSOME APPROACH
• Clarification
– Software validation process should not be
confused with any other validation requirements,
such as process or product validation
– Does not cover any specific safety or efficacy
issues with respect to product being
manufactured
15

16. © Copyright John Cachat
Section 3.
Context for Software Validation
• 3.1. Definitions and Terminology
– 3.1.1 Requirements and Specifications
– 3.1.2 Verification and Validation
– 3.1.3 IQ/OQ/PQ
• 3.2. Software Development as Part of System Design
• 3.3. Software is Different from Hardware
• 3.4. Benefits of Software Validation
• 3.5 Design Review
16

17. © Copyright John Cachat
3.1. Definitions and Terminology
17
http://www.fda.gov/iceci/inspections/inspectionguides/ucm074875.htm

18. © Copyright John Cachat
3.1.1 Requirements and Specifications
• A requirement can be any need or expectation for a
system or for its software.
• Requirements reflect the stated or implied needs of
the customer, and may be market-based, contractual,
or statutory, as well as an organization's internal
requirements.
• There can be many different kinds of requirements
(e.g., design, functional, implementation, interface,
performance, or physical requirements).
• A specification is defined as “a document that states
requirements.”
18

19. © Copyright John Cachat
3.1.2 Verification and Validation
• Treats “verification” and “validation” as separate
and distinct terms
• Software verification provides objective evidence
that the design outputs of a particular phase of
the software development life cycle meet all of
the specified requirements for that phase.
• Software validation to be “confirmation by
examination and provision of objective evidence
that software specifications conform to user
needs and intended uses, and that the particular
requirements implemented through software
can be consistently fulfilled.”
19

20. © Copyright John Cachat
3.1.2 Verification and Validation
• Software verification - a developer
cannot test forever & testing does
not guarantee no defects
• Software validation - it is hard to
know how much evidence is enough.
• Software validation is a matter of
developing a “level of confidence”
• How much “confidence?” How
much risk?
20

21. © Copyright John Cachat
3.1.3 IQ/OQ/PQ
• Installation qualification (IQ) - the system has
been built and installed correctly and that all
required supporting services are available and
connected correctly.
• Operational qualification (OQ) - demonstrate
that the newly acquired software functions as
expected; that all parts and components operate
correctly
• Performance qualification (PQ) - demonstrate
compliance with all requirements given in the
User Requirements Specification document
21

22. © Copyright John Cachat
3.2. Software Development as Part of System Design
• Software validation must be considered within
the context of the overall design validation for
the system
• End user rarely cares about the software
• The user's needs and intended uses from
which the product is developed
– Correct blood pressure reading
– Anti-lock brakes work
– Airplane controls work
22

23. © Copyright John Cachat
3.3. Software is Different from Hardware
• Seemingly insignificant changes in software code can create
unexpected and very significant problems elsewhere in the
software program The vast majority of software problems
are traceable to errors made during the design and
development process.
• One of the most significant features of software is
branching, i.e., the ability to execute alternative series of
commands, based on differing inputs.
• Software also has the speed and ease with which it can be
changed concern regarding change management
• Software failures often occur without advanced warning
(no noise, vibration, etc)
23

24. © Copyright John Cachat
3.3. Software is Different from Hardware
• Because of its complexity,
the software development
process should be
controlled more tightly
than hardware.
24

25. © Copyright John Cachat
3.4. Benefits of Software Validation
• This is the business part
– decreased failure rates
– fewer recalls and corrective actions
– less risk to patients and users, and
– reduced liability to manufacturers
25
This Car Runs on Code
It takes dozens of microprocessors running
100 million lines of code to get a premium
car out of the driveway.
As Much Software Code as an Airbus A380

26. © Copyright John Cachat
3.5 Design Review
• Design reviews are documented,
comprehensive, and systematic
examinations of a design to evaluate the
adequacy of the design requirements,
to evaluate the capability of the design
to meet these requirements, and to
identify problems.
• FMEA - Failure mode effect analysis
• SET – Success every time
26

27. © Copyright John Cachat
FMEA
27

28. © Copyright John Cachat
Section 4.
Principles of Software Validation
4.1. Requirements
4.2. Defect Prevention
4.3. Time and Effort
4.4. Software Life Cycle
4.5. Plans
4.6. Procedures
4.7. Software Validation after a Change
4.8. Validation Coverage
4.9. Independence of Review
4.10. Flexibility and Responsibility
28

29. © Copyright John Cachat
Section 4.
Software Validation Summary
• Software testing is a necessary activity.
• In most cases software testing by itself is not sufficient to
establish confidence that the software is fit for its intended
use
• Validation coverage should be based on the software's
complexity and safety risk - not on firm size or resource
constraints.
• The final conclusion that the software is validated should
be based on evidence collected from planned efforts
conducted throughout the software lifecycle
• Whenever software is changed, a validation analysis
should be conducted not just for validation of the individual
change, but also to determine the extent and impact of
that change on the entire software system.
29

30. © Copyright John Cachat
Section 5. Activities and Tasks
5.1. Software Life Cycle Activities
5.2. Typical Tasks Supporting Validation
5.2.1. Quality Planning
5.2.2. Requirements
5.2.3. Design
5.2.4. Construction or Coding
5.2.5. Testing by the Software Developer
5.2.6. User Site Testing
5.2.7. Maintenance and Software Changes
30

31. © Copyright John Cachat
Software Life Cycle
• Quality Planning
• System Requirements Definition
• Detailed Software Requirements Specification
• Software Design Specification
• Construction or Coding
• Testing
• Installation
• Operation and Support
• Maintenance - Change Control, Revision Control
• Retirement
31

32. © Copyright John Cachat
Quality Planning Tasks
• Risk Management Plan
• Configuration Management Plan
• Software Quality Assurance Plan (Software Verification & Validation Plan)
• Verification and Validation Tasks, and Acceptance Criteria
• Schedule and Resource Allocation
• Reporting Requirements
– Formal Design Review Requirements
– Other Technical Review Requirements
• Problem Reporting and Resolution Procedures
• Other Support Activities
32

33. © Copyright John Cachat
5.2.2 Requirements
• All software system inputs;
• All software system outputs;
• All functions that the software system will perform;
• All performance requirements that the software will meet, (e.g.,
data throughput, reliability, and timing);
• Definition of all external and user interfaces, as well as any
internal software-to-system interfaces;
• How users will interact with the system;
• What constitutes an error and how errors should be handled;
• Required response times;
• The intended operating environment for the software, if this is a
design constraint
• All ranges, limits, defaults, and specific values that the software
will accept; and
• All safety related requirements, specifications, features, or
functions that will be implemented in software.
33

34. © Copyright John Cachat
Testing Coverage
• Statement Coverage
• Decision (Branch) Coverage
• Condition Coverage
• Multi-Condition Coverage
• Loop Coverage
• Path Coverage
• Data Flow Coverage
34

35. © Copyright John Cachat
Section 6.
Validation Of Automated Process Equipment
And
Quality System Software
• 6.1. How Much Validation Evidence Is Needed?
• 6.2. Defined User Requirements
• 6.3. Validation of Off-the-Shelf Software and
Automated Equipment
35

36. © Copyright John Cachat
6.1. How Much Validation Evidence Is
Needed?
• The level of validation effort should be
commensurate with the risk posed by the
automated operation
• The extent of validation evidence needed for such
software depends on the device manufacturer's
documented intended use of that software
• COTS - consider auditing the vendor's design and
development methodologies used in the
construction of the software and assess the
development and validation documentation
generated for the COTS software
36

37. © Copyright John Cachat
IEEE Technical Resources
• SRS - Software Requirements Specification IEEE 830
• SQAP - Software Quality Assurance Plan IEEE 730
• SCMP- Software Configuration Management Plan IEEE 828
• STD - Software Test Documentation IEEE 829
• SVVP - Software Validation & Verification Plan IEEE 1012
• SDD - Software Design Description IEEE 1016
• SPMP - Software Project Management Plan IEEE 1058
37

38. © Copyright John Cachat
Software Best Practices
• Develop software iteratively
• Manage requirements
• Use component-based architectures
• Visually model software
• Verify and validate
• Software change control process
• Document, document, document
38

39. © Copyright John Cachat
Summary
• You DO NOT have to
validate all software
• You validate based on risk
• Software is not the same as
hardware
• Plan for the entire software
life cycle
39

40. © Copyright John Cachat
About Us
John Cachat
jmc@peproso.com
Contact
Proven expertise in business
information systems
Rapid Solution
Development™ process
Services
Assess Current Status
Develop Short and Long
Term Plans
Develop Specific Solutions
to Your Problems
Assist in ROI Analysis
40

41. © Copyright John Cachat
Software Validation: The Do’s and Don’ts
&
Contact:
John Cachat
jmc@peproso.com
Copy of Presentation
&
Request a Demo
Visit:
http://peproso.com/webinars
Future Webinars
41