Role of Testing in Certification ProcessDocument Transcript
ROLE OF TESTING IN CERTIFICATION PROCESS, JANUARY 2011 1 Role of Testing in Certiﬁcation Process Rishu Seth (Bachelor of Engineering, Computer Science) University of Applied Sciences - Frankfurt am Main, Germany Email: firstname.lastname@example.org Abstract—This paper gives an overview of the procedure References 5of testing during veriﬁcation and validation which identiﬁesimportant defects, ﬂaws or errors in the application code thatmust be ﬁxed during software development process and resulting L IST OF F IGURESin successfull certiﬁcation process for that software.There are 1 V-Model of Software Testing  . . . . . . . . . 3different types of testings available for different types of processes 2 Software Testing Phases  . . . . . . . . . . . . 3according to the requirement of the software.Then this paperlists and describes the basic certiﬁcation problems like Relia- 3 DO178B V & V Process  . . . . . . . . . . . 4bilty Certiﬁcation Problem and Veriﬁcation Methods Drawbacks 4 Design Control Model  . . . . . . . . . . . . . 5which are contemporary in today’s world. Also, it states somebasic standards and their requirements for testing. I. I NTRODUCTION  Ertiﬁcation - A written guarantee that a system or compo- C ONTENTS C nent complies with its speciﬁed requirements and is ac- ceptable for operational use. Certiﬁcation is usually carried outI Introduction  1 by government agencies or other organizations with a national I-A Certiﬁcation Process  . . . . . . . . . 2 standing. Certiﬁcation can be applied to either organizations I-A1 Veriﬁcation . . . . . . . . . . 2 or individuals, tools or methods, or systems or products. I-A2 Veriﬁcation Techniques . . . 2 Certiﬁcation with regard to organisation aims at making sure I-A3 Validation . . . . . . . . . . 2 that the organisation is making use of certain standards and I-A4 Validation Techniques . . . . 2 criterias to achieve a level of expertise. Though it is not completely applicable to all areas because measuring theII Role of Testing in Certiﬁcation Process 2 procedures is easy as compared to measuring the competence II-A Why to do Software Testing?  . . . . 2 with which they are executed. So rather than being applied II-B Who Does The Testing? . . . . . . . . . 2 to design, certiﬁcation is applied to areas such as quality II-C Test Speciﬁcation Techniques  . . . . 2 assurance and testing to make sure that the aim is fullﬁlled. II-D The V-Model of Software Testing . . . 2 Apart from organisations, certiﬁcation can also be applied to II-D1 Unit testing . . . . . . . . . 3 individuals, enabling them to be in a certain profession like II-D2 System testing . . . . . . . . 3 doctors, lawyers, accountants and civil engineers or to tools II-D3 Integration testing . . . . . . 3 for being used in different software development processes by II-D4 User Acceptance Testing . . 3 deﬁning some basic requirements for both. For example, DO- II-D5 Production Veriﬁcation Testing 3 178B/ED-12B does not deﬁne any speciﬁc tools to be used but gives certain requirements for tools that are to be used toIII Principal Certiﬁcation Problems 3 gain certiﬁcation. III-A Reliability Certiﬁcation Problem  . . 3 Finally systems or products may also be certiﬁed. There is III-B Veriﬁcation Methods Drawbacks  . . 4 an issue with certiﬁcation process to whether methodolgy is to be certiﬁed rather than the artifact. This is a basic issueIV Requirements for Testing based on Standards 4 with products consisting of softwares specially safety critical IV-A Aerospace: RTCA/DO-178B (EURO- softwares, because software testing is so difﬁcult and also just CAE ED12B)  . . . . . . . . . . . . 4 veriﬁes that software is made following some guidelines and IV-B Biomedical Engineering: IEC 60601-1- has met some standard speciﬁed by certifying agency. This 4  . . . . . . . . . . . . . . . . . . . 4 does not necessarily mean that the system is correct. IV-B1 Hardware Veriﬁcation . . . . 4 There are different reasons why a product needs certiﬁca- IV-B2 Software veriﬁcation . . . . . 4 tion. Sometimes certiﬁcation is required for legal reasons. IV-B3 System validation . . . . . . 5 For example, before an aircraft is allowed to ﬂy, it must IV-C Generic Standard: IEC 61508  . . . . 5 obtain a license. Being certiﬁed would also be important for IV-C1 Overview of Requirements . 5 commercial reasons like having a sales advantage. One of the main reasons for certiﬁcation is to show competence in speciﬁcV Conclusion 5 areas.
ROLE OF TESTING IN CERTIFICATION PROCESS, JANUARY 2011 2A. Certiﬁcation Process  in space due to a data conversion error. Investigators dis- 1) Veriﬁcation: Veriﬁcation - The process of evaluating a covered that software on the spacecraft performed certainsystem or component to determine whether the product of a calculations in English units (yards) when it should havegiven development phase satisfys the conditions imposed at used metric units (meters). • In June 1996 the ﬁrst ﬂight of the European Spacethe start of that phase. i.e. Agency’s Ariane 5 rocket failed shortly after launching,"Are we building the product right?" resulting in an uninsured loss of $500,000,000. The disaster was traced to the lack of exception handling for 2) Veriﬁcation Techniques: There are many different veri- a ﬂoating-point error when a 64-bit integer was convertedﬁcation techniques but they all basically fall into two major to a 16-bit signed integer.categories - Software testing answers questions that development testing • Dynamic testing and code reviews can’t. They are as follows: • Static testing • Does it really work as expected?But veriﬁcation simply demonstrates whether the output of a • Does it meet the user’s requirements?phase conforms to the input of a phase as opposed to showing • Is it what the users expect?that the output is actually correct. Also it does not detect errors • Is it compatible with our other systems?resulting from wrong input speciﬁcation and these errors may • How does it perform?effect in later development stages. So veriﬁcation alone is not • How does it scale when more users are added?sufﬁcient and we need validation as well to be more sure that • Which areas need more work?the system is operational according to speciﬁcations. • Is it ready for release? 3) Validation: Validation - The process of evaluating asystem or component during or at the end of the development Factors involved in testing are -process to determine whether it satisﬁes speciﬁed require- • Business requirementsments. i.e. • Functional design requirements • Technical design requirements"Are we building the right product?" • Programmer code 4) Validation Techniques: Validation usually takes place at • Hardware conﬁgurationthe end of the development cycle, and looks at the completesystem as opposed to veriﬁcation, which focuses on smaller B. Who Does The Testing?sub-systems. Various validation techniques are: Software testing is not a one person job. It takes a team, • Formal methods but the team may be larger or smaller depending on the size • Fault injection and complexity of the application being tested. The actual • Dependability analysis programmers that wrote the code should have a reduced role • Hazard analysis in testing their own written code. • Risk analysis C. Test Speciﬁcation Techniques  II. ROLE OF T ESTING IN C ERTIFICATION P ROCESS Test speciﬁcation techniques can be split up into two groups Testing always means comparing. It requires an item to are as follows:be tested and terms of reference with which the item must • White Box Testing - In this testing, techniques arecomply. It provides an insight into the difference between the based on the program code, the program descriptions oractual status and the required status of the item. technical design. Knowledge about the internal structureAccording to International Standard Organisation testing is of the system plays an important role. Other terms useddescribed as - for this kind of techniques are Glass-box or Structural"Technical operation that consists of the determination of one testing.or more characteristics of a given product, process or service • Black Box Testing - In this testing, techniques are basedaccording to a speciﬁed procedure" . on functional speciﬁcations and quality requirements. The whole system is viewed as it would be in actual case.InA. Why to do Software Testing?  this kind of testing, knowledge about the structure of the system is not used but the judgement is made merely The most basic and instinctive response of people is "To from a functional point of view of the system.ﬁnd the bugs!", but programmers and developers know that forﬁnding bugs ’Debugging’ is there. But actually “bug” is reallya problem in the code, whereas software testing is focused on D. The V-Model of Software Testingﬁnding defects in the ﬁnal product. Examples of important It is not advisable to leave Software Testing for the enddefects that better testing would have found: of the project. So the V-Model of testing incorporates testing • In October 1999 the $125 million NASA Mars Climate into the entire software development life cycle. In a diagram Orbiter— an interplanetary weather satellite — was lost of the V-Model, the V proceeds down and then up, from left to
ROLE OF TESTING IN CERTIFICATION PROCESS, JANUARY 2011 3 identiﬁes unexpected changes to existing processes introduced by the new application. For mission critical applications the importance of this testing cannot be overstated. Figure 2. Software Testing Phases  The V-Model of testing identiﬁes ﬁve software testing phases, each with a certain type of test associated with it. EachFigure 1. V-Model of Software Testing  testing phase and each individual test should have speciﬁc entry criteria that must be met before testing can begin and speciﬁc exit criteria that must be met before the test or phaseright depicting the basic sequence of development and testing can be certiﬁed as successfull. The entry and exit criteria areactivities. deﬁned by the Test Coordinators and listed in the Test Plan. The given V-Model ﬁgure 1 description given below. Then after successfull completion of test phases and testing 1) Unit testing: A series of stand-alone tests are conducted procedures, the certiﬁcation procedure comes in to existenceduring Unit Testing. Each test examines an individual com- which is then quite simple because it depends on successfullponent that is new or has been modiﬁed. A unit test is also testing processes.called a module test because it tests the individual units of So we can say that sucessfull testing process lays the founda-code that comprise the application. tion for successfull certiﬁcation process. 2) System testing: System Testing tests all components andmodules that are new or changed and are needed to form a III. P RINCIPAL C ERTIFICATION P ROBLEMScomplete system. The system test may require involvement ofother systems but it should be minimum to reduce the risk of This section highlights some problems that arise duringexternally-induced problems. The emphasis in system testing certiﬁcation process.is validating and verifying the functional design speciﬁcationand seeing how all the modules work together. A. Reliability Certiﬁcation Problem  3) Integration testing: Like system testing, integration test- There has been a phenomenal growth in the utilization ofing also tests all the components that are new or changed and object-oriented technology for developing software systemsare needed to form a complete system, but it also requires in last decades. It’s popularity is due to its support for datainvolvement of other systems and interfaces with other appli- abstraction, information hiding, extensional programming andcations including those owned by an outside vendor, external in particular reusable software.partners, or the customer. It has various subtypes- It’s major merit is its ﬂexibity to produce reusable modules, • Compatibility Testing - It ensures that application works but a thing to be kept in mind is that components are often with differently conﬁgured systems. not reused if their reliability cannot be guaranteed. Therefore • Performance Testing - Performance tests are used to eval- it is essential to realize that reliability certiﬁcation is a must uate and understand the application’s scalability when, when discussing reusability, but the problem is that no speciﬁc for example, more users are added or the volume of data test methods are prescribed by current object-oriented/based increases. development methods. This problem has just been of major • Stress Testing - Stress Testing is performance testing at interest as it has been realized that object orientation in itself higher than normal simulated loads. Stressing runs the is not sufﬁcient to create high quality software. system or application beyond the limits of its speciﬁed For testing there are two major alternatives: requirements to determine the load under which it fails • Black Box Testing - It takes an external view of the and how it fails. system and test cases are generated without knowledge 4) User Acceptance Testing: It is also callad as Beta testing of the interior of the system.or end user testing. It is where, testing moves from the hands • White box testing - It aims at covering paths in theof the IT department into those of the business users who then code or all lines in the code or maximising some otherperform the real world testing. coverage measure. 5) Production Veriﬁcation Testing: Production veriﬁcation The main objective of most testing techniques is to validatetesting is a ﬁnal opportunity to determine if the software that the system fulﬁlls the requirements, the focus is mostlyis ready for release. As a sort of full dress rehearsal, it on functional requirements whereas reliability certiﬁcation
ROLE OF TESTING IN CERTIFICATION PROCESS, JANUARY 2011 4focuses on detecting the faults that cause the most frequentfailures, hence maximising the growth in reliability.So often there are problems with providing reliability certi-ﬁcation to a product that is intended to be reused even afterperforming modern testing techniques.B. Veriﬁcation Methods Drawbacks  Software veriﬁcation approaches fall mainly in two groups: • Dynamic • StaticAlmost every existing standard uses these two approaches toverify software as a part of safety analysis. However theseveriﬁcation techniques are not advanced enough in relation to Figure 3. DO178B V & V Process the safety integrity levels needed for the software.Formal veriﬁcation techniques also have some major draw-backs.They are not entirely practical.For example, showing life cycle activities as appropriate.consistency between requirements and code does not ensure Veriﬁcation of coding and integration process involves reviewconﬁdence in safety since most of safety problems originates and testing of the source code implemented as per the Softwarefrom ﬂaws in requirements. Another drawback is the feasi- Design Data. The review comments and errors identiﬁed frombility, since few formal veriﬁcations applied to real programs this process are fed back to previous life cycle activities asrequire massive effort for relatively small software. appropriate.Only practical demonstrations can validate the usability of Veriﬁcation of integration process involves testing of the objectsome of the veriﬁcation methods and we know that these two code on Instruction Set Simulator/ Target Emulator, Targetveriﬁcation techniques directly lead to certiﬁcation process, board for compliance. The test results from this process areso often there are problems with certiﬁcation due to these fed back to previous life cycle activities as appropriate. Indrawbacks. general all errors that are reported are managed and tracked to closure.IV. R EQUIREMENTS FOR T ESTING BASED ON S TANDARDS And at last Software Veriﬁcation Cases and Procedures as This section highlights requirements with regard to testing well as the Software Veriﬁcation Results are veriﬁed forfor some basic standards used all over the world in different completeness and correctness in the Veriﬁcation of Veriﬁcationﬁelds. Process Results.Every software has different requirements for testing depend-ing on its own ﬁeld. B. Biomedical Engineering: IEC 60601-1-4 A. Aerospace: RTCA/DO-178B (EUROCAE ED12B)  It is a safety standard used in the ﬁeld of Biomedical It is an international standard relating to the safety and Engineering. Medical devices go through many stages ofairworthiness of software for avionics. During planning stage product testing before they are available to customers.of this standard various requirements related to testing in this Veriﬁcation and validation activities are repetitive,standard are mentioned in Software veriﬁcation plan (SVP). comprehensive, well planned and documented. OutputsDO-178B distinguishes between testing and veriﬁcation. are tested against design speciﬁcations.Testing - Process of testing per segment,does not ensureabsence of errors. The Figure 4 explains the verﬁcation and validation processVeriﬁcation - Veriﬁcation on the other hand is a generic term for Biomedical standards.for activities like Reviews, Analyses, and Testing. 1) Hardware Veriﬁcation: Safety and Electromagnetic The ﬁgure 3 represents the Veriﬁcation Process as required Compatibility (EMC) are two areas of focus during medicalby DO-178B and indicates the veriﬁcation activities at the end device hardware veriﬁcation including functional and perfor-of each of the processes like Requirement Process, Design mance testing, simulations, visual inspections, worst case/faultProcess, Coding and Software Integration Process and the tree analysis. All of the veriﬁcation tests must be applied toHardware Integration Process. all possible product variations.Veriﬁcation of requirement process involves review/analysis of 2) Software veriﬁcation: International requirements deﬁnedSoftware Requirement Data. The review comments from this in IEC 60601-1-4 speciﬁes the design controls that must beprocess are fed back to the previous life cycle activities as in place for medical software development. The software’sappropriate. actual performance must meet the corresponding design-inputVeriﬁcation of Design process involves review and analysis of document’s requirements. For most time it is a manual pro-the design that is provided in the Software Design Data. The cess and software developers maintain spreadsheets that linkreview comments from this process are fed back to previous veriﬁcation test results to the associated requirements.
ROLE OF TESTING IN CERTIFICATION PROCESS, JANUARY 2011 5 V. C ONCLUSION Software Testing has the potential to save time and money by identifying problems early and to improve customer satis- faction and safety by delivering a more error free product. Although Software Testing process is a crucial element in the development of embedded system, it plays more sub- stantial role in highly safety critical systems for supporting certiﬁcation process. In conclusion, to make optimum use of software testing, the veriﬁcation phase should be initialized from the very ﬁrst preliminary requirements analysis stage where requirements analysis and design reviews are done till the concluding stage where functional testing and environment modelling is ﬁnally done. Testing according to the new development methodoligies like object oriented development methods and some other modern softwares that come in group of those safety critical embedded systems which are intended to be used in different domains, have to be improved or we can say that be more intense andFigure 4. Design Control Model  speciﬁc to ensure more safety than it ever did.For example, in case of reliability certiﬁcation because companies cannot reuse software components without knowing how reliable 3) System validation: After the software and hardware they are, veriﬁcation and validation thus should be performeddesigns pass veriﬁcation activities, validation of the integrated on different modules or objects separately to assure theirsystems begins. To ensure that a medical device complies with reliability level.the governing rules and regulations of medical device software Todays world is revolutionized by modern technology whichvalidation, validation engineers constantly add, update, and emphasizes the human life to a great extent. A well planned,develop more detailed and quality-driven validation procedures comprehensive and documented requirements for veriﬁcationto increase the probability of ﬁnding errors. and validation process is implemented to make the safetyC. Generic Standard: IEC 61508  critical systems like Aviation and Medical more reliable and trustworthy before they are brought into use. IEC-61508 is an international standard of rules applied in I presume that with implementing and adhering new technolo-industry and titles as "Functional safety of programmable gies , the requirements for veriﬁcation and validation shouldelectronic safety-related systems". AdaTEST95 and Cantata++ be proactively modernized for advancement of the product andare used to meet the veriﬁcation and testing requirements of customer satisfaction.the standard. 1) Overview of Requirements: This standard has 7 parts ACKNOWLEDGEMENTSeach identifying different processes and the parts which iden-tify veriﬁcation, validation and test requirements relevant to I would like to specially thank Prof. Dr. Matthias WagnerAdaTEST95 and Cantata++ are: and my fellow colleagues for continuously supporting and • IEC 61508 Part 3: Software Requirements being continuously a source of motivation for me. • IEC 61508 Part 7: Over View of Techniques and Mea- sures R EFERENCESAdaTEST95 and Cantata++ is centered on a dynamic test  Veriﬁcation/Validation/Certiﬁcation, Carnegie Mellon University, 18-harness. The test harness can be used to support testing at all 849b Dependable Embedded Systems, Spring 1999, Author: Eushiuan Tranlevels from module testing through to full integration testing.  Software Testing Fundamentals—Concepts, Roles, and Terminology,IEC 61508 correctly demands planning of dynamic testing as John E. Bentley, Wachovia Bank, Charlotte NCpart of design process for each software object.The product’s  Software Testing, A guide to the TMAP Approach, Martin Pol, Ruud Teunissen, Erik van Veenendaalstructured test scripts can be used as Software Design Test  Reliability Certiﬁcation of Software Components, Claes Wohlin andSpeciﬁcations and Software Module Test Speciﬁcations as they Björn Regnel, Department of Communication Systems, Lund Instituteare easily readable by quality assurance staff. of Technology, Lund University, Box 118, SE-221 00 Lund, Sweden.  Software Safety Certiﬁcation: A Multi-domain Problem, PatriciaAdaTEST 95 and Cantata++ are well suited to the develop- Rodriguez-Dapena, European Space Agency, Copyrighted IEEEment of the software and facilitate a high degree of automation  Applying DO178B for IV & V of Safety critical Software, White Paper,of the veriﬁcation and test techniques required for effective use Sreekumar Panicker, Wipro Technologies  Testing Medical Devices, Written by Gary Powalisz, GEof the standard. Healthcare Available: http://www.evaluationengineering.com/index.AdaTEST 95 and Cantata++ have been developed to the php/solutions/instrumentation/testing-medical-devices.html. Last accesshighest practical standard for software verifcation tools and on:11/01/2011  IPL Testing Tools and IEC 61508, IPL Information Processing Ltd.,provide comprehensive functionality and they are the only Eveleigh House, Grove Street, UKtools developed to such high standards.