Role of Testing in Certification Process


Published on

Published in: Technology
  • Be the first to comment

Role of Testing in Certification Process

  1. 1. ROLE OF TESTING IN CERTIFICATION PROCESS, JANUARY 2011 1 Role of Testing in Certification Process Rishu Seth (Bachelor of Engineering, Computer Science) University of Applied Sciences - Frankfurt am Main, Germany Email: Abstract—This paper gives an overview of the procedure References 5of testing during verification and validation which identifiesimportant defects, flaws or errors in the application code thatmust be fixed during software development process and resulting L IST OF F IGURESin successfull certification process for that software.There are 1 V-Model of Software Testing [2] . . . . . . . . . 3different types of testings available for different types of processes 2 Software Testing Phases [2] . . . . . . . . . . . . 3according to the requirement of the software.Then this paperlists and describes the basic certification problems like Relia- 3 DO178B V & V Process [6] . . . . . . . . . . . 4bilty Certification Problem and Verification Methods Drawbacks 4 Design Control Model [7] . . . . . . . . . . . . . 5which are contemporary in today’s world. Also, it states somebasic standards and their requirements for testing. I. I NTRODUCTION [1] Ertification - A written guarantee that a system or compo- C ONTENTS C nent complies with its specified requirements and is ac- ceptable for operational use. Certification is usually carried outI Introduction [1] 1 by government agencies or other organizations with a national I-A Certification Process [1] . . . . . . . . . 2 standing. Certification can be applied to either organizations I-A1 Verification . . . . . . . . . . 2 or individuals, tools or methods, or systems or products. I-A2 Verification Techniques . . . 2 Certification with regard to organisation aims at making sure I-A3 Validation . . . . . . . . . . 2 that the organisation is making use of certain standards and I-A4 Validation Techniques . . . . 2 criterias to achieve a level of expertise. Though it is not completely applicable to all areas because measuring theII Role of Testing in Certification Process 2 procedures is easy as compared to measuring the competence II-A Why to do Software Testing? [2] . . . . 2 with which they are executed. So rather than being applied II-B Who Does The Testing? . . . . . . . . . 2 to design, certification is applied to areas such as quality II-C Test Specification Techniques [3] . . . . 2 assurance and testing to make sure that the aim is fullfilled. II-D The V-Model of Software Testing . . . 2 Apart from organisations, certification can also be applied to II-D1 Unit testing . . . . . . . . . 3 individuals, enabling them to be in a certain profession like II-D2 System testing . . . . . . . . 3 doctors, lawyers, accountants and civil engineers or to tools II-D3 Integration testing . . . . . . 3 for being used in different software development processes by II-D4 User Acceptance Testing . . 3 defining some basic requirements for both. For example, DO- II-D5 Production Verification Testing 3 178B/ED-12B does not define any specific tools to be used but gives certain requirements for tools that are to be used toIII Principal Certification Problems 3 gain certification. III-A Reliability Certification Problem [4] . . 3 Finally systems or products may also be certified. There is III-B Verification Methods Drawbacks [5] . . 4 an issue with certification process to whether methodolgy is to be certified rather than the artifact. This is a basic issueIV Requirements for Testing based on Standards 4 with products consisting of softwares specially safety critical IV-A Aerospace: RTCA/DO-178B (EURO- softwares, because software testing is so difficult and also just CAE ED12B) [6] . . . . . . . . . . . . 4 verifies that software is made following some guidelines and IV-B Biomedical Engineering: IEC 60601-1- has met some standard specified by certifying agency. This 4 [7] . . . . . . . . . . . . . . . . . . . 4 does not necessarily mean that the system is correct. IV-B1 Hardware Verification . . . . 4 There are different reasons why a product needs certifica- IV-B2 Software verification . . . . . 4 tion. Sometimes certification is required for legal reasons. IV-B3 System validation . . . . . . 5 For example, before an aircraft is allowed to fly, it must IV-C Generic Standard: IEC 61508 [8] . . . . 5 obtain a license. Being certified would also be important for IV-C1 Overview of Requirements . 5 commercial reasons like having a sales advantage. One of the main reasons for certification is to show competence in specificV Conclusion 5 areas.
  2. 2. ROLE OF TESTING IN CERTIFICATION PROCESS, JANUARY 2011 2A. Certification Process [1] in space due to a data conversion error. Investigators dis- 1) Verification: Verification - The process of evaluating a covered that software on the spacecraft performed certainsystem or component to determine whether the product of a calculations in English units (yards) when it should havegiven development phase satisfys the conditions imposed at used metric units (meters). • In June 1996 the first flight of the European Spacethe start of that phase. i.e. Agency’s Ariane 5 rocket failed shortly after launching,"Are we building the product right?" resulting in an uninsured loss of $500,000,000. The disaster was traced to the lack of exception handling for 2) Verification Techniques: There are many different veri- a floating-point error when a 64-bit integer was convertedfication techniques but they all basically fall into two major to a 16-bit signed integer.categories - Software testing answers questions that development testing • Dynamic testing and code reviews can’t. They are as follows: • Static testing • Does it really work as expected?But verification simply demonstrates whether the output of a • Does it meet the user’s requirements?phase conforms to the input of a phase as opposed to showing • Is it what the users expect?that the output is actually correct. Also it does not detect errors • Is it compatible with our other systems?resulting from wrong input specification and these errors may • How does it perform?effect in later development stages. So verification alone is not • How does it scale when more users are added?sufficient and we need validation as well to be more sure that • Which areas need more work?the system is operational according to specifications. • Is it ready for release? 3) Validation: Validation - The process of evaluating asystem or component during or at the end of the development Factors involved in testing are -process to determine whether it satisfies specified require- • Business requirementsments. i.e. • Functional design requirements • Technical design requirements"Are we building the right product?" • Programmer code 4) Validation Techniques: Validation usually takes place at • Hardware configurationthe end of the development cycle, and looks at the completesystem as opposed to verification, which focuses on smaller B. Who Does The Testing?sub-systems. Various validation techniques are: Software testing is not a one person job. It takes a team, • Formal methods but the team may be larger or smaller depending on the size • Fault injection and complexity of the application being tested. The actual • Dependability analysis programmers that wrote the code should have a reduced role • Hazard analysis in testing their own written code. • Risk analysis C. Test Specification Techniques [3] II. ROLE OF T ESTING IN C ERTIFICATION P ROCESS Test specification techniques can be split up into two groups Testing always means comparing. It requires an item to are as follows:be tested and terms of reference with which the item must • White Box Testing - In this testing, techniques arecomply. It provides an insight into the difference between the based on the program code, the program descriptions oractual status and the required status of the item. technical design. Knowledge about the internal structureAccording to International Standard Organisation testing is of the system plays an important role. Other terms useddescribed as - for this kind of techniques are Glass-box or Structural"Technical operation that consists of the determination of one testing.or more characteristics of a given product, process or service • Black Box Testing - In this testing, techniques are basedaccording to a specified procedure" [3]. on functional specifications and quality requirements. The whole system is viewed as it would be in actual case.InA. Why to do Software Testing? [2] this kind of testing, knowledge about the structure of the system is not used but the judgement is made merely The most basic and instinctive response of people is "To from a functional point of view of the system.find the bugs!", but programmers and developers know that forfinding bugs ’Debugging’ is there. But actually “bug” is reallya problem in the code, whereas software testing is focused on D. The V-Model of Software Testingfinding defects in the final product. Examples of important It is not advisable to leave Software Testing for the enddefects that better testing would have found: of the project. So the V-Model of testing incorporates testing • In October 1999 the $125 million NASA Mars Climate into the entire software development life cycle. In a diagram Orbiter— an interplanetary weather satellite — was lost of the V-Model, the V proceeds down and then up, from left to
  3. 3. ROLE OF TESTING IN CERTIFICATION PROCESS, JANUARY 2011 3 identifies unexpected changes to existing processes introduced by the new application. For mission critical applications the importance of this testing cannot be overstated. Figure 2. Software Testing Phases [2] The V-Model of testing identifies five software testing phases, each with a certain type of test associated with it. EachFigure 1. V-Model of Software Testing [2] testing phase and each individual test should have specific entry criteria that must be met before testing can begin and specific exit criteria that must be met before the test or phaseright depicting the basic sequence of development and testing can be certified as successfull. The entry and exit criteria areactivities. defined by the Test Coordinators and listed in the Test Plan. The given V-Model figure 1 description given below. Then after successfull completion of test phases and testing 1) Unit testing: A series of stand-alone tests are conducted procedures, the certification procedure comes in to existenceduring Unit Testing. Each test examines an individual com- which is then quite simple because it depends on successfullponent that is new or has been modified. A unit test is also testing processes.called a module test because it tests the individual units of So we can say that sucessfull testing process lays the founda-code that comprise the application. tion for successfull certification process. 2) System testing: System Testing tests all components andmodules that are new or changed and are needed to form a III. P RINCIPAL C ERTIFICATION P ROBLEMScomplete system. The system test may require involvement ofother systems but it should be minimum to reduce the risk of This section highlights some problems that arise duringexternally-induced problems. The emphasis in system testing certification validating and verifying the functional design specificationand seeing how all the modules work together. A. Reliability Certification Problem [4] 3) Integration testing: Like system testing, integration test- There has been a phenomenal growth in the utilization ofing also tests all the components that are new or changed and object-oriented technology for developing software systemsare needed to form a complete system, but it also requires in last decades. It’s popularity is due to its support for datainvolvement of other systems and interfaces with other appli- abstraction, information hiding, extensional programming andcations including those owned by an outside vendor, external in particular reusable, or the customer. It has various subtypes- It’s major merit is its flexibity to produce reusable modules, • Compatibility Testing - It ensures that application works but a thing to be kept in mind is that components are often with differently configured systems. not reused if their reliability cannot be guaranteed. Therefore • Performance Testing - Performance tests are used to eval- it is essential to realize that reliability certification is a must uate and understand the application’s scalability when, when discussing reusability, but the problem is that no specific for example, more users are added or the volume of data test methods are prescribed by current object-oriented/based increases. development methods. This problem has just been of major • Stress Testing - Stress Testing is performance testing at interest as it has been realized that object orientation in itself higher than normal simulated loads. Stressing runs the is not sufficient to create high quality software. system or application beyond the limits of its specified For testing there are two major alternatives: requirements to determine the load under which it fails • Black Box Testing - It takes an external view of the and how it fails. system and test cases are generated without knowledge 4) User Acceptance Testing: It is also callad as Beta testing of the interior of the system.or end user testing. It is where, testing moves from the hands • White box testing - It aims at covering paths in theof the IT department into those of the business users who then code or all lines in the code or maximising some otherperform the real world testing. coverage measure. 5) Production Verification Testing: Production verification The main objective of most testing techniques is to validatetesting is a final opportunity to determine if the software that the system fulfills the requirements, the focus is mostlyis ready for release. As a sort of full dress rehearsal, it on functional requirements whereas reliability certification
  4. 4. ROLE OF TESTING IN CERTIFICATION PROCESS, JANUARY 2011 4focuses on detecting the faults that cause the most frequentfailures, hence maximising the growth in reliability.So often there are problems with providing reliability certi-fication to a product that is intended to be reused even afterperforming modern testing techniques.B. Verification Methods Drawbacks [5] Software verification approaches fall mainly in two groups: • Dynamic • StaticAlmost every existing standard uses these two approaches toverify software as a part of safety analysis. However theseverification techniques are not advanced enough in relation to Figure 3. DO178B V & V Process [6]the safety integrity levels needed for the software.Formal verification techniques also have some major draw-backs.They are not entirely practical.For example, showing life cycle activities as appropriate.consistency between requirements and code does not ensure Verification of coding and integration process involves reviewconfidence in safety since most of safety problems originates and testing of the source code implemented as per the Softwarefrom flaws in requirements. Another drawback is the feasi- Design Data. The review comments and errors identified frombility, since few formal verifications applied to real programs this process are fed back to previous life cycle activities asrequire massive effort for relatively small software. appropriate.Only practical demonstrations can validate the usability of Verification of integration process involves testing of the objectsome of the verification methods and we know that these two code on Instruction Set Simulator/ Target Emulator, Targetverification techniques directly lead to certification process, board for compliance. The test results from this process areso often there are problems with certification due to these fed back to previous life cycle activities as appropriate. Indrawbacks. general all errors that are reported are managed and tracked to closure.IV. R EQUIREMENTS FOR T ESTING BASED ON S TANDARDS And at last Software Verification Cases and Procedures as This section highlights requirements with regard to testing well as the Software Verification Results are verified forfor some basic standards used all over the world in different completeness and correctness in the Verification of Verificationfields. Process Results.Every software has different requirements for testing depend-ing on its own field. B. Biomedical Engineering: IEC 60601-1-4 [7]A. Aerospace: RTCA/DO-178B (EUROCAE ED12B) [6] It is a safety standard used in the field of Biomedical It is an international standard relating to the safety and Engineering. Medical devices go through many stages ofairworthiness of software for avionics. During planning stage product testing before they are available to customers.of this standard various requirements related to testing in this Verification and validation activities are repetitive,standard are mentioned in Software verification plan (SVP). comprehensive, well planned and documented. OutputsDO-178B distinguishes between testing and verification. are tested against design specifications.Testing - Process of testing per segment,does not ensureabsence of errors. The Figure 4 explains the verfication and validation processVerification - Verification on the other hand is a generic term for Biomedical standards.for activities like Reviews, Analyses, and Testing. 1) Hardware Verification: Safety and Electromagnetic The figure 3 represents the Verification Process as required Compatibility (EMC) are two areas of focus during medicalby DO-178B and indicates the verification activities at the end device hardware verification including functional and perfor-of each of the processes like Requirement Process, Design mance testing, simulations, visual inspections, worst case/faultProcess, Coding and Software Integration Process and the tree analysis. All of the verification tests must be applied toHardware Integration Process. all possible product variations.Verification of requirement process involves review/analysis of 2) Software verification: International requirements definedSoftware Requirement Data. The review comments from this in IEC 60601-1-4 specifies the design controls that must beprocess are fed back to the previous life cycle activities as in place for medical software development. The software’sappropriate. actual performance must meet the corresponding design-inputVerification of Design process involves review and analysis of document’s requirements. For most time it is a manual pro-the design that is provided in the Software Design Data. The cess and software developers maintain spreadsheets that linkreview comments from this process are fed back to previous verification test results to the associated requirements.
  5. 5. ROLE OF TESTING IN CERTIFICATION PROCESS, JANUARY 2011 5 V. C ONCLUSION Software Testing has the potential to save time and money by identifying problems early and to improve customer satis- faction and safety by delivering a more error free product. Although Software Testing process is a crucial element in the development of embedded system, it plays more sub- stantial role in highly safety critical systems for supporting certification process. In conclusion, to make optimum use of software testing, the verification phase should be initialized from the very first preliminary requirements analysis stage where requirements analysis and design reviews are done till the concluding stage where functional testing and environment modelling is finally done. Testing according to the new development methodoligies like object oriented development methods and some other modern softwares that come in group of those safety critical embedded systems which are intended to be used in different domains, have to be improved or we can say that be more intense andFigure 4. Design Control Model [7] specific to ensure more safety than it ever did.For example, in case of reliability certification because companies cannot reuse software components without knowing how reliable 3) System validation: After the software and hardware they are, verification and validation thus should be performeddesigns pass verification activities, validation of the integrated on different modules or objects separately to assure theirsystems begins. To ensure that a medical device complies with reliability level.the governing rules and regulations of medical device software Todays world is revolutionized by modern technology whichvalidation, validation engineers constantly add, update, and emphasizes the human life to a great extent. A well planned,develop more detailed and quality-driven validation procedures comprehensive and documented requirements for verificationto increase the probability of finding errors. and validation process is implemented to make the safetyC. Generic Standard: IEC 61508 [8] critical systems like Aviation and Medical more reliable and trustworthy before they are brought into use. IEC-61508 is an international standard of rules applied in I presume that with implementing and adhering new technolo-industry and titles as "Functional safety of programmable gies , the requirements for verification and validation shouldelectronic safety-related systems". AdaTEST95 and Cantata++ be proactively modernized for advancement of the product andare used to meet the verification and testing requirements of customer satisfaction.the standard. 1) Overview of Requirements: This standard has 7 parts ACKNOWLEDGEMENTSeach identifying different processes and the parts which iden-tify verification, validation and test requirements relevant to I would like to specially thank Prof. Dr. Matthias WagnerAdaTEST95 and Cantata++ are: and my fellow colleagues for continuously supporting and • IEC 61508 Part 3: Software Requirements being continuously a source of motivation for me. • IEC 61508 Part 7: Over View of Techniques and Mea- sures R EFERENCESAdaTEST95 and Cantata++ is centered on a dynamic test [1] Verification/Validation/Certification, Carnegie Mellon University, 18-harness. The test harness can be used to support testing at all 849b Dependable Embedded Systems, Spring 1999, Author: Eushiuan Tranlevels from module testing through to full integration testing. [2] Software Testing Fundamentals—Concepts, Roles, and Terminology,IEC 61508 correctly demands planning of dynamic testing as John E. Bentley, Wachovia Bank, Charlotte NCpart of design process for each software object.The product’s [3] Software Testing, A guide to the TMAP Approach, Martin Pol, Ruud Teunissen, Erik van Veenendaalstructured test scripts can be used as Software Design Test [4] Reliability Certification of Software Components, Claes Wohlin andSpecifications and Software Module Test Specifications as they Björn Regnel, Department of Communication Systems, Lund Instituteare easily readable by quality assurance staff. of Technology, Lund University, Box 118, SE-221 00 Lund, Sweden. [5] Software Safety Certification: A Multi-domain Problem, PatriciaAdaTEST 95 and Cantata++ are well suited to the develop- Rodriguez-Dapena, European Space Agency, Copyrighted IEEEment of the software and facilitate a high degree of automation [6] Applying DO178B for IV & V of Safety critical Software, White Paper,of the verification and test techniques required for effective use Sreekumar Panicker, Wipro Technologies [7] Testing Medical Devices, Written by Gary Powalisz, GEof the standard. Healthcare Available: 95 and Cantata++ have been developed to the php/solutions/instrumentation/testing-medical-devices.html. Last accesshighest practical standard for software verifcation tools and on:11/01/2011 [8] IPL Testing Tools and IEC 61508, IPL Information Processing Ltd.,provide comprehensive functionality and they are the only Eveleigh House, Grove Street, UKtools developed to such high standards.