Automatic Testing Tool
Introduction <ul><li>The tool provides facilities for:   </li></ul><ul><li>Dynamic Testing : executing the software under ...
 
Functional Overview <ul><li>Dynamic Testing  facilities allow both host and target.    </li></ul><ul><ul><ul><ul><li>Cover...
<ul><li>Users can define their coverage analysis, and static analysis criteria. </li></ul><ul><li>Dynamic Testing </li></u...
 
<ul><li>There must be some form of specification for the software </li></ul><ul><li>The testing tool must handle test case...
<ul><li>The Solution </li></ul>
<ul><li>A  test script  controls tests.  </li></ul><ul><li>The test script is compiled and linked with Test Harness (TH) a...
The Test Harness <ul><li>Test Harness (TH) provides facilities to run, verify the results of, document, and repeat dynamic...
 
<ul><li>When a test is run, a Results file is produced detailing every step of the test and highlighting any failures.  </...
<ul><li>TH verifies data using  CHECK  directives. These cause the comparison of a data item with its expected value.  </l...
<ul><li>Similarly, external data areas may be simulated and checked.    </li></ul><ul><li>Timing Analysis </li></ul><ul><l...
Results <ul><li>TH directives mark the progress of the test run in the output.  </li></ul><ul><li>Any  unexpected  event i...
<ul><li>Test Results For : example  </li></ul><ul><li>Results File : example.ctr </li></ul><ul><li>Tests Run At : Feb 17 0...
 
Test Script Generation <ul><li>The Test Script generator (TS) prepares dynamic test scripts for execution with TH.  </li><...
 
<ul><li>Test scripts produced by TS feature positive and negative data checking.  </li></ul><ul><li>TS automatically codes...
Coverage Analysis <ul><li>Coverage analysis measures the proportion of software executed during dynamic testing.    </li><...
Static Analysis <ul><li>Static Analysis provides an assessment of various non-functional features relating to the software...
<ul><li>Data areas and types, which are declared but not used, are highlighted.  </li></ul><ul><li>Users can define their ...
Code Complexity <ul><li>Many metrics are supported:   </li></ul><ul><li>McCabe's measure and Myers' extension   </li></ul>...
Using Analysis <ul><li>Test Analysis may be used in many different ways and, if required, be extended by the user to meet ...
<ul><ul><li>An extension to TH, allowing the user to fully integrate dynamic testing with coverage analysis and static ver...
 
<ul><li>The  pre-processor  and  instrumenter  are used to produce instrumented source code that is source code containing...
Stand-Alone Analysis <ul><li>Test Analysis can also be used stand-alone.  </li></ul><ul><li>Developers can use their own o...
 
Coverage Analysis Results <ul><li>Coverage Analysis Within a Test Script </li></ul><ul><li>Users have access to coverage a...
 
<ul><li>>> WARNING: Switch executed with unknown case value </li></ul><ul><li>Total of decision outcomes = 14 </li></ul><u...
Static Analysis Results <ul><li>Static analysis results can be accessed in a number of ways:   </li></ul><ul><ul><li>as pa...
<ul><li>Static Analysis Measures for &quot;control.c a03m00&quot; </li></ul><ul><li>SOURCE_LINES                     557 <...
<ul><li>. </li></ul><ul><li>. </li></ul><ul><li>MAXIMUM_NESTING_LEVEL            4 </li></ul><ul><li>AVERAGE_NESTING_LEVEL...
<ul><li>HALSTEAD_NUM_UNIQUE_OPERANDS     75 </li></ul><ul><li>HALSTEAD_TOTAL_NUM_OPERANDS      221 </li></ul><ul><li>. </l...
<ul><li>UNUSED_LOCAL_TYPES               0 </li></ul><ul><li>DATA ANALYSIS </li></ul><ul><li>-----------------------------...
<ul><li>ct02_dirsep_ca           extern             1 </li></ul><ul><li>ct02_hdrftr_pca          extern             6 </li...
<ul><li>vl_error_b                                      10 </li></ul><ul><li>vl_error_n                                   ...
<ul><li>MAX_CONDITION_COMP_NESTING       3 </li></ul><ul><li>==================== Overall Measures For The File ==========...
Upcoming SlideShare
Loading in...5
×

12 Automated Testing Tools

483

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
483
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
21
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

12 Automated Testing Tools

  1. 1. Automatic Testing Tool
  2. 2. Introduction <ul><li>The tool provides facilities for:  </li></ul><ul><li>Dynamic Testing : executing the software under test in order to verify its compliance </li></ul><ul><li>Coverage Analysis : measuring the proportion of software exercised by dynamic testing  </li></ul><ul><li>Static Analysis : examining source code to assess its complexity </li></ul>
  3. 4. Functional Overview <ul><li>Dynamic Testing facilities allow both host and target.   </li></ul><ul><ul><ul><ul><li>Coverage Analysis facilities allow to assess the coverage of individual software units, programs/tasks and entire software systems. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Static Analysis: Software that has been thoroughly dynamically tested can still have problems. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Static analysis provides information on qualities such as maintainability and compliance with coding standards.   </li></ul></ul></ul></ul><ul><li>The number of code statements and the McCabe's cyclomatic complexity metric are calculated.  </li></ul>
  4. 5. <ul><li>Users can define their coverage analysis, and static analysis criteria. </li></ul><ul><li>Dynamic Testing </li></ul><ul><li>The Testing Problem </li></ul>
  5. 7. <ul><li>There must be some form of specification for the software </li></ul><ul><li>The testing tool must handle test cases generated using </li></ul><ul><ul><li>functional (black-box) </li></ul></ul><ul><ul><li>structural (white-box) </li></ul></ul><ul><ul><li>isolation   </li></ul></ul><ul><ul><li>Integration testing </li></ul></ul><ul><ul><li>positive testing </li></ul></ul><ul><ul><li>negative testing   </li></ul></ul>
  6. 8. <ul><li>The Solution </li></ul>
  7. 9. <ul><li>A test script controls tests. </li></ul><ul><li>The test script is compiled and linked with Test Harness (TH) and the software under test. </li></ul><ul><li>This produces an executable program that produces a test Results file.  </li></ul><ul><li>The benefits of this approach are:  </li></ul><ul><ul><ul><li>Documentation : The tool provides documented testing.  </li></ul></ul></ul><ul><ul><ul><li>Repeatability : Tests can be easily.  </li></ul></ul></ul><ul><ul><ul><li>Maintainability : Tests Scripts are easy to understand and updated.  </li></ul></ul></ul>
  8. 10. The Test Harness <ul><li>Test Harness (TH) provides facilities to run, verify the results of, document, and repeat dynamic tests. </li></ul><ul><li>TH consists of a set of library directives accessed from the test script. </li></ul><ul><li>The test script calls the software under test and the TH directives embedded in the script check the effects of the call on environment.  </li></ul>
  9. 12. <ul><li>When a test is run, a Results file is produced detailing every step of the test and highlighting any failures. </li></ul><ul><li>A Results table is displayed summarizing the results for each test case and providing total figures. </li></ul><ul><li>An overall statement of test pass or fail is provided and returned to the command shell.   </li></ul><ul><li>Checking Values </li></ul><ul><li>Most important aspect of dynamic testing is checking that the outputs from the software under test are as expected. </li></ul>
  10. 13. <ul><li>TH verifies data using CHECK directives. These cause the comparison of a data item with its expected value.  </li></ul><ul><li>Simulating External Software </li></ul><ul><li>Isolation testing implies that calls to external units and external data must be simulated.  </li></ul><ul><li>External calls can be simulated, ensuring that they are made in the expected order and that input parameters have the correct values at each call. </li></ul><ul><li>Return values can be individually set on different calls to the same simulated function. </li></ul>
  11. 14. <ul><li>Similarly, external data areas may be simulated and checked.  </li></ul><ul><li>Timing Analysis </li></ul><ul><li>Not only the correct functioning of software determines its acceptability. </li></ul><ul><li>Some applications need to perform certain activities both correctly and within defined time constraints.  </li></ul><ul><li>The tool permits execution times to be recorded and tests passed or failed depending on the performance of the software.  </li></ul>
  12. 15. Results <ul><li>TH directives mark the progress of the test run in the output. </li></ul><ul><li>Any unexpected event is highlighted by a message. </li></ul><ul><li>For example: a failed check will be marked with FAILED and a diagnostic will give both the actual and expected values of the item being checked.  </li></ul><ul><li>The output below contains a typical section of a TH output file: </li></ul>
  13. 16. <ul><li>Test Results For : example  </li></ul><ul><li>Results File : example.ctr </li></ul><ul><li>Tests Run At : Feb 17 09:38:32 </li></ul><ul><li>Start Test 001 </li></ul><ul><li>   EXECUTE: my_function, </li></ul><ul><li>     Expected calls = 1 </li></ul><ul><li>     START_STUB : my_stub </li></ul><ul><li>         CALL_REF/ACTION: Action 1, Call 1 </li></ul><ul><li>         Check PASSED : my_stub_string </li></ul><ul><li>         Item = &quot;Hello, world&quot; </li></ul><ul><li>     END_STUB : my_stub </li></ul><ul><li>   DONE : my_function </li></ul><ul><li>Check FAILED : my_external </li></ul><ul><li>Expected 0x0000712B 28971  </li></ul><ul><li>Item 0x000080E8 33000  </li></ul><ul><li>End Test 001 </li></ul>
  14. 18. Test Script Generation <ul><li>The Test Script generator (TS) prepares dynamic test scripts for execution with TH. </li></ul><ul><li>TS takes information from a test case definition file and generates a test script. </li></ul><ul><li>The test case definition file specifies test cases, establishes initial conditions and expected results.  </li></ul><ul><li>TS can produce comprehensive test script templates which can be the basis of manually coded test scripts.  </li></ul><ul><li>TS works by scanning the test case definition file and the software under test to produce the test script.  </li></ul>
  15. 20. <ul><li>Test scripts produced by TS feature positive and negative data checking. </li></ul><ul><li>TS automatically codes check routines for user defined types and stubs for external functions/classes.  </li></ul><ul><li>Analysis </li></ul><ul><li>Test Analysis (TA) provides the user with Coverage Analysis and Static Analysis features. </li></ul>
  16. 21. Coverage Analysis <ul><li>Coverage analysis measures the proportion of software executed during dynamic testing.  </li></ul><ul><ul><li>statement coverage </li></ul></ul><ul><ul><li>decision (or branch ) coverage </li></ul></ul><ul><ul><li>boolean expression coverage </li></ul></ul><ul><ul><li>call pair coverage : measures the proportion of calls to other functions which have been exercised. </li></ul></ul><ul><ul><li>call coverage : measures that expected functions have been called.  </li></ul></ul><ul><ul><li>data value coverage : checks that program variables have held a series of (user defined) values during the testing process.  </li></ul></ul>
  17. 22. Static Analysis <ul><li>Static Analysis provides an assessment of various non-functional features relating to the software.  </li></ul><ul><li>Enforcement of coding standards   </li></ul><ul><li>Measurement of code complexity and structure .   </li></ul><ul><li>Coding Standards </li></ul><ul><li>The metrics examined are incorporated into overall test pass/fail criteria. </li></ul><ul><ul><li>For example, users may check that no goto statements (or labels) are used, that only one return statement is present and that there are no switch statements without a default .  </li></ul></ul>
  18. 23. <ul><li>Data areas and types, which are declared but not used, are highlighted. </li></ul><ul><li>Users can define their static analysis metrics to check on the use of code. </li></ul><ul><ul><li>For instance, these facilities can be used to restrict access to certain library routine calls (such as malloc ).   </li></ul></ul>
  19. 24. Code Complexity <ul><li>Many metrics are supported:  </li></ul><ul><li>McCabe's measure and Myers' extension  </li></ul><ul><li>Essential McCabe's  </li></ul><ul><li>Hansen's measure of software complexity by the pair  </li></ul><ul><li>Halstead's software science metrics  </li></ul><ul><li>Harrison's scope ratio   </li></ul>
  20. 25. Using Analysis <ul><li>Test Analysis may be used in many different ways and, if required, be extended by the user to meet client specific requirements.  </li></ul><ul><li>The Analysis comprises:  </li></ul><ul><ul><li>a special C pre-processor ( TP ).  </li></ul></ul><ul><ul><li>an Instrumenter program, to analyse source code files and insert coverage 'probes' ( TI );  </li></ul></ul><ul><ul><li>an additional library of test directives, which may be incorporated into a test script ( the TA library ).  </li></ul></ul><ul><li>The Analysis can be used in two main ways. </li></ul>
  21. 26. <ul><ul><li>An extension to TH, allowing the user to fully integrate dynamic testing with coverage analysis and static verification. </li></ul></ul><ul><li>This diagram illustrates the use of Cantata Analysis in this way:  </li></ul>
  22. 28. <ul><li>The pre-processor and instrumenter are used to produce instrumented source code that is source code containing probes to facilitate the collection of coverage data. </li></ul><ul><li>The instrumenter produces an annotated code listing containing the source code and a static analysis report.  </li></ul><ul><li>The instrumented source code is compiled and linked with a test script and the TA library. </li></ul><ul><li>When run, the resultant executable produces test results which includes both static analysis, coverage analysis information, and dynamic test results.   </li></ul>
  23. 29. Stand-Alone Analysis <ul><li>Test Analysis can also be used stand-alone. </li></ul><ul><li>Developers can use their own or a third party tool test software, while generating analysis reports using Test Analysis:   </li></ul>
  24. 31. Coverage Analysis Results <ul><li>Coverage Analysis Within a Test Script </li></ul><ul><li>Users have access to coverage and static analysis results from within the test script. </li></ul><ul><li>The next example shows a simple decision coverage report, indicating execution of each decision:   </li></ul>
  25. 33. <ul><li>>> WARNING: Switch executed with unknown case value </li></ul><ul><li>Total of decision outcomes = 14 </li></ul><ul><li>Total outcomes exercised at least once = 6 </li></ul><ul><li>Decision coverage = 42% </li></ul><ul><li>>> WARNING: DECISION COVERAGE INCOMPLETE </li></ul><ul><li>Stand-Alone Coverage Analysis </li></ul><ul><li>In stand-alone mode, coverage analysis can be used to provide coverage reporting from units, modules or complete application programs.  </li></ul>
  26. 34. Static Analysis Results <ul><li>Static analysis results can be accessed in a number of ways:  </li></ul><ul><ul><li>as part of the instrumenter list file  </li></ul></ul><ul><ul><li>as a comma separated value ( .CSV ) file, which can be exported to spreadsheet or database packages  </li></ul></ul><ul><ul><li>from within the TH test script, where they may be checked  </li></ul></ul><ul><li>An example of each of these formats is shown next.   </li></ul>
  27. 35. <ul><li>Static Analysis Measures for &quot;control.c a03m00&quot; </li></ul><ul><li>SOURCE_LINES                     557 </li></ul><ul><li>CODE_LINES                       198 </li></ul><ul><li>COMMENT_LINES                    353 </li></ul><ul><li>BLANK_LINES                      45 </li></ul><ul><li>EXPRESSION_STATEMENTS            53 </li></ul><ul><li>FOR_LOOP_STATEMENTS              3 </li></ul><ul><li>WHILE_LOOP_STATEMENTS            0 </li></ul><ul><li>DO_LOOP_STATEMENTS               0 </li></ul><ul><li>IF_STATEMENTS                    13 </li></ul><ul><li>SWITCH_STATEMENTS                1 </li></ul><ul><li>RETURN_STATEMENTS                1 </li></ul><ul><li>GOTO_STATEMENTS                  0 </li></ul><ul><li>STATEMENTS                       74 </li></ul><ul><li>DECLARATIONS                     13 </li></ul><ul><li>COMMENTS                         215 </li></ul>
  28. 36. <ul><li>. </li></ul><ul><li>. </li></ul><ul><li>MAXIMUM_NESTING_LEVEL            4 </li></ul><ul><li>AVERAGE_NESTING_LEVEL            0.58 </li></ul><ul><li>. </li></ul><ul><li>. </li></ul><ul><li>. </li></ul><ul><li>MCCABE                           19 </li></ul><ul><li>ESSENTIAL_MCCABE                 1 </li></ul><ul><li>MYERS_MCCABE_LOWER               19 </li></ul><ul><li>MYERS_MCCABE_UPPER               21 </li></ul><ul><li>HANSEN_CYCLOMATIC_NUM            18 </li></ul><ul><li>HANSEN_OPERATOR_COUNT            98 </li></ul><ul><li>HARRISON_SCOPE_RATIO             0.45 </li></ul><ul><li>HALSTEAD_NUM_UNIQUE_OPRS         21 </li></ul><ul><li>HALSTEAD_TOTAL_NUM_OPERATORS     194 </li></ul>
  29. 37. <ul><li>HALSTEAD_NUM_UNIQUE_OPERANDS     75 </li></ul><ul><li>HALSTEAD_TOTAL_NUM_OPERANDS      221 </li></ul><ul><li>. </li></ul><ul><li>. </li></ul><ul><li>. </li></ul><ul><li>CLASSES                          0 </li></ul><ul><li>NEW                              0 </li></ul><ul><li>DELETE                           0 </li></ul><ul><li>THROW                            0 </li></ul><ul><li>TRY_CATCH                        0 </li></ul><ul><li>ANONYMOUS_UNIONS                 0 </li></ul><ul><li>PARAMETERS                       2 </li></ul><ul><li>UNUSED_PARAMETERS                0 </li></ul><ul><li>AUTOMATICS                       13 </li></ul><ul><li>STATICS                          0 </li></ul><ul><li>UNUSED_DATA                      0 </li></ul><ul><li>LOCAL_TYPES                      0 </li></ul>
  30. 38. <ul><li>UNUSED_LOCAL_TYPES               0 </li></ul><ul><li>DATA ANALYSIS </li></ul><ul><li>------------------------------------------------- </li></ul><ul><li>Definitions and Declarations Outside of Any Function </li></ul><ul><li>------------------------------------------------- </li></ul><ul><li>Name                     Flags              No. of References </li></ul><ul><li>-----              ----------------- </li></ul><ul><li>a03m01                   function           9 </li></ul><ul><li>a03m02                   function           5 </li></ul><ul><li>a03m03                   function           2 </li></ul><ul><li>. </li></ul><ul><li>. </li></ul><ul><li>. </li></ul>
  31. 39. <ul><li>ct02_dirsep_ca           extern             1 </li></ul><ul><li>ct02_hdrftr_pca          extern             6 </li></ul><ul><li>ct02_notmdte_ca          extern             0 UNUSED </li></ul><ul><li>. </li></ul><ul><li>. </li></ul><ul><li>Definitions and Declarations Within Function : a03m00 </li></ul><ul><li>-------------------------------------------- </li></ul><ul><li>Name                     Flags                  No. of References </li></ul><ul><li>                 ----------------- </li></ul><ul><li>cr_argv_pca              parameter              1 </li></ul><ul><li>cr_numargc_n             parameter              0 UNUSED </li></ul><ul><li>vl_baselen_i                                    2 </li></ul>
  32. 40. <ul><li>vl_error_b                                      10 </li></ul><ul><li>vl_error_n                                      4 </li></ul><ul><li>vl_exit_z                                       0 UNUSED </li></ul><ul><li>vl_fileloop_i                                   6 </li></ul><ul><li>vl_format_pca                                   4  </li></ul><ul><li>============== Overall Preprocessor Measures For The File =============  </li></ul><ul><li>TOTAL_MACROS                     1272 </li></ul><ul><li>MACROS_TO_BE_EXPANDED            1270 </li></ul><ul><li>MACROS_NOT_TO_BE_EXPANDED        2 </li></ul><ul><li>SUBSTITUTED_MACROS               111 </li></ul><ul><li>UNSUBSTITUTED_MACROS             0 </li></ul><ul><li>DIRECT_INCLUDE_FILES             10 </li></ul><ul><li>INDIRECT_INCLUDE_FILES           1 </li></ul><ul><li>MAX_INCLUDE_NESTING              2 </li></ul>
  33. 41. <ul><li>MAX_CONDITION_COMP_NESTING       3 </li></ul><ul><li>==================== Overall Measures For The File ==================== </li></ul><ul><li>FILE_SOURCE_LINES                767 </li></ul><ul><li>FILE_CODE_LINES                  238 </li></ul><ul><li>FILE_COMMENT_LINES               465 </li></ul><ul><li>FILE_BLANK_LINES                 105 </li></ul><ul><li>FILE_STATEMENTS                  74 </li></ul><ul><li>FILE_DECLARATIONS                50 </li></ul><ul><li>FILE_COMMENTS                    309 </li></ul><ul><li>FILE_FUNCTIONS                   1 </li></ul><ul><li>FILE_CHECKSUM                    846162015 </li></ul><ul><li>FILE_CLASSES                     0 </li></ul><ul><li>FUNCTION_CLASSES                 0 </li></ul><ul><li>FILE_FRIENDS                     0 </li></ul><ul><li>FUNCTIONS_IN_CLASSES             0 </li></ul>
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×