Copyright McCabe


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Slide 10 shows the integration between McCabe IQ and configuration management tools. McCabe’s current partners in the configuration management space are Merant, Rational, and CA. There are two key areas of synergy between McCabe IQ and configuration management tools. First, we can set up triggers in the configuration management tools so that McCabe IQ is called to analyze software quality as each software change is made, so we can monitor and manage software quality when the software changes, rather than just storing the changed software itself. Next, McCabe IQ and the configuration management tool can combine to provide a test environment in the same way that you currently have production and debugging environments. For example, in addition to a nightly production build, you can set up a nightly instrumented build that monitors test execution. Then, as your testers, who may not know anything about McCabe IQ or configuration management, execute tests, you can analyze the effectiveness of their testing and help refocus their efforts as appropriate.
  • Slide 11 shows the integration between McCabe IQ and test automation. McCabe’s current partner in the test automation space is Mercury Interactive. McCabe IQ analyzes source code to focus testing effort on the highest-risk software, and this information is used by TestDirector when planning and managing testing activity. McCabe IQ also produces test executables that monitor testing progress, so that you can develop a set of WinRunner scripts that effectively test your application rather than just pushing all the buttons and selecting all the menu choices. Just pushing buttons and selecting menu choices typically exercises less than 30% of the application logic. For non-GUI applications, the McCabe ReTest component automates test case execution and verification. Putting it all together, McCabe IQ allows you to manage your automated testing effort based on software risk, and make sure your automated test suites provide a thorough, effective test of your application.
  • Slide 15 shows the McCabe Compare add-on. McCabe Compare helps identify and maintain duplicate and near-duplicate software modules by comparing metrics, names, logic structure, data references, and other characteristics. Since similar software modules often contain similar errors, McCabe Compare helps propagate software changes everywhere they are needed.
  • Slide 16 shows the McCabe Change add-on. McCabe Change identifies software change at the module level, so it can be used along with quality and testing information to form a complete profile of technical risk. In the report at the right, there are several low-complexity changed modules, and several high-complexity unchanged modules, but only one high-complexity changed module, which should be the main focus of review and testing.
  • Slide 17 shows the McCabe Test product. McCabe Test analyzes source code and tracks execution paths during testing using source-level instrumentation. It analyzes testing thoroughness at many levels of detail, from specific paths through individual source code modules up to program summary statistics. It also helps increase testing thoroughness by identifying paths that remain to be tested. By using the results of detailed structural code analysis to guide testing, McCabe Test focuses test effort on high-risk software, and uses precise, objective metrics to monitor testing progress. The result is that more errors can be found and fixed during the testing phase, before the software is released. Also, the test progress metrics allow you to estimate the time and effort necessary to finish testing, and to know when you can stop testing and ship with confidence. The top diagram is a structure chart in which the colors, percentages, and progress bars indicate testing progress, showing testing thoroughness in the context of program structure. The diagram at the lower left is the detailed control flow graph structure of a single module, with an untested path highlighted in blue. The report fragment to the lower right shows the detailed sequence of decision outcomes corresponding to that untested path, with source line numbers, control flow graph node numbers, and source code decision expressions given for reference.
  • Slice 19 shows the McCabe Slice add-on. McCabe Slice traces functionality to implementation by identifying and isolating the software that was executed during specific functional transactions. This helps isolate business rules for reengineering, and also helps isolate errors for debugging. The diagram shows the section of code that was executed in response to a particular functional scenario, and was not executed by any of a collection of other, related functional scenarios. This factors out all common processing and isolates the software that is unique to the target functionality. The left side of the diagram shows the unique control flow fragment highlighted in red on the control flow graph, and the right side shows the highlighted source code.
  • Slide 21 gives a summary of the McCabe IQ component products and add-ons. McCabe QA helps improve quality with precise, objective metrics. McCabe Data analyzes data impact to focus reengineering and testing efforts. McCabe compare helps eliminate or consistently maintain duplicate and near-duplicate code. McCabe Change helps focus on the impact of new and changed software. McCabe Test helps increase test effectiveness and focus testing efforts on high-risk software areas. McCabe TestCompress helps increase test efficiency by identifying small subsets of large production data sets that can be used as test data. McCabe Slice traces functionality to code and isolates the subset of code that implements particular functional transactions. And McCabe ReTest automates regression testing for non-GUI software.
  • Initiate Exercise I. After each concept, click on the Battlemap and illustrate the concept.
  • Initiate Exercise I. After each concept, click on the Battlemap and illustrate the concept.
  • Initiate Exercise I. After each concept, click on the Battlemap and illustrate the concept.
  • With the conclusions of the previous slide in mind, the testing challenges can be summarised as on this slide.
  • This slide is self explanatory!
  • This slide is simply to show the change in mode of the Battlemap from Standard to Coverage mode.
  • This slide is self-explanatory.
  • Testing results can be examined at both the structural (Battlemap) level, and at the flowgraph level.
  • Additional functional tests can often be derived simply by examination of the Battlemap in Coverage mode. A first step is to isolate the untested calling hierarchies, and determine whether the names of untested modules provide any insight into extra functional tests. Frequently the assistance of someone who knows the software is invaluable at this stage, although it is not necessary to gain value from the coverage Battlemap. Functional testers can often challenge developers as to why certain modules remain untested even after a full functional test run - this type of dialog between testers and developers can be healthy and useful - although caution should be used to keep functional tests “unbiased” towards the internals of the software. Further work can be undertaken on the partially tested modules if necessary, although this will usually require knowledge of the code to derive high level functional tests from untested paths in modules.
  • Following the derivation of the very large number of theoretical execution paths through software, the question is: How can a sensible set of theoretical tests be derived for a software component?
  • In order to examine code coverage we will use two simple code examples. The simple question is “Which of the two flowgraphs is more complex?” (Obviously the answer is B!) The next question is “Therefore which requires more testing?”. Again the answer is clearly B.
  • When examining code coverage, we can see that minimum 2 tests are required to cover all the code for example A. When looking at example B, we can also see that 2 similar tests will also cover all the code. In fact, if there were 1,000,000 decisions in sequence in example B, then 2 tests would still cover all the code. Clearly, code coverage has nothing to do with the amount of logic in a piece of code. In fact, Code coverage is just a mathematical side effect of executing code and is not a good mechanism for determining the testedness of code.
  • This is where McCabe found himself in 1977. He was asked by the NSA to look at large FORTRAN systems and determine how they should theoretically be tested. He took a simple flowgraph (as shown) and determined that there were (in this case) 2 tests required to cover all the code - which is a minimum level for testing. He then determined that one extra test (as shown) could be performed. This extra test will not execute any additional code (because it has already all been covered by the first 2 tests), but it will test the first decision against the second to ensure that they are not tied together (dependent). If the fourth (and final) test is performed in addition to the first three, there is no additional new result, apart from the fact that all possible execution paths have been executed. Given the previous determination that the total number of possible execution paths quickly grows very large, doing all possible tests is too many. This number of tests (in this case 3) can be measured for any flowgraph, and this number is called Complexity …..
  • When looking at the earlier types of testing which are more focussed on the code internals, cyclomatic path derivation can be a useful tool. Recalling from previous sections, complexity is equal to the number of linearly independent test paths through a module. An important result is that a full set of cyclomatic paths represents a “meaningful subset of all possible execution paths”, i.e. that any conceivable execution path can be obtained using a combination of the paths in a cyclomatic set. Additionally, because cyclomatic paths can be derived using McCabe Test, they can be used easily to derive additional tests that could be added to an initial set.
  • McCabe Test can produce the untested cyclomatic paths - that is the cyclomatic paths which remain to be executed. These are displayed as both a flowgraph and as a decision table. The mechanism to show these paths is by right clicking on a tested module, and then selecting ‘Test Paths’. The diagrams and reports which appear when ‘Test Paths’ is selected is configurable under ‘Preferences->Testing’ - the default is all cyclomatic paths...
  • Alternative reports available under ‘Test Paths’ include a report of untested branches. This report can be more useful for less rigorous testing approaches, and more quickly displays the untested sections of code and the associated decisions. Note the ‘# Exec’ column which describes the number of times each branch has been executed.
  • Need to introduce System Level Integration Paths. Assumption is that S0 and S1 have been covered in the metrics section. This system has S1 of 6 - meaning there are 6 end-to-end paths which would cover all the iv(G) paths in each module.
  • Need to describe how the graphical Integration paths work within McCabe Test.
  • Description of textual report.
  • High level verification is implicit in coverage analysis. Examination of coverage reports is a useful starting point for verifying that code has been sufficiently tested.
  • When looking at coverage reports, there are several different mechanisms for measuring coverage. McCabe Test supports/provides four major types.
  • Coverage Reports at System Level can be used to determine when testing is complete. In particular, using incremental test coverage analysis (examining the difference in coverage between test sets) provides a powerful mechanism for assessing a good time to stop testing. Caution should be exercised, however, if using this technique as coverage increments will not decay linearly. It is important to ensure that basic (level one) functional testing is complete before using coverage increments to determine an end-point for testing.
  • The assumption that all the code needs to be tested equally is an invalid one. All systems have critical or important code, with the remainder being less important either because it is run less often, or because it is already well tested or perhaps that the delivered functionality is less important. It is important to focus limited testing resources onto the critical code.
  • The technique is to import coverage across the whole analysis, then to bind the previously created “critical” class. The Combined Coverage report (right click on bound class) for this class can be used to assess the testedness of code elements at a high level. Then zoom into the class using the Battlemap would allow the contents of the class to be viewed. Remaining tests should be focussed on increasing coverage of the critical elements. Coverage in other areas of the system will be a side effect of trying to increase the coverage of the critical elements.
  • Finally, this approach of grouping elements by criticality can be refined, by creating several groups (classes) containing modules of increasing criticality. Examining the coverage of each of the criticality groups (classes) in turn will provide a detailed breakdown of the spread of coverage across the system. This technique is more useful for providing a risk assessment of the completeness of coverage. It is difficult to derive additional tests from several different groups of critical elements - experience shows that the best method is to stick to a single small group of critical modules to derive additional tests as described in the previous slide.
  • This slide should be self-explanatory. The reason this is introduced here is because it becomes relevant later - although load/save testdata is not specific to ‘When to Stop Testing’ only! Note: If two analyses contain common code (source files) it is possible to import coverage between programs. [the way to do this is to export a repos testdata item, then change the program name of the existing testdata item(so that it can be imported into the second program) - then reimport the original testdata item. The result is duplicate coverage in 2 programs. In the second program only identical code elements will have coverage imported for them]
  • Let’s assume that a software application has been analysed and coverage achieved. If changes are made to the code, then the parsers will detect a change in the date stamp of the source files, and will reparse the modified files. If any modules inside these changed files have been modified, then the parser will internally mark the module as being changed. If the existing coverage results are then imported into the changed project (or simply if you change to coverage mode), the modified code modules will have had all coverage removed from them. However, the unchanged code modules will retain the previous coverage. This can be used in order to focus testing on the changed code only.
  • Metrics Trending (VQT Only!) is useful to ensure improving test coverage between releases. A powerful technique is to utilise the change detection described in the previous slide, and then import previous test results. The objective at each release is to retest the application to achieve similar or better coverage results than the last version. In this manner the changes will have been tested, and an improving test schedule will be easy to maintain.
  • A description of McCabe Change. [a brazen attempt to get the client to buy more licenses!]
  • Self explanatory
  • Copyright McCabe

    1. 1. Management Overview 9861 Broken Land Parkway Fourth Floor Columbia, Maryland 21046 800-638-6316 [email_address] 1-800-634-0150
    2. 2. Agenda <ul><li>McCabe IQ Overview </li></ul><ul><li>Software Measurement Issues </li></ul><ul><li>McCabe Concepts </li></ul><ul><li>Software Quality Metrics </li></ul><ul><li>Software Testing </li></ul><ul><li>Questions and Answers </li></ul>
    3. 3. About McCabe & Associates 20 Years of Expertise Global Presence Analyzed Over 25 Billion Lines of Code
    4. 4. McCabe IQ process flow Analysis platform Target platform Source code Compile and run Execution log Effective Testing Quality Management Instrumented source code McCabe IQ
    5. 5. McCabe IQ and Configuration Management <ul><li>Merant PVCS </li></ul><ul><li>Rational ClearCase </li></ul><ul><li>CA Endevor </li></ul>McCabe IQ Execution Log Test Environment Effective Testing Quality Management <ul><li>Monitor quality as software changes </li></ul><ul><li>Manage test environment </li></ul>
    6. 6. McCabe IQ and Test Automation McCabe IQ <ul><li>Mercury Interactive: </li></ul><ul><li>TestDirector </li></ul><ul><li>WinRunner </li></ul>Source code Test executable Execution log Risk Management Test Management GUI Test Automation Effective Testing <ul><li>Risk-driven test management </li></ul><ul><li>Effective, automated testing </li></ul>Non-GUI Test Automation
    7. 7. McCabe IQ Components McCabe IQ Framework ( metrics, data, visualization, testing, API ) TESTING McCabe Test McCabe TestCompress McCabe Slice McCabe ReTest QUALITY ASSURANCE McCabe QA McCabe Data McCabe Compare McCabe Change Source Code Parsing Technology (C, C++, Java, Visual Basic, COBOL, Fortran, Ada)
    8. 8. McCabe QA <ul><li>McCabe QA measures software quality with industry-standard metrics </li></ul><ul><ul><li>Manage technical risk factors as software is developed and changed </li></ul></ul><ul><ul><li>Improve software quality using detailed reports and visualization </li></ul></ul><ul><ul><li>Shorten the time between releases </li></ul></ul><ul><ul><li>Develop contingency plans to address unavoidable risks </li></ul></ul>
    9. 9. McCabe Data <ul><li>McCabe Data pinpoints the impact of data variable modifications </li></ul><ul><ul><li>Identify usage of key data elements and data types </li></ul></ul><ul><ul><li>Relate data variable changes to impacted logic </li></ul></ul><ul><ul><li>Focus testing resources on the usage of selected data </li></ul></ul>
    10. 10. McCabe Compare <ul><li>McCabe Compare identifies reusable and redundant code </li></ul><ul><ul><li>Simplify maintenance and re-engineering of applications through the consolidation of similar code modules </li></ul></ul><ul><ul><li>Search for software defects in similar code modules, to make sure they’re fixed consistently throughout the software </li></ul></ul>
    11. 11. McCabe Change <ul><li>McCabe Change identifies new and changed modules </li></ul><ul><ul><li>Manage change with more precision than the file-level information from CM tools </li></ul></ul><ul><ul><li>Work with a complete technical risk profile </li></ul></ul><ul><ul><ul><li>Complex? </li></ul></ul></ul><ul><ul><ul><li>Poorly tested? </li></ul></ul></ul><ul><ul><ul><li>New or changed? </li></ul></ul></ul><ul><ul><li>Focus review and test efforts </li></ul></ul>
    12. 12. McCabe Test <ul><li>McCabe test maximizes testing effectiveness </li></ul><ul><ul><li>Focus testing on high-risk areas </li></ul></ul><ul><ul><li>Objectively measure testing effectiveness </li></ul></ul><ul><ul><li>Increase the failure detection rate during internal testing </li></ul></ul><ul><ul><li>Assess the time and resources needed to ensure a well-tested application </li></ul></ul><ul><ul><li>Know when to stop testing </li></ul></ul>
    13. 13. McCabe Slice <ul><li>McCabe Slice traces functionality to implementation </li></ul><ul><ul><li>Identifies code that implements specific functional transactions </li></ul></ul><ul><ul><li>Isolates code that is unique to the implementation of specific functional transactions </li></ul></ul><ul><ul><li>Helps extract business rules for application redesign </li></ul></ul>
    14. 14. McCabe IQ Components Summary <ul><li>McCabe QA : Improve quality with metrics </li></ul><ul><li>McCabe Data : Analyze data impact </li></ul><ul><li>McCabe Compare : Eliminate duplicate code </li></ul><ul><li>McCabe Change : Focus on changed software </li></ul><ul><li>McCabe Test : Increase test effectiveness </li></ul><ul><li>McCabe TestCompress : Increase test efficiency </li></ul><ul><li>McCabe Slice : Trace functionality to code </li></ul><ul><li>McCabe ReTest : Automate regression testing </li></ul>
    15. 15. Software Measurement Issues <ul><li>Risk management </li></ul><ul><li>Software metrics </li></ul><ul><li>Complexity metrics </li></ul><ul><li>Complexity metric evaluation </li></ul><ul><li>Benefits of complexity measurement </li></ul>
    16. 16. Software Risk Management <ul><li>Software risk falls into two major categories </li></ul><ul><ul><li>Non-technical risk: how important is the system? </li></ul></ul><ul><ul><ul><li>Usually known early </li></ul></ul></ul><ul><ul><li>Technical risk: how likely is the system to fail? </li></ul></ul><ul><ul><ul><li>Often known too late </li></ul></ul></ul><ul><li>Complexity analysis quantifies technical risk </li></ul><ul><ul><li>Helps quantify reliability and maintainability </li></ul></ul><ul><ul><ul><li>This helps with prioritization, resource allocation, contingency planning, etc. </li></ul></ul></ul><ul><ul><li>Guides testing </li></ul></ul><ul><ul><ul><li>Focuses effort to mitigate greatest risks </li></ul></ul></ul><ul><ul><ul><li>Helps deploy testing resources efficiently </li></ul></ul></ul>
    17. 17. Software Metrics Overview <ul><li>Metrics are quantitative measures </li></ul><ul><ul><li>Operational: cost, failure rate, change effort, … </li></ul></ul><ul><ul><li>Intrinsic: size, complexity, … </li></ul></ul><ul><li>Most operational metrics are known too late </li></ul><ul><ul><li>Cost, failure rate are only known after deployment </li></ul></ul><ul><ul><li>So, they aren’t suitable for risk management </li></ul></ul><ul><li>Complexity metrics are available immediately </li></ul><ul><ul><li>Complexity is calculated from source code </li></ul></ul><ul><li>Complexity predicts operational metrics </li></ul><ul><ul><li>Complexity correlates with defects, maintenance costs, ... </li></ul></ul>
    18. 18. Complexity Metric Evaluation <ul><li>Good complexity metrics have three properties </li></ul><ul><ul><li>Descriptive: objectively measure something </li></ul></ul><ul><ul><li>Predictive: correlate with something interesting </li></ul></ul><ul><ul><li>Prescriptive: guide risk reduction </li></ul></ul><ul><li>Consider lines of code </li></ul><ul><ul><li>Descriptive: yes, measures software size </li></ul></ul><ul><ul><li>Predictive, Prescriptive: no </li></ul></ul><ul><li>Consider cyclomatic complexity </li></ul><ul><ul><li>Descriptive: yes, measures decision logic </li></ul></ul><ul><ul><li>Predictive: yes, predicts errors and maintenance </li></ul></ul><ul><ul><li>Prescriptive: yes, guides testing and improvement </li></ul></ul>
    19. 19. Benefits of Complexity Measurement <ul><li>Complexity metrics are available from code </li></ul><ul><ul><li>They can even be estimated from a design </li></ul></ul><ul><li>They provide continuous feedback </li></ul><ul><ul><li>They can identify high-risk software as soon as it is written or changed </li></ul></ul><ul><li>They pinpoint areas of potential instability </li></ul><ul><ul><li>They can focus resources for reviews, testing, and code improvement </li></ul></ul><ul><li>They help predict eventual operational metrics </li></ul><ul><ul><li>Systems with similar complexity metric profiles tend to have similar test effort, cost, error frequency, ... </li></ul></ul>
    20. 20. McCabe Concepts <ul><li>Definition: In C and C++, a module is a function or subroutine with a single entry point and a single exit point. A module is represented by a rectangular box on the Battlemap. </li></ul>main function a function c function d printf Difficult to maintainable module Difficult to test module Well-designed, testable module Library module
    21. 21. Analyzing a Module <ul><li>For each module, an annotated source listing and flowgraph is generated. </li></ul><ul><li>Flowgraph - an architectural diagram of a software module’s logic. </li></ul>1 main() 2 { 3 printf(“example”); 4 if (y > 10) 5 b(); 6 else 7 c(); 8 printf(“end”); 9 } Stmt Code Number main Flowgraph node :statement or block of sequential statements condition end of condition edge : flow of control between nodes 1-3 4 5 7 8-9 Battlemap main b c printf
    22. 22. Flowgraph Notation (C) if (i) ; if (i) ; else ; if (i || j) ; do ; while (i); while (i) ; switch(i) { case 0: break; ... } if (i && j) ;
    23. 23. Flowgraph and Its Annotated Source Listing 0 1* 2 3 4* 5 6* 7 8 9 Origin information Node correspondence Metric information Decision construct
    24. 24. Low Complexity Software <ul><li>Reliable </li></ul><ul><ul><li>Simple logic </li></ul></ul><ul><ul><ul><li>Low cyclomatic complexity </li></ul></ul></ul><ul><ul><li>Not error-prone </li></ul></ul><ul><ul><li>Easy to test </li></ul></ul><ul><li>Maintainable </li></ul><ul><ul><li>Good structure </li></ul></ul><ul><ul><ul><li>Low essential complexity </li></ul></ul></ul><ul><ul><li>Easy to understand </li></ul></ul><ul><ul><li>Easy to modify </li></ul></ul>
    25. 25. Moderately Complex Software <ul><li>Unreliable </li></ul><ul><ul><li>Complicated logic </li></ul></ul><ul><ul><ul><li>High cyclomatic complexity </li></ul></ul></ul><ul><ul><li>Error-prone </li></ul></ul><ul><ul><li>Hard to test </li></ul></ul><ul><li>Maintainable </li></ul><ul><ul><li>Can be understood </li></ul></ul><ul><ul><li>Can be modified </li></ul></ul><ul><ul><li>Can be improved </li></ul></ul>
    26. 26. Highly Complex Software <ul><li>Unreliable </li></ul><ul><ul><li>Error prone </li></ul></ul><ul><ul><li>Very hard to test </li></ul></ul><ul><li>Unmaintainable </li></ul><ul><ul><li>Poor structure </li></ul></ul><ul><ul><ul><li>High essential complexity </li></ul></ul></ul><ul><ul><li>Hard to understand </li></ul></ul><ul><ul><li>Hard to modify </li></ul></ul><ul><ul><li>Hard to improve </li></ul></ul>
    27. 27. Would you buy a used car from this software? <ul><li>Problem: There are size and complexity boundaries beyond which software becomes hopeless </li></ul><ul><ul><li>Too error-prone to use </li></ul></ul><ul><ul><li>Too complex to fix </li></ul></ul><ul><ul><li>Too large to redevelop </li></ul></ul><ul><li>Solution: Control complexity during development and maintenance </li></ul><ul><ul><li>Stay away from the boundary </li></ul></ul>
    28. 28. Important Complexity Measures <ul><li>Cyclomatic complexity: v(G) </li></ul><ul><ul><li>Amount of decision logic </li></ul></ul><ul><li>Essential complexity: ev(G) </li></ul><ul><ul><li>Amount of poorly-structured logic </li></ul></ul><ul><li>Module design complexity: iv(G) </li></ul><ul><ul><li>Amount of logic involved with subroutine calls </li></ul></ul><ul><li>Data complexity: sdv </li></ul><ul><ul><li>Amount of logic involved with selected data references </li></ul></ul>
    29. 29. Cyclomatic Complexity <ul><li>The most famous complexity metric </li></ul><ul><li>Measures amount of decision logic </li></ul><ul><li>Identifies unreliable software, hard-to-test software </li></ul><ul><li>Related test thoroughness metric, actual complexity, measures testing progress </li></ul>
    30. 30. <ul><li>Cyclomatic complexity , v - A measure of the decision logic of a software module. </li></ul><ul><ul><li>Applies to decision logic embedded within written code. </li></ul></ul><ul><ul><li>Is derived from predicates in decision logic. </li></ul></ul><ul><ul><li>Is calculated for each module in the Battlemap. </li></ul></ul><ul><ul><li>Grows from 1 to high, finite number based on the amount of decision logic. </li></ul></ul><ul><ul><li>Is correlated to software quality and testing quantity; units with higher v , v>10 , are less reliable and require high levels of testing. </li></ul></ul>Cyclomatic Complexity
    31. 31. Cyclomatic Complexity 1 4 2 6 7 8 9 11 13 14 15 3 5 10 12 region method regions = 11 Beware of crossing lines R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 19 23 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 20 21 22 23 24 edges and node method e = 24, n = 15 v = 24 -15 +2 v = 11  =2  =1  =1  =2  =1  =1  =1  =1 predicate method v =  + 1 v = 11
    32. 32. Vital Signs and High v’s Risks of increasing v TIME 5 20 15 10 <ul><li>Higher risk of failures </li></ul><ul><li>Difficult to understand </li></ul><ul><li>Unpredictable expected results </li></ul><ul><li>Complicated test environments including more test drivers </li></ul><ul><li>Knowledge transfer constraints to new staff </li></ul>
    33. 33. Essential Complexity <ul><li>Measures amount of poorly-structured logic </li></ul><ul><li>Remove all well-structured logic, take cyclomatic complexity of what’s left </li></ul><ul><li>Identifies unmaintainable software </li></ul><ul><li>Pathological complexity metric is similar </li></ul><ul><ul><li>Identifies extremely unmaintainable software </li></ul></ul>
    34. 34. <ul><li>Essential complexity , ev - A measure of “structuredness” of decision logic of a software module. </li></ul><ul><ul><li>Applies to decision logic embedded within written code. </li></ul></ul><ul><ul><li>Is calculated for each module in the Battlemap. </li></ul></ul><ul><ul><li>Grows from 1 to v based on the amount of “unstructured” decision logic. </li></ul></ul><ul><ul><li>Is associated with the ability to modularize complex modules. </li></ul></ul><ul><ul><li>If ev increases , then the coder is not using structured programming constructs. </li></ul></ul>Essential Complexity
    35. 35. Essential Complexity - Unstructured Logic Branching out of a loop Branching in to a loop Branching into a decision Branching out of a decision
    36. 36. Essential Complexity - Flowgraph Reduction <ul><li>Essential complexity, ev, is calculated by reducing the module flowgraph. Reduction is completed by removing decisions that conform to single-entry, single-exit constructs. </li></ul>Cyclomatic Complexity = 4 Essential Complexity = 1
    37. 37. Essential Complexity <ul><li>Flowgraph and reduced flowgraph after structured constructs have been removed, revealing decisions that are unstructured. </li></ul>v = 5 Reduced flowgraph v = 3 Therefore ev of the original flowgraph = 3 Superimposed essential flowgraph
    38. 38. Essential Complexity <ul><li>Essential complexity helps detect unstructured code. </li></ul>Good designs Can quickly deteriorate! v = 10 ev = 1 v = 11 ev = 10
    39. 39. Vital Signs and High ev’s Risks of increasing ev TIME 1 10 6 3 <ul><li>Intricate logic </li></ul><ul><li>Conflicting decisions </li></ul><ul><li>Unrealizable test paths </li></ul><ul><li>Constraints for architectural improvement </li></ul><ul><li>Difficult knowledge transfer to new staff </li></ul>
    40. 40. How to Manage and Reduce v and ev TIME Decreasing and managing v and ev 1 20 15 10 <ul><li>Emphasis on design architecture and methodology </li></ul><ul><li>Development and coding standards </li></ul><ul><li>QA procedures and reviews </li></ul><ul><li>Peer evaluations </li></ul><ul><li>Automated tools </li></ul><ul><li>Application portfolio management </li></ul><ul><li>Modularization </li></ul>
    41. 41. Module Design Complexity How Much Supervising Is Done?
    42. 42. Module design complexity <ul><li>Measures amount of decision logic involved with subroutine calls </li></ul><ul><li>Identifies “managerial” modules </li></ul><ul><li>Indicates design reliability, integration testability </li></ul><ul><li>Related test thoroughness metric, tested design complexity, measures integration testing progress </li></ul>
    43. 43. <ul><li>Module design complexity , iv - A measure of the decision logic that controls calls to subroutines. </li></ul><ul><ul><li>Applies to decision logic embedded within written code. </li></ul></ul><ul><ul><li>Is derived from predicates in decision logic associated with calls. </li></ul></ul><ul><ul><li>Is calculated for each module in the Battlemap. </li></ul></ul><ul><ul><li>Grows from 1 to v based on the complexity of calling subroutines. </li></ul></ul><ul><ul><li>Is related to the degree of &quot;integratedness&quot; between a calling module and its called modules. </li></ul></ul>Module Design Complexity
    44. 44. Module Design Complexity <ul><li>Module design complexity, iv, is calculated by reducing the module flowgraph. Reduction is completed by removing decisions and nodes that do not impact the calling control over a module’s immediate subordinates. </li></ul>
    45. 45. Module Design Complexity <ul><li>Example: </li></ul>main proge progd iv = 3 Therefore, iv of the original flowgraph = 3 Reduced Flowgraph v = 3 proge() progd() main v = 5 proge() progd() main() { if (a == b) progd(); if (m == n) proge(); switch(expression) { case value_1: statement1; break; case value_2: statement2; break; case value_3: statement3; } } do not impact calls
    46. 46. Data complexity <ul><li>Actually, a family of metrics </li></ul><ul><ul><li>Global data complexity (global and parameter), specified data complexity, date complexity </li></ul></ul><ul><li>Measures amount of decision logic involved with selected data references </li></ul><ul><li>Indicates data impact, data testability </li></ul><ul><li>Related test thoroughness metric, tested data complexity, measures data testing progress </li></ul>
    47. 47. Data complexity calculation Paths Conditions Pb : 1-2-3-4-9-3-4-9-12 C1 = T, C2 = T, C2 = F P2 : 1-2-12 C1 = F P3 : 1-2-3-4-9-12 C1 = T, C2 = F v = 6 M : data complexity = 3 M : => Data A Data A C1 C2 C1 C2 C3 C4 C5 1 2 5 7 8 3 10 11 12 4* 9 6 3 2 1 12 4* 9
    48. 48. Module Metrics Report v, number of unit test paths for a module Total number of test paths for all modules iv, number of integration tests for a module Average number of test paths for each module
    49. 49. Common Testing Challenges <ul><li>Deriving Tests </li></ul><ul><ul><li>Creating a “Good” Set of Tests </li></ul></ul><ul><li>Verifying Tests </li></ul><ul><ul><li>Verifying that Enough Testing was Performed </li></ul></ul><ul><ul><li>Providing Evidence that Testing was Good Enough </li></ul></ul><ul><li>When to Stop Testing </li></ul><ul><li>Prioritizing Tests </li></ul><ul><ul><li>Ensuring that Critical or Modified Code is Tested First </li></ul></ul><ul><li>Reducing Test Duplication </li></ul><ul><ul><li>Identifying Similar Tests That Add Little Value & Removing Them </li></ul></ul>
    50. 50. An Improved Testing Process Requirements Static Identification of Test Paths Implementation Black Box White Box Sub-System or System Analysis Test Scenarios
    51. 51. What is McCabe Test? Source Code Parsing The McCabe Tools Execute Code Trace Info Build Executable Import Requirements Tracing Test Coverage Untested Paths Database Instrumented Source Code Export
    52. 52. Coverage Mode <ul><li>Color Scheme Represents Coverage </li></ul>No Trace File Imported
    53. 53. Coverage Results <ul><li>Colors Show “Testedness” </li></ul><ul><li>Lines Show Execution Between Modules </li></ul><ul><li>Color Scheme: </li></ul><ul><ul><li>Branches </li></ul></ul><ul><ul><li>Paths </li></ul></ul><ul><ul><li>Lines of Code </li></ul></ul>Trace File Imported Partially Tested Tested Untested 3 67% My_Func1ion
    54. 54. Coverage Results at Unit Level Module _ >Slice
    55. 55. Deriving Functional Tests <ul><li>Examine Partially Tested Modules </li></ul><ul><li>Module Names Provide Insight into Additional Tests </li></ul><ul><li>Visualize Untested Modules </li></ul>Module Name ‘search’
    56. 56. Deriving Tests at the Unit Level 18 times <ul><li>Too Many Theoretical Tests! </li></ul><ul><li>What is the Minimum Number of Tests? </li></ul><ul><li>What is a “Good” Number of Tests? </li></ul>Statistical Paths = 10 18 0 10 18 Minimum yet effective testing? Too Few Tests Too Many Tests
    57. 57. Code Coverage Example ‘A’ Example ‘B’ Which Function Is More Complex?
    58. 58. Using Code Coverage Example ‘A’ Example ‘B’ 2 Tests Required Code Coverage Is Not Proportional to Complexity 2 Tests Required
    59. 59. McCabe's Cyclomatic Complexity McCabe's Cyclomatic Complexity v(G) Number of Linearly Independent Paths One Additional Path Required to Determine the Independence of the 2 Decisions
    60. 60. Deriving Tests at the Unit Level Complexity = 10 <ul><li>Minimum 10 Tests Will: </li></ul><ul><li>Ensure Code Coverage </li></ul><ul><li>Test Independence of Decisions </li></ul>
    61. 61. Unit Level Test Paths - Baseline Method <ul><li>The baseline method is a technique used to locate distinct paths within a flowgraph. The size of the basis set is equal to v(G). </li></ul>C E B G A F D M=N O=P X=Y S=T Basis set of paths Path conditions P1: ABCBDEF Pb: M=N,O=P,S=T,O not = P P2: AGDEF P2: M not = N, X=Y P3: ABDEF P3: M=N,O not = P P4: ABCF P4: M=N,O=P,S not = T P5: AGEF P5: M not = N,X not = Y v = 5
    62. 62. Structured Testing Coverage E F G H A B C D M N O P I J K L R1 R2 R3 R4 R5 1. Generates independent tests Basis set P1: ACDEGHIKLMOP P2: ABD… P3: ACDEFH… P4: ACDEGHIJL… P5: ACDEGHIKLMNP 2. Code coverage - frequency of execution Node A B C D E F G H I J K L M N O P Count 5 1 4 5 5 1 4 5 5 1 4 5 5 1 4 5
    63. 63. Other Baselines - Different Coverage E F G H A B C D M N O P I J K L R1 R2 R3 R4 R5 Previous code coverage - frequency of execution Node A B C D E F G H I J K L M N O P Count 5 1 4 5 5 1 4 5 5 1 4 5 5 1 4 5 Same number of tests; which coverage is more effective? 1. Generates independent tests Basis set P1: ABDEFHIJLMNP P2: ACD… P3: ABDEGH… P4: ABDEGHIKL… P5: ABDEGHIKLMOP 2. Code coverage - frequency of execution Node A B C D E F G H I J K L M N O P Count 5 4 1 5 5 4 1 5 5 4 1 5 5 4 1 5
    64. 64. Untested Paths at Unit Level <ul><li>Cyclomatic Test Paths </li></ul><ul><ul><li>M odule _ > T est Paths </li></ul></ul><ul><ul><li>Complete Test Paths by Default </li></ul></ul><ul><li>Configurable Reports </li></ul><ul><ul><li>Preferences _ >Testing </li></ul></ul><ul><ul><li>Modify List of Graph/Test Path Flowgraphs </li></ul></ul>Module _ >Test Paths Remaining Untested Test Paths
    65. 65. Untested Branches at Unit Level Preferences _ >Testing (Add ‘Tested Branches’ Flowgraph to List) Module _ >Test Paths Number of Executions for Decisions Untested Branches
    66. 66. Untested Paths at Higher Level <ul><li>System Level Integration Paths </li></ul><ul><ul><li>Based on S 1 </li></ul></ul><ul><ul><li>End-to-End Execution </li></ul></ul><ul><ul><li>Includes All iv(G) Paths </li></ul></ul>S 1 = 6
    67. 67. Untested Paths at Higher Level <ul><li>System Level Integration Paths </li></ul><ul><ul><li>Displayed Graphically </li></ul></ul><ul><ul><li>Textual Report </li></ul></ul><ul><ul><li>Theoretical Execution Paths </li></ul></ul><ul><ul><li>Show Only Untested Paths </li></ul></ul>S 1 = 6
    68. 68. Untested Paths at Higher Level <ul><li>Textual Report of End-to-End Decisions </li></ul>Decision Values with Line/Node # Module Calling List
    69. 69. Verifying Tests <ul><li>Use Coverage to Verify Tests </li></ul><ul><ul><li>Store Coverage Results in Repository </li></ul></ul><ul><li>Use Execution Flowgraphs to Verify Tests </li></ul>
    70. 70. Verifying Tests Using Coverage <ul><li>Four Major Coverage Techniques: </li></ul><ul><ul><li>Code Coverage </li></ul></ul><ul><ul><li>Branch Coverage </li></ul></ul><ul><ul><li>Path Coverage </li></ul></ul><ul><ul><li>Boolean Coverage (MC/DC) </li></ul></ul>67% 23% 0% 35% 100%
    71. 71. When to Stop Testing <ul><li>Coverage to Assess Testing Completeness </li></ul><ul><ul><li>Branch Coverage Reports </li></ul></ul><ul><li>Coverage Increments </li></ul><ul><ul><li>How Much New Coverage for Each New Test of Tests? </li></ul></ul>
    72. 72. When to Stop Testing <ul><li>Is All of the System Equally Important? </li></ul><ul><li>Is All Code in An Application Used Equally? </li></ul><ul><ul><li>10% of Code Used 90% of Time </li></ul></ul><ul><ul><li>Remaining 90% Only Used 10% of Time </li></ul></ul><ul><li>Where Do We Need to Test Most? </li></ul>
    73. 73. When to Stop Testing / Prioritizing Tests <ul><li>Locate “Critical” Code </li></ul><ul><ul><li>Important Functions </li></ul></ul><ul><ul><li>Modified Functions </li></ul></ul><ul><ul><li>Problem Functions </li></ul></ul><ul><li>Mark Modules </li></ul><ul><ul><li>Create New “Critical” Group </li></ul></ul><ul><li>Import Coverage </li></ul><ul><li>Assess Coverage for “Critical” Code </li></ul><ul><ul><li>Coverage Report for “Critical” Group </li></ul></ul><ul><ul><li>Examine Untested Branches </li></ul></ul>32 67% Runproc 39 52% Search 56 My_Func1ion
    74. 74. Criticality Coverage <ul><li>Optionally Use Several “Critical” Groups </li></ul><ul><ul><li>Increasing Levels </li></ul></ul><ul><ul><li>Determine Coverage for Each Group </li></ul></ul><ul><ul><li>Focus Testing Effort on Critical Code </li></ul></ul>Coverage Insufficient Testing? Criticality Useful as a Management Technique 30% 25% 90% 70% 50%
    75. 75. When to Stop Testing <ul><li>Store Coverage in Repository </li></ul><ul><ul><li>With Name & Author </li></ul></ul><ul><li>Load Coverage </li></ul><ul><ul><li>Multiple Selections </li></ul></ul><ul><ul><li>Share Between Users </li></ul></ul><ul><ul><li>Import Between Analyses with Common Code </li></ul></ul>T esting _ > L oad/Save Testing Data
    76. 76. Testing the Changes Version 1.0 - Coverage Results Version 1.1 - Previous Coverage Results Imported Into New Analysis Changed Code <ul><li>Import Previous Coverage Results Into New Analysis: </li></ul><ul><ul><li>Parser Detects Changed Code </li></ul></ul><ul><ul><li>Coverage Removed for Modified or New Code </li></ul></ul>
    77. 77. Testing the Changes <ul><li>Store Coverage for Versions </li></ul><ul><ul><li>Use Metrics Trending to Show Increments </li></ul></ul><ul><ul><li>Objective is to Increase Coverage between Releases </li></ul></ul>Incremental Coverage
    78. 78. McCabe Change <ul><li>Marking Changed Code </li></ul><ul><ul><li>Reports Showing Change Status </li></ul></ul><ul><ul><li>Coverage Reports for Changed Modules </li></ul></ul><ul><li>Configurable Change Detection </li></ul><ul><ul><li>Standard Metrics </li></ul></ul><ul><ul><li>“ String Comparison” </li></ul></ul>Changed Code
    79. 79. Manipulating Coverage <ul><li>Addition/Subtraction of slices </li></ul><ul><ul><li>The technique: </li></ul></ul>
    80. 80. Slice Manipulation <ul><li>Slice Operations </li></ul><ul><li>Manipulate Slices Using Set Theory </li></ul><ul><li>Export Slice to File </li></ul><ul><ul><li>List of Executed Lines </li></ul></ul><ul><li>Must be in Slice Mode </li></ul>
    81. 81. Review <ul><li>McCabe IQ Products </li></ul><ul><li>Metrics </li></ul><ul><ul><li>cyclomatic complexity, v </li></ul></ul><ul><ul><li>essential complexity, ev </li></ul></ul><ul><ul><li>module design complexity, iv </li></ul></ul><ul><li>Testing </li></ul><ul><ul><li>Deriving Tests </li></ul></ul><ul><ul><li>Verifying Tests </li></ul></ul><ul><ul><li>Prioritizing Tests </li></ul></ul><ul><ul><li>When is testing complete? </li></ul></ul><ul><li>Managing Change </li></ul>