Your SlideShare is downloading. ×
ISTQB Technical Test Analyst 2012 Training - Structure-Based Testing
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

ISTQB Technical Test Analyst 2012 Training - Structure-Based Testing

5,973
views

Published on

This is a free module from my course ISTQB CTAL Technical Test Analyst revised to 2012 syllabus. If you need full training feel free to contact me by email (amraldo@hotmail.com) or by mobile …

This is a free module from my course ISTQB CTAL Technical Test Analyst revised to 2012 syllabus. If you need full training feel free to contact me by email (amraldo@hotmail.com) or by mobile (+201223600207).

Published in: Technology, Business

6 Comments
17 Likes
Statistics
Notes
No Downloads
Views
Total Views
5,973
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
6
Likes
17
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • A decision is made by a complex expression that eventually evaluates to TRUE or FALSE.An atomic condition is defined as “the simplest form of code that can result in a TRUE or FALSE outcome.”
  • The first 2 techniques are covered in the ISTQB CTFL. Starting from the 2nd technique till the 6th technique are based on decision predicates. They derive test cases based on control flow.Except the first 3 techniques, all techniques are rigorous (أدق). Starting from technique 3 to techniques 6, thoroughness (شمولية) of test increases. They require more tests to be defined in order to achieve their intended coverage and to find more subtle instances of this type of defect.
  • x, D, F, A, and B would each need a test case where it evaluated to TRUE and one where it evaluated to FALSE.(C==D) needs two test cases.(A>B) needs two test cases.(X+Y==-1) needs two test cases.(!(D)==TRUE) needs two test cases.
  • You can notice in the table that 100% decision coverage is achieved, 100% condition coverage is achieved, and that the independent effect of A, B, and C on the final decision outcome is shown.
  • 4 test cases are needed: TTT, FTT, TFT, TTF.
  • 4 test cases are needed: FFF, TFF, TFF, FFT.
  • Case 3 is an example of a grouped functionality.Case 4 is an example of a more complex construct where feedback is applied. Case 5 is an example of a bit-wise operation.
  • There may be a very serious bug issue with short circuiting. If an atomic condition is supplied by a called function and it is short-circuited out of evaluation, then any side effects that might have occurred through its execution are lost.
  • Exercise 1: FXXXX, TFXXX, TTFFX, TTFTF, TTFTT, and TTTXXExercise 2: FFXXX, FTFFX, FTFTF, FTFTT, FTTXF, FTTXT, TXFFX, TXFTF, TXFTT, TXTXF, and TXTXT
  • Believe it or not, they are infinite.
  • Is it achievable all the time?! It is doubtful.
  • The code should be modeled as a CFG to deduce the following test cases: A = 11, B = 12, C = 13 A = 11, B = 0, C = 0A = 0, B = 12, C = 13A = 0, B = 0, C =0
  • LCSAJ 1 can only be executed if forced by a debugger.LCSAJ 2 will execute once at the start of the loop. It is concerned with the first iteration of the loop only. LCSAJ 3 can only be executed if forced by a debugger.LCSAJ 4 will execute only once at the end of the loop. LCSAJ 5 will execute many times in the loop as long as totals[val] != MAXCOUNT at the start of the iteration.LCSAJ 6 will execute if totals[val] == MAXCOUNT.LCSAJ 7 will execute 749 time.LCSAJ 8 will occur once after the loop ends.
  • Connected components is 1 in our examples. How it should be calculated is beyond our scope.
  • Create a CFG, calculate C (4), and finally find test cases.
  • The higher the level, the higher the confidence in the software
  • Transcript

    • 1. ISTQB Advanced Level 2012 – Technical Test Analyst
    • 2. Objectives  Build on ISTQB CTFL in the area of technical test analysis  Prepare for the ISTQB CTAL – Technical Test Analyst exam 31/08/20122 ISTQB Advanced Level 2012 - Technical Test Analyst
    • 3. Prerequisites  ISTQB CTFL or equivalent  Practical experience in SW testing 31/08/20123 ISTQB Advanced Level 2012 - Technical Test Analyst
    • 4. Notes  Ask any time.  Turn your cell silent. 31/08/20124 ISTQB Advanced Level 2012 - Technical Test Analyst
    • 5. References  ISTQB CTAL – TTA syllabus version 2012 31/08/20125 ISTQB Advanced Level 2012 - Technical Test Analyst
    • 6. Outline  The Technical Test Analyst's Tasks in Risk-Based Testing  Structure-Based Testing  Analytical Techniques  Quality Characteristics for Technical Testing  Reviews  Test Tools and Automation 31/08/20126 ISTQB Advanced Level 2012 - Technical Test Analyst
    • 7. Outline  The Technical Test Analyst's Tasks in Risk-Based Testing  Structure-Based Testing  Analytical Techniques  Quality Characteristics for Technical Testing  Reviews  Test Tools and Automation 31/08/20127 ISTQB Advanced Level 2012 - Technical Test Analyst
    • 8. Structure-Based Testing  Introduction  Condition Testing  Decision Condition Testing  Modified Condition/Decision Coverage (MC/DC) Testing  Multiple Condition Testing  Path Testing  API Testing  Selecting a Structure-Based Technique 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 8
    • 9. Structure-Based Testing  Introduction  Condition Testing  Decision Condition Testing  Modified Condition/Decision Coverage (MC/DC) Testing  Multiple Condition Testing  Path Testing  API Testing  Selecting a Structure-Based Technique 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 9
    • 10. What is Structure-Based Testing?  AKA white box or code-based testing  Uses the code, the data and the architecture and/or system flow as the basis for test design  Techniques used in structure- based testing:  Derive test cases systematically  Focus on a particular aspect of the structure 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 10
    • 11. Examples of Structure-Based Testing Test Basis  Component level  Code structure  Statements  Decisions  Conditions  Integration level  Call tree  System level  Menus structure  Business process  Web page structure 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 11
    • 12. Why Structure-Based Testing?  Structure-based testing provides a coverage criteria.  A criteria can be measured and associated with an objective.  Achieving full coverage != Complete test  It only means technique being used no longer suggests any useful tests for the structure under consideration.  Complete test is project context dependent. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 12
    • 13. When Structure-Based Testing should be Used?  Specification-based testing will never test all the code.  Structure-based testing should be used on top of specification based testing to measure the coverage and hence add extra tests to increase the coverage. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 13
    • 14. Atomic Conditions vs. Decisions  A is an atomic condition.  B is a atomic condition.  C is a atomic condition.  (A &&B) || C is a decision.  Logical combination of atomic conditions makes a decision predicate. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 14 if ((A && B) || C){ ... }
    • 15. Examples of Structure-Based Testing Techniques 1. Statement testing 2. Decision/Branch testing 3. Condition testing 4. Decision Condition testing 5. Modified Condition/Decision Coverage (MC/DC) testing 6. Multiple Condition testing 7. Path testing 8. API testing 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 15
    • 16. Structure-Based Testing  Introduction  Condition Testing  Decision Condition Testing  Modified Condition/Decision Coverage (MC/DC) Testing  Multiple Condition Testing  Path Testing  API Testing  Selecting a Structure-Based Technique 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 16
    • 17. What is Condition Testing?  Considers how decisions are made rather than decision predicates  Answers the question “Could there be bugs hiding in the condition itself?”  Makes sure that each atomic condition is tested both ways, TRUE and FALSE 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 17
    • 18. Condition Testing Coverage  % of conditions possibilities exercised by a test suite  Condition coverage % = (# of conditions possibilities executed/total # of conditions possibilities ) * 100  Example:  Program has 100 conditions, thus 200 conditions possibilities  Tests exercise 112 conditions possibilities  Condition coverage = 56% 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 18
    • 19. How is Condition Testing Done? 1. Identify atomic conditions. 2. Find the minimal test cases to achieve 100% condition coverage for each atomic condition. 3. Combine test cases of the atomic conditions to find the minimal test cases to test the decision. Make sure all test cases identified in step 2 are executed. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 19
    • 20. Exercise: Identify Min # of Test Cases to Achieve 100% Condition Coverage 1. X 2. D && F 3. (A || B) && (C == D) 4. (A > B) || ( X + Y == - 1) && (! (D) == TRUE) 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 20
    • 21. Condition Testing Applicability  Probably interesting only in the abstract  Understanding it is necessary to achieving greater levels of coverage that build upon it. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 21
    • 22. Condition Testing Limitations  Achieving 100% condition coverage does not necessitate 100% decision coverage.  100% condition coverage necessitates 100% decision coverage iff decisions are consisted of single atomic expressions. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 22 if (A && B){ ... } else { ... } if (X){ ... } else { ... } TC1: A = T and B = F TC2: A = F and B = T TC1: X = T TC2: X = F
    • 23. Structure-Based Testing  Introduction  Condition Testing  Decision Condition Testing  Modified Condition/Decision Coverage (MC/DC) Testing  Multiple Condition Testing  Path Testing  API Testing  Selecting a Structure-Based Technique 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 23
    • 24. What is Decision Condition Testing?  Testing must achieve condition coverage and decision coverage.  100% decision condition coverage guarantees 100% statement coverage, 100% decision coverage, and 100% condition coverage.  A thoughtful choice of test data values for atomic conditions may result in achieving this level of coverage without adding extra test cases beyond that needed to achieve condition coverage. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 24 if (A && B){ ... } else { ... } TC1: A = T and B = T TC2: A = F and B = F
    • 25. Decision Condition Testing Coverage  100% decision condition coverage = 100% decision coverage + 100% condition coverage 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 25
    • 26. How is Decision Condition Testing Done? 1. Identify atomic conditions. 2. Find the minimal test cases to achieve 100% condition coverage for each atomic condition. 3. Combine test cases of the atomic conditions to find the minimal test cases to test the decision keeping in mind 100% decision coverage is achieved. Make sure all test cases identified in step 2 are executed. 4. Add/Modify any test cases needed if 100% decision coverage is not achieved while keeping the 100% condition coverage 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 26
    • 27. Exercise: Identify Min # of Test Cases to Achieve 100% Decision Condition Coverage 1. X 2. D && F 3. (A || B) && (C == D) 4. (A > B) || ( X + Y == - 1) && (! (D) == TRUE) 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 27
    • 28. Decision Condition Testing Applicability  Code is important but not critical.  What is important and what is critical?!!! 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 28
    • 29. Decision Condition Testing Limitations  May require more test cases compared to previous techniques  May consume more time compared to previous techniques 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 29
    • 30. Structure-Based Testing  Introduction  Condition Testing  Decision Condition Testing  Modified Condition/Decision Coverage (MC/DC) Testing  Multiple Condition Testing  Path Testing  API Testing  Selecting a Structure-Based Technique 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 30
    • 31. What is MC/DC Testing?  Created by Boeing for use when specific languages were to be used in safety-critical programming  Provides a stronger level of control flow coverage  Guarantees 100% decision condition coverage and adds to it 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 31
    • 32. MC/DC Coverage 1. At least one test where the decision outcome would change if the atomic condition X were TRUE 2. At least one test where the decision outcome would change if the atomic condition X were FALSE 3. Each different atomic condition has tests that meet requirements 1 and 2. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 32
    • 33. MC/DC Coverage in Other Words 1. 100% condition coverage 2. 100% decision coverage 3. Each atomic condition must affect the outcome decision independently while the other atomic conditions are held fixed. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 33
    • 34. MC/DC Testing by Example  (A || B) && C 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 34 A B C (A or B) and C Test 1 T F T T Test 2 F T T T Test 3 F F T F Test 4 T F F F
    • 35. C/C’s of MC/DC Testing  Number of test cases for a decision is unique.  There may be more than a test group fulfilling the MC/DC coverage.  Causes a lot of testing but not an exhaustive testing  Assuming N unique atomic conditions, MC/DC can usually be achieved in N+1 unique test cases 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 35
    • 36. Logical Expressions and their Truth Tables 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 36
    • 37. MC/DC Testing for an And Gate  N inputs and gate requires n+1 test cases.  All inputs are true.  Each input is set exclusively to false.  Exercise: Find test cases for (A && B && C) 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 37
    • 38. MC/DC Testing for an Or Gate  N inputs or gate requires n+1 test cases.  All inputs are false.  Each input is set exclusively to true.  Exercise: Find test cases for (A || B || C) 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 38
    • 39. MC/DC Testing for an Xor Gate  No rule  For a 2-input xor gate, any 3 out of the 4 possible combinations can achieve the MC/DC coverage.  Only 1 combination can detect an OR mistyped as XOR. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 39
    • 40. MC/DC Testing for a Not Gate  Only 2 test cases  31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 40
    • 41. MC/DC Testing for Comparators  <, >, <=, >=, ==, !=  Needs 3 test cases:  Output is false and inputs are close to the boundary.  Output is true and inputs are close the boundary.  The inputs are at the boundary.  MC/DC + discovering typing mistakes  Exercise: Find test cases for x >= 25000 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 41
    • 42. MC/DC Testing for an If-Else Statement  Needs the following test cases:  Inputs to make decision true.  Inputs to force decision false.  Inputs to exercise any of the previous items in the decision.  Exercise: if(Z) statement 1; else statement 2; 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 42
    • 43. MC/DC Testing for a While Loop  Needs the following test cases:  Inputs to make decision true.  Inputs to force decision false.  Inputs to exercise any of the previous items in the decision.  What about the for loop?  Exercise: while(Z) statement 1; 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 43
    • 44. MC/DC Testing for a Do-While Loop  Needs the following test cases:  Inputs to make decision true.  Inputs to force decision false  Inputs to exercise any of the previous items in the decision.  Exercise: do statement 1; while(Z); 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 44
    • 45. How is MC/DC Testing Done? 1. Represent source code graphically in terms of logic gates. 2. Identify test inputs and output relation. 1. Draw truth table 3. Identify observation point and control input(s), then eliminate masked test cases. 4. Determine MC/DC for each logic gate starting from the input moving towards the output.  Exercise: Determine MC/DC for (B && C) || D. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 45
    • 46. 1- Representing the Code Graphically 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 46
    • 47. 2- I/O Relation ID B C D A 1 F F F F 2 F F T T 3 F T F F 4 F T T T 5 T F F F 6 T F T T 7 T T F T 8 T T T T 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 47
    • 48. 3- Observation Point, Control Input, Masked Test Case  An observation point is the point at which we observe the output of concern.  A control input is when its value is known, the value at the observation point can be calculated regardless of the other inputs.  Masked test cases are test cases where the we care only for the control input and others inputs are do not care. They are in the essence equivalent. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 48
    • 49. 3- Observation Point, Control Input, Masked Test Case cont’d  Masking tests rules:  W and true = W  W or false = W  W xor false = W  W xor true = !W 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 49
    • 50. 3- Observation Point, Control Input, Masked Test Case cont’d ID B C D A 1 F F F F 2 F F T T 3 F T F F 4 F T T T 5 T F F F 6 T F T T 7 T T F T 8 T T T T 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 50 Observation Point Control Input when = T Masked Test Cases
    • 51. 4- Determining MC/DC for the Gates  By combining the results in the table, we can use any of the following tests :  2, 3, 5, 7  3, 4, 5, 7  3, 5, 6, 7 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 51 Valid Inputs Missing Test Cases And gate TT: row 7 TF: row 5 FT: row 3 Or gate FF: rows 1, 3, or 5 TF: row 7 FT: rows 2, 4, or 6
    • 52. Exercise: Identify Min # of Test Cases to Achieve 100% MC/DC Coverage 1. Z = (A || B) && (C || D) 2. Z = (A && !B) || (C || D) 3. A = (B || C) && D; E = (X && Y) || C; Z = A&& E 4. O = (O || I) && !R 5. Z = X & Y 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 52
    • 53. MC/DC Testing Applicability  Aerospace SW industry  Safety-critical systems  In general, it is used when dealing with safety critical SW where any failure may cause a catastrophe. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 53
    • 54. MC/DC Testing Limitations  Short-circuiting may affect the ability to attain MC/DC coverage since some required tests may not be achievable.  Achieving MC/DC coverage may be complicated when the atomic conditions are coupled in the expression.  Depending on the decision statement in the code, it may not be possible to vary the value of the coupled term such that it alone causes the decision outcome to change. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 54
    • 55. Short Circuiting  In some programming languages or some compiler options, the right hand operand is evaluated iff it is needed to determine the expression value.  Q: If func() is never evaluated, the question then becomes, can we still achieve MC/DC coverage?  A: May be as some tests may be not achievable at all. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 55 if (A || func()){ ... }
    • 56. Short Circuiting cont’d  Many organizations evaluated it in different ways.  If your project is subject to regulatory statutes, consult how that regulatory body wants you to deal with the short circuiting.  For example, NASA claims that short circuit logic can be dealt with by applying the masking MC/DC approach explained earlier, where a case-by- case basis analysis of the decision is done. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 56
    • 57. Coupled Atomic Conditions  Multiple occurrences of a condition in an expression is called coupling.  A and !A can not be varied independently as required by MC/DC.  If your project is subject to regulatory statutes, consult how that regulatory body wants you to deal with the short circuiting.  There are 2 approaches for facing such situation: 1. Unique Cause MC/DC approach, where the term condition in the definition of MC/DC is deemed to mean uncoupled condition. 2. Masking MC/DC approach explained earlier, where a case-by-case basis analysis of the decision is done. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 57 if (A || (! A && B)){ ... }
    • 58. Structure-Based Testing  Introduction  Condition Testing  Decision Condition Testing  Modified Condition/Decision Coverage (MC/DC) Testing  Multiple Condition Testing  Path Testing  API Testing  Selecting a Structure-Based Technique 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 58
    • 59. What is Multiple Condition Testing?  Test every possible combination of atomic conditions in a decision!  Truly exhaustive testing 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 59
    • 60. How is Multiple Condition Testing Done?  Create a truth table with every combination.  Does not take much analysis  # of test cases = 2# of atomic conditions 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 60
    • 61. Multiple Condition Testing by Example  (A || B) && C 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 61 A B C (A or B) and C Test 1 F F F F Test 2 F F T F Test 3 F T F F Test 4 F T T T Test 5 T F F F Test 6 T F T T Test 7 T T F F Test 8 T T T T
    • 62. Is it Really a Truth Table?  Short circuiting and coupling are still applicable.  If the language uses short-circuiting, the number of actual test cases will often be reduced, depending on the order and grouping of logical operations that are performed on the atomic conditions. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 62
    • 63. Exercise: Identify Min # of Test Cases to Achieve 100% Multiple Condition Coverage  Assume we have short circuit logic compiler 1. A && B && (C || (D && E)) 2. ((A || B) && (C || D)) && E 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 63
    • 64. Multiple Condition Testing Applicability  Testing embedded software which was expected to run reliably without crashing for long periods of time  Telephone switches that were expected to last 30 years  This type of testing is likely to be replaced by MC/DC testing in most critical applications. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 64
    • 65. Multiple Condition Testing Limitations  Because the number of test cases can be derived directly from a truth table containing all of the atomic conditions, this level of coverage can easily be determined.  However, the sheer number of test cases required makes MC/DC coverage more applicable to most situations. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 65
    • 66. Structure-Based Testing  Introduction  Condition Testing  Decision Condition Testing  Modified Condition/Decision Coverage (MC/DC) Testing  Multiple Condition Testing  Path Testing  API Testing  Selecting a Structure-Based Technique 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 66
    • 67. What is Path Testing?  Identifying paths through the code and then creating tests to cover them  In some cases we will get some of the same tests we got through control- flow testing.  On the other hand, we might come up with some other interesting tests. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 67
    • 68. Brute Force Path Testing  Test every independent path through the system.  Loops are the killers.  Needs infinite time and infinite resources!  However, it is realistic to perform some path testing.  Exercise: How many paths do exist in the shown figure? 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 68
    • 69. How can Loops be Tested in Path Testing?  A significant and meaningful reduction is needed. 1. Execute the loop zero times. 2. Execute the loop one time. 3. Execute the loop n times.  n is very small and typical loop value. 4. Execute the loop m times  m is the maximum number of loop times. 5. Test m-1, and m+1 times 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 69
    • 70. Path Testing Coverage  % of paths exercised by a test suite  Path coverage % = (# of paths executed/total # of paths ) * 100  Example:  Program has 74 paths  Tests exercise 30 paths  Path coverage = 40%  Notes:  100% path coverage (disregarding loops) = 100 % statement coverage + 100% decision coverage  Path testing provides more thorough testing than decision coverage, with a relatively small increase in the number of tests . 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 70
    • 71. How is Path Testing Done? 1. Beizer’s Method of Path Testing 2. Linear Code Sequence And Jump Testing – LCSAJ Testing 3. Basis Path/Cyclomatic Complexity Testing 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 71
    • 72. Beizer’s Method 1. Pick the first path as the simplest, functionally sensible path from entry to exit. 2. Pick each additional path as a small variation on the previous path.  Try to change only one branch in the path.  Favor short paths over long paths when possible.  Favor paths that make functional sense over ones that do not. 3. Pick paths that don't make functional sense only when required for coverage.  Such paths might be extraneous and should be questioned. 4. Use intuition when choosing paths. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 72
    • 73. Notes about Beizer’s Method  Some path segments are likely to be executed more than once.  The key point is to test every possible branch through the code at least once and possibly many times. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 73
    • 74. Exercise: Identify Min # of Test Cases to Achieve 100% Path Coverage Using Beizer’s Method 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 74 A = getchar(); B = getchar(); C = getchar(); if(A > 10) { printf(“Case 1n”); } if((B > 1) || (C > 2)){ printf(“Case 1n”); }
    • 75. What is a LCSAJ Testing?  AKA as decision-to-decision testing  Concerned with testing a linear sequence of executable statements of code followed by a jump to another executable statement of code. This might happen after a decision or a goto statement in the code.  Devised by Professor Michael Hennell of the University of Liverpool in 1976  Relatively high overhead methodology for testing 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 75
    • 76. What is a LCSAJ?  Small block of code that: 1. Has a start which can be a start of a module or a line of code jumped to from another LCSAJ 2. Has an end which can be a module end or a line of code where a jump occurs 3. Has a target line of the jump 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 76
    • 77. LCSAJ Testing Coverage  % of LCSAJs exercised by a test suite  LCSAJ coverage % = (# of LCSAJs executed/total # of LCSAJs ) * 100  Example:  Program has 42 LCSAJs  Tests exercise 30 LCSAJs  LCSAJ coverage = 71 % 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 77
    • 78. LCSAJ Example Code and Table 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 78 1:#include <stdlib.h> 2:#include <string.h> 3:#include <math.h> 4:#define MAXCOLUMNS 26 5:#define MAXROWS 20 6:#define MAXCOUNT 90 7:#define ITERATIONS 750 8:int main(void){ 9: int count = 0, totals[MAXCOLUMNS], val = 0; 10: memset(totals, 0, MAXCOLUMNS * sizeof(int)); 11: while(count < ITERATIONS){ 12: val = abs(rand()) % MAXCOLUMNS; 13: totals[val] += 1; 14: if(totals[val] > MAXCOUNT){ 15: totals[val] = MAXCOUNT; 16: } 17: count++; 18: } 19: return(0); 20:} LCSAJ # Start End Jump To 1 8 11 19 2 8 14 17 3 8 18 11 4 11 11 19 5 11 14 17 6 11 18 11 7 17 18 11 8 19 19 -1
    • 79. Exercise: Identify Min # of Test Cases to Achieve 100% LCSAJ Coverage  For the previous example, identify the min # of test cases to achieve 100% LCSAJ coverage. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 79
    • 80. Notes About LCSAJ Testing  Not all LCSAJs are really executable.  They can be executed by special intervention using debuggers for example.  A lot of time may be spent to achieve 100% LCSAJ coverage.  LCSAJ testing is really brittle to code and requires expensive maintenance.  LCSAJs testing in most cases leads to a partial path coverage. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 80
    • 81. What is Basis Path/Cyclomatic Complexity Testing?  Suggestion is to test the structure of the code to a reasonable amount  Devised by Thomas McCabe in 1976  Any SW module has a small # of unique, independent paths (excluding iterations) called basis paths.  Code structure can be tested by testing the basis paths because all the infinite different paths are just using/reusing the basis paths.  AKA minimal path testing 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 81
    • 82. Basis Path and Basis Path Set  Basis path is a unique independent path through the module with no iterations or loops allowed.  A basis path traverses at least one edge that no other path does.  Basis path set is the smallest # of basis paths that cover the structure of the code.  100% basis path coverage = 100% statement coverage + 100% decision coverage 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 82
    • 83. Cyclomatic Complexity  Cyclomatic refers to an imaginary loop from the end of a module of code back to the beginning of it.  Cyclomatic complexity # is the # of loops needed to cycle through the loop until the structure of the code has been completely covered.  Depends on the decisions in the module not on the module size  Cyclomatic complexity # = # of test cases we need to test the set of basis paths = # of basis paths  The higher the complexity, the more likely there will be a higher bug count 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 83
    • 84. Example: Calculating Cyclomatic Complexity 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 84 EuclidGCD (int a, int b){ int t; if (a <= 0) return -1; if (b <= 0) return -1; if (b > a) { t = a; a = b; b = t; } t = a % b; while (t != 0) { a = b; b = t; t = a % b; } return b; } End Start a <= 0 b <= 0 b > a t != 0 A B C D E F
    • 85. Example: Calculating Cyclomatic Complexity cont’d  C = # of Regions + 1 = 4 + 1 = 5  C = # of Edges - # of Nodes + 2 x # of Connected Components = 9 – 6 + 2 = 5  C = # of branching + # of looping + 1 = 3 + 1 + 1  Used for large code constructs and does not need CFGs  if, for, while, or do-while constructs count as 1.  case block counts as 1.  else and default block count as 0. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 85 End Start a <= 0 b <= 0 b > a t != 0 A C D E F B
    • 86. Example: Deriving Basis Paths and Test Cases  Basis paths are:  ABF  ABCF  ABCDEF  ABCDEF  ABCDEEF  Basis tests are:  -5, 2  -1  2, -5  -1  16, 8  8  4, 8  4  20, 8  4 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 86 End Start a <= 0 b <= 0 b > a t != 0 A C D E F B
    • 87. Exercise: Identify Min # of Test Cases to Achieve 100% Basis Path Testing Coverage 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 87 int main(int MaxCols, int Iterations, int MaxCount){ int count = 0, totals[MaxCols], val = 0; memset(totals, 0, MaxCols * sizeof(int)); if (MaxCount > Iterations){ while(count < Iterations){ val = abs(rand()) % MaxCols; totals[val] += 1; if(totals[val] > MaxCount){ totals[val] = MAXCOUNT; } count++; } } return(0); }
    • 88. Path Testing Applicability  Partial path testing is often performed in safety critical SW.  It is a good addition to the other methods covered earlier because it looks at paths through the SW rather than just at the way decisions are made. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 88
    • 89. Path Testing Limitations  While it is possible to use a control flow graph to determine the paths, realistically a tool is required to calculate them for complex modules. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 89
    • 90. Structure-Based Testing  Introduction  Condition Testing  Decision Condition Testing  Modified Condition/Decision Coverage (MC/DC) Testing  Multiple Condition Testing  Path Testing  API Testing  Selecting a Structure-Based Technique 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 90
    • 91. What is an API?  An Application Programming Interface (API) is code which enables communication between different processes, programs and/or systems.  APIs are often utilized in a client/server relationship where one process supplies some kind of functionality to other processes. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 91
    • 92. What is API Testing?  In certain respects API testing is quite similar to testing a GUI.  Though GUI is rarely involved.  The focus is on the evaluation of input values and returned data.  Setting (complex) initial environment as GUI is mostly not involved is very common. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 92
    • 93. Aspects of API Testing Aspect Description -ve testing Trying to use API interfaces in ways for which they were not intended Testing APIs robust error handling to avoid incorrect operation Combinatorial testing Combining the APIs as well as their parameters in different ways because APIs are often used in conjunction with other APIs Availability/Reliability testing APIs are loosely coupled. Transactions can be lost or timed-out. Thorough testing of the recovery and retry mechanisms to ensure that all services have a very high availability Requires reliability testing by the API author as well as infrastructure support 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 93
    • 94. API Testing Coverage  API testing is a type of testing rather than a measurement/coverage technique.  No specific coverage level  Coverage example could be input/output coverage. 1. Use equivalence partitioning to determine valid/invalid input/output partitions. 2. Use equivalence partitioning testing and boundary value testing to design test cases that cover all valid/invalid input/output partitions. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 94
    • 95. API Testing Applicability  In distributed systems or remote processing  OS calls  SOA  RPC  Web services  Testing systems of systems 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 95
    • 96. API Testing Limitations  Usage of specialized tools as there is no GUI involvement  Tools are needed for:  Initial environment setup  Data marshaling  API invocation  Result determination 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 96
    • 97. Structure-Based Testing  Introduction  Condition Testing  Decision Condition Testing  Modified Condition/Decision Coverage (MC/DC) Testing  Multiple Condition Testing  Path Testing  API Testing  Selecting a Structure-Based Technique 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 97
    • 98. Testing Principle #6: Testing is Context Dependent 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 98  The context of the software system under test should guide the technical test analyst.  Code criticality α Level of coverage α Time & resources  Sometimes the level of coverage required may be derived from applicable standards that apply to the software system.  DO-178 B in USA (ED-12B in Europe)  IEC-61508 Code Criticality Level of Coverage Time & Resources
    • 99. Comparisons can Help the Selection  There are 3 ways of comparisons:  Pros and cons general summary  Subsumes  Check-list based  Note that API testing is not included as it is not relate to code. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 99
    • 100. Techniques Pros & Cons Technique Pros Cons Statement Easy to understand & apply Can isolate dead code Low maintenance required Limited value for decisions & loops Many paths untested Ad-hoc testing can achieve ~70% statement coverage Decision Easy to understand & apply Good cost-benefit ratio Does not account for condition combinations Condition No real advantage over decision testing May omit decisions if not carefully done Decision/Condition Easy to understand Good cost-benefit ratio Needs careful choice of test cases MC/DC Less test cases compared to multiple condition More difficult to understand Multiple Condition High thoroughness High # of test cases Multiple conditions could have been coded as consecutive single conditions LCSAJ Gives different angle of path coverage Difficult to apply Highly brittle to code Path Very throughout Good for procedure testing Practically impossible to achieve high level of coverage 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 100
    • 101. Subsumes - Which Technique Implicitly Includes Others? 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 101 Multiple Condition MC/DC Decision Condition Decision Condition Statement LCSAJ Path
    • 102. Check-List Comparison Technique Thorough Understandable Tool Support Maintainability Statement 1 5 5 5 Decision 2 5 5 5 Condition 2 5 4 5 Decision Condition 2 5 4.5 5 MC/DC 4 3 3 5 Multiple Condition 3 4 4 5 LCSAJ 3 1 1 2 Path 5 2 4 2 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 102
    • 103. DO-178 B  Used in an airborne environment 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 103 Failure Level Description Testing Technique A: Catastrophic Failure may cause lack of critical function needed to safely fly or land the plane MC/DC B: Hazardous Failure may have a large –ve impact on safety or performance Decision Testing and MC/DC optional C: Major Failure is significant, but less serious than A or B Statement testing @ minimum D: Minor Failure is noticeable, but with less impact than C E: No effect failure has no impact on safety
    • 104. IEC-61508  For the functional safety of programmable, electronic, safety-related systems (automotive, rail, manufacturing process, nuclear power plants, and machinery) 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 104 SIL Testing Technique 1 (Least critical) Statement and branch coverage recommended 2 Statement coverage highly recommended, branch coverage recommended 3 Statement and branch coverage highly recommended 4 (most critical) MC/DC highly recommended
    • 105. Testing Multi-Systems  In modern systems, it is rare that all processing will be done on a single system.  API testing should be instituted anytime some of the processing is going to be done remotely.  The criticality of the system should determine how much effort should be invested in API testing. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 105
    • 106. One Final Word  Dynamic techniques should not be used in isolation.  They should be mixed and combined.  Some techniques can be useful in identifying the test conditions, others more suited for minimizing test conditions, while others are more suited for deriving test cases from test conditions. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 106
    • 107. Combining Techniques Example 1. Using decision tables to identify test conditions from requirements.  Test conditions will be every column in this case. 2. Transform the decision table into a binary decision table.  This may result in the # of test conditions. 3. Identify a binary equation to relate actions to decisions. 4. Apply MC/DC to reduce the # of test conditions. 5. Identify valid/invalid ranges for every condition and action using equivalence portioning. 6. Apply equivalence partitioning and boundary value analysis to derive test cases to cover the test conditions. 31/08/2012 ISTQB Advanced Level 2012 - Technical Test Analyst 107