Università degli Studi dell’AquilaL20: White Box/Structural Testing                                      Henry Muccini    ...
The material in these slides may be freely reproduced anddistributed, partially or totally, as far as an explicitreference...
AGENDARecap                              Fault, Error, Failure chainWhite Box Testing                                   St...
Recap“Systematic” testing vs. “ad hoc”testing →Repeatable,   MeasurableSoftware Testing Process →Test   Selection, Test Ex...
Fundamental Testing QuestionsTest Case:  How is test described / recorded?Test Criteria:  What should we test?Test Oracle:...
Fault, Error, FailureFault: anomaly in source code  →   may or may not produce a failure  →   a “bug”Error: inappropriate ...
THE FAULT-FAILURE MODEL                                         Error                              Failure       Fault    ...
WHITE BOX/STRUCTURAL TESTING
How to test thisJava program?
Selection of test suite is based on someelements in the code                              (Test) InputsAssumption: Executi...
Idea:  →   A program is a graph  →   A graph can be traversed, by using some inputs  →   A test suite is adequate if I gua...
Control Flow  →   The partial order of statement execution, as defined by the      semantics of the languageData Flow  →  ...
1   function P return INTEGER is 2   begin 3      X, Y: INTEGER; 4      READ(X); READ(Y); 5      while (X > 10) loop 6    ...
6                  7            10                     F            T                  T            T                    F...
printSum(int a, int b) {       Test this case   int result = a + b;   if (result > 0)     printcol(“red”, result); and thi...
Defined in terms of → test   requirementsResult in → test specifications → test casesSelection VS adequacy/evaluation crit...
test specificationsprintSum(int a, int b) {       a+b>0   int result = a + b;   if (result > 0)     printcol(“red”, result...
test casesprintSum(int a, int b) {       a == 3                               b == 9   int result = a + b;   if (result > ...
Test requirements: Statements in programCstmts = (number of executed statements)           (number of statements)
printSum(int a, int b) {    int result = a + b;    if (result > 0)      printcol(“red”, result);    else if (result < 0)  ...
a == 3  b == 9  printSum(int a, int b) {    int result = a + b;    if (result > 0)      printcol(“red”, result);    else i...
a == 3   a == -5  b == 9   b == -8  printSum(int a, int b) {    int result = a + b;    if (result > 0)      printcol(“red”...
Test hypothesis: →By  executing a path once, potential faults related to it will  be revealed (independently from the inpu...
Only about 1/3 of NASA statements were executedunder test before software was released (Stucki 1973)Microsoft reports 80-9...
a == 3   a == -5  b == 9   b == -8  printSum(int a, int b) {    int result = a + b;    if (result > 0)      printcol(“red”...
a == 3   a == -5  b == 9   b == -8  printSum(int a, int b) {     int result = a + b;     if (result > 0)       printcol(“r...
Test requirements: Branches in the programCbranches = (number of traversed branches)               (number of branches)
a == 3     a == -5   b == 9     b == -8  printSum(int a, int b) {     int result = a + b;     if (result > 0)       printc...
a == 3   a == -5  b == 9   b == -8  printSum(int a, int b) {     int result = a + b;     if (result > 0)       printcol(“r...
a == 3   a == -5   a == 0  b == 9   b == -8   b == 0  printSum(int a, int b) {     int result = a + b;     if (result > 0)...
entry       • Consider test cases                                        {(x=5,y=5), (x=5, y=-5)}1. void main() {         ...
entry       • Consider test cases                                        {(x=5,y=5), (x=5, y=-5)}1. void main() {         ...
Test requirements: Truth values assumed bybasic conditions      Cbc = (number of boolean values assumed by all basic condi...
entry       • Consider test cases                                        {(x=0,y=-5), (x=5, y=5)}1. void main() {         ...
Test requirements: Branches and truth valuesassumed by basic conditions                  if ( ( a || b ) && c ) { … }     ...
Key idea: Test important combinations ofconditions, avoiding exponential blowupA combination is “important” if each basicc...
MC/DC criterion: For each basic condition C, there aretwo test cases in which the truth values of allconditions except C a...
( a && b && c )Test case    a       b       c      outcome   1        True    True    True     True   2        True    Tru...
( a && b && c )Test case    a       b       c      outcome   1        True    True    True     True   2        True    Tru...
MC/DC criterion: For              ( a && b && c )                 each basic conditionTest case    a       b       c      ...
( a && b && c )Test case    a       b       c      outcome   1        True    True    True     True   2        True    Tru...
( a && b && c )Test case    a       b       c      outcome   1        True    True    True     True   2        True    Tru...
( a && b && c )Test case    a       b       c      outcome   1        True    True    True     True   2        True    Tru...
( a && b && c )Test case    a       b       c      outcome   1        True    True    True     True   2        True    Tru...
( a && b && c )Test case    a       b       c      outcome   1        True    True    True     True   2        True    Tru...
( a && b && c )Test case    a       b       c      outcome   1        True    True    True     True   2        True    Tru...
( a && b && c )Test case    a       b       c      outcome   1        True    True    True     True   2        True    Tru...
Tradeoff between number of required testcases and thoroughness of the testRequired by both US and European qualitystandard...
Path coverage: cover each path in the programLoop coverage (given n): cover all paths that contain at  most n iterations o...
Most widely used as adequacy criteria because fullyautomatable! Not all graph paths correspond to actual program paths →so...
White Box Testing (Introduction to)
White Box Testing (Introduction to)
White Box Testing (Introduction to)
White Box Testing (Introduction to)
White Box Testing (Introduction to)
Upcoming SlideShare
Loading in...5
×

White Box Testing (Introduction to)

384

Published on

An introductory lecture to white box testing

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
384
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
33
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

White Box Testing (Introduction to)

  1. 1. Università degli Studi dell’AquilaL20: White Box/Structural Testing Henry Muccini DISIM, University of L’Aquila www.henrymuccini.com, henry.muccini@univaq.it
  2. 2. The material in these slides may be freely reproduced anddistributed, partially or totally, as far as an explicitreference or acknowledge to the material author ispreserved.Partially based on material from Alex Orso, Mauro Pezze’,Michal Young, and Andreas Zeller
  3. 3. AGENDARecap Fault, Error, Failure chainWhite Box Testing Statement Coverage Branch Coverage Basic Condition Coverage MC/CD Coverage
  4. 4. Recap“Systematic” testing vs. “ad hoc”testing →Repeatable, MeasurableSoftware Testing Process →Test Selection, Test Execution, Oracle, AdequacyType of testing →Unit,Integration, System →Performance, Stress →Regression →WB, BB
  5. 5. Fundamental Testing QuestionsTest Case: How is test described / recorded?Test Criteria: What should we test?Test Oracle: Is the test correct?Test Adequacy: How much is enough?Test Process: Is testing complete and effective?Test Driver: used to execute the system on the SUTStubs: to reproduce missing objects, for integration testing
  6. 6. Fault, Error, FailureFault: anomaly in source code → may or may not produce a failure → a “bug”Error: inappropriate development action → action introduced by a fault → Cause of a faultFailure: incorrect/unexpected behavior or output → incident is the symptoms revealed by execution → failures are usually classified
  7. 7. THE FAULT-FAILURE MODEL Error Failure Fault affects the delivered service sw internal state must be is corrupted activated (latent or manifest)A fault will manifest itself as a failure if all of the 3 following conditions hold:1) EXECUTION: the faulty piece of code is executed2) INFECTION: the internal state of the program is corrupted (error state)3) PROPAGATION: the error propagates to program output PIE MODEL (Voas, IEEE TSE Aug. 1992)
  8. 8. WHITE BOX/STRUCTURAL TESTING
  9. 9. How to test thisJava program?
  10. 10. Selection of test suite is based on someelements in the code (Test) InputsAssumption: Executing the faulty“element” is a necessary condition forrevealing a fault Source CodeThere exist several examples → Control flow (statement, branch, basis path, path) Output → Condition (simple, multiple) Internal behavior → Loop → Dataflow (all-uses, all-du-paths) → Fault based (mutation)
  11. 11. Idea: → A program is a graph → A graph can be traversed, by using some inputs → A test suite is adequate if I guarantee certain coverage ─ Metric: percentage of coverage achieved Objective: Cover the software structure
  12. 12. Control Flow → The partial order of statement execution, as defined by the semantics of the languageData Flow → The flow of values from definitions of a variable to its uses Graph representation of control flow and data flow relationships
  13. 13. 1 function P return INTEGER is 2 begin 3 X, Y: INTEGER; 4 READ(X); READ(Y); 5 while (X > 10) loop 6 X := X – 10; 7 exit when X = 10; 8 end loop; 9 if (Y < 20 and then X mod 2 = 0) then10 Y := Y + 20;11 else12 Y := Y – 20;13 end if;14 return 2*X + Y;15 end P;
  14. 14. 6 7 10 F T T T F 9 T2,3,4 5 9a 9´ 9b 14 F F 12
  15. 15. printSum(int a, int b) { Test this case int result = a + b; if (result > 0) printcol(“red”, result); and this one else if (result < 0) printcol(“blue”, result); [else do nothing] and this one!}
  16. 16. Defined in terms of → test requirementsResult in → test specifications → test casesSelection VS adequacy/evaluation criteria
  17. 17. test specificationsprintSum(int a, int b) { a+b>0 int result = a + b; if (result > 0) printcol(“red”, result); a + b < 0 else if (result < 0) printcol(“blue”, result); [else do nothing] a + b == 0}
  18. 18. test casesprintSum(int a, int b) { a == 3 b == 9 int result = a + b; if (result > 0) a == -5 printcol(“red”, result); b == -8 else if (result < 0) printcol(“blue”, result); a == 0 [else do nothing] b == 0}
  19. 19. Test requirements: Statements in programCstmts = (number of executed statements) (number of statements)
  20. 20. printSum(int a, int b) { int result = a + b; if (result > 0) printcol(“red”, result); else if (result < 0) printcol(“blue”, result); }Coverage: 0%
  21. 21. a == 3 b == 9 printSum(int a, int b) { int result = a + b; if (result > 0) printcol(“red”, result); else if (result < 0) printcol(“blue”, result); }Coverage: 71%
  22. 22. a == 3 a == -5 b == 9 b == -8 printSum(int a, int b) { int result = a + b; if (result > 0) printcol(“red”, result); else if (result < 0) printcol(“blue”, result); }Coverage: 100%
  23. 23. Test hypothesis: →By executing a path once, potential faults related to it will be revealed (independently from the input)Ideally: →To exhaustively cover all possible paths along the program graph representationApproximations to path coverage →Coverage criteria
  24. 24. Only about 1/3 of NASA statements were executedunder test before software was released (Stucki 1973)Microsoft reports 80-90% statement coverageBoeing must get 100% statement coverage (feasible)for all softwareUsually can about 85% coverage; 100% is harder → Unreachable code; dead code → Complex sequences → Not enough resources
  25. 25. a == 3 a == -5 b == 9 b == -8 printSum(int a, int b) { int result = a + b; if (result > 0) printcol(“red”, result); else if (result < 0) printcol(“blue”, result); }Coverage: 100%
  26. 26. a == 3 a == -5 b == 9 b == -8 printSum(int a, int b) { int result = a + b; if (result > 0) printcol(“red”, result); else if (result < 0) printcol(“blue”, result); [missing branch] }Coverage: 100%
  27. 27. Test requirements: Branches in the programCbranches = (number of traversed branches) (number of branches)
  28. 28. a == 3 a == -5 b == 9 b == -8 printSum(int a, int b) { int result = a + b; if (result > 0) printcol(“red”, result); else if (result < 0) printcol(“blue”, result); [missing branch] }Coverage: ?
  29. 29. a == 3 a == -5 b == 9 b == -8 printSum(int a, int b) { int result = a + b; if (result > 0) printcol(“red”, result); else if (result < 0) printcol(“blue”, result); [missing branch] }Coverage: 88%
  30. 30. a == 3 a == -5 a == 0 b == 9 b == -8 b == 0 printSum(int a, int b) { int result = a + b; if (result > 0) printcol(“red”, result); else if (result < 0) printcol(“blue”, result); [missing branch] }Coverage: 100%
  31. 31. entry • Consider test cases {(x=5,y=5), (x=5, y=-5)}1. void main() { 32. float x, y;3. read(x); 44. read(y); !(X=0 X=0 ||5. if(x==0)||(y>0) ||y>0) y>06. y = y/x; 7 67. else x = y+2; 88. write(x);9. write(y); 910.} exit
  32. 32. entry • Consider test cases {(x=5,y=5), (x=5, y=-5)}1. void main() { 3 • The test suite is adequate2. float x, y; for branch coverage, but3. read(x); 4 does not reveal the fault4. read(y); !(X=0 X=0 || ||y>0) y>0 at statement 65. if(x==0)||(y>0) • Predicate 5 can be true6. y = y/x; 7 6 or false operating on only7. else x = y+2; 8 one condition8. write(x);9. write(y);10.} 9 ⇓ exit Basic condition coverage
  33. 33. Test requirements: Truth values assumed bybasic conditions Cbc = (number of boolean values assumed by all basic conditions) (number of boolean values of all basic conditions) x y T T F F
  34. 34. entry • Consider test cases {(x=0,y=-5), (x=5, y=5)}1. void main() { 3 • The test suite is2. float x, y; adequate for basic3. read(x); 4 condition coverage4. read(y); !(X=0 X=0 || x y5. if(x==0)||(y>0) ||y>0) y>06. y = y/x; T T 7 67. else x = y+2; F F 88. write(x); • but is not adequate9. write(y); 9 for branch coverage.10.} exit ⇓ Branch and condition coverage
  35. 35. Test requirements: Branches and truth valuesassumed by basic conditions if ( ( a || b ) && c ) { … } a b c Outcome T T T T F F F F
  36. 36. Key idea: Test important combinations ofconditions, avoiding exponential blowupA combination is “important” if each basiccondition is shown to independently affect theoutcome of each decision
  37. 37. MC/DC criterion: For each basic condition C, there aretwo test cases in which the truth values of allconditions except C are the same, and the compoundcondition as a whole evaluates to True for one of thosetest cases and False for the other
  38. 38. ( a && b && c )Test case a b c outcome 1 True True True True 2 True True False False 3 True False True False 4 True False False False 5 False True True False 6 False True False False 7 False False True False 8 False False False False
  39. 39. ( a && b && c )Test case a b c outcome 1 True True True True 2 True True False False 3 True False True False 4 True False False False 5 False True True False 6 False True False False 7 False False True False 8 False False False False
  40. 40. MC/DC criterion: For ( a && b && c ) each basic conditionTest case a b c outcome C, there are two test 1 True True True True cases in which the 2 True True False False truth values of all 3 True False True False conditions except C 4 True False False False are the same, and the 5 False True True False compound condition 6 False True False False as a whole evaluates 7 False False True False to True for one of 8 False False False False those test cases and False for the other
  41. 41. ( a && b && c )Test case a b c outcome 1 True True True True 2 True True False False 3 True False True False 4 True False False False 5 False True True False 6 False True False False 7 False False True False 8 False False False False 1 True True True True 5 False True True False
  42. 42. ( a && b && c )Test case a b c outcome 1 True True True True 2 True True False False 3 True False True False 4 True False False False 5 False True True False 6 False True False False 7 False False True False 8 False False False False 1 True True True True 5 False True True False
  43. 43. ( a && b && c )Test case a b c outcome 1 True True True True 2 True True False False 3 True False True False 4 True False False False 5 False True True False 6 False True False False 7 False False True False 8 False False False False 1 True True True True 5 False True True False
  44. 44. ( a && b && c )Test case a b c outcome 1 True True True True 2 True True False False 3 True False True False 4 True False False False 5 False True True False 6 False True False False 7 False False True False 8 False False False False 1 True True True True 5 False True True False 3 True False True False
  45. 45. ( a && b && c )Test case a b c outcome 1 True True True True 2 True True False False 3 True False True False 4 True False False False 5 False True True False 6 False True False False 7 False False True False 8 False False False False 1 True True True True 5 False True True False 3 True False True False
  46. 46. ( a && b && c )Test case a b c outcome 1 True True True True 2 True True False False 3 True False True False 4 True False False False 5 False True True False 6 False True False False 7 False False True False 8 False False False False 1 True True True True 5 False True True False 3 True False True False
  47. 47. ( a && b && c )Test case a b c outcome 1 True True True True 2 True True False False 3 True False True False 4 True False False False 5 False True True False 6 False True False False 7 False False True False 8 False False False False 1 True True True True 5 False True True False 3 True False True False 2 True True False False
  48. 48. Tradeoff between number of required testcases and thoroughness of the testRequired by both US and European qualitystandards in aviation
  49. 49. Path coverage: cover each path in the programLoop coverage (given n): cover all paths that contain at most n iterations of any loop in the programData-flow coverage: cover data-flow relations in the program (du pairs)Mutation coverage: cover (kill) mutated versions of the program
  50. 50. Most widely used as adequacy criteria because fullyautomatable! Not all graph paths correspond to actual program paths →some paths could be not executable →Conditions may not be satisfiable due to interdependent conditions →Paths may not be executable due to interdependent decisions The testing output is strongly related to the testinginputs
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×