Syllabus                                                 8/19/10 3:52 PM



 CSE 565
 Software Verification, Validation an...
23.   Testing standards
24.   Test process improvement
25.   Test process patterns
[week 01 – Class 01 – Thursday]                                     8/19/10 3:52 PM



 Testing Background


 1. Objective...
-   VALIDATION: Tends to answer the question "are we building the
    right product? Is this software do what it needs to ...
If you cannot make a exhaustive poll then you have to assure you
    have a good sample (representative and valid statisti...
[week 01 – Class 01 – Thursday]                                    8/19/10 3:52 PM



 Testing Process Activities


 1. So...
3. Testing Attitudes

-   Independence. IV&V, independence test, or independence team.

-   Customer perspective: understa...
[week 02 – Class 02 – Tuesday]                               8/24/10 2:48 PM




 0. From the last class

 Reading Papers
...
   level of independence

d) Faster and Cheaper
   Lean manufacturing, lean testing, lean software development :-o

   T...
   19. (Personnel issues). Testers are not domain experts. We need
    to have the people who really understand the appli...
•   5. (Foundational) Use ODC (Orthogonal Defect Classification).
    This technique is used to improve test effectiveness...
And then finally some how 1 /1000 error per LOC

    Then worry about variation en configuration

•   Review the Table 1 a...
[week 02 – Class 02 – Tuesday]                                8/24/10 2:48 PM




 1. Testing Techniques

 There are two t...
Example: software for AMT system; examples of use-cases: withdraw,
    deposit, check balance.

    For the use-case withd...
[week 02 – Class 03 – Thursday]                                                                                      8/26/...
2. Equivalent Partitioning Technique

1. Identify Partition.
2. Write test covering as many uncovered valid partitions as ...
EP requires at least 5 test cases for this example (according with the
    three steps described before)



   Example Tw...
For this create equivalent partitions (EP).
Our INPUT is PIN

Think about ATTRIBUTES like:

length
- 1. 4-8 is (v)
- 2. <4...
Then create test data:


     Test         PIN
     1            1234 (VVV)
     2            123 (IVV)
     3            ...
Testers need Requirements as Development!



Create coverage matrix:


 Partitions             Test cases
                ...
[week 03 – Class 04 – Tuesday]                                  8/31/10 2:59 PM




 0. From the last class

    We talk ...
1. Paper 4. An Introduction to Scenario Testing
   by Cem Kaner
   For each scenario 1 or more test

   Scenarios are hi...
-1, 0, 1
    39, 40, 41
    99, 100, 101




3. Paper 5. A review of Boundary Value Analysis
   Techniques by David J. Coe...
41-100             V
    <0                 I
    >100               I
    ~integer           I


EP defines the following...
1. Functional Testing
1.1. Sampling Strategies (Scenario| Use Case)-Based
1.1.3. Decision Trees, Decision Tables and DOE

...
Customer
   A                      V    V
   B                      V                  V
   C                      V      ...
Other way is to reduce the number of inputs, and instead of testing all
    5 inputs at the same time, we look for which i...
o        A
                   1-9
                   10-99
                   99-100
         o B
                    ...
[week 03 – Class 05 – Thursday]                               9/2/10 2:49 PM




 0. From the last class

    Last class ...
   Standard shipping (6th case)

   Decision table

                 1   2   3   4   5   6   7   8   9   10   11   12   ...
Rates
    - In-state: $50/hr for first 12hrs and $600 >12
    - Out-state: $500/hrs for first 12hrs and $6000 > 12
    - D...
In          X          X                 X               X
     Out              X             X                 X        ...
Positive Integer   V    V                          V

      Spaces                         V            V

     Supplier

...
   Create Valid Inputs and Valid Combinations
   Then Crete Invalid inputs and Invalid Combination separately.

    The ...
[week 04 – Class 06 – Tuesday]                                  9/7/10 3:00 PM




 0. From the last class

    We have b...
But, what happens if:

    1. lost of power (in any point of time)



   Example Two:

    The “call waiting” feature in ...
These examples are to illustrate these techniques scenario-based. And
see what happen when an asynchronous event happen du...
[week 04 – Class 06 – Tuesday]                                  9/7/10 3:00 PM




 1. Functional Testing
 1.2. Requiremen...
- abstract model (simulation)
    4. Tools supports

   SUT (system under test)

   For Software Architecture:
    Requi...
   We are going to illustrate how to do Model-Based testing with State
    Machines

   Example: ATM Machine




       ...
   State machine Coverage: to reach each state and test each stimuli
    or event from the state.

   We need to be sure...
A-> 3 -> B (B-3)

    where the first letter is the state reached and the number indicates the
    event.

    3. 2nd and ...
T5 could be 3,5,5,5

The goal is not the minimum but if we can it is ok.
In this case we have 5 test cases to cover 6 path...
[week 04 – Class 07 – Thursday]                                                 9/9/10 2:56 PM




 1. Continue with Model...
   STEP 2. Sequences

    1,4
    1,5,6
    1,5,7
    1,5,8
    2,
    3,9



   STEP 3. Test Cases

    All of them sho...
[week 04 – Class 07 – Thursday]                                  9/9/10 2:56 PM




 1. Functional Testing
 1.4. Design of...
   DOE – Pair Wise Combination Testing

   Steps:

    1. Identify the inputs parameters to the SUT (systematic under Te...
   Example Three

    CPU - A B C
    OS – D E H
    DB – F, G

    All combinations are 18

   With Pair wise combinati...
   See Table 9.




3. Paper 10. Automated Combinatorial Test Methods –
Beyond Pair wise Testing

   They review pair wi...
This is also used for Interaction Testing between Black boxes.
[week 05 – Class 08 – Tuesday]                                  9/14/10 2:57 PM




 1. Paper 11. An Innovative Approach f...
   Tend to be techniques to be applied into lower level (small units of
    code) unit, services, components.

    Not fo...
   Flow Chart = Control Flow Diagram.
    Test predicate (diamond)
    Statement (oval)

   4 statement in the previous ...
   EJEMPLO UN MAPA CUADRICULAR CON CALLES Y CADA INTERSECION
    ES DECIDIR UNA DIRECCION

   Even If I use structured p...
But that not enough for Decision-Condition Coverage.
    We need to open the test predicate and look inside: there are 5
 ...
if4
T
F

if6
T
F

if8
T
F



For multiple-condition coverage C1 y C2 make combinations of T y F


For conditions C3 y C4 s...
4 con T, T
5
6T
7

4 con F, T

    Multiple-condition Coverage: 100% NO
    Decision-condition Coverage: 100% NO
    De...
[week 05 – Class 09 – Thursday]                               9/16/10 2:53 PM




 0. Last Class

    We review different...
TF

    So F F can not occurs!




1. Problem

•   Code coverage for midterm exam

          if x <10 or y>50
    (1)   th...
Decision Coverage
We have 2 decisions points (ifs)


   Decision    Test Case
   1T          T1(T o F), T2(F o T),T3(T o F...
So, we have:

•   Multiple-Condition Coverage: 5/8 = 62.5%
•   Decision-Condition Coverage and Decision Coverage: 11/12 = ...
( 1)   read (,y)
( 2)   while x!=0
( 3)   y <- x + y
( 4)   if y>10
( 5)   then put (y)
( 6)   else put (x)               ...
many times we will run the loop we could mathematically define how
many paths we have.



3. Anomaly Detection

   Static...
[week 06 – Class 10 – Tuesday]                                9/21/10 2:57 PM




 0. Last Class

    Continue with Testi...
   (1)
    A1 = A0 + B0
    B1 = B0
    C1 = C0 = (undefined)

   (2)
    A2 = A1 = A0 + B0
    B2 = A1 - B1 = A0 + B0 –...
In these cases we need to execute the code and do the symbolic
  execution for each of the paths.

  So if we do the True-...
x3 = x0 + 15
    y3 = x0 + 10

    x5 = x3 + 10 = x0+25
    y5 = x0 + 10

    The numbers correspond to the sentence numbe...
y = -0.5

    This are the values to test the path

   Example Two (FT Path):

    Calculate the symbolic variables.

   ...
And the conditions are:

initial:
(x0 > 7) and (y1 > 0) and (x3 + y3 > 10)

then,
(x0 > 7) and (x0 > -10) and (x0 > -15/2)...
[week 06 – Class 10 – Tuesday]                                  9/21/10 2:57 PM




 1. Complexity Measure: McCabe Number....
Then s2
    Else if y>0
                                                   1	
  
      Then s3
      Else s4
             ...
Example One:

   This are the 4 basis path in the example:

    (1)   F   T T (arbitrary)
    (2)   T   _T
    (3)   F   ...
   In the past classes we talk about Code coverage (white box).

   We mention: Control and Data flow coverage.

   Dat...
[week 06 – Class 11 – Thursday]                                 9/23/10 2:55 PM




 0. Last class

    Control Flow Cove...
   For each node:
       o def(i) : set of variables for node i which are globally defined
       o c-use(i): set of vari...
c-use(1)   =   0 (nothing)
    c-use(2)   =   0 (nothing)
    c-use(3)   =   {y}
    c-use(4)   =   {y}

    p-use(TP1) = ...
Go back to Example Two:

    def(1) = {x, y, z}
    def(2) = {y}
    def(3) = {w}
    def(4)= {w}
                        ...
3. All-Uses Coverage

   Goal: develop test cases such that for each definition, there is a
    definition use path (du) ...
   Search {i1. I2…} en c-use(M) = {i1, i2…} and add the numbers M to
    the set

    dcu(x,1) = {}
    dcu(y,1) = {}
   ...
100% condition coverage

All definitions criteria is satisfied if for each definition in the program
some use of this defi...
201008 Software Testing Notes (part 1/2)
Upcoming SlideShare
Loading in...5
×

201008 Software Testing Notes (part 1/2)

14,640

Published on

Published in: Technology
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
14,640
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
0
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide

201008 Software Testing Notes (part 1/2)

  1. 1. Syllabus 8/19/10 3:52 PM CSE 565 Software Verification, Validation and Testing Department of Computer Science and Engineering Ira A. Fulton School of Engineering Tempe, Arizona 85287-4411 Topics 01. Week 01 - Testing background 02. Week 01 - Testing process activities 03. Week 02 - Requirements-based testing techniques 04. Feature interaction testing 05. Exploratory testing 06. Structure-based testing techniques 07. Integration testing 08. System testing 09. Regression testing 10. Usability testing 11. Performance testing 12. Security testing 13. Design of Experiments 14. Testing tools 15. Software Reliability Engineering 16. Reliability models 17. Statistical testing 18. Test planning 19. Risk based testing 20. Tracking testing progress 21. Testing measures 22. Test documentation
  2. 2. 23. Testing standards 24. Test process improvement 25. Test process patterns
  3. 3. [week 01 – Class 01 – Thursday] 8/19/10 3:52 PM Testing Background 1. Objective of Testing a. - Minimize risks b. - Find faults (things that not works) c. - How robust is the software: Do not fail and if fails it recover very fast d. - Assess performance 2. When To Stop Testing? a. - Demonstrate requirements are met b. - When it is good enough???? Monthly Meetings Of IEEE-CS and ACM - Sponsor the main conferences - Phoenix (search for it!) 3. Definitions IEEE - RELIABILITY: Probability that a give SW application will operate for some given time period without SW failure. MTBF - mean time between failure. It is a related concept with Reliability. Agencies which certified SW. Each of them have standards: FAA. Aviation Agency. Standard D0178b. Probability (10^-9). B787 is another standard (Boeing) FDA is another agency.
  4. 4. - VALIDATION: Tends to answer the question "are we building the right product? Is this software do what it needs to do? (cumple requerimientos de usuario) - VERIFICATION: Are we building the product right? Does this meets it requirements. It is the easy one. (esta implementado acorde a diseño) Industry talks about V&V concepts which stands for Validation & Verification. Other concept is IV&V which stands for Independent V&V - TESTING: Examination of the behavior of a program by executing it on sample data sets. Testers VS Pollsters Analogies [Poll - encuestas] [Scientific pollsters such as CNN, NBC News. They take a representative group for the poll, where the demographics are well defined which means that the MOE (margin of error) is less than 3.] Pollsters are paid to make predictions, while Testers …. does tester make predictions?. Testers are expected to do predictions about the reliability of the SW. Both Testers and Pollsters works with significant constrains, mainly money and time. Pollsters cannot do an exhaustive poll. Testers cannot either do exhaustive testing. Some cases maybe when the SW is very simple and has very specific inputs.
  5. 5. If you cannot make a exhaustive poll then you have to assure you have a good sample (representative and valid statistically). The same happened for a test process. 4. Effective Testing - Examination of the behavior of a program by executing it on REPRESENTATIVE sample data sets. - ROI (Return of Investment): Is a development effort but also a V&V effort. How many effort we put on doing the development and the V&V. Time used for different activities 1/3 - A&D 1/3 – Coding 1/3 - V&V Cost = f(productivity)…The SW productivity is 1 LOC / hr. ROI, on software is all about the effort we put on doing the V&V. 5. Measures Of Reliability - CRUD - Customer Reported Unique Defects. It is expressed by defects/KLOC. In 2010 the average is 1/KLOC. defects/Kloc = 1 or 0.1 (2010) - Another way to measure this is using 6sigma =). (six sigma) = 3.14/million. Even here is A LOT! See The Risk Digest http://catless.ncl.ac.uk/risks
  6. 6. [week 01 – Class 01 – Thursday] 8/19/10 3:52 PM Testing Process Activities 1. Software Testing Process vs. Development Process - Development process a. Requirements [What to build – use case scenarios] b. Design [Pick the Arch patterns. How to meet requirements] c. Implementation d. V&V These 4 activities on development has a analogy on testing process. - Testing process a. Test objectives (plan, conf management) b. Sampling strategy [Test Design] c. Write test cases and test scripts d. Execute test Testing is a mature process and requires specifics skills. There are some other activities that are analogous and that affects both, testing and development: a. Planning b. Configuration Management c. Changes of requirements d. Review 2. Testing Limitations And Constraints 1. Constraint by the requirements (missing or bad reqs by example) 2. exhaustive testing is not possible 3. Limited time and money
  7. 7. 3. Testing Attitudes - Independence. IV&V, independence test, or independence team. - Customer perspective: understand the customer; know about the domain. - Demonstrate that the system works. Go through all the requirements and be sure that we meet them all. This is kind of clean process. - Dirty testing (try to break the system) - Professionalism. Testing should be an add value to the product. Should be something where a person can have a career. The different paths of career on SW could be: development, mgnt, testing. It could be a good idea to get a “Licensing of SE”, there is a certification on Texas (fee, test, and educational background). The test covers: development and reason about correctness (testing, implementation, prove of correctness). Note: A good test engineer has: - - an ability to take the point of view of the customer, - a strong desire for quality, - a 'test to break' attitude, - and an attention to detail.
  8. 8. [week 02 – Class 02 – Tuesday] 8/24/10 2:48 PM 0. From the last class Reading Papers  Classic Mistakes in Testing  Testing Best Practices  Crosstalk about Quality Testing Attitudes  Professionalism: testing is a carrier, as develop and as management.  License as SE in Texas! :-0 (develop and reasoning about correctness). Include testing, Proof of correctness and Implement  Testing is not a course into university :-0 but there are a lot of books! 1. Testing Principles a) Do not throw away your test. They are going to be useful in the future. Reuse, maintain your product (the product will be out for a while, it is better to save your case of testing) regression analysis. b) Probability of the existence of more errors in a section of code is proportional to the # of errors already found.  Remember we have 1 error/Kloc  Errors tend to generate clusters (because complexity, dependences of the code, skills of the developer of that section of code, poor or changing requirements that affect that part). c) A necessary part of a test case is the definition of the expected result A test case consist on: input and expected results.  Example: Testing a new optimized C++ compiler. The input would be a file1 with code, and the exit a file2 with code that satisfied defined characteristics.
  9. 9.  level of independence d) Faster and Cheaper Lean manufacturing, lean testing, lean software development :-o  This has to be with technology, techniques, and reuse. 2. Paper 1. Classic testing mistakes by Brian Marick • Brian Marick is a king of leader in the area  [Page 24] Some classic testing mistakes  2. (Role of Testing). Thinking that the purpose of testing is to find bugs. There is not problem in the state. However we should remind that the objective of testing is to find “important bugs”.  4. (Role of Testing). Not reporting usability problems. It is common to report only functionality errors. Remain that the user do not use a tool functionally but by tasks.  7. (Role of Testing). Starting testing too late (bug detection not bug reduction). Testing should begin in front. We should write test at the same time we are writing requirements. You write your test cases before you write your code. The idea is that developer have more information about what it is expected.  10. (Planning the complete testing effort). Putting stress and load testing off to the last minute. It is a problem due the fact there would not be enough time to fix any issue.  11. (Planning the complete testing effort). Not testing the documentation. Normally the testing is done to the requirements, but it is also necessary to test the documentation (installations procedures and configuration instructions).
  10. 10.  19. (Personnel issues). Testers are not domain experts. We need to have the people who really understand the application.  23. (Personnel issues). A physical separation between developers and testers. Separated groups. It is not good to have this separation. 3. Paper 2. Testing Best Practices by Ram Chillarege • Report 28 best practices • 1. (Incremental) Asses software reliability via statistical testing. One way to talk about reliability is MTBF (Mean Time Between Failures), the only way to measure this is through statistical testing. • 2. (Basis) Develop an agile test design. Agile means: flexible, short cycles, small pieces at the same time. • 3. (Foundational) Utilize model-base testing-techniques Whle working on the development we use Model Base Software Engineering (MBSE) and Software Development, this means have models such as UML, state machine, Petri nets, and then from here move to the code. UML - "state machine" go to test and code! Something similar happen for testing. We need models and the move from there to testing cases. • 4. (Incremental) Cross-functional teams. It is a very good practice to have tester and developers together at the same time. Use (IPDT) - Integrated Product Development Team
  11. 11. • 5. (Foundational) Use ODC (Orthogonal Defect Classification). This technique is used to improve test effectiveness. For each founded defect we will have attributes associated with it. Defect Attributes – Defect classification.  6. (Basis) Automated Test This is the first homework   7. (Foundational) Perform scenario-based testing. Scenario comes from Uses cases and Use-case modeling.  8. (Foundational) Emphasize usability testing  9. (Basis) Test over multiple platforms 4. Paper 3. Software Quality Challenge by W. Humphrey • Some analogies: Compare SW with other human generated media (paper/book) 1 defect per page. (software) 1,000,000 LOC – 40,000 pages / errors • [Page 4, paragraph 3] The problem with the software is that there is a lot of combinations and permutations of possible errors. So many permutations to test all cases software options State of the art in SW development. 100 defect / 1000 lines Then we filter errors: first the compiler, then static analyzer (search for error patterns), then inspection, then unit testing (UT), then Integration Testing(IT), then System Testing (ST) And we can obtain 10-20/1000 error per LOC
  12. 12. And then finally some how 1 /1000 error per LOC Then worry about variation en configuration • Review the Table 1 and Table 2. (testing variations and possible paths through a network). Think on how to test all possible options on the software. • [Page 5] The eight elements of software quality management 1. Establish quality policies, goals and plans. (V&V). 2. Properly training, couch and support. Make sure that all the team understand the principles, objectives and tools for testing. 3. Establish and maintain a requirements quality-management process (V&V). 4. Establish and maintain statistical control of the Software Engineering process. 5. Review, inspect and evaluate all artifacts (test). Evaluate the products and process! 6. Evaluate all defects for correction and prevent other similar problems. (ODC - Orthogonal Defect Classification) 7. Establish and maintain Configuration Management (CM) System (testing). 8. Continually improve the Process. (testing), postmortem and root case analysis (CMMI, PSP) This course is going to be focused on Software Reliability!
  13. 13. [week 02 – Class 02 – Tuesday] 8/24/10 2:48 PM 1. Testing Techniques There are two types of testing techniques  Functional testing: (black box oriented = tests the functionality of an application as opposed to its internal structures or workings). This are applicable on any level. Typical black-box test design techniques include: Decision table testing, All-pairs testing, State transition tables, Equivalence partitioning, Boundary value analysis.  Structural Testing: (white box oriented = tests internal structures or workings of an application as opposed to its functionality). The tester chooses inputs to exercise paths through the code and determine the appropriate outputs. It is analogous to testing nodes in a circuit White-box test design techniques include: Control-flow testing, Data flow testing, Branch testing, Path testing. We are going to review some of the Functional Testing techniques. 2. Functional Testing (black box) Useful for all levels of Testing! a) Scenario-base Testing: directly tined to use-cases (use case driven testing). The requirements are expressed on use-cases. (histories with several steps)
  14. 14. Example: software for AMT system; examples of use-cases: withdraw, deposit, check balance. For the use-case withdraw could be different scenarios: • normal (sunshine case) • alternative 1(no cash) • alternative 2(wrong PIN) • alternative 3 (bad something) Sampling Strategies for used for case-based testing b) Requirement-base testing. Is an alternative for Scenario-base testing. The requirements are in a document as a list of Shall’s (system shall do this), then for each of the req. we match one or more testing (we use a matrix of reqs vs test). This are functional requirements Make a table with requirements and testing associated with each requirement. (a table) 3. Structural Testing (white box) 4. Testing Levels  Unit Testing  Integration Testing  System Testing
  15. 15. [week 02 – Class 03 – Thursday] 8/26/10 2:57 PM 1. Functional Testing 1.1. Sampling Strategies (Scenario| Use Case)-Based 1.1.1. Equivalence Partitioning  technique that divides the input data of a software unit (function, service, component, program) into partitions of data from which test cases can be derived. In principle, test cases are designed to cover each partition at least once. This technique tries to define test cases that uncover classes of errors, thereby reducing the total number of test cases that must be developed.  Example: Domain = integer 32 bits -­                          0                  +   Test Cases: 1. -5 -> 5 2. 0 -> 0 3. 5 -> 5 4. max 5. min Partition: negative, positive and zero. Take a sample of each partition (valid and invalid partition) Invalid Partition - greater than max - less that min - characters - float
  16. 16. 2. Equivalent Partitioning Technique 1. Identify Partition. 2. Write test covering as many uncovered valid partitions as possible. 3. For each invalid partition write a test that cover one and only one at time.  Assumption: Input are Independent. BUT this assumption is normally not True.  Example One: Test a function f(x, y) that receive two input values in the ranges of X: 1 .. 10; Y: 50 .. 100 Partitions Test cases 1 2 3 4 5 X 1...10 V V V V <1 I I >10 I I Y 50…100 V V V V <50 I I >100 I I remember step 3, only one invalid value at one time Test X Y 1 5 55 2 -1 60 3 5 45 4 5
  17. 17. EP requires at least 5 test cases for this example (according with the three steps described before)  Example Two: (scenario testing) ATM system: scenario one –withdraw; scenario two –deposit; scenario three - atm pin change. Let us see a simple use case for amt pin change. And try to do scenario-base testing: User System 1. validate the user 2. display options 3. Pin change 4. Prompt new pin 5. enter pin 6. Validate pin 7. Reenter 8. Confirm/Update Scenarios: a) normal one b) alternative 1 (format problem) c) alternative 2 (pin not match) How to test normal and alternative scenarios? We need extra requirements What is a valid PIN? 4-8 digits 1st and last must be different cannot have 3 or more digits the same
  18. 18. For this create equivalent partitions (EP). Our INPUT is PIN Think about ATTRIBUTES like: length - 1. 4-8 is (v) - 2. <4 is (i) - 3. >8 is (i) first/last - 4. They are different (v) - 5. They are the same (i) number of same digits - 6. <3 (v) - 7. >=3 (i) AS developer you think in architecture, design and algorithm AS tester you identify partitions and make sampling Then create coverage matrix: Partitions Test cases 1 2 3 4 5 Length 4-8 V V V V <4 I I >8 I I First/Last Different V V V V V Equals I I # same digits <3 V V V V V >=3 I I
  19. 19. Then create test data: Test PIN 1 1234 (VVV) 2 123 (IVV) 3 123456789 (IVV) 4 1231(VIV) 5 244443(VVI) Testing is all about sampling! For the normal scenario 1 test {t1} For alternative 1 (invalid pin) {t2, t3, t4, t5}  Problem (in class): A Payroll System with 2 input: - Id (5 digits) - Hours Worked And three Output - Id (5 digits) - Type employee, is based on the Id 00000 - 29999 engineer 30000 – 99999 support - Gross pay $10 hr for 1st 40 hr $15 hr after 40 hr Additional information: - max number hours a person can work – 100 hr. If exceeded then send error. - min number – 0 hr - format of the hrs is integer and round down!
  20. 20. Testers need Requirements as Development! Create coverage matrix: Partitions Test cases 1 2 3 4 5 6 7 8 Id 5 digit V V V V V V Engineer V V V V Support V V V <5 I I >5 I I ~digit I I Hours 0-40 V V V V 41-100 V V V <0 I I >100 I I ~integer I I Remember the steps: 1 the matrix; 2 the valid as many as possible; 3 one invalid at time. Valid cases are tested with two test cases Then go for the invalids (only one each time).
  21. 21. [week 03 – Class 04 – Tuesday] 8/31/10 2:59 PM 0. From the last class  We talk about Functional Testing – Scenario-Based (Sampling Strategies) Equivalence Partitioning and work with some examples  Let us check the homework about the “Registration System” Partitions Test cases 1 2 3 4 5 6 7 8 Line # 5 digit in DB V V V V <5 I I >5 I I ~digit I I 5 digit not in DB I I Semester # 1 V V 2 V V 3 V V other I I Output (status) Cancel V V Open V V Closed V V Output are test variations. We want to try each of them. They are like States. Create samples for each semester and each Status. But not combinations, because we assume are independents.
  22. 22. 1. Paper 4. An Introduction to Scenario Testing by Cem Kaner  For each scenario 1 or more test  Scenarios are histories that are motivating, and credible, complex, and easy to evaluate.  Differences between requirements analysis and create test cases: Tester exploits disagreement. The main idea is to create histories (original and also taken from others similar systems) with objects inside, with user (good and bad) interacting with the objects, with system events 1. Functional Testing 1.1. Sampling Strategies (Scenario| Use Case)-Based 1.1.2. Boundary Value Analysis  Each of the inputs of the scenario must be tested with max, min and values around the max and min. (test the edges and around the edges of EP) • Example (Payroll system from the previous session): Partitions Hours 0-40 V 41-100 V <0 I >100 I ~integer I (1) Test the Edges means: <0 ; 0-40 ; 41 -100 ; >100 (2) Test around the Edges:
  23. 23. -1, 0, 1 39, 40, 41 99, 100, 101 3. Paper 5. A review of Boundary Value Analysis Techniques by David J. Coe.  Fault – (in Spanish: defecto) incorrect step , process or data definition in a computer program. A fault causes Failure. Detect fault before cause a Failure.  Failure – (in Spanish: falla) inability of a program or component to meet its function.  Software testing identifies Failures, which indicate the presence of one or more faults. (The goal of testing is to find faults!)  {FAULT - tolerant} Software – Detect the fault and avoid the Failure!  Single Variable and Multi-Variable 4. Merging EP and BVA  EP where input independent.  BVA where  EP x BVA (both at the same time) Go back to the problem: Partitions Hours 0-40 V
  24. 24. 41-100 V <0 I >100 I ~integer I EP defines the following sections <0 0-40 41-100 >100 BVA defines the followings values for each section -1 0, 1, 39, 40 41, 42, 99, 100 101 Hit the border and around there. BVA is done with INPUT but can also be about OUTPUT Example: Suppose that the maximum check amount that the system should generate is $9,999.99 [max] if the program is Input – Process – Output then create input that generate $9,999.99 9,999.90 OK --- NO .97 10000.00 reject --- NO .03 Output BVA is testing (validation and verification) but is more about Review & Inspection.
  25. 25. 1. Functional Testing 1.1. Sampling Strategies (Scenario| Use Case)-Based 1.1.3. Decision Trees, Decision Tables and DOE  DOE = Design of Experiments  This family of testing techniques are for test combinations of DEPENDENT Input!  Example Inputs: - Customer [A,B,C] - Order [1- 1000] Output: - Discount Rules: 1. Customer A gets 0% discount to < 10 gets 5% discount to 10 – 99 gets 10% discount >= 100 2. Customer B 5% < 10 15% 10-99 25% >= 100 3. Customer C 0% < 10 20% 10-99 25% >=100 With EP, we can do 3 testing cases Partitions Test Cases 1 2 3
  26. 26. Customer A V V B V V C V V Order 1-9 V V 10-99 V V 100-1000 V V The question is, are those three cases enough to test the whole system. Is it possible to ship the system with only this test? The answer is NO! But, in this case the type of customer and the amount of the order are not independent. So it is not possible to use EP. Decision Tables. 1 2 3 4 5 6 7 8 9 A X X X B X X X C X X X 1-9 X X X 10-99 X X X 10-100 X X X No think about a problem where you have five inputs values that are not independent. F (A, B, C, D, E) and A have 4 partitions, B 5, C 10, D 4 and E 3… this makes 2400 test cases table, which is huge. One way to reduce this is reduces the terms. For instance reviewing the partitions of C we can reduce from 10 to 3.
  27. 27. Other way is to reduce the number of inputs, and instead of testing all 5 inputs at the same time, we look for which inputs should be really tested together. And we might find out that A,B,C has to be tested together but not with D and E.  Example In the previous example: If we add the D (I) case to the customer, and we add the values out of the boundaries, this make 4 partitions for customer and 5 to order amount, what makes 20 test cases. But in fact instead of having the 20 of them, we can go with only 12, the 9 basics and then only three extras WITH ERRORS INDEPENDENTS! 1 2 3 4 5 6 7 8 9 10 11 12 A V X X X X B V X X X X C V X X X D (I) I X 1-9 V X X X X 10-99 V X X X 10- V X X X 100 <1 I X >1000 I X This technique becomes practical doing “assumptions” Columns becomes test cases! Decision Tree  Customer
  28. 28. o A  1-9  10-99  99-100 o B  1-9  10-99  99-100 o C  1-9  10-99  99-100 NOTICE that the decision tree generates the same 9 test cases as the table decision. Each PATH correspond with one Test Case! Trees and Tables are analogous!
  29. 29. [week 03 – Class 05 – Thursday] 9/2/10 2:49 PM 0. From the last class  Last class we talk about different forms of testing and use 3 techniques, combinatorial, decision table and trees.  We finish with a simple problem about online order system and we obtain 4 test cases with decision tree and 8 with the table.  Putting all Together: EP+BV+DTT The tree in the product node -> featured -> ship USA split in two cases: Yes and No. Yes -> Free No -> Standard Shipping The same happends to Qty node  The tree have 6 paths • customer o 1 time  free earth shipping (1st case) o ~ 1 time  product  featured • Shipping at USA o free us shipping (2nd case) • Nota USA o Standard Shipping  ~featured • qty o >100  Shipping at USA  Free US Shipping  Not a USA  Standard Shipping o <=100
  30. 30.  Standard shipping (6th case)  Decision table 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 st 1 time Yes V V V V V V V V No V V V V V V V V Product Featured V V V V V V V V Not F. V V V V V V V V USA Add Yes V V V V V V V V No V V V V V V V V QTY <100 V V V V V V V V >100 V V V V V V V V We have 16 vs. 6? Remember that for “first time user”, I do not care about Anything? Most of the time 6 test cases would be sufficient, however we might want to do the 16 just to be sure that the programmers do not do anything wrong. 1. First Problem in Class Scenario-based Testing to a Tuition System Inputs - Status (instate, outstate) - Hours # - Dependent of university employment
  31. 31. Rates - In-state: $50/hr for first 12hrs and $600 >12 - Out-state: $500/hrs for first 12hrs and $6000 > 12 - Dependant: Receive 50% discount not matter if it is in-state or out-st  Decision Tree • Status o In-state  Dependant  Yes • Hours <=12 o $25/hr • Hours >12 o 300/hr  No • Hours <=12 o $50/hr • Hours >12 o 600/hr o Out-state  Dependant  Yes • Hours <=12 o $250/hr • Hours >12 o 3000/hr  No • Hours <=12 o $500/hr • Hours >12 o 6000/hr  Decision Table 1 2 3 4 5 6 7 8 Status
  32. 32. In X X X X Out X X X X Hours <=12 X X X X >12 X X X X Depend Yes X X X X No X X X X In this case both tree and table generate the same amount of test cases. When building a tree it could get different number of test cases depending on with which input you start. The table is recommended to force us in think about combinations! But Trees show sequences! 2. Second Problem in Class Combining EP x BVA x DT See the document: Data Base Exercise. (entry validation) First step: look the inputs. All the inputs are the ones underlined. 1 2 3..5 6 7 7 8 T Part # In list V V V V V No in list I Part status code New V V Old V Unknown V V Req. Inventory
  33. 33. Positive Integer V V V Spaces V V Supplier In DB and PP V In DB not in PP V NO in DB Auto Reorder Flag Y V N V Reorder Qty Positive Integer V V Other Remaining Inv T Positive Integer V V Other Purchase Terms Positive Integer V Blank V Part price Dollar value V V V Other (same color means dependency) They are not independent so not use EP Decision Table could generate 1152 combinations :-( But they are not all dependent. So we can combine! Independents: Part # y Part Price. Dependents Part Status Code y Req. Inventory. Dependents Supplier y Purchase Term. Dependents Auto Reorder Flag, Reorder Qty y Remaining Inventory.  We are going to have 3 to 5 valid Test Cases, creating valid situations  Case 6 is an invalid option for input independent 1  Case 7 is an invalid combination of input 2 and 3 that are dependents
  34. 34.  Create Valid Inputs and Valid Combinations  Then Crete Invalid inputs and Invalid Combination separately. The final table has approximately 5 test cases for valid and a total of 13 to 14 test cases. And at the end BVA :-o This three techniques together in real problems
  35. 35. [week 04 – Class 06 – Tuesday] 9/7/10 3:00 PM 0. From the last class  We have been working with sampling data and synchronous testing techniques.  REQ BASED TESTING TECHNIQUES are applicable to all levels: Unit Testing, System Testing, etc.  For sampling we see: EP, BV, Decision tree and Decision table 1. Asynchronous Testing  The king of testing that we will when there are input that can take place in any point in time during the execution.  Start wit a simple example  TIME LINE: system engineering tool for looking in a scenario.  Example One: ATM system in a “normal” scenario. Sequence: 1. Customer insert a card 2. Customer request withdraw 3. Update account 4. Dispose cash 5. Take cash
  36. 36. But, what happens if: 1. lost of power (in any point of time)  Example Two: The “call waiting” feature in a cel phone. A talking with B, C tries to call A. We will have a time line for a normal scenario. 1. A enter digits for B 2. A presses send button 3. Network connects to B 4. B line is ringing 5. B answer 6. A and B taking During this time line could happen synchronous events. For instant, C calling A. BUT what happened if: C could call A during different point in the time line.  C calls when A is entering the digits. A’s phone will ring.  C calls just in the same time A hit the send button. A will answer C’s calling.  C calls when the network is trying to connects B. You might lost the call.  C calls when B line is ringing. If A get the call, does B get on hold ?
  37. 37. These examples are to illustrate these techniques scenario-based. And see what happen when an asynchronous event happen during a time-line. These asynchronous events could just happen or interrupt the time-line.
  38. 38. [week 04 – Class 06 – Tuesday] 9/7/10 3:00 PM 1. Functional Testing 1.2. Requirement-Based Techniques 1.2.1. Model-Based Testing (Paper 6,7,8) Use the model to generate the solution UML tool to create the model of the system And we can create the implementation from the model! 2. Paper 6. Gold Practice, Model Based Testing  MBT is the automatic generation of efficient test procedures using models.  Model, expected input, expected output, run test, compare output.  Techniques to build the model: decision table, finite state machine, grammars, Markov chains, state charts (UML). 3. Paper 7. A survey of Model-Driven Testing Techniques  UML for models  UML Testing Profile (extension of UML) (BPEL) – Business Process Execution Language. Used for services! Both describe behavior!  Survey: 1. Based on languages used 2. Degree of automatic generation of test scenario 3. Testing target: - implementation “code”
  39. 39. - abstract model (simulation) 4. Tools supports  SUT (system under test)  For Software Architecture: Requirements -> Design in UML  For Software Testing: Requirements -> Design in UML Testing Profile But this is not the goal of the class!! 4. Paper 8. Model Based Testing Using Software Architecture  Software Architecture - use UML - or use ADL (architecture description language)  Syntaxes and semantics  The paper talk about a technique called acme develop by ABLE/CMU  Example with a client / server system  The idea connect model and test  UML Test Profile feed into Tools, like JUnit Now let put the theory in practice! 5. State Machines
  40. 40.  We are going to illustrate how to do Model-Based testing with State Machines  Example: ATM Machine 1.  Invalid  card  /   Wait  for  a   error   card   2.  Stolen  card   A   3.  Valid  card  /   prompt  for  a  PIN   Capture   the  card   Waiting  for   a  pin   F   5.  Bad  PIN  /   B   Retry   4.  Valid    PIN   5.  Bad  PIN     4.  Valid    PIN   Waiting  for   Waiting  for  a   a  pin  /  2nd   transaction   try   5.  Bad  PIN     C   D   Waiting  for   4.  Valid    PIN   a  pin  /  3nd   try   E    So if this is the way the system works Test Each Path. For each state try each input or stimulates
  41. 41.  State machine Coverage: to reach each state and test each stimuli or event from the state.  We need to be sure that each state is tested and covered.  Try to test each event/state/event combination (3, 1)(4,1) (3,2)(4,2) (3,7)(4,7)  Let us see a Simple Way to attempt to achieve this coverage Testing cover creation  3 step process: 1. develop a state testing tree from the final state machine 2. identify test sequences from the tree (paths) 3. develop tests that the contain the sequences (start  end/terminal state)  STEP 1. Develop State Testing Tree 1. Start state = ROOT. From the example it would be A. 2. 1st. level. Identify the events and new states reached from the root. From the example it would includes: A ->1 -> A (A-1) A-> 2 -> F (F-2)
  42. 42. A-> 3 -> B (B-3) where the first letter is the state reached and the number indicates the event. 3. 2nd and remaining levels, expand each state not previously expanded in the tree B -> 4 -> D (D-4) B -> 5 -> C (C-5) C -> 4 -> D (D-4) C -> 5 -> E (E-5) E -> 4 -> D (D-4) E -> 5 -> F (E-5)  STEP 2. Identify Test Sequences (Path) The terminal states in this example are D and F. 1 2 3,4 3,5,4 3,5,5,4 3,5,5,5  Step 3. Develop Test Cases That Contains The Sequences We need to include the sequences in the exact order!! T1 could be 1, 3,4 this include (1) (3,4) T2 could be 2 T3 could be 3,4,5 T4 could be 3,5,5,4
  43. 43. T5 could be 3,5,5,5 The goal is not the minimum but if we can it is ok. In this case we have 5 test cases to cover 6 paths. 3,4 is different from 3,5,4 (remember same order)
  44. 44. [week 04 – Class 07 – Thursday] 9/9/10 2:56 PM 1. Continue with Model Based Testing Techniques. 1   2   A   B   4   5   3   C   6   D   7   9   E   8    STEP 1. Build the tree A -> 1 -> B A -> 2 -> A A -> 3 -> C B-> 4 -> A B-> 5 -> D C ->9 -> E D -> 6 -> C D -> 7 ->D D -> 8 -> E Do not extend previous visited nodes! Take care of LEVELS
  45. 45.  STEP 2. Sequences 1,4 1,5,6 1,5,7 1,5,8 2, 3,9  STEP 3. Test Cases All of them should start with A and finish on E 1,5,8 3,9 1,4,3,9 2,1,4,3,9 1,5,7,7,8 1,5,6,9 2,3,9
  46. 46. [week 04 – Class 07 – Thursday] 9/9/10 2:56 PM 1. Functional Testing 1.4. Design of Experiments (DOE)  So far we have covered this: EP BVA Decision Tree and Decision Tables Model-Based Testing State Based Testing (state machine)  Now we will review a new technique: Design of experiments (DoE)  Sometimes called Pair-wise Interaction testing  We have to be careful when we use this tech  Is sound cool but some times could be difficult and cause more problems than solutions  This is for decision tables that are Too large (a lot combinations)  Example One: Let see the following problem: Performance testing for a car: Engine (3 options): 3.0, 3.8, 5.0 Transmission (2 options): manual, auto Body style (2 options): 2Doors, 4Doors Tires (2 options): normal, high performance  Having all this inputs we will have 24 possible configurations (3*2*2*2 options = 24 configurations) But It could be exhaustive  DOE ( it is a sampling technique). Instead of test all, use small sample.
  47. 47.  DOE – Pair Wise Combination Testing  Steps: 1. Identify the inputs parameters to the SUT (systematic under Test) 2. Partition the inputs/ create samples (remember EP) 3. Specify constraints prohibit combinations. In some systems something can not happen together, for instance having two buttons to be pressed at the same time. This would a restriction. 4. Develop test wich satisfy: For any two parameters/inputs P1 and P2 and for any partition value V1 for P1 and V2 for P2 there is a test where P1 has the value V1 and P2 has the value V2.  Example Two: P1 (3 values v1, v2, v3) and P2 (v4, v5) , P3 ( 4 partitions), P4 (1 partition), P5 (2 partition) in SUT (system under test) so 48 different combinations.  Example One (continuation): Take the column with more values P1 combine with a second column. Verify that combinations are satisfied. Engine (3) Transition(2) Body(2) Tires(2) 3.0 A 2D N 3.0 M 4D HP 3.8 A 2D HP 3.8 M 4D N 5.0 A 4D HP 5.0 M 2D N You can do 24 cases (all) but DOE warranty that this 6 are enough.
  48. 48.  Example Three CPU - A B C OS – D E H DB – F, G All combinations are 18  With Pair wise combination testing at least 9.  The parameters with the most number of values (3 each) that would generate 9 combination, this would be the minimum number of combinations.  To create the list, first create all possible combinations of the two partitions with the greater number of values. And then complete the third column assuring that we have a combination that covers all options. NOTE: The table below is not complete, it is needed to complete the third column.  CREATE AN ORTOGONAL ARRAY   Software for this: AETG, Allpairs, tvguwm. A D F A E G A H F B D F B E G B H G C D G C E F C H F 2. Paper 9. The AETG System: An Approach to Testing Based on Combinational Design  Describe a tool to do DOE  Basic combinatorial design paradigm
  49. 49.  See Table 9. 3. Paper 10. Automated Combinatorial Test Methods – Beyond Pair wise Testing  They review pair wise as an introductions  Unlike pair wise technique instead of testing in pair, we can take groups of inputs, for instance P1-P2-P3 or P1-P2-P4. This would help, but it is kind of difficult. So here is were automation comes =)!. 4. Problem about Database  8 inputs with 2 o 3 partitions each one.  Applying pair wise will be 9 test (because we have 2 cases with 3 partitions (take the 2 with more partitions and multiply).  If we use pair-wise combination, we will have as minimum as 9 test cases, but does this test cases would be enough?... No due the dependency we have in the inputs. This happen even if we use a 3x or 4x flavor of pair-wise. Here maybe it is do not going to work … Why? Because dependences This technique must be a very careful!!! Pair wise combinations cold cause problems. THIS is good for configuration testing! Something that make sense I try this and that! This software interact with this device driver by example or for performance or stress testing
  50. 50. This is also used for Interaction Testing between Black boxes.
  51. 51. [week 05 – Class 08 – Tuesday] 9/14/10 2:57 PM 1. Paper 11. An Innovative Approach for Testing Bioinformatics Programs Using Metamorphic Testing.  Bioinformatics programs: organize and analyze large & complex biological dataset.  Invoke complex algorithms to extract useful information.  Test case: input -> expected results  Testing “oracle” problem 1- N-version programming Input to several version of the program (v1, v2, v3) and compare. (version means working implementations)  Metamorphic Testing:  Been able to look at the input domain and search “REASONABLE” outputs.  Define properties about the output.  Another example: graph analysis: search for shortest path (G,a,b)  Another example: graphics (huge data sets) Moving a Light source to cause a shadow  In summary, the idea that testing need inputs and predefined outputs works very well in theory, but there are a lot of fields where this do not work in the same way. So the methods reviewed above could help for these fields. 2. Structure-Based Testing (White Box Testing)  A white box testing methods gets its name due the fact it see the code. Unlike black box testing methods that are all based only in requirements.
  52. 52.  Tend to be techniques to be applied into lower level (small units of code) unit, services, components. Not for entire systems Not fot Integration or Qualification Level of Testing Two categories  Static Techniques Analysis of the code. Example: symbolic execution.  Dynamic Techniques Execute the code: run for test. • Structure - control flow. o Statement coverage o Decision coverage o Decision-Condition Coverage o Multiple decision Coverage • Data flow 3. Statement Coverage  Develop test cases such that every statement is execute at least once  Example if a <10 or b>5 (1) then x<-50 (2) else x <-0; if w = 5 or y>0 (3) then z<-48 (4) else z<-5;
  53. 53.  Flow Chart = Control Flow Diagram. Test predicate (diamond) Statement (oval)  4 statement in the previous code  Path Testing to execute every Path into the code! Testing strategy execute all path into the code  Loops are going to make the numbers of paths bigger. So path coverage is complicate.  Values of (a, b, w, y) to execute all statements (with 2 cases we do it)! a = 0, b=0, w=0, y=0 (execute 1 and 4) a=10, b=4, w=5, y = don’t care (execute 2 and 3) Whit these two tests cases we are covering all the statements.  However, this kind of testing is the minimum need. FAA D0178B standard / best practices: 1. statements 100% statement coverage!  Compilers help us to do that. By example the PROFILER measure the time of each statement. 4. Decision (branch) Coverage  Develop test cases such that each branch (decision point) is executed at least once.  Decision point -> conditional statement = test predicates = diamonds
  54. 54.  EJEMPLO UN MAPA CUADRICULAR CON CALLES Y CADA INTERSECION ES DECIDIR UNA DIRECCION  Even If I use structured programming (no GOTO) then if I have NOT 100% statement coverage then I have 100% test predicated. s1 if x<10 then s2 s3 x =5 I have 100% statement coverage But NOT 100% decision coverage, false case is MISSING!  Statement Coverage Do Not Imply Decision Coverage!  Decision Coverage satisfies Statement but Statement not necessarily satisfies Decision! So Software must cover 100% Decision Coverage 3. Decision-Condition Coverage  Develop test such that each condition in a decision takes on all possible outcomes and each decision takes on all possible outcomes  This is like Decision++ (the second part of the definition, is Decision Coverage) if x<10 or y>50 and z!=0 or flag= T and status = NIL then S1 else S2 2 test to statement coverage 2 test to decision coverage
  55. 55. But that not enough for Decision-Condition Coverage. We need to open the test predicate and look inside: there are 5 conditions! For each condition we need to test their TRUE and FALSE cases We need 2 cases. All false and All True. 4. Multiple Condition  Develop test to execute all combinations of conditions within a decision. This is the most powefull. 5. Example, Test Binary Search Algorithm) (1) start <-1; (2) end <-num; (3) found <- false; (4) while starts <= end and not found //(C1) and (C2) (5) middle <-(start = end)/2 (6) if key > table[middle] //(C3) (7) then start <- middle +1 (8) else if key = table[middle] //(c4) (9) then found <-T (10) LOC <- middle; (11) else end <- middle -1 We have 4 conditions, and we have 11 statements (each row). We have three decision points and we have one case with multiple (2) conditions.
  56. 56. if4 T F if6 T F if8 T F For multiple-condition coverage C1 y C2 make combinations of T y F For conditions C3 y C4 simply try one T and one F of each one Example of Execution 10 20 30 40 50 60 70 Key=55 1 2 3 4 with T, T 5 6T 7 4 done before T, T 5 6F 8 with F 11
  57. 57. 4 con T, T 5 6T 7 4 con F, T  Multiple-condition Coverage: 100% NO  Decision-condition Coverage: 100% NO  Decision Coverage: NO  Statement coverage: NO Now execute this: 10 20 30 40 50 60 70 Key = 40 1 2 3 4 with T, T 5 6 with F 8 with T 9 10 4 with T,F  Coverage is cumulative 4. Multiple-condition Coverage NOT 100% (75%) 3. Decision-condition Coverage 100% 2. Decision Coverage 100% 1. Statement Coverage 100%
  58. 58. [week 05 – Class 09 – Thursday] 9/16/10 2:53 PM 0. Last Class  We review different types of Coverage in particular Control-Flow related.  We review the Binary Search Algorithm  We do not have 100% Multiple-Condition Coverage. What can we do?  Test Case 3. Search for the missing case that give me 100% Multiple- condition coverage. Do not EXIST! Loop Entry 1 2 T T circle middle E diamante T y F if T start E if False diamante if if true found E if false end E So I have 3 path to test T {impact condition 1} FT {impact condition 2} FF {impact condition 1} But any single path affect both conditions! TT FT
  59. 59. TF So F F can not occurs! 1. Problem • Code coverage for midterm exam if x <10 or y>50 (1) then S1 (2) else S2 if w=50 or z>10 (3) then S3 (4) else S4 • Whit this four test cases x y w Z T1 0 0 0 0 T2 11 75 50 12 T3 5 15 10 5 T4 15 75 50 3 The question is, do we have 100% coverage with this four test cases? Statement Coverage We have 4 statements (each s) Statement Test Case 1 T1, T2, T3,T4 2 3 T2, T4 4 T1, T3
  60. 60. Decision Coverage We have 2 decisions points (ifs) Decision Test Case 1T T1(T o F), T2(F o T),T3(T o F),T4(F o T) 1F 2T T2(T o T), T4(F o T), 2F T1 (F o F), T3(F o F), Decision-Condition Coverage We have 4 conditions (underlined) Decision Test Case 1T T1, T3 1F T2, T4 2T T2, T4 2F T1, T3 3T T2, T4 3F T1, T3 4T T2 4F T1, T3, T4 Multiple-Condition Coverage We have 2 multiple conditions (line 1 and 4) Decision Test Case 1 TT 1 TF T1, T3 1 FT T2, T4 1 FF 2 TT T2 2 TF T4 2 FT 2 FF T1,T3
  61. 61. So, we have: • Multiple-Condition Coverage: 5/8 = 62.5% • Decision-Condition Coverage and Decision Coverage: 11/12 = 91.6% • Sentence Coverage: 3/4 = 75% 2. Path Expressions • Convert Control Flow Graph to path expression! a   a   a   1   2   3   4   5   a   a    We can label the edges  We have 4 paths for 1 to 5 Approach to convert a Control Flow Graph (CFG) into a Path Expression: 1. Combine serial links by “multiplying” path expressions 2. combine parallel links by “adding”, “+” their path expressions 3. Remove self loop by replacing them with a link of the form x* • Lets make it more meaningful Example:
  62. 62. ( 1) read (,y) ( 2) while x!=0 ( 3) y <- x + y ( 4) if y>10 ( 5) then put (y) ( 6) else put (x) 1   a   ( 7) if x>10 m   ( 8) then x<-x-5 11   2   ( 9) else x<-x-1 (10) read (x,y) b   (11) end; 3   Convert the code into graph c   d   4   e   5   6   g   l   f   7   h   i   8   9   j   k   10   The nodes on the CFG are not diamonds and ovals, but circles that represent the nodes. The labeling of the edges is totally arbitrary. And then into Path Expression a ( bc (df + eg) (hj + ik) l ) * m So, whit this notation we “somehow” can define how many paths we have from 1 to 11. In this case it is a infinite situation. So if we define how
  63. 63. many times we will run the loop we could mathematically define how many paths we have. 3. Anomaly Detection  Static analysis to find potential errors in code!  It is all about to find anomalies in the code, such as open a file that is already open, or close a file that is already close. Or assign a value to a variable in the statement 10 e.g. x=3 and the next line (11) we assign another different value e.g. x=5.  There is a theorem that helps to define these situations. Huans Theorem Let A,B,C be nonempty sets of character sequences whose smallest string is at least 1 character long. Let T be a 2-character string. Then if T is a substring of AB^NC, then T will appear in AB^2C.  We are going to get sophisticated on the CFG, where edges’ labels are not going to be arbitrary but they will represent what is going on in that path. With this kind of labels it would be easier to find any anomaly. Hvans theorem works with the path expressions to find anomalies.
  64. 64. [week 06 – Class 10 – Tuesday] 9/21/10 2:57 PM 0. Last Class  Continue with Testing Approaches  We looking at control flow graph , path expression, data anomalies, continue with that and introduce symbolic execution. 1. Symbolic Execution  Definition from Paper 3 (homework list): Symbolic execution represents values of program inputs with symbolic values instead of concrete (initialized) data and executes the program by manipulating program expressions involving the symbolic values. (like algebra)  Uses • Probe of correctness • Generate test data • Anomalies Detection Example One: (0) input A, B (1) A <- A + B (2) B <- A - B (3) A <- 2 * A + B (4) C <- A + 4  (0) A0 – value of A after execute step 0 (defined) B0 (defined) C0 (undefined)
  65. 65.  (1) A1 = A0 + B0 B1 = B0 C1 = C0 = (undefined)  (2) A2 = A1 = A0 + B0 B2 = A1 - B1 = A0 + B0 – B0 = A0 C2 = C1 = C2 = (undefined)  (3) A3 = 2 *A2 + B2 = 3A0 + 2B0 B3 = A0 C3 = (undefined)  (4) A4 = 3A0 + 2B0 B4 = A0 C4 = 3A0 + 2B0 + 4 Example Two: F   T   if (x<=0) or (y<=0) then x<-x^2; y <- y^2 1   2   else x<-x+1; y <-y+1; endif; if (x<1) or (y<1) T   F   then x<-x+1; y <- y+1 else x<-x-1; y <-y-1; 3   4   endif; Then, create a Control Flow Diagram (flowchart diagram) Now for symbolic execution we have 4 paths into the diagram
  66. 66. In these cases we need to execute the code and do the symbolic execution for each of the paths. So if we do the True-True case we will have: TT path: x1 = X0^2 y1 = y0^2 x3 = x1+1 = X0^2 + 1 y3 = y1+1 = y0^2 + 1 Example Three: read   read(x, y) F   if(x>7) T   (1) then y<-x+10 (2) else y<-x-10; 1   2   if y > 0 (3) then x<-y+5 T   F   (4) else x<-y-5 3   4   if x + y > 10 (5) then x<-x+10 (6) else x<-x-10; T   F   5   6   TTT Path: x1 = x0 y1 = x0 + 10
  67. 67. x3 = x0 + 15 y3 = x0 + 10 x5 = x3 + 10 = x0+25 y5 = x0 + 10 The numbers correspond to the sentence number  The main use of this is to create test data, in particular path conditions Things gets ugly when we add loops, references inside data structures.  Assuming things simple this is useful to create test data, in particular Path Conditions: Necessary Values for the path to be covered.  Use in conjunction with the branch condition for all the branches along the path with symbolic variables substituted in. 3. Path Condition  What values do we need in a path to be transverse.  Conjunction of the branch conditions for all the branches along the path with symbolic variables substituted in.  Example Two (TT Path):  We identify 4 paths. For TT Path ( (x0<=0) or (y0<=0) )  ( (x1 < 1) or (y1<1) ) Then, What test data we need to drive down the path? ( (x0<=0) or (y0<=0))  ( (x0^2<1) or (y0^2<1) ) x = -0.5
  68. 68. y = -0.5 This are the values to test the path  Example Two (FT Path): Calculate the symbolic variables. x2 = x0 + 1 y2 = y0 + 1 x3 = x0 + 2 y3 = y0 + 2 Then define the path condition ( (x0>0) ^ (y0>0) ) ^ ( (x0<0) or (y0<0) ) Then, create test data for that! This show that, this is a infeasible path, it is a path that can not to be executed (FT is a path invisible THAT CAN NOT BE EXECUTED) 4. Problem in class: Path Condition  Develop the path condition for TTT path in code from example 3 x1 = x0 y1 = x0 + 10 x3 = x0 + 15 y3 = x0 + 10 x5 = x3 + 10 = x0+25 y5 = x0 + 10
  69. 69. And the conditions are: initial: (x0 > 7) and (y1 > 0) and (x3 + y3 > 10) then, (x0 > 7) and (x0 > -10) and (x0 > -15/2) Easy, just make substitutions!
  70. 70. [week 06 – Class 10 – Tuesday] 9/21/10 2:57 PM 1. Complexity Measure: McCabe Number.  Now we are going to talk about Complexity measure this is a tool that take a piece a of code and generate a complexity number for this code. This is know as McCabe number. This is equal to the complexity of a graph.  Code -> # (calculate mcCabe complexity number)  Based of [graph]  The formal definition is V(G) = edges – nodes + 2(connected components) = e –n + 2p  The informal definition is V(G) = test predicates + 1, where test predicates (points of decisions)  For the example 3 is 4 (3if plus one)  If a piece of code has a high complexity number it means that it has a lot of test predicates which means it has a lot of points of decision. On companies that develop software they define a threshold for this complexity and if a code gets a number that is over this threshold the code is flag. It does not means that the code is wrong but it has to be reviewed more careful. McCabe complexity testing Strategy Structured testing V(G) = #of basis paths that must be tested = minimum # of test Let see an example S1 If x<10
  71. 71. Then s2 Else if y>0 1   Then s3 Else s4 T   F   If z=5 Then s6 Else s7 2   T   F   3   4   T   F   We have 6 paths (3 x 2). Complexity = 4 (diamantes + 1) 5   6   So, the recommendation us to test 4 cases  So for the code above we will choose the basis paths, that we call them basis paths.  This paths are the ones which can be used to create all other paths. Approach: 1. Select an arbitrary path 2. Flip the first condition, attempt to return to 1st path 3. Reset first condition, flip the second condition, attempt to return to 1st path 4. Repeat (no audio since last exercise, audio continue here)
  72. 72. Example One:  This are the 4 basis path in the example: (1) F T T (arbitrary) (2) T _T (3) F FT (4) F TF This would be the four basis paths according with McCabe technique. We have 4 basis paths due the fact the complexity of the code is 4. read   Example Two (with 3 consecutive ifs)  Complexity : 4 (diamonds + 1) F   T    For basics Paths 1   2   (1) T T T (2) F T T (3) T F T T   F   (4) T T F 3   4   T   F   5   6  
  73. 73.  In the past classes we talk about Code coverage (white box).  We mention: Control and Data flow coverage.  Data Flow Coverage, look definitions an uses of data and create paths. (next class)
  74. 74. [week 06 – Class 11 – Thursday] 9/23/10 2:55 PM 0. Last class  Control Flow Coverage: McCabe Testing, Decision coverage, Multiple Condition, etc.  Today Data flow coverage. 1. Data Flow Coverage  Even 100% Control Flow coverage, that means even 100% multiple condition coverage do not warranty no errors. Try to do all path coverage it still do not warranty no errors)  Data flow coverage extend the capability of detect errors. Data is going to search data problems like point with cero division Definition: Variable Occurrence Classifications  def: (variable defined) x:=0  c- use: (computational use) variable used in a computation. y = x +1  p-use: (predicate use) variable is used in a predicate. If x<1 Approach: 1 Classify Variables. 2. Construct Control Flow Graph (CFG) with annotations about (def/use)
  75. 75.  For each node: o def(i) : set of variables for node i which are globally defined o c-use(i): set of variables for which node i contains a c-use o p-use(TP) : set of variable for which the TP contains a p-use (TP = test predicate) Example One: if x<10 then y<-0; z<-0 T   F   else y<-5; y<-10 endif 1   2    In this example we can see that we have nodes in our graph that have multiple statements. Node (1) has 2 statements Y=0 and z = 0. 1   Example Two: T   F   (1) get x,z y <- 0 TP1   if x > 10 2   (2) then y <- 15; if z > 0 (3) then w <- y + 1 T   F   (4) else w <- y - 1 TP2   def(1) = {x,y,z} 3   4   def(2) = {y} def(3) = {w} def(4)= {w}
  76. 76. c-use(1) = 0 (nothing) c-use(2) = 0 (nothing) c-use(3) = {y} c-use(4) = {y} p-use(TP1) = {x} p-use(TP2) = {z}  Then, a new definition, CLEAR PATH w. r. t (whit respect to) "some variable" Definition: Clear Path (def-clr) It is a path between two nodes i and j with intermediate nodes (i, N1, N2, … Nm, j) which contains no definitions of x in (N1,… , Nm) Definition: Definition Computation Use (dcu) dcu(x, i) Where i is any node, and x is defined in i dcu(x, i) is the set of all nodes j such that x is in c-use(j) and for which there is a definition-clear path w.r.t x from i to j Definition: Definition Predicate Use Path (dpu) dpu (x, i) [definition /predicate use set] is the set of all TP such that x is in p-use(TP) and for which there is a def- clr path w.r.t. x from i to TP Definition: Use Path (du) It is a path (n,…. Nj, Nk) is a du path w.r.t. x if N1 has a global definition of X and either: 1. Nk has a c-use of x and (n1, … nj) is a def-clr path w.r.t. x 2. Mk is a TP and has a p-use of X and (ni, ..nj) is a def-clr path w.r.t. x (the path from definition to use )
  77. 77. Go back to Example Two: def(1) = {x, y, z} def(2) = {y} def(3) = {w} def(4)= {w} 1   c-use(1) = 0 (nothing) T   F   c-use(2) = 0 (nothing) c-use(3) = {y} TP1   c-use(4) = {y} 2   p-use(Tp1) = {x} p-use(Tp2) = {z} T   F    dcu(var, node) nodes that use the “var” defined in “node” TP2   dcu (x,1) = empty 3   4   dcu (z,1) = empty dcu (y,1) = {3,4} dcu (y,2) = {3,4} dcu (w,3) = vacio dcu (w,4) = vacio  dpu(var, node) dpu (x,1) = {TP1} dpu(z,1) = {Tp2}  Test Coverage Criteria for data flow means "all-uses coverage" We want 100%
  78. 78. 3. All-Uses Coverage  Goal: develop test cases such that for each definition, there is a definition use path (du) to every use of the definition. 1   Problem One get x, y; T   F   a <- 0; b <- 0; TP1   2   3   if x>10 then w <-a + 1; b <- 4 else w <-b + 1; a <- 4; T   F   if y>10 then z <- a + w else z <- b + w; TP2   4   5   def(1) = {x,y,a,b} def(2) = {w,b} def(3) = {w,a} def(4) = {z} def(5) = {z} c-use(1) = empty c-use(2) = {a} c-use(3) = {b} c-use(4) = {a, w} c-use(5) = {b, w} p-use(TP1) = {x} p-use(TP2) = {y}  Create a dcu for each pair def(N) = {i1, i2….}
  79. 79.  Search {i1. I2…} en c-use(M) = {i1, i2…} and add the numbers M to the set dcu(x,1) = {} dcu(y,1) = {} dcu(a,1) = {2, 4} dcu(b,1) = {3, 5} dcu(w,2) = {4} dcu(b,2) = {5} dcu(w,3) = {4} dcu(a,3) = {5} dcu(z,4) = {} dcu(z,5) = {} dpu(x,1) = TP1 dpu(y,1) = TP2 X Y Coverage Test 1. TT 15 15 Def(a1) & c-use(a2) & c-use(a4) Test 2. FT 5 15 Def(a1) & c-use(a4)  For the 4 paths. Test EVERY definition  Give me all-uses coverage with respect to a but not respect to b You need to do this for all-uses. The same use cases give me all te missing ones  X Y Test 1 15 15 Test 2 5 15 Test 3 5 5 Test 4 15 5 What control – Flow Coverage do I have? 100% statement coverage
  80. 80. 100% condition coverage All definitions criteria is satisfied if for each definition in the program some use of this definition is executed by at least one input in the test suite.

×