Software Testing Testing: Our Experiences

601 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
601
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
17
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Software Testing Testing: Our Experiences

  1. 1. Testing: Our Experiences Test Case Software Testing Software to be tested Output Test Case When to Stop? A Real Testing Example Generation Test Cases Test Case Just a list. {1,3,2} A sorted list. SPECS: {1,2,3} Takes a list Repeated entry. Software {3,2,3} of numbers; to be tested Empty list. returns a {} Sorting Negative numbers. Sorting sorted list. Verification {-1, -2} Program Program Output {1,{} 3} {2, 2, 3, {-2, -1} Output No Test Coverage Enough? Yes Test Case Automated Testing Automated Testing Generation Test Case Test Case Generator Test Case Software Software to be tested to be tested Verification Test Specs Verifier Output Output OR Test Oracle No Test Coverage Coverage Enough? Evaluator Yes 1
  2. 2. Testing the New Version Regression Testing Original Original Original Original Test Software Test Software Cases Cases New New Modified Test Modified Test Software Software Cases Cases What is Testing? Types of Testing • Process of determining whether a • Execution-based Testing task has been correctly carried out • Non-execution based Testing [Schach ’96] • Goals of testing – Reveal Faults • Correctness • Reliability • Usability Conflicting Goals? • Robustness • Performance • Discussion Execution-based Testing Black-box Testing • Generating and Executing Test • Discussion: MAC/ATM Machine Cases on the Software Example • Types of Execution-based Testing – Specs – Testing to Specifications • Cannot withdraw more than $300 • Black-box Testing • Cannot withdraw more than your account balance – Testing to Code Balance • Glass-box (White-box) Testing x Software 2
  3. 3. White-box Testing Discussion • Example • Which is superior? Generate test cases x: 1..1000; to cover each statement • Each technique has its strengths – 1 INPUT-FROM-USER(x); Use both If (x <= 300) { 2 INPUT-FROM-FILE(BALANCE); If (x <= BALANCE) 3 GiveMoney x; 4 else Print “You don’t have $x in your account!!”} else 5 Print “You cannot withdraw more than $300”; 6 Eject Card; Determining Adequacy Surprise Quiz • Determine test cases so that each • Statement coverage print statement is executed at • Branch coverage least once begin • Path coverage input(x); • All-def-use-path coverage if (x < 100) if x>=100 x<100 print “Line 1”; 1 x>=100 if else { x<50 x>=100 x>=50 if (x < 50) print “Line 2” else print “Line 3”; 2 3 } end Non-execution Based Simulation • Walkthroughs • Integration with system hardware is – Manual simulation by team leader central to the design • Inspections – Developer narrates the reading • Model the external hardware • Key Idea • Model the interface – Review by a team of experts: Syntax checker? • Code Readings • Examples • Formal Verification of Correctness • Discussion – Very Expensive – Justified in Critical Applications • Semi-formal: Some Assertions 3
  4. 4. Boundary-value Analysis Testing Approaches • Partition the program domain into • Top-down input classes • Bottom-up • Choose test data that lies both • Big Bang inside each input class and at the boundary of each class • Unit testing • Select input that causes output at each class boundary and within each • Integration testing class • Stubs • Also known as stress testing • System testing Mutation Testing Test Case Generation • Errors are introduced in the • Test Input to the Software program to produce “mutants” • Some researchers/authors also • Run test suite on all mutants and define the test case to contain the the original program expected output for the test input Category-partition Method Steps • Key idea • Decompose the functional specification into functional units – Method for creating functional test – Characteristics of functional units suites • They can be tested independently – Role of test engineer • Examples – A top-level user command • Analyze the system specification – Or a function • Write a series of formal test specifications • Decomposition may require several stages – Automatic generator • Similar to high-level decomposition done • Produces test descriptions by software designers – May be reused, although independent decomposition is recommended 4
  5. 5. Steps Steps • Examine each functional unit • “Test cases are chosen to maximize – Identify parameters chances of finding errors” • Explicit input to the functional unit • For each parameter & environmental – Environmental conditions condition – Find categories • Characteristics of the system’s state • Major property or characteristic • Test Cases • Examples – Browsers, Operating Systems, array size – Specific values of parameters • For each category – And environmental conditions – Find choices » Examples: (IE 5.0, IE 4.5, Netscape 7.0), (Windows NT, Linux), (100, 0, -1) Steps AI Planning Method • Develop “Formal Test Specification” • Key Idea for each functional unit – Input to Command-driven software is a – List of categories sequence of commands – Lists of choices within each category – The sequence is like a plan • Constraints • Scenario to test • Automatically produces a set of – Initial state “test frames” – Goal state – Consists of a set of choices Example Preconditions & Effects • VCR command-line software • Rewind – Precondition: If at end of tape • Commands – Effects: At beginning of tape – Rewind • Play • If at the end of tape – Precondition: If at beginning of tape – Play – Effects: At end of tape • If fully rewound • Eject – Eject – Precondition: If at end of tape • If at the end of tape – Effects: VCR has no tape – Load • Load • If VCR has no tape – Precondition: If VCR has no tape – Effects: VCR has tape 5
  6. 6. Preconditions & Effects Initial and Goal States • Rewind • Initial State – Precondition: end_of_tape – end_of_tape – Effects: ¬end_of_tape • Play • Goal State – Precondition: ¬end_of_tape – ¬end_of_tape – Effects: end_of_tape • Plan? • Eject – Precondition: end_of_tape – Rewind – Effects: ¬has_tape • Load – Precondition: ¬has_tape – Effects: has_tape Initial and Goal States Test Coverage & Adequacy • Initial State • How much testing is enough? – ¬end_of_tape & has_tape • When to stop testing • Goal State • Test data selection criteria – ¬has_tape • Test data adequacy criteria • Plan? – Stopping rule – Degree of adequacy – Play – Eject • Test coverage criteria • Objective measurement of test quality Preliminaries Goodenough & Gerhart [‘75] • Test data selection • What is a software test adequacy – What test cases criterion • Test data adequacy criteria – Predicate that defines “what – When to stop testing properties of a program must be • Examples exercised to constitute a thorough – Statement Coverage test”, i.e., one whose successful execution implies no errors in a tested – Branch coverage program – Def-use coverage – Path coverage 6
  7. 7. Uses of test adequacy Categories of Criteria • Objectives of testing • Specification based – All-combination criterion • In terms that can be measured • choices – For example branch coverage – Each-choice-used criterion • Two levels of testing • Program based – First as a stopping rule – Statement – Then as a guideline for additional test – Branch cases • Note that in both the above types, the correctness of the output must be checked against the specifications Others • Random testing • Statistical testing • Interface based 7

×