Black box testing (an introduction to)
Upcoming SlideShare
Loading in...5
×
 

Black box testing (an introduction to)

on

  • 1,683 views

An introduction to black box testing, as part of the Advanced Software Engineering course at www.di.univaq.it/muccini/SE+/2012

An introduction to black box testing, as part of the Advanced Software Engineering course at www.di.univaq.it/muccini/SE+/2012

Statistics

Views

Total Views
1,683
Views on SlideShare
1,683
Embed Views
0

Actions

Likes
1
Downloads
25
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Black box testing (an introduction to) Black box testing (an introduction to) Presentation Transcript

  • Università degli Studi dell’AquilaL21: Black Box/Functional Testing Henry Muccini DISIM, University of L’Aquila www.henrymuccini.com, henry.muccini@univaq.it
  • The material in these slides may be freely reproduced anddistributed, partially or totally, as far as an explicitreference or acknowledge to the material author ispreserved.Partially based on material from Alex Orso, Mauro Pezze’,and Michal Young.
  • AGENDAWhite box testing Representative values Systematic vs Random Testing Equivalence Partitioning Category Partition Method
  • Where is the boundary between testing and other (analysis/verification) methods? What distinguishes software testing from “other” approaches is execution. This requires the ability to: → launch the tests on some executable version (traceability problem), and → analyse (observe and evaluate) the results (not granted for embedded systems)I am not saying testing is superior and we should disregard other approaches: on the contrary, testing and other methods should complement/support each other
  • Any software test campaign involves a trade-off between → Limiting resource utilization (time, effort) ⇒ as few tests as possible → Augmenting our confidence ⇒ as many tests as possibleTwo research challenges: → Determining a feasible and meaningful stopping rule → Evaluating test effectiveness (reliability, “coverage notion”, ....very tough problem
  • BLACK BOX/FUNCTIONAL TESTING
  • Functional testing: Deriving test cases from programspecifications ─ Functional refers to the source of information used in test case design, not to what is testedAlso known as: → specification-based testing (from specifications) → black-box testing (no view of the code)Functional specification = description of intendedprogram behavior → either formal or informal(C) 2007 MAURO PEZZÈ & MICHAL YOUNG
  • Exploit some knowledge to choose samples that are morelikely to include “special” or trouble-prone regions of theinput space → Failures are sparse in the whole input space ... → ... but we may find regions in which they are denseDesirable case: Each fault leads to failures that are dense(easy to find) in some class of inputs → sampling each class in the quasi-partition selects at least one input that leads to a failure, revealing the fault → seldom guaranteed; we depend on experience-based heuristics (C) 2007 MAURO PEZZÈ & MICHAL YOUNG
  • Failures are sparse Failure (valuable test case) ... but dense in some in the space of parts of the space No failure possible inputs ...The space of possible input values (the haystack) If we systematically test some Functional testing is one way of cases from each part, we will drawing pink lines to isolate include the dense parts regions with likely failures (C) 2007 MAURO PEZZÈ & MICHAL YOUNG
  • Random (uniform): → Pickpossible inputs uniformly → Avoids designer bias ─ A real problem: The test designer can make the same logical mistakes and bad assumptions as the program designer (especially if they are the same person) → But treats all inputs as equally valuable(C) 2007 MAURO PEZZÈ & MICHAL YOUNG
  • Defects are not distributed uniformly!•  Assume our printSum program fails if bothinputs are 0!•  Random sampling is unlikely to choose a = 0and b = 0!We want bias!!
  • Systematic (non-uniform): → Try to select inputs that are especially valuable → Usually by choosing representatives of classes that are apt to fail often or not at allFunctional testing is systematic testing
  • •An independently testable feature (ITF) is afunctionality that can be tested independently of otherfunctionalities of the software under test.•It need not correspond to a unit or subsystem of thesoftware.•For example, a file sorting utility may be capable ofmerging two sorted files, and it may be possible to testthe sorting and merging functionalities separately,even though both features are implemented by muchof the same source code.
  • It is recommended to devise separate test cases foreach functionality of the system, whenever possible.The design of different tests for differentfunctionalities can simplify the test generationproblemThus, identifying functional features does notcorrespond to identifying single modules at the designlevel, but rather to suitably slicing the specifications tobe able to attack their complexity incrementallyFeature-based view of the system
  • Functional testing uses the specification (formal orinformal) to partition the input space → E.g., specification of “roots” program suggests division between cases with zero, one, and two real rootsTest each category, and boundaries betweencategories → No guarantees, but experience suggests failures often lie at the boundaries (as in the “roots” program)(C) 2007 MAURO PEZZÈ & MICHAL YOUNG
  • •Try to select inputs that are especially valuable! • Usually by choosing representatives of equivalence classes!•Assumption: inputs in different categories behaveequivalently ⇒I can select one input per category
  • Consider a software module that is intended to accept thename of a grocery item and a list of the different sizes theitem comes in, specified in ounces. The specifications statethat the item name is to be alphabetic characters 2 to 15characters in length. Each size may be a value in the rangeof 1 to 48, whole numbers only. The sizes are to beentered in ascending order (smaller sizes first). Amaximum of five sizes may be entered for each item. Theitem name is to be entered first, followed by a comma,then followed by a list of sizes. A comma will be used toseparate each size. Spaces (blanks) are to be ignoredanywhere in the input.Taken from: http://users.csc.calpoly.edu/~jdalbey/205/Resources/grocerystore.
  • Consider a software module that is intended to accept thename of a grocery item and a list of the different sizes theitem comes in, specified in ounces.The specifications state that the item name is to be alphabeticcharacters 2 to 15 characters in length. Each size may be avalue in the range of 1 to 48, whole numbers only.The sizes are to be entered in ascending order (smaller sizesfirst). A maximum of five sizes may be entered for each item.The item name is to be entered first, followed by a comma,then followed by a list of sizes. A comma will be used toseparate each size. Spaces (blanks) are to be ignoredanywhere in the input.
  • Item name is alphabetic (valid)Item name is not alphabetic (invalid)Item name is less than 2 characters in length (invalid)Item name is 2 to 15 characters in length (valid)Item name is greater than 15 characters in length (invalid)Size value is less than 1 (invalid)Size value is in the range 1 to 48 (valid)Size value is greater than 48 (invalid)Size value is a whole number (valid)Size value is a decimal (invalid)Size value is numeric (valid)Size value includes nonnumeric characters (invalid)Size values entered in ascending order (valid)Size values entered in nonascending order (invalid)No size values entered (invalid)One to five size values entered (valid)More than five sizes entered (invalid)Item name is first (valid)…
  • Expected# Test Data Classes Covered Outcome1 xy,1 T 1,4,7,9,11,13,16,18,20,222 AbcDefghijklmno,1,2,3 ,4,48 T 1,4,7,9,11,13,16,18,20,233 a2x,1 F 24 A,1 F 35 abcdefghijklmnop F 56 Xy,0 F 67 XY,49 F 88 Xy,2.5 F 109 xy,2,1,3,4,5 F 1410 Xy F 1511 XY,1,2,3,4,5,6 F 1712 1,Xy,2,3,4,5 F 1913 XY2,3,4,5,6 F 2114 AB,2#7 F 12
  • Item name is alphabetic (valid)Item name is not alphabetic (invalid)Item name is less than 2 characters in length (invalid)Item name is 2 to 15 characters in length (valid)Item name is greater than 15 characters in length (invalid)Size value is less than 1 (invalid)Size value is in the range 1 to 48 (valid)Size value is greater than 48 (invalid)Size value is a whole number (valid)Size value is a decimal (invalid)Size value is numeric (valid)Size value includes nonnumeric characters (invalid)Size values entered in ascending order (valid)Size values entered in nonascending order (invalid)No size values entered (invalid)One to five size values entered (valid)More than five sizes entered (invalid)Item name is first (valid)…
  • The main idea is to partition the input domain offunction being tested, and then select test data foreach class of the partition.The problem of all the existing techniques is the lack ofsystematic.Solution: the Category Partition Method
  • Steps: ITF (Independently testable feature) Identify parameters and Environmental conditions Parameters: Explicit input to the functional unit Environmental Conditions: Characteristics of the system’s state Find categories that characterize each parameter and environment condition. Every category should be partitioned into distinct choices Test Spec
  • Command: find (a pattern into a file) Syntax: find <pattern> <file>Function: The find command is used to locate one or more instance of a givenpattern in a text file. All lines in the file that contain the pattern are written tostandard output. A line containing the pattern is written only once, regardless of thenumber of times the pattern occurs in it.The pattern is any sequence of characters whose length does not exceed themaximum length of a line in the file .To include a blank in the pattern, the entirepattern must be enclosed in quotes (“).To include quotation mark in the pattern ,twoquotes in a row (“ “) must be used.
  • Example:find john myfile display lines in the file myfile which contain johnfind “john smith” in myfile display lines in the file myfile which contain john smithfind “john”” smith” in myfile display lines in the file myfile which contain john” smith
  • Parameters (explicit input to the functional unit)Example: sizefind john myfile display lines in the file myfile which contain john quote emb blankfind “john smith” in myfile display lines in the file myfile which contain john smith emb quotesfind “john”” smith” in myfile display lines in the file myfile which contain john” smith
  • Parameters:Pattern size: empty single character many character longer than any line in the fileQuoting: pattern is quoted pattern is not quoted pattern is improperly quotedEmbedded blanks: no embedded blank one embedded blank several embedded blanks
  • Embedded quotes: no embedded quotes one embedded quotes several embedded quotes File name: good file name no file with this nameEnvironments (characteristics of the system’s state): Number of occurrence of pattern in file: none exactly one Total Tests frames: more than one 1944 Pattern occurrences on target line: one more than one
  • Pattern size : emptyQuoting : pattern is quotedEmbedded blanks : several embedded blanksEmbedded quotes : no embedded quoteFile name : good file nameNumber of occurrence of pattern in file : nonePattern occurrence on target line : oneHowever, not all combinations of value classescorrespond to reasonable test case specifications!!!
  • • Properties– [property A, B, …]– A and B are property names– E.g., [property Empty]• Selector expression– [if A]– E.g., [if Empty]
  • Parameters: Pattern size: empty [ property Empty ] single character [ property NonEmpty ] many character [ property NonEmpty ] longer than any line in the file [ property NonEmpty ] Quoting: pattern is quoted [ property quoted ] pattern is not quoted [ if NonEmpty ] pattern is improperly quoted [ if NonEmpty ] Embedded blanks: no embedded blank [ if NonEmpty ] one embedded blank [ if NonEmpty and Quoted ] several embedded blanks [ if NonEmpty and Quoted ]
  • Embedded quotes: no embedded quotes [ if NonEmpty ] one embedded quotes [ if NonEmpty ] several embedded quotes [ if NonEmpty ]File name: good file name no file with this nameEnvironments: Total Tests frames: Number of occurrence of pattern in file: 678 none [ if NonEmpty ] exactly one [ if NonEmpty ] [ property Match] more than one [ if NonEmpty ] [ property Match ] Pattern occurrences on target line: one [ if Match ] more than one [ if Match ]
  • The label [error] indicates a value class that need betried only once, in combination with non-error valuesof other parameters.
  • Parameters: Pattern size: empty [ property Empty ] single character [ property NonEmpty ] many character [ property NonEmpty ] longer than any line in the file [ error ] Quoting: pattern is quoted [ property quoted ] pattern is not quoted [ if NonEmpty ] pattern is improperly quoted [ error ] Embedded blanks: no embedded blank [ if NonEmpty ] one embedded blank [ if NonEmpty and Quoted ] several embedded blanks [ if NonEmpty and Quoted ]
  • Embedded quotes: no embedded quotes [ if NonEmpty ] one embedded quotes [ if NonEmpty ] several embedded quotes [ if NonEmpty ] [ single ]File name: good file name no file with this name [ error ]Environments: Total Tests frames: Total Tests frames: Number of occurrence of pattern in file: 40 125 none [ if NonEmpty ] exactly one [ if NonEmpty ] [ property Match] more than one [ if NonEmpty ] [ property Match ] Pattern occurrences on target line: [ single ] one [ if Match ] more than one [ if Match ] [ single ]
  • • Use a constraint solver• Choose specific values that satisfy theconstraints
  • A completely different approach: →We cannot realistically presume to find and remove the last failure. →Then, we should invest test resources to find out the “biggest” ones.“Goodness” here means to minimise the userperceived faults, i.e. →Tryto find earlier those bugs that are more frequently encountered (neglect “small” bugs) →Not suitable for safety critical applications
  • Identify: • Typical usage scenario • Frequency of those scenariosTest more, more frequent scenarios