Chapter 9
Testing the System
Shari L. Pfleeger
Joann M. Atlee
4th Edition
4th Edition
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
Contents
9.1 Principles of system testing
9.2 Function testing
9.3 Performance testing
9.4 Reliability, availability, and maintainability
9.5 Acceptance testing
9.6 Installation testing
9.7 Automated system testing
9.8 Test documentation
9.9 Testing safety-critical systems
9.10 Information systems example
9.11 Real-time example
9.12 What this chapter means for you
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
Chapter 9 ObjectivesFunction testingPerformance testingAcceptance testingSoftware reliability, availability, and maintainabilityInstallation testingTest documentationTesting safety-critical systems
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.1 Principles of System Testing
Source of Software Faults During Development
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.1 Principles of System Testing System Testing ProcessFunction testing: does the integrated system perform as promised by the requirements specification?Performance testing: are the non-functional requirements met?Acceptance testing: is the system what the customer expects?Installation testing: does the system run at the customer site(s)?
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.1 Principles of System Testing
System Testing Process (continued)Pictorial representation of steps in testing process
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.1 Principles of System Testing
Techniques Used in System TestingBuild or integration planRegression testingConfiguration managementversions and releasesproduction system vs. development systemdeltas, separate files and conditional compilationchange control
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.1 Principles of System Testing
Build or Integration PlanDefine the subsystems (spins) to be testedDescribe how, where, when, and by whom the tests will be conducted
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.1 Principles of System Testing
Example of Build Plan for Telecommunication SystemSpinFunctionsTest StartTest EndOExchange1 September15 September1Area code30 September15 October2State/province/district25 October5 November3Country10 November20 November4International1 December15 December
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.1 Principles of System Testing
Example Number of Spins for Star Network
Spin 0: test the central computer’s general functions
Spin 1: test the central computer’s message-translation function
Spin 2: test the central computer’s message-assimilation function
Spin 3: test each outlying computer in the stand alone mode
Spin 4: test the outlying c.
internship ppt on smartinternz platform as salesforce developer
Chapter 9Testing the SystemShari L. PfleegerJo.docx
1. Chapter 9
Testing the System
Shari L. Pfleeger
Joann M. Atlee
4th Edition
4th Edition
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
Contents
9.1 Principles of system testing
9.2 Function testing
9.3 Performance testing
9.4 Reliability, availability, and maintainability
9.5 Acceptance testing
2. 9.6 Installation testing
9.7 Automated system testing
9.8 Test documentation
9.9 Testing safety-critical systems
9.10 Information systems example
9.11 Real-time example
9.12 What this chapter means for you
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
Chapter 9 ObjectivesFunction testingPerformance
testingAcceptance testingSoftware reliability, availability, and
maintainabilityInstallation testingTest documentationTesting
safety-critical systems
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.1 Principles of System Testing
Source of Software Faults During Development
3. Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.1 Principles of System Testing System Testing
ProcessFunction testing: does the integrated system perform as
promised by the requirements specification?Performance
testing: are the non-functional requirements met?Acceptance
testing: is the system what the customer expects?Installation
testing: does the system run at the customer site(s)?
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.1 Principles of System Testing
System Testing Process (continued)Pictorial representation of
steps in testing process
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.1 Principles of System Testing
Techniques Used in System TestingBuild or integration
planRegression testingConfiguration managementversions and
releasesproduction system vs. development systemdeltas,
separate files and conditional compilationchange control
4. Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.1 Principles of System Testing
Build or Integration PlanDefine the subsystems (spins) to be
testedDescribe how, where, when, and by whom the tests will be
conducted
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.1 Principles of System Testing
Example of Build Plan for Telecommunication
SystemSpinFunctionsTest StartTest EndOExchange1
September15 September1Area code30 September15
October2State/province/district25 October5
November3Country10 November20 November4International1
December15 December
5. Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.1 Principles of System Testing
Example Number of Spins for Star Network
Spin 0: test the central computer’s general functions
Spin 1: test the central computer’s message-translation function
Spin 2: test the central computer’s message-assimilation
function
Spin 3: test each outlying computer in the stand alone mode
Spin 4: test the outlying computer’s message-sending function
Spin 5: test the central computer’s message-receiving function
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.1 Principles of System Testing
Regression TestingIdentifies new faults that may have been
introduced as current one are being correctedVerifies a new
version or release still performs the same functions in the same
manner as an older version or release
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
6. 9.1 Principles of System Testing
Regression Testing StepsInserting the new codeTesting
functions known to be affected by the new codeTesting essential
function of m to verify that they still work properlyContinuing
function testing m + 1
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.1 Principles of System Testing
Configuration ManagementVersions and releasesProduction
system vs. development systemDeltas, separate files and
conditional compilationChange control
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.1 Principles of System Testing
Test TeamProfessional testers: organize and run the
testsAnalysts: who created requirementsSystem designers:
understand the proposed solutionConfiguration management
specialists: to help control fixesUsers: to evaluate issues that
arise
7. Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.2 Function Testing
Purpose and RolesCompares the system’s actual performance
with its requirementsDevelops test cases based on the
requirements document
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.2 Function Testing
Cause-and-Effect GraphA Boolean graph reflecting logical
relationships between inputs (causes), and the outputs (effects)
or transformations (effects)
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.2 Function Testing
Notation for Cause-and-Effect Graph
8. Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.2 Function Testing
Cause-and-Effect Graphs ExampleINPUT: The syntax of the
function is LEVEL(A,B) where A is the height in meters of the
water behind the dam, and B is the number of centimeters of
rain in the last 24-hour periodPROCESSING: The function
calculates whether the water level is within a safe range, is too
high, or is too lowOUTPUT: The screen shows one of the
following messages
1. “LEVEL = SAFE” when the result is safe or low
2. “LEVEL = HIGH” when the result is high
3. “INVALID SYNTAX”
depending on the result of the calculation
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.2 Function Testing
Cause-and-Effect Graphs Example (Continued)Causes
The first five characters of the command “LEVEL”
The command contains exactly two parameters separated by a
comma and enclosed in parentheses
The parameters A and B are real numbers such that the water
9. level is calculated to be LOW
The parameters A and B are real numbers such that the water
level is calculated to be SAFE
The parameters A and B are real numbers such that the water
level is calculated to be HIGH
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.2 Function Testing
Cause-and-Effect Graphs Example (Continued)Effects
1. The message “LEVEL = SAFE” is displayed on the screen
2. The message “LEVEL = HIGH” is displayed on the screen
The message “INVALID SYNTAX” is printed out
Intermediate nodes
1. The command is syntactically valid
2. The operands are syntactically valid
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.2 Function Testing
Cause-and-Effect Graphs of LEVEL Function Example
Exactly one of a set of conditions can be invoked
At most one of a set of conditions can be invoked
At least one of a set of condition can be invoked
10. One effects masks the observance of another effect
Invocation of one effect requires the invocation of another
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.2 Function Testing
Decision Table for Cause-and-Effect Graph of LEVEL
FunctionTest 1Test 2Test 3Test 4Test 5Cause 1IIISICause
2IIIXSCause 3ISSXXCause 4SISXXCause 5SSIXXEffect
1PPAAAEffect 2AAPAAEffect 3AAAPP
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.3 Performance Tests
Purpose and RolesUsed to examine the calculationthe speed of
responsethe accuracy of the resultthe accessibility of the
dataDesigned and administrated by the test team
11. Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.3 Performance Tests
Types of Performance Tests
Stress tests
Volume tests
Configuration tests
Compatibility tests
Regression tests
Security tests
Timing tests
Environmental tests
Quality tests
Recovery tests
Maintenance tests
Documentation tests
Human factors (usability) tests
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.4 Reliability, Availability, and Maintainability
DefinitionSoftware reliability: operating without failure under
given condition for a given time intervalSoftware availability:
operating successfully according to specification at a given
point in timeSoftware maintainability: for a given condition of
use, a maintenance activity can be carried out within stated time
12. interval, procedures and resources
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.4 Reliability, Availability, and Maintainability
Different Level of Failure SeverityCatastrophic: causes death or
system lossCritical: causes severe injury or major system
damageMarginal: causes minor injury or minor system
damageMinor: causes no injury or system damage
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.4 Reliability, Availability, and Maintainability
Failure DataTable of the execution time (in seconds) between
successive failures of a command-and-control
systemInterfailure Times (Read left to right, in
rows)330113811159291112151385077241088867012026114325
55242684221801011466001536552426822765176584573009726
34522551971936798161351148212331343571932363136974802
32330365122254310165293794412981029030052928116082810
11445296175510641783860983707338687242323293014618431
22611800865143530143108031101247943700875245729189744
73864461229909481082227548255091001010713717906150332
110456485485116018644116
13. Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.4 Reliability, Availability, and Maintainability
Failure Data (Continued)Graph of failure data from previous
table
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.4 Reliability, Availability, and Maintainability
Uncertainty Inherent from Failure DataType-1 uncertainty: how
the system will be usedType-2 uncertainty: lack of knowledge
about the effect of fault removal
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
14. 9.4 Reliability, Availability, and Maintainability
Measuring Reliability, Availability, and MaintainabilityMean
time to failure (MTTF)Mean time to repair (MTTR)Mean time
between failures (MTBF)
MTBF = MTTF + MTTRReliability R =
MTTF/(1+MTTF)Availability A = MTBF
(1+MTBF)Maintainability M = 1/(1+MTTR)
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.4 Reliability, Availability, and Maintainability
Reliability Stability and GrowthProbability density function f or
time t, f (t): when the software is likely to failDistribution
function: the probability of failureF(t) = ∫ f (t) dtReliability
Function: the probability that the software will function
properly until time tR(t) = 1- F(t)
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.4 Reliability, Availability, and Maintainability
Uniformity Density FunctionUniform in the interval from
t=0..86,400 because the function takes the same value in that
interval
15. Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.4 Reliability, Availability, and Maintainability
Sidebar 9.4 Difference Between Hardware and Software
ReliabilityComplex hardware fails when a component breaks
and no longer functions as specifiedSoftware faults can exist in
a product for long time, activated only when certain conditions
exist that transform the fault into a failure
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.4 Reliability, Availability, and Maintainability
Reliability PredictionPredicting next failure times from past
history
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.4 Reliability, Availability, and Maintainability
16. Elements of a Prediction SystemA prediction model: gives a
complete probability specification of the stochastic processAn
inference procedure: for unknown parameters of the model
based on values of t₁, t₂, …, ti-1A prediction procedure:
combines the model and inference procedure to make
predictions about future failure behavior
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.4 Reliability, Availability, and Maintainability
Reliability ModelThe Jelinski-Moranda model: assumesno type-
2 uncertaintycorrections are perfectfixing any fault contributes
equally to improving the reliabilityThe Littlewood modeltreats
each corrected fault’s contribution to reliability as independent
variableuses two source of uncertainty
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.4 Reliability, Availability, and Maintainability
Successive Failure Times for Jelinski-MorandaIMean Time to
ith failureSimulated Time to ith
Failure12211224413261342845303063377737118426494854105
634116718312838313111171416719015333436
17. Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.5 Acceptance Tests
Purpose and RolesEnable the customers and users to determine
if the built system meets their needs and expectationsWritten,
conducted and evaluated by the customers
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.5 Acceptance Tests
Types of Acceptance TestsPilot test: install on experimental
basisAlpha test: in-house testBeta test: customer pilotParallel
testing: new system operates in parallel with old system
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
18. 9.4 Reliability, Availability, and Maintainability
Result of Acceptance TestsList of requirementsare not
satisfiedmust be deletedmust be revisedmust be added
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.6 Installation TestingBefore the testingConfigure the
systemAttach proper number and kind of devicesEstablish
communication with other systemThe testingRegression tests: to
verify that the system has been installed properly and works
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.7 Automated System Testing
SimulatorPresents to a system all the characteristics of a device
or system without actually having the device or system
availableLooks like other systems with which the test system
must interfaceProvides the necessary information for testing
without duplication the entire other system
19. Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.8 Test DocumentationTest plan: describes system and plan
for exercising all functions and characteristicsTest specification
and evaluation: details each test and defines criteria for
evaluating each featureTest description: test data and
procedures for each testTest analysis report: results of each test
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.8 Test Documentation
Documents Produced During Testing
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.8 Test Documentation
Test PlanThe plan begins by stating its objectives, which
shouldguide the management of testingguide the technical effort
required during testingestablish test planning and
schedulingexplain the nature and extent of each testexplain how
the test will completely evaluate system function and
performancedocument test input, specific test procedures, and
20. expected outcomes
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.8 Test Documentation
Parts of a Test Plan
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.8 Testing Documentation
Test-Requirement Correspondence Chart
TestRequirement 2.4.1:
Generate and
Maintain Database Requirement 2.4.2:
Selectively Retrieve
DataRequirement 2.4.3:
Produced Specialized
Reports1. Add new recordX2. Add fieldX3. Change fieldX4.
Delete recordX5. Delete fieldX6. Create indexXRetrieve record
with a requested7. Cell numberX8. Water heightX9. Canopy
heightX10. Ground coverX11, Percolation rateX12. Print full
databaseX13. Print directoryX14. Print keywordsX15. Print
simulation summaryX
21. Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.8 Test Documentation
Sidebar 9.8 Measuring Test Effectiveness and EfficiencyTest
effectiveness can be measured by dividing the number of faults
found in a given test by the total number of faults foundTest
efficiency is computed by dividing the number of faults found
22. in testing by the effort needed to perform testing
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.8 Test Documentation
Test DescriptionIncludingthe means of controlthe datathe
procedures
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.8 Test Documentation
Test Description Example
INPUT DATA:
Input data are to be provided by the LIST program. The
program generates randomly a list of N words of alphanumeric
characters; each word is of length M. The program is invoked
by calling
RUN LIST(N,M)
in your test driver. The output is placed in global data area
LISTBUF. The test datasets to be used for this test are as
follows:
Case 1: Use LIST with N=5, M=5
Case 2: Use LIST with N=10, M=5
Case 3: Use LIST with N=15, M=5
23. Case 4: Use LIST with N=50, M=10
Case 5: Use LIST with N=100, M=10
Case 6: Use LIST with N=150, M=10
INPUT COMMANDS:
The SORT routine is invoked by using the command
RUN SORT (INBUF,OUTBUF) or
RUN SORT (INBUF)
OUTPUT DATA:
If two parameters are used, the sorted list is placed in OUTBUF.
Otherwise, it is placed in INBUF.
SYSTEM MESSAGES:
During the sorting process, the following message is displayed:
“Sorting ... please wait ...”
Upon completion, SORT displays the following message on the
screen:
“Sorting completed”
To halt or terminate the test before the completion message is
displayed, press CONTROL-C on the keyboard.
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.8 Test Documentation
Test Script for Testing The “change field”
Step N: Press function key 4: Access data file.
Step N+1: Screen will ask for the name of the date file.
Type ‘sys:test.txt’
Step N+2: Menu will appear, reading
* delete file
* modify file
* rename file
24. Place cursor next to ‘modify file’ and press RETURN
key.
Step N+3: Screen will ask for record number. Type ‘4017’.
Step N+4: Screen will fill with data fields for record 4017:
Record number: 4017 X: 0042 Y: 0036
Soil type: clay Percolation: 4 mtrs/hr
Vegetation: kudzu Canopy height: 25 mtrs
Water table: 12 mtrs Construct: outhouse
Maintenance code: 3T/4F/9R
Step N+5: Press function key 9: modify
Step N+6: Entries on screen will be highlighted. Move cursor to
VEGETATION field. Type ‘grass’ over ‘kudzu’ and press
RETURN key.
Step N+7: Entries on screen will no longer be highlighted.
VEGETATION field should now read ‘grass’.
Step N+8: Press function key 16: Return to previous screen.
Step N+9: Menu will appear, reading
* delete file
* modify file
* rename file
To verify that the modification has been
recorded,place cursor next to ‘modify file’ and press RETURN
key.
Step N+10: Screen will ask for record number. Type ‘4017’.
Step N+11: Screen will fill with data fields for record 4017:
Record number: 4017 X: 0042 Y: 0036
Soil type: clay Percolation: 4 mtrs/hr
Vegetation: grass Canopy height: 25 mtrs
Water table: 12 mtrs Construct: outhouse
Maintenance code: 3T/4F/9R
Pfleeger and Atlee, Software Engineering: Theory and Practice
25. Chapter 9.*
9.8 Test Documentation
Test Analysis ReportDocuments the result of testProvides
information needed to duplicate the failure and to locate and fix
the source of the problemProvides information necessary to
determine if the project is completeEstablish confidence in the
system’s performance
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.8 Test Documentation
Problem Report FormsLocation: Where did the problem
occur?Timing: When did it occur?Symptom: What was
observed?End result: What were the consequences?Mechanism:
How did it occur?Cause: Why did it occur?Severity: How much
was the user or business affected?Cost: How much did it cost?
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.10 Information Systems Example
The Piccadilly SystemMany variables, many different test cases
to considerAn automated testing tool may be useful
26. Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.10 Information Systems Example
Things to Consider in Selecting a Test
ToolCapabilityReliabilityCapacityLearnabilityOperabilityPerfor
manceCompatibilityNonintrusiveness
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.10 Information Systems Example
Sidebar 9.13 Why Six-Sigma Efforts Do Not Apply to
SoftwareA six-sigma quality constraint says that in a billion
parts, we can expect only 3.4 to be outside the acceptable
rangeIt is not apply to software becausePeople are variable, the
software process inherently contains a large degree of
uncontrollable variationSoftware either conforms or it does not,
there are no degree of conformanceSoftware is not the result of
a mass-production process
27. Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.11 Real-Time Example
Ariane-5 FailureSimulation might help preventing the
failureCould have generated signals related to predicted flight
parameters while turntable provided angular movement
Pfleeger and Atlee, Software Engineering: Theory and Practice
Chapter 9.*
9.12 What This Chapter Means for YouShould anticipate testing
from the very beginning of the system life cycleShould think
about system functions during requirement analysisShould use
fault-tree analysis, failure modes and effect analysis during
designShould build safety case during design and code
reviewsShould consider all possible test cases during testing
History of Social Problems
You are the director of an agency that serves populations with
the following social problems:
· Substance abuse,
· Child abuse,
· Elderly health and social care,
· Issues in corrections,
· Homelessness,
· School behavior-related issues, and
· Poverty.
28. You are asked by your supervisor to train new volunteers at
your agency. You need to provide the volunteers background
education on your agency, the social problem, and the
populations being served. The topic is: “The History of Human
Services” pertaining to specific social problems in the
community.
Choose one of the aforementioned social problems that your
agency serves and discuss the following items:
1. Describe the issues related to the social problem at this point
in time.
2. Discuss the timeline of major historical issues of the
problem.
3. Provide two solutions to the problem.
4. Explain the predominant human service concerns during this
current year compared to the very start of your timeline?
Your training manual should be 2-3 pages in length. You are
welcome to include graphics for visual interest, use colorful
font, and apply APA Style citations and references when
thoughts, ideas, graphics, or another author’s work is utilized.