Principles of Software TestingPart ISatzinger, Jackson, and Burd
Testing Testing is a process of identifying defects Develop test cases and test data A test case is a formal description of• A starting state• One or more events to which the softwaremust respond• The expected response or ending state Test data is a set of starting states andevents used to test a module, group ofmodules, or entire system
Testing discipline activities
Test types and detected defects
Unit Testing The process of testing individualmethods, classes, or components beforethey are integrated with other software Two methods for isolated testing of units Driver• Simulates the behavior of a method thatsends a message to the method beingtested Stub• Simulates the behavior of a method thathas not yet been written
Integration Testing Evaluates the behavior of a group ofmethods or classes Identifies interface compatibility, unexpectedparameter values or state interaction, and run-time exceptions System test Integration test of the behavior of an entiresystem or independent subsystem Build and smoke test System test performed daily or several times aweek
Usability Testing Determines whether a method, class,subsystem, or system meets user requirements Performance test Determines whether a system or subsystem canmeet time-based performance criteria• Response time specifies the desired or maximumallowable time limit for software responses toqueries and updates• Throughput specifies the desired or minimumnumber of queries and transactions that must beprocessed per minute or hour
User Acceptance Testing Determines whether the system fulfills userrequirements Involves the end users Acceptance testing is a very formal activityin most development projects
Who Tests Software? Programmers Unit testing Testing buddies can test other’s programmer’scode Users Usability and acceptance testing Volunteers are frequently used to test betaversions Quality assurance personnel All testing types except unit and acceptance Develop test plans and identify needed changes
Part IIPrinciples of Software Testing forTestersModule 0: About This Course
Course Objectives After completing this course, you will be a moreknowledgeable software tester. You will be able tobetter: Understand and describe the basic concepts offunctional (black box) software testing. Identify a number of test styles and techniques andassess their usefulness in your context. Understand the basic application of techniques used toidentify useful ideas for tests. Help determine the mission and communicate thestatus of your testing with the rest of your project team. Characterize a good bug report, peer-review the reportsof your colleagues, and improve your own report writing. Understand where key testing concepts apply within thecontext of the Rational Unified Process.
Course Outline0 – About This Course1 – Software Engineering Practices2 – Core Concepts of Software Testing3 – The RUP Testing Discipline4 – Define Evaluation Mission5 – Test and Evaluate6 – Analyze Test Failure7 – Achieve Acceptable Mission8 – The RUP Workflow As Context
Principles of Software Testing forTestersModule 1: Software Engineering Practices(Some things Testers should know about them)
Objectives Identify some common softwaredevelopment problems. Identify six software engineering practicesfor addressing common softwaredevelopment problems. Discuss how a software engineeringprocess provides supporting context forsoftware engineering practices.
Symptoms of Software Development Problems User or business needs not met Requirements churn Modules don’t integrate Hard to maintain Late discovery of flaws Poor quality or poor user experience Poor performance under load No coordinated team effort Build-and-release issues
Software Engineering Practices Reinforce Each OtherValidates architecturaldecisions early onAddresses complexity of design/implementation incrementallyMeasures quality early and oftenEvolves baselines incrementallyEnsures users involvedas requirements evolveDevelop IterativelyManage RequirementsUse Component ArchitecturesModel Visually (UML)Continuously Verify QualityManage ChangeSoftware EngineeringPractices
Principles of Software Testing forTestersModule 2: Core Concepts of SoftwareTesting
Objectives Introduce foundation topics of functionaltesting Provide stakeholder-centric visions ofquality and defect Explain test ideas Introduce test matrices
Module 2 Content OutlineDefinitions Defining functional testing Definitions of quality A pragmatic definition of defect Dimensions of quality Test ideas Test idea catalogs Test matrices
Functional Testing In this course, we adopt a common, broadcurrent meaning for functional testing. It is Black box Interested in any externally visible ormeasurable attributes of the software other thanperformance. In functional testing, we think of theprogram as a collection of functions We test it in terms of its inputs and outputs.
How Some Experts Have Defined Quality Fitness for use (Dr. Joseph M. Juran) The totality of features and characteristics of aproduct that bear on its ability to satisfy a givenneed (American Society for Quality) Conformance with requirements (Philip Crosby) The total composite product and servicecharacteristics of marketing, engineering,manufacturing and maintenance through whichthe product and service in use will meetexpectations of the customer (Armand V.Feigenbaum) Note absence of “conforms tospecifications.”
Quality As Satisfiers and Dissatisfiers Joseph Juran distinguishes betweenCustomer Satisfiers and Dissatisfiers askey dimensions of quality: Customer Satisfiers• the right features• adequate instruction Dissatisfiers• unreliable• hard to use• too slow• incompatible with the customer’s equipment
A Working Definition of QualityQuality is value to some person.---- Gerald M. Weinberg
Change Requests and Quality A “defect” – in the eyes of a projectstakeholder– can include anything aboutthe program that causes the program tohave lower value. It’s appropriate to report any aspect of thesoftware that, in your opinion (or in theopinion of a stakeholder whose interestsyou advocate) causes the program to havelower value.
Dimensions of Quality: FURPSReliability e.g., Test the applicationbehaves consistently andpredictably.Performance e.g., Test onlineresponse under averageand peak loadingFunctionality e.g., Test the accurateworkings of eachusage scenarioUsability e.g., Test application fromthe perspective ofconvenience to end-user.Supportability e.g., Test the ability tomaintain and supportapplication underproduction use
A Broader Definition of Dimensions of Quality Accessibility Capability Compatibility Concurrency Conformance tostandards Efficiency Installability anduninstallability Localizability Maintainability Performance Portability Reliability Scalability Security Supportability Testability UsabilityCollectively, these are often called Qualities of Service,Nonfunctional Requirements, Attributes, or simply the -ilities
Test Ideas A test idea is a brief statement thatidentifies a test that might be useful. A test idea differs from a test case, in thatthe test idea contains no specification of thetest workings, only the essence of the ideabehind the test. Test ideas are generators for test cases:potential test cases are derived from a testideas list. A key question for the tester or test analystis which ones are the ones worth trying.
Exercise 2.3: Brainstorm Test Ideas (1/2) We’re about to brainstorm, so let’s review… Ground Rules for Brainstorming The goal is to get lots of ideas. You brainstormtogether to discover categories of possible tests—good ideas that you can refine later. There are more great ideas out there than you think. Don’t criticize others’ contributions. Jokes are OK, and are often valuable. Work later, alone or in a much smaller group, toeliminate redundancy, cut bad ideas, and refine andoptimize the specific tests. Often, these meetings have a facilitator (who runs themeeting) and a recorder (who writes good stuff ontoflipcharts). These two keep their opinions tothemselves.
Exercise 2.3: Brainstorm Test Ideas (2/2) A field can accept integer values between20 and 50. What tests should you try?
A Test Ideas List for Integer-Input Tests Common answers to the exercise would include:Test Why it’s interesting Expected result20 Smallest valid value Accepts it19 Smallest -1 Reject, error msg0 0 is always interesting Reject, error msgBlank Empty field, what’s it do? Reject? Ignore?49 Valid value Accepts it50 Largest valid value Accepts it51 Largest +1 Reject, error msg-1 Negative number Reject, error msg4294967296 2^32, overflow integer? Reject, error msg
Discussion 2.4: Where Do Test Ideas Come From? Where would you derive Test Ideas Lists? Models Specifications Customer complaints Brainstorm sessions among colleagues
A Catalog of Test Ideas for Integer-Input tests Nothing Valid value At LB of value At UB of value At LB of value - 1 At UB of value + 1 Outside of LB of value Outside of UB of value 0 Negative At LB number of digits or chars At UB number of digits or chars Empty field (clear the defaultvalue) Outside of UB number of digitsor chars Non-digits Wrong data type (e.g. decimalinto integer) Expressions Space Non-printing char (e.g.,Ctrl+char) DOS filename reserved chars(e.g., " * . :") Upper ASCII (128-254) Upper case chars Lower case chars Modifiers (e.g., Ctrl, Alt, Shift-Ctrl, etc.) Function key (F2, F3, F4, etc.)
The Test-Ideas Catalog A test-ideas catalog is a list of related testideas that are usable under manycircumstances. For example, the test ideas for numeric inputfields can be catalogued together and used forany numeric input field. In many situations, these catalogs aresufficient test documentation. That is, anexperienced tester can often proceed withtesting directly from these without creatingdocumented test cases.
Apply a Test Ideas Catalog Using a Test MatrixField nameField nameField name
Review: Core Concepts of Software Testing What is Quality? Who are the Stakeholders? What is a Defect? What are Dimensions of Quality? What are Test Ideas? Where are Test Ideas useful? Give some examples of a Test Ideas. Explain how a catalog of Test Ideas couldbe applied to a Test Matrix.
Principles of Software Testing forTestersModule 4: Define Evaluation Mission
So? Purpose of Testing? The typical testing group has two keypriorities. Find the bugs (preferably in priority order). Assess the condition of the whole product(as a user will see it). Sometimes, these conflict The mission of assessment is the underlyingreason for testing, from management’sviewpoint. But if you aren’t hammering hard onthe program, you can miss key risks.
Missions of Test Groups Can Vary Find defects Maximize bug count Block premature product releases Help managers make ship / no-ship decisions Assess quality Minimize technical support costs Conform to regulations Minimize safety-related lawsuit risk Assess conformance to specification Find safe scenarios for use of the product (findways to get it to work, in spite of the bugs) Verify correctness of the product Assure quality
A Different Take on Mission: Public vs. Private Bugs A programmer’s public bug rate includes allbugs left in the code at check-in. A programmer’s private bug rate includesall the bugs that are produced, including theones fixed before check-in. Estimates of private bug rates have rangedfrom 15 to 150 bugs per 100 statements. What does this tell us about our task?
Defining the Test Approach The test approach (or “testing strategy”)specifies the techniques that will be used toaccomplish the test mission. The test approach also specifies how thetechniques will be used. A good test approach is: Diversified Risk-focused Product-specific Practical Defensible
Heuristics for Evaluating Testing Approach James Bach collected a series of heuristicsfor evaluating your test approach. Forexample, he says: Testing should be optimized to find importantproblems fast, rather than attempting to find allproblems with equal urgency. Please note that these are heuristics – theywon’t always the best choice for yourcontext. But in different contexts, you’ll finddifferent ones very useful.
What Test Documentation Should You Use? Test planning standards and templates Examples Some benefits and costs of using IEEE-829standard based templates When are these appropriate? Thinking about your requirements for testdocumentation Requirements considerations Questions to elicit information about testdocumentation requirements for your project
Write a Purpose Statement for Test Documentation Try to describe your core documentationrequirements in one sentence that doesn’thave more than three components. Examples: The test documentation set will primarilysupport our efforts to find bugs in this version,to delegate work, and to track status. The test documentation set will support ongoingproduct and test maintenance over at least 10years, will provide training material for newgroup members, and will create archivessuitable for regulatory or litigation use.
Review: Define Evaluation Mission What is a Test Mission? What is your Test Mission? What makes a good Test Approach (TestStrategy)? What is a Test Documentation Mission? What is your Test Documentation Goal?
Principles of Software Testing forTestersModule 5: Test & Evaluate
Test and Evaluate – Part One: Test In this module, we drill intoTest and Evaluate This addresses the “How?”question: How will you test thosethings?
Test and Evaluate – Part One: Test This module focuseson the activityImplement Test Earlier, we coveredTest-Idea Lists, whichare input here In the next module,we’ll cover AnalyzeTest Failures, thesecond half of Testand Evaluate
Review: Defining the Test Approach In Module 4, we covered Test Approach A good test approach is:DiversifiedRisk-focusedProduct-specificPracticalDefensible The techniques you apply should followyour test approach
Discussion Exercise 5.1: Test Techniques There are as many as 200 published testingtechniques. Many of the ideas areoverlapping, but there are common themes. Similar sounding terms often mean differentthings, e.g.: User testing Usability testing User interface testing What are the differences among thesetechniques?
Dimensions of Test Techniques Think of the testing you do in terms of fivedimensions: Testers: who does the testing. Coverage: what gets tested. Potential problems: why youre testing (whatrisk youre testing for). Activities: how you test. Evaluation: how to tell whether the test passedor failed. Test techniques often focus on one or twoof these, leaving the rest to the skill andimagination of the tester.
Test Techniques—Dominant Test Approaches Of the 200+ published Functional Testingtechniques, there are ten basic themes. They capture the techniques in actual practice. In this course, we call them: Function testing Equivalence analysis Specification-based testing Risk-based testing Stress testing Regression testing Exploratory testing User testing Scenario testing Stochastic or Random testing
“So Which Technique Is the Best?”TestersCoveragePotential problemsActivitiesEvaluationTechnique ATechnique BTechnique CTechnique ETechnique FTechnique GTechnique H Each hasstrengths andweaknesses Think interms ofcomplement There is no“one true way” Mixingtechniquescan improvecoverageTechnique D
InceptionInception ElaborationElaboration ConstructionConstruction TransitionTransitionApply Techniques According to the LifeCycle Test Approach changes over the project Some techniques work well in early phases;others in later ones Align the techniques to iteration objectivesA limited set of focused tests Many varied testsA few components of software under test Large system under testSimple test environment Complex test environmentFocus on architectural & requirement risks Focus on deployment risks
Module 5 Agenda Overview of the workflow: Test and Evaluate Defining test techniques Individual techniques Function testing Equivalence analysis Specification-based testing Risk-based testing Stress testing Regression testing Exploratory testing User testing Scenario testing Stochastic or Random testing Using techniques together
At a Glance: Function TestingTag line Black box unit testingObjective Test each function thoroughly, one at atime.Testers AnyCoverage Each function and user-visible variablePotential problems A function does not work in isolationActivities Whatever worksEvaluation Whatever worksComplexity SimpleHarshness VariesSUT readiness Any stage
Strengths & Weaknesses: Function Testing Representative cases Spreadsheet, test each item in isolation. Database, test each report in isolation Strengths Thorough analysis of each item tested Easy to do as each function is implemented Blind spots Misses interactions Misses exploration of the benefits offered by theprogram.
At a Glance: Equivalence Analysis (1/2)Tag line Partitioning, boundary analysis, domaintestingObjectiveThere are too many test cases to run.Use stratified sampling strategy toselect a few test cases from a hugepopulation.Testers AnyCoverageAll data fields, and simple combinationsof data fields. Data fields include input,output, and (to the extent they can bemade visible to the tester) internal andconfiguration variablesPotential problems Data, configuration, error handling
At a Glance: Equivalence Analysis (2/2)ActivitiesDivide the set of possible values of a field intosubsets, pick values to represent each subset.Typical values will be at boundaries. Moregenerally, the goal is to find a “bestrepresentative” for each subset, and to runtests with these representatives.Advanced approach: combine tests of several“best representatives”. Several approaches tochoosing optimal small set of combinations.Evaluation Determined by the dataComplexity SimpleHarshnessDesigned to discover harsh single-variabletests and harsh combinations of a fewvariablesSUT readiness Any stage
Strengths & Weaknesses: Equivalence Analysis Representative cases Equivalence analysis of a simple numeric field. Printer compatibility testing (multidimensional variable,doesn’t map to a simple numeric field, but stratifiedsampling is essential) Strengths Find highest probability errors with a relatively small setof tests. Intuitively clear approach, generalizes well Blind spots Errors that are not at boundaries or in obvious specialcases. The actual sets of possible values are oftenunknowable.
Optional Exercise 5.2: GUI Equivalence Analysis Pick an app that you know and some dialogs MS Word and its Print, Page setup, Font format dialogs Select a dialog Identify each field, and for each field• What is the type of the field (integer, real, string, ...)?• List the range of entries that are “valid” for the field• Partition the field and identify boundary conditions• List the entries that are almost too extreme and tooextreme for the field• List a few test cases for the field and explain why thevalues you chose are the most powerfulrepresentatives of their sets (for showing a bug)• Identify any constraints imposed on this field by otherfields
At a Glance: Specification-Based TestingTag line Verify every claimObjective Check conformance with every statement inevery spec, requirements document, etc.Testers AnyCoverage Documented reqts, features, etc.PotentialproblemsMismatch of implementation to specActivities Write & execute tests based on the spec’s.Review and manage docs & traceabilityEvaluation Does behavior match the spec?Complexity Depends on the specHarshness Depends on the specSUT readiness As soon as modules are available
Strengths & Weaknesses: Spec-Based Testing Representative cases Traceability matrix, tracks test cases associated witheach specification item. User documentation testing Strengths Critical defense against warranty claims, fraud charges,loss of credibility with customers. Effective for managing scope / expectations ofregulatory-driven testing Reduces support costs / customer complaints byensuring that no false or misleading representations aremade to customers. Blind spots Any issues not in the specs or treated badly in thespecs /documentation.
Traceability Tool for Specification-Based TestingStmt 1 Stmt 2 Stmt 3 Stmt 4 Stmt 5Test 1 X X XTest 2 X XTest 3 X X XTest 4 X XTest 5 X XTest 6 X XThe Traceability Matrix
Optional Exercise 5.5: What “Specs” Can You Use? Challenge: Getting information in the absence of a spec What substitutes are available? Example: The user manual – think of this as a commercialwarranty for what your product does. What other “specs” can you/should you beusing to test?
Exercise 5.5—Specification-Based Testing Here are some ideas for sources that youcan consult when specifications areincomplete or incorrect. Software change memos that come with newbuilds of the program User manual draft (and previous version’smanual) Product literature Published style guide and UI standards
Definitions—Risk-Based Testing Three key meanings:1. Find errors (risk-based approach to the technicaltasks of testing)2. Manage the process of finding errors (risk-basedtest management)3. Manage the testing project and the risk posed by(and to) testing in its relationship to the overallproject (risk-based project management) We’ll look primarily at risk-based testing (#1),proceeding later to risk-based test management. The project management risks are veryimportant, but out of scope for this class.
At a Glance: Risk-Based TestingTag line Find big bugs firstObjective Define, prioritize, refine tests in terms ofthe relative risk of issues we could test forTesters AnyCoverage By identified riskPotential problems Identifiable risksActivities Use qualities of service, risk heuristics andbug patterns to identify risksEvaluation VariesComplexity AnyHarshness HarshSUT readiness Any stage
Strengths & Weaknesses: Risk-Based Testing Representative cases Equivalence class analysis, reformulated. Test in order of frequency of use. Stress tests, error handling tests, security tests. Sample from predicted-bugs list. Strengths Optimal prioritization (if we get the risk list right) High power tests Blind spots Risks not identified or that are surprisingly more likely. Some “risk-driven” testers seem to operate subjectively.• How will I know what coverage I’ve reached?• Do I know that I haven’t missed something critical?
Optional Exercise 5.6: Risk-Based Testing You are testing Amazon.com(Or pick another familiar application) First brainstorm: What are the functional areas of the app? Then evaluate risks:• What are some of the ways that each of thesecould fail?• How likely do you think they are to fail? Why?• How serious would each of the failure types be?
At a Glance: Stress TestingTag line Overwhelm the productObjectiveLearn what failure at extremes tellsabout changes needed in theprogram’s handling of normal casesTesters SpecialistsCoverage LimitedPotential problems Error handling weaknessesActivities SpecializedEvaluation VariesComplexity VariesHarshness ExtremeSUT readiness Late stage
Strengths & Weaknesses: Stress Testing Representative cases Buffer overflow bugs High volumes of data, device connections, longtransaction chains Low memory conditions, device failures, viruses, othercrises Extreme load Strengths Expose weaknesses that will arise in the field. Expose security risks. Blind spots Weaknesses that are not made more visible by stress.
At a Glance: Regression TestingTag line Automated testing after changesObjective Detect unforeseen consequences of changeTesters VariesCoverage VariesPotentialproblemsSide effects of changesUnsuccessful bug fixesActivities Create automated test suites and run againstevery (major) buildComplexity VariesEvaluation VariesHarshness VariesSUT readiness For unit – early; for GUI - late
Strengths & Weaknesses—Regression Testing Representative cases Bug regression, old fix regression, general functionalregression Automated GUI regression test suites Strengths Cheap to execute Configuration testing Regulator friendly Blind spots “Immunization curve” Anything not covered in the regression suite Cost of maintaining the regression suite
At a Glance: Exploratory TestingTag line Simultaneous learning, planning, andtestingObjectiveSimultaneously learn about theproduct and about the test strategiesto reveal the product and its defectsTesters ExplorersCoverage Hard to assessPotential problems Everything unforeseen by plannedtesting techniquesActivities Learn, plan, and test at the same timeEvaluation VariesComplexity VariesHarshness VariesSUT readiness Medium to late: use cases must work
Strengths & Weaknesses: Exploratory Testing Representative cases Skilled exploratory testing of the full product Rapid testing & emergency testing (including thrown-over-the-wall test-it-today) Troubleshooting / follow-up testing of defects. Strengths Customer-focused, risk-focused Responsive to changing circumstances Finds bugs that are otherwise missed Blind spots The less we know, the more we risk missing. Limited by each tester’s weaknesses (can mitigate thiswith careful management) This is skilled work, juniors aren’t very good at it.
At a Glance: User TestingTag line Strive for realismLet’s try real humans (for a change)Objective Identify failures in the overallhuman/machine/software system.Testers UsersCoverage Very hard to measurePotential problems Items that will be missed by anyoneother than an actual userActivities Directed by userEvaluation User’s assessment, with guidanceComplexity VariesHarshness LimitedSUT readiness Late; has to be fully operable
Strengths & Weaknesses—User Testing Representative cases Beta testing In-house lab using a stratified sample of target market Usability testing Strengths Expose design issues Find areas with high error rates Can be monitored with flight recorders Can use in-house tests focus on controversial areas Blind spots Coverage not assured Weak test cases Beta test technical results are mixed Must distinguish marketing betas from technical betas
At a Glance: Scenario TestingTag line Instantiation of a use caseDo something useful, interesting, and complexObjective Challenging cases to reflect real useTesters AnyCoverage Whatever stories touchPotentialproblemsComplex interactions that happen in real useby experienced usersActivities Interview stakeholders & write screenplays,then implement testsEvaluation AnyComplexity HighHarshness VariesSUT readiness Late. Requires stable, integrated functionality.
Strengths & Weaknesses: Scenario Testing Representative cases Use cases, or sequences involving combinations of usecases. Appraise product against business rules, customer data,competitors’ output Hans Buwalda’s “soap opera testing.” Strengths Complex, realistic events. Can handle (help with)situations that are too complex to model. Exposes failures that occur (develop) over time Blind spots Single function failures can make this test inefficient. Must think carefully to achieve good coverage.
At a Glance: Stochastic or Random Testing (1/2)Tag lineMonkey testingHigh-volume testing with new cases allthe timeObjectiveHave the computer create, execute,and evaluate huge numbers of tests.The individual tests are not all thatpowerful, nor all that compelling.The power of the approach lies inthe large number of tests.These broaden the sample, andthey may test the program over along period of time, giving us insightinto longer term issues.
At a Glance: Stochastic or Random Testing (2/2)Testers MachinesCoverage Broad but shallow. Problems withstateful apps.Potential problems Crashes and exceptionsActivities Focus on test generationEvaluation Generic, state-basedComplexity Complex to generate, but individualtests are simpleHarshness Weak individual tests, but hugenumbers of themSUT readiness Any
Combining Techniques (Revisited) A test approach should be diversified Applying opposite techniques can improvecoverage Often one technique canextend anotherTestersCoveragePotential problemsActivitiesEvaluationTechnique GTechnique ATechnique BTechnique CTechnique ETechnique FTechnique HTechnique D
Applying Opposite Techniques to Boost CoverageRegression Inputs:• Old test cases andanalyses leading to newtest cases Outputs:• Archival test cases,preferably welldocumented, and bugreports Better for:• Reuse across multi-version productsExploration Inputs:• models or other analysesthat yield new tests Outputs• scribbles and bug reports Better for:• Find new bugs, scoutnew areas, risks, or ideasContrast these two techniquesExploration Regression
Applying Complementary Techniques Together Regression testing alone suffers fatigue The bugs get fixed and new runs add little info Symptom of weak coverage Combine automation w/ suitable variance E.g. Risk-based equivalence analysis Coverage of the combinationcan beat sum of the parts EquivalenceRisk-basedRegression
How To Adopt New Techniques1. Answer these questions: What techniques do you use in your test approachnow? What is its greatest shortcoming? What one technique could you add to make thegreatest improvement, consistent with a good testapproach:• Risk-focused?• Product-specific?• Practical?• Defensible?2. Apply that additional technique until proficient3. Iterate