Software testing

PDF generated using the open source mwlib toolkit. See http://code.pediapress.com/ for more information.
PDF generated at: Mon, 17 Mar 2014 03:44:18 UTC
Software testing

Contents
Articles
Introduction 1
Software testing 1
Black-box testing 19
Exploratory testing 20
San Francisco depot 22
Session-based testing 23
Scenario testing 25
Equivalence partitioning 26
Boundary-value analysis 28
All-pairs testing 30
Fuzz testing 33
Cause-effect graph 36
Model-based testing 37
Web testing 41
Installation testing 43
White-box testing 44
White-box testing 44
Code coverage 46
Modified Condition/Decision Coverage 50
Fault injection 51
Bebugging 56
Mutation testing 57
Testing of non functional software aspects 61
Non-functional testing 61
Software performance testing 62
Stress testing 68
Load testing 70
Volume testing 73
Scalability testing 73
Compatibility testing 74

Portability testing 75
Security testing 75
Attack patterns 77
Pseudolocalization 81
Recovery testing 83
Soak testing 83
Characterization test 84
Unit testing 85
Unit testing 85
Self-testing code 91
Test fixture 92
Method stub 94
Mock object 95
Lazy systematic unit testing 98
Test Anything Protocol 99
XUnit 102
List of unit testing frameworks 104
SUnit 139
JUnit 140
CppUnit 143
Test::More 144
NUnit 145
NUnitAsp 147
CsUnit 149
HtmlUnit 151
Test automation 152
Test automation 152
Test bench 157
Test execution engine 158
Test stubs 160
Testware 161
Test automation framework 162
Data-driven testing 167
Modularity-driven testing 168
Keyword-driven testing 168
Hybrid testing 171

Lightweight software test automation 172
Testing process 173
Software testing controversies 173
Test-driven development 175
Agile testing 185
Bug bash 187
Pair Testing 188
Manual testing 189
Regression testing 191
Ad hoc testing 193
Sanity testing 193
Integration testing 195
System testing 196
System integration testing 198
Acceptance testing 200
Risk-based testing 204
Software testing outsourcing 205
Tester driven development 207
Test effort 207
Testing artefacts 209
IEEE 829 209
Test strategy 211
Test plan 214
Traceability matrix 216
Test case 219
Test data 221
Test suite 222
Test script 223
Test harness 224
Static testing 225
Static testing 225
Software review 226
Software peer review 228
Software audit review 229
Software technical review 230

Management review 231
Software inspection 231
Fagan inspection 233
Software walkthrough 236
Code review 237
Automated code review 239
Code reviewing software 239
Static code analysis 240
List of tools for static code analysis 243
GUI testing and review 249
GUI software testing 249
Usability testing 252
Think aloud protocol 257
Usability inspection 258
Cognitive walkthrough 258
Heuristic evaluation 261
Pluralistic walkthrough 265
Comparison of usability evaluation methods 268
References
Article Sources and Contributors 270
Image Sources, Licenses and Contributors 277
Article Licenses
License 278

1
Introduction
Software testing
Software development process
A software developer at work
Core activities
•• Requirements
•• Specification
•• Architecture
•• Construction
•• Design
•• Testing
•• Debugging
•• Deployment
•• Maintenance
Methodologies
•• Waterfall
•• Prototype model
•• Incremental
•• Iterative
•• V-Model
•• Spiral
•• Scrum
•• Cleanroom
•• RAD
•• DSDM
•• RUP
•• XP
•• Agile
•• Lean
•• Dual Vee Model
•• TDD
•• FDD
•• DDD

Software testing 2
Supporting disciplines
•• Configuration management
•• Documentation
•• Quality assurance (SQA)
•• Project management
•• User experience
Tools
•• Compiler
•• Debugger
•• Profiler
•• GUI designer
•• Modeling
•• IDE
•• Build automation
•• v
•• t
• e [1]
Software testing is an investigation conducted to provide stakeholders with information about the quality of the
product or service under test.
[2]
Software testing can also provide an objective, independent view of the software to
allow the business to appreciate and understand the risks of software implementation. Test techniques include, but
are not limited to the process of executing a program or application with the intent of finding software bugs (errors
or other defects).
Software testing can be stated as the process of validating and verifying that a computer
program/application/product:
•• meets the requirements that guided its design and development,
•• works as expected,
•• can be implemented with the same characteristics,
•• and satisfies the needs of stakeholders.
Software testing, depending on the testing method employed, can be implemented at any time in the software
development process. Traditionally most of the test effort occurs after the requirements have been defined and the
coding process has been completed, but in the Agile approaches most of the test effort is on-going. As such, the
methodology of the test is governed by the chosen software development methodology.
[citation needed]
Overview
Testing can never completely identify all the defects within software.
[3]
Instead, it furnishes a criticism or
comparison that compares the state and behavior of the product against oracles—principles or mechanisms by which
someone might recognize a problem. These oracles may include (but are not limited to) specifications, contracts,
[4]
comparable products, past versions of the same product, inferences about intended or expected purpose, user or
customer expectations, relevant standards, applicable laws, or other criteria.
A primary purpose of testing is to detect software failures so that defects may be discovered and corrected. Testing
cannot establish that a product functions properly under all conditions but can only establish that it does not function
properly under specific conditions. The scope of software testing often includes examination of code as well as
execution of that code in various environments and conditions as well as examining the aspects of code: does it do
what it is supposed to do and do what it needs to do. In the current culture of software development, a testing
organization may be separate from the development team. There are various roles for testing team members.
Information derived from software testing may be used to correct the process by which software is developed.

Software testing 3
Every software product has a target audience. For example, the audience for video game software is completely
different from banking software. Therefore, when an organization develops or otherwise invests in a software
product, it can assess whether the software product will be acceptable to its end users, its target audience, its
purchasers and other stakeholders. Software testing is the process of attempting to make this assessment.
Defects and failures
Not all software defects are caused by coding errors. One common source of expensive defects is requirement gaps,
e.g., unrecognized requirements which result in errors of omission by the program designer. Requirement gaps can
often be non-functional requirements such as testability, scalability, maintainability, usability, performance, and
security.
Software faults occur through the following processes. A programmer makes an error (mistake), which results in a
defect (fault, bug) in the software source code. If this defect is executed, in certain situations the system will produce
wrong results, causing a failure.
[5]
Not all defects will necessarily result in failures. For example, defects in dead
code will never result in failures. A defect can turn into a failure when the environment is changed. Examples of
these changes in environment include the software being run on a new computer hardware platform, alterations in
source data, or interacting with different software. A single defect may result in a wide range of failure symptoms.
Input combinations and preconditions
A fundamental problem with software testing is that testing under all combinations of inputs and preconditions
(initial state) is not feasible, even with a simple product.
[][6]
This means that the number of defects in a software
product can be very large and defects that occur infrequently are difficult to find in testing. More significantly,
non-functional dimensions of quality (how it is supposed to be versus what it is supposed to do)—usability,
scalability, performance, compatibility, reliability—can be highly subjective; something that constitutes sufficient
value to one person may be intolerable to another.
Software developers can't test everything, but they can use combinatorial test design to identify the minimum
number of tests needed to get the coverage they want. Combinatorial test design enables users to get greater test
coverage with fewer tests. Whether they are looking for speed or test depth, they can use combinatorial test design
methods to build structured variation into their test cases. Note that "coverage", as used here, is referring to
combinatorial coverage, not requirements coverage.
Economics
A study conducted by NIST in 2002 reports that software bugs cost the U.S. economy $59.5 billion annually. More
than a third of this cost could be avoided if better software testing was performed.
[7]
It is commonly believed that the earlier a defect is found, the cheaper it is to fix it. The following table shows the
cost of fixing the defect depending on the stage it was found. For example, if a problem in the requirements is found
only post-release, then it would cost 10–100 times more to fix than if it had already been found by the requirements
review. With the advent of modern continuous deployment practices and cloud-based services, the cost of
re-deployment and maintenance may lessen over time.

Software testing 4
Cost to fix a defect Time detected
Requirements Architecture Construction System test Post-release
Time introduced Requirements 1× 3× 5–10× 10× 10–100×
Architecture – 1× 10× 15× 25–100×
Construction – – 1× 10× 10–25×
The data from which this table is extrapolated is scant. Laurent Bossavit says in his analysis:
The “smaller projects” curve turns out to be from only two teams of first-year students, a sample size so small
that extrapolating to “smaller projects in general” is totally indefensible. The GTE study does not explain its
data, other than to say it came from two projects, one large and one small. The paper cited for the Bell Labs
“Safeguard” project specifically disclaims having collected the fine-grained data that Boehm’s data points
suggest. The IBM study (Fagan’s paper) contains claims which seem to contradict Boehm’s graph, and no
numerical results which clearly correspond to his data points.
Boehm doesn’t even cite a paper for the TRW data, except when writing for “Making Software” in 2010, and
there he cited the original 1976 article. There exists a large study conducted at TRW at the right time for
Boehm to cite it, but that paper doesn’t contain the sort of data that would support Boehm’s claims.
Roles
Software testing can be done by software testers. Until the 1980s, the term "software tester" was used generally, but
later it was also seen as a separate profession. Regarding the periods and the different goals in software testing,
[8]
different roles have been established: manager, test lead, test analyst, test designer, tester, automation developer,
and test administrator.
History
The separation of debugging from testing was initially introduced by Glenford J. Myers in 1979. Although his
attention was on breakage testing ("a successful test is one that finds a bug") it illustrated the desire of the software
engineering community to separate fundamental development activities, such as debugging, from that of verification.
Dave Gelperin and William C. Hetzel classified in 1988 the phases and goals in software testing in the following
stages:
• Until 1956 – Debugging oriented
[9]
• 1957–1978 – Demonstration oriented
[10]
• 1979–1982 – Destruction oriented
[11]
• 1983–1987 – Evaluation oriented
[12]
• 1988–2000 – Prevention oriented
[13]
Testing methods
Static vs. dynamic testing
There are many approaches to software testing. Reviews, walkthroughs, or inspections are referred to as static
testing, whereas actually executing programmed code with a given set of test cases is referred to as dynamic testing.
Static testing is often implicit, as proofreading, plus when programming tools/text editors check source code
structure or compilers (pre-compilers) check syntax and data flow as static program analysis. Dynamic testing takes
place when the program itself is run. Dynamic testing may begin before the program is 100% complete in order to
test particular sections of code and are applied to discrete functions or modules. Typical techniques for this are either

Software testing 5
using stubs/drivers or execution from a debugger environment.
Static testing involves verification, whereas dynamic testing involves validation. Together they help improve
software quality. Among the techniques for static analysis, mutation testing can be used to ensure the test-cases will
detect errors which are introduced by mutating the source code.
The box approach
Software testing methods are traditionally divided into white- and black-box testing. These two approaches are used
to describe the point of view that a test engineer takes when designing test cases.
White-Box testing
White-box testing (also known as clear box testing, glass box testing, transparent box testing and structural
testing) tests internal structures or workings of a program, as opposed to the functionality exposed to the end-user. In
white-box testing an internal perspective of the system, as well as programming skills, are used to design test cases.
The tester chooses inputs to exercise paths through the code and determine the appropriate outputs. This is analogous
to testing nodes in a circuit, e.g. in-circuit testing (ICT).
While white-box testing can be applied at the unit, integration and system levels of the software testing process, it is
usually done at the unit level. It can test paths within a unit, paths between units during integration, and between
subsystems during a system–level test. Though this method of test design can uncover many errors or problems, it
might not detect unimplemented parts of the specification or missing requirements.
Techniques used in white-box testing include:
• API testing (application programming interface) – testing of the application using public and private APIs
• Code coverage – creating tests to satisfy some criteria of code coverage (e.g., the test designer can create tests to
cause all statements in the program to be executed at least once)
• Fault injection methods – intentionally introducing faults to gauge the efficacy of testing strategies
• Mutation testing methods
• Static testing methods
Code coverage tools can evaluate the completeness of a test suite that was created with any method, including
black-box testing. This allows the software team to examine parts of a system that are rarely tested and ensures that
the most important function points have been tested.
[14]
Code coverage as a software metric can be reported as a
percentage for:
• Function coverage, which reports on functions executed
• Statement coverage, which reports on the number of lines executed to complete the test
100% statement coverage ensures that all code paths, or branches (in terms of control flow) are executed at least
once. This is helpful in ensuring correct functionality, but not sufficient since the same code may process different
inputs correctly or incorrectly.
Black-box testing
Black box diagram
Black-box testing treats the software as a "black box", examining
functionality without any knowledge of internal implementation.
The testers are only aware of what the software is supposed to do,
not how it does it. Black-box testing methods include: equivalence
partitioning, boundary value analysis, all-pairs testing, state
transition tables, decision table testing, fuzz testing, model-based testing, use case testing, exploratory testing and
specification-based testing.

Software testing 6
Specification-based testing aims to test the functionality of software according to the applicable requirements. This
level of testing usually requires thorough test cases to be provided to the tester, who then can simply verify that for a
given input, the output value (or behavior), either "is" or "is not" the same as the expected value specified in the test
case. Test cases are built around specifications and requirements, i.e., what the application is supposed to do. It uses
external descriptions of the software, including specifications, requirements, and designs to derive test cases. These
tests can be functional or non-functional, though usually functional.
Specification-based testing may be necessary to assure correct functionality, but it is insufficient to guard against
complex or high-risk situations.
One advantage of the black box technique is that no programming knowledge is required. Whatever biases the
programmers may have had, the tester likely has a different set and may emphasize different areas of functionality.
On the other hand, black-box testing has been said to be "like a walk in a dark labyrinth without a flashlight."
Because they do not examine the source code, there are situations when a tester writes many test cases to check
something that could have been tested by only one test case, or leaves some parts of the program untested.
This method of test can be applied to all levels of software testing: unit, integration, system and acceptance. It
typically comprises most if not all testing at higher levels, but can also dominate unit testing as well.
Visual testing
The aim of visual testing is to provide developers with the ability to examine what was happening at the point of
software failure by presenting the data in such a way that the developer can easily ﬁnd the information he or she
requires, and the information is expressed clearly.
At the core of visual testing is the idea that showing someone a problem (or a test failure), rather than just describing
it, greatly increases clarity and understanding. Visual testing therefore requires the recording of the entire test
process – capturing everything that occurs on the test system in video format. Output videos are supplemented by
real-time tester input via picture-in-a-picture webcam and audio commentary from microphones.
Visual testing provides a number of advantages. The quality of communication is increased dramatically because
testers can show the problem (and the events leading up to it) to the developer as opposed to just describing it and
the need to replicate test failures will cease to exist in many cases. The developer will have all the evidence he or she
requires of a test failure and can instead focus on the cause of the fault and how it should be fixed.
Visual testing is particularly well-suited for environments that deploy agile methods in their development of
software, since agile methods require greater communication between testers and developers and collaboration
within small teams.
[citation needed]
Ad hoc testing and exploratory testing are important methodologies for checking software integrity, because they
require less preparation time to implement, while the important bugs can be found quickly. In ad hoc testing, where
testing takes place in an improvised, impromptu way, the ability of a test tool to visually record everything that
occurs on a system becomes very important.Wikipedia:Please clarify
[citation needed]
Visual testing is gathering recognition in customer acceptance and usability testing, because the test can be used by
many individuals involved in the development process.
[citation needed]
For the customer, it becomes easy to provide
detailed bug reports and feedback, and for program users, visual testing can record user actions on screen, as well as
their voice and image, to provide a complete picture at the time of software failure for the developer.

Software testing 7
Grey-box testing
Grey-box testing (American spelling: gray-box testing) involves having knowledge of internal data structures and
algorithms for purposes of designing tests, while executing those tests at the user, or black-box level. The tester is
not required to have full access to the software's source code.Wikipedia:Verifiability Manipulating input data and
formatting output do not qualify as grey-box, because the input and output are clearly outside of the "black box" that
we are calling the system under test. This distinction is particularly important when conducting integration testing
between two modules of code written by two different developers, where only the interfaces are exposed for test.
However, tests that require modifying a back-end data repository such as a database or a log file does qualify as
grey-box, as the user would not normally be able to change the data repository in normal production
operations.
[citation needed]
Grey-box testing may also include reverse engineering to determine, for instance, boundary
values or error messages.
By knowing the underlying concepts of how the software works, the tester makes better-informed testing choices
while testing the software from outside. Typically, a grey-box tester will be permitted to set up an isolated testing
environment with activities such as seeding a database. The tester can observe the state of the product being tested
after performing certain actions such as executing SQL statements against the database and then executing queries to
ensure that the expected changes have been reflected. Grey-box testing implements intelligent test scenarios, based
on limited information. This will particularly apply to data type handling, exception handling, and so on.
Testing levels
Tests are frequently grouped by where they are added in the software development process, or by the level of
specificity of the test. The main levels during the development process as defined by the SWEBOK guide are unit-,
integration-, and system testing that are distinguished by the test target without implying a specific process model.
Other test levels are classified by the testing objective.
Unit testing
Unit testing, also known as component testing, refers to tests that verify the functionality of a specific section of
code, usually at the function level. In an object-oriented environment, this is usually at the class level, and the
minimal unit tests include the constructors and destructors.
These types of tests are usually written by developers as they work on code (white-box style), to ensure that the
specific function is working as expected. One function might have multiple tests, to catch corner cases or other
branches in the code. Unit testing alone cannot verify the functionality of a piece of software, but rather is used to
assure that the building blocks the software uses work independently of each other.
Unit testing is a software development process that involves synchronized application of a broad spectrum of defect
prevention and detection strategies in order to reduce software development risks, time, and costs. It is performed by
the software developer or engineer during the construction phase of the software development lifecycle. Rather than
replace traditional QA focuses, it augments it. Unit testing aims to eliminate construction errors before code is
promoted to QA; this strategy is intended to increase the quality of the resulting software as well as the efficiency of
the overall development and QA process.
Depending on the organization's expectations for software development, unit testing might include static code
analysis, data flow analysis metrics analysis, peer code reviews, code coverage analysis and other software
verification practices.

Software testing 8
Integration testing
Integration testing is any type of software testing that seeks to verify the interfaces between components against a
software design. Software components may be integrated in an iterative way or all together ("big bang"). Normally
the former is considered a better practice since it allows interface issues to be located more quickly and fixed.
Integration testing works to expose defects in the interfaces and interaction between integrated components
(modules). Progressively larger groups of tested software components corresponding to elements of the architectural
design are integrated and tested until the software works as a system.
Component interface testing
The practice of component interface testing can be used to check the handling of data passed between various units,
or subsystem components, beyond full integration testing between those units. The data being passed can be
considered as "message packets" and the range or data types can be checked, for data generated from one unit, and
tested for validity before being passed into another unit. One option for interface testing is to keep a separate log file
of data items being passed, often with a timestamp logged to allow analysis of thousands of cases of data passed
between units for days or weeks. Tests can include checking the handling of some extreme data values while other
interface variables are passed as normal values. Unusual data values in an interface can help explain unexpected
performance in the next unit. Component interface testing is a variation of black-box testing, with the focus on the
data values beyond just the related actions of a subsystem component.
System testing
System testing, or end-to-end testing, tests a completely integrated system to verify that it meets its requirements.
For example, a system test might involve testing a logon interface, then creating and editing an entry, plus sending or
printing results, followed by summary processing or deletion (or archiving) of entries, then logoff.
In addition, the software testing should ensure that the program, as well as working as expected, does not also
destroy or partially corrupt its operating environment or cause other processes within that environment to become
inoperative (this includes not corrupting shared memory, not consuming or locking up excessive resources and
leaving any parallel processes unharmed by its presence).
[citation needed]
Acceptance testing
At last the system is delivered to the user for Acceptance testing.
Testing Types
Installation testing
An installation test assures that the system is installed correctly and working at actual customer's hardware.
Compatibility testing
A common cause of software failure (real or perceived) is a lack of its compatibility with other application software,
operating systems (or operating system versions, old or new), or target environments that differ greatly from the
original (such as a terminal or GUI application intended to be run on the desktop now being required to become a
web application, which must render in a web browser). For example, in the case of a lack of backward compatibility,
this can occur because the programmers develop and test software only on the latest version of the target
environment, which not all users may be running. This results in the unintended consequence that the latest work
may not function on earlier versions of the target environment, or on older hardware that earlier versions of the target
environment was capable of using. Sometimes such issues can be fixed by proactively abstracting operating system

Software testing 9
functionality into a separate program module or library.
Smoke and sanity testing
Sanity testing determines whether it is reasonable to proceed with further testing.
Smoke testing consists of minimal attempts to operate the software, designed to determine whether there are any
basic problems that will prevent it from working at all. Such tests can be used as build verification test.
Regression testing
Regression testing focuses on finding defects after a major code change has occurred. Specifically, it seeks to
uncover software regressions, as degraded or lost features, including old bugs that have come back. Such regressions
occur whenever software functionality that was previously working, correctly, stops working as intended. Typically,
regressions occur as an unintended consequence of program changes, when the newly developed part of the software
collides with the previously existing code. Common methods of regression testing include re-running previous sets
of test-cases and checking whether previously fixed faults have re-emerged. The depth of testing depends on the
phase in the release process and the risk of the added features. They can either be complete, for changes added late in
the release or deemed to be risky, or be very shallow, consisting of positive tests on each feature, if the changes are
early in the release or deemed to be of low risk. Regression testing is typically the largest test effort in commercial
software development,
[15]
due to checking numerous details in prior software features, and even new software can be
developed while using some old test-cases to test parts of the new design to ensure prior functionality is still
supported.
Acceptance testing
Acceptance testing can mean one of two things:
1. A smoke test is used as an acceptance test prior to introducing a new build to the main testing process, i.e. before
integration or regression.
2. Acceptance testing performed by the customer, often in their lab environment on their own hardware, is known as
user acceptance testing (UAT). Acceptance testing may be performed as part of the hand-off process between any
two phases of development.
[citation needed]
Alpha testing
Alpha testing is simulated or actual operational testing by potential users/customers or an independent test team at
the developers' site. Alpha testing is often employed for off-the-shelf software as a form of internal acceptance
testing, before the software goes to beta testing.
Beta testing
Beta testing comes after alpha testing and can be considered a form of external user acceptance testing. Versions of
the software, known as beta versions, are released to a limited audience outside of the programming team. The
software is released to groups of people so that further testing can ensure the product has few faults or bugs.
Sometimes, beta versions are made available to the open public to increase the feedback field to a maximal number
of future users.
[citation needed]

Software testing 10
Functional vs non-functional testing
Functional testing refers to activities that verify a specific action or function of the code. These are usually found in
the code requirements documentation, although some development methodologies work from use cases or user
stories. Functional tests tend to answer the question of "can the user do this" or "does this particular feature work."
Non-functional testing refers to aspects of the software that may not be related to a specific function or user action,
such as scalability or other performance, behavior under certain constraints, or security. Testing will determine the
breaking point, the point at which extremes of scalability or performance leads to unstable execution. Non-functional
requirements tend to be those that reflect the quality of the product, particularly in the context of the suitability
perspective of its users.
Destructive testing
Destructive testing attempts to cause the software or a sub-system to fail. It verifies that the software functions
properly even when it receives invalid or unexpected inputs, thereby establishing the robustness of input validation
and error-management routines.
[citation needed]
Software fault injection, in the form of fuzzing, is an example of
failure testing. Various commercial non-functional testing tools are linked from the software fault injection page;
there are also numerous open-source and free software tools available that perform destructive testing.
Software performance testing
Performance testing is generally executed to determine how a system or sub-system performs in terms of
responsiveness and stability under a particular workload. It can also serve to investigate, measure, validate or verify
other quality attributes of the system, such as scalability, reliability and resource usage.
Load testing is primarily concerned with testing that the system can continue to operate under a specific load,
whether that be large quantities of data or a large number of users. This is generally referred to as software
scalability. The related load testing activity of when performed as a non-functional activity is often referred to as
endurance testing. Volume testing is a way to test software functions even when certain components (for example a
file or database) increase radically in size. Stress testing is a way to test reliability under unexpected or rare
workloads. Stability testing (often referred to as load or endurance testing) checks to see if the software can
continuously function well in or above an acceptable period.
There is little agreement on what the specific goals of performance testing are. The terms load testing, performance
testing, scalability testing, and volume testing, are often used interchangeably.
Real-time software systems have strict timing constraints. To test if timing constraints are met, real-time testing is
used.
Usability testing
Usability testing is needed to check if the user interface is easy to use and understand. It is concerned mainly with
the use of the application.
Accessibility testing
Accessibility testing may include compliance with standards such as:
•• Americans with Disabilities Act of 1990
•• Section 508 Amendment to the Rehabilitation Act of 1973
• Web Accessibility Initiative (WAI) of the World Wide Web Consortium (W3C)

Software testing 11
Security testing
Security testing is essential for software that processes confidential data to prevent system intrusion by hackers.
Internationalization and localization
The general ability of software to be internationalized and localized can be automatically tested without actual
translation, by using pseudolocalization. It will verify that the application still works, even after it has been translated
into a new language or adapted for a new culture (such as different currencies or time zones).
Actual translation to human languages must be tested, too. Possible localization failures include:
• Software is often localized by translating a list of strings out of context, and the translator may choose the wrong
translation for an ambiguous source string.
•• Technical terminology may become inconsistent if the project is translated by several people without proper
coordination or if the translator is imprudent.
•• Literal word-for-word translations may sound inappropriate, artificial or too technical in the target language.
• Untranslated messages in the original language may be left hard coded in the source code.
• Some messages may be created automatically at run time and the resulting string may be ungrammatical,
functionally incorrect, misleading or confusing.
• Software may use a keyboard shortcut which has no function on the source language's keyboard layout, but is
used for typing characters in the layout of the target language.
• Software may lack support for the character encoding of the target language.
• Fonts and font sizes which are appropriate in the source language may be inappropriate in the target language; for
example, CJK characters may become unreadable if the font is too small.
•• A string in the target language may be longer than the software can handle. This may make the string partly
invisible to the user or cause the software to crash or malfunction.
• Software may lack proper support for reading or writing bi-directional text.
•• Software may display images with text that was not localized.
• Localized operating systems may have differently named system configuration files and environment variables
and different formats for date and currency.
Development testing
Development Testing is a software development process that involves synchronized application of a broad spectrum
of defect prevention and detection strategies in order to reduce software development risks, time, and costs. It is
performed by the software developer or engineer during the construction phase of the software development
lifecycle. Rather than replace traditional QA focuses, it augments it. Development Testing aims to eliminate
construction errors before code is promoted to QA; this strategy is intended to increase the quality of the resulting
software as well as the efficiency of the overall development and QA process.
Depending on the organization's expectations for software development, Development Testing might include static
code analysis, data flow analysis metrics analysis, peer code reviews, unit testing, code coverage analysis,
traceability, and other software verification practices.

Software testing 12
Testing process
Traditional waterfall development model
A common practice of software testing is that testing is performed by an independent group of testers after the
functionality is developed, before it is shipped to the customer. This practice often results in the testing phase being
used as a project buffer to compensate for project delays, thereby compromising the time devoted to testing.
Another practice is to start software testing at the same moment the project starts and it is a continuous process until
the project finishes.
Agile or Extreme development model
In contrast, some emerging software disciplines such as extreme programming and the agile software development
movement, adhere to a "test-driven software development" model. In this process, unit tests are written first, by the
software engineers (often with pair programming in the extreme programming methodology). Of course these tests
fail initially; as they are expected to. Then as code is written it passes incrementally larger portions of the test suites.
The test suites are continuously updated as new failure conditions and corner cases are discovered, and they are
integrated with any regression tests that are developed. Unit tests are maintained along with the rest of the software
source code and generally integrated into the build process (with inherently interactive tests being relegated to a
partially manual build acceptance process). The ultimate goal of this test process is to achieve continuous integration
where software updates can be published to the public frequently.
This methodology increases the testing effort done by development, before reaching any formal testing team. In
some other development models, most of the test execution occurs after the requirements have been defined and the
coding process has been completed.
Top-down and bottom-up
Bottom Up Testing is an approach to integrated testing where the lowest level components (modules, procedures,
and functions) are tested first, then integrated and used to facilitate the testing of higher level components. After the
integration testing of lower level integrated modules, the next level of modules will be formed and can be used for
integration testing. The process is repeated until the components at the top of the hierarchy are tested. This approach
is helpful only when all or most of the modules of the same development level are ready.
[citation needed]
This method
also helps to determine the levels of software developed and makes it easier to report testing progress in the form of
a percentage.
[citation needed]
Top Down Testing is an approach to integrated testing where the top integrated modules are tested and the branch of
the module is tested step by step until the end of the related module.
In both, method stubs and drivers are used to stand-in for missing components and are replaced as the levels are
completed....
A sample testing cycle
Although variations exist between organizations, there is a typical cycle for testing. The sample below is common
among organizations employing the Waterfall development model. The same practices are commonly found in other
development models, but might not be as clear or explicit.
• Requirements analysis: Testing should begin in the requirements phase of the software development life cycle.
During the design phase, testers work to determine what aspects of a design are testable and with what parameters
those tests work.
• Test planning: Test strategy, test plan, testbed creation. Since many activities will be carried out during testing, a
plan is needed.

Software testing 13
• Test development: Test procedures, test scenarios, test cases, test datasets, test scripts to use in testing software.
• Test execution: Testers execute the software based on the plans and test documents then report any errors found
to the development team.
• Test reporting: Once testing is completed, testers generate metrics and make final reports on their test effort and
whether or not the software tested is ready for release.
• Test result analysis: Or Defect Analysis, is done by the development team usually along with the client, in order
to decide what defects should be assigned, fixed, rejected (i.e. found software working properly) or deferred to be
dealt with later.
• Defect Retesting: Once a defect has been dealt with by the development team, it is retested by the testing team.
AKA Resolution testing.
• Regression testing: It is common to have a small test program built of a subset of tests, for each integration of
new, modified, or fixed software, in order to ensure that the latest delivery has not ruined anything, and that the
software product as a whole is still working correctly.
• Test Closure: Once the test meets the exit criteria, the activities such as capturing the key outputs, lessons
learned, results, logs, documents related to the project are archived and used as a reference for future projects.
Automated testing
Many programming groups are relying more and more on automated testing, especially groups that use test-driven
development. There are many frameworks to write tests in, and continuous integration software will run tests
automatically every time code is checked into a version control system.
While automation cannot reproduce everything that a human can do (and all the ways they think of doing it), it can
be very useful for regression testing. However, it does require a well-developed test suite of testing scripts in order to
be truly useful.
Testing tools
Program testing and fault detection can be aided significantly by testing tools and debuggers. Testing/debug tools
include features such as:
•• Program monitors, permitting full or partial monitoring of program code including:
• Instruction set simulator, permitting complete instruction level monitoring and trace facilities
• Program animation, permitting step-by-step execution and conditional breakpoint at source level or in machine
code
• Code coverage reports
• Formatted dump or symbolic debugging, tools allowing inspection of program variables on error or at chosen
points
•• Automated functional GUI testing tools are used to repeat system-level tests through the GUI
• Benchmarks, allowing run-time performance comparisons to be made
• Performance analysis (or profiling tools) that can help to highlight hot spots and resource usage
Some of these features may be incorporated into an Integrated Development Environment (IDE).

Software testing 14
Measurement in software testing
Usually, quality is constrained to such topics as correctness, completeness, security,
[citation needed]
but can also
include more technical requirements as described under the ISO standard ISO/IEC 9126, such as capability,
reliability, efficiency, portability, maintainability, compatibility, and usability.
There are a number of frequently used software metrics, or measures, which are used to assist in determining the
state of the software or the adequacy of the testing.
Testing artifacts
The software testing process can produce several artifacts.
Test plan
A test specification is called a test plan. The developers are well aware what test plans will be executed and
this information is made available to management and the developers. The idea is to make them more cautious
when developing their code or making additional changes. Some companies have a higher-level document
called a test strategy.
Traceability matrix
A traceability matrix is a table that correlates requirements or design documents to test documents. It is used to
change tests when related source documents are changed, to select test cases for execution when planning for
regression tests by considering requirement coverage.
Test case
A test case normally consists of a unique identifier, requirement references from a design specification,
preconditions, events, a series of steps (also known as actions) to follow, input, output, expected result, and
actual result. Clinically defined a test case is an input and an expected result. This can be as pragmatic as 'for
condition x your derived result is y', whereas other test cases described in more detail the input scenario and
what results might be expected. It can occasionally be a series of steps (but often steps are contained in a
separate test procedure that can be exercised against multiple test cases, as a matter of economy) but with one
expected result or expected outcome. The optional fields are a test case ID, test step, or order of execution
number, related requirement(s), depth, test category, author, and check boxes for whether the test is
automatable and has been automated. Larger test cases may also contain prerequisite states or steps, and
descriptions. A test case should also contain a place for the actual result. These steps can be stored in a word
processor document, spreadsheet, database, or other common repository. In a database system, you may also
be able to see past test results, who generated the results, and what system configuration was used to generate
those results. These past results would usually be stored in a separate table.
Test script
A test script is a procedure, or programing code that replicates user actions. Initially the term was derived from
the product of work created by automated regression test tools. Test Case will be a baseline to create test
scripts using a tool or a program.
Test suite
The most common term for a collection of test cases is a test suite. The test suite often also contains more
detailed instructions or goals for each collection of test cases. It definitely contains a section where the tester
identifies the system configuration used during testing. A group of test cases may also contain prerequisite
states or steps, and descriptions of the following tests.
Test fixture or test data
In most cases, multiple sets of values or data are used to test the same functionality of a particular feature. All
the test values and changeable environmental components are collected in separate files and stored as test data.

Software testing 15
It is also useful to provide this data to the client and with the product or a project.
Test harness
The software, tools, samples of data input and output, and configurations are all referred to collectively as a
test harness.
Certifications
Several certification programs exist to support the professional aspirations of software testers and quality assurance
specialists. No certification now offered actually requires the applicant to show their ability to test software. No
certification is based on a widely accepted body of knowledge. This has led some to declare that the testing field is
not ready for certification. Certification itself cannot measure an individual's productivity, their skill, or practical
knowledge, and cannot guarantee their competence, or professionalism as a tester.
Software testing certification types
• Exam-based: Formalized exams, which need to be passed; can also be learned by self-study [e.g., for ISTQB or
QAI]
• Education-based: Instructor-led sessions, where each course has to be passed [e.g., International Institute for
Software Testing (IIST)].
Testing certifications
• Certified Associate in Software Testing (CAST) offered by the QAI
• CATe offered by the International Institute for Software Testing
• Certified Manager in Software Testing (CMST) offered by the QAI
• Certified Test Manager (CTM) offered by International Institute for Software Testing
• Certified Software Tester (CSTE) offered by the Quality Assurance Institute (QAI)
• Certified Software Test Professional (CSTP) offered by the International Institute for Software Testing
• CSTP (TM) (Australian Version) offered by K. J. Ross & Associates
[16]
• ISEB offered by the Information Systems Examinations Board
• ISTQB Certified Tester, Foundation Level (CTFL) offered by the International Software Testing Qualification
Board
• ISTQB Certified Tester, Advanced Level (CTAL) offered by the International Software Testing Qualification
Board
• TMPF TMap Next Foundation offered by the Examination Institute for Information Science
• TMPA TMap Next Advanced offered by the Examination Institute for Information Science
Quality assurance certifications
• CMSQ offered by the Quality Assurance Institute (QAI).
• CSQA offered by the Quality Assurance Institute (QAI)
• CSQE offered by the American Society for Quality (ASQ)
• CQIA offered by the American Society for Quality (ASQ)

Software testing 16
Controversy
Some of the major software testing controversies include:
What constitutes responsible software testing?
Members of the "context-driven" school of testing believe that there are no "best practices" of testing, but
rather that testing is a set of skills that allow the tester to select or invent testing practices to suit each unique
situation.
Agile vs. traditional
Should testers learn to work under conditions of uncertainty and constant change or should they aim at process
"maturity"? The agile testing movement has received growing popularity since 2006 mainly in commercial
circles,
[17][18]
whereas government and military software providers use this methodology but also the
traditional test-last models (e.g. in the Waterfall model).
[citation needed]
Exploratory test vs. scripted
Should tests be designed at the same time as they are executed or should they be designed beforehand?
Manual testing vs. automated
Some writers believe that test automation is so expensive relative to its value that it should be used
sparingly.
[19]
More in particular, test-driven development states that developers should write unit-tests, as
those of XUnit, before coding the functionality. The tests then can be considered as a way to capture and
implement the requirements.
Software design vs. software implementation
Should testing be carried out only at the end or throughout the whole process?
Who watches the watchmen?
The idea is that any form of observation is also an interaction—the act of testing can also affect that which is
being tested.
[20]
Related processes
Software verification and validation
Software testing is used in association with verification and validation:
•• Verification: Have we built the software right? (i.e., does it implement the requirements).
•• Validation: Have we built the right software? (i.e., do the requirements satisfy the customer).
The terms verification and validation are commonly used interchangeably in the industry; it is also common to see
these two terms incorrectly defined. According to the IEEE Standard Glossary of Software Engineering
Terminology:
Verification is the process of evaluating a system or component to determine whether the products of a given
development phase satisfy the conditions imposed at the start of that phase.
Validation is the process of evaluating a system or component during or at the end of the development process
to determine whether it satisfies specified requirements.
According to the ISO 9000 standard:
Verification is confirmation by examination and through provision of objective evidence that specified
requirements have been fulfilled.
Validation is confirmation by examination and through provision of objective evidence that the requirements
for a specific intended use or application have been fulfilled.

Software testing 17
Software quality assurance (SQA)
Software testing is a part of the software quality assurance (SQA) process.
[]
In SQA, software process specialists and
auditors are concerned for the software development process rather than just the artifacts such as documentation,
code and systems. They examine and change the software engineering process itself to reduce the number of faults
that end up in the delivered software: the so-called "defect rate". What constitutes an "acceptable defect rate"
depends on the nature of the software; A flight simulator video game would have much higher defect tolerance than
software for an actual airplane. Although there are close links with SQA, testing departments often exist
independently, and there may be no SQA function in some companies.
[citation needed]
Software testing is a task intended to detect defects in software by contrasting a computer program's expected results
with its actual results for a given set of inputs. By contrast, QA (quality assurance) is the implementation of policies
and procedures intended to prevent defects from occurring in the first place.
References
[1] http://en.wikipedia.org/w/index.php?title=Template:Software_development_process&action=edit
[2] Exploratory Testing (http://www.kaner.com/pdfs/ETatQAI.pdf), Cem Kaner, Florida Institute of Technology, Quality Assurance Institute
Worldwide Annual Software Testing Conference, Orlando, FL, November 2006
[3] Software Testing (http://www.ece.cmu.edu/~koopman/des_s99/sw_testing/) by Jiantao Pan, Carnegie Mellon University
[4] Leitner, A., Ciupa, I., Oriol, M., Meyer, B., Fiva, A., "Contract Driven Development = Test Driven Development – Writing Test Cases"
(http://se.inf.ethz.ch/people/leitner/publications/cdd_leitner_esec_fse_2007.pdf), Proceedings of ESEC/FSE'07: European Software
Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering 2007, (Dubrovnik, Croatia),
September 2007
[5] Section 1.1.2, Certified Tester Foundation Level Syllabus (http://www.istqb.org/downloads/syllabi/SyllabusFoundation.pdf),
International Software Testing Qualifications Board
[6] Principle 2, Section 1.3, Certified Tester Foundation Level Syllabus (http://www.bcs.org/upload/pdf/istqbsyll.pdf), International
Software Testing Qualifications Board
[7] Software errors cost U.S. economy $59.5 billion annually (http://www.abeacha.com/NIST_press_release_bugs_cost.htm), NIST report
[8][8] see D. Gelperin and W.C. Hetzel
[9][9] until 1956 it was the debugging oriented period, when testing was often associated to debugging: there was no clear difference between
testing and debugging.
[10] From 1957–1978 there was the demonstration oriented period where debugging and testing was distinguished now – in this period it was
shown, that software satisfies the requirements.
[11] The time between 1979–1982 is announced as the destruction oriented period, where the goal was to find errors.
[12] 1983–1987 is classified as the evaluation oriented period: intention here is that during the software lifecycle a product evaluation is
provided and measuring quality.
[13][13] From 1988 on it was seen as prevention oriented period where tests were to demonstrate that software satisfies its specification, to detect
faults and to prevent faults.
[14] Introduction (http://www.bullseye.com/coverage.html#intro), Code Coverage Analysis, Steve Cornett
[15] Paul Ammann; Jeff Offutt (2008). Introduction to Software Testing (http://books.google.com/books?id=leokXF8pLY0C&pg=PA215). p.
215 of 322 pages.
[16] K. J. Ross & Associates (http://www.kjross.com.au/cstp/)
[17] “We’re all part of the story” (http://stpcollaborative.com/knowledge/272-were-all-part-of-the-story) by David Strom, July 1, 2009
[18] IEEE article about differences in adoption of agile trends between experienced managers vs. young students of the Project Management
Institute (http://ieeexplore.ieee.org/Xplore/login.jsp?url=/iel5/10705/33795/01609838.pdf?temp=x). See also Agile adoption study
from 2007 (http://www.ambysoft.com/downloads/surveys/AgileAdoption2007.ppt)
[19] An example is Mark Fewster, Dorothy Graham: Software Test Automation. Addison Wesley, 1999, ISBN 0-201-33140-3.
[20] Microsoft Development Network Discussion on exactly this topic (http://channel9.msdn.com/forums/Coffeehouse/
402611-Are-you-a-Test-Driven-Developer/)

Software testing 18
Further reading
• Bertrand Meyer, "Seven Principles of Software Testing," Computer, vol. 41, no. 8, pp. 99–101, Aug. 2008,
doi:10.1109/MC.2008.306; available online (http://se.ethz.ch/~meyer/publications/testing/principles.pdf).
External links
• Software testing tools and products (http://www.dmoz.org/Computers/Programming/Software_Testing/
Products_and_Tools) on the Open Directory Project
• "Software that makes Software better" Economist.com (http://www.economist.com/science/tq/displaystory.
cfm?story_id=10789417)

19
Black-box testing
Black-box testing
Black box diagram
Black-box testing is a method of software testing that examines
the functionality of an application (e.g. what the software does)
without peering into its internal structures or workings (see
white-box testing). This method of test can be applied to virtually
every level of software testing: unit, integration, system and
acceptance. It typically comprises most if not all higher level testing, but can also dominate unit testing as well.
[citation needed]
Test procedures
Specific knowledge of the application's code/internal structure and programming knowledge in general is not
required. The tester is aware of what the software is supposed to do but is not aware of how it does it. For instance,
the tester is aware that a particular input returns a certain, invariable output but is not aware of how the software
produces the output in the first place.
Test cases
Test cases are built around specifications and requirements, i.e., what the application is supposed to do. Test cases
are generally derived from external descriptions of the software, including specifications, requirements and design
parameters. Although the tests used are primarily functional in nature, non-functional tests may also be used. The
test designer selects both valid and invalid inputs and determines the correct output without any knowledge of the
test object's internal structure.
Test design techniques
Typical black-box test design techniques include:
• Decision table testing
•• All-pairs testing
•• State transition Analysis
•• Equivalence partitioning
•• Boundary value analysis
• Cause–effect graph
•• Error guessing

Hacking
In penetration testing, black-box testing refers to a methodology where an ethical hacker has no knowledge of the
system being attacked. The goal of a black-box penetration test is to simulate an external hacking or cyber warfare
attack.
References
External links
• BCS SIGIST (British Computer Society Specialist Interest Group in Software Testing): Standard for Software
Component Testing (http://www.testingstandards.co.uk/Component Testing.pdf), Working Draft 3.4, 27.
April 2001.
Exploratory testing
Exploratory testing is an approach to software testing that is concisely described as simultaneous learning, test
design and test execution. Cem Kaner, who coined the term in 1983,
[1]
now defines exploratory testing as "a style of
software testing that emphasizes the personal freedom and responsibility of the individual tester to continually
optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result
interpretation as mutually supportive activities that run in parallel throughout the project."
[2]
While the software is being tested, the tester learns things that together with experience and creativity generates new
good tests to run. Exploratory testing is often thought of as a black box testing technique. Instead, those who have
studied it consider it a test approach that can be applied to any test technique, at any stage in the development
process. The key is not the test technique nor the item being tested or reviewed; the key is the cognitive engagement
of the tester, and the tester's responsibility for managing his or her time.
[3]
History
Exploratory testing has always been performed by skilled testers. In the early 1990s, ad hoc was too often
synonymous with sloppy and careless work. As a result, a group of test methodologists (now calling themselves the
Context-Driven School) began using the term "exploratory" seeking to emphasize the dominant thought process
involved in unscripted testing, and to begin to develop the practice into a teachable discipline. This new terminology
was first published by Cem Kaner in his book Testing Computer Software and expanded upon in Lessons Learned in
Software Testing. Exploratory testing can be as disciplined as any other intellectual activity.
Description
Exploratory testing seeks to find out how the software actually works, and to ask questions about how it will handle
difficult and easy cases. The quality of the testing is dependent on the tester's skill of inventing test cases and finding
defects. The more the tester knows about the product and different test methods, the better the testing will be.
To further explain, comparison can be made of freestyle exploratory testing to its antithesis scripted testing. In this
activity test cases are designed in advance. This includes both the individual steps and the expected results. These
tests are later performed by a tester who compares the actual result with the expected. When performing exploratory
testing, expectations are open. Some results may be predicted and expected; others may not. The tester configures,
operates, observes, and evaluates the product and its behaviour, critically investigating the result, and reporting
information that seems likely to be a bug (which threatens the value of the product to some person) or an issue
(which threatens the quality of the testing effort).

In reality, testing almost always is a combination of exploratory and scripted testing, but with a tendency towards
either one, depending on context.
According to Cem Kaner & James Marcus Bach, exploratory testing is more a mindset or "...a way of thinking about
testing" than a methodology.
[4]
They also say that it crosses a continuum from slightly exploratory (slightly
ambiguous or vaguely scripted testing) to highly exploratory (freestyle exploratory testing).
[5]
The documentation of exploratory testing ranges from documenting all tests performed to just documenting the bugs.
During pair testing, two persons create test cases together; one performs them, and the other documents.
Session-based testing is a method specifically designed to make exploratory testing auditable and measurable on a
wider scale.
Exploratory testers often use tools, including screen capture or video tools as a record of the exploratory session, or
tools to quickly help generate situations of interest, e.g. James Bach's Perlclip.
Benefits and drawbacks
The main advantage of exploratory testing is that less preparation is needed, important bugs are found quickly, and at
execution time, the approach tends to be more intellectually stimulating than execution of scripted tests.
Another major benefit is that testers can use deductive reasoning based on the results of previous results to guide
their future testing on the fly. They do not have to complete a current series of scripted tests before focusing in on or
moving on to exploring a more target rich environment. This also accelerates bug detection when used intelligently.
Another benefit is that, after initial testing, most bugs are discovered by some sort of exploratory testing. This can be
demonstrated logically by stating, "Programs that pass certain tests tend to continue to pass the same tests and are
more likely to fail other tests or scenarios that are yet to be explored."
Disadvantages are that tests invented and performed on the fly can't be reviewed in advance (and by that prevent
errors in code and test cases), and that it can be difficult to show exactly which tests have been run.
Freestyle exploratory test ideas, when revisited, are unlikely to be performed in exactly the same manner, which can
be an advantage if it is important to find new errors; or a disadvantage if it is more important to repeat specific
details of the earlier tests. This can be controlled with specific instruction to the tester, or by preparing automated
tests where feasible, appropriate, and necessary, and ideally as close to the unit level as possible.
Usage
Exploratory testing is particularly suitable if requirements and specifications are incomplete, or if there is lack of
time. The approach can also be used to verify that previous testing has found the most important defects.
References
[1] Kaner, Falk, and Nguyen, Testing Computer Software (Second Edition), Van Nostrand Reinhold, New York, 1993. p. 6, 7-11.
[2] Cem Kaner, A Tutorial in Exploratory Testing (http://www.kaner.com/pdfs/QAIExploring.pdf), p. 36.
[3] Cem Kaner, A Tutorial in Exploratory Testing (http://www.kaner.com/pdfs/QAIExploring.pdf), p. 37-39, 40- .
[4] Cem Kaner, James Bach, Exploratory & Risk Based Testing, www.testingeducation.org (http://www.testingeducation.org), 2004, p. 10
[5] Cem Kaner, James Bach, Exploratory & Risk Based Testing, www.testingeducation.org (http://www.testingeducation.org), 2004, p. 14

External links
• James Bach, Exploratory Testing Explained (http://www.satisfice.com/articles/et-article.pdf)
• Cem Kaner, James Bach, The Nature of Exploratory Testing (http://www.testingeducation.org/a/nature.pdf),
2004
• Cem Kaner, James Bach, The Seven Basic Principles of the Context-Driven School (http://www.
context-driven-testing.com)
• Jonathan Kohl, Exploratory Testing: Finding the Music of Software Investigation (http://www.methodsandtools.
com/archive/archive.php?id=65), Kohl Concepts Inc., 2007
• Chris Agruss, Bob Johnson, Ad Hoc Software Testing (http://www.testingcraft.com/ad_hoc_testing.pdf)
San Francisco depot
San Francisco depot is a mnemonic for the SFDPO software exploratory testing heuristic that was popularized by
James Bach. SFDPO stands for Structure, Function, Data, Platform and Operations. Each of these represents a
different aspect of a software product.
Structure
Structure is what the entire product is. This is its physical files, utility programs, physical materials such as user
docs, specifications and design docs, etc.
Function
Function is what the product does. This is the product's features. How does it handle errors? What is its UI? How
does it interface with the operating system?
Data
Data is what the product processes. What kinds of input does it process? This can be input from the user, the file
system, etc. What kind of output or reports does it generate? Does it come with default data? Is any of its input
sensitive to timing or sequencing?
Platform
Platform is what the product depends upon. What operating systems and related service packs, browsers, runtime
libraries, plug-ins, languages and locales, etc. does it run on? Does the user need to configure the environment? Does
it depend on third-party components?
Operations
Operations are scenarios in which the product will be used. Who are the application's users and what are their
patterns and sequences of input? Where and how will they use it? What are the different ways a user can use the
product's features?

San Francisco depot 23
External links
• How Do You Spell Testing?
[1]
References
[1] http://www.satisfice.com/articles/sfdpo.shtml
Session-based testing
Session-based testing is a software test method that aims to combine accountability and exploratory testing to
provide rapid defect discovery, creative on-the-fly test design, management control and metrics reporting. The
method can also be used in conjunction with scenario testing. Session-based testing was developed in 2000 by
Jonathan and James Bach.
Session-based testing can be used to introduce measurement and control to an immature test process and can form a
foundation for significant improvements in productivity and error detection. Session-based testing can offer benefits
when formal requirements are not present, incomplete, or changing rapidly.
Elements of session-based testing
Mission
The mission in Session Based Test Management identifies the purpose of the session, helping to focus the session
while still allowing for exploration of the system under test. According to Jon Bach, one of the co-founders of the
methodology, the mission tells us “what we are testing or what problems we are looking for.”
[1]
Charter
A charter is a goal or agenda for a test session. Charters are created by the test team prior to the start of testing, but
they may be added or changed at any time. Often charters are created from a specification, test plan, or by examining
results from previous sessions.
Session
An uninterrupted period of time spent testing, ideally lasting one to two hours. Each session is focused on a charter,
but testers can also explore new opportunities or issues during this time. The tester creates and executes test cases
based on ideas, heuristics or whatever frameworks to guide them and records their progress. This might be through
the use of written notes, video capture tools or by whatever method as deemed appropriate by the tester.
Session report
The session report records the test session. Usually this includes:
•• Charter.
•• Area tested.
•• Detailed notes on how testing was conducted.
• A list of any bugs found.
•• A list of issues (open questions, product or project concerns)
•• Any files the tester used or created to support their testing
•• Percentage of the session spent on the charter vs investigating new opportunities.
•• Percentage of the session spent on:

Session-based testing 24
•• Testing - creating and executing tests.
•• Bug investigation / reporting.
•• Session setup or other non-testing activities.
•• Session Start time and duration.
Debrief
A debrief is a short discussion between the manager and tester (or testers) about the session report. Jonathan Bach
uses the aconymn PROOF to help structure his debriefing. PROOF stands for:-
•• Past. What happened during the session?
•• Results. What was achieved during the session?
•• Obstacles. What got in the way of good testing?
•• Outlook. What still needs to be done?
• Feelings. How does the tester feel about all this?
[2]
Parsing results
With a standardized Session Report, software tools can be used to parse and store the results as aggregate data for
reporting and metrics. This allows reporting on the number of sessions per area or a breakdown of time spent on
testing, bug investigation, and setup / other activities.
Planning
Testers using session-based testing can adjust their testing daily to fit the needs of the project. Charters can be added
or dropped over time as tests are executed and/or requirements change.
References
[1] First published 11/2000 in STQE magazine, today known as Better Software http://www.stickyminds.com/BetterSoftware/magazine.asp
[2] http://www.satisfice.com/articles/sbtm.pdf
External links
• Session-Based Test Management Site (http://www.satisfice.com/sbtm/)
• How to Manage and Measure ET (http://www.quardev.com/content/whitepapers/
how_measure_exploratory_testing.pdf)
• Session-Based Test Lite (http://www.quardev.com/articles/sbt_lite)
• Adventures in Session-Based Testing (http://www.workroom-productions.com/papers/AiSBTv1.2.pdf)
• Session-Based Test Management (http://www.satisfice.com/articles/sbtm.pdf)
• Better Software Magazine (http://www.stickyminds.com/BetterSoftware/magazine.asp)

Scenario testing 25
Scenario testing
Scenario testing is a software testing activity that uses scenarios: hypothetical stories to help the tester work through
a complex problem or test system. The ideal scenario test is a credible, complex, compelling or motivating story the
outcome of which is easy to evaluate. These tests are usually different from test cases in that test cases are single
steps whereas scenarios cover a number of steps.
History
Kaner coined the phrase scenario test by October 2003. He commented that one of the most difficult aspects of
testing was maintaining step-by-step test cases along with their expected results. His paper attempted to find a way to
reduce the re-work of complicated written tests and incorporate the ease of use cases.
A few months later, Buwalda wrote about a similar approach he had been using that he called "soap opera testing".
Like television soap operas these tests were both exaggerated in activity and condensed in time. The key to both
approaches was to avoid step-by-step testing instructions with expected results and instead replaced them with a
narrative that gave freedom to the tester while confining the scope of the test.
Methods
System scenarios
In this method only those sets of realistic, user activities that cover several components in the system are used as
scenario tests. Development of system scenario can be done using:
[citation needed]
1. Story lines
2.2. State transitions
3. Business verticals
4.4. Implementation story from customers
Use-case and role-based scenarios
In this method the focus is on how a user uses the system with different roles and
environment.Wikipedia:Verifiability
References

Equivalence partitioning
Equivalence partitioning (also called Equivalence Class Partitioning or ECP) is a software testing technique that
divides the input data of a software unit into partitions of equivalent data from which test cases can be derived. In
principle, test cases are designed to cover each partition at least once. This technique tries to define test cases that
uncover classes of errors, thereby reducing the total number of test cases that must be developed. An advantage of
this approach is reduction in the time required for testing a software due to lesser number of test cases.
Equivalence partitioning is typically applied to the inputs of a tested component, but may be applied to the outputs in
rare cases. The equivalence partitions are usually derived from the requirements specification for input attributes that
influence the processing of the test object.
The fundamental concept of ECP comes from equivalence class which in turn comes from equivalence relation. A
software system is in effect a computable function implemented as an algorithm in some implementation
programming language. Given an input test vector some instructions of that algorithm get covered, ( see code
coverage for details ) others do not. This gives the interesting relationship between input test vectors:- is an
equivalence relation between test vectors if and only if the coverage foot print of the vectors are exactly
the same, that is, they cover the same instructions, at same step. This would evidently mean that the relation cover
would partition the input vector space of the test vector into multiple equivalence class. This partitioning is called
equivalence class partitioning of test input. If there are equivalent classes, only vectors are sufficient to fully
cover the system.
The demonstration can be done using a function written in C:
int safe_add( int a, int b )
{
int c = a + b;
if ( a >= 0 && b >= 0 && c < 0 )
{
fprintf ( stderr, "Overflow!n" );
}
if ( a < 0 && b < 0 && c >= 0 )
{
fprintf ( stderr, "Underflow!n" );
}
return c;
}
On the basis of the code, the input vectors of are partitioned. The blocks we need to cover are the overflow
statement and the underflow statement and neither of these 2. That gives rise to 3 equivalent classes, from the code
review itself.

Demonstrating Equivalence Class Partitioning
To solve the input problem, we take refuge in the inequation
we note that there is a fixed size of Integer (computer science) hence,
the z can be replaced with:-
and
with and
The values of the test vector at the strict condition of the equality that
is and are called
the boundary values, Boundary-value analysis has detailed information about it. Note that the graph only covers the
overflow case, first quadrant for X and Y positive values.
In general an input has certain ranges which are valid and other ranges which are invalid. Invalid data here does not
mean that the data is incorrect, it means that this data lies outside of specific partition. This may be best explained by
the example of a function which takes a parameter "month". The valid range for the month is 1 to 12, representing
January to December. This valid range is called a partition. In this example there are two further partitions of invalid
ranges. The first invalid partition would be <= 0 and the second invalid partition would be >= 13.
... -2 -1 0 1 .............. 12 13 14 15 .....
--------------|-------------------|---------------------
invalid partition 1 valid partition invalid partition 2
The testing theory related to equivalence partitioning says that only one test case of each partition is needed to
evaluate the behaviour of the program for the related partition. In other words it is sufficient to select one test case
out of each partition to check the behaviour of the program. To use more or even all test cases of a partition will not
find new faults in the program. The values within one partition are considered to be "equivalent". Thus the number
of test cases can be reduced considerably.
An additional effect of applying this technique is that you also find the so-called "dirty" test cases. An inexperienced
tester may be tempted to use as test cases the input data 1 to 12 for the month and forget to select some out of the
invalid partitions. This would lead to a huge number of unnecessary test cases on the one hand, and a lack of test
cases for the dirty ranges on the other hand.
The tendency is to relate equivalence partitioning to so called black box testing which is strictly checking a software
component at its interface, without consideration of internal structures of the software. But having a closer look at
the subject there are cases where it applies to grey box testing as well. Imagine an interface to a component which
has a valid range between 1 and 12 like the example above. However internally the function may have a
differentiation of values between 1 and 6 and the values between 7 and 12. Depending upon the input value the
software internally will run through different paths to perform slightly different actions. Regarding the input and
output interfaces to the component this difference will not be noticed, however in your grey-box testing you would
like to make sure that both paths are examined. To achieve this it is necessary to introduce additional equivalence
partitions which would not be needed for black-box testing. For this example this would be:
... -2 -1 0 1 ..... 6 7 ..... 12 13 14 15 .....
--------------|---------|----------|---------------------
invalid partition 1 P1 P2 invalid partition 2
valid partitions

To check for the expected results you would need to evaluate some internal intermediate values rather than the
output interface. It is not necessary that we should use multiple values from each partition. In the above scenario we
can take -2 from invalid partition 1, 6 from valid partition P1, 7 from valid partition P2 and 15 from invalid partition
2.
Equivalence partitioning is not a stand alone method to determine test cases. It has to be supplemented by boundary
value analysis. Having determined the partitions of possible inputs the method of boundary value analysis has to be
applied to select the most effective test cases out of these partitions.
Further reading
• The Testing Standards Working Party website
[1]
• Parteg
[2]
, a free test generation tool that is combining test path generation from UML state machines with
equivalence class generation of input values.
•• [3]
References
[1] http://www.testingstandards.co.uk
[2] http://parteg.sourceforge.net
[3] http://books.google.co.in/books/about/Software_Testing_Techniques.html
Boundary-value analysis
Boundary value analysis is a software testing technique in which tests are designed to include representatives of
boundary values. The idea comes from the Boundary (topology). Given that we have a set of test vectors to test the
system, a topology can be defined on that set. Those inputs which belong to the same equivalence class as defined by
the equivalence partitioning theory would constitute the basis (topology). Given that the basis sets are neighbors as
defined in neighbourhood (mathematics), there would exist a boundary between them. The test vectors on either side
of the boundary are called boundary values. In practice this would require that the test vectors can be ordered, and
that the individual parameters follows some kind of order ( either partial order or total order ).
Formal Definition
Formally the boundary values can be defined as below:- Let the set of the test vectors be . Let's
assume that there is an ordering relation defined over them, as . Let be two equivalent classes. Assume
that test vector and . If or then the classes are in
neighbourhood (mathematics) and the values are boundary values.
In plainer English, values on the minimum and maximum edges of an equivalence partition are tested. The values
could be input or output ranges of a software component, can also be the internal implementation. Since these
boundaries are common locations for errors that result in software faults they are frequently exercised in test cases.
Application
The expected input and output values to the software component should be extracted from the component
specification. The values are then grouped into sets with identifiable boundaries. Each set, or partition, contains
values that are expected to be processed by the component in the same way. Partitioning of test data ranges is
explained in the equivalence partitioning test case design technique. It is important to consider both valid and invalid
partitions when designing test cases.

The demonstration can be done using a function written in c language
int safe_add( int a, int b )
{
int c = a + b ;
if ( a >= 0 && b >= 0 && c < 0 )
{
fprintf ( stderr, "Overflow!n");
}
if ( a < 0 && b < 0 && c >= 0 )
{
fprintf ( stderr, "Underflow!n");
}
return c;
}
On the basis of the code, the input vectors of are partitioned. The blocks we need to cover are the overflow
statement and the underflow statement and neither of these 2. That gives rise to 3 equivalent classes, from the code
review itself.
Demonstrating Boundary Values (Orange)
we note that there is a fixed size of Integer (computer science) hence:-
We note that the input parameter a and b both are integers, hence total
order exists on them. When we compute the equalities:-
we get back the values which are on the boundary, inclusive, that is
these pairs of are valid combinations, and no underflow or
overflow would happen for them.
On the other hand:-
gives pairs of which are invalid combinations, Overflow would occur for
them. In the same way:-
gives pairs of which are invalid combinations, Underflow would occur for
them.
Boundary values (drawn only for the overflow case) are being shown as the orange line in the right hand side figure.
For another example, if the input values were months of the year, expressed as integers, the input parameter 'month'
might have the following partitions:
... -2 -1 0 1 .............. 12 13 14 15 .....
--------------|-------------------|-------------------
invalid partition 1 valid partition invalid partition 2
The boundary between two partitions is the place where the behavior of the application changes and is not a real
number itself. The boundary value is the minimum (or maximum) value that is at the boundary. The number 0 is the
maximum number in the first partition, the number 1 is the minimum value in the second partition, both are
boundary values. Test cases should be created to generate inputs or outputs that will fall on and to either side of each
boundary, which results in two cases per boundary. The test cases on each side of a boundary should be in the
smallest increment possible for the component under test, for an integer this is 1, but if the input was a decimal with

2 places then it would be .01. In the example above there are boundary values at 0,1 and 12,13 and each should be
tested.
Boundary value analysis does not require invalid partitions. Take an example where a heater is turned on if the
temperature is 10 degrees or colder. There are two partitions (temperature<=10, temperature>10) and two boundary
values to be tested (temperature=10, temperature=11).
Where a boundary value falls within the invalid partition the test case is designed to ensure the software component
handles the value in a controlled manner. Boundary value analysis can be used throughout the testing cycle and is
equally applicable at all testing phases.
References
• The Testing Standards Working Party
[1]
website.
All-pairs testing
In computer science, all-pairs testing or pairwise testing is a combinatorial method of software testing that, for
each pair of input parameters to a system (typically, a software algorithm), tests all possible discrete combinations of
those parameters. Using carefully chosen test vectors, this can be done much faster than an exhaustive search of all
combinations of all parameters, by "parallelizing" the tests of parameter pairs.
Rationale
Assume that the test functionWikipedia:Please clarify has parameters given in a set
. The range of the parameters are given by . Let's assume that
. We notice that the set of choices of ranges can be a multiset Wikipedia:Please clarify,
because there can be multiple parameters having same number of choices.
Let's define as one of the maximum of the multiset . Then, the number of pair-wise test cases on this
test function would be:-
Plainly that would mean, if the and then the number of tests is
typically O(nm). Where n and m are the number of possibilities for each of the two parameters with the most choices.
Parameter Name Value 1 Value 2 Value 3 Value 4
Enabled True False * *
Choice Type 1 2 3 *
Category a b c d
In this case the parameters are Enabled with choices range of 2, Choice Type with 3, and Category with 4. That
would mean: Hence, n = 4, m = 3 and number of tests would be 12. The pict tool generated
pairwise test cases on the input looks like:-

Enabled Choice Type Category
True 3 a
True 1 d
False 1 c
False 2 d
True 2 c
False 2 a
False 1 a
False 3 b
True 2 b
True 3 d
False 3 c
True 1 b
The below table would generate a multiset :
Parameter Name Value 1 Value 2 Value 3 Value 4
Enabled True False * *
Choice Type 1 2 3 4
Category a b c d
In this case the parameters are Enabled with choices range of 2, Choice Type with 4, and Category with 4. That
would mean:- and it is a multiset. Hence, n = 4, m = 4 and number of tests would be 16, which are
shown in the below table:-
Enabled Choice Type Category
True 3 b
False 3 d
True 2 a
True 1 c
False 4 b
False 1 a
False 2 b
True 4 d
True 4 a
True 2 d
False 4 c
False 1 b
True 2 c
False 1 d
False 3 a
True 3 c

The reasoning behind all-pairs testing is this: the simplest bugs in a program are generally triggered by a single input
parameter. The next simplest category of bugs consists of those dependent on interactions between pairs of
parameters, which can be caught with all-pairs testing. Bugs involving interactions between three or more parameters
are progressively less common, while at the same time being progressively more expensive to find by exhaustive
testing, which has as its limit the exhaustive testing of all possible inputs.
This can be further generalized.
[citation needed]
The idea is to apply sorting to the set so that
gets ordered too. Let the sorted set be a tuple :-
Now we can take the set and call it the pairwise testing. Generalizing further we can
take the set and call it the 3-wise testing. Eventually, we can say
T-wise testing.
The N-wise testing then would just be, all possible combinations from the above formula.
One of the main strengths of combinatorial technique is that it enables a significant reduction of the number of test
cases without compromising functional coverage. Many testing methods regard all-pairs testing of a system or
subsystem as a reasonable cost-benefit compromise between often computationally infeasible higher-order
combinatorial testing methods, and less exhaustive methods which fail to exercise all possible pairs of parameters.
Because no testing technique can find all bugs, all-pairs testing is typically used together with other quality
assurance techniques such as unit testing, symbolic execution, fuzz testing, and code review.
Notes
External links
• Combinatorialtesting.com; Includes clearly written introductions to pairwise and other, more thorough, methods
of combinatorial testing (http://www.combinatorialtesting.com)
• Hexawise.com - Pairwise test case generating tool with both free and commercial versions (also provides more
thorough 3-way, 4-way, 5-way, and 6-way coverage solutions) (http://hexawise.com/)
• Pairwise Testing Comes of Age - Review including history, examples, issues, research (http://testcover.com/
pub/background/stareast2008.ppt)
• Pairwise Testing: Combinatorial Test Case Generation (http://www.pairwise.org/)
• Pairwise testing (http://www.developsense.com/testing/PairwiseTesting.html)
• All-pairs testing (http://www.mcdowella.demon.co.uk/allPairs.html)
• Pairwise and generalized t-way combinatorial testing (http://csrc.nist.gov/acts/)
• TestApi - .NET API library for testing, providing a variation generation API (http://testapi.codeplex.com)
• NTestCaseBuilder - another .NET API library; focussed purely on combinatoric testing and scalability of test case
generation (http://github.com/sageserpent-open/NTestCaseBuilder)
• JCombinatorial - an open-source library that facilitates all-pairs testing with JUnit (http://code.google.com/p/
jcombinatorial/)
• "rdExpert Software for Orthogonal Array Testing" (http://www.phadkeassociates.com/
index_rdexperttestplanning.htm). Phadke Associates, Inc. "Commercial toolset for Orthogonal Array and
PairWise Testing."
• Pairwise Testing in the Real World: Practical Extensions to Test-Case Scenarios (http://msdn.microsoft.com/
en-us/library/cc150619.aspx)
• A Laboratory for Combinatorial Interaction Testing (http://code.google.com/a/eclipselabs.org/p/citlab/)

Fuzz testing 33
Fuzz testing
Fuzz testing or fuzzing is a software testing technique, often automated or semi-automated, that involves providing
invalid, unexpected, or random data to the inputs of a computer program. The program is then monitored for
exceptions such as crashes, or failing built-in code assertions or for finding potential memory leaks. Fuzzing is
commonly used to test for security problems in software or computer systems.
The field of fuzzing originates with Barton Miller at the University of Wisconsin in 1988. This early work includes
not only the use of random unstructured testing, but also a systematic set of tools to evaluate a wide variety of
software utilities on a variety of platforms, along with a systematic analysis of the kinds of errors that were exposed
by this kind of testing. In addition, they provided public access to their tool source code, test procedures and raw
result data.
There are two forms of fuzzing program, mutation-based and generation-based, which can be employed as white-,
grey-, or black-box testing. File formats and network protocols are the most common targets of testing, but any type
of program input can be fuzzed. Interesting inputs include environment variables, keyboard and mouse events, and
sequences of API calls. Even items not normally considered "input" can be fuzzed, such as the contents of databases,
shared memory, or the precise interleaving of threads.
For the purpose of security, input that crosses a trust boundary is often the most interesting. For example, it is more
important to fuzz code that handles the upload of a file by any user than it is to fuzz the code that parses a
configuration file that is accessible only to a privileged user.
History
The term "fuzz" or "fuzzing" originates from a 1988 class project, taught by Barton Miller at the University of
Wisconsin. The project developed a basic command-line fuzzer to test the reliability of Unix programs by
bombarding them with random data until they crashed. The test was repeated in 1995, expanded to include testing of
GUI-based tools (such as the X Window System), network protocols, and system library APIs. Follow-on work
included testing command- and GUI-based applications on both Windows and Mac OS X.
One of the earliest examples of fuzzing dates from before 1983. "The Monkey" was a Macintosh application
developed by Steve Capps prior to 1983. It used journaling hooks to feed random events into Mac programs, and was
used to test for bugs in MacPaint.
Another early fuzz testing tool was crashme, first released in 1991, which was intended to test the robustness of Unix
and Unix-like operating systems by executing random machine instructions.
Uses
Fuzz testing is often employed as a black-box testing methodology in large software projects where a budget exists
to develop test tools. Fuzz testing is one of the techniques that offers a high benefit-to-cost ratio.
The technique can only provide a random sample of the system's behavior, and in many cases passing a fuzz test may
only demonstrate that a piece of software can handle exceptions without crashing, rather than behaving correctly.
This means fuzz testing is an assurance of overall quality, rather than a bug-finding tool, and not a substitute for
exhaustive testing or formal methods.
As a gross measurement of reliability, fuzzing can suggest which parts of a program should get special attention, in
the form of a code audit, application of static code analysis, or partial rewrites.

Fuzz testing 34
Types of bugs
As well as testing for outright crashes, fuzz testing is used to find bugs such as assertion failures and memory leaks
(when coupled with a memory debugger). The methodology is useful against large applications, where any bug
affecting memory safety is likely to be a severe vulnerability.
Since fuzzing often generates invalid input it is used for testing error-handling routines, which are important for
software that does not control its input. Simple fuzzing can be thought of as a way to automate negative testing.
Fuzzing can also find some types of "correctness" bugs. For example, it can be used to find incorrect-serialization
bugs by complaining whenever a program's serializer emits something that the same program's parser rejects. It can
also find unintentional differences between two versions of a program or between two implementations of the same
specification.
Techniques
Fuzzing programs fall into two different categories. Mutation-based fuzzers mutate existing data samples to create
test data while generation-based fuzzers define new test data based on models of the input.
The simplest form of fuzzing technique is sending a stream of random bits to software, either as command line
options, randomly mutated protocol packets, or as events. This technique of random inputs continues to be a
powerful tool to find bugs in command-line applications, network protocols, and GUI-based applications and
services. Another common technique that is easy to implement is mutating existing input (e.g. files from a test suite)
by flipping bits at random or moving blocks of the file around. However, the most successful fuzzers have detailed
understanding of the format or protocol being tested.
The understanding can be based on a specification. A specification-based fuzzer involves writing the entire array of
specifications into the tool, and then using model-based test generation techniques in walking through the
specifications and adding anomalies in the data contents, structures, messages, and sequences. This "smart fuzzing"
technique is also known as robustness testing, syntax testing, grammar testing, and (input) fault injection. The
protocol awareness can also be created heuristically from examples using a tool such as Sequitur. These fuzzers can
generate test cases from scratch, or they can mutate examples from test suites or real life. They can concentrate on
valid or invalid input, with mostly-valid input tending to trigger the "deepest" error cases.
There are two limitations of protocol-based fuzzing based on protocol implementations of published specifications:
1) Testing cannot proceed until the specification is relatively mature, since a specification is a prerequisite for
writing such a fuzzer; and 2) Many useful protocols are proprietary, or involve proprietary extensions to published
protocols. If fuzzing is based only on published specifications, test coverage for new or proprietary protocols will be
limited or nonexistent.
Fuzz testing can be combined with other testing techniques. White-box fuzzing uses symbolic execution and
constraint solving. Evolutionary fuzzing leverages feedback from an heuristic (E.g., code coverage in grey-box
harnessing, or a modeled attacker behavior in black-box harnessing) effectively automating the approach of
exploratory testing.

Software testing

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (9)

Similar to Software testing

Similar to Software testing (20)

More from Abrianto Nugraha

More from Abrianto Nugraha (20)

Recently uploaded

Recently uploaded (20)

Software testing