2. 2
In this presentation I will
Explain why this subject matter
should be of value to you in your
current role
Give an overview of where Black-
Box testing fits into software testing
theory
Outline how Black-Box testing can be
applied
3. 3
What is Black-Box Testing?
Black-Box testing is a general testing
strategy that is part of a larger conceptual
framework of test theory
It provides a system for evaluating the
effectiveness of various methodologies to
meet software testing goals
It provides a mind set for increasing the
effectiveness of individual testing efforts
4. 4
… a couple questions before we get
started.
What is the role of our department?
Is our role to produce documentation?
Is our role to make product decisions?
Is our role to invent and follow processes?
Is our role to create and execute test plans?
Is our role to create and write testing tools?
Is our role to validate products are bug-free?
5. 5
Or is our role to… Find and get fixed as
many of a products problems (bugs) as
possible before it goes to market?
Aggressively find ways to break the product
Provide management quality information for
evaluating risk
Provide programmers timely information so
they can fix and prevent bugs
Enable the company to ship as bug-free and
high-quality a product as possible given market
demands and resource constraints
7. 7
Do the strategies and methods that follow
from an understanding of Black-Box testing
help find and fix bugs?
While bug counts are not especially
valid or reliable as a metric… as a rough
indicator they may tell us something.
Does my record as a tester provide an
indication that some understanding of
test theory is useful?
Beyond an understanding of test theory
and practice do I have any special or
unique advantages over other testers?
8. 8
From a technical perspective I am at a
disadvantage relative to most “Engineers” in
our department
•I have almost no practical programming
knowledge or experience
•My networking and systems knowledge
and experience are minimal relative to
most other team members
•I started with no gaming industry
experience or other trade specific
knowledge
9. 9
Last Six Months
ABS “bugs” Found and Fixed
0
5
10
15
20
25
30
35
40
X Y Z Avrg. P Q R Jason
ABS Found ABS Fixed
10. 10
In terms of measurable results relevant to
our primary role as a department
Twice as many ABS issues found as
average
Three times as many ABS issues fixed as
average
As a case study these results indicate a
likely effect for my testing methods and
strategy
16. 16
Assume the software is broken.
Finding and fixing bugs accounts for 40-80% of software
industry development costs [Kaner, Falk & Nguyen;
1993]
Operations and Maintenance costs account for 67% of a
software products life-cycle costs. After the development
stage it costs over 50 times as much to fix issues [Kaner,
Falk & Nguyen; 1993]
“Bug-Free” software released to testers from
programmers is estimated to contain one to three bugs per
hundred statements [Beizer; 1990]
We are more likely to find the issues a product has if we
assume such issues exist. If we believe the software is
robust and are just validating that it is market ready we
are more likely to dismiss problems as flukes.
17. 17
“Complete” testing is not possible (the
domain is too large and complex)
You can’t test every path.
A twenty Statement program can contain 100 trillion unique paths.
The universe is estimated to be only 40 trillion seconds old
[Kaner, Falk & Nguyen; 1993] [Meyers; 1979]
You can’t catch all design problems
To do so you would have to predict all possible system interactions
and end-user scenarios
You can’t even prove the logic
With potential system and component interactions you can only
prove logic to be internally consistent
18. 18
Effective Testing Requires a
Strategy
A strategy is a long term plan of action
designed to achieve a set of goals
Creating a good strategy requires a clear
understanding of goals
A good strategy includes a number of tactics
(methods) to achieve the goal
A strategy employs simplifying concepts or
organizing principles
Implicitly, a strategy is an adaptable toolbox of
heuristics as opposed to a rigid set of rules
20. 20
White-box testing
Testing strategy based on the structure of the
object being tested
Requires access to and understanding of the
source code (For example-- execute each line of
code and check expected outcomes)
Programmers are estimated to catch 99% of their
coding bugs using White-Box methods before
releasing software to testers
Generally most effective at the unit test level
21. 21
Black-Box Testing
A testing strategy based on system
behavior and requirements
Theoretically no knowledge of how an
object operates or it’s structure is needed
Views software from a functional end-user
like perspective as opposed to asking if the
software’s internal logic is correct
Generally most effective at the system test
level.
22. 22
Hybrid Testing
Combination of Black-Box and White-Box
testing.
Optimal for integration testing at the
component level
23. 23
Some methods used with both Black &
White Box Testing
Verification of Specs
Accuracy of Computations
Usability – Use case /scenario
Boundary Conditions - both valid and invalid
Equivalence partitioning
All-pairs testing
Performance
State Transitions
Load: Volume, Stress, Storage
Error Recovery
Security
24. 24
A Good Strategic Mind Set for Testing
Have a reliable process to validate and break products in alignment
with stated and unstated requirements
Get a testable product design with clear understanding of product
requirements
Use Sanity testing, Smoke testing , disaster checks and other
methods to find bugs early (especially the serious ones).
Remember that testing is never “complete”. Full verification is not
possible. All non-trivial programs have bugs. The job of testing is to
find those bugs, especially the serious ones. The more of the bugs
you can find the better.
25. 25
Strategic Considerations Regarding
Test Plans
Test plans/scripts decline in effectiveness with usage
(known as the “pesticide effect”). More issues are often
found writing test plans than executing them
Test plans are labor intensive both to maintain and
execute
Test plans are generally more effective when viewed as a
tour bus (get off the bus and poke around if you see
something unusual). Use purely as a validation process
that must be completed should be minimized / avoided
Essential validation checks should be automated when
possible
26. 26
Some Other Considerations
Have a good oracle or way of determining if an
issue is a bug or not.
Present issues in a way that engineers and
management will want to get them fixed.
Fixing this one is trivial… it should be easy
It’s not really a corner case, have you thought about…
I need your help figuring out how to reproduce this one…
Lobby for the serious ones. Being without
decision making authority, but responsible for
quality puts testers in a politically challenging
position. You need to be an effective advocate.
27. 27
Goals of Software Testing
Give programmers information to fix and
prevent bugs
Give management information to evaluate
risk
Get as bug-free a product as warranted
