A look at both standard and exploratory testing, introductions and what tests should contain.

1. Explore!
Software Testing In An Exploratory Style:
Learning and
Discoveries

2. Why Test?
Minimizes the risk gap
Outside perspective
Edge cases
User comfort
Does it work
Financial costs
Build The Right Thing
Does it meet the needs of a
user?
Build It Correctly
Does it function as it should?

3. Why Test?
Software is a relationship between people – who are both
rational and emotional.
Projects are completed, with both uncertainty and time
pressures.
We give it our best, but there will always be errors,
inexperience, and carelessness.
We focus on not misleading either colleagues or clients,
knowingly or by neglect.

4. Why Test?
Testing is an activity – and performance. Not a
remainder.
It discovers the status and threats of a product, so correct
decisions can be made about it.
Testers accept responsibility for their work, not overall
product quality – but can contribute to it.
Credible, cost-effective testing is the goal, and
communication of anything that might interfere with it.

5. Costs?

NIST report estimates that business loses between
$22 and $59 billion annually (in 2002 dollars) from
non-functional products

Stecklein, J. et al. (2004) conducted a study for NASA type of products
and found that if finding an error in requirements phase costs 1 unit then
it increases so significantly that the same defect if undetected until
operations phase can cost to fix anywhere from 29 units to more than
1,500 units. So if a unit costs $1000 then at requirements phase a defect
would cost a $1,000 to fix while in the operations it can cost from
$29,000 to $150,000 to fix.

6. A good tester is always learning how to
Be comfortable with
uncertainty and change
Design tests
Learn from other people
To effectively use testing
tools
Envision what the
product can be
Identify potential bugs in
the product
Construct and share
stories of the tests

7. The Testing Mentality

You are an agent first, but
a curious one

And one who takes pride
in their work.

You can provide
information, and analyze
the results.
These skills allow you to be
an effective, and efficient
critic of a check, and
explore options to write
them better, or find a new
and different way to
check.

8. Standards and Models

IEEE Std 1012-2012 IEEE Standard for System and Software
Verification and Validation

SEI Capability Maturity Model (CMM)

ISO SPICE

ISO 12207

European Cooperation for Space Standardization (ECSS)

GDPR

9. Criteria are chosen
by both risk/safety level and
organization or products.
Testing is done under scrutiny,
with uncertainty,
in a timely, ethical manner.

10. Testing Activities

11. Testing is not just finding out what a specific
activity does.
You need to be able to evaluate and model the product
- and question this model -
then create the conditions where you can model it,
and gain confidence, credibility, and motivation.
This way, risks are communicated, so informed decisions
can be made.
If this is done early and often, the quality of the end product
can be improved.

12. Testing is in your head!
Specifications
Product
Report
Coverage
Use models (technical, domain, and product ones), think
both deep and wide. And don't discount your instincts.
Good testing doesn't just happen in physical space!

13. Testing throughout development is vital
Inspection and
analysis of user
requirements
Test system
requirements and
setup
Tasks, schedules,
test scope
division of works
Test check:
inspection and
analysis
Test check:
design and
implementation
Test execution
and
documentation

14. Check versus Suite

A test check examines
one aspect of a system:
for example, using
various inputs on one
page to test the reaction
of the system.

This is testing a specific
behavior.

A test suite, like its
musical namesake,
covers all important
aspects of what needs to
be tested.

This is testing the entire
system.

15. Test Plans Include

Manual testing: A human goes in, and does the tests

Automatic testing: a program is made to run tests

The test plan will include such things as who is
responsible, what is to be tested, how it should be
tested, and milestones to meet

16. Common Information for Tests

Unique Test Case name and/or ID

Test Scenario and test summary or description

Prerequisites & setup

Test data & inputs

Execution Steps

17. Common Information for Tests

Expected behavior

Assumptions
– Any assumptions that are made about the system or
the test case.

Actual results

Status

18. Writing a good test check
Test one thing only
Make it exact and
accurate
Make it clear and easy to
understand
Make it small
Make it independent
No extra steps or words
Make it traceable to
requirements
Make it repeatable
Make it consistent,
internally and over the
test suite

19. Learn from your checks: they can be
improved, they can be wrong,
OR
they can show a bug that isn't important.
Checks show something can work,
not that it will.

20. Elements of Testing
Make accurate models of the product and the risks
Consider what to cover
Determine which procedures cover the product, and
apply the oracles (quality checks)
Configure and operate the testing system(s)
Observe, evaluate and report the ACTUAL test results

21. ___ coverage is how completely you have examined the
product, with a model of ___
What aspects of the product were tested?
What risks did you look for?
What requirements have you tested for?

22. Test Factors
Determine which of these test factors are the most
important for each test.
Model Specific Factors Specific procedure Test
Incidental Factors

23. SFDIPOT
(San Francisco Depot)
Coverage Includes:
Structure Platform
Function Operations
Data Time
Interfaces

24. Product Elements
Structural Coverage
Test what it's made of
Example:
Print Testing
Files associated with Printing
Code modules that trigger printing
Code statements and branches in the printing modules

25. Product Elements
Functional Coverage
Test what it does
Example:
Print Testing
Print, page setup, and print preview
Print range, print copies, zoom
Print all, print current page, print specific page

26. Product Elements
Data Coverage
Test what it does it to
Example:
Print Testing
Types of documents
Item in document, size and structure
Data about how to print (zoom, copy count)

27. Product Elements
Interface Coverage
Test how we interact with it
Example:
Print Testing
Menus, windows, dialogs, and controls
Launch and control interfaces
Print to or from files
USB printer interface protocol(s)
Physical connections between printer & computer or network

28. Product Elements
Platform Coverage
Test what it depends on
Example:
Print Testing
Printers, spoolers, network behavior
Computers
Operating Systems
Printer drivers

29. Product Elements
Operations Coverage
Test how it's actually used
Example:
Print Testing
Use defaults
Use realistic environments
Use realistic scenarios
Use complex flow