Software testing: an introduction - 2017

www.unamur.be
Software Testing:
an Introduction
Xavier Devroey
<x.d.m.devroey@tudelft.nl>
SERG, Delft University of Technology, The Netherlands
INFO M110 : Ingénierie du logiciel
2017 - 2018

Error, fault, failure
Developper
1. makes
errors
2. failure
3. caused by
a fault (bug)

Error, fault, failure, incident
Developper
User
4. reports the incident
5. assigned to
Bug tracker

Why you should care about testing

Moth, found in one of the relays of Harvard's Mark II,
taped to a log book (September 1947).

Vol 501 d'Ariane 5 (1996)
• Carrying 370 millions $
• Flight for about 40 sec.
• Integer overflow due to a
64-bit floating point number
conversion into a 16-bit
signed integer
https://fr.wikipedia.org/wiki/Vol_501_d%27Ariane_5

Millennium bug (1999)
• Date format problem
https://en.wikipedia.org/wiki/Year_2000_problem
01 DATE.
05 DAY PIC 99.
05 MONTH PIC 99.
05 YEAR PIC 99.
• Year declared on 2 digits

WoW Corrupted Blood Incident (2005)
• Boss casts a highly
contagious hit point-draining
spell (Corrupted Blood)
https://en.wikipedia.org/wiki/Corrupted_Blood_incident
• Spell intended to last only
seconds and functions only
within the boss area
• … but spreads through
hunter pets and NPCs

Heartbleed (2014)
• Security bug in OpenSSL
• Buffer overread
• Missing bounds check
https://en.wikipedia.org/wiki/Heartbleed

Seville Airbus A400M Atlas crash (2015)
https://en.wikipedia.org/wiki/2015_Seville_Airbus_A400M_Atlas_crash
• Three engines stopped 
working
• Engines software
configuration issue
« […] a weakness in the test procedure of planes before they fly, or a
problem that results from the implementation of these procedures. »
• Pre-delivery test flight

A test engineer walks
into a bar and
• orders a beer
• orders 0 beers
• orders 9999999 beers
• orders a lizard
• orders -1 beers
• orders a "sfdeljknesv"
— Bill Sempf (@sempf)

Menu
System 
under 
testSpecification

Menu
System 
under 
testSpecification
Test case

Menu
System 
under 
testSpecification
Test case
Output

Menu
System 
under 
testSpecification
Test case
Output
Pass or fail?

Software testing consists of the dynamic verification  
that a program provides expected behaviours  
on a finite set of test cases, suitably selected from the
usually infinite execution domain.
[SWEBOK v3]

vs static
Formal, semiformal, or informal
• compiler warnings (code analysis),
• javadoc warnings (documentation analysis),
• design inspection (model analysis),
• theorem proving (e.g., Event-B),
• code analysis
• Code review
• IDE warnings
• FindBugs (http://findbugs.sourceforge.net)

usually infinite execution domain.The oracle problem
“An oracle is any human or mechanical agent that
decides whether a program behaved correctly in a
given test and accordingly results in a verdict of
“pass” or “fail”.” [SWEBOK v3]

“Program testing can be used to show the presence of bugs,
but never to show their absence” Dijkstra (1970)
Exhaustive testing is not possible in practice !!!
Trade-off between limited resources, schedules and unlimited test
requirements
Prioritization based on risk, usage, cost, …

“I have a lot of test cases, thus my program is bug free”
WRONG !!!
How to select test cases ?
• Selection criteria
• I/O coverage (black box testing)
• No access to the source code
• Code coverage (white box testing)
• Access to source code
• Test engineer expertise
How to assess
the quality of
your test cases?

Source of test generation
• Black box testing
• Based on specification documents
• No access to the source code
• White box testing
• Access to the source code
• Model-based testing
Characteristic being tested
• Functional
• is the output correct for a given input ?
• Non-functional
• performance, robustness, security, usability, etc.

Purpose of the tests
• Testing a single element
• Unit testing (e.g., a single class)
• Component testing
• Integration of multiple elements
• Integration testing
• System testing
• Test with users
• Acceptance/Beta testing
• After an evolution of the system
• Regression testing

Requirements
speciﬁcation
Design
Coding and
unit testing
Integration/
System
testing
Acceptance
testing
Release
Integration/
System
test plan
Acceptance
test plan
Maintenance and
regression testing

Testing in Agile process
Testing from the beginning
• No commit without test!
• Have a clear testing policy
• Are your tests good enough?
• Use dedicated tools
Continuous Integration Dashboard
Cobertura

Specify requirements as tests
• State what system
• should do
• and should not do

Testers and developers are allies
• Testers are developers
• developers are testers

Testers and developers are allies
Test often and in small chunks
• Unit and integration tests are important
• Code design matters!!!

Testing Driven Development (TDD)
Red
• Write automated tests (examples) from user stories
• Tests fail
Green
• Developers design and write code to make the tests pass
• Tests are run frequently
Refactor
• Refactor code
• Without changing interfaces or behaviour
• Tests must pass

The Telecommunication Software & Systems Group
(TSSG) case study

Agile process implementation
The Telecommunication Software & Systems Group (TSSG)
case study
From Dowling, P., & McGrath, K. (2015). Using free and open source tools to manage
software quality. Communications of the ACM, 58(7), 51–55. doi:10.1145/2755503
• Open Source tools
• Support team’s activities management
• Language: Java

Agile process implementation
The Telecommunication Software & Systems Group (TSSG)
case study
Project Management
Requirements Management
Issues Tracking
Development environment
Building
Unit testing
Code
coverage
Cobertura
Continuous Integration
Code
Repository
Test environment
Functional
Testing
Test Case
Management
Load Testing
Security
Testing

public class Message { 
private long id = -1;
private String author;
private String message; 
public Message(String author, String message) {
this.author = author;
this.message = message;
}
// Gets and Sets ...
/**
   * Appends the given text at the end of this' message.
   * A space is added before the given text.
   *
   * @param text The test to append.
   */
public void compose(String text) {
this.message = message + " " + text;
}
}

• Collec'on of classes to perform unit tes'ng
• Uses annota'ons (e.g., @Test)

public class MessageTest {
@BeforeClass
public static void setUpClass() {
// Executed once before all tests
}
@AfterClass
public static void tearDownClass() {
// Executed once after all tests
} 
@Before
public void setUp() {
// Executed before each test
} 
@After
public void tearDown() {
// Executed after each test
} 
@Test
public void testCompose() {
// Write test here
}
}

@Test
public void testComposeString() {
Message msg = new Message("Me", "despicable");
msg.compose("me");
}
Is the compose method covered by the test case?
Is it enough?
Yes
No
There are no assertions on the result

@Test
msg.compose("me");
assertThat(msg.getMessage(), equalTo("despicable me"));
}
JUnit assertions Hamcrest matchers
asserTrue(c);
assertFalse(c);
assertEquals(expected, actual);
assertNull(o);
assertNotNull(o);
fail();
assertThat(T actual, matcher);
assertThat(T actual, equalTo(T operand));
assertThat(T actual, not(T operand));
assertThat(Iterable<? super T> c, 
hasItem(T item));
assertThat(T actual, nullValue());
assertThat(T actual, notNullValue());

Core
• anything - always matches, useful if you don't care what the object under test is
Logical
• allOf - matches if all matchers match, short circuits (like Java &&)
• anyOf - matches if any matchers match, short circuits (like Java ||)
• not - matches if the wrapped matcher doesn't match and vice versa
Object
• equalTo - test object equality using Object.equals
• hasToString - test Object.toString
• instanceOf, isCompatibleType - test type
• notNullValue, nullValue - test for null
• sameInstance - test object identity
Collections
• hasEntry, hasKey, hasValue - test a map contains an entry, key or value
• hasItem, hasItems - test a collection contains elements
• hasItemInArray - test an array contains an element
Number
• closeTo - test floating point values are close to a given value
• greaterThan, greaterThanOrEqualTo, lessThan, lessThanOrEqualTo - test ordering
Text
• equalToIgnoringCase - test string equality ignoring case
• equalToIgnoringWhiteSpace - test string equality ignoring differences in runs of whitespace
• containsString, endsWith, startsWith - test string matching

Class 
Under 
Test
DAO
Class
Database

Fake
Test implementation ≠ production implementation
public class FakeMessageDAO extends MessageDAO{

private static long id = 0;
private final Map<Long, Message> messages;
public FakeMessageDAO(DAOFactory factory) {
messages = new HashMap<>();
} 
@Override
public Message create(Message value) throws DAOException {
value.setId(id); messages.put(id, value); id++;
return value;
}
@Override
public void delete(Message value) throws DAOException {…} 
@Override
public Message find(long id) throws DAOException {…}
…
}

Fake
Test database instance
Design your code to
• Externalize DB access information .properties
• Manage DB connections
BasicDataSource source = new BasicDataSource();
source.setUrl("jdbc:derby:memory:sample;create=true");
source.setDriverClassName("org.apache.derby.jdbc.EmbeddedDriver");
...
source.getConnection();
Connection pooling
using commons-dbcp
Connection management
using commons-dbutils

Stub
Test implementation holds predefined data
public class StubMessageDAO extends MessageDAO { 
@Override
public Message create(Message value) throws DAOException {
value.setId(42l);
return value;
}
 
@Override
public Message find(long id) throws DAOException {
return new Message(id, "Me", "Despicable");
}
@Override
public List<Message> find(String author) throws DAOException {
List<Message> result = new ArrayList<>();
result.add(new Message(9, author, "One ring to rule them all"));
result.add(new Message(42, author, "Life, the Universe and Everything"));
return result;
}
…
}

Mock
Registers calls it receives
@Test
public void testMocking() throws Exception {
// Initialise mock object
MessageDAO dao = mock(MessageDAO.class);
when(dao.find(42)).thenReturn(new Message(42, "Me", "Despicable"));
// Check that method has been called
verify(dao).find(42);
}
// Call the method
Message msg = dao.find(42);
assertThat(msg.getMessage(), equalTo("Despicable"));
Should you always check method calls?

be.unamur:myfirstwebapp:0.0.1-SNAPSHOT
MessageTest.java
FakeMessageDAO.java
StubMessageDAO.java
MockMessageDAO.java
PrintMessagesServletTest.java
Simple example of JUnit test class
Fake MessageDAO object with a Map implementation
MessageDAO stub with predefined data
Example of mock usage applied to MessageDAO class
Complete example of a servlet unit test with mocking and
a fake DAOFactory using a in memory database

Requirements coverage
Req. 
1
Req.
1.1
Req.
1.2
Req. 
2
Req.
2.1
Test1 x x
Test2 x
Test3 x
Test4 x x
• Coverage matrix

Input/output domain coverage
public void compose(String text);
public void compose(String t1, String t2);
• Equivalence partitioning
Normal case: "Despicable you"
Limit cases: "" null "azrtyuiop^$ù m;21@98#3!"
• t1
• t2
☛ Combinatorial explosion with the number of parameters
(based on pre/post conditions)

Combinatorial Interaction Testing (CIT)
t1 t2 t3
"Despicable you" "Despicable you" "Despicable you"
"" "" ""
null null null
"azrtyuiop^$ù" "azrtyuiop^$ù" "azrtyuiop^$ù"
Hyp.: most faults are caused by interactions of at most two factors
Pairwise testing: cover all pairs of values
"Despicable you" "Despicable you" "Despicable you"
"Despicable you" "" ""
"Despicable you" null null
"Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù"
"" "Despicable you" ""
"" "" "Despicable you"
"" null "azrtyuiop^$ù"
"" "azrtyuiop^$ù" null
null "Despicable you" null
null "" "azrtyuiop^$ù"
null null "Despicable you"
null "azrtyuiop^$ù" ""
"azrtyuiop^$ù" "Despicable you" "azrtyuiop^$ù"
"azrtyuiop^$ù" "" null
"azrtyuiop^$ù" null ""
"azrtyuiop^$ù" "azrtyuiop^$ù" "Despicable you"
16 test cases
instead of 64

app.
http://cse.unl.edu/~citportal/
http://ctweb.abstracta.com.uy/combinatorial.jsp
Hyp.: most faults are caused by interactions of at most t factors
T-wise testing: cover all t-uples of values
Integration and System testing
Which
configuration
should I use
for my tests?

type
Android
version
Screen Processor
tablet 4.0.4 XLarge ARM Cortex
smartphone 4.1 Large Intel x86
phablet 4.1.1 Normal
4.1.2 Small
… or using a feature model
http://familiar-project.github.io
https://featureide.github.io
http://www.skalup.com/
http://research.henard.net/SPL/PLEDGE/
http://martinfjohansen.com/models2011/spltool/
http://cse.unl.edu/~citportal/

Code coverage
Line coverage: percentage of lines covered by tests
https://github.com/cobertura/cobertura
Achieving line coverage  
=  
executing each line at least once

Code coverage
Branch coverage: percentage of lines covered by tests
https://github.com/cobertura/cobertura
Achieving branch
coverage  
=  
executing each branch
at least once
Is it enough?
No

@Test
msg.compose("me");
}
There are no assertions on the result
Cobertura
mvn cobertura:cobertura

SUT
System under testTest cases
executed
on
t1 t2
t3 t4
Mutation operators introduce small syntactic errors

SUT
M1
Mutation
operator
Mutants
t1 t2
t3 t4

SUT
M1
Mutation
operator M2
Mutants
t1 t2
t3 t4

SUT
M1
Mutation
operator M2
M3
Mutants
t1 t2
t3 t4

Test cases
M1
M2
M3
executed
on
Mutants
t1 t2
t3 t4

M1
M2
M3
Mutation score = 2/3
MutantsTest cases
t1 t2
t3 t4

Hypothesis
Competent programmer hypothesis
Programmers write programs that are almost perfect
• Faults are syntactically small
• Test catches artificial mutants test catches real mutants
(faults)
Coupling effect
Big effects arising from bugs in the software are closely
coupled to small and simple bugs.
• Small mutations emulates real bugs in the software
http://www0.cs.ucl.ac.uk/staff/M.Harman/exe10.html
Javalanche µJava Major

CONDITIONALS_BOUNDARY
Operators
NEGATE_CONDITIONALS
MATH
INCREMENTS
< → <=
if(…) → if(true)
REMOVE_CONDITIONALS == → !=
+ → -
i++ → i--
INVERT_NEGS -i → i
INLINE_CONSTS true → false
RETURN_VALS return o; → return null;
...

mvn org.pitest:pitest-maven:mutationCoverage
mvn clean test
Results in target/pit-reports/

Program execution using symbolic values
{x : int}
{x : int | x < 0}
{x : int | x < 0}
int foo(int x){
if(x < 0){
return -1/x;
}
return 1/x;
}

int foo(int x){
if(x < 0){
return -1/x;
}
return 1/x;
}
Program execution using symbolic values
{x : int}
{x : int | x > 0} v {x : int | x = 0}
test case 1 test case 2

Test case generation 
= 
a search optimization
problem
To find a (near) optimal
solution
Search guided  
using a meta-heuristic
0
20
40
60
80
t1 t2 t3 t4 t5 t6 t7
Potential solutions
Usefulness

Test case generation 
= 
a search optimization
problem
Want to minimize/maximize objective function
Classic techniques:
• linear programming  
(e.g., simplex)
• branch and bound
• etc.
Meta-heuristics:
• genetic algorithms
• hillclimbing
• etc.

Search guided  
using a meta-heuristic
• Problem-independent
• Approximates solution
• Non deterministic
• Requires to:
• represent solutions
• encoding/decoding
• evaluate solutions
• using a fitness function

• Pick a random candidate in the search space
• Find the neighbours
• Measure the fitness of the candidate and its neighbours
• Either
• Pick the best neighbour (if any), and iterate.
• Or
• If there is no better neighbour, (and the fitness is not
ideal yet), re-start
Hillclimbing

Hillclimbing
Global maximum
f(x)
x
Global minimum

Global maximum
f(x)
x
Local maximum
Hillclimbing

Evaluate Fitness
Selection Crossover
Initialize Population
F = 0
|| T
Mutation
Create the Next Generation
Reinsertion
Genetic algorithms

Random initial
test suites
Test suites
evolution
Minimize test
suite with
maximized
coverage

• mvn compile evosuite:generate
• mvn evosuite:export -DtargetFolder=target/
generated-test-sources/evosuite
• mvn test
Generates tests in .evosuite/ folder

Test environment
Functional
Testing
Test Case
Management
Load Testing
Security
Testing
Performance 
Testing
Functional
testing
Security
Testing NMap

Selenium UI testing
• open: opens a page using a URL
• click/clickAndWait: performs a click operation (and waits for a page to load)
• verifyTitle/assertTitle: verifies an expected page title
• verifyTextPresent: verifies expected text is somewhere on the page
• verifyElementPresent: verifies an expected UI element, as defined by its
HTML tag, is present on the page
• verifyText: verifies expected text and its corresponding HTML tag are present
on the page
• verifyTable: verifies a table’s expected contents
• waitForPageToLoad: pauses execution until an expected new page loads.
Called automatically when clickAndWait is used
• waitForElementPresent: pauses execution until an expected UI element, as
defined by its HTML tag, is present on the page

Test with/by the user
• Functionality
• Performance
• Security
• …
Acceptance test plan
• Which features?
• Which data?
• Which KPI?
• …

White box Black box Model-based
Functional Non-functional
• performance
• robustness
• security
• usability
Unit Integration System Acceptance Regression

Testing is important
• User satisfaction
• Money
• Life
Software is developed for users
Use proper tools
… including QA tools

There is always one more bug…

Software testing: an introduction - 2017

More Related Content

What's hot

Similar to Software testing: an introduction - 2017

More from XavierDevroey

Recently uploaded

Software testing: an introduction - 2017