Software testing: an introduction - 2016

www.unamur.be
Software Testing: an Introduction
Xavier Devroey <xavier.devroey@unamur.be>
INFO M110 Ingénierie du logiciel
Namur, Belgique

Plan
• Introduction
• Software testing typology
– Source of test generation
– Characteristic being tested
– Lifecycle phase
• Agile process implementation: the TSSG case study
• Unit testing using Maven and JUnit
• Code review
• Test-case assessment
21/10/2016 xavier.devroey@unamur.be 2

Error, fault, failure, and incident
Error
Dev.

Error
Fault Exec.
Failure (bug)
Dev.

Error
Fault Exec.
Failure (bug)
Reports
Incident
Bug Tracker
Dev.
User

Software testing: definition [SWEBOK v3]
“Software testing consists of the dynamic verification
that a program provides expected behaviours
on a finite set of test cases, suitably selected from the
usually infinite execution domain.”

Testing process
Req.
Test engineer
Test
case
defines

Testing process
Req.
Test engineer
Test
case
defines
Pgm.
executed on

Dynamic verification
• Testing implies executing the program under test
• Contrary to static analysis
– Formal, semiformal, or informal
– Examples:
• compiler warnings (code analysis),
• javadoc warnings (documentation analysis),
• design inspection (model analysis),
• theorem proving (e.g., Event-B),
• code analysis
– Code review
– IDE warnings
– FindBugs http://findbugs.sourceforge.net

Expected behaviours
• It must be possible to decide whether the observed
outcomes for a given input is acceptable or not
• The oracle problem
– “An oracle is any human or mechanical agent that decides
whether a program behaved correctly in a given test and
accordingly results in a verdict of “pass” or “fail”.” [SWEBOK v3]

Finite set of test cases
• Exhaustive testing is not possible in practice
– Infinite number of test cases
– Select a subset of all possible tests
– Trade-off between limited resources, schedules and unlimited
test requirements
• Prioritization based on risk, usage, cost, …
“Program testing can be used to show the presence of bugs,
but never to show their absence” Dijkstra (1970)

Suitably selected
• How to select test-cases ?
• Selection criteria
– Input/output domains coverage (black box testing)
• Domain partitioning techniques
• Based on specification documents
• No access to the source code
– Code coverage: statement coverage, decision coverage, path
coverage, etc. (white box testing)
• Access to source code
– Test engineer expertise

Types of testing
• Source of test generation
– Black box testing
• Based on specification documents
• No access to the source code
– White box testing
• Access to the source code
– Model-based testing
• Characteristic being tested
– Functional testing: is the output correct for a given input ?
– Non-functional testing: Performance, robustness, security,
usability, etc.

Types of testing
• Lifecycle phase
– Coding
• Unit testing
• Component testing
– Integration
• Integration testing
– System integration
• System testing
– Release
• Acceptance/Beta testing
– Maintenance
• Regression testing

Testing in the waterfall model
Requirements
speciﬁcation
Design
Coding and
unit testing
Integration testing
System testing
Acceptance
testing
Release
Maintenance and
regression testing

Testing in the V-model
Requirements
speciﬁcation
Design
Coding and
unit testing
Integration/
System
testing
Acceptance
testing
Release
Integration/
System
test plan
Acceptance
test plan
Maintenance and
regression testing

Testing in an Agile process
1. Include testing activities from the beginning
2. Specify requirements in terms of tests
3. Testers and developers are not enemies
4. Test often and in small chunks

Test Driven Development (TDD)
• Red
– Write automated tests
(examples) from user stories
– Tests fail (as the functionality
is not implemented yet)
• Green
– Developers design and write
code to make the tests pass
– Tests are run frequently
• Refactor
– Refactor code without
changing interfaces or
behaviour (tests must pass)
Red
GreenRefactor

AGILE PROCESS IMPLEMENTATION
The Telecommunication Software & Systems Group
(TSSG) case study

Agile process implementation: TSSG
• From Dowling, P., & McGrath, K. (2015). Using free and
open source tools to manage software quality.
Communications of the ACM, 58(7), 51–55.
doi:10.1145/2755503
• Open Source tools
• Support team’s activities management
• Language: Java

Project Management
Requirements Management
Issues Tracking

Project Management
Issues Tracking
Development environment
Building
Unit testing
Code
coverage
Cobertura
Continuous Integration
Code
Repository

Project Management
Issues Tracking
Building
Unit testing
Code
coverage
Cobertura
Code
Repository
Test environment
Functional
Testing
Test Case
Management
Load Testing
Security
Testing

Workflow
• Developer forks the code
base
• Implements changes
• Integrates possible
changes from code repo.
• Runs a complete build
– Including Unit tests
• Submits changes to code
repo.
• CI server performs
integration tests
Do NOT break the build!

Project Management
Issues Tracking
Building
Unit testing
Code
coverage
Cobertura
Code
Repository
Test environment
Functional
Testing
Test Case
Management
Load Testing
Security
Testing

Maven 3: Introduction
• Project Lifecycle management tool
– Project description in Project Object Model (pom.xml)
– Build automation (including unit testing phase)
– Project modularization
– Dependency management
– Deployment automation
– Reporting (code quality check, documentation generation,
website generation, test coverage, etc.)
• Convention over configuration (… like Java EE)
“Systems, libraries, and frameworks should assume reasonable
defaults. Without requiring unnecessary configuration, systems
should “just work”.” [Maven by example]

Installation
• See https://maven.apache.org/install.html
• Use command mvn –v to check installation
• Local Maven files are in ~/.m2
• ~/.m2/settings.xml contains access configurations
• ~/.m2/repository contains the local Maven
repository
$ mvn -version
Apache Maven 3.1.1
Maven home: /usr/local/apache-maven-3.1.1
Java version: 1.8.0_31, vendor: Oracle Corporation
Java home: /Library/Java/jdk1.8.0_31.jdk/Contents/Home/jre
Default locale: fr_FR, platform encoding: UTF-8
OS name: "mac os x”

First Maven project (using NetBeans)

Maven project structure
• myfirstmavenproject/
– pom.xml
– src/
• main/
– java/
» Here goes your Java source code
– resources/
» Here goes other resources (files, etc.) included in the .jar/.war
• test/
– java/
» Here goes the JUnit tests
– resources/
» Here goes other resources used by the tests
– sub-module-1/
• pom.xml
• src/
– sub-module-2/
• …

Pom.xml

Pom.xml
Identifier
• Maven identifiers have to be unique
• Format: groupId:artifactId:version
• GroupId should identify the organisation using Java package naming
convention
• ArtifactId should identify the project
• Version should be encoded on 3 number XX.XX.XX
• “-SNAPSHOT” is used to indicate the currently under
development version

Pom.xml
Type of the artefact
• Packaging indicates to Maven the type of artifact produced during
build
• “jar” indicates a .jar file (executable or not)
• “pom” indicates a maven project without compiled code (used by a
parent project)
• “war” indicates a .war file that will be deployed on a web server

Pom.xml
Properties used by project and sub-projects
• Properties are used as global constants
• Implicit properties
• ${basedir} current project root directory
• ${project.version} current project version
• ${project.groupId} current project groupId
• Declared properties
• Should contain dependencies versions

Sub-modules
• Parent project (should have pom packaging)
• Sub-module has to declare parent project
• In this case: sub-module-1 version is inherited
<project …>
…
<packaging>pom</packaging>
<modules>
<module>sub-module-1</module>
</modules>
</project>
<project …>
<parent>
<groupId>be.unamur.info</groupId>
<artifactId>myfirstmavenproject</artifactId>
<version>0.0.1-SNAPSHOT</version>
</parent>
<artifactId>sub-module-1</artifactId>
…
</project>

Dependencies
• Maven central repository
– http://repo.maven.apache.org/maven2
– http://search.maven.org (to search an artifact)
• Declare dependency in a Maven project
<project …>
…
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>

mvn <phase>
• compile
– Compile source files
• test
– Launch JUnit tests in
src/test/java/
• package
– Create .jar or .war or … file in
target/
• install
– Install artefact in local Maven
repository
• deploy
– Deploy artefact using
deployment configuration

JUnit: introduction
• Collection of classes to perform unit testing
• Uses annotations (e.g., @Test)
import static org.junit.Assert.assertEquals;
import org.junit.Test;
public class CalculatorTest {
@Test
public void evaluatesExpression() {
Calculator calculator = new Calculator();
int sum = calculator.evaluate("1+2+3");
assertEquals(6, sum);
}
}
java -cp .;junit-4.12.jar;hamcrest-core-1.3.jar org.junit.runner.JUnitCore CalculatorTest

Anatomy of a Junit test class
import …
public class MyClassTest{
private static final Logger logger = LoggerFactory.getLogger(JHConwayLifeCellTest.class);
@Rule
public TestRule watcher = new TestWatcher() {
@Override protected void starting(Description description) {
logger.info(String.format("Starting test: %s()...”, description.getMethodName()));};};
@BeforeClass
public static void setUpClass() {logger.info("Setting up before class");}
@AfterClass
public static void tearDownClass() {logger.info("Tearing down after class");}
@Before
public void setUp() {logger.info("Setting up before test");}
@After
public void tearDown() {logger.info("Tearing down after test");}
@Test
public void testMyMethod() {assertTrue(true);}
}

Assertions
• JUnit
– asserTrue(c); assertFalse(c); assertEquals(expected, actual);
assertNull(o); assertNotNull(o); fail()
– assertThat(T actual, org.hamcrest.Matcher<T> matcher)
• Hamcrest matchers (org.hamcrest.Matchers)
– assertThat(T actual, equalTo(T operand))
– assertThat(T actual, not(T operand))
– assertThat(Iterable<? super T> c, hasItem(T item))
– assertThat(T actual, equalTo(T operand))
– assertThat(T actual, nullValue())
– assertThat(T actual, notNullValue())

Hamcrest common matchers
• Core
– anything - always matches, useful if you don't care what the object under test is
• Logical
– allOf - matches if all matchers match, short circuits (like Java &&)
– anyOf - matches if any matchers match, short circuits (like Java ||)
– not - matches if the wrapped matcher doesn't match and vice versa
• Object
– equalTo - test object equality using Object.equals
– hasToString - test Object.toString
– instanceOf, isCompatibleType - test type
– notNullValue, nullValue - test for null
– sameInstance - test object identity
• Collections
– hasEntry, hasKey, hasValue - test a map contains an entry, key or value
– hasItem, hasItems - test a collection contains elements
– hasItemInArray - test an array contains an element
• Number
– closeTo - test floating point values are close to a given value
– greaterThan, greaterThanOrEqualTo, lessThan, lessThanOrEqualTo - test ordering
• Text
– equalToIgnoringCase - test string equality ignoring case
– equalToIgnoringWhiteSpace - test string equality ignoring differences in runs of whitespace
– containsString, endsWith, startsWith - test string matching

Hands-on: Maven, JUnit and Hamcrest
• be.unamur:myfirstwebapp
• Class to test: be.unamur.myfirstwebapp.Message
• Test coverage using Cobertura
– mvn cobertura:cobertura
– Report in target/site/cobertura/index.html

Hands-on: test a data access object
• Using a test database
– Derby (https://db.apache.org/derby/)
– H2 (http://www.h2database.com/)
– DB server test instance (MySQL, Oracle, etc.)
• Code designed to
– Externalize DB access information (e.g., using a property file)
– Manage DB connections (using a DataSource object)
• Connection pooling using commonds-dbcp
(https://commons.apache.org/proper/commons-dbcp/)
• In this case SQL execution using commonds-dbutils
(https://commons.apache.org/proper/commons-dbutils/)
• Exercice: Add test to cover “if” branch in
MessageDAO.update

Hands-on: test a servlet
• Using mock objects to emulate servlet environment
• Code designed to
– Strictly separate model (Beans + DAOs), view (JSPs), and controller
(Servlets)
• Mockito: tasty mocking framework for unit tests in Java
– http://site.mockito.org
• Create the mock
– HttpServletRequest request = mock(HttpServletRequest.class);
• Stub method calls
– when(request.getParameter("author")).thenReturn("Albert");
• Check methods were called with given arguments
– verify(context).getRequestDispatcher("/WEB-INF/messages.jsp");

CODE REVIEW
A static analysis detour

Code review
• Over-the-shoulder reviewing
– sometimes called stepmother reviewing
• Pair programming

Code review
• Pull request
Dev.
Team
Project
master
Creates the
feature in a
dedicated branch
in local repository
Pushes the
branch to central
rep. and ﬁlls a
pull request
Reviews the
code, discusses it,
and alters it
Merges the
feature into the
oﬃcial rep. and
closes the pull
request

Code review
• Pull request
• Rubber duck debugging

Code analysis: FindBugs
• http://findbugs.sourceforge.net
• Called after Maven test phase
– mvn test findbugs:findbugs findbugs:gui

Selenium
• User Interface (UI) testing framework for web application
– Integration and acceptance
– black-box
– (mainly used for) regression testing
Testing Environment
Selenium WebDriver
Selenium IDE
1. Design and/or
execute test-cases
2. Export

Installation
• See http://www.seleniumhq.org/docs/02_selenium_ide.jsp

Selenium Commands
• open
– opens a page using a URL
• click/clickAndWait
– performs a click operation, and optionally waits for a new page to load
• verifyTitle/assertTitle
– verifies an expected page title
• verifyTextPresent
– verifies expected text is somewhere on the page
• verifyElementPresent
– verifies an expected UI element, as defined by its HTML tag, is present on the page
• verifyText
– verifies expected text and its corresponding HTML tag are present on the page
• verifyTable
– verifies a table’s expected contents
• waitForPageToLoad
– pauses execution until an expected new page loads. Called automatically when clickAndWait is used
• waitForElementPresent
– pauses execution until an expected UI element, as defined by its HTML tag, is present on the page

EVALUATION OF THE TESTS
PERFORMED
Are my test cases suitably selected?

Testing process
Req.
Test engineer
Test
case
defines
Pgm.
executed on
Good enough?

Coverage
• Requirements coverage
• Input domain coverage
– Equivalence partitioning
– Combinatorial testing (e.g., pairwise testing)
• Code coverage
– Statement/block coverage
– Decision/branch coverage
– condition coverage
– modified condition/decision coverage (MC/DC)
– Path coverage
– E.g., Cobertura, JaCoCo, Clover
• mvn clean clover:setup test clover:aggregate clover:clover

Mutation testing
• Mimic typical errors by introducing small modifications in
the program
• Based on 2 hypothesis:
– Competent programmer hypothesis
– Coupling effect
• Mutation operators introduce small syntactic errors
Original program:
if (a && x < 5) {
foo();
} else {
fee();
}
Mutant 1:
if (a || x < 5) {
foo();
} else {
fee();
}
Mutant 2:
if (a && x < 5) {
fee();
} else {
fee();
}

Mutation Testing
SUT
Mut. Op.
Mut. Op.

Mutation Testing
M1 M2
M3 M4

Mutation Testing
M1 M2
M3 M4
Test
case

Mutation Testing
M1 M2
M3 M4
Test
case
Count( )
Mutation score =
Total Nb. Mut.

Hands-on: test cases assessment
• http://pitest.org
• Called after Maven test phase
– mvn org.pitest:pitest-maven:mutationCoverage

FROM TEST SUITE EVALUATION
… to test suite augmentation

Symbolic execution
• Execute the program with symbolic values
– Constraints on the variables to follow defined path
• http://klee.github.io (C)
• Pex (C#); CodeHunt (C# and Java)
int foo(int x){
if(x < 0){
return -1/x;
}
return 1/x;
}

Symbolic execution
{x : int}
{x : int | x < 0}
{x : int | x < 0}
int foo(int x){
if(x < 0){
return -1/x;
}
return 1/x;
}

Symbolic execution
{x : int}
{x : int | x > 0} v {x : int | x = 0}
int foo(int x){
if(x < 0){
return -1/x;
}
return 1/x;
}

Search-based software testing
• Use metaheuristics (e.g., evolutionary algorithms) to
– generate test cases (or test data);
– prioritize test cases;
– minimize test suites; etc.
• For instance, we want to generate input data for a given
function, we need:
– A search algorithm: hill climbing or genetic algorithm
– A representation of the program and input values
– Search operators: find the neighbours of the input values
– A fitness function: decision coverage
• This requires instrumentation of the program (to measure coverage
when executing tests)

Hands-on: test cases generation
• http://www.evosuite.org/
• Generates tests in .evosuite/ folder
• mvn compile evosuite:generate
• mvn evosuite:export -DtargetFolder=target/generated-
test-sources/evosuite
• mvn test

• Mathur, A. P. (2008). Foundations of software testing. Pearson Education.
• Utting, M., & Legeard, B. (2007). Practical Model-Based Testing: A Tools Approach.
Morgan Kaufmann.
• Bourque, P., & Fairley, R. E. (2014). Guide to the Software Engineering Body of
Knowledge (SWEBOK(R)): Version 3.0. IEEE Computer Society.
• Dowling, P., & McGrath, K. (2015). Using free and open source tools to manage
software quality. Communications of the ACM, 58(7), 51–55. doi:10.1145/2755503
• Tim O'Brien et al. Maven by Example. Sonatype,
http://www.sonatype.com/books/mvnex-book/reference/public-book.html
• http://junit.org
• http://hamcrest.org
• http://site.mockito.org
• http://www.seleniumhq.org
• http://pitest.org
• http://klee.github.io
• https://www.microsoft.com/en-us/research/project/pex-and-moles-isolation-and-
white-box-unit-testing-for-net/
• http://www.evosuite.org/
• https://openclassrooms.com/courses/creez-votre-application-web-avec-java-ee

… last tricks
• Google guava
– https://github.com/google/guava/
• Apache commons
– https://commons.apache.org
• Simple Logging Facade for Java (SLF4J)
– http://www.slf4j.org
• Jetty
– http://www.eclipse.org/jetty/
– mvn jetty:run

Software testing: an introduction - 2016

Recommended

Recommended

More Related Content

Similar to Software testing: an introduction - 2016

Similar to Software testing: an introduction - 2016 (20)

More from XavierDevroey

More from XavierDevroey (10)

Recently uploaded

Recently uploaded (20)

Software testing: an introduction - 2016