Design For Testability

Giovanni Asproni
email: gasproni@asprotunity.com
twitter: @gasproni
linkedin: http://www.linkedin.com/in/gasproni
Design For Testability
What, Why, How
1
/dev/winter 2016 Cambridge

“...is the degree to which a software artifact (i.e. a
software system, software module, requirements- or
design document) supports testing in a given test
context. If the testability of the software artifact is
high, then finding faults in the system (if it has any) by
means of testing is easier.”
Software Testability
2
http://en.wikipedia.org/wiki/Software_testability

Why?
• Testability improves other qualities of the system, as
well as delivery time and cost
• Testability and system design depend on each other
3

Qualities
• Maintainability
• Evolvability
• Use-it-again-ility
• I refuse to call it“reusability”
• Supportability
• Diagnosability
• Delivery time
• Cost
4

Testability Depends On Design (And Vice Versa)
5
“To be tested a system has
to be designed to be
tested”

Design for Test (aka "Design for Testability" or "DFT") is a
name for design techniques that add certain testability
features to a microelectronic hardware product design. The
premise of the added features is that they make it easier to
develop and apply manufacturing tests for the designed
hardware. The purpose of manufacturing tests is to
validate that the product hardware contains no defects
that could, otherwise, adversely aﬀect the product’s
correct functioning.
6
Design For Testability
http://en.wikipedia.org/wiki/Design_for_testing

Examples of Software Testability Features
• Dependency injection (technique not frameworks!)
• Configuration parameters
• There are more…
7

“All architecture is design but
not all design is architecture”
Grady Booch
Design And Architecture
8
http://handbookofsoftwarearchitecture.com/?p=63
By Grady_Booch,_CHM_2011_2.jpg: vonguard
from Oakland, Nmibia derivative work: YMS
[CC BY-SA 2.0 (http://creativecommons.org/
licenses/by-sa/2.0)], via Wikimedia Commons

Every software-intensive system has an architecture. In
some cases that architecture is intentional, while in
others it is accidental. Most of the time it is both, born of
the consequences of a myriad of design decisions made
by its architects and its developers over the lifetime of a
system, from its inception through its evolution. In that
sense, the architecture of a system is the naming of the
most significant design decisions that shape a system,
where we measure significant by cost of change and by
impact upon use.
Design And Architecture
9
http://handbookofsoftwarearchitecture.com/?p=63

Architecture Is For (1/2)
• Modelling, and reasoning about, important
aspects of the system
• Quality attributes: scalability, performance,
latency, testability, etc., often, cannot (easily) be
retrofitted
10

Architecture Is For (2/2)
• Recording and communicating decisions
• Guiding lower level design and implementation
decisions
• E.g., avoid“TDDing”the system in the wrong
direction
• Splitting work among teams (Conway’s Law)
11

Organizations which design
systems are constrained to
produce designs which are
copies of the communication
structures of these organizations.
Conway’s Law
12
http://www.melconway.com/Home/Committees_Paper.html

14
Testing shows the presence,
not the absence of bugs.
Edsger Dijkstra
"Edsger Wybe Dijkstra" by Hamilton Richards - manuscripts of Edsger W. Dijkstra,
University Texas at Austin. Licensed under CC BY-SA 3.0 via Wikimedia Commons -
http://commons.wikimedia.org/wiki/File:Edsger_Wybe_Dijkstra.jpg#/media/
File:Edsger_Wybe_Dijkstra.jpg

15
In theory there is no diﬀerence between
theory and practice. But, in practice, there
is.

16
“More than the act of testing, the act of
designing tests is one of the best bug
preventers known. The thinking that
must be done to create a useful test can
discover and eliminate bugs before they
are coded - indeed, test-design thinking
can discover and eliminate bugs at every
stage in the creation of software, from
conception to specification, to design,
coding and the rest.”
Boris Beizer

Well Chosen Tests
• Give us confidence changes don’t break existing
functionality
• Scale much better than formal methods and
formal inspections
• And support refactoring activities more
eﬃciently
• Most tests can be automated
17

Diagnosability
18
Diagnose: identify the nature of (an illness
or other problem) by examination of the
symptoms
Testable software is more diagnosable

Cost Of Software
• costtotal = costdevelop + costmaintain
• costmaintain = costunderstand + costchange + costtest + costdeploy
• The above costs are related with each other
19

Cost Of Maintenance
• Usually costmaintain/costtotal ~ 60%
• It is important to write maintainable software
20
“Frequently Forgotten Fundamental Facts about Software Engineering”Robert Glass, IEEE Software, May/June 2001,
http://www.eng.auburn.edu/~hendrix/comp6710/readings/Forgotten_Fundamentals_IEEE_Software_May_2001.pdf

As of 2011, the average cost per function point in the United States
is about $1,000 to build software applications and another $1,000
to maintain and support them for five years: $2,000 per function
point in total. For projects that use eﬀective combinations of
defect prevention and defect removal activities and achieve high
quality levels, average development costs are only about $700 per
function point and maintenance, and support costs drop to about
$500 per function point: $1,200 per function point in total.
21
How Quality Aﬀects Costs
From“The Economics of Software Quality”,
Capers Jones and Olivier Bonsignour

22
How Quality Aﬀects Delivery Time
% of defects removed
Developmenttime
Most organisations are
somewhere around this point
Fastest schedule
~95 100
Source Steve McConnell: http://www.stevemcconnell.com/articles/art04.htm

23
“Quality is an accelerator”
John Cliﬀord, Construx Software

Testability Measures
25
• Observability: how easy it is to infer internal states
from external outputs
• Controllability: how easy it is to set some internal
states of the system using external inputs

Seams
26
Seam: a seam is a place where you
can alter behaviour in your
program without editing it in that
place.
Enabling point: every seam has an
enabling point, a place where you
can make a decision to use one or
another.

Some Seam Types
27
• Object seams
• Link seams
• Preprocessing seams
• Test hooks

29
public class EventServer implements Runnable { 
 
...
 
private EventProducer eventProducer; 
public EventServer(EventProducer eventProducer) { 
this.eventProducer = eventProducer; 
... 
} 
 
@Override 
public void run() { 
... 
while (isRunning()) { 
try {
... 
Event event = eventProducer.next(); 
... 
} catch (Throwable e) { 
... 
} 
} 
} 
... 
}
Seam
Enabling point

31
public class EventServer implements Runnable { 
 
...
 
private EventProducer eventProducer; 
public EventServer() { 
this.eventProducer = new SpecialEventProducer(); 
... 
} 
 
@Override 
public void run() { 
... 
while (isRunning()) { 
try {
... 
Event event = eventProducer.next(); 
... 
} catch (Throwable e) { 
... 
} 
} 
} 
... 
}
Seam
Defined in a diﬀerent jar

A test hook is code used to configure the system to
behave diﬀerently during testing.
Test Hooks
33

public class AClass { 
... 
public void method() { 
...
// Initialise testing using
// some external property
boolean testing = ... 
if (testing) {
// do something
}
else {
// do something else
}
} 
... 
}
34
Hook

36
@Test 
public void generatesProxyCorrectly() { 
 
final int value = 10; 
final String expectedMethodName =
ServiceInterface.class.getMethods()[0].getName(); 
final Object[] args = {value}; 
 
serviceCaller = context.mock(ServiceCaller.class); 
proxyMaker = new ServiceProxyMaker(serviceCaller); 
 
context.checking(new Expectations() {{ 
oneOf(serviceCaller).call(args, expectedMethodName,
serviceAddress, Void.TYPE); 
will(returnValue(null)); 
}}); 
 
ServiceInterface generatedProxy =
proxyMaker.make(serviceAddress,
ServiceInterface.class).service(); 
generatedProxy.call(value); 
context.assertIsSatisfied(); 
}

public class AClass { 
... 
public void method() { 
...
// Initialise testing using
// some external property
boolean testing = ... 
if (testing) {
// do something
}
else {
// do something else
}
} 
... 
}
37
Hook

38
public class AClassTest { 
…
@Test 
public void methodDoesSomethin() {
// Set testing mode to true
…
// Set some other test context
…
AClass subject = new ACLass(…) 
subject.method()
// Assert something
…
} 
... 
}

Principle: Keep Test Logic Out of Production Code
39
“Testing is about verifying the behavior of a system. If
the system behaves diﬀerently when under test, then
how can we be certain that the production code
actually works? Even worse, the test hooks could
cause the software to fail in production!”
Test hooks violate this principle.
They must be used only in
special situations, e.g., a
stepping stone in making
legacy code testable.

Test Doubles
• Sometimes we need to test classes that interact with other objects which are diﬃcult to control
• The real object has nondeterministic behaviour (e.g., random number generator)
• The real object is diﬃcult to set up (e.g., requiring a certain file system, database, or network
environment)
• The real object has behaviour that is hard to trigger (e.g., a network error)
• The real object is slow
• The real object has (or is) a user interface
• The test needs to ask the real object about how it was used (e.g., confirm that a callback
function was actually called)
• The real object does not yet exist (e.g., interfacing with other teams or new hardware
systems)
40

Test Doubles
• Dummies
• Stubs
• Spies
• Fakes
• Mocks
41

Dummies
• Dummy objects are passed around but never
actually used
• E.g., a mandatory argument in a constructor never
used during a specific test
42

Stubs
• Stubs provide canned answers to calls made during
the test
43

Spies
• Spies are stubs that also record some information
based on how they were called. (e.g., the number of
times a method has been called)
44

45
public class OrderStateTest {
private static String TALISKER = "Talisker";
private WarehouseStub warehouse = new WarehouseStub();
@Test
public void orderIsFilledIfEnoughInWarehouse() {
Order order = new Order(TALISKER, 50);
order.ﬁll(warehouse);
assertTrue(order.isFilled())
assertTrue(warehouse.removeCalled);
}
@Test
public void orderDoesNotRemoveIfNotEnough() {
assertFalse(order.isFilled());
assertFalse(warehouse.removeCalled);
}
}
Adapted from: http://martinfowler.com/articles/mocksArentStubs.html
public class WarehouseStub implements Warehouse {
public boolean removeCalled = false;
public void hasInventory(String brand, int amount) {
return “Talisker”.equals(brand) && amount <= 50
}
public void remove(String brand, int amount) {
removeCalled = true;
}
…..
}

Fakes
• Fake objects actually have working
implementations, but take some shortcuts which
make them not suitable for production (e.g., an in
memory database)
46

47
public class OrderStateTest {
private static String TALISKER = "Talisker";
private Warehouse warehouse = new WarehouseFake();
@Test
public void orderIsFilledIfEnoughInWarehouse() {
assertTrue(order.isFilled())
}
@Test
public void orderDoesNotRemoveIfNotEnough() {
assertFalse(order.isFilled());
}
}
Adapted from: http://martinfowler.com/articles/mocksArentStubs.html
public class WarehouseFake implements Warehouse {
private HashMap<String, Integer> inventoryByBrand;
public WarehouseFake() {
inventoryByBrand =
new HashMap<>(){{put(“Talisker”, 50);}}
}
public void hasInventory(String brand, int required) {
available = inventoryByBrand.get(brand)
return available != null && required <= available;
}
public void remove(String brand, int amount) {
available = inventoryByBrand.get(brand)
if (available == null || amount > available) {
// Manage the error…
}
else {
inventoryByBrand.put(brand, available - amount);
}
}
…..
}

Mocks
• Mocks are objects pre-programmed with
expectations which form a specification of the calls
they are expected to receive
48

49
@Test 
 
 
 
}}); 
 
}

Mocks Aren’t Stubs
• Mocks are meant for testing behaviour
• Stubs and all other doubles are generally used for
testing state
• Classic TDD vs Mockist TDD (Classic school vs
London School of TDD)
50
http://martinfowler.com/articles/mocksArentStubs.html

APIs
• APIs for testing
• To query / change the state of the system
• To query / change the configuration of the system
• Notifications
• Logs
• They often become support APIs
51

Some Examples
• Time providers
• Notifications of asynchronous events
• Timers to check performance parameters
• Etc.
52

53
System
Test System
Entry points for standard users
Entry points for testing

A big part of designing a software system for testability is
about deciding where to put the seams and their
enabling points, and to create the necessary APIs for
testing
54

...But There Is More
• Ease of building the system
• Deployability
• Configurability
55

Deployability
• How reliably and easily can the system be deployed in
a particular environment?
• Does it require some special hardware configuration?
• Does it require some special OS configuration?
• Does it require a special directory structure?
• Etc.
56

Configurability
• How easily can paths, dbs, and other external resources be
configured? E.g.:
• Are external resources hardcoded?
• Are paths fixed?
• Can resources be configured at compile time or runtime?
• Too few or too many (or the wrong) options could be a
problem
57

Tests And Runtimes
• Tests and system (partially) in same runtime
• Allows the use of dependency injection at the
language level
• Tests and system in separate runtimes
• Dependencies are setup via configuration
• May rely more on special APIs for testing
58

59
From:“Growing Object Oriented Software Guided By Tests”

60
Same Runtime: Manage Global State Carefully
• Global application state should not spill over
• Frameworks must play nicely
• Avoid singletons and global variables

Same Or Separate Runtimes?
• Same runtimes
• More fine grained functional testing
• Easier TDD at all levels
• Separate runtimes
• Better for testing qualities (aka“non-
functionals”)
• Proper full system testing
61

62
Test First at all levels makes it easier to design a
system for testability

63
But you will probably need also some
upfront design

Design Approaches
• Big upfront design
• No upfront design
• Rough upfront design
64

Design Approaches (A Diﬀerent
Perspective)
• Architecture-indiﬀerent
design
• Architecture-focused design
• Architecture hoisting
65

Big Upfront Design
“BDUF design for testability is hard
because it is difficult to know what the
tests will need in the way of control
points and observation points on the
SUT. We can easily build software that
is difficult to test. We can also spend a
lot of time designing in testability
mechanisms that are either insufficient
or unnecessary. Either way, we will
have spent a lot of effort with nothing
to show for it.”
From:“xUnit Test Patterns”

No Upfront Design
• Works if the design“emerges”by coding
and refactoring
• Can be quite eﬀective in creating a
testable system
• Requires a higher level of expertise
• Challenging for teams bigger than 3-4
people
• Doesn’t work for large systems

Rough Upfront Design
• Helps the team to keep the big picture in mind
• Pull in the same direction
• E.g., helps in choosing the right refactoring paths
• Useful to hoist important qualities
• Design changes and evolves as necessary

Architectural Drivers
• Qualities (aka, non-functional requirements, or ilities)
• Scalability
• Security
• Performance
• Usability
• Etc.
69

70
“All quality requirements
should be quantified.”
Tom Gilb

Start Small!
71
“A complex system that
works is invariably found to
have evolved from a simple
system that worked”

Travel Light
• No speculative design
• Remove“reusable”and“flexible”from your
dictionary!
72

Travel Light: Beware Of
Frameworks
73
• Can be very useful...
• ...but they can impair testability (among other
things)

A Walking Skeleton is a tiny implementation of
the system that performs a small end-to-end
function. It need not use the final architecture,
but it should link together the main
architectural components. The architecture
and the functionality can then evolve in
parallel.
74
Walking Skeleton
From: http://alistair.cockburn.us/Walking+skeleton

75
Walking Skeleton
Database
Client
GUI Domain layer Server
Walking skeleton

Risk Based Testing
• Least testing
• Low likelihood of failure, low consequences of failure
• Moderate testing
• Low likelihood of failure, high consequences of failure
• High likelihood of failure, low consequences of failure
• Most testing
• High likelihood of failure, High consequences of
failure
76

What Kind Of Tests?
77
• Manual Tests?
• System Tests?
• Component Tests?
• Unit Tests?

78
Tests are mainly for maintainability
and evolution.

Tests Need To
• Be reasonably simple to create
• Or people will not create enough of them, or, perhaps, create just
some easy, but not necessarily high value, ones
• Provide feedback at the right granularity level (observability and
controllability)
• Running an entire distributed system to check the implementation
of a single class or method is inconvenient, to say the least
• Easy and fast to run
• Or they won’t be executed as often as they should be (if at all)
79
http://asprotunity.com/blog/system-tests-can-kill-your-project/

80
@Test 
 
 
 
}}); 
 
}

81
System
Test System
Entry points for standard users
Entry points for testing

• Can be quite complicated to setup and run, especially for distributed systems
• Can be very slow, making the feedback loop quite long
• May be very diﬃcult or even impossible to run on a developer’s machine
• Don’t scale well compared to other kinds of automated tests
• Are unhelpful in pinpointing the hiding places of bugs found in production
• Their coarse-grained nature makes them unsuitable for checking that a single
component, class, function, or method has been implemented correctly.
82
System Tests (Limitations)

Can be very diﬃcult to use with
distributed teams (especially in large
projects)
System Tests (Limitations)
83

Concluding Remarks
• Implications On Design
• Added complexity on some dimensions
• But added simplicity in others
• More stuﬀ to take care of
• Tests need to be maintained
• ...But it’s still cheaper than running the application
manually to smoke test changes...
85

Take aways
• Testability improves other qualities of the system, as
well as delivery time and cost
• Testability and system design depend on each other
• Test First at all levels helps
86

87
Books
ptg6985396
From the Library of Giovanni Asproni

Design For Testability

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to Design For Testability

Similar to Design For Testability (20)

More from Giovanni Asproni

More from Giovanni Asproni (8)

Recently uploaded

Recently uploaded (20)

Design For Testability