Testability is the degree to which a system can be effectively and efficiently tested. This key software attribute indicates whether testing (and subsequent maintenance) will be easy and cheap—or difficult and expensive. In the worst case, a lack of testability means that some components of the system cannot be tested at all. Testability is not free; it must be explicitly designed into the system through adequate design for testability. Peter Zimmerer describes influencing factors (controllability, visibility, operability, stability, simplicity) and constraints (conflicting nonfunctional requirements, legacy code), and shares his experiences implementing and testing highly-testable software. Peter offers practical guidance on the key actions: (1) designing well-defined control and observation points in the architecture, and (2) specifying testability needs for test automation early. He shares creative and innovative approaches to overcome failures caused by deficiencies in testability. Peter presents a new, comprehensive strategy for testability design that can be implemented to gain the benefits in a cost-efficient manner.

1.
TE
Half‐day Tutorial
6/4/2013 8:30 AM
"Design for Testability:
A Tutorial for Devs and Testers"
Presented by:
Peter Zimmerer
Siemens AG
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888‐268‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Peter Zimmerer
Siemens AG
Peter Zimmerer is a principal engineer at Siemens AG, Corporate Technology, in Munich,
Germany. For more than twenty years Peter has been working in the field of software testing
and quality engineering. He performs consulting, coaching, and training on test management
and test engineering practices in real-world projects, driving research and innovation in this
area. An ISTQB® Certified Tester Full Advanced Level, Peter is a member of the German
Testing Board, has authored several journal and conference contributions, and is a frequent
speaker at international conferences. Contact Peter at peter.zimmerer@siemens.com.

3. Corporate Technology
Design for Testability:
A Tutorial for Devs and Testers
Agile Development Conference & Better Software Conference West
Las Vegas, NV USA
Vegas NV,
June 2-7, 2013
Peter Zimmerer
Principal Engineer
Siemens AG
Corporate Technology
81739 Munich, Germany
peter.zimmerer@siemens.com
peter zimmerer@siemens com
http://www.siemens.com/corporate-technology/
Unrestricted. © Siemens AG 2013. All rights reserved.
Contents
 What is testability all about?
 What are the influencing factors and constraints of testability?
 Why is testability so important?
 Who is responsible and who must be involved to realize design for
testability?
 How can we improve, foster, and sustain the testability of a system?
 Ho can we inno ate and renovate design for testability in our
How
e innovate
reno ate
testabilit
o r
projects?
 A new, comprehensive strategy on
design for testability over the whole lifecycle of a system
Page 2
June 4, 2013
Peter Zimmerer
Unrestricted. © Siemens AG 2013. All rights reserved.
Corporate Technology

4. A story on missing testability …
The Therac-25 was a radiation therapy machine
produced by Atomic Energy of Canada Limited
(AECL) after the Therac 6 and Therac 20 units.
Therac-6
Therac-20 units
It was involved in at least six accidents between
1985 and 1987, in which patients were given
massive overdoses of radiation, approximately
i
d
f di ti
i t l
100 times the intended dose. These accidents
highlighted the dangers of software control of
safety-critical systems, and they have become
a standard case study in health informatics and
software engineering
engineering.
A commission concluded that the primary reason should be attributed to the bad
software design and development practices and not explicitly to several coding
practices,
errors that were found. In particular, the software was designed so that it was
realistically impossible to test it in a clean automated way.
http://en.wikipedia.org/wiki/Therac-25
Page 3
June 4, 2013
Peter Zimmerer
Unrestricted. © Siemens AG 2013. All rights reserved.
Corporate Technology
Testability – Theoretical definitions
Testability is the ease of validation, that the software meets the requirements. Testability is broken
down into: accountability, accessibility, communicativeness, self-descriptiveness, structuredness.
Barry W. Boehm: Software Engineering Economics, Prentice Hall,1981
Testability is the degree to which, and ease with which, software can be effectively tested.
Donald G. Firesmith: Testing Object-Oriented Software, Proceedings of the Object EXPO Europe, London (UK), July 1993
The degree to which a system or component facilitates the establishment of test criteria and the
performance of t t t d t
f
f tests to determine whether those criteria have been met.
i
h th th
it i h
b
t
The degree to which a requirement is stated in terms that permit establishment of test criteria and
performance of tests to determine whether those criteria have been met.
IEEE Standard Computer Dictionary: A Compilation of IEEE Standard Computer Glossaries. New York NY 1990
Glossaries
York, NY,
The testability of a program P is the probability that if P contains faults, P will fail under test.
Dick Hamlet, Jeffrey Voas: Faults on its Sleeve. Amplifying Software Reliability Testing, in Proceedings of the 1993 International
Symposium on Software Testing and Analysis (ISSTA), Cambridge, Massachusetts, USA, June 28–30, 1993
Software testability refers to the ease with which software can be made to demonstrate its faults
through (typically execution-based) testing. Len Bass, Paul Clement, Rick Kazmann: Software Architecture in Practice
The degree to which an objective and feasible test can be designed to determine whether a
requirement is met. ISO/IEC 12207
The capability of the software product to enable modified software to be validated. ISO/IEC 9126-1
Degree of effectiveness and efficiency with which test criteria can be established for a system,
product or component and tests can be performed to determine whether those criteria have been
met. ISO/IEC 25010, adapted from ISO/IEC/IEEE 24765
Page 4
June 4, 2013
Peter Zimmerer
Unrestricted. © Siemens AG 2013. All rights reserved.
Corporate Technology

5. Testability – Factors (1)
Control(lability)
 The better we can control the system (in isolation),
the more and better testing can be done, automated, and optimized
 Ability to apply and control the inputs to the system under test
or place it in specified states (for example reset to start state, roll back)
state
 Interaction with the system under test (SUT) through control points
Control(lability)
Visibility / observability
Visibility / observability
y
y
 What you see is what can be tested
 Ability to observe the inputs, outputs, states, internals, error conditions,
resource utilization, and other side effects of th system under test
tili ti
d th
id ff t f the
t
d t t
 Interaction with the system under test (SUT) through observation points
Page 5
June 4, 2013
Peter Zimmerer
Unrestricted. © Siemens AG 2013. All rights reserved.
Corporate Technology
Testability – Factors (2)
Availability, operability
 The better it works, the more efficiently it can be tested
 Bugs add overhead for analysis and reporting to testing
 No bugs block the execution of the tests
Simplicity, consistency, and decomposability




The less there is to test, the more quickly we can test it
Standards, (code / design) guidelines, naming conventions
Layering, modularization, isolation, loose coupling, separation of concern, SOLID
y
g,
,
,
p g, p
,
Internal software quality (architecture, design, code), technical debt
Stability
 The fewer the changes, the fewer the disruptions to testing
changes
 Changes are infrequent, controlled and do not invalidate existing tests
 Software recovers well from failures
Understandability, knowledge (of expected results)
U d
t d bilit k
l d ( f
t d
lt )
 The more (and better) information we have, the smarter we will test
 Design is well understood, good and accurate technical documentation
@Microsoft: SOCK – Simplicity, Observability, Control, Knowledge
Page 6
June 4, 2013
Peter Zimmerer
Unrestricted. © Siemens AG 2013. All rights reserved.
Corporate Technology

6. Testability – Factors (3)
Visibility /
Observability
Control(lability)
Answer
Question
Availability
Operability
Stability
Oracle
Simplicity
Understandability
g
Knowledge
Page 7
June 4, 2013
Peter Zimmerer
Unrestricted. © Siemens AG 2013. All rights reserved.
Corporate Technology
Testability – Constraints
How we design and build the system affects testability
Testability i th k t
T t bilit is the key to cost-effective test automation
t ff ti t t t
ti
Testability is often a better investment than automation
 Test environment: stubs, mocks, fakes, dummies, spies
 Test oracle: assertions (design by contract)
 Implemented either inside the SUT or inside the test automation
Challenges for testability
 Size, complexity, structure, variants, 3rd party components, legacy code
 Regulatory aspects
 You cannot log every interface (huge amount of data)
 C fli t with other non-functional requirements like encapsulation,
Conflicts ith th
f
ti
l
i
t lik
l ti
performance or security, and impacts system behavior
 Non-determinism: concurrency, threading, race conditions, timeouts,
Non determinism:
message latency, shared and unprotected data
Page 8
June 4, 2013
Peter Zimmerer
Unrestricted. © Siemens AG 2013. All rights reserved.
Corporate Technology

7. Testability is affected / required by new technologies
y
y
g
Example: Cloud computing
Cloud testability – the negative
– Information hiding and remoteness
– Complexity and statefulness
– Autonomy and adaptiveness
– Dependability and performance
– Paradigm infancy (temporary)
Cloud testability – the positive
+ Computational power
+ Storage
+ Virtualization
Testing requires lots of resources
and the cloud is certainly powerful enough to handle it
Reference: Tariq M Ki
R f
T i M. King, A
Annaji S G i D id F li
ji S. Ganti, David Froslie:
Towards Improving the Testability of Cloud Application Services, 2012
Page 9
June 4, 2013
Peter Zimmerer
Unrestricted. © Siemens AG 2013. All rights reserved.
Corporate Technology
Design for Testability (DfT) – Why?
Testability ≈ How easy / effective / efficient / expensive is it to test?
Can it be tested at all?
Increase depth and quality of tests – Reduce cost, effort, time of tests
 More bugs detected (earlier) and better root cause analysis with less effort
 Support testing of error handling and error conditions, for example to test how a
component reacts to corrupted data
 Better and more realistic estimation of testing efforts
non functional
 Testing of non-functional requirements, test automation, regression testing
 Cloud testing, testing in production (TiP)
Reduce cost, effort, and time for debugging, diagnosis, maintenance
 T i ll underestimated b managers, not easy t measure h
Typically d
ti t d by
t
to
honestly
tl
Provide building blocks for self-diagnosing / self-correcting software
Possible savings ~10% of total development budget
(Stefan Jungmayr, http://www.testbarkeit.de/)
Simplify, accelerate,
Simplify accelerate and power up the testing
Focus the testing on the real stuff … and have more fun2013. All rights reserved.
in testing …
Unrestricted. © Siemens AG
Page 10
June 4, 2013
Peter Zimmerer
Corporate Technology

8. Edu
ucate stakeholders on ben
s
nefit of DfT
f
Design for Testability (DfT) – Who?
Joint venture between architects (+ developers) and testers
 Collaboration between different stakeholders
R
Requires accountability and clear ownership
i
t bilit
d l
hi
Testability must be built-in by architects ( developers)
built in
(+
 Pro-active strategy in tactical design:
design the system with testability as a key design criterion
T t
Testers (test automation engineers) must define testability requirements
(t t t
ti
i
)
t d fi t t bilit
i
t
to enable effective and efficient test automation
Contractual agreement between design and test
Balancing production code and test code
B l
i
d ti
d
dt t d
 Production code and test code must fit together
 Manage the design of production code and test code
as codependent assets
Page 11
June 4, 2013
Peter Zimmerer
Unrestricted. © Siemens AG 2013. All rights reserved.
Corporate Technology
Edu
ucate stakeholders on ben
s
nefit of DfT
f
Design for Testability (DfT) – How?
Suitable testing architecture, good design principles
Interaction with the system under test through
well-defined control points and observation points
interfaces, ports, hooks, mocks,
Additional (scriptable) interfaces ports hooks mocks interceptors
for testing purposes (setup, configuration, simulation, recovery)
Coding guidelines, naming conventions
guidelines
Internal software quality (architecture, code)
Control(lability)
Visibility / observability
Built-in lf t t (BIST), b ilt i t t
B ilt i self-test (BIST) built-in test (BIT)
Consistency checks (assertions, design by contract, deviations)
Logging and tracing (AOP, counters, monitoring, probing, profiling)
L
i
dt
i (AOP
t
it i
bi
fili )
Diagnosis and dump utilities (internal states, resource utilization)
Think test-first (TDD): how can I test this?Unrestricted. © Siemens AG 2013. All rights reserved.
Page 12
June 4, 2013
Peter Zimmerer
Corporate Technology

9. Architectural and design patterns (1)
Tester
Use layered architectures, reduce number of dependencies
Use good design principles
 Strong cohesion, loose coupling, separation of concerns
SUT
Real or simulated
environment
Component orientation and adapters ease integration testing
 Components are the units of testing
 Component interface methods are subject to black-box testing
Avoid / resolve cyclic dependencies between components
 Combine or split components
 Dependency inversion via callback interface (observer-observable
design pattern)
Example: MSDN Views Testability guidance
The purpose is to provide information about how to increase the
Model View Presenter
testability surface of Web applications using the Model-View-Presenter
(M-V-P) pattern. http://msdn.microsoft.com/en-us/library/cc304742.aspx
Page 13
June 4, 2013
Peter Zimmerer
Unrestricted. © Siemens AG 2013. All rights reserved.
Corporate Technology
Architectural and design patterns (2)
Provide appropriate test hooks and factor your design in a way that
lets test code interrogate and control the running system
 I l t and encapsulate d
Isolate d
l t dependencies on th external environment
d
i
the t
l
i
t
 Use patterns like dependency injection, interceptors, introspective
 Use configurable factories to retrieve service providers
 Declare and pass along parameters instead of hardwire references to
service providers
D l
Declare i t f
interfaces th t can be i l
that
b implemented b t t classes
t d by test l
 Declare methods as overridable by test methods
 Avoid references to literal values
 Shorten lengthy methods by making calls to replaceable helper methods
Promote repeatable, reproducible behavior
P
t
t bl
d ibl b h i
 Provide utilities to support deterministic behavior (e.g. use seed values)
 That’s just good design practice!
Page 14
June 4, 2013
Peter Zimmerer
Unrestricted. © Siemens AG 2013. All rights reserved.
Corporate Technology

10. Design for Testability (DfT) patterns
Dependency injection
The client provides the depended-on object
to th
t the SUT
Dependency lookup
The SUT asks another object to return the depended-on object before it
uses it
Humble object
We extract the logic into a separate easy-to-test component that is
easy to test
decoupled from its environment
Test h k
T t hook
We modify the SUT to behave differently during the test
Reference: Gerard Meszaros: xUnit Test Patterns: Refactoring Test Code, Addison-Wesley, 2007
http://xunitpatterns.com/
Page 15
June 4, 2013
Peter Zimmerer
Unrestricted. © Siemens AG 2013. All rights reserved.
Corporate Technology
Use principles and practices from clean code developer (CCD)
p
p
http://www.clean-code-developer.de/
http://www.clean-code-developer.com/
Page 16
June 4, 2013
Peter Zimmerer
Unrestricted. © Siemens AG 2013. All rights reserved.
Corporate Technology

11. Interfaces can act as
 control points
Attributes of testability interfaces  observation points
Indirect Input
Product API
Product Events
Test API
System
under
Test (SUT)
Depended
Components
Indirect Output
Test Events
 Manual vs. automatable
 Lightweight vs. intrusive
 Tester owned vs. developer owned
vs
 Realistic vs. artificial
vs
 Control-only vs. visibility-only vs. both
Information only
 Information-only vs. knowledge
 Debug-only (logs / asserts) vs. release build
 Internal (embedded) vs. external vs. shipping (documented) feature
 NFR (e.g. security) harmless vs. NFR conflict (e.g. security hole)
Page 17
June 4, 2013
Unrestricted. © Siemens AG 2013. All rights reserved.
Peter Zimmerer
Corporate Technology
Logging and tracing
Check the architecture and design.
Check the dynamic behavior
(i.e.
(i communication) of the system.
i ti ) f th
t
Detect bugs, including sporadic bugs
like race conditions.
Distributed System
...
Event
...
Event
Event Collection
Analyze execution, data flow, and system state.
Follow error propagation and reproduce faults.
F ll
ti
d
d
f lt
For this, tracing is essential.
Tracing is important for unit testing,
integration testing, system testing,
system monitoring, diagnosis, …
t
it i
di
i
Start early with a logging and tracing
Page 18
June 4, 2013
Peter Zimmerer
Trace
visualization
Trace
Trace
analysis
Relationship between
trace extraction, trace visualization,
concept! and trace analysis
Unrestricted. © Siemens AG 2013. All rights reserved.
Corporate Technology

12. Further benefits of good logging and tracing practices
Avoid analysis paralysis – a situation where the test team spends
as much time investigating test failures as they do testing
Support failure matching and (automated) failure analysis
Avoid duplicates in the bug database by using rules of analysis
when deciding if a failure has been previously observed
Backbone of reliable failure analysis
Example: Windows Error Reporting (WER)
 Small, well defined set of parameters
A t
Automated classification of issues
t d l
ifi ti
fi
 Similar crash reports are grouped together
Reduced amount of data collected from customers:
bucketing
Page 19
June 4, 2013
Peter Zimmerer
Unrestricted. © Siemens AG 2013. All rights reserved.
Corporate Technology
Diagnosis and dump utilities (internal states, resource utilization)
Detect, identify, analyze, isolate, diagnose, and reproduce
a bug (location, factors, triggers, constraints, root cause)
in diff
i different contexts as effective and as efficient as possible
t
t t
ff ti
d
ffi i t
ibl
CommunicationD
iagnosisExample
 Unit / component testing:
bug in my unit/component or external unit/component or environment?
 Integration testing of several subsystems:
bug in
b i my subsystem or external subsystem or environment?
b
t
t
l b
t
i
t?
 System of systems testing:
bug in which part of the whole system of systems?
 Product line testing in product line engineering (PLE):
bug in domain engineering part or application engineering part?
 B in system under t t (SUT) or test system?
Bug i
t
d test
t t
t ?
 quality of test system is important as well ...
 SUT in operation:
maintenance, serviceability at the customer site
Page 20
June 4, 2013
Peter Zimmerer
Unrestricted. © Siemens AG 2013. All rights reserved.
Corporate Technology

13. Edu
ucate stakeholders on ben
s
nefit of DfT
f
Strategy for Design for Testability (DfT) over the lifecycle (1)
 Consistently defined and well-known in the project to drive a common
understanding, awareness, and collaboration with clear accountability
 Addressed and specified as one important non-functional requirement
from the beginning
 Investigated by elicitation techniques (interviews, observation,
brainstorming, story writing, prototyping, personas, reuse, apprenticing)
 I l d d and elaborated i th product b kl
Included d l b t d in the
d t backlog
risk based
 Defined, specified, traced, and updated as a risk item in the risk-based
testing strategy with well-defined mitigation tasks
B l
Balanced with regards t conflicting non-functional requirements
d ith
d to
fli ti
f
ti
l
i
t
 Implemented dependent on the specific domain, system, and (software)
technologies that are available
Page 21
June 4, 2013
Peter Zimmerer
Unrestricted. © Siemens AG 2013. All rights reserved.
Corporate Technology
Edu
ucate stakeholders on ben
s
nefit of DfT
f
Strategy for Design for Testability (DfT) over the lifecycle (2)
 Specified and elaborated in a testability guideline for architects, PhilippeKruchten
nal_4+1ViewMode
developers, testers ( DfT – How?)
 Testability view*/profile in the architectural design approach (ADA)
view /profile
 Description how to recognize characteristics of the ADA in an artifact
to support an analysis and check of the claimed ADA
 A set of controls (controllables) associated with using th ADA
t f
t l (
t ll bl )
i t d ith i the
Control+Observatio
nPoints@Architecture
 A set of observables associated with using the ADA
 A set of testing firewalls within the architecture used for regression testing
(firewalls enclose the set of parts that must be retested, they are imaginary
boundaries that limits retesting of impacted parts for modified software)
 A set of preventive principles (e g memory / time partitioning):
(e.g.
specific system behavior and properties guaranteed by design (e.g. scheduling,
interrupt handling, concurrency, exception handling, error management)
 A fault model for the ADA: failures that can occur failures that cannot occur
occur,
 Analysis (architecture-based or code-based) corresponding to the fault model to
detect ADA-related faults in the system (e.g. model checking)
 Tests made redundant or de-prioritized based on fault model and analysis
*Additional view to the 4+1 View Model of Software Architecture by Philippe Kruchten
Page 22
June 4, 2013
Peter Zimmerer
Unrestricted. © Siemens AG 2013. All rights reserved.
Corporate Technology

14. Edu
ucate stakeholders on ben
s
nefit of DfT
f
Strategy for Design for Testability (DfT) over the lifecycle (3)
 Included as one important criteria in milestones and quality gates, e.g.:
 Check control and observation points in the architecture
 Ch k t t bilit requirements f sustaining t t automation
Check testability
i
t for
t i i test t
ti
 Check logging and tracing concept to provide valuable information
 Check testability support in coding guidelines
 Check testability needs (especially visibility) for regression testing
I
Investigated and explored i static t ti
ti t d d
l d in t ti testing:
architecture interrogation (interviews, interactive workshops),
architecture reviews, QAW, ATAM*, code reviews
ATAM ,
 Investigated and explored in dynamic testing:
special t t bilit t
i l testability tour i scenario t ti (
in
i testing (application t i )
li ti touring)
 Neglecting testability means increasing technical debt … $$ …
*Architecture-centric methods from SEI, see http://www.sei.cmu.edu/
Page 23
June 4, 2013
Peter Zimmerer
Unrestricted. © Siemens AG 2013. All rights reserved.
Corporate Technology
Summary – Testability and Design for Testability (DfT)
 What?
 Why?
 Who?
 How?
 Strategy for Design for Testability (DfT) over Siemens AG 2013. All rights reserved.
the lifecycle
Unrestricted. ©
Page 24
June 4, 2013
Peter Zimmerer
Corporate Technology