State-of-the-art testing approaches typically include different testing levels like reviews, unit testing, component testing, integration testing, system testing, and acceptance testing. There is also common sense that typically unit testing is done by developers (they are responsible to check the quality of their units at least to some extent) and system testing is done by professional independent testers. But, who is responsible to adequately test the architecture which is one of the key artifacts in developing and maintaining flexible, powerful, and sustainable products and systems? History has shown that too many project failures and troubles are caused by deficiencies in the architecture.Furthermore, what does the term architecture testing mean and why is this term seldom used? To answer these questions, Peter describes what architecture testing is all about and explains a list of pragmatic practices and experiences to implement it successfully. He offers practical advice on the required tasks and activities as well as the needed involvement, contributions, and responsibilities of software architects in the area of testing – because a close cooperation between testers and architects is the key to drive and sustain a culture of prevention rather than detection across the lifecycle. Finally, if we claim to be in pursuit of quality then adequate architecture testing is not only a lever for success but a necessity. And this results not only in better quality but also speeds up development by facilitating change and decreasing maintenance efforts.

1. Copyright © Siemens AG 2011. All rights reserved.
Corporate Technology
EuroSTAR 2011
Manchester, England
November 24, 2011
Peter Zimmerer
Principal Engineer
Siemens AG
Corporate Technology
81739 Munich, Germany
peter.zimmerer@siemens.com
http://www.siemens.com/corporate-technology/
Architecture Testing: Wrongly Ignored!

2. Contents
Motivation
Architecture testing
Practices and tasks in architecture testing
 Involvement of Software Architects in testing activities
Summary

3. Motivation
The system‘s architecture is important …
 Vehicle for communication among stakeholders – including testers!
 Providing system context, scoping, and partitioning of development work
 Foundation for many lifecycle artifacts – including the test basis!
 Representing early design decisions and constraints:
most critical to get right, hardest to change
 First design artifact addressing (enabling or inhibiting) objectives,
non-functional requirements, and quality attributes
 Key for systematic reuse, integration with legacy / third party software,
managing change, cost-effective maintenance and evolution
 Poor architectures are a major cause for project failures and disaster
… so, we should test it thoroughly!

4. Architecture testing Glossary V2.1, April 2010@
Glossary V2.0, Dec 2007
A search in the glossary for “architecture“ results in only one hit:
design-based testing:
An approach to testing in which test cases are designed based on the
architecture and/or detailed design of a component or system (e.g.
tests of interfaces between components or systems).

5. Architecture testing
Requirements testing
Architecture testing
Document testing
Unit testing
Integration testing
System testing
Acceptance testing
1,010,000
102,000
191,000
4,900,000
1,630,000
2,770,000
1,840,000
Result hits* @ :
@
*Hits counted on September 12, 2011 – ratio still about the same since 2005 …

6. Testing strategy – Based on risks
Base the testing strategy on business goals and priorities
 Risk-based testing strategy (RBT)
Risk identification
Risk = P × D
 P Probability of failure
 Frequency of use
 Chance of failure:
criticality & complexity at implementation & usage, lack of quality
 D Damage (consequence & cost for business & test & usage)
Risk analysis – Product risk analysis workshop
Risk response – Test objectives, test levels, test design methods …

7. Product risk analysis for software architects
Identify especially all risks related to the architecture
 Architectural requirements (functional and non-functional)
 Architectural quality attributes
 Architectural use cases, features, and functions
 Architectural decisions
 Design decisions
 Technology selections
 Third-party component selections (open source)
 Bug categories
Actively participate in a product risk analysis workshop as one
important stakeholder

8. Test levels – Example V model
System
Requirements
Architecture,
Design
Unit
Specification
Coding
Acceptance
Testing
System
Testing
Integration
Testing
Unit
Testing
User
Requirements
based on

9. Test levels – Example V model with architecture testing
System
Requirements
Architecture,
Design
Unit
Specification
Acceptance
Testing
System
Testing
Integration
Testing
Unit
Testing
User
Requirements
Code Quality
Management
Architecture
interrogation:
interviews,
interactive
workshops
Architecture testing
is any testing of architecture and architectural artifacts
 mapping of architectural risks to test levels
Test case design
Analysis,
reviews,
previews,
inspections
Load model
specification
based on
Coding
Static testing
MTW,
QAW,
ATAM
Prototyping,
simulation
From SEI, see
www.sei.cmu.edu/

10. Test-driven development (TDD) ≈ Preventive Testing
Preventive testing is built upon the observation that
one of the most effective ways of specifying something is
to describe (in detail) how you would accept (test) it
if someone gave it to you.
David Gelperin, Bill Hetzel (<1990)
Real TDD means to let testing drive and influence the architecture
and design as well
 Do not use TDD only on the level of unit and acceptance testing

11. Testability – Factors
Control(lability)
 The better we can control the system,
the more and better testing can be done, automated, and optimized
 Ability to apply and control the inputs to the system under test
or place it in specified states (for example reset to start state)
 Interaction with the system under test (SUT) through control points
Visibility / observability
 What you see is what can be tested
 Ability to observe the inputs, outputs, states, internals, error conditions,
resource utilization, and other side effects of the system under test
 Interaction with the system under test (SUT) through observation points
Control(lability)
Visibility / observability

12. Design for Testability (DfT) – Who?
Joint venture between architects (+ developers) and testers
 Collaboration between different stakeholders
Testability must be built-in by architects (+ developers)
 Pro-active strategy:
design the system with testability as a key design criterion
 Testers (test automation engineers) must define testability
requirements to enable effective and efficient test automation
Contractual agreement between design and test
Balancing production code and test code
 Production code and test code must fit together
 Manage the design of production code and test code
as codependent assets
EducatestakeholdersonbenefitofDfT

13. Test design methods
Test design methods are not for testers only …
Check and foster the usage of test design methods in the
development team
Use test design methods
 To provide an adequate test basis
 In architecture reviews, for example scenario testing
 To design test cases for the architecture,
especially for integration testing
 To prevent bugs (think test-first …)
The act of designing tests is one of the
most effective bug preventers known.
Boris Beizer, 1983

14. Integration testing
The goal of integration testing is to test in a grey-box manner
 The interaction of components and subsystems
 The interaction and embedding with the environment and system
configuration
 The dynamic behavior and communication of the system
 Control flow and data flow
 The architecture and design as specified in the
Software Architecture Description document
Select an appropriate integration strategy
that supports and drives the goals and
benefits of integration testing
Address and follow integration testing needs
in the architecture (including testability)
Integration testing
Integration
Architecture

15. Test architectures, test automation
The architecture of a test system may be even more complex than
the architecture of the system under test
Use good architecture practices to define, specify, create, realize,
and maintain the test (automation) system as well
An efficient quality check of software is not possible
in real project life without the right test automation!
 Incremental, iterative, agile processes
 Regression testing
 Maintenance – refactoring, redesign, reengineering

16. Regression testing – The big questions are always …
Software Architect:
Which parts of the architecture must be tested again
after a (code) change?
Test Manager:
Which test cases must be executed again
after a (code) change?
Remark:
Change can be a code change
but also any change in the environment or system configuration
Use knowledge about the architecture and design of the system
during change and impact analysis
 to identify risky areas
 to select required regression test cases effectively

17. Architectural quality – internal software quality
and code quality management
Checking of architectural conformance
 Validate the formalized description
 Evaluate reported architecture violations
 Simulate a restructuring of a system (refactoring, redesign,
reengineering)
Identification of suspicious components and trends (software aging)
 Detect potential problems early enough
Analyzing the impact of design changes and design alternatives
 Use it as input for well-founded decisions
Code comprehension
 Get a well-rounded knowledge
of the real state of the system
Avoid this:

18. Practices and tasks in architecture testing
Understand the mission and the value of testing and promote it
Risk-based testing strategy (test levels)
Test-driven development
Design for testability
Test design methods
Integration testing
Test architectures, test automation
Regression testing
Architectural quality – internal software quality and code quality
management

19. Summary
The Software Architect cooperates closely with the Test Manager
 To define, motivate, drive, and enforce a comprehensive
understanding of and attitude to testing and quality in the whole team
 To sustain a culture of prevention rather than detection across the
lifecycle
The Software Architect and the Test Manager address "quality"
from different viewpoints:
 Software Architect:
Build quality into the system (or rather into the architecture)
and do architecture testing
 Test Manager:
Provide information related to quality (feedback)