The document discusses evaluating the sustainability of software architectures. It examines how scenario-based evaluation methods and architecture-level metrics can be used to assess sustainability. Specifically, it looks at how methods like ATAM and ALMA incorporate sustainability criteria through change scenarios. It also identifies over 40 metrics that measure qualities like encapsulation, modularity, and size which impact sustainability. The document concludes there are opportunities for more empirical research on methods and metrics, integrating approaches, and codifying experiences to better support sustainability evaluation in practice.

1. Sustainability Evaluation
of Software Architectures
Heiko Koziolek
Industrial Software Systems
ABB Corporate Research, Germany
1

2. 2 2

3. • Technology
2004 • Technology
2011
– GUI: Microsoft Visual Basic – GUI: Microsoft WPF
– Middleware: Classic OPC – Middleware: OPC UA
– OS: Windows XP – OS: Windows 7
• Functionality (select.) • Functionality (select.)
– Basic alarm management – Rich alarm management
– No system integration – Multi-system integration
– No online updates – Online updates
• Code & Documentation • Code & Documentation
– Defined APIs and layers – Some layer violations
– Design rationale present – Some design rationale lost
– Up-to-date documentation – Outdated documentation
3

4. Measuring sustainability?
4

5. State of practice:
Techniques for architecture evaluation
Experience 83%
Prototyping 70%
Scenarios 54%
Checklists 40%
Simulation 17%
Questionaires 16%
Metrics 15%
Others 11%
Math. models 5%
[Source: AliBabar & Gorton 2009, survey with 88 architects]
5

6. Research Questions
• RQ1:
Scenario-based methods
& sustainability evaluation?
• RQ2:
What kind of architecture-level metrics?
• RQ3:
Implications for industry & research?
6

7. Search
• 14 conferences, 12 journals, 10 books
• Included only methods/metrics
from renowned venues
• Excluded dormant or unvalidated
methods/metrics
7

8. Research Questions
• RQ1:
Scenario-based methods
& sustainability evaluation?
• RQ2:
What kind of architecture-level metrics?
• RQ3:
Implications for industry & research?
8

9. Scenario-based Methods
• SAAM (5%) • ALMA (1%)
• ATAM (5%) • ESAAMI
Practitioners
• ARID • SAAF applying
the method
• SAAMER • ASAAM [AliBabar &
• PASA • SALUTA Gorton2009]
• SBAR • SACAM
• SAAMCS dormant
• DOSAM
• ISAAMCR • SBAR
9

10. Sustainability Evaluation Criteria
Change scenario Ripple effect Variation point
specification analysis analysis
Architecture Existing tools &
Tradeoff analysis
improvement artifacts

11. Sustainability Evaluation Criteria
Change scenario Ripple effect Variation point
specification analysis analysis
Architecture Existing tools &
Tradeoff analysis
improvement artifacts

12. ATAM ALMA
Change
Sustainability evaluation
scenario spec.
Ripple effect
analysis
Variation point
analysis
Architecture
improvement
Tradeoff
analysis
Existing
artifacts/tools
Return on
Investment
12

13. ATAM ALMA
Change
Sustainability evaluation
Quality attribute Top-down, bottom-up,
scenario spec. scenario template no template
Ripple effect Manual, based on Manual, based on
analysis experience experience
Variation point n/a n/a
analysis
Architecture Modifiability Tactics, n/a
improvement Styles
Tradeoff Utility tree with n/a
analysis stakeholder preferences
Existing No support No support
artifacts/tools
Return on Cost: 30-70 person days Cost: unknown
Investment Benefit: not quantified Benefit: not quantified
13

14. Findings RQ1: Scenarios
• ATAM: more features
• ALMA: more specific for sustainability
• Manual ripple effect analysis
• No reverse engineering
• No tool integration
• Return on Investment unknown
• In practice: use best of both methods
14

15. Research Questions
• RQ1:
Scenario-based methods
& sustainability evaluation?
• RQ2:
What kind of architecture-level metrics?
• RQ3:
Implications for industry & research?
15

16. Architectural metrics categories
Similarity of Independent
Encapsulation
Purpose Compil. & Testing
Acyclic [Sarkar2007]
Size
16 Dependencies

17. Architectural metrics categories
Similarity of Independent
Encapsulation
Purpose Compil. & Testing
Acyclic [Sarkar2007]
Size
17 Dependencies

18. Architecture Metrics
Similarity of
Purpose
Encapsulation
Independent
Compilation
& Testing
Acyclic
Dependencies
Size
18

19. Architecture Metrics
Similarity of Concept Domination Metric, Concept Coherency Metic,
API Function Usage Index, Concern Diffusion over Arch. Components,
Purpose Concern Scope, Concern Overlap
Encapsulation Ratio of Cohesive Interactions, Import Coupling, Export Coupling,
Modularization Quality, Afferent Couplings, Efferent Couplings,
Component-level Interlacing Between Concerns,
Lack of Concern-based Cohesion, Module Interaction Index,
Non-API Function Closedness Index, Implicit Dependency Index,
Base class fragility index, Inheritance-based inter-module coupling,
Not-programming-to-interfaces Index, Association-induced coupling,
State Access Violation Index, Entropy of an architectural slicing,
Architecture Slicing Cohesion, Decision Volatility
Independent Cumulative Component Dependency, Average Cumulative Component
Dependency, Normalized Cumulative Component Dependency,
Compilation Coupling of a module, Intra-module coupling of a module, Cohesion of a
& Testing module, Abstractness, Instability, Distance from the Main Sequence,
Module Interaction Stability Index, Normalized Testability Dependency
Metric, Plugin Pollution Index, Change impact, Independence Level
Acyclic Package Dependency Cycles, Cyclic Dependencies Index, Layer
Organization Index, Excessive Structural Complexity
Dependencies
Size Module Size Boundness Index, Module Size Uniformity Index
19

20. Architecture Metrics
Similarity of Concept Domination Metric, Concept Coherency Metic,
API Function Usage Index, Concern Diffusion over Arch. Components,
Purpose Concern Scope, Concern Overlap
Encapsulation Ratio of Cohesive Interactions, Import Coupling, Export Coupling,
Modularization Quality, Afferent Couplings, Efferent Couplings,
Component-level Interlacing Between Concerns,
Lack of Concern-based Cohesion, Module Interaction Index,
Non-API Function Closedness Index, Implicit Dependency Index,
Base class fragility index, Inheritance-based inter-module coupling,
Not-programming-to-interfaces Index, Association-induced coupling,
State Access Violation Index, Entropy of an architectural slicing,
Architecture Slicing Cohesion, Decision Volatility
Independent Cumulative Component Dependency, Average Cumulative Component
Dependency, Normalized Cumulative Component Dependency,
Compilation Coupling of a module, Intra-module coupling of a module, Cohesion of a
& Testing module, Abstractness, Instability, Distance from the Main Sequence,
Module Interaction Stability Index, Normalized Testability Dependency
Metric, Plugin Pollution Index, Change impact, Independence Level
Acyclic Package Dependency Cycles, Cyclic Dependencies Index, Layer
Organization Index, Excessive Structural Complexity
Dependencies
Size Module Size Boundness Index, Module Size Uniformity Index
20

21. Example: Module Interact. Idx
call call call ������ ������
������������ ∈ ������1 … ������������ ������������������������ (������ ������ )
������������������ ������ =
������������������������ (������)
1+1
= ≈ 0.67
API Non-API 3
������
[Source: Sarkar2007 IEEE TSE]
21

22. Example: Comparing Systems
[Source: Sarkar2007 IEEE TSE]
22

23. Findings RQ2
• >40 architecture-level
source code metrics in literature
• Measures for architecture
and modularization quality
• Success factors for application:
– Mix of metrics
– Combination with
class-level / process metrics
• Limitations:
– Empirical validation
– Tool support
23

24. Research Questions
• RQ1:
Scenario-based methods
& sustainability evaluation?
• RQ2:
What kind of architecture-level metrics?
• RQ3:
Implications for industry & research?
24

25. Findings RQ3: Implications
• More empirical research
• Method integration
• Effective formal models
• Codify experiences
• Explore other
approaches (simulation?)
25

26. Conclusions
• Sustainability Evaluation of Architectures
– change scenarios:
basic support for sustainability criteria
– architectural code metrics:
measure modularization, lack empirical evidence
• Research opportunities
– combined methods
– empirical research
26

27. 27

28. Existing surveys
• Surveys on scenario- • Surveys / papers
based method on code metrics
– Dobrica 2002 – McCabe 1976
– Ali Babar 2004 – Halstead 1977
– Kazman 2005 – Chidamber 1994
– Breivold 2010 – Riaz 2009
– ... – ...
28 28

29. ATAM ALMA
Goals Sensitivity & Tradeoff Change impact analysis,
Analysis maint. effort prediction
Architectural Process, data-flow, uses, Any
description physical & module view
Process Comprehensive Limited
support
Tool support n/a n/a
Knowledge Recommended n/a
repository
Validation > 20 industrial case 7 industrial case studies
studies documented (before 2004)
29