This document discusses model-based testing of self-adaptive software systems. It begins by motivating the need for new testing methods for these complex systems. It then performs a failure analysis to develop a failure domain model and fault dependency graph. From this, it derives 6+1 requirements that testing of self-adaptive systems should assure. Finally, it discusses future work in instantiating the requirements for specific projects and developing appropriate testing models and coverage criteria.

1. Department of Computer Sience, Software Technology Group
Valencia, 31.05.2013
Towards Systematic Model-based Testing of
Self-adaptive Software
Georg Püschel, Sebastian Götz, Claas Wilke, Uwe Aßmann

2. Towards Systematic MBT of SAS Folie Nr. 2 von XYZ
Outline
1) SAS requires a new MBT foundation
2) Failure Analysis -- Method and Application
3) Result: SAS Testing Requirements
4) Conclusion and Future Work
02.06.2013

3. Towards Systematic MBT of SAS Folie Nr. 3 von XYZ
Motivation I: Why do we need…?
 Energy Auto Tuning models and architecture:
„CoolSoftware“
 Multi-NFP Auto-Tuning: ”Highly Adaptive Energy-Efficient
Computing” (HAEC)
 Role-based SAS architecture: „Smart Application Grids“
(SMAGS)
 Cyber-physical Systems including SAS behaviour: „Visual
and Interactive Cyber-Physical Systems Control and
Integration“ (VICCI)
 All projects research about requirements and design.
 To complete the SE process, testing is also required.
02.06.2013

4. Towards Systematic MBT of SAS Folie Nr. 4 von XYZ
Motivation II: Why should we generally….?
 SAS conform to a feedback loop principle (e.g., MAPE-K [KC03])
 Challenge:
“Creating validation and verification techniques to
test and evaluate control loops’ behaviour and
automatically detect unintended interactions.” [Cheng+08]
 Are SAS special to testing?
 As in design much more complex information has to be
specified.
 As SAS run control loops, testing methods can be
specifically tailored to it.
 Because of both, models should be to generate test cases.
(MBT – Model-based Testing)
02.06.2013

5. Towards Systematic MBT of SAS Folie Nr. 5 von XYZ
Related Work: What was done before?
 Gap (1): Besides challenge statements, analysis is rare.
 Gap (2): Only a minority considers the interaction between running system
processes and adaptation. (Transactions? Order of actions?)
 Gap (3): Generalization?
 Gap (4): Many approaches base on @runtime self-testing.
 But: For industry-ready and safe systems (e.g., with certification) we
require black box testing in advance (but before deployment)!
Some related approaches/methods:
 DiVA offers a comprehensive strategy, split in two phases [DiVA]:
 Early validation@design time: adaptation logic and context model are
executed in a simulation
 Operational validation: deals with context changes, uses Multi-
dimensional Coverage Arrays (MDCA)
 Testing strategy for context-adaptive systems: „Artificial Shake Table Testing“
coverage strategy to produce context test data [MB09]
 Further strategies generate test cases from context and control flows
[TSE+04][Wang+07]
 Cheng„s explicit test model of the total state space of an SAS. [Cheng+08]
02.06.2013

6. Towards Systematic MBT of SAS Folie Nr. 6 von XYZ
Long-term goals
02.06.2013
here
• failure analysis
• requirements
future
• instantiation
• generalized SAS testing

7. Towards Systematic MBT of SAS Folie Nr. 7 von XYZ
Failure Analysis prerequisites:
Requirements and Assumptions
 Following steps are performed in Failure Mode
and Effects Analysis (FMEA) [FMEA]:
1. Build Failure Domain Model (dimensions&properties)
2. Investigate scenarios
(3.) visualize fault propagation in a Fault Dependency
Tree/Graph
But: We require a common architectural
perspective of SAS!
 Assumption: We do only consider SAS fulfilling the
MAPE-K principle.
02.06.2013

8. Towards Systematic MBT of SAS Folie Nr. 8 von XYZ
Failure Analysis, Step (1): Failure Domain Model for SAS
02.06.2013
Deviation from exepected
behaviour at a system‘s interface.
Inconsistent part of total system
state (inner state+perceivable
external state).
An error‘s cause.
Propagation to
another component.

9. Towards Systematic MBT of SAS Folie Nr. 9 von XYZ
Failure Analysis, Step (1): Failure Domain Model for SAS
02.06.2013
dimensions/properties
Standard dimensions, but important
to keep track of.
With or without inclusion of
knowledge model?
Which components of the control
loop are affected.
Especially if the failure includes
manipulating the outer reality.
Failures can also be associated with
non-optimal fulfilled goals.
Event-driveness/Determintation of
adaptation initiation.

10. Towards Systematic MBT of SAS Folie Nr. 10 von XYZ
Failure Analysis, Step (2): Scenarios
02.06.2013
check
Monitor Plan
Scheduler
Configuration
planner
(Planner)
Adaptation
logic execution
(Analyzer)
Event
monitoring
& processing
(Monitor)
Analyse
Action queue
synch
System
actions
PRE
PLAN
SCHED
Sensors Effectors
SENS
EFFECT
POST
ADAPT
Execute
Executor
RECONF
TRIG
EVENT
„K“

11. Towards Systematic MBT of SAS Folie Nr. 11 von XYZ
Failure Analysis, Step (3): Fault Dependency Graph
 the FDG visualizes the casual chain of
failure propagation
 the cyclic propagation is estabilished
through the stateful knowledge memory
and potential interdependencies of sensing
manipulated outer/physical reality
(dashed arrow)
 PRE and TRIG are coupled in both
directections such that these scenarios
must be tested together
02.06.2013

12. Towards Systematic MBT of SAS Folie Nr. 12 von XYZ
 Task: State requirements which can be mapped to respective scenarios.
 Requirements as assurance tasks/goals.
Assure…
1) …correct sensor interpretation. (SENS)
2) …correct adaptation initiation. (TRIG/PRE/ADAPT)
3) …correct adaptation planning. (PLAN)
4) …consistent interaction between adaptation and
system behavior. (SCHED)
5) …consistent adaptation execution. (POST/RECONF)
6) …correct system behavior. (EVENT/EFFECT)
Extra:
7) Find contructive (to generate) and analytic (to measure) coverage criteria for
SAS.
Result: SAS Testing Requirements
02.06.2013

13. Towards Systematic MBT of SAS Folie Nr. 13 von XYZ
Conclusion and Future Work
 We applied FMEA to the MAPE-K concept.
 Failure Domain Model: properties of failures in
SAS
 Fault Dependency Graph shows some special
causal interdependencies in SAS
 6+1 derived requirements for SAS testing
 its a basic toolset for SAS testers
 Future Work:
 instantiate requirements for our (initially named)
projects
 … other to existing SAS frameworks
 therefore, develop appropriate testing models and
coverage criteria
02.06.2013

14. Towards Systematic MBT of SAS Folie Nr. 14 von XYZ
FIN– Q/A?
02.06.2013
Thank you for your attendence.

15. Towards Systematic MBT of SAS Folie Nr. 15 von XYZ
Funding
This research has received funding…
• …within the project #100084131 by the
European Social Fund (ESF) and the German
Federal State of Saxony,
• …by Deutsche Forschungsgemeinschaft
(DFG) within CRC 912 (HAEC)
• …as well as T-Systems Multimedia Solutions
GmbH.
02.06.2013

16. Towards Systematic MBT of SAS Folie Nr. 16 von XYZ
References
[KC03] J. O. Kephart and D. M. Chess, “The vision of autonomic computing,” Computer, vol. 36, no. 1, Jan.
2003, pp. 41–50.
[Cheng+08] B. H. C. Cheng, D. Lemos, H. Giese, P. Inverardi, and J. M. et al., “Software engineering for self-
adaptive systems: A research roadmap,” in Dagstuhl Seminar 08031 on Software Engineering for Self-
Adaptive Systems, 2008.
[FMEA] H. E. Roland and B. Moriarty, System Safety Egnineering and Managemnent, 2nd edn. John Wiley &
Sons, Chichester, 1990, ch. Failure Mode and Effect Analysis.
[STA07] H. Sozer, B. Tekinerdogan, and M. Aksit, Archtitecting dependable systems IV. Springer, 2007, ch.
Extending failure models and effects analysis approach for reliability analysis at the software architecture
design level.
[TSA08] B. Tekinerdogan, H. Sozer, and M. Aksit, “Software architecture reliability analysis using failure
scenarios,” Journal of Systems and Software, vol. 81 (4), 2008, pp. 558–575.
[MB09] F. Munoz and B. Baudry, “Artificial table testing dynamically adaptive systems,” 2009.
[DiVA] A. Maaß, D. Beucho, and A. Solberg, “Adaptation model and validation framework final version (DiVA
deliverable D4.3),” 2010.
[Tse+04] T. Tse, S. Yau, W. Chan, H. Lu, and T. Chen, “Testing context-sensitive middleware-based software
applications,” 28th Annual International Computer Software and Applications Conference, 2004, pp. 458–466.
[Wang+07] Z. Wang, S. Elbaum, and D. S. Rosenblum, “Automated generation of context-aware tests,” 29th
International Conference on Software Engineering (ICSE), 2007, pp. 406–415.
02.06.2013