This PhD dissertation by Zoltán Micskei focuses on developing languages and frameworks for specifying test artifacts in software testing, particularly for high availability middleware and mobile systems. It presents new scientific results on robustness testing methodologies, semantic frameworks for UML sequence diagrams, and the creation of a test language for mobile systems. The document also discusses various challenges and approaches in testing while evaluating the robustness of different middleware implementations.

1. Languages and frameworks for
specifying test artifacts
PhD dissertation public defense
Zoltán Micskei
Advisors: István Majzik (BME)
Hélène Waeselynck (LAAS-CNRS)

2. Software testing concepts
Test framework = test artifacts + supporting tools
2
Testing process:
Test artifacts:
Specification,
requirements
Test cases Verdicts
Test
execution
Specification,
requirements
Test
requirements
Test purposes
Test suite
(test cases) Test traces Verdicts
Test approach
Test
execution
Test
evaluation
Test oracle

3. Test languages
 Specific languages for each artifact:
o E.g. U2TP, TTCN-3, ATML …
 Requirements and purposes:
o Graphical scenario languages
o MSC, LSC, UML 2 Sequence Diagrams
3

4. New application domains (1)
High availability (HA) middleware systems
 Standardizing HA functions
o SA Forum AIS specification
 Characteristics
o State-based nature
o Robustness is a key factor
 Challenges
o Test languages do not address robustness testing
o HA middleware is a complex system
4

5. New application domains (2)
Context-aware mobile computing systems
 Mobile computing system
o laptop, phone, intelligent car…
 Characteristics
o Context awareness
o Dynamic, evolving environment
o Communication with unknown partners
 Challenges
o Describing dynamic configurations
o Extensions for describing contexts
5

6. Common design approach
6

7. Common design approach
7

8. Contents
 Preliminaries and objectives
 New scientific results
o Robustness testing of standard-based HA middleware
o Semantic choices in UML 2 Sequence Diagrams
o A test language and framework for mobile systems
 Conclusion
8

9. Research question
 Review of previous API robustness testing
 Identifying sources for activating failures
9

10. Test approach
10

11. Systematic design of test tools
1. Collecting test artifacts’
requirements
2. Designing the new
test languages
3. Developing test tools
11
E.g.: type-specific testing
Required:
• Definition of functions
• Definition of types
• Selecting test values

12. Case studies
 Objective: Can the test framework be used to
assess the robustness of HA middleware systems?
o Qualitative observations
 Unit of analysis: OpenAIS, OpenSAF, SAFE4TRY
 Data collection: exit and error codes, logs
 Case studies performed:
1. Can the complex techniques detect more failures?
2. Can the three techniques uncover different failures?
3. Can the different implementations be compared?
12

13. Thesis 1
 Thesis 1: Robustness testing of HA middleware
o Following a systematic method I identified potential
activation modes of robustness faults (including
activation through stateless API, stateful API and
underlying services), designed languages to represent
the related test artifacts, and developed algorithms for
tools that use these languages to generate test data.
o I implemented the languages and tools in a test
framework, which can compare the robustness of
standard specification-based middleware
implementations.
 Collaboration: Francis Tam (Nokia Research Center)
13

14. Related publications
 Z. Micskei, H. Madeira, A. Avritzer, I. Majzik, M. Vieira, and N. Antunes. “Robustness Testing
Techniques and Tools”. In: Resilience Assessment and Evaluation of Computing Systems. Ed. by K.
Wolter, A. Avritzer, M. Vieira, and A. van Moorsel. Springer, 2012, pp. 323–339. doi: 10.1007/978-3-
642-29032-9_16
 Z. Micskei, I. Majzik, and F. Tam. “Comparing Robustness of AIS-Based Middleware Implementations”.
In: International Service Availability Symposium (ISAS 2007). Ed. by M. Malek, M. Reitenspieß, and A.
van Moorsel. Vol. 4526. LNCS. Springer, May 2007, pp. 20–30. doi:10.1007/978-3-540-72736-1_3
 A. Kövi and Z. Micskei. “Robustness Testing of Standard Specifications-Based HA Middleware”. In:
30th International Conference on Distributed Computing Systems Workshops (ICDCSW). IEEE, June
2010, pp. 302–306. doi: 10.1109/ICDCSW.2010.73
 Z. Micskei, I. Majzik, and F. Tam. “Robustness Testing Techniques for High Availability Middleware
Solutions”. In: International Workshop on Engineering of Fault Tolerant Systems (EFTS2006).
Luxembourg, Luxembourg: University of Luxembourg, June 2006, pp. 55–66
 Z. Micskei. “Robustness Comparison of High Availability Middleware Systems”. In: 14th PhD Mini-
Symposium. Budapest University of Technology and Economics. Budapest, Hungary: Department of
Measurement and Information Systems, Feb. 2007, pp. 70–73
 Z. Micskei. “Robustness Testing of High Availability Middleware Solutions”. In: 13th PhD Mini-
Symposium. Budapest University of Technology and Economics. Budapest, Hungary: Department of
Measurement and Information Systems, Feb. 2006, pp. 36–37
 Z. Micskei. “Nagy Rendelkezésre Állást Biztosító Köztes Rétegek Robosztusság Tesztelése”. In: Proc. of
Tavaszi Szél 2006. Kaposvár, Hungary: Doktoranduszok Országos Szövetsége, May 2006, pp. 295–298
14

15. Contents
 Preliminaries and objectives
 New scientific results
o Robustness testing of standard-based HA middleware
o Semantic choices in UML 2 Sequence Diagrams
o A test language and framework for mobile systems
 Conclusion
15

16. Research question
 Review of the OMG spec. and 13 formal semantics
 Categorizing semantic choices
 Evaluating the different options for the choices
16

17. Goal: easy to use presentation of results
 Notation inspired by feature models:
 „Test” diagrams and traces:
17
sd c2
a : A b : B
m1
Are the following traces valid, invalid or
inconclusive?
!m1.?m1.!m2.?m2.!m3.?m3.!m1.?m1
!m1.?m1.!m4.?m4.!m2.?m2.!m3.?m3
!m1.?m1.!m3.?m3.!m2.?m2.!m3.?m3
!m1.?m1.!m1.?m1.!m2.?m2.!m3.?m3
m3
m2

18. List of semantic choices
18

19. Thesis 2
 Thesis 2: I identified and categorized the semantic
choices and available options in UML 2 Sequence
Diagrams. I gave a structured framework with an
easy to use feature-model like representation of
the available options that can be used to adapt
the semantics of the language to a specific
purpose.
19

20. Related publications
 Z. Micskei and H. Waeselynck. “The many meanings of UML 2
Sequence Diagrams: a survey”. In: Software and Systems Modeling
10.4 (2011), pp. 489–514. doi: 10.1007/s10270-010-0157-9
 Z. Micskei and H. Waeselynck. A survey of UML 2.0 sequence
diagrams’ semantics. Tech. rep. 08389. Laboratoire d’Analyse et
d’Architecture des Systemes (LAAS), Aug.2008, pp.1–37
20

21. Contents
 Preliminaries and objectives
 New scientific results
o Robustness testing of standard-based HA middleware
o Semantic choices in UML 2 Sequence Diagrams
o A test language and framework for mobile systems
 Conclusion
21

22. Research question
 Case study:
Mobile Group
Membership
Protocol (GMP)
22
Test language and framework for mobile systems
Characteristics
Research question
Context awareness
Communication with
local broadcast
How can dynamic communication structures be specified in testing artifacts?
Dynamic, evolving
environment

23. Proposed notation for context information
 Two views: event (left) and spatial (right)
 Extending graphical scenario languages
23

24. Test approach and framework
24
Requirements
TERMOS language
- event view: messages
- spatial view: configurations
Execution environment
Network
simulator
Context
controller
Application execution
SUT
Test
components
Execution trace
Trace evaluation Verdict
GraphSeq tool
Configuration matching

25. TERMOS language
 Test Requirement language for Mobile Setting
 Defining the modeling language
o Abstract syntax: profile + restricting UML
o Concrete syntax: existing graphical elements
o Well-formedness: constraints for verification
o Semantics: based on Thesis 2’s framework
(goal: evaluating test traces),
formal operational semantics
25

26. Formal semantics: example
26
„Pre-processing”
„Unwinding”

27. Thesis 3
 Thesis 3: I designed a test requirement language that can
be used in the domain of mobile systems.
o I defined the syntax of the language using extensions to the
UML Sequence Diagrams’ metamodel, and its semantics using
an automaton-based formal operational semantics.
o The language is capable of expressing local broadcasts and
changes in the communication topology, and has the necessary
syntactic and semantic choices to make the specified
requirements checkable.
 Collaboration: Nicolas Rivière, Minh Duc Nguyen (LAAS)
27

28. Related publications
 G. Pintér, Z. Micskei, A. Kövi, Z. Égel, I. Kocsis, G. Huszerl, and A. Pataricza.
“Model-Based Approaches for Dependability in Ad-Hoc Mobile Networks and
Services”. In: Architecting Dependable Systems V. Springer, 2008, pp. 150–174.
doi: 10.1007/978-3-540-85571-2_7
 H. Waeselynck, Z. Micskei, N. Rivière, Á. Hamvas, and I. Nitu. “TERMOS: a Formal
Language for Scenarios in Mobile Computing Systems”. In: Mobile and
Ubiquitous Systems: Computing, Networking, and Services (MobiQuitous 2010).
Ed. by P. Sénac, M. Ott, and A. Seneviratne. Sydney, Australia, Dec. 2010, pp.
285–296. doi: 10.1007/978-3-642-29154-8_24
 H. Waeselynck, Z. Micskei, M.D. Nguyen, and N.Rivière. “Mobile Systems from a
Validation Perspective: a Case Study”. In: Sixth International Symposium on
Parallel and Distributed Computing (ISPDC ’07). Ed. by D. Kranzlmüller, W.
Schreiner, and J. Volkert. Hagenberg, Austria: IEEE Computer Society, July 2007,
pp. 85–92. doi: 10.1109/ISPDC.2007.37
 Z. Micskei. “Specifying Tests for Ad-Hoc Mobile Systems”. In: 15th PhD Mini-
Symposium. Budapest University of Technology and Economics. Budapest,
Hungary: Department of Measurement and Information Systems, Feb. 2008, pp.
32–35
 Z. Micskei, H. Waeselynck, M. D. Nguyen, and N. Rivière. Analysis of a group
membership protocol for Ad-hoc networks. Tech. rep. 06797. Laboratoire
d’Analyse et d’Architecture des Systemes (LAAS), Nov. 2006, pp. 1–42
28

29. Contents
 Preliminaries and objectives
 New scientific results
o Robustness testing of standard-based HA middleware
o Semantic choices in UML 2 Sequence Diagrams
o A test language and framework for mobile systems
 Conclusion
29

30. Application of new results
 Comparing robustness of HA middleware
o OpenAIS, OpenSAF, SAFE4TRY implementations
o Reporting faults to the developers
o Publication of the test suite and the results
 Testing mobile systems
o TERMOS language: HIDENETS EU FP6 project
o Extending results in the EU ARTEMIS R3-COP project
30

31. Summary of new results
31