1.
13th CREST Open Workshop<br />12th-13th of May 2011, London<br />Search-Based Software Engineering for Model-Based Testing<br />Event-B in a Nutshell<br />Test Generation <br />Approaches<br />SBT Challenges<br />Test Data Generation<br />Finite Model <br />Learning<br />Conclusions<br />Alin Stefanescu - University of Pitesti, Romania<br />*) joint work with colleagues from DEPLOY project<br />

2.
[Event-B in a Nutshell]<br />

3.
Event-B history<br /><ul><li>Jean-Raymond Abrial (1938- )

4.
Inventor of the Z and B formal methods.

5.
Z – developed in the 70s

6.
B – developed in the 90s, successfully deployed in industry

7.
Event-B – born with the 21st century

8.
Evolution of B for system level specification

9.
Developement supported by French and European projects:</li></ul> FP6 RODIN and FP7 DEPLOY<br />

10.
DEPLOY project (2008-2012) – funded by FP7 <br />DEPLOY :: Industrial deployment of advanced systemengineering methods for high productivity and dependability using formal methods <br /><ul><li>4 industrial partners

11.
Bosch, Siemens, SAP, SSF

12.
3 industrial service providers

13.
Systerel, ClearSy, Cetic

14.
7 academic partners

15.
Newcastle, Aabo, Düsseldorf, ETH Zurich,

16.
Southampton, Pitesti, Bucharest</li></ul>http://www.deploy‐project.eu<br />

17.
Rodin platform for Event-B<br /><ul><li>Extension of Eclipse IDE (Java-based)

18.
Theorem proving as core technology

19.
Many other Rodin Plug-ins

20.
ProB:animation, consistency and model-checking

21.
Animators (AnimB)

22.
Decomposition

23.
Modularisation

24.
Team-work

25.
Code generation

26.
UML-B

27.
etc.</li></li></ul><li>Event-B in a nutshell<br />Event-B model<br /><ul><li>State-transition model(like ASM, B, Z)

28.
set theory as mathematical language

29.
refinement as basic modeling approach

30.
Contexts

31.
carrier sets (domains)

32.
constants

33.
axioms

34.
Machines

35.
global variables

36.
invariants

37.
events that update the variables

38.
Events

39.
local parameters

40.
guards

41.
actions</li></ul>ITEMS := CONTEXT {{it1}, {it34}, {it36}, {it67}, {it89}, {it11}, {it354}, {it876},{it321}, {it333}, {it78}, {it787}, {it7878}, {it2342}, {it3453}, {it6786}, {it1232}, {it7765}, {it7098})<br />items : Powerset(ITEMS)<br />

42.
[Test Generation Approaches]<br />

43.
Test generation based on Event-B<br />Model-Based Testing (MBT)<br />Model-based testing (MBT) is a newlyintroduced topic in DEPLOY<br />priority topic for industrial partners like SAP<br />challenges due to the sheer size of the state space of real-life scenarios<br />Weinvestigate search-based testing (SBT) techniques for Event-B.<br />

44.
Future MBT plugin in RODIN<br />Event-B model<br />MBT Plug-in<br />MBT <br />Users<br />{ Model-checking }<br />{ Constraint-based }<br />Extra <br />test <br />information<br />{ Search-based }<br />Test cases<br />University of Pitesti and University of Dusseldorf<br />Tool developers:<br />

45.
Test generation from Event-B<br />SBT Opportunity! <br />1. Generate a set of tests<br />(sequence of events with concrete param.)<br />2. Optimize test suite<br />(according to some criteria) – if still needed<br />Event-B model<br />ev2(..),ev5(..),...ev3()<br />ev4(..),ev2(..),...ev4(..)<br />...<br />...<br />...<br />...<br />...<br />...<br />...<br />...<br />ev3(..),ev7(..),...ev5(..)<br />...<br />...<br />...<br />...<br />...<br />ev6(..),ev5(..),...ev8<br />Global variables:<br /> var1, var2, var3, ...<br />Events<br /> ev1(p11,...), <br /> ev2(p21,...), <br /> ev3(p31,...),<br /> ... <br />ev2(..),ev5(..),...ev3()<br />ev4(..),ev2(..),...ev4(..)<br />...<br />ev3(..),ev7(..),...ev5(..)<br />

46.
What is the explicit state space<br />State Space of <br />the Event-B model<br />(3,4,{a,b},...)<br />Abstract machine<br />Event-B model<br />ev3(5)<br />Global variables:<br /> var1, var2, var3, ...<br />Events<br /> ev1(p11,...), <br /> ev2(p21,...), <br /> ev3(p31,...),<br /> ... <br />...<br />...<br />...<br />...<br />States given by the values of global variables<br />Transitions labeled by events with concrete parameters<br />

47.
Test generation from Event-B<br />State Space of <br />the Event-B model<br />(3,4,{a,b},...)<br />Approach 1:<br />Explore the state space using the <br /> ProB model checker<br />state space explosion mainly due to data<br />Try: guide the search<br />ev3(5)<br />...<br />...<br />...<br /> SBT Opportunity! <br />

48.
Test generation from Event-B – part II<br />State Space of <br />the Event-B model<br />(3,4,{a,b},...)<br />Approach 2:<br />Explore state space ignoring the data<br />(i.e. local parameters)<br />Problem 1: still large state space<br />Then: construct approximations of state space <br /> up to depth K using finite automata <br />Try 1: using machine learning and static analysis<br />Try 2: using evolutionary algorithms?<br />Problem 2: infeasible sequences<br />Try 1: constraint solving for path feasibility<br />Try 2: test data generation with metaheuristics<br /> ev3<br />...<br />...<br />...<br /> SBT Opportunities! <br />

49.
[Search-Based Testing Challenges]<br />

50.
Let’s take a look at some of specific challenges for Event-B...<br />More details in:<br />A. Stefanescu, F. Ipate, R. Lefticaru, C. Tudose. <br />Towards Search-Based Testing for Event-B Models. <br />To appear in Proc. of 4th International Workshop on Search-Based Software Testing (SBST), 2011.<br />

51.
No explicit state space<br />Fact<br />Event-B has no explicit states like the EFSMs <br />no control state (as in EFSMs)<br />Problem<br />Large (possibly infinite) state space<br />testing coverage criteria must be defined<br />only recent work addressing SBT for EFSMs<br />Possible ideas:<br />coverage of all events (or a given subset of them) or coverage of all test paths of length < K<br />many other coverages possible, so industrial guidance is needed<br />consider the class of Event-B models with a special state variable (see industrial use cases from SAP, SSF, Bosch and UML-B models) <br />

52.
Non-numerical types<br />Fact<br />Event-B is based on set theory<br />set relations, powersets, functions, set comprehensions, products, records, etc.<br />Complex structured data (e.g. business domain)<br />Problem<br />fitness functions in literature mostly defined for numerical types<br />Possible solutions<br />design new fitness functions for set-based (non-numerical) types<br />efficient encoding of mixed non-numerical/numerical test data<br />

53.
Hierarchical models<br />Fact<br />Event-B supports different types of hierarchy<br />refinement from abstract to concrete levels<br />model decomposition<br />modularity <br />most industrial models use some sort of hierarchy (due to size) <br />Problem<br />no much previous work on SBT addressing hierarchical models<br />Possible ideas:<br />adapt existing work on test selection for hierarchical state machines<br />use the existing ProB model checker that can partially deal with hierarchy<br />

54.
Non-determinism<br />Fact<br />Event-B has different types of non-determinism<br />:| or :∈ operators (e.g. x :∈ {item1, ..., item20})<br />non-deterministic choice of the event to be executed when several enabled<br />non-deterministic choice of parameters (ANY construct) <br />non-deterministic initialisation of variables satisfying the set of invariants<br />Problem<br />no much previous work on SBT addressing non-deterministism<br />Possible ideas:<br />devise fitness functions that improve the chance of choosing a given path in a non-deterministic model (under certain assumptions)<br />make the non-determinism visible (model instrumentation)<br />

55.
[Test Data Generation]<br />

56.
Generating test data for a path<br />Problem<br />Given one path of events, provide the test data (event parameters) that enables the execution of the path.<br />Approach<br />genetic algorithms<br />encoding of sets into binary genes<br />mixed choromosomes (numerical and binary genes)<br />More details in:<br />I. Dinca, A. Stefanescu, F. Ipate, R. Lefticaru, C. Tudose. <br />Test Data Generation for Event-B Models using Genetic Algorithms. <br />In Proc. of 2nd International Conference on Software Engineering and Computer Systems (ICSECS'11). CCIS Series, vol. 181, pp. 76-90, Springer, 2011.<br />

57.
Test data generation with genetic algorithms<br />Event-B model<br />Encoding of variables<br />…<br />0<br />1<br />1<br />0<br />1<br />0<br />0<br />1<br />1<br />0<br />1<br />0<br />1<br />0<br />1<br />0<br />0<br />1<br />0<br />1<br />1<br />0<br />0<br />1<br />1<br />1<br />1<br />0<br />1<br />items<br />ITEMS := CONTEXT {{it1}, {it34}, {it36}, {it67}, {it89}, {it11}, {it354}, {it876},{it321}, {it333}, {it78}, {it787}, {it7878}, {it2342}, {it3453}, {it6786}, {it1232}, {it7765}, {it7098})<br />items : Powerset(ITEMS)<br />“Chromosome”<br />Crossover<br />0<br />Selection<br />Mutation<br />Fitness evaluation<br />End?<br />Fitness functions<br />Simulator (ProB)<br />

58.
Fitness functions for one path<br />fitness := approach level + normalized branch level<br />New objective functions for set types<br />Classical Tracey’s objective for numerical types<br />

59.
Examples from the benchmark<br />

60.
Statistical results<br />Statistical comparison of<br /><ul><li> Genetic Algorithms (GA) and

61.
Random Testing (RT)

62.
on 18 paths covering 5 Event-B models

63.
using statistical test like t-test and U-test</li></ul>And (of course) the winner is:<br /><ul><li>GA performs significantly better than RT on most paths </li></ul>Note: We are currently evaluating constraint-solving (mature for Event-B). It It seems to be quicker for small to medium path (with exceptions).<br />

64.
[Finite Model Learning]<br />

65.
Generating finite models from Event-B<br />Problem<br />There is no explicit state space of an Event-B model<br />Approach<br />Finite automata learning (adapted L* algorithm )<br />Aproximation through cover automata<br />K-bound on the length of executions<br />Use finite automata for conformance test generation<br />More details in:<br />F. Ipate, I. Dinca, A. Stefanescu:<br />Model Learning and Test Generation for Event-B using Cover Automata. <br />Submitted to SEFM’11.<br />

66.
First experiments<br />SBT Opportunity?! <br />Preliminary approach<br />Approximation through cover automata for bound l<br />Incremental -> fits very well with model refinements<br />Minimal finite automata<br /> Sometimes difficult to find counterexamples (to the approximation)<br />Scales for medium size models:<br />Boundl<br />ev3<br />...<br />...<br />...<br />

67.
[Conclusions]<br />

68.
Opportunities for Search-Based Techniques<br />To wrap-up opportunities:<br /><ul><li>Test suite minimisation with multi-objective optimisation

69.
Test data generation for one path with search-based algorithms

70.
Construct finite models with evolutionary algorithms

71.
Combine ProB model-checker with meta-heuristics

72.
Combine ProB constraint-solver with meta-heuristics

73.
Experiment with different search algorithms (PSO, ACO, SA,...)</li></ul>To be answered until end of project (April 2012): <br />Which of the above work good in practice?<br />