Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Slides for a talk on search-based testing for Event-B models


Published on

This a talk I gave at the 13th CREST Worshop. See more details at:

Published in: Education
  • Be the first to comment

  • Be the first to like this

Slides for a talk on search-based testing for Event-B models

  1. 1. 13th CREST Open Workshop<br />12th-13th of May 2011, London<br />Search-Based Software Engineering for Model-Based Testing<br />Event-B in a Nutshell<br />Test Generation <br />Approaches<br />SBT Challenges<br />Test Data Generation<br />Finite Model <br />Learning<br />Conclusions<br />Alin Stefanescu - University of Pitesti, Romania<br />*) joint work with colleagues from DEPLOY project<br />
  2. 2. [Event-B in a Nutshell]<br />
  3. 3. Event-B history<br /><ul><li>Jean-Raymond Abrial (1938- )  
  4. 4. Inventor of the Z and B formal methods.
  5. 5. Z – developed in the 70s
  6. 6. B – developed in the 90s, successfully deployed in industry
  7. 7. Event-B – born with the 21st century
  8. 8. Evolution of B for system level specification
  9. 9. Developement supported by French and European projects:</li></ul> FP6 RODIN and FP7 DEPLOY<br />
  10. 10. DEPLOY project (2008-2012) – funded by FP7 <br />DEPLOY :: Industrial deployment of advanced systemengineering methods for high productivity and dependability using formal methods <br /><ul><li>4 industrial partners
  11. 11. Bosch, Siemens, SAP, SSF
  12. 12. 3 industrial service providers
  13. 13. Systerel, ClearSy, Cetic
  14. 14. 7 academic partners
  15. 15. Newcastle, Aabo, Düsseldorf, ETH Zurich,
  16. 16. Southampton, Pitesti, Bucharest</li></ul>http://www.deploy‐<br />
  17. 17. Rodin platform for Event-B<br /><ul><li>Extension of Eclipse IDE (Java-based)
  18. 18. Theorem proving as core technology
  19. 19. Many other Rodin Plug-ins
  20. 20. ProB:animation, consistency and model-checking
  21. 21. Animators (AnimB)
  22. 22. Decomposition
  23. 23. Modularisation
  24. 24. Team-work
  25. 25. Code generation
  26. 26. UML-B
  27. 27. etc.</li></li></ul><li>Event-B in a nutshell<br />Event-B model<br /><ul><li>State-transition model(like ASM, B, Z)
  28. 28. set theory as mathematical language
  29. 29. refinement as basic modeling approach
  30. 30. Contexts
  31. 31. carrier sets (domains)
  32. 32. constants
  33. 33. axioms
  34. 34. Machines
  35. 35. global variables
  36. 36. invariants
  37. 37. events that update the variables
  38. 38. Events
  39. 39. local parameters
  40. 40. guards
  41. 41. actions</li></ul>ITEMS := CONTEXT {{it1}, {it34}, {it36}, {it67}, {it89}, {it11}, {it354}, {it876},{it321}, {it333}, {it78}, {it787}, {it7878}, {it2342}, {it3453}, {it6786}, {it1232}, {it7765}, {it7098})<br />items : Powerset(ITEMS)<br />
  42. 42. [Test Generation Approaches]<br />
  43. 43. Test generation based on Event-B<br />Model-Based Testing (MBT)<br />Model-based testing (MBT) is a newlyintroduced topic in DEPLOY<br />priority topic for industrial partners like SAP<br />challenges due to the sheer size of the state space of real-life scenarios<br />Weinvestigate search-based testing (SBT) techniques for Event-B.<br />
  44. 44. Future MBT plugin in RODIN<br />Event-B model<br />MBT Plug-in<br />MBT <br />Users<br />{ Model-checking }<br />{ Constraint-based }<br />Extra <br />test <br />information<br />{ Search-based }<br />Test cases<br />University of Pitesti and University of Dusseldorf<br />Tool developers:<br />
  45. 45. Test generation from Event-B<br />SBT Opportunity! <br />1. Generate a set of tests<br />(sequence of events with concrete param.)<br />2. Optimize test suite<br />(according to some criteria) – if still needed<br />Event-B model<br />ev2(..),ev5(..),...ev3()<br />ev4(..),ev2(..),...ev4(..)<br />...<br />...<br />...<br />...<br />...<br />...<br />...<br />...<br />ev3(..),ev7(..),...ev5(..)<br />...<br />...<br />...<br />...<br />...<br />ev6(..),ev5(..),...ev8<br />Global variables:<br /> var1, var2, var3, ...<br />Events<br /> ev1(p11,...), <br /> ev2(p21,...), <br /> ev3(p31,...),<br /> ... <br />ev2(..),ev5(..),...ev3()<br />ev4(..),ev2(..),...ev4(..)<br />...<br />ev3(..),ev7(..),...ev5(..)<br />
  46. 46. What is the explicit state space<br />State Space of <br />the Event-B model<br />(3,4,{a,b},...)<br />Abstract machine<br />Event-B model<br />ev3(5)<br />Global variables:<br /> var1, var2, var3, ...<br />Events<br /> ev1(p11,...), <br /> ev2(p21,...), <br /> ev3(p31,...),<br /> ... <br />...<br />...<br />...<br />...<br />States given by the values of global variables<br />Transitions labeled by events with concrete parameters<br />
  47. 47. Test generation from Event-B<br />State Space of <br />the Event-B model<br />(3,4,{a,b},...)<br />Approach 1:<br />Explore the state space using the <br /> ProB model checker<br />state space explosion mainly due to data<br />Try: guide the search<br />ev3(5)<br />...<br />...<br />...<br /> SBT Opportunity! <br />
  48. 48. Test generation from Event-B – part II<br />State Space of <br />the Event-B model<br />(3,4,{a,b},...)<br />Approach 2:<br />Explore state space ignoring the data<br />(i.e. local parameters)<br />Problem 1: still large state space<br />Then: construct approximations of state space <br /> up to depth K using finite automata <br />Try 1: using machine learning and static analysis<br />Try 2: using evolutionary algorithms?<br />Problem 2: infeasible sequences<br />Try 1: constraint solving for path feasibility<br />Try 2: test data generation with metaheuristics<br /> ev3<br />...<br />...<br />...<br /> SBT Opportunities! <br />
  49. 49. [Search-Based Testing Challenges]<br />
  50. 50. Let’s take a look at some of specific challenges for Event-B...<br />More details in:<br />A. Stefanescu, F. Ipate, R. Lefticaru, C. Tudose. <br />Towards Search-Based Testing for Event-B Models. <br />To appear in Proc. of 4th International Workshop on Search-Based Software Testing (SBST), 2011.<br />
  51. 51. No explicit state space<br />Fact<br />Event-B has no explicit states like the EFSMs <br />no control state (as in EFSMs)<br />Problem<br />Large (possibly infinite) state space<br />testing coverage criteria must be defined<br />only recent work addressing SBT for EFSMs<br />Possible ideas:<br />coverage of all events (or a given subset of them) or coverage of all test paths of length < K<br />many other coverages possible, so industrial guidance is needed<br />consider the class of Event-B models with a special state variable (see industrial use cases from SAP, SSF, Bosch and UML-B models) <br />
  52. 52. Non-numerical types<br />Fact<br />Event-B is based on set theory<br />set relations, powersets, functions, set comprehensions, products, records, etc.<br />Complex structured data (e.g. business domain)<br />Problem<br />fitness functions in literature mostly defined for numerical types<br />Possible solutions<br />design new fitness functions for set-based (non-numerical) types<br />efficient encoding of mixed non-numerical/numerical test data<br />
  53. 53. Hierarchical models<br />Fact<br />Event-B supports different types of hierarchy<br />refinement from abstract to concrete levels<br />model decomposition<br />modularity <br />most industrial models use some sort of hierarchy (due to size) <br />Problem<br />no much previous work on SBT addressing hierarchical models<br />Possible ideas:<br />adapt existing work on test selection for hierarchical state machines<br />use the existing ProB model checker that can partially deal with hierarchy<br />
  54. 54. Non-determinism<br />Fact<br />Event-B has different types of non-determinism<br />:| or :∈ operators (e.g. x :∈ {item1, ..., item20})<br />non-deterministic choice of the event to be executed when several enabled<br />non-deterministic choice of parameters (ANY construct) <br />non-deterministic initialisation of variables satisfying the set of invariants<br />Problem<br />no much previous work on SBT addressing non-deterministism<br />Possible ideas:<br />devise fitness functions that improve the chance of choosing a given path in a non-deterministic model (under certain assumptions)<br />make the non-determinism visible (model instrumentation)<br />
  55. 55. [Test Data Generation]<br />
  56. 56. Generating test data for a path<br />Problem<br />Given one path of events, provide the test data (event parameters) that enables the execution of the path.<br />Approach<br />genetic algorithms<br />encoding of sets into binary genes<br />mixed choromosomes (numerical and binary genes)<br />More details in:<br />I. Dinca, A. Stefanescu, F. Ipate, R. Lefticaru, C. Tudose. <br />Test Data Generation for Event-B Models using Genetic Algorithms. <br />In Proc. of 2nd International Conference on Software Engineering and Computer Systems (ICSECS'11). CCIS Series, vol. 181, pp. 76-90, Springer, 2011.<br />
  57. 57. Test data generation with genetic algorithms<br />Event-B model<br />Encoding of variables<br />…<br />0<br />1<br />1<br />0<br />1<br />0<br />0<br />1<br />1<br />0<br />1<br />0<br />1<br />0<br />1<br />0<br />0<br />1<br />0<br />1<br />1<br />0<br />0<br />1<br />1<br />1<br />1<br />0<br />1<br />items<br />ITEMS := CONTEXT {{it1}, {it34}, {it36}, {it67}, {it89}, {it11}, {it354}, {it876},{it321}, {it333}, {it78}, {it787}, {it7878}, {it2342}, {it3453}, {it6786}, {it1232}, {it7765}, {it7098})<br />items : Powerset(ITEMS)<br />“Chromosome”<br />Crossover<br />0<br />Selection<br />Mutation<br />Fitness evaluation<br />End?<br />Fitness functions<br />Simulator (ProB)<br />
  58. 58. Fitness functions for one path<br />fitness := approach level + normalized branch level<br />New objective functions for set types<br />Classical Tracey’s objective for numerical types<br />
  59. 59. Examples from the benchmark<br />
  60. 60. Statistical results<br />Statistical comparison of<br /><ul><li> Genetic Algorithms (GA) and
  61. 61. Random Testing (RT)
  62. 62. on 18 paths covering 5 Event-B models
  63. 63. using statistical test like t-test and U-test</li></ul>And (of course) the winner is:<br /><ul><li>GA performs significantly better than RT on most paths </li></ul>Note: We are currently evaluating constraint-solving (mature for Event-B). It It seems to be quicker for small to medium path (with exceptions).<br />
  64. 64. [Finite Model Learning]<br />
  65. 65. Generating finite models from Event-B<br />Problem<br />There is no explicit state space of an Event-B model<br />Approach<br />Finite automata learning (adapted L* algorithm )<br />Aproximation through cover automata<br />K-bound on the length of executions<br />Use finite automata for conformance test generation<br />More details in:<br />F. Ipate, I. Dinca, A. Stefanescu:<br />Model Learning and Test Generation for Event-B using Cover Automata. <br />Submitted to SEFM’11.<br />
  66. 66. First experiments<br />SBT Opportunity?! <br />Preliminary approach<br />Approximation through cover automata for bound l<br />Incremental -> fits very well with model refinements<br />Minimal finite automata<br /> Sometimes difficult to find counterexamples (to the approximation)<br />Scales for medium size models:<br />Boundl<br />ev3<br />...<br />...<br />...<br />
  67. 67. [Conclusions]<br />
  68. 68. Opportunities for Search-Based Techniques<br />To wrap-up opportunities:<br /><ul><li>Test suite minimisation with multi-objective optimisation
  69. 69. Test data generation for one path with search-based algorithms
  70. 70. Construct finite models with evolutionary algorithms
  71. 71. Combine ProB model-checker with meta-heuristics
  72. 72. Combine ProB constraint-solver with meta-heuristics
  73. 73. Experiment with different search algorithms (PSO, ACO, SA,...)</li></ul>To be answered until end of project (April 2012): <br />Which of the above work good in practice?<br />