Software Architecture-based Testing and Model-checking -

891 views
853 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
891
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
34
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Software Architecture-based Testing and Model-checking -

  1. 1. Software Architecture-based Testing and Model-checking - ECI 2005, University of Buenos Aires - Course Web-site: [www.di.univaq.it/muccini/ECI05/] Lecture 5: Model-Checking driven Testing Lecturer: Henry Muccini Assistant Professor, Computer Science Department University of L'Aquila - Italy muccini@di.univaq.it [www.di.univaq.it/muccini] – [www.HenryMuccini.com] Copyright Notice » The material in these slides may be freely reproduced and distributed, partially or totally, as far as an explicit reference or acknowledge to the material author is preserved. Henry Muccini SEA Group 2 © 2005 by H. Muccini / ECI 2005 Course Acknowledgment » This work is joined with Patrizio Pelliccione (University of L’Aquila), Pierluigi Pierini (Siemens CNX), and Antonio Bucchiarone (ISTI – CNR) » Published in ITM 2004, Lecture Notes in Computer Science, LNCS, vol. 3236, pp. 351 - 365 (2004). SEA Group 3 © 2005 by H. Muccini / ECI 2005 Course 1
  2. 2. Agenda » Introduction and Motivations: - ModTest and TeStor » ModTest in Siemens C.N.X. » Challenges and Future Work SEA Group 4 © 2005 by H. Muccini / ECI 2005 Course Considerations » Different Analysis require different notations - testing, model-checking, performance and reliability analysis require specific formalisms and annotations on UML-based models [UML&SA04] - There is a huge relationship between what we specify and what kind of analysis we may perform > Modeling for documenting vs. Modeling for analysis …But… SEA Group 5 © 2005 by H. Muccini / ECI 2005 Course Considerations » different Analysis techniques are usually related - supposing an industry is interested in deadlock analysis and performance analysis, a complete result is obtained only using two different ADLs. » Modeling is really expensive - we want to reuse the same model for many analysis techniques SEA Group 6 © 2005 by H. Muccini / ECI 2005 Course 2
  3. 3. Integrating Analysis Techniques… validate the SA model conformance with respect to selected functional properties Charmy Project [www.di.univaq.it/charmy] provide confidence on the implementation fulfillment to its architectural spec SA-based Testing SEA Group 7 © 2005 by H. Muccini / ECI 2005 Course ModTest: Model-Checking driven Testing [ITM04, CBSE05,TR_March05] SEA Group 8 © 2005 by H. Muccini / ECI 2005 Course Software Model Checking and Software Testing » Model Checking: - It checks whether a certain property is valid for a certain model of a system [Ruys_PhDThesis] > Model checking is a model-based, automatic technique that, given a finite-state model M of a system and a property P, checks the validity of P in M » Testing: - “Software testing consists of the dynamic verification of the behavior of a program on a finite set of test cases, suitably selected from the usually infinite executions domain, against the specified expected behavior” [Bertolino_SWEBOK] SEA Group 9 © 2005 by H. Muccini / ECI 2005 Course 3
  4. 4. Advantages Model Checking Testing exhaustive approach to clever selection of limited and completely check the system relevant test cases - completely automated - usually left to the tester experience skills on formal methods skills on formal methods generally not required state explosion problem test case identification problem only model-based code-based, model-based, specification-based SEA Group 10 © 2005 by H. Muccini / ECI 2005 Course Goals and Motivations > 1/2 » General Goal: - integration of model-checking and testing to model- provide an useful tool to test modern complex software systems » In related approaches: - By using model-checking features, counter-examples are produced, successively used to derive test cases - Main Limitations: > P1 : due to models complexity, the model checker techniques become inapplicable, thus not allowing to identify test cases; > P2 : even on little examples, the number of generated test cases causes the intractability SEA Group 11 © 2005 by H. Muccini / ECI 2005 Course Goals and Motivations > 2/2 » Our Goal: To apply Model-checking and Testing in a Software Architecture-based (SA) process, where: > Model-checking techniques are used to validate the SA model conformance with respect to selected functional properties + avoiding state explosion problem > while testing techniques are used to provide confidence on the implementation fulfillment to its architectural specification + Test case selection driven by model-checking SEA Group 12 © 2005 by H. Muccini / ECI 2005 Course 4
  5. 5. Our Proposal validate the SA model conformance with respect to selected functional properties Charmy provide confidence on the implementation fulfillment to its architectural spec SEA Group SA-based Testing 13 © 2005 by H. Muccini / ECI 2005 Course SA-based model-checking Charmy Project [www.di.univaq.it/charmy] Property validate the SA model expressed through FMS Model of conformance to scenarios the SA selected functional properties » Model-checking: > It checks whether a certain property is valid for a certain FSM model of a system [Ruys_PhDThesis] SEA Group 14 © 2005 by H. Muccini / ECI 2005 Course Charmy: Informally Requirements Software Architecture Translate them into Buchi Property Automata M.C. SPIN Translate them True or False SEA Group into Promela 15 © 2005 by H. Muccini / ECI 2005 Course 5
  6. 6. SA-based Testing SA-based Testing [ICSE01, FASE04,TSE04] » Goal: - Provide confidence on the implementation fulfillment to its architectural specification SEA Group 16 © 2005 by H. Muccini / ECI 2005 Course SA-based Testing in ModTest SEA Group 17 © 2005 by H. Muccini / ECI 2005 Course Question » How can we integrate both techniques? - Rephrasing: how can we use model-checking results for generating test specifications? » Usually: - By focussing on model-checking driven testing, counter-examples are produced and successively used to derive test cases » Our idea: - To use the properties to derive test specifications, by recovering missing information from state machines > In this way, the SA is validated with respect to requirements and the SEA Group implementation conformance to the SA is tested 18 © 2005 by H. Muccini / ECI 2005 Course - SA-based analysis process 6
  7. 7. The TeStor Algorithm » TeStor: - Inputs: > sequence diagram (inSD) representing properties - test generation directives > components’ behavioral models (in terms of components' state machines), - the model of the software under test - Outputs: > a set of test specifications, still in the form of sequence diagrams (outSD) SEA Group 19 © 2005 by H. Muccini / ECI 2005 Course TeStor objective Model of the system Test directives Test specification m9 m3 m8 m1 SEA Group 20 © 2005 by H. Muccini / ECI 2005 Course TeStor objective Model of the system Test directives Test specification m9 m3 m8 m1 SEA Group 21 © 2005 by H. Muccini / ECI 2005 Course 7
  8. 8. How TeStor can achieve this goal » How TeStor could work: - Parallel composition of the state-machines models, and their traversal (like model-checkers), or - Challenge: to avoid parallel composition > To limit state explosion problems… SEA Group 22 © 2005 by H. Muccini / ECI 2005 Course The Algorithm » The TeStor algorithm can be split into two macro-steps: - State machines (SM) Linearization > Decomposes SM in a set of linear traces - Test Sequence Generation > Looks at each linearized trace in order to identify the sup-trace of the inSD. > This macro-step is composed by a Validation part, which checks when and how sup-traces need to be combined to produce the outSD. > The Merge algorithm glues together the validated traces SEA Group 23 © 2005 by H. Muccini / ECI 2005 Course Linearization » Starting from the initial state in the components’ state diagrams; - It creates a trace at any time a state with a branch is reached or an already visited state is reached » The algorithm is iterated, starting from the previously reached state, until unvisited states still exist. m8 m8 m1 m7 S0 à S0 S3 à S4à S5à S0 S5 S1 S4 S3 S2 m9 m3 m10 SEA Group m9 S3 à S3 S0 à S1à S2à S3 24 © 2005 by H. Muccini / ECI 2005 Course 8
  9. 9. From State Machines to Traces SEA Group 25 © 2005 by H. Muccini / ECI 2005 Course Test Sequence Generation » Starting from the traces generated from each component state machine - For each message “mi” in the inSD, TeStor identifies those traces which contain it > Validate algorithm C2’s i.s. = S0 C3’s i.s. = S0 Property C3 C2 C1 C4 m9 m3 m10 m8 m1 m7 SEA Property to be verified e) Group S0 –m9> S1 S0 –m9> S0 26 © 2005 by H. Muccini / ECI 2005 CourseS0 --m9-> S1 -- m3->S2 --m10->S3 --m9->S3 Test Sequence Generation » Starting from the traces generated from each component state machine - For each message “mi” in the inSD, TeStor identifies those traces which contain it > Validate algorithm Property > Merge algoritm C3 C2 C1 C4 m9 m3 m10 m8 m1 m7 SEA Property to be verified e) Group 27 © 2005 by H. Muccini / ECI 2005 Course 9
  10. 10. Supporting Tools The TeStor algorithm has been implemented has a plug-in component for Charmy, our validation framework for architectural analysis. SEA Group 28 © 2005 by H. Muccini / ECI 2005 Course ModTest: Model-Checking driven Testing Test directives validate the SA model conformance with respect to selected functional properties Model of the system Charmy [www.di.univaq.it/charmy] TeStor provide confidence on the implementation fulfillment to its architectural spec SA-based Testing SEA Group 29 © 2005 by H. Muccini / ECI 2005 Course ModTest in Siemens C.N.X. SEA Group 30 © 2005 by H. Muccini / ECI 2005 Course 10
  11. 11. Siemens CNX : main research areas » Siemens CNX S.p.a. is a Siemens R&D lab; its mission is the design and development of SDH(1) TLC equipments » relevant research areas: - Formal design methodologies - System and software performance analysis - Test design methodologies - Intelligent agent application - Network Processors - Ethernet first mile - Optics and cristal properties - Electromagnetic compatibility SEA Group 1) SDH Synchronous Digital Hierarchy 31 © 2005 by H. Muccini / ECI 2005 Course Test Design Methodology > objective »Improve the tets design process System System Requirements Requirements System Architecture feedbacks System Tests System Architec ture Design Design Design NOK NOK Model NOK Review Review Check OK OK OK Implementation Modification Test Implelemtation Test Generation Implementation/ Equipment Develop Requests Design Engine Equipment Develop NOK Test Test NOK Exec Exec OK OK System System Release Release SEA Group a b 32 © 2005 by H. Muccini / ECI 2005 Course Case Study > some definitions » A SDH Network Element (NE, i.e. equipment) is modeled using the functional model standardized by ETSI and ITU-T. » The functional model is built around two specific concepts: - “network layering", with a client/server relationship between adjacent layers; - “atomic functions“ (connection, termination and adaptation), to specify the behavior of each layer. » applicative functions should reside on top of a layer providing specific processing on transmitted information » A “virtual network connection” can be established between mate network layers (or atomic/applicative functions) belonging to different NEs by means of transport services offered by the SEA Group underlying layers 33 © 2005 by H. Muccini / ECI 2005 Course 11
  12. 12. Case Study > some definitions SEA Group 34 © 2005 by H. Muccini / ECI 2005 Course Case Study > EOW architecture » The EOW supports a telephone EOW eowSN HS Node link between NEs using dedicated voice channels defined on the CM SDH frame (i.e. the “EOW SubNetwork” [eowSN]); HS » An EOW node consists of: CM HS - A “handset” (HS) that manage the physical phone device; CM - a “conference manager” (CM) that control the handset connection to the EOW subnetwork; SEA Group 35 © 2005 by H. Muccini / ECI 2005 Course Case Study > EOW components Handset (HS) timeout localNumSign1 Congestion Busy CM1 callRequest [cbusy==true] k oo k onHook oo of fH HS1 call1 o nH !lo c a ?c all lN um S eowKeyDigit ig n config onHook offHook Init Config Idle Check localNumSign2 call onHook t im timeout Request eou [cbusy==false] CM2 ?call t e o nH InConference HS2 call2 oo o offHook k w ll be r) eowKeyDigit Ringing ?ca n um igit, S it (d n yDig S ig N Dialing Ke Nu m e ow !loc al localNumSign3 call CM3 Request [(digit==x)&&(number!=1)]/!callRequest(digit,number) HS3 call3 ?callRequest(digit,number) S1 S2 all ue ]/!c eowKeyDigit ? lo = tr =0 ca sy bu it= lN l/c um [dig al /!c Si 1 )] gn = e r! ?eowKeyDigit(digit,number) /c b b ! callR um S1 S2 us S4 e qu & (n y= e st )& fa (dig =x !callRequest(digit,number) ls e it,n u i t= mb ig e r) [( d S3 eowSubnetwork SEA Group Conference Manager (CM) 36 © 2005 by H. Muccini / ECI 2005 Course 12
  13. 13. Case Study > Functional Requirements » EOW Functional Requirements/Properties: A) when an operator makes a call dialling a selective number, the target operator must receive the call. B) it must be possible to enter a busy conference (with the special number-sign key) when a call is already in progress. C) It must be always possible to exit to the conference (cleanly terminate a call). SEA Group 37 © 2005 by H. Muccini / ECI 2005 Course Case Study > Functional Requirements SEA Group 38 © 2005 by H. Muccini / ECI 2005 Course Case Study Results Interactive simulation & Test generation » Simulation without constraint will result in an intractable number of traces; » Simulation conditioned by the given properties; » Up to 36 test traces was extracted; - Most of them are eligible to become test cases; » Test selection focus on some optimization criteria like: - Maximization of system coverage, - Minimization of global number of tests - Minimization of test lenght (i.e. number of steps) SEA Group 39 © 2005 by H. Muccini / ECI 2005 Course 13
  14. 14. Some Considerations Advantages: » Model complexity and the state explosion reduction obtained by: SA- level model chekcing, iterative approach and abstraction ; » Charmy → easy to use, practical, approach to model-checking, hiding the modeling complexity; » interactive simulation → we may identify traces of interest for testing the whole system or just a relevant subsystem. » test specifications are identified from the architectural model (not from requirements) - Easiest alignment between SA and Test specifications; - Easiest control of the design steps and evolution SEA Group 40 © 2005 by H. Muccini / ECI 2005 Course Some Considerations Limitations: » The Test Generator Engine can be automated; its implementation is in progress. » The executable tests implementation from the generated test specifications is not automated yet. We approach this point with the aim to automate also this step. » Models dimension and complexity still remain an issue, even if the iterative approach reduces the state explosion problem. SEA Group 41 © 2005 by H. Muccini / ECI 2005 Course Challenges SEA Group 42 © 2005 by H. Muccini / ECI 2005 Course 14
  15. 15. Challenge: SA-centric Analysis Process SEA Group 43 © 2005 by H. Muccini / ECI 2005 Course Charmy and TeStor [TR_March05] SEA Group 44 © 2005 by H. Muccini / ECI 2005 Course QuARS, ModTest, CowTest and UIT [QoSA05] SEA Group 45 © 2005 by H. Muccini / ECI 2005 Course 15
  16. 16. Future Work SEA Group 46 © 2005 by H. Muccini / ECI 2005 Course Lessons Learned » From this experience we learned integration is possible: - Analysis integration: ModTest > Future work will integrate other analysis techniques » But we also need: - Notation extension: It is possible to extend the same UML- based notation - Tool extension: we can add a plugin implementing the new analysis technique SEA Group 47 © 2005 by H. Muccini / ECI 2005 Course Dually: Putting in Synergy UML 2.0 and ADLs » A framework which i) identifies a core set of architectural elements always required (A0), ii) creates an UML profile for A0, iii) provides extensibility mechanisms to add modeling concepts needed for specific analysis. iv) describes how semantic links mechanisms can be kept between different notations. SEA Group It is impossible to identify a unique Different Analysis require 48 © 2005 by H. Muccini / representing language for ECI 2005 Course SAs different notations 16
  17. 17. Notation Extension: General Idea Model Transformation = = + = = SEA Group 49 © 2005 by H. Muccini / ECI 2005 Course Notation Extension: DUALLY [TR_May05] Identify a core set of Create an UML profile able to model the core architectural elements architectural elements always required previously identified Describe how semantic links Provide extensibility mechanisms can be kept mechanisms to add modeling concepts between different notations needed for specific SEA Group analysis 50 © 2005 by H. Muccini / ECI 2005 Course The roles of ADL, UML and XML A DL UM L XML xArch xArch & ACME DUALLY DUALLY ADLs Profile for XML A0 Experience DUALLY XML modeling extensions extensions SEA Group 51 © 2005 by H. Muccini / ECI 2005 Course 17
  18. 18. Tool Extension: Charmy Studio SEA Group 52 © 2005 by H. Muccini / ECI 2005 Course Wish List: Tool·one: integration of analysis techniques UML/SA profile UML/SA » Integration of for Model- profile for UML/SA profile for Performance Other Checking Testing many analysis Input filters techniques DUALLY DUALLY DUALLY » Integration of standard editing and Profile for A 0 XML many UML- analysis tools based notations Modeling XML extensions extensions » Integration of Plugged Editing into Semantic Relations different DUALLY tools analysis tools Model- checking Analysis notation tools Tes ting Performance feedback notation feedback notation feedback Model- Performance Other Checking tool Testing tool tool SEA Group 53 © 2005 by H. Muccini / ECI 2005 Course 18

×