SoftTest Ireland: Model Based Testing - January 27th 2011


Published on

Visit and sign up for Irish Software Testing event updates.

These are the slides from our Model Based Testing talk in IBEC.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Although several definitions exists, most of them carry the same basic idea about an model describing a system of some sort and some way of using that model to get test cases (or rather test sequences).
  • The model describes the functionality of a system, or rather the functionality we want to test. You see the possible paths through the system visualized by arrows (aka edges) and nodes (aka vertexes). Traversing the model is quite simple; do something in an edge and verify that your actions turn out right in the following vertex, and on to the next one…. One beauty of this approach is that the code will be fairly small functions of “do:ing”, that would be Edges, and mainly “verifying”, Vertexes. I say mainly because one usually need to do some things to be able to verify, like open a customer dialog and search for latest transaction or create a database connection to check a certain post in a datatabel etc.
  • That is an interesting question! And a potential trap! Instinct will tell you not to model to “ shallow”, an extreme case would look like Edge_StartGoogle, Vertex_GoogleHome, Edge_NavToImagesAndVideosAndLoginGMailWithValidAndInvalidDataAndNavToHome and finally Vertex_ValidateAllWeJustDidt! Such a model will not benefit from anything. The data you want to verify is very far back and even more serious is the total lack of possible, “random”, paths through the system The opposite is not too good either, by cutting all and everything into small pieces you will end up with a too big model representing fairly little functionality compared to its size and loosing the overview benefits of the model. OK, so what is the appropriate medium way? I’d say use your knowledge, experience and gut feeling and fine-tune from that. Remember that you haven’t scripted anything yet so changes are fast and cheap!
  • You will be one that discover the misalignment of understanding between developers, testers, SME:s, product owners and the alike, and you didn’t even start to automate any tests yet! Now, that is proactive testing! (couldn’t we get this benefit with “classic” automation? Sure, just try getting the parties mentioned to read your test cases….compared to browsing through a model on the and informally discuss the functionality. I’ll bet even your boss will glance at the model and nod in pseudo-understanding….)
  • No UML GraphWalker  own ruleset in conjunction with  GraphML , is easier to get started with than UML. As testers, we do not need all functionality that UML has to offer. No exit/stop points The idea behind this, is that we want long, unpredictable test sequences. We do not want to walk the same path every time we execute a test. We want variation, spiced with randomness. This will create a better 'test coverage' of the system under test. The way to tell  GraphWalker  to stop generating test sequences are done by means of Stop Criterias, passed as arguments to the tool. Online GraphWalker  supports online test sequence generation. Using the online mode, the tool is capable of  testing non-deterministic systems. In essence, this means that the path walked through the model is decided runtime, during the actual test execution. This is very helpful if your test execution tool needs to communicate with the model during the test. Event-driven GraphWalker  will support (from version 2.5.1-SNAPSHOT) the possibility to switch model caused by an event. For example, let's say we have a model that executes the navigation of the GUI of a mobile phone. At any point in that execution, an incoming call will be such an event that will switch from navigating the GUI, to a model that handles the call.
  • SoftTest Ireland: Model Based Testing - January 27th 2011

    1. 1. Model-based Testing Introduction and Hands-on session Stefan Ekman Dublin, 2011-01-27
    2. 2. Model Based Testing What is Model-based Testing?
    3. 3. What is MBT? <ul><li>Wikipedia: </li></ul><ul><li>“ Model-based testing is software testing in which test </li></ul><ul><li>cases are derived in whole or in part from a model that </li></ul><ul><li>describes some (usually functional) aspects of the system </li></ul><ul><li>under test (SUT).” </li></ul>
    4. 4. Advantages <ul><li>Model-based testing is easy to understand from both the business and developer communities </li></ul><ul><li>Model-based testing separates (business-) logic from testing code </li></ul><ul><li>Model-based testing increases the test coverage with the same effort as “Classic” test automation </li></ul><ul><li>Model-based testing is the fastest way to get use of automated test </li></ul>
    5. 5. Advantages <ul><li>Model-based testing enables us to switch testing tool if needed or support multiple platforms using the same model </li></ul><ul><li>Model-based testing focuses on requirement coverage, not how many test cases you executed last week etc. </li></ul><ul><li>Model-based testing proved to positively affect the maintenance problem, the nemesis of all test automation. </li></ul>
    6. 6. The Model Modelling your tests
    7. 7. Tools / Environment . GraphML Model with yEd GraphWalker MBT engine Selenium Java or SOAP QTP Via SOAP Hudson Build system
    8. 9. Code behind <ul><li>public void e_Nav_GoogleImages() </li></ul><ul><li>{ </li></ul><ul><li>log .info(&quot;Edge: e_Nav_GoogleImages&quot;); </li></ul><ul><li>selenium .click( “link=Images&quot; ); </li></ul><ul><li>selenium .waitForPageToLoad(30000); </li></ul><ul><li>} </li></ul>
    9. 10. Modelling <ul><li>What should the model represent? </li></ul><ul><li>I suggest that you start with the model representing the end usage/user functionality of the system. That will usually resemble your traditional functional test case approach. There are several other approaches for you to explore later in you new MBT career… </li></ul>
    10. 11. Modelling <ul><li>To what level do we model? </li></ul>
    11. 12. Modelling <ul><li>Who creates the model? </li></ul><ul><li>Start out yourself to begin with </li></ul><ul><li>Present the model – show it to SME’s and developers for input </li></ul><ul><li>If the system is non-existing, you probably would consider user stories, product backlog items, use-cases or the equivalent requirement's information. </li></ul>
    12. 13. Modelling <ul><li>During this process you will collect the first benefits of working with MBT! </li></ul>
    13. 14. GraphWalker <ul><li>GraphWalker is a tool for generating offline and online test sequences from Finite State Machines and Extended Finite State Machines. </li></ul><ul><li> </li></ul><ul><li>Lightweight MBT – No UML </li></ul><ul><li>No exit/stop points </li></ul><ul><li>Online </li></ul><ul><li>Event-driven </li></ul><ul><li>GraphWalker is an Open Source MBT engine created by Kristian Karl and are stable and used in large IT testing environment. </li></ul>
    14. 15. GraphWalker - Stop Criteria's <ul><li>REACHED_EDGE </li></ul><ul><li>REACHED_VERTEX </li></ul><ul><li>EDGE_COVERAGE </li></ul><ul><li>VERTEX_COVERAGE </li></ul><ul><li>TEST_LENGTH </li></ul><ul><li>TEST_DURATION </li></ul><ul><li>REACHED_REQUIREMENT </li></ul><ul><li>REQUIEMENT_COVERAGE </li></ul><ul><li>NEVER </li></ul>
    15. 16. GraphWalker – XML configuration <ul><li><?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?> </li></ul><ul><li><!DOCTYPE MBTINIT SYSTEM &quot;mbt_setup.dtd&quot; > </li></ul><ul><li><MBTINIT EXECUTOR=&quot;java:com.paddypower.mbt.BetPlacementRegression&quot; EXTENDED=&quot;true&quot; GUI=“true&quot;> </li></ul><ul><li><MODEL PATH=&quot;model/BetPlacementRegression.graphml&quot;/> </li></ul><ul><li><CLASS PATH=&quot;JavaPPautomationbin&quot;/> </li></ul><ul><li><CLASS PATH=&quot;.servlet-2.3.jar&quot;/> </li></ul><ul><li><CLASS PATH=&quot;seleniumselenium-java-client-driver.jar&quot;/> </li></ul><ul><li><CLASS PATH=&quot;seleniumselenium-server-2.0a6.jar&quot;/> </li></ul><ul><li><SCRIPT> </li></ul><ul><li>PortNumber = 5555; </li></ul><ul><li>DBconnectionString = &quot;jdbc:informix-sqli://;user=user;password=12345&quot;; </li></ul><ul><li>MAX_SELECTIONS = 10; </li></ul><ul><li></SCRIPT> </li></ul><ul><li><GENERATOR TYPE=&quot;RANDOM&quot;> </li></ul><ul><li><CONDITION TYPE=&quot;REACHED_VERTEX&quot; VALUE=&quot;v_Done&quot;/> </li></ul><ul><li></GENERATOR> </li></ul><ul><li></MBTINIT> </li></ul>
    16. 17. GraphWalker - The Walks <ul><li>RANDOM </li></ul><ul><li>A_STAR Will try to generate the shortest possible test </li></ul><ul><li>sequence through a model with given stop </li></ul><ul><li>condition. The algorithm only works well on </li></ul><ul><li>smaller models. </li></ul><ul><li>SHORTEST_NON_OPTIMIZED </li></ul>
    17. 18. GraphWalker - Edge <ul><li>An edge is a transition. Could be a click on a button, a timeout, a server API call. </li></ul><ul><li>An edge can have a label, but it is not mandatory. </li></ul><ul><li>• The label of an edge will map against a method, function or sub routine during test execution. </li></ul><ul><li>As a rule of thumb, the best practice is to start the name with 'e_'. The reason for this, is that it is easier to recognize the function in the test execution tool, as an edge. For example: e_StartApplication e_EnterBasicView </li></ul><ul><li>The same name of an edge, can be re-used in the model. </li></ul>
    18. 19. GraphWalker - Vertex <ul><li>A vertex is some kind of state that is possible to verify using some kind of oracle. </li></ul><ul><li>A vertex must always have a label. </li></ul><ul><li>The label of a vertex is mapped against a method, function or sub routine during test execution. </li></ul><ul><li>As a rule of thumb, the best practice is to start the name with 'v_'. The reason for this, is that it is easier to recognize the function in the test execution tool, as a vertex. For example: v_ApplicationStarted v_BasicView </li></ul><ul><li>The same name of a vertex, can be re-used in the model. </li></ul>
    19. 20. GraphWalker - Labels <ul><li>Label [cond. expr.] / Statement1;Statement2; </li></ul><ul><li>White spaces not allowed. </li></ul><ul><li>Follow the naming convention for the implementing programming </li></ul><ul><li>language. </li></ul><ul><li>Example: </li></ul><ul><li>e_StartApplication </li></ul><ul><li>Example: </li></ul><ul><li>e_EnterBasicView </li></ul>
    20. 21. GraphWalker - Guards <ul><li>Label [cond.expr.] / Statement1;Statement2; </li></ul><ul><li>Conditional expression that returns a Boolean value. </li></ul><ul><li>The guard indicates if the edge is accessible in the current state of the </li></ul><ul><li>machine. The language is either Java or JavaScript. </li></ul><ul><li>Example: </li></ul><ul><li>e_Evaluate [x>0 && y<=10] </li></ul><ul><li>Example: </li></ul><ul><li>e_Checkout [shoppingBasket.contains(&quot;Cerials&quot;)] </li></ul>
    21. 22. GraphWalker - Statements <ul><li>Label [cond.expr.] / Statement1;Statement2; </li></ul><ul><li>Any applicable Java or JavaScript statements </li></ul><ul><li>Example: </li></ul><ul><li>e_Calculate / x=5;y++; </li></ul><ul><li>Example: </li></ul><ul><li>e_AddCerials / shoppingBasket.add(&quot;Cerials&quot;); </li></ul>
    22. 23. GraphWalker – Keyword Start <ul><li>Used by Vertices only. </li></ul><ul><li>Each graph must have a start vertex, and that vertex holds this </li></ul><ul><li>keyword. There is only one vertex in a graph, holding this keyword. </li></ul><ul><li>The Start vertex can only have one out-edge. </li></ul><ul><li>If the graph is the main-graph, that edge must have a </li></ul><ul><li>label. </li></ul><ul><li>If the graph is a sub-graph, that edge cannot have a label. </li></ul>
    23. 24. GraphWalker - REQTAG <ul><li>Used by Vertices </li></ul><ul><li>Used by GraphWalker for keeping track of which </li></ul><ul><li>requirements are fulfilled during testing. </li></ul><ul><li>In the example above, if v_BookInfomation is passed, it </li></ul><ul><li>would mean the requirement with tag UC01 2.2.3 has </li></ul><ul><li>passed. </li></ul>v_BookInfomation REQTAG=UC01 2.2.3
    24. 25. GraphWalker - Weight <ul><li>Used by Edges only. </li></ul><ul><li>Used by MBT during random walks during test sequence generation. It holds a real value between 0 and 1, and represents the probability that a specific edge should be chosen. A value of 0.05, would mean a 5% chance of that edge to be selected during a run. </li></ul><ul><li>Note: It only works with the RANDOM generator </li></ul>v_BookInfomation REQTAG=UC01 2.2.3 e_NavA weight=0.25 e_NavB
    25. 26. GraphWalker – Keyword Blocked <ul><li>Used by both Vertices and Edges. </li></ul><ul><li>A vertex or an edge with the key word BLOCKED, will be </li></ul><ul><li>excluded from the model when walked. </li></ul>v_BookInfomation BLOCKED
    26. 27. Metrics <ul><li>Standard test metrics does not apply so well. </li></ul><ul><li>Test cases are not meaningful when running online tests. </li></ul><ul><li>MBT tends to drive the requirement work. </li></ul>
    27. 28. GraphWalker – QTP & SOAP <ul><li>WebService(&quot;SoapServicesService&quot;).SetTOProperty &quot;WSDL&quot;, &quot;http://myComputerName:9090/mbt-services?WSDL&quot; If ( not WebService(&quot;SoapServicesService&quot;).Reload()  )Then     Reporter.ReportEvent micFail, &quot;MBT failure&quot;, &quot;MBT encountered an error. See the MBT log files for information.&quot;     ExitTest End If Do until not WebService(&quot;SoapServicesService&quot;).HasNextStep()     action = WebService(&quot;SoapServicesService&quot;).GetNextStep()     If len(action) > 0 Then         If not Eval( action ) Then             Reporter.ReportEvent micFail, &quot;Script failure&quot;, &quot;The script encountered an error when trying to run function: &quot; & action             ExitTest         End if     End If Loop Reporter.ReportEvent micDone, &quot;MBT Statistics&quot;, WebService(&quot;SoapServicesService&quot;).GetStatistics() </li></ul>
    28. 29. 5 min break.. <ul><li>I am not in the office at the moment. Please send any work to be translated. </li></ul>
    29. 30. Hands on session <ul><li>Setup the environment </li></ul><ul><li>Create a model </li></ul><ul><li>Generate code from model </li></ul><ul><li>Automate with selenium </li></ul><ul><li>Create XML configuration file for GraphWalker </li></ul><ul><li>Run the model! </li></ul>
    30. 31. Files <ul><li>1. Create directory MBT i.e C:MBT </li></ul><ul><li>2. Create directory Models under in the MBT folder </li></ul><ul><li>3. Copy GraphWalker and Selenium jar files to the directory selenium-java-client-driver.jar selenium-server.jar graphwalker-2.5.1-SNAPSHOT-standalone.jar </li></ul>
    31. 32. Model <ul><li>Visit with Firefox </li></ul><ul><li>Record with Selenium IDE by clicking on all elements and as well right clicking the result </li></ul><ul><li>Switch off recording. </li></ul><ul><li>Options > Format > JUnit4 (Java) </li></ul>
    32. 33. Model <ul><li>Start yEd to create the model </li></ul>
    33. 34. Generate the Java Code <ul><li>/** </li></ul><ul><li>* This method implements the {EDGE_VERTEX} '{LABEL}' </li></ul><ul><li>*/ </li></ul><ul><li>public void {LABEL}() </li></ul><ul><li>{ </li></ul><ul><li> &quot;{EDGE_VERTEX}: {LABEL}&quot; ); </li></ul><ul><li>throw new RuntimeException( &quot;The {EDGE_VERTEX}: {LABEL} is not implemented yet!&quot; ); </li></ul><ul><li>} </li></ul><ul><li>java -jar graphwalker-2.5.1-SNAPSHOT-standalone.jar source -f Models/mbttest.graphml -t java.template.txt > javacode.txt </li></ul>
    34. 35. Eclipse – Create the Java project <ul><li>Create new Java Project; MbtTest </li></ul><ul><li>Create new Java Package; ie.mbt.test </li></ul><ul><li>Create new Class; MBTtest </li></ul><ul><li>Copy paste the contents of the generated Java code from the model into the created class. </li></ul>
    35. 36. The java code <ul><li>Include references to Selenium and GraphWalker in your project. Right click on your project and choose Properties. Under Java Build Path choose Libraries and then click Add External JARs.. Choose the two jar files: </li></ul><ul><li>graphwalker-2.5.1-SNAPSHOT-standalone.jar </li></ul><ul><li>selenium-java-client-driver.jar </li></ul><ul><li>private HttpCommandProcessor proc = new HttpCommandProcessor(&quot;localhost&quot;,4444, &quot;*chrome&quot;, “”); </li></ul><ul><li>private Selenium selenium = new DefaultSelenium(proc); </li></ul><ul><li>private Logger log = Util.setupLogger(MBTtest.class); </li></ul><ul><li>// import org.apache.log4j.Logger; </li></ul><ul><li>// import org.graphwalker.Util; </li></ul>
    36. 37. Writing the Java Code <ul><li>In e_init we start the selenium object. </li></ul><ul><li>selenium.start(); </li></ul><ul><li>;;); </li></ul><ul><li>Delete the Throw exception! </li></ul><ul><li>Update all e_r11, e_r12 etc </li></ul><ul><li>;Radio1_1&quot;); </li></ul><ul><li>row1 = Integer. parseInt (selenium.getText(“r11&quot;)); </li></ul>
    37. 38. Writing the Java Code <ul><li>Create the utility GetResult </li></ul><ul><li>public boolean GetResult() </li></ul><ul><li>{ </li></ul><ul><li>int rowTotal = row1 + row2 + row3; </li></ul><ul><li>int calcTotal = Integer. parseInt (selenium.getText(&quot;totalDiv&quot;)); </li></ul><ul><li>if (rowTotal==calcTotal) </li></ul><ul><li>return true ; </li></ul><ul><li>else </li></ul><ul><li>return false ; </li></ul><ul><li>} </li></ul><ul><li>IMPORTANT! </li></ul><ul><li>Compile you class after any changes, CTRL + B </li></ul>
    38. 39. GraphWalker XML <ul><li><?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?> </li></ul><ul><li><!DOCTYPE MBTINIT SYSTEM &quot;mbt_setup.dtd&quot; > </li></ul><ul><li><MBTINIT EXECUTOR=&quot;java:ie.mbt.test.MBTtest&quot; EXTENDED=&quot;true&quot;> </li></ul><ul><li><MODEL PATH=&quot;models/MBTTest.graphml&quot;/> </li></ul><ul><li><CLASS PATH=&quot;MbtTestbin&quot;/> </li></ul><ul><li><CLASS PATH=&quot;selenium-java-client-driver.jar&quot;/> </li></ul><ul><li><CLASS PATH=&quot;selenium-server.jar&quot;/> </li></ul><ul><li><GENERATOR TYPE=&quot;RANDOM&quot;> </li></ul><ul><li><CONDITION TYPE=&quot;TEST_LENGTH&quot; VALUE=&quot;1000&quot;/> </li></ul><ul><li></GENERATOR> </li></ul><ul><li></MBTINIT> </li></ul>
    39. 40. Selenium Server <ul><li>java -jar selenium-server.jar </li></ul><ul><li>Will start with default settings on port 4444 </li></ul>
    40. 41. Start the scripts.. <ul><li>Command line to start GraphWalker with the correct settings: </li></ul><ul><li>java -jar graphwalker-2.5.1-SNAPSHOT-standalone.jar gui xml -f MBTTest.xml </li></ul>
    41. 42. Questions? <ul><li>More info at: </li></ul><ul><li> </li></ul><ul><li> </li></ul><ul><li> </li></ul><ul><li>Email: </li></ul>
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.