Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Agile Testing Practices

5,010 views

Published on

A look at different testing approachs for testing applications within Agile teams

Agile Testing Practices

  1. 1. Leveraging EmergingTechnologies in Agile Teams … and some testing tools you should try at home Paul King Director, ASERT @paulk_asert paulk@asert.com.au
  2. 2. Topics  Introduction • Using testing DSLs • All combinations and all pairs • Auto generated tests© ASERT 2006-2011 • Model-based testing • Business/logic rules • Example driven testing • Further information AgileAust 2011 - 2
  3. 3. Emerging Technology Areas • Development • Deployment & Mgmt – Continuous integration – Continuous deployment – Convention over – Build optimisation configuration – Cloud computing – Language evolution – Framework Support Agile© ASERT 2006-2011 evolution via improved – Concurrency Management of – Configuration Complexity management • A few favourites • Testing – Grails – ATDD/BDD – Gpars – Testing DSLs – Gradle – Scripting – Tableaux AgileAust 2011 - 3
  4. 4. Advances in the Testing Arena Unit Testing Mock/interaction testing Techniques State-based testing Testing DSLs ATDD/BDD Data-driven Logic-driven© ASERT 2006-2011 Integration Testing Model-driven Performance testing Acceptance Testing All-pairs & Web drivers combinations Non-web drivers Gpars Test runners AgileAust 2011 - 4
  5. 5. Groovy and Testing Tool Spectrum* Utilities Runners AllPairs, Combinations Native Groovy, JUnit, TestNG, Spock, EasyB, Polyglot languages JBehave, Cucumber, Robot Framework, SLIM Logic programming Threads, Parallel / Web Database SOAP / Other Concurrency libraries Drivers Drivers REST Drivers Data-driven libraries Drivers Networking libraries WebTest DbUnit FEST© ASERT 2006-2011 XML Processing GroovyWS WebDriver DataSets Email Read/write files / JWebUnit SqlUnit XML-RPC FTP Excel / Word / CSV Reporting, Logging Tellurium groovy.sql CXF AntUnit Selenium JPA Axis2 Telnet HtmlUnit JDO JAX-WS SSH Tools Watij BigTable JAX-RS Exec iTest2, SoapUI, Twist, HttpBuilder JDBC WindowLicker IDEs, JMeter, Text Cyberneko editors, Recorders, Sahi, Build Tools, CI * Tools/libraries/frameworks dont always neatly fall into one category – still useful conceptually AgileAust 2011 - 5
  6. 6. Application under Test© ASERT 2006-2011 AgileAust 2011 - 6
  7. 7. Concept Manual HTTP Request / Response Web Server© ASERT 2006-2011 Automated Driver Runner <webtest name="myTest"> <steps> <invoke description="get Login Page" url="login" /> <verifyTitle description="we should see the login title" text="Login Page" /> </steps> </webtest> Read Script HTTP Request / Response AgileAust 2011 - 7
  8. 8. Driver Category • Real browser invoker • Browser Emulators – Runs on platform – Can simulate multiple supported by real browsers browser – Less platform – May need multiple restrictions platforms, e.g. IE6/IE7 – Good for CI© ASERT 2006-2011 – Uses actual JavaScript – Easier to not download engine images, resources – Can be easier to use – Ability to optimise with test recorders JavaScript interactions – Automation – More extensible capabilities differ – Ability to disable across browsers JavaScript – Can typically get to all – Scope for parallelism aspects of browser AgileAust 2011 - 8
  9. 9. What is Groovy? • “Groovy is like a super version of Java. It can leverage Javas enterprise capabilities but also has cool productivity features like closures, DSL support, builders and dynamic typing.”© ASERT 2006-2011 Groovy = Java – boiler plate code + mostly dynamic typing + closures + domain specific languages + builders + metaprogramming + GDK library AgileAust 2011 - 9
  10. 10. Topics • Introduction  Using testing DSLs • All combinations and all pairs • Auto generated tests© ASERT 2006-2011 • Model-based testing • Business/logic rules • Example driven testing • Further information AgileAust 2011 - 10
  11. 11. Prefer scripted over tool centric/recorders • Scripting • Tools/Recorders – Choose open – Lock-in to vendor tools languages – Testing often done – Developer friendly after development – Test-first friendly – Tests separate from – Test using whatever code© ASERT 2006-2011 levels make sense – Proprietary scripting – Tests with code languages – Reduce testing – User-interface documentation focussed • Hard-coded details – Refactor friendly leads to fragile tests – Run with every change – No or hard refactoring as part of continuous integration AgileAust 2011 - 11
  12. 12. Prefer business talk over tech talk • Avoid • Prefer import org.openqa.selenium.By import org.openqa.selenium.htmlunit.HtmlUnitDriver def driver = new HtmlUnitDriver() driver.get(http://localhost:8080/postForm) assert driver.title == Welcome to SimpBlog // fill in query form and submit it Source: Evolving Web-Based Test Automation into Agile Business Specifications, Mugridge et al driver.findElement(By.name(title)). sendKeys(Bart was here) driver.findElement(By.name(content)).© ASERT 2006-2011 sendKeys(Cowabunga dude!) def select = driver. findElement(By.name(category)) select. findElements(By.tagName("option")). find{ it.text == Home }.setSelected() driver.findElement(By.name(btnPost)).click() ... given we are on the blog entry page when entering Bart was here as the title and entering Home as the category then posting 5 appears AgileAust 2011 - 12
  13. 13. Topics • Introduction • Using testing DSLs  All combinations and all pairs • Auto generated tests© ASERT 2006-2011 • Model-based testing • Business/logic rules • Example driven testing • Further information AgileAust 2011 - 13
  14. 14. Workshop • Our blog application sometimes has errors but only for certain combinations of Author, Category and Content. • How should we test it?© ASERT 2006-2011 AgileAust 2011 - 14
  15. 15. Workshop • Our blog application sometimes has errors but only for certain combinations of Author, Category and Content. • How should we test it?© ASERT 2006-2011 What about: * When a new Category or Author is added in the future? * Tests take too long to run? AgileAust 2011 - 15
  16. 16. All Combinations • Description test(MacOS, 4G, 250G) – Dont have a bunch test(Linux, 4G, 250G) of hard-coded, hard test(Vista, 4G, 250G) to maintain manual test(MacOS, 8G, 500G) test data or even test(Linux, 8G, 500G) manually generated test(Vista, 8G, 500G) // 30 more rows CSV file© ASERT 2006-2011 – Much better to generate test cases [ from succinct [MacOS, Linux, Vista], [2G, 4G, 6G, 8G], expressions of [250G, 350G, 500G] what you are trying ].combinations().each{ to achieve os, mem, disk -> test(os, mem, disk) } AgileAust 2011 - 16
  17. 17. All Combinations Case Study def combos = [ ["Bart", "Homer", "Marge", "Lisa", "Maggie"], ["Work", "School", "Home", "Travel", "Food"], ["foo", "bar", "baz"] ].combinations() println "Found ${combos.size()} combos" combos.each { author, category, content ->© ASERT 2006-2011 postAndCheck author, category, content } def postAndCheck(author, category, content) { // ... details not shown ... } Found 75 combos AgileAust 2011 - 17
  18. 18. All Pairs • Description – Sometimes called pairwise testing or orthogonal array testing© ASERT 2006-2011 – Technique to limit the explosion of test cases by identifying samples of important classes of test cases (equivalence classes) • providing maximum coverage with minimum testing – Instead of all combinations, systematically use pair- wise combinations of interactions between objects • as most faults result from adverse two-way interactions AgileAust 2011 - 18
  19. 19. SimpBlog Case Study... def cases = new AllPairs().generate( author: ["Bart", "Homer", "Marge", "Lisa", "Maggie"], category: ["Work", "School", "Home", "Travel", "Food"], content: ["foo", "bar", "baz"]) println "Found ${cases.size()} cases" cases.each { next ->© ASERT 2006-2011 println next // just for debugging purposes postAndCheck next.author, next.category, next.content } def postAndCheck(author, category, content) { // ... details not shown ... } AgileAust 2011 - 19
  20. 20. ...SimpBlog Case Study Found 18 cases [content:bar, category:Food, author:Bart] [content:bar, category:School, author:Homer] [content:foo, category:Work, author:Bart] [content:baz, category:School, author:Homer] [content:bar, category:Home, author:Maggie] [content:foo, category:School, author:Marge] [content:bar, category:Work, author:Bart]© ASERT 2006-2011 [content:baz, category:Travel, author:Bart] [content:foo, category:Home, author:Homer] [content:bar, category:Travel, author:Marge] [content:baz, category:Work, author:Homer] [content:bar, category:Travel, author:Lisa] [content:baz, category:Travel, author:Maggie] [content:baz, category:Home, author:Marge] [content:baz, category:Food, author:Homer] [content:baz, category:Travel, author:Lisa] [content:foo, category:Food, author:Maggie] [content:foo, category:Travel, author:Lisa] AgileAust 2011 - 20
  21. 21. All Combinations/Pairs: Going Further • Advanced options – N-Wise – Compulsory combinations – Excluding combinations – Prioritising combinations – Shuffling© ASERT 2006-2011 • Sites – http://en.wikipedia.org/wiki/All-pairs_testing – http://www.pairwise.org/ • (content starting to age but still useful) • Tools – SpecExplorer – http://code.google.com/p/jwise/ AgileAust 2011 - 21
  22. 22. Topics • Introduction • Using testing DSLs • All combinations and all pairs  Auto generated tests© ASERT 2006-2011 • Model-based testing • Business/logic rules • Example driven testing • Further information AgileAust 2011 - 22
  23. 23. Workshop • Creating good test data is taking a long time • Errors seem to occur when the system is used with data that we haven’t© ASERT 2006-2011 tried before • What else can we do? AgileAust 2011 - 23
  24. 24. Workshop • Creating good test data is taking a long time • Errors seem to occur when the system is used with data that we haven’t© ASERT 2006-2011 What about: tried before * When fields change in • What else can we do? the future? * Measuring coverage? * Repeatable tests? AgileAust 2011 - 24
  25. 25. QuickCheck • Test Java/Groovy programs more declaratively using automatic random test data generation • Replace manually selected scenario-based tests with specification-based testing • Generators available: – primitive types,© ASERT 2006-2011 collections, POJOs – value ranges – lists, sets and array – distinct values – distributions – determinism (random, deterministic ) – generator strategies (composition of generator values (oneOf, frequency, list, array, nullsAnd), transformation, mutation) – value frequencies (frequency, oneOf) AgileAust 2011 - 25
  26. 26. QuickCheck Samples… 682 Ant Bee for (words in someNonEmptyLists(strings())) { 141 assert words*.size().sum() == words.sum().size() Jul } Dog Dog def pets = fixedValues([Ant, Bee, Cat, Dog]) Dog def nums = excludeValues(integers(100, 999, INVERTED_NORMAL), 500..599)© ASERT 2006-2011 801 def months = new Generator<String>() { 177 Generator<Date> genDate = dates() String next() { genDate.next().format("MMM") } Ant } Dog def gen = new DefaultFrequencyGenerator(pets, 50) 951 gen.add(nums, 30) Cat gen.add(months, 20) Aug 20.times { Bee def next = gen.next().toString() Bee println next Oct assert next.size() == 3 } Dog 241 AgileAust 2011 - 26
  27. 27. …QuickCheck Samples… 682 Ant Bee for (words in someNonEmptyLists(strings())) { 141 assert words*.size().sum() == words.sum().size() Jul } Dog Dog def pets = fixedValues([Ant, Bee, Cat, Dog]) Dog def nums = excludeValues(integers(100, 999, INVERTED_NORMAL), 500..599)© ASERT 2006-2011 801 def months = new Generator<String>() { 177 Generator<Date> genDate = dates() String next() { genDate.next().format("MMM") } Ant } Dog def gen = new DefaultFrequencyGenerator(pets, 50) 951 gen.add(nums, 30) Cat gen.add(months, 20) Aug 20.times { Bee def next = gen.next().toString() Bee println next Oct assert next.size() == 3 } Dog 241 AgileAust 2011 - 27
  28. 28. …QuickCheck Samples… 682 Ant Bee for (words in someNonEmptyLists(strings())) { 141 assert words*.size().sum() == words.sum().size() Jul } Dog Dog def pets = fixedValues([Ant, Bee, Cat, Dog]) Dog def nums = excludeValues(integers(100, 999, INVERTED_NORMAL), 500..599)© ASERT 2006-2011 801 def months = new Generator<String>() { 177 Generator<Date> genDate = dates() String next() { genDate.next().format("MMM") } Ant } Dog def gen = new DefaultFrequencyGenerator(pets, 50) 951 gen.add(nums, 30) Cat gen.add(months, 20) Aug 20.times { Bee def next = gen.next().toString() Bee println next Oct assert next.size() == 3 } Dog 241 AgileAust 2011 - 28
  29. 29. …QuickCheck Samples 682 Ant Bee for (words in someNonEmptyLists(strings())) { 141 assert words*.size().sum() == words.sum().size() Jul } Dog Dog def pets = fixedValues([Ant, Bee, Cat, Dog]) Dog def nums = excludeValues(integers(100, 999, INVERTED_NORMAL), 500..599)© ASERT 2006-2011 801 def months = new Generator<String>() { 177 Generator<Date> genDate = dates() String next() { genDate.next().format("MMM") } Ant } Dog def gen = new DefaultFrequencyGenerator(pets, 50) 951 gen.add(nums, 30) Cat gen.add(months, 20) Aug 20.times { Bee def next = gen.next().toString() Bee println next Oct assert next.size() == 3 } Dog 241 AgileAust 2011 - 29
  30. 30. QuickCheck: SimpBlog Case Study… • Approach – Auto-selected values for author, category – Auto-generated values for title, content def authors = ["Bart", "Homer", "Lisa", "Marge", "Maggie"] def categories = ["Home", "Work", "Food", "Travel"]© ASERT 2006-2011 10.times { postAndCheck anyString(), anyFixedValue(categories), anyFixedValue(authors), anyString() } def postAndCheck(title, category, author, content) { ... } AgileAust 2011 - 30
  31. 31. …QuickCheck: SimpBlog Case Study© ASERT 2006-2011 Screenshot of fixed titles version AgileAust 2011 - 31
  32. 32. Test Generation: Going Further • AutoMocks • Auto domain classes • Assertions • http://en.wikipedia.org/wiki/QuickCheck – Erlang, Scheme, Common Lisp, Perl, Python, Clojure,© ASERT 2006-2011 Scala, Ruby, Java, F#, Standard ML, JavaScript, C++ AgileAust 2011 - 32
  33. 33. Topics • Introduction • Using testing DSLs • All combinations and all pairs • Auto generated tests© ASERT 2006-2011  Model-based testing • Business/logic rules • Example driven testing • Further information AgileAust 2011 - 33
  34. 34. Workshop • We recently added AJAX to the SimpBlog application and some of our users are reporting issues • Depending on the order in which the blog form is filled out there can sometimes be strange error© ASERT 2006-2011 messages displayed • What testing strategies can we use in this scenario? AgileAust 2011 - 34
  35. 35. Workshop • We recently added AJAX to the SimpBlog application and some of our users are reporting issues • Depending on the order in which the blog form is filled out there can sometimes be strange error© ASERT 2006-2011 messages displayed What about: * When a new • What testing strategies can we Category or use in this scenario? Author is added in the future? * What does coverage mean? AgileAust 2011 - 35
  36. 36. Model-based testing • Deriving test suites from source code is usually deemed impractical • Instead develop models which describe certain characteristics of the desired system behaviour© ASERT 2006-2011 • May involve: – theorem proving – constraint logic programming – model checking – event-flow models – markov chain models AgileAust 2011 - 36
  37. 37. ModelJUnit... • Description – Supports model-based testing – Allows you to write simple finite state machine (FSM) models or extended finite state machine (EFSM) models in Java or Groovy© ASERT 2006-2011 – You can then generate tests from those models and measure various model coverage metrics AgileAust 2011 - 37
  38. 38. ...ModelJUnit... // require modeljunit.jar import nz.ac.waikato.modeljunit.coverage.* import nz.ac.waikato.modeljunit.* class VendingMachineModel implements FsmModel { def state = 0 // 0,25,50,75,100 void reset(boolean testing) {state = 0} boolean vendGuard() {state == 100} @Action void vend() {state = 0} boolean coin25Guard() {state <= 75}© ASERT 2006-2011 @Action void coin25() {state += 25} boolean coin50Guard() {state <= 50} @Action void coin50() {state += 50} } def tester = new RandomTester(new VendingMachineModel()) tester.buildGraph() def metrics = [new ActionCoverage(), new StateCoverage(), new TransitionCoverage(), new TransitionPairCoverage()] metrics.each { tester.addCoverageMetric it } tester.addListener "verbose" tester.generate 20 println nMetrics Summary: tester.printCoverage() AgileAust 2011 - 38
  39. 39. ...ModelJUnit ... done (0, coin50, 50) done (50, coin25, 75) done (75, coin25, 100) done Random reset(true) done (0, coin50, 50) done (50, coin25, 75) done (75, coin25, 100) done (100, vend, 0) ...© ASERT 2006-2011 done (0, coin50, 50) done (50, coin50, 100) Metrics Summary: done (100, vend, 0) action coverage: 3/3 done (0, coin25, 25) state coverage: 5/5 done (25, coin25, 50) transition coverage: 7/8 done Random reset(true) transition-pair coverage: 8/12 ... done (0, coin50, 50) done (50, coin25, 75) done (75, coin25, 100) done (100, vend, 0) done (0, coin50, 50) done (50, coin25, 75) ... AgileAust 2011 - 39
  40. 40. Interlude: Seven Bridges of Königsberg • Königsberg, Prussia (now Kaliningrad, Russia) spans the Pregel River including two large islands connected by 7 bridges • Find a walk through the city crossing each bridge once and only once. The islands can not be reached by any route other than the bridges,© ASERT 2006-2011 and every bridge must be crossed completely every time. Source: http://en.wikipedia.org/wiki/Seven_Bridges_of_K%C3%B6nigsberg AgileAust 2011 - 40
  41. 41. Interlude: Seven Bridges of Königsberg • Königsberg, Prussia (now Kaliningrad, Russia) spans the Pregel River including two large islands connected by 7 bridges • Find a walk through the city crossing each bridge once and only once. The islands can not be reached by any route other than the bridges,© ASERT 2006-2011 and every bridge must be crossed completely every time. Euler showed there is no solution. Source: http://en.wikipedia.org/wiki/Seven_Bridges_of_K%C3%B6nigsberg AgileAust 2011 - 41
  42. 42. Interlude: Königsberg Bridges Variation 8. The Blue Prince contrives a stealthy plan to build an eighth bridge so that he can begin in the evening at his Schloß, walk the bridges, and end at the Gasthaus to brag of his victory. Of course, he wants the Red Prince to be unable to duplicate the feat from the Red Castle. Where does the Blue Prince build the eighth bridge? 9. The Red Prince, infuriated by his brothers Gordian solution to the problem, wants to build a ninth bridge,© ASERT 2006-2011 enabling him to begin at his Schloß, walk the bridges, and end at the Gasthaus to rub dirt in his brothers face. As an extra bit of revenge, his brother should then no longer be able to walk the bridges starting and ending at his Schloß as before. Where does the Red Prince build the ninth bridge? 10. The Bishop has watched this furious bridge building with dismay. It upsets the towns Weltanschauung and, worse, contributes to excessive drunkenness. He wants to build a tenth bridge that allows all the inhabitants to walk the bridges and return to their own beds. Where does the Bishop build the tenth bridge? Source: http://en.wikipedia.org/wiki/Seven_Bridges_of_K%C3%B6nigsberg AgileAust 2011 - 42
  43. 43. ModelJUnit: SimpBlog Case Study... • Does the order in which form information is entered affect the application? ... void reset(boolean testing) { authorSelected = false // require modeljunit.jar, htmlunit.jar categorySelected = false© ASERT 2006-2011 import nz.ac.waikato.modeljunit.coverage.* titleEntered = false import nz.ac.waikato.modeljunit.* contentEntered = false import com.gargoylesoftware.htmlunit.WebClient client = new WebClient() url = http://localhost:8080/postForm class SimpBlogModel implements FsmModel { page = client.getPage(url) boolean authorSelected = false assert Welcome to SimpBlog == page.titleText boolean categorySelected = false form = page.getFormByName(post) boolean titleEntered = false } ... boolean contentEntered = false int count = 0 def client, page, form // Special known method, allows equivalence class definition // example states: __ __ __ __, AU __ __ __, AU CA TI CO def getState() { "${authorSelected ? AU : __ }${categorySelected ? CA : __ }" + "${titleEntered ? TI : __ }${contentEntered ? CO : __ }" } ... AgileAust 2011 - 43
  44. 44. ...ModelJUnit: SimpBlog Case Study... ... boolean "enter title Guard"() { !titleEntered } @Action void "enter title "() { titleEntered = true form.getInputByName(title).setValueAttribute("Title ${count++}") } boolean enterContentGuard() { !contentEntered } @Action void enterContent() { contentEntered = true form.getTextAreaByName(content).setText("Content ${count++}") } boolean chooseAuthorGuard() { !authorSelected } @Action void chooseAuthor() { authorSelected = true // simple version just Lisa© ASERT 2006-2011 form.getSelectByName(author).getOptions().find{ it.text == Lisa }.setSelected(true) } boolean pickCategoryGuard() { !categorySelected } @Action void pickCategory() { categorySelected = true // simple version just Home form.getSelectByName(category).getOptions().find{ it.text == Home }.setSelected(true) } boolean "submit post Guard"() { categorySelected && authorSelected && titleEntered && contentEntered } @Action void "submit post "() { def result = form.getInputByName(btnPost).click() assert result.getElementsByTagName(h1).item(0).textContent.matches(Post.*: Title .*) // could do more asserts here reset(true) } } // end of SimpBlogModel class definition ... AgileAust 2011 - 44
  45. 45. ...ModelJUnit: SimpBlog Case Study... def tester = new RandomTester(new SimpBlogModel()) tester.buildGraph() def metrics = [ new ActionCoverage(), new StateCoverage(), new TransitionCoverage(), new TransitionPairCoverage() ] metrics.each {© ASERT 2006-2011 tester.addCoverageMetric it } tester.addListener "verbose" tester.generate 50 println nMetrics Summary: tester.printCoverage() def graphListener = tester.model.getListener("graph") graphListener.printGraphDot "simpblog.dot" println "nGraph contains " + graphListener.graph.numVertices() + " states and " + graphListener.graph.numEdges() + " transitions." AgileAust 2011 - 45
  46. 46. ...ModelJUnit: SimpBlog Case Study... done ( __ __ __ __ , pickCategory, __ CA __ __ ) ... done ( __ CA __ __ , enterContent, __ CA __ CO ) done ( __ __ __ __ , pickCategory, __ CA __ __ ) done ( __ CA __ CO , enter title , __ CA TI CO ) done ( __ CA __ __ , enterContent, __ CA __ CO ) done ( __ CA TI CO , chooseAuthor, AU CA TI CO ) done ( __ CA __ CO , chooseAuthor, AU CA __ CO ) done ( AU CA TI CO , submit post , __ __ __ __ ) done ( AU CA __ CO , enter title , AU CA TI CO ) done ( __ __ __ __ , pickCategory, __ CA __ __ ) done ( AU CA TI CO , submit post , __ __ __ __ ) done ( __ CA __ __ , chooseAuthor, AU CA __ __ ) done ( __ __ __ __ , chooseAuthor, AU __ __ __ ) done ( AU CA __ __ , enter title , AU CA TI __ ) done ( AU __ __ __ , pickCategory, AU CA __ __ ) done ( AU CA TI __ , enterContent, AU CA TI CO ) done ( AU CA __ __ , enterContent, AU CA __ CO ) done ( AU CA TI CO , submit post , __ __ __ __ ) done ( AU CA __ CO , enter title , AU CA TI CO ) done ( __ __ __ __ , chooseAuthor, AU __ __ __ ) done ( AU CA TI CO , submit post , __ __ __ __ ) done ( AU __ __ __ , pickCategory, AU CA __ __ ) done ( __ __ __ __ , chooseAuthor, AU __ __ __ ) done ( AU CA __ __ , enter title , AU CA TI __ ) done ( AU __ __ __ , enter title , AU __ TI __ ) done ( AU CA TI __ , enterContent, AU CA TI CO ) done ( AU __ TI __ , pickCategory, AU CA TI __ ) done ( AU CA TI CO , submit post , __ __ __ __ ) done ( AU CA TI __ , enterContent, AU CA TI CO )© ASERT 2006-2011 done ( __ __ __ __ , enterContent, __ __ __ CO ) done ( AU CA TI CO , submit post , __ __ __ __ ) done ( __ __ __ CO , pickCategory, __ CA __ CO ) done Random reset(true) done ( __ CA __ CO , chooseAuthor, AU CA __ CO ) done ( __ __ __ __ , pickCategory, __ CA __ __ ) done ( AU CA __ CO , enter title , AU CA TI CO ) done ( __ CA __ __ , enterContent, __ CA __ CO ) done ( AU CA TI CO , submit post , __ __ __ __ ) done ( __ CA __ CO , enter title , __ CA TI CO ) done ( __ __ __ __ , pickCategory, __ CA __ __ ) done ( __ CA TI CO , chooseAuthor, AU CA TI CO ) done ( __ CA __ __ , enter title , __ CA TI __ ) done ( __ CA TI __ , chooseAuthor, AU CA TI __ ) Metrics Summary: done ( AU CA TI __ , enterContent, AU CA TI CO ) action coverage: 5/5 done ( AU CA TI CO , submit post , __ __ __ __ ) state coverage: 12/16 done ( __ __ __ __ , chooseAuthor, AU __ __ __ ) transition coverage: 19/33 done ( AU __ __ __ , pickCategory, AU CA __ __ ) transition-pair coverage: 26/56 done ( AU CA __ __ , enter title , AU CA TI __ ) done ( AU CA TI __ , enterContent, AU CA TI CO ) Graph contains 16 states and 33 transitions. done ( AU CA TI CO , submit post , __ __ __ __ ) ... AgileAust 2011 - 46
  47. 47. ...ModelJUnit: SimpBlog Case Study...© ASERT 2006-2011 Advanced Simplified version version (just Lisa) AgileAust 2011 - 47
  48. 48. ...ModelJUnit: SimpBlog Case Study© ASERT 2006-2011 AgileAust 2011 - 48
  49. 49. Model-based testing: Going further • Model-based testing – http://www.cs.waikato.ac.nz/~marku/mbt/modeljunit/ – http://en.wikipedia.org/wiki/Model_based_testing – http://home.mit.bme.hu/~micskeiz/pages/modelbased_testing.html – A Taxonomy of Model-Based Testing http://www.cs.waikato.ac.nz/pubs/wp/2006/uow-cs-wp-2006-04.pdf • Related algorithms© ASERT 2006-2011 – Job scheduling problem – Travelling salesman problem – Chinese postman problem – Vehicle routing problem – Spanning trees and Hamiltonian paths – http://www.nada.kth.se/~viggo/problemlist/compendium.html – http://en.wikipedia.org/wiki/List_of_terms_relating_to_algorithm s_and_data_structures AgileAust 2011 - 49
  50. 50. Topics • Introduction • Using testing DSLs • All combinations and all pairs • Auto generated tests© ASERT 2006-2011 • Model-based testing  Business/logic rules • Example driven testing • Further information AgileAust 2011 - 50
  51. 51. Workshop • Set up some test cases representing the Simpsons weekly blogging habits: – They never blog on the same day – Marge blogs only on a Saturday or Sunday© ASERT 2006-2011 – Maggie blogs only on a Tuesday or Thursday – Lisa blogs only on a Monday, Wednesday or Friday – Bart blogs only on the day after Lisa – Homer only blogs if noone else blogged the previous day and doesnt allow anyone to blog the next day AgileAust 2011 - 51
  52. 52. Workshop • Set up some test cases representing the Simpsons weekly blogging habits: – They never blog on the same day – Marge blogs only on a Saturday or Sunday© ASERT 2006-2011 – Maggie blogs only on a Tuesday or What about: Thursday * When the – Lisa blogs only on a Monday, habits change? Wednesday or Friday – Bart blogs only on the day after Lisa – Homer only blogs if noone else blogged the previous day and doesnt allow anyone to blog the next day AgileAust 2011 - 52
  53. 53. Constraint/Logic Programming... • Description – Style of programming where relations between variables are stated in the form of constraints – First made popular by logic programming languages such as Prolog but the style is now also used outside logic programming specific languages© ASERT 2006-2011 – Constraints differ from the common primitives of other programming languages in that they do not specify one or more steps to execute but rather the properties of a solution to be found – Popular libraries used with Groovy supporting constraint programming include Gecode/J, Choco and tuProlog – Well look at Choco as an example AgileAust 2011 - 53
  54. 54. ...Constraint/Logic Programming...© ASERT 2006-2011 Source: http://xkcd.com/287/ AgileAust 2011 - 54
  55. 55. ...Constraint/Logic Programming... // requires choco 2.1.0-basic.jar from http://choco.emn.fr/ import static choco.Choco.* import choco.kernel.model.variables.integer.IntegerVariable def m = new choco.cp.model.CPModel() def s = new choco.cp.solver.CPSolver() def menu = [ Found a solution:© ASERT 2006-2011 Mixed fruit : 215, 7 * Mixed fruit French fries : 275, Found a solution: Side salad : 335, 1 * Mixed fruit Hot wings : 355, 2 * Hot wings Mozzarella sticks : 420, 1 * Sampler plate Sampler plate : 580 ] def numOrdered = new IntegerVariable[menu.size()] def priceEach = new int[menu.size()] def sum = 1505 ... AgileAust 2011 - 55
  56. 56. ...Constraint/Logic Programming ... menu.eachWithIndex { name, price, i -> // number ordered >= 0 // number ordered * price <= sum numOrdered[i] = makeIntVar(name, 0, sum.intdiv(price)) priceEach[i] = price } m.addConstraint(eq(scalar(numOrdered, priceEach), sum))© ASERT 2006-2011 s.read(m) def more = s.solve() while (more) { println "Found a solution:" numOrdered.each { def v = s.getVar(it) if (v.val) println " $v.val * $v.name" } more = s.nextSolution() } AgileAust 2011 - 56
  57. 57. SimpBlog Case Study... def m = new choco.cp.model.CPModel() def s = new choco.cp.solver.CPSolver() daysOfWeek = ["Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"] def bart = makeIntVar(Bart, 0, 6)© ASERT 2006-2011 def homer = makeIntVar(Homer, 0, 6) def marge = makeIntVar(Marge, 0, 6) def lisa = makeIntVar(Lisa, 0, 6) def maggie = makeIntVar(Maggie, 0, 6) def simpsons = [bart, homer, marge, lisa, maggie] ... AgileAust 2011 - 57
  58. 58. ...SimpBlog Case Study... ... // They never blog on the same day for (i in 0..<simpsons.size()) for (j in 0..<i) m.addConstraint(neq(simpsons[i], simpsons[j])) // Marge blogs only on a Saturday or Sunday m.addConstraint(or(eq(marge, 0), eq(marge, 6))) // Maggie blogs only on a Tuesday or Thursday m.addConstraint(or(eq(maggie, 2), eq(maggie, 4)))© ASERT 2006-2011 // Lisa blogs only on a Monday, Wednesday or Friday m.addConstraint(or(eq(lisa, 1), eq(lisa, 3), eq(lisa, 5))) // Bart blogs only on the day after Lisa m.addConstraint(eq(plus(lisa, 1), bart)) // Homer only blogs if noone else blogged the previous // day and doesnt allow anyone to blog the next day m.addConstraint(and(distanceNEQ(homer, marge, 1), distanceNEQ(homer, bart, 1), distanceNEQ(homer, maggie, 1), distanceNEQ(homer, lisa, 1))) ... AgileAust 2011 - 58
  59. 59. ...SimpBlog Case Study ... s.read(m) def more = s.solve() if (!more) println "No Solutions Found" else println pad("Solutions:") + simpsons.collect{ pad(it.name) }.join() while (more) { print pad("") println simpsons.collect {© ASERT 2006-2011 def v = s.getVar(it) pad(daysOfWeek[v.val]) }.join() more = s.nextSolution() } def pad(s) { s.padRight(12) } Solutions: Bart Homer Marge Lisa Maggie Thursday Saturday Sunday Wednesday Tuesday Tuesday Saturday Sunday Monday Thursday Saturday Tuesday Sunday Friday Thursday Thursday Sunday Saturday Wednesday Tuesday AgileAust 2011 - 59
  60. 60. Logic solvers: Going further • Available tools – JSR-331: Java Constraint Programming API http://jcp.org/en/jsr/detail?id=331 – JaCoP: Java Constraint Programming solver http://jacop.osolpro.com/ – ChocoSolver© ASERT 2006-2011 http://www.emn.fr/z-info/choco-solver/ AgileAust 2011 - 60
  61. 61. Topics • Introduction • Using testing DSLs • All combinations and all pairs • Auto generated tests© ASERT 2006-2011 • Model-based testing • Business/logic rules  Example driven testing • Further information AgileAust 2011 - 61
  62. 62. Canoo WebTest • Description – Open source tool for automated testing of web applications – Declarative approach in XML or testing DSL in Groovy – Has Test Recorder – Excellent reporting options – Ant-based under the covers <target name="login" > <testSpec name="normal" >© ASERT 2006-2011 &config; <steps> <invoke stepid="get Login Page" url="login.jsp" /> <verifytitle stepid="we should see the login title" text="Login Page" /> <setinputfield stepid="set user name" name="username" value="scott" /> <setinputfield stepid="set password" name="password" value="tiger" /> <clickbutton stepid="Click the submit button" label="let me in" /> <verifytitle stepid="Home Page follows if login ok" text="Home Page" /> </steps> </testSpec> </target> AgileAust 2011 - 63
  63. 63. Firefox Recorder© ASERT 2006-2011 AgileAust 2011 - 64
  64. 64. EasyB Example ... • When run will be marked as pending – perfect for ATDD scenario "Bart posts a new blog entry", { given "we are on the create blog entry page" when "I have entered Bart was here as the title" and "I have entered Cowabunga Dude! into the content" and "I have selected Home as the category"© ASERT 2006-2011 and "I have selected Bart as the author" and "I click the Create Post button" then "I expect the entry to be posted" } AgileAust 2011 - 65
  65. 65. ...EasyB Example... description "Post Blog Entry Feature" narrative "for feature", { as_a "Blogger" i_want "to be able to post a blog" so_that "I can keep others informed" } before "posting blog", { given "we are on the create blog entry page", {© ASERT 2006-2011 webClient = new com.gargoylesoftware.htmlunit.WebClient() page = webClient.getPage(http://localhost:8080/postForm) } } scenario "Bart was here blog", { when "I have entered Bart was here as the title", { form = page.getFormByName(post) form.getInputByName(title).setValueAttribute( Bart was here (and so was EasyB)) } ... AgileAust 2011 - 66
  66. 66. ...EasyB Example... ... and "I have entered Cowabunga Dude! into the content", { form.getTextAreaByName(content).setText(Cowabunga Dude!) } and "I have selected Home as the category", { form.getSelectByName(category).getOptions().find { it.text == Home }.setSelected( } and "I click the Create Post button", { result = form.getInputByName(btnPost).click() }© ASERT 2006-2011 then "I expect the entry to be posted", { // check blog post details assert result.getElementsByTagName(h1).item(0).textContent.matches(Post.*: Bart wa def h3headings = result.getElementsByTagName(h3) assert h3headings.item(1).textContent == Category: Home // traditional style h3headings.item(2).textContent.shouldBe Author: Bart // BDD style // expecting: <table><tr><td><p>Cowabunga Dude!</p></td></tr></table> def cell = result.getByXPath(//TABLE//TR/TD)[0] def para = cell.firstChild assert para.textContent == Cowabunga Dude! // para.shouldHave textContent: Cowabunga Dude! } } AgileAust 2011 - 67
  67. 67. ...EasyB Example© ASERT 2006-2011 AgileAust 2011 - 68
  68. 68. JBehave Example... Given we are on the create blog entry page When I have entered "Bart was here" as the title And I have entered "Cowabunga Dude!" as the content And I have selected "Home" from the "category" dropdown And I have selected "Bart" from the "author" dropdown And I click the Create Post button Then I should see a heading message matching "Post.*: Bart was here.*" import org.jbehave.scenario.Scenario© ASERT 2006-2011 import org.jbehave.scenario.steps.Steps class NewPostScenario extends Scenario { NewPostScenario() { super([new CreateBlogSteps()] as Steps[]) } } Scenario: Given we are on the create blog entry page When I have entered "Bart was here" as the title And I have entered "Cowabunga Dude!" as the content And I have selected "Home" from the "category" dropdown And I have selected "Bart" from the "author" dropdown And I click the Create Post button Then I should see a heading message matching "Post.*: Bart was here.*" AgileAust 2011 - 69
  69. 69. ...JBehave Example... import org.jbehave.scenario.steps.Steps import org.jbehave.scenario.annotations.* import com.gargoylesoftware.htmlunit.WebClient class CreateBlogSteps extends Steps { def page, form, result @Given("we are on the create blog entry page") void gotoEntryPage() { page = new WebClient().getPage(http://localhost:8080/postForm) }© ASERT 2006-2011 @When(I have entered "$title" as the title) void enterTitle(String title) { form = page.getFormByName(post) form.getInputByName(title). setValueAttribute(title + (and so was JBehave)) } @When(I have entered "$content" as the content) void enterContent(String content) { form.getTextAreaByName(content).setText(content) } ... AgileAust 2011 - 70
  70. 70. JBehave Web Runner...© ASERT 2006-2011 AgileAust 2011 - 71
  71. 71. Interlude: Einstein’s Riddle… • Wikipedia: The zebra puzzle is a well- known logic puzzle – Often called Einsteins Puzzle or Einsteins Riddle – Said to have been invented by Albert© ASERT 2006-2011 Einstein as a boy – Claim is that Einstein said “Only 2 percent of the worlds population can solve it.” AgileAust 2011 - 72
  72. 72. Interlude: …Einstein’s Riddle • Some premises: – The British person lives in the red house – The Swede keeps dogs as pets – The Dane drinks tea – The green house is on the left of the white house – The green homeowner drinks coffee – The man who smokes Pall Mall keeps birds –© ASERT 2006-2011 The owner of the yellow house smokes Dunhill – The man living in the center house drinks milk – The Norwegian lives in the first house – The man who smokes Blend lives next to the one who keeps cats – The man who keeps the horse lives next to the man who smokes Dunhill – The man who smokes Bluemaster drinks beer – The German smokes Prince – The Norwegian lives next to the blue house – The man who smokes Blend has a neighbor who drinks water • And a question: Who owns the fish? AgileAust 2011 - 73
  73. 73. Einstein’s Riddle : Prolog % from http://www.baptiste-wicht.com/2011/09/solve-einsteins-riddle-using-prolog % Preliminary definitions persons(0, []) :- !. persons(N, [(_Men,_Color,_Drink,_Smoke,_Animal)|T]) :- N1 is N-1, persons(N1,T). person(1, [H|_], H) :- !. person(N, [_|T], R) :- N1 is N-1, person(N1, T, R). % The Brit lives in a red house hint1([(brit,red,_, _, _)|_]). hint1([_|T]) :- hint1(T).© ASERT 2006-2011 % The Swede keeps dogs as pets hint2([(swede,_,_,_,dog)|_]). hint2([_|T]) :- hint2(T). % The Dane drinks tea hint3([(dane,_,tea,_,_)|_]). hint3([_|T]) :- hint3(T). % The Green house is on the left of the White house hint4([(_,green,_,_,_),(_,white,_,_,_)|_]). hint4([_|T]) :- hint4(T). % The owner of the Green house drinks coffee. hint5([(_,green,coffee,_,_)|_]). hint5([_|T]) :- hint5(T). ... AgileAust 2011 - 74
  74. 74. Einstein’s Riddle : Groovy testing DSL the Dane drinks tea the German smokes prince the Swede keeps dogs© ASERT 2006-2011 the Brit has a red house the Norwegian owns the first house the man from the centre house drinks milk the owner of the green house drinks coffee the owner of the yellow house smokes dunhill the person known to smoke pallmall rears birds the man known to smoke bluemaster drinks beer the green house is on the left side of the white house the man known to smoke blends lives next to the one who keeps cats the man known to keep horses lives next to the man who smokes dunhill the man known to smoke blends lives next to the one who drinks water the Norwegian lives next to the blue house AgileAust 2011 - 75
  75. 75. SLIM with Groovy© ASERT 2006-2011 AgileAust 2011 - 76
  76. 76. © ASERT 2006-2011 SpiderFixture from FitLibrary (FitNesse) ZiBreve… ZiBreve Source: Evolving Web-Based Test Automation into Agile Business Specifications, Mugridge et alAgileAust 2011 - 77
  77. 77. …ZiBreve© ASERT 2006-2011 Source: Evolving Web-Based Test Automation into Agile Business Specifications, Mugridge et al AgileAust 2011 - 78
  78. 78. Example driven testing • Bridging the communication Gap – Specification by example and agile acceptance testing by Gojko Adzic • Specification by Example – How successful teams deliver the right software by Gojko Adzic: http://manning.com/adzic/© ASERT 2006-2011 • Exploration Through Example – http://www.exampler.com/blog/ (Brian Marick) • Evolving Web-Based Test Automation into Agile Business Specifications Rick Mugridge et al – http://www.mdpi.com/1999-5903/3/2/159/pdf AgileAust 2011 - 79
  79. 79. Topics • Introduction • Using testing DSLs • All combinations and all pairs • Auto generated tests© ASERT 2006-2011 • Model-based testing • Business/logic rules • Example driven testing  Further information AgileAust 2011 - 80
  80. 80. Conclusion • Don’t be afraid to use technology – For development, testing, deployment • Embrace continuous learning • There are many ways to capture business requirements© ASERT 2006-2011 – “Deciding what to build” – Find the best way to represent your requirements – Prefer machine understandable, executable solutions AgileAust 2011 - 81
  81. 81. More Information© ASERT 2006-2011 82
  82. 82. About the Author Dr Paul King leads ASERT, an organisation based in Brisbane, Australia which provides software development, training and mentoring services to customers wanting to embrace new technologies, harness best practices and innovate. He has been contributing to open source projects for nearly 20 years and is an active committer on numerous projects including Groovy. Paul speaks at international© ASERT 2006-2011 conferences, publishes in software magazines and journals, and is a co-author of Mannings best-seller: Groovy in Action. Paul King Director, ASERT @paulk_asert paulk@asert.com.au 83

×