Make Your Testing Groovy

8,918 views

Published on

Using the Groovy dynamic language for primarily functional / acceptance / customer / BDD testing with a forward looking perspective. Also considers polyglot options. The techniques and lessons learned can be applied to other kinds of testing and are also applicable to similar languages. Drivers and Runners discussed include: Native Groovy, HttpBuilder, HtmlUnit, WebTest, Watij, Selenium, WebDriver , Tellurium, JWebUnit, JUnit, TestNG, Spock, EasyB, JBehave, Cucumber, Robot Framework and FitNesse/Slim. Also looks at JMeter, ScalaCheck, Choco, AllPairs and ModelJUnit

Published in: Technology

Make Your Testing Groovy

  1. 1. ©ASERT2006-2016 Make your testing Groovy Dr Paul King Groovy Lead for Object Computing Inc. @paulk_asert http:/slideshare.net/paulk_asert/make-tests-groovy https://github.com/paulk-asert/MakeTestingGroovy
  2. 2. ©ASERT2006-2016 Why test with Groovy?
  3. 3. Why test with Groovy? • Wrong first question! • … but great second question?  • Consider first the task at hand and the organization and people fit! ©ASERT2006-2016
  4. 4. “People Fit” / “Organization Fit” • People – Developers (familiarity with languages) – Testers (tools language familiarity) – Responsibility to create/run/maintain – BAs/SMEs (ability to read technical tests) – Consider both development and maintenance – Expected feedback from tests (JUnit/Reports) • Organization – Maturity level – Degree of automation – Tools in use – Need for adaptability ©ASERT2006-2016
  5. 5. Why Test With Groovy? • Advantages – Easy to learn – Particularly good for Java shops – Easy to plug and play different testing tools – Good community & tools for professional agile • Disadvantages – Requires JVM – Less advantages if your developers using Python, .Net, PHP – Maybe your testers already know Ruby ©ASERT2006-2016
  6. 6. Scripting/Dynamic Languages • Advantages – Lend themselves to succinct code/DSLs – Powerful – Increased Refactoring – Increased Reuse – Less prone to Brittleness – Flexibility for tool integration – Open APIs provide extensibility • Disadvantages – Can be too low level – Sometimes less tool support ©ASERT2006-2016
  7. 7. Test Characteristics… • Coverage/Traceability – Requirement coverage/traceability – Code coverage: line, branch, path, state – Transactional Tracing • Purpose – Unit, Integration, System, Customer • Manageability – Removing duplication – Managing lifecycle: shared setUp/tearDown (@Before @After) ©ASERT2006-2016
  8. 8. …Test Characteristics • Handling test data – Data-driven, databases, Spreadsheets, tables, keywords, random, model-driven – Large Test Volumes – Speed of feedback, performance testing • Tool evolution, longevity, cost, support, documentation – Open Source/Commercial, Critical mass/popularity • Combinability ©ASERT2006-2016
  9. 9. Key Testing Practices… • Use testing DSL’s • Look to move up the testing stack – It used to be all about the driver – Now the driver is hidden in the framework or tool stack • Apply good testing practices – Pareto analysis, bug clusters, mutation testing, test early, all pairs/equivalence partitions/orthogonal array testing, risk-based test selection, coding for testability, use CI, boundary value analysis, defensive programming ©ASERT2006-2016
  10. 10. …Key Testing Practices • Plug and play testing tools – Run different drivers with different runners and different tools • Complement automated tests with exploration • Expand testing scope – Test environment readiness, test deployments ©ASERT2006-2016
  11. 11. Groovy and Testing Tool Spectrum* ©ASERT2006-2016 * Tools/libraries/frameworks don't always neatly fall into one category – still useful conceptually
  12. 12. ©ASERT2006-2016 What is Groovy? “Groovy is like a super version of Java. It leverages Java features but adds productivity features and provides great flexibility and extensibility.” Groovy = Java – boiler plate code + closures (1st class FP) + extensible type system + runtime & compile-time metaprogramming + flexible grammar (DSLs) + scripting & GDK library
  13. 13. Groovy starter ©ASERT2006-2016 System.out.println("Hello, World!"); // supports Java syntax println 'Hello, World!' // but can remove some syntax String name = 'Guillaume' // Explicit typing/awareness println "$name, I'll get the car." // Gstring (interpolation) def longer = """${name}, the car is in the next row.""" // multi-line, implicit type assert 0.5 == 1/2 // BigDecimal equals() assert 0.1 + 0.2 == 0.3 // and arithmetic def printSize(obj) { // implicit/duck typing print obj?.size() // safe dereferencing } def pets = ['ant', 'bee', 'cat'] // literal list syntax pets.each { pet -> // closure support assert pet < 'dog' // overloading '<' on String } // or: for (pet in pets)...
  14. 14. Target audience • Testing for developers – Organising tests – Running tests – Data generation – Mocking – What can be tested – Code coverage • Testing beyond developers – Structuring – Better reporting – Data-driven – Domain specific language (DSL) – Test coverage
  15. 15. Testing Frameworks • None • JUnit 3 • JUnit 4 • JUnit 5 • TestNG • Spock
  16. 16. Testing Frameworks • None ??? • JUnit 3 • JUnit 4 • JUnit 5 • TestNG • Spock
  17. 17. Testing Frameworks • None ??? – Groovy deems testing so important that it comes with built-in testing: – Built-in asserts, mocks – Built-in JUnit 3 GroovyTestCase and utilities – Built-in runners for tests – Bundled JUnit 4 • JUnit 3 • JUnit 4 • JUnit 5 • TestNG • Spock
  18. 18. Built-in assertions ©ASERT2006-2016 import static Converter.celsius assert 20 == celsius(68) assert 35 == celsius(95) assert -17 == celsius(0).toInteger() assert 0 == celsius(32) class Converter { static celsius (fahrenheit) { (fahrenheit - 32) * 5 / 9 } }
  19. 19. Built-in assertions • But what about errors? • Groovy’s power Assert mechanism gives a friendly description of what went wrong ©ASERT2006-2016
  20. 20. Built-in assertions • But what about errors? • Groovy’s power Assert mechanism gives a friendly description of what went wrong ©ASERT2006-2016
  21. 21. GroovyTestCase (extends JUnit 3) ©ASERT2006-2016 import org.junit.Assert import static Converter.celsius import static org.junit.Assert.assertEquals class SimpleGroovyTest extends GroovyTestCase { void testShouldConvert() { assert Converter.celsius(68) == 20 assert Converter.celsius(212) == 100, "Should convert boiling" assertEquals("Should convert freezing", 0.0, celsius(32.0)) assertEquals("Should convert nearly freezing", 0.0, celsius(32.1), 0.1) } }
  22. 22. JUnit4 ©ASERT2006-2016 import org.junit.Test import static org.junit.Assert.assertEquals import static Converter.celsius class SimpleJUnit4Test { @Test void shouldConvert() { assert celsius(68) == 20 assert celsius(212) == 100, "Should convert boiling" assertEquals("Should convert freezing", 0.0, celsius(32.0)) assertEquals("Also for nearly freezing", 0.0, celsius(32.1), 0.1) } }
  23. 23. JUnit5 ©ASERT2006-2016 @Grab('org.junit.platform:junit-platform-runner:1.0.0-M2') @Grab('org.junit.jupiter:junit-jupiter-engine:5.0.0-M2') import org.junit.jupiter.api.Test import org.junit.platform.runner.JUnitPlatform import org.junit.runner.RunWith import static Converter.celsius @RunWith(JUnitPlatform) class ConverterJUnit5Tests { @Test void freezing() { assert celsius(32) == 0 } @Test void boiling() { assert celsius(212) == 100 } }
  24. 24. JUnit5 ©ASERT2006-2016 @Grab('org.junit.platform:junit-platform-runner:1.0.0-M2') @Grab('org.junit.jupiter:junit-jupiter-engine:5.0.0-M2') import org.junit.jupiter.api.Test import org.junit.platform.runner.JUnitPlatform import org.junit.runner.RunWith import static Converter.celsius @RunWith(JUnitPlatform) class ConverterJUnit5Tests { @Test void freezing() { assert celsius(32) == 0 } @Test void boiling() { assert celsius(212) == 100 } }
  25. 25. Spock - BDD style ©ASERT2006-2016 @Grab('org.spockframework:spock-core:1.0-groovy-2.4') import spock.lang.Specification class GivenWhenThenSpec extends Specification { def "test adding a new item to a set"() { given: def items = [4, 6, 3, 2] as Set when: items << 1 then: items.size() == 5 } }
  26. 26. Parameterized ©ASERT2006-2016 import org.junit.Test import org.junit.runner.RunWith import org.junit.runners.Parameterized import org.junit.runners.Parameterized.Parameters import static Converter.celsius @RunWith(Parameterized) class DataDrivenJUnitTest { private c, f, scenario @Parameters static scenarios() {[ [0, 32, 'Freezing'], [20, 68, 'Garden party conditions'], [35, 95, 'Beach conditions'], [100, 212, 'Boiling'] ]*.toArray()} DataDrivenJUnitTest(c, f, scenario) this.c = c this.f = f this.scenario = scenario } @Test void convert() { def actual = celsius(f) def msg = "$scenario: ${f}°F should convert into ${c}°C" assert c == actual, msg } }
  27. 27. Spock ©ASERT2006-2016 @Grab('org.spockframework:spock-core:1.0-groovy-2.4') import spock.lang.* import static Converter.celsius class SpockDataDriven extends Specification { def "test temperature scenarios"() { expect: celsius(tempF) == tempC where: scenario | tempF || tempC 'Freezing' | 32 || 0 'Garden party conditions' | 68 || 20 'Beach conditions' | 95 || 35 'Boiling' | 212 || 100 } }
  28. 28. Spock - Celsius ©ASERT2006-2016 @Grab('org.spockframework:spock-core:1.0-groovy-2.4') import spock.lang.* import static Converter.celsius class SpockDataDriven extends Specification { @Unroll def "Scenario #scenario: #tempFºF should convert to #tempCºC"() { expect: celsius(tempF) == tempC where: scenario | tempF || tempC 'Freezing' | 32 || 0 'Garden party conditions' | 68 || 20 'Beach conditions' | 95 || 34 'Boiling' | 212 || 100 } }
  29. 29. Spock - Celsius ©ASERT2006-2016 @Grab('org.spockframework:spock-core:1.0-groovy-2.4') import spock.lang.* import static Converter.celsius class SpockDataDriven extends Specification { @Unroll def "Scenario #scenario: #tempFºF should convert to #tempCºC"() { expect: celsius(tempF) == tempC where: scenario | tempF || tempC 'Freezing' | 32 || 0 'Garden party conditions' | 68 || 20 'Beach conditions' | 95 || 34 'Boiling' | 212 || 100 } }
  30. 30. Spock - Mocks ©ASERT2006-2016 @Grab('org.spockframework:spock-core:1.0-groovy-2.4') import spock.lang.Specification class SpockMock extends Specification { def "buy ticket for a movie theater"() { given: def purchase = new Purchase("Lord of the Rings", 2) MovieTheater theater = Mock() theater.hasSeatsAvailable("Lord of the Rings", 2) >> true when: purchase.fill(theater) then: purchase.completed 1 * theater.purchaseTicket("Lord of the Rings", 2) } }
  31. 31. Spock - Mocks ©ASERT2006-2016 @Grab('org.spockframework:spock-core:1.0-groovy-2.4') import spock.lang.Specification class SpockMock extends Specification { def "buy ticket for a movie theater"() { given: def purchase = new Purchase("Lord of the Rings", 2) MovieTheater theater = Mock() theater.hasSeatsAvailable("Lord of the Rings", 2) >> true when: purchase.fill(theater) then: purchase.completed 1 * theater.purchaseTicket("Lord of the Rings", 2) } }
  32. 32. Spock - Mocks ©ASERT2006-2016 @Grab('org.spockframework:spock-core:1.0-groovy-2.4') import spock.lang.Specification class SpockMockWildcards extends Specification { def "cannot buy a ticket when the movie is sold out"() { given: def purchase = new Purchase("Lord of the rings", 2) MovieTheater theater = Mock() when: theater.hasSeatsAvailable(_, _) >> false purchase.fill(theater) then: !purchase.completed 0 * theater.purchaseTicket(_, _) } }
  33. 33. Spock - Mocks ©ASERT2006-2016 @Grab('org.spockframework:spock-core:1.0-groovy-2.4') import spock.lang.Specification class SpockMockClosureChecks extends Specification { def "on couples night tickets are sold in pairs"() { given: def purchase = new Purchase("Lord of the Rings", 2) MovieTheater theater = Mock() theater.hasSeatsAvailable("Lord of the Rings", 2) >> true when: purchase.fill(theater) then: 1 * theater.purchaseTicket(_, { it % 2 == 0 }) } }
  34. 34. Assertion frameworks • None • JUnit 3 assertions • Hamcrest • FEST • Google truth assertThat(googleColors).containsNoneOf(PINK, BLACK, WHITE, ORANGE) assert ![PINK, BLACK, WHITE, ORANGE].any { color -> color in googleColors }
  35. 35. Property-based testing • Agile testing game (TDD) – Minimum test code to steer design of minimal production code with desired business functionality but 100% code coverage – “Grey box” testing – Rarely used with functional programming ©ASERT2006-2016
  36. 36. Property-based testing • Agile testing game (TDD) – Minimum test code to steer design of minimal production code with desired business functionality but 100% code coverage – “Grey box” testing – Rarely used with functional programming – Instead validate certain properties ©ASERT2006-2016
  37. 37. Property-based testing ©ASERT2006-2016 @Grab('net.java.quickcheck:quickcheck:0.6') import static net.java.quickcheck.generator.PrimitiveGenerators.* import static java.lang.Math.round import static Converter.celsius def gen = integers(-40, 240) def liquidC = 0..100 def liquidF = 32..212 100.times { int f = gen.next() int c = round(celsius(f)) assert c <= f assert c in liquidC == f in liquidF }
  38. 38. Property-based testing with Spock ©ASERT2006-2016 https://github.com/Bijnagte/spock-genesis
  39. 39. Property-based testing: spock genesis ©ASERT2006-2016 @Grab('com.nagternal:spock-genesis:0.6.0') @GrabExclude('org.codehaus.groovy:groovy-all') import spock.genesis.transform.Iterations import spock.lang.Specification import static Converter.celsius import static java.lang.Math.round import static spock.genesis.Gen.integer class ConverterSpec extends Specification { def liquidC = 0..100 def liquidF = 32..212 @Iterations(100) def "test phase maintained"() { given: int tempF = integer(-40..240).iterator().next() when: int tempC = round(celsius(tempF)) then: tempC <= tempF tempC in liquidC == tempF in liquidF } …
  40. 40. Property-based testing: spock genesis ©ASERT2006-2016 @Grab('com.nagternal:spock-genesis:0.6.0') @GrabExclude('org.codehaus.groovy:groovy-all') import spock.genesis.transform.Iterations import spock.lang.Specification import static Converter.celsius import static java.lang.Math.round import static spock.genesis.Gen.integer class ConverterSpec extends Specification { def liquidC = 0..100 def liquidF = 32..212 @Iterations(100) def "test phase maintained"() { given: int tempF = integer(-40..240).iterator().next() when: int tempC = round(celsius(tempF)) then: tempC <= tempF tempC in liquidC == tempF in liquidF } … … @Iterations(100) def "test order maintained"() { given: int tempF1 = integer(-273..999).iterator().next() int tempF2 = integer(-273..999).iterator().next() when: int tempC1 = round(celsius(tempF1)) int tempC2 = round(celsius(tempF2)) then: (tempF1 <=> tempF2) == (tempC1 <=> tempC2) } }
  41. 41. Case Study
  42. 42. Case Study
  43. 43. Case Study
  44. 44. Case Study
  45. 45. Case Study
  46. 46. Case Study
  47. 47. Case Study
  48. 48. Case Study
  49. 49. Case Study
  50. 50. Case Study
  51. 51. Case Study
  52. 52. Case Study
  53. 53. Case Study
  54. 54. All Combinations
  55. 55. All Combinations Case Study
  56. 56. All Pairs
  57. 57. All Pairs Case Study
  58. 58. GPars • Library classes and DSL allowing you to handle tasks concurrently: – Data Parallelism map, filter, reduce functionality in parallel with parallel array support – Asynchronous functions extend the Java executor services to enable multi-threaded closure processing – Dataflow Concurrency supports natural shared-memory concurrency model, using single-assignment variables – Actors provide Erlang/Scala-like actors including "remote" actors on other machines – Safe Agents provide a non-blocking mt-safe reference to mutable state; like "agents" in Clojure 8
  59. 59. Case Study with GPars
  60. 60. Constraint/Logic Programming • Description – Style of programming where relations between variables are stated in the form of constraints – First made popular by logic programming languages such as Prolog but the style is now also used outside logic programming specific languages – Constraints differ from the common primitives of other programming languages in that they do not specify one or more steps to execute but rather the properties of a solution to be found – Popular libraries used with Groovy supporting constraint programming include Gecode/J, Choco and tuProlog – We'll look at Choco as an example
  61. 61. Case Study with Constraint Programming • You have been asked to set up some test cases representing the Simpsons’ weekly blogging habits • After some careful study you observe the following strange behavior – They never blog on the same day – Marge blogs only on a Saturday or Sunday – Maggie blogs only on a Tuesday or Thursday – Lisa blogs only on a Monday, Wednesday or Friday – Bart blogs only on the day after Lisa – Homer only blogs if noone else blogged the previous day and doesn't allow anyone to blog the next day
  62. 62. Case Study with Constraint Programming
  63. 63. Case Study with Constraint Programming
  64. 64. Case Study with Constraint Programming
  65. 65. ModelJUnit
  66. 66. ModelJUnit
  67. 67. ModelJUnit
  68. 68. Case Study with ModelJUnit
  69. 69. Case Study with ModelJUnit
  70. 70. Case Study with ModelJUnit
  71. 71. Case Study with ModelJUnit
  72. 72. Case Study with ModelJUnit
  73. 73. Testing DSLs • Low-level DSL • Higher-level DSL post blog from Bart with title "Bart rulz!" and category School and content "Cowabunga Dude!"
  74. 74. More Information: Groovy in Action

×