Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Making property-based testing easier to read for humans

946 views

Published on

Agile practices have taught us that both stakeholder involvement and early testing are key to quality software. However, it is usually the case that tools for good communication are not that good for testing, and vice-versa.

In this talk, readSpec (one of the results of the PF7 EU PROWESS project) is presented, a tool that attempts to fill in this gap.

Published in: Technology
  • Login to see the comments

  • Be the first to like this

Making property-based testing easier to read for humans

  1. 1. Making property-based testing easier to read for humans Laura M. Castro February 18th, 2016
  2. 2. Stakeholder involvement & software testing The challenge of communication Active stakeholder participation is an agile best practice1 Early testing is also an agile best practice1 1 Agile practices are considered mainstream, with adoption rates ≈50%2 of 30
  3. 3. Stakeholder involvement & software testing The challenge of communication Active stakeholder participation is an agile best practice1 stakeholder involvement requires inclusive modeling (UML activity diagrams, user stories) Early testing is also an agile best practice1 best represented by Test-Driven Development (TDD) 1 Agile practices are considered mainstream, with adoption rates ≈50%2 of 30
  4. 4. Stakeholder involvement & software testing The challenge of communication Active stakeholder participation is an agile best practice1 stakeholder involvement requires inclusive modeling (UML activity diagrams, user stories) Early testing is also an agile best practice1 best represented by Test-Driven Development (TDD) stakeholder involvement + early testing = Behaviour-Driven Development (BDD) 1 Agile practices are considered mainstream, with adoption rates ≈50%2 of 30
  5. 5. Behaviour-Driven Development Feature: Deletion of element from list In order to operate lists As a user of the lists module I want to delete an element from a list Scenario: Delete integer from list of integers Given I have the integer 3 And I have the list of integers [0,8,6] When I use the function lists:delete/2 Then the resulting list should not contain 3 3 of 30
  6. 6. Behaviour-Driven Development The behaviour becomes test case given("I have the integer (d+)", S, [Num]) -> {ok, S ++ [list_to_integer(Num)]}; ... when("I use the function (w+) from the module (w+)", S, [F, M]) -> {ok, apply(list_to_atom(M), list_to_atom(F), S)}. then("the resulting list should not contain (d+)", S, [Num]) -> case not lists:member(list_to_integer(Num), S) of true -> {ok, S}; false -> {failed, S} end. 4 of 30
  7. 7. Behaviour-Driven Development Is this catching on? 5 of 30
  8. 8. Behaviour-Driven Development Is this catching on? 6 of 30
  9. 9. Behaviour-Driven Development Pros and cons Good as communication tool a number of implementations in many languages: C++, Clojure, Erlang, Java, JavaScript, Lua, .NET, PHP,… Not as good as testing tool run as many test cases as you can write well-known problems of “traditional” testing: specification, implementation, maintainability, effectiveness 7 of 30
  10. 10. Property-Based Testing (PBT) Strategy for improved testing: abstraction + automation write properties instead of test cases properties are used to automatically generate, run, and diagnose many, many test cases 8 of 30
  11. 11. Property-Based Testing A very simple example How do you write properties? Let’s start simple: Example: delete an element from a list 9 of 30
  12. 12. Property-Based Testing A very simple example How do you write properties? Let’s start simple: Example: delete an element from a list Desired functionality? If I delete an element from a list, it shouldn't be there anymore. 9 of 30
  13. 13. Property-Based Testing A very simple example How do you write properties? Let’s start simple: Example: delete an element from a list Desired functionality? If I delete an element from a list, it shouldn't be there anymore. Which item? Which list? Any. 9 of 30
  14. 14. Property-Based Testing A very simple example prop_simple() -> ?FORALL(I, item(), ?FORALL(L, list(item()), not lists:member(I, lists:delete(I, L)))). 10 of 30
  15. 15. Property-Based Testing A very simple example prop_simple() -> ?FORALL(I, item(), ?FORALL(L, list(item()), not lists:member(I, lists:delete(I, L)))). 11 of 30
  16. 16. Property-Based Testing A very simple example prop_simple() -> ?FORALL(I, item(), ?FORALL(L, list(item()), not lists:member(I, lists:delete(I, L)))). 12 of 30
  17. 17. Property-Based Testing A very simple example prop_simple() -> ?FORALL(I, item(), ?FORALL(L, list(item()), [] == [ X || X <- lists:delete(I, L), X == I])). 13 of 30
  18. 18. Property-Based Testing A very simple example prop_simple() -> ?FORALL(I, item(), ?FORALL(L, list(item()), not lists:member(I, lists:delete(I, L)))). 14 of 30
  19. 19. Property-Based Testing A very simple example > eqc:quickcheck(simple_eqc:prop_simple()). ................................................ ................................................ .... OK, passed 100 tests 15 of 30
  20. 20. Property-Based Testing A very simple example > eqc:quickcheck(simple_eqc:prop_simple()). ................................................ .................... Failed! After 69 tests. 15 [-13,-15,-2,-17,-20,15,15] Shrinking xxxxxxxx.xx.xxxxxxxx (2 times) 15 [15,15] 16 of 30
  21. 21. Property-Based Testing A more complex example How complex can it get? Models for stateful components. Example: EVM registry register(Name, Pid) -> ok unregister(Name) -> ok whereis(Name) -> Pid | undefined 17 of 30
  22. 22. Property-Based Testing A more complex example How complex can it get? Models for stateful components. Example: EVM registry register(Name, Pid) -> ok unregister(Name) -> ok whereis(Name) -> Pid | undefined Desired functionality? If I perform a sequence of registry actions, I expect compliance with model at all times. 17 of 30
  23. 23. Property-Based Testing A more complex example How complex can it get? Models for stateful components. Example: EVM registry register(Name, Pid) -> ok unregister(Name) -> ok whereis(Name) -> Pid | undefined Desired functionality? If I perform a sequence of registry actions, I expect compliance with model at all times. Which sequences of actions? Any. 17 of 30
  24. 24. Property-Based Testing A more complex example prop_registry() -> ?FORALL(Cmds, commands(registry_model), begin ... % setup code {H, S, TResult} = run_commands(registry_model,Cmds), ... % cleanup code TResult == ok end). 18 of 30
  25. 25. Property-Based Testing A more complex example prop_registry() -> ?FORALL(Cmds, commands(registry_model), begin ... % setup code {H, S, TResult} = run_commands(registry_model,Cmds), ... % cleanup code TResult == ok end). 19 of 30
  26. 26. Property-Based Testing A more complex example register_pre(S) -> S#state.pids /= []. ... unregister_next(S,_,[Name]) -> S#state{regs = lists:keydelete(Name,1,S#state.regs)}. ... whereis_post(S, [Name], Result) -> case lists:keyfind(Name, 1, S#state.regs) of {Name,Value} -> Res =:= Value; false -> Res =:= undefined end. 20 of 30
  27. 27. Property-Based Testing Pros and cons Good as testing tool run as many test cases as you have time to execute run different test cases every time get reduced counterexamples, for focused debugging get a functional specification of the software a number of implementations in many languages: Clojure, Lisp, Erlang, ML, Prolog, Scala, Scheme, Smalltalk,… Not as good as communication tool 21 of 30
  28. 28. readSpec making PBT easier to read for humans Get (semi)natural language descriptions for sample test cases that test properties produce 1. Write PBT artifacts (properties, models) 2. Get readSpec to translate them into cucumber-like features implemented in Erlang uses Quviq QuickCheck’s coverage-based suite generation Available at: https://github.com/prowessproject/readspec 22 of 30
  29. 29. readSpec A very simple example: what do you get? File: simple.feature FEATURE: simple Simple QuickCheck properties ... SCENARIO: Deleting an integer from a list should result in a list that does not contain that integer. GIVEN I have the integer 6 AND I have the list [-1, 2, 13, 0, 5] THEN IS FALSE THAT the integer 6 is in the result of calling lists:delete/2 with 6 and [-1, 2, 13, 0, 5]. ... 23 of 30
  30. 30. readSpec A very simple example: what do you get? File: prop_simple.counterexample.feature GIVEN I have the integer 15 AND I have the list [15, 15] THEN the integer 15 is in the result of calling lists:delete/2 with 15 and [15, 15]. 24 of 30
  31. 31. readSpec A very simple example: what do you get? ********************************************************** CALL: register/2 ********************************************************** POSSIBILITIES ... ================================ *** REQUIREMENTS *** [*] The field called "pids" must be equal to the empty list *** RESULT *** the literal 'false' ================================ OTHERWISE ... ================================ *** RESULT *** the literal 'true' 25 of 30
  32. 32. VoDKATV: industrial pilot study 26 of 30
  33. 33. VoDKATV: industrial pilot study Research questions RQ 1) Can readSpec produce efficient text representations of test properties? RQ 2) Are readSpec text representations an effective way to communicate among stakeholders? 27 of 30
  34. 34. VoDKATV: industrial pilot study Measurements RQ 1) Amount of time to generate the textual representation RQ 1) LOC of textual representations w.r.t. LOC of corresponding test properties/models RQ 2) Stakeholders average rating of usefulness (would use them instead of current way of communic.?) RQ 2) Stakeholders average rating of usability (appropriate? easy to understand/follow?) 28 of 30
  35. 35. VoDKATV: industrial pilot study Results Amount of time needed in the order of milliseconds Nature of output (verbosity, number of lines) appropriate Simple cases: high degree of influence of personal skills and background on deeming text more helpful than property code Complex cases: some people still cannot make sense of either; the rest consistently agree the text is more helpful enough to provide the insights needed to suggest more tests! 29 of 30
  36. 36. readSpec: making PBT easier to read for humans Final quotes “readSpec looks human-friendly” “simple cases are too verbose for tech people (code is shorter)” “helps in understanding tests better, especially when they’re complex” “it’s interesting when cases reach certain complexity” “presents requirements in an explicit way, that can slip your mind o/w” “this is very helpful for people who are not familiar with the code” “only with a good level of maturity and experience one could be more comfortable with QC properties and models” 30 of 30

×