1.
Making property-based testing
easier to read for humans
Laura M. Castro
February 18th, 2016

2.
Stakeholder involvement & software testing
The challenge of communication
Active stakeholder participation is an agile best practice1
Early testing is also an agile best practice1
1
Agile practices are considered mainstream, with adoption rates ≈50%2 of 30

3.
Stakeholder involvement & software testing
The challenge of communication
Active stakeholder participation is an agile best practice1
stakeholder involvement requires inclusive modeling
(UML activity diagrams, user stories)
Early testing is also an agile best practice1
best represented by Test-Driven Development (TDD)
1
Agile practices are considered mainstream, with adoption rates ≈50%2 of 30

4.
Stakeholder involvement & software testing
The challenge of communication
Active stakeholder participation is an agile best practice1
stakeholder involvement requires inclusive modeling
(UML activity diagrams, user stories)
Early testing is also an agile best practice1
best represented by Test-Driven Development (TDD)
stakeholder involvement + early testing =
Behaviour-Driven Development (BDD)
1
Agile practices are considered mainstream, with adoption rates ≈50%2 of 30

5.
Behaviour-Driven Development
Feature: Deletion of element from list
In order to operate lists
As a user of the lists module
I want to delete an element from a list
Scenario: Delete integer from list of integers
Given I have the integer 3
And I have the list of integers [0,8,6]
When I use the function lists:delete/2
Then the resulting list should not contain 3
3 of 30

6.
Behaviour-Driven Development
The behaviour becomes test case
given("I have the integer (d+)", S, [Num]) ->
{ok, S ++ [list_to_integer(Num)]};
...
when("I use the function (w+) from the module (w+)", S, [F, M]) ->
{ok, apply(list_to_atom(M), list_to_atom(F), S)}.
then("the resulting list should not contain (d+)", S, [Num]) ->
case not lists:member(list_to_integer(Num), S) of
true -> {ok, S};
false -> {failed, S}
end.
4 of 30

7.
Behaviour-Driven Development
Is this catching on?
5 of 30

8.
Behaviour-Driven Development
Is this catching on?
6 of 30

9.
Behaviour-Driven Development
Pros and cons
Good as communication tool
a number of implementations in many languages:
C++, Clojure, Erlang, Java, JavaScript, Lua, .NET, PHP,…
Not as good as testing tool
run as many test cases as you can write
well-known problems of “traditional” testing:
specification, implementation, maintainability, effectiveness
7 of 30

10.
Property-Based Testing (PBT)
Strategy for improved testing: abstraction + automation
write properties instead of test cases
properties are used to automatically
generate, run, and diagnose many, many test cases
8 of 30

11.
Property-Based Testing
A very simple example
How do you write properties? Let’s start simple:
Example: delete an element from a list
9 of 30

12.
Property-Based Testing
A very simple example
How do you write properties? Let’s start simple:
Example: delete an element from a list
Desired functionality?
If I delete an element from a list,
it shouldn't be there anymore.
9 of 30

13.
Property-Based Testing
A very simple example
How do you write properties? Let’s start simple:
Example: delete an element from a list
Desired functionality?
If I delete an element from a list,
it shouldn't be there anymore.
Which item? Which list? Any.
9 of 30

14.
Property-Based Testing
A very simple example
prop_simple() ->
?FORALL(I, item(),
?FORALL(L, list(item()),
not lists:member(I, lists:delete(I, L)))).
10 of 30

15.
Property-Based Testing
A very simple example
prop_simple() ->
?FORALL(I, item(),
?FORALL(L, list(item()),
not lists:member(I, lists:delete(I, L)))).
11 of 30

16.
Property-Based Testing
A very simple example
prop_simple() ->
?FORALL(I, item(),
?FORALL(L, list(item()),
not lists:member(I, lists:delete(I, L)))).
12 of 30

17.
Property-Based Testing
A very simple example
prop_simple() ->
?FORALL(I, item(),
?FORALL(L, list(item()),
[] == [ X || X <- lists:delete(I, L), X == I])).
13 of 30

18.
Property-Based Testing
A very simple example
prop_simple() ->
?FORALL(I, item(),
?FORALL(L, list(item()),
not lists:member(I, lists:delete(I, L)))).
14 of 30

19.
Property-Based Testing
A very simple example
> eqc:quickcheck(simple_eqc:prop_simple()).
................................................
................................................
....
OK, passed 100 tests
15 of 30

20.
Property-Based Testing
A very simple example
> eqc:quickcheck(simple_eqc:prop_simple()).
................................................
.................... Failed! After 69 tests.
15
[-13,-15,-2,-17,-20,15,15]
Shrinking xxxxxxxx.xx.xxxxxxxx (2 times)
15
[15,15]
16 of 30

21.
Property-Based Testing
A more complex example
How complex can it get? Models for stateful components.
Example: EVM registry
register(Name, Pid) -> ok
unregister(Name) -> ok
whereis(Name) -> Pid | undefined
17 of 30

22.
Property-Based Testing
A more complex example
How complex can it get? Models for stateful components.
Example: EVM registry
register(Name, Pid) -> ok
unregister(Name) -> ok
whereis(Name) -> Pid | undefined
Desired functionality?
If I perform a sequence of registry actions,
I expect compliance with model at all times.
17 of 30

23.
Property-Based Testing
A more complex example
How complex can it get? Models for stateful components.
Example: EVM registry
register(Name, Pid) -> ok
unregister(Name) -> ok
whereis(Name) -> Pid | undefined
Desired functionality?
If I perform a sequence of registry actions,
I expect compliance with model at all times.
Which sequences of actions? Any.
17 of 30

24.
Property-Based Testing
A more complex example
prop_registry() ->
?FORALL(Cmds, commands(registry_model),
begin
... % setup code
{H, S, TResult} = run_commands(registry_model,Cmds),
... % cleanup code
TResult == ok
end).
18 of 30

25.
Property-Based Testing
A more complex example
prop_registry() ->
?FORALL(Cmds, commands(registry_model),
begin
... % setup code
{H, S, TResult} = run_commands(registry_model,Cmds),
... % cleanup code
TResult == ok
end).
19 of 30

26.
Property-Based Testing
A more complex example
register_pre(S) ->
S#state.pids /= [].
...
unregister_next(S,_,[Name]) ->
S#state{regs = lists:keydelete(Name,1,S#state.regs)}.
...
whereis_post(S, [Name], Result) ->
case lists:keyfind(Name, 1, S#state.regs) of
{Name,Value} -> Res =:= Value;
false -> Res =:= undefined
end.
20 of 30

27.
Property-Based Testing
Pros and cons
Good as testing tool
run as many test cases as you have time to execute
run different test cases every time
get reduced counterexamples, for focused debugging
get a functional specification of the software
a number of implementations in many languages:
Clojure, Lisp, Erlang, ML, Prolog, Scala, Scheme, Smalltalk,…
Not as good as communication tool
21 of 30

28.
readSpec
making PBT easier to read for humans
Get (semi)natural language descriptions
for sample test cases that test properties produce
1. Write PBT artifacts (properties, models)
2. Get readSpec to translate them into cucumber-like features
implemented in Erlang
uses Quviq QuickCheck’s coverage-based suite generation
Available at:
https://github.com/prowessproject/readspec
22 of 30

29.
readSpec
A very simple example: what do you get?
File: simple.feature
FEATURE: simple
Simple QuickCheck properties
...
SCENARIO: Deleting an integer from a list should
result in a list that does not contain that integer.
GIVEN I have the integer 6
AND I have the list [-1, 2, 13, 0, 5]
THEN IS FALSE THAT the integer 6 is in
the result of calling lists:delete/2
with 6 and [-1, 2, 13, 0, 5].
...
23 of 30

30.
readSpec
A very simple example: what do you get?
File: prop_simple.counterexample.feature
GIVEN I have the integer 15
AND I have the list [15, 15]
THEN the integer 15 is in
the result of calling lists:delete/2 with 15 and [15, 15].
24 of 30

31.
readSpec
A very simple example: what do you get?
**********************************************************
CALL: register/2
**********************************************************
POSSIBILITIES ...
================================
*** REQUIREMENTS ***
[*] The field called "pids" must be equal to the empty list
*** RESULT ***
the literal 'false'
================================
OTHERWISE ...
================================
*** RESULT ***
the literal 'true'
25 of 30

32.
VoDKATV: industrial pilot study
26 of 30

33.
VoDKATV: industrial pilot study
Research questions
RQ 1) Can readSpec produce efficient text representations of
test properties?
RQ 2) Are readSpec text representations an effective way to
communicate among stakeholders?
27 of 30

34.
VoDKATV: industrial pilot study
Measurements
RQ 1) Amount of time to generate the textual representation
RQ 1) LOC of textual representations w.r.t.
LOC of corresponding test properties/models
RQ 2) Stakeholders average rating of usefulness
(would use them instead of current way of communic.?)
RQ 2) Stakeholders average rating of usability
(appropriate? easy to understand/follow?)
28 of 30

35.
VoDKATV: industrial pilot study
Results
Amount of time needed in the order of milliseconds
Nature of output (verbosity, number of lines) appropriate
Simple cases: high degree of influence of personal skills and
background on deeming text more helpful than property code
Complex cases: some people still cannot make sense of
either; the rest consistently agree the text is more helpful
enough to provide the insights needed to suggest more tests!
29 of 30

36.
readSpec: making PBT easier to read for humans
Final quotes
“readSpec looks human-friendly”
“simple cases are too verbose for tech people (code is shorter)”
“helps in understanding tests better, especially when they’re complex”
“it’s interesting when cases reach certain complexity”
“presents requirements in an explicit way, that can slip your mind o/w”
“this is very helpful for people who are not familiar with the code”
“only with a good level of maturity and experience one could be more
comfortable with QC properties and models”
30 of 30