The document discusses property-based testing (PBT), a technique that automates the generation of specific test cases based on general models, making testing more efficient and effective. It outlines how PBT can handle stateful systems, utilize tools for generating data, and handle counterexamples, stressing the importance of adopting PBT methodologies in software testing. Additionally, it highlights ongoing research and future expectations in applying PBT to domains such as web UI testing and robotics.

1. Targeting Property-Based Testing
inside out
Laura M. Castro
Bournemouth, July 16th

2. A lot of people and companies still test things "manually"... writing test
cases one by one... having to decide which data to use... having to
implement what to expect... and still missing lots of bugs that slip
through to production software

3. Property-Based Testing: the practice
o What is Property-Based Testing (PBT)?
o A testing technique that uses a general model to automatically derive specific
test cases for a SUT

4. A "model"?

5. Property-Based Testing: the practice
o What is Property-Based Testing (PBT)?
o A testing technique that uses a general model to automatically derive specific
test cases for a SUT
o If the SUT has no state, the model is a
universally quantified description of a
property that the SUT is expected to
hold
o If the SUT does have state, the model
is a state machine implementation
that holds the state of the SUT relevant
for testing purposes, and describes how
it evolves when subjected to calls to its
API, together with indications of when
such calls can be made (preconditions)
and the effects they are expected to
have (postconditions)
?FORALL(I, nonzero_integer(),
sut:divide(I, I) == 1).
?FORALL(I, integer(),
?FORALL(L, list(integer()),
not lists:member(I, sut:remove_all(I, L))).
?FORALL(L1, list(integer()),
?FORALL(L2, list(integer()),
begin
L1L2 = sut:join(L1, L2),
((length(L1L2) >= length(L1)) andalso
(length(L1L2) >= length(L2)))
end)).

6. Property-Based Testing: the practice
o What is Property-Based Testing (PBT)?
o A testing technique that uses a general model to automatically derive specific
test cases for a SUT
o If the SUT has no state, the model is a
universally quantified description of a
property that the SUT is expected to
hold
o If the SUT does have state, the model
is a state machine implementation
that holds the state of the SUT relevant
for testing purposes, and describes how
it evolves when subjected to calls to its
API, together with indications of when
such calls can be made (preconditions)
and the effects they are expected to
have (postconditions)
?FORALL(I, nonzero_integer(),
sut:divide(I, I) == 1).
?FORALL(I, integer(),
?FORALL(L, list(integer()),
not lists:member(I, sut:remove_all(I, L))).
?FORALL(L1, list(integer()),
?FORALL(L2, list(integer()),
begin
L1L2 = sut:join(L1, L2),
((length(L1L2) >= length(L1)) andalso
(length(L1L2) >= length(L2)))
end)).
next_state(S,_,
{call,sut,configure,[Seconds]}) ->
S#alarm{value = Seconds};
…
precondition(_,{call,sut,_,_}) ->
service_alarm_started();
…
postcondition(S,{call,sut,_,_},_Res) when
S#alarm.elapsed_time == S#alarm.value ->
service_alarm_fired();
…

7. Property-Based Testing: the practice
o What is Property-Based Testing (PBT)?
o PBT tools provide library utilities for model definition (including data generators
for basic data types, functions to sample them), and functions to run any number
of desired specific test cases
> eqc:quickcheck(sut_props:prop_divide()).
.........................................................................
............................
OK, passed 101 tests
true
> eqc:quickcheck(eqc:numtests(10000,sut_props:prop_divide())).
.........................................................................
...........................(x10).........................................
...........................................................(x100)........
.........................................................................
........
OK, passed 10000 tests
true

8. Property-Based Testing: the practice
o What is Property-Based Testing (PBT)?
o PBT tools can also implement shrinking, the (local) minimisation of
counterexamples, as well as counterexample saving and re-checking of previously
failing properties
> eqc:quickcheck(sut_props:prop_remove()).
.........................................Failed! After 41 tests.
{-8,[11,-10,-8,10,-8]}
Shrinking xxxxxxxxxxxxx..xxxxxxxxxxxxxxxxxxxxxx(2 times)
{-8,[-8,-8]}
False
> C = eqc:counterexample().
[{-8,[-8,-8]}]
…
> eqc:recheck(sut_props:prop_divide(), C).
OK, test passed this time.
true

9. Sorry, you still need to debug the error...
and it might be an error in the model!

10. This sounds too difficult...

11. Experience from previous projects
o ProTest (FP7 ICT 2007, 215868)
o Adopting new methodologies/tools is not
trivial, should be championed & assisted
o SMEs are particularly sensitive to time
pressures, team changes
o PROWESS (FP7 ICT 2011, 317820)
o Specific software applications have their
own particularities w.r.t. testing
o Domain-specific, yet generally-applicable
test models can be defined to increase
effectiveness and efficiency

12. Property-Based Testing: recent research
o While in many cases randomly generating inputs/calls is effective enough to
find bugs and increase our confidence in the SUT, it is not exactly efficient
o Sometimes one gets stuck finding the same bug again and again
o Quviq QuickCheck allows users to run a property for a given amount of time and report
the different counterexamples found later, instead of stopping when the first property
violation is detected
o Sometimes it takes a long time to find a bug
o This can be avoided sometimes by tampering with data generators
o PropEr allows users to specify a search strategy in order to steer the generation of data
for the test cases

13. Property-Based Testing: recent research
o Automatically testing web-based UIs using webdriver
o Monkey testing is easy, allows to find crashes the user should never see
o Specific models are still hard to write for new adopters of PBT
, especially front-end
o Validation of SDN (software-defined networks) policies
o Different sources of policies (i.e. decentralized specification)
o Policies added/removed dynamically (i.e. runtime-dynamic specification)
o Complexity grows as does network size (not real time)

14. Let's do testing for robotics!

15. Expectations for upcoming projects
o Robot cooperation surely has its
own specificities that we can
capture in domain-specific test
models
o Expressivity
o Effectiveness
o Robot cooperation brings relevant
aspects to the testing table:
o Security
o Fault-tolerance
o Isolation

16. Disclaimer
o All images related to the motion picture "Inside Out (2015)" are protected by
copyright, which is believed to belong to the distributor of the film, Walt
Disney Studios Motion Pictures, the publisher of the film or the graphic artist