Finding edge cases and hidden bugs is hard for a fixed test data. It would be nice to think about tests as proofs, where certain properties of our code would be tested against generated sets of data. Is this possible? Can a framework prepare test data for us and find examples that do not work? During this talk, we will explore a technique called Property-based testing (PBT) and how to apply it in our day to day work in Java. We will look how to build good properties and check few non academic examples of properties.

1. Andrzej Jóźwiak
andrzej.jozwiak@gmail.com
https://github.com/artificialrevelations
Property
Based Tests
And Where
To Find Them

2. meetup.juglodz.pl
yt.juglodz.pl
fb.juglodz.pl
2

3. Exploits of a Mom: https://xkcd.com/327/ 3

4. Poll: Unit tests ...
A. are exhaustive
B. guarantee the correctness
C. help find unknown cases
D. all of the above
E. it really depends on the context
4

5. Would you like to play a game?
Can your unit tests ...
5

6. find out that the car brakes do not work
when volume on the radio is changed?
Can your unit tests ...
Experiences with QuickCheck: https://www.cs.tufts.edu/~nr/cs257/archive/john-hughes/quviq-testing.pdf 6

7. Bypassing Android lockscreen: https://sites.utexas.edu/iso/2015/09/15/android-5-lockscreen-bypass/
find out that putting enough text in the
Android lock screen text field can crash it
and give full access to the phone?
Can your unit tests ...
7

8. A Formal Specification of a File Synchronizer: https://www.cis.upenn.edu/~bcpierce/papers/unisonspec.pdf
find out that your secure file storage is
not secure and is losing data?
Can your unit tests ...
8

9. find out that going back and forth through
a list of available voices can cause the
app to play an incorrect voice?
Can your unit tests ...
my own child sure can!
9
This is a thing that we found out in our own app

10. 10
Twitter: https://twitter.com/cbirchall/status/1209156343904587779
All unit tests passing do not always guarantee correctness...

11. 11
And now for an academic example...
public static <A> List<A> reverse(final List<A> original) {
final int size = original.size();
final List<A> reversed = new ArrayList<>(size);
// going from the last element to the first of the
original list
for (int i = size - 1; i >= 0; i--) {
// thus adding the elements in the reversed order
reversed.add(original.get(i));
}
return reversed;
}
What are the properties of the code here?
Github: https://git.io/JfDGP

12. 12
Lists reverse function example based tests
Github: https://git.io/JfDGa

13. 13
Let’s build an example property...
Github: https://git.io/JfDZS
@Test
public void reverse_reversed_equals_original() {
// given:
final List<String> tested = Arrays.asList("This", "is", "a", "test");
// when:
final List<String> result = reverse(reverse(tested));
// then:
assertEquals(tested, result);
// Should we create more examples?
}

14. 14
Let’s build a property based testing framework ...
We will notice soon enough that something is missing

15. 15
Let’s build a property based testing framework ...
private interface Property<A> {
boolean check(A value);
}
private static <A> Property<List<A>> reverseOfReversedEqualsOriginal ()
{
return original -> reverse(reverse(original)).equals(original);
}
Github: https://git.io/JfDZS

16. 16
Let’s build a property based testing framework ...
private interface Generator<A> {
A generate(Random seed);
}
// A very simple generator that produces lists:
// - with size varying from 0 to 100
// - with elements being numbers from 0 to 100 as strings
private Generator<List<String>> randomStringListsGenerator () {
return seed -> {
final int randomSize = randInt(seed, 0, 100);
final List<String> randomizedList = new
ArrayList<>(randomSize);
for (int i = 0; i < randomSize; i++) {
randomizedList .add(String.valueOf(randInt(seed, 0, 100)));
}
return randomizedList ;
};
}
Github: https://git.io/JfDZS

17. 17
Let’s build a property based testing framework ...
private static <A> void quickCheck(final long seedValue,
final int numberOfTries,
final Property<A> property,
final Generator<A> generator) {
final Random seed = new Random(seedValue);
for (int i = 0; i < numberOfTries; i++) {
final A tested = generator.generate(seed);
final boolean result = property.check(tested);
if (!result) {
final StringBuilder builder =
new StringBuilder()
.append("Property test failed")
.append("nSeed value: ")
.append(seedValue)
.append("nExample data: ")
.append(tested);
throw new AssertionError(builder);
}
}
}
Github: https://git.io/JfDZS

18. 18
Let’s build a property based testing framework ...
@Test
public void custom_reverse_property_test() {
final int numberOfTries = 1000;
final long seedValue = new Random().nextLong();
final Property<List<String>> reverseProperty =
reverseOfReversedEqualsOriginal();
final Generator<List<String>> generator =
randomStringListsGenerator();
quickCheck(seedValue, numberOfTries, reverseProperty, generator);
}
Github: https://git.io/JfDZS

19. 19
We are lucky! The example only has 19 elements, what
would happen if it had 1000 or 10000 elements?
private static <A> Property<List<A>> reverseWrongProperty() {
return original -> reverse(original).equals(original);
}
Github: https://git.io/JfDZS
java.lang.AssertionError: Property test failed
Seed value: -2569510089704470893
Example data: [40, 15, 20, 30, 36, 35, 55, 99, 89, 93, 67, 27,
31, 95, 26, 6, 84, 23, 92]
How can we know our little framework is correct?

20. 20
What about a professional solution?
Github: https://git.io/JfDZS
@Property
@Report(Reporting.GENERATED)
public boolean broken_reverse_property
(@ForAll List<?> original) {
return Lists.reverse(original).equals(original);
}
|-------------------jqwik-------------------
tries = 1 | # of calls to property
checks = 1 | # of not rejected calls
generation-mode = RANDOMIZED | parameters are randomly generated
after-failure = PREVIOUS_SEED | use the previous seed
seed = -1616208483702989146 | random seed to reproduce generated values
sample = [[Any[0], Any[1]]]
original-sample = [[Any[64], Any[226], Any[319], Any[71], Any[351], Any[137],
Any[9], Any[262], Any[239], Any[485], Any[23], Any[265], Any[108], Any[348],
Any[202], Any[365], Any[147], Any[347], Any[133]]]
It also found a 19 element example but managed to shrink it to 2

21. 21
Poll: Our framework was missing ...
A. a way to shrink the example to smallest possible
size
B. a way to shrink the values used by the example
C. pretty printing of the output
D. all of the above
E. nothing, It was perfect!
F. I bet there is something more!

22. 22
Our framework was missing ...

23. Are there well established categories of
properties?
23
Before we leave the realm of academic examples...
and the Math behind them is minimal! I promise.

24. 24
Property Category: Encode/Decode

25. 25
Property Category: Encode/Decode
Twitter: https://github.com/btlines/pbdirect

26. 26
Property Category: Encode/Decode
Twitter: https://github.com/btlines/pbdirect

27. 27
Property Category: Invariants

28. 28
Property Category: Invariants
size of the collection: some operations on collections do not change their size but
only modify the order of elements or their state like sorting, mapping
content of the collection: some operations on collections do not change their
content (do not remove or add elements), it’s possible to find all elements but just in a
different order
order of elements: some operations change the characteristics of elements but do
not change their order
domain specific: premium users will always have a special discount regardless of the
amount of items on their shopping cart. We can always find properties of our model
that do not changed regardless of operations applied.

29. 29
Property Category: Idempotence

30. 30
Property Category: Idempotence
absolute value of an integer: abs(abs(x)) = abs(x)
floor, ceiling functions: ceil(ceil(x)) = ceil(x)
database search: searching for an element in a database multiple times should give
us the same results each time. This only holds in a case where there are no updates in
between.
sorting: sorting a collection multiple times should give us the same result, sorting an
already sorted collection should give us the same result, the same applies for filtering,
searching for distinct elements etc.
crosswalk button: pressing a crosswalk button multiple times will not change the end
result of the lights changing from red to green.

31. 31
Property Category: Different paths lead to the same destination

32. 32
Property Category: Different paths lead to the same destination
commutative property of addition: x + y = y + x, for all x, y ∈ R
putting socks on your feet: does it matter in which order we will put our socks on? If
the thing that we would like end up with are both feet in socks, then it does not matter
if we start with left or right foot!
applying discount code to the shopping cart: does it matter if we apply the discount
code to an empty cart (ofc if the API allows to start shopping like this) before we add
the items? The result should be the same for when we apply the discount to already
filled shopping cart.
putting ingredients on your pizza: does it matter if we start with pineapple or ham if
we want to have great pizza? I know I know this example is controversial!
filtering and sorting the results: the end result should be the same if we start from
sorting and then filter out the unwanted results. Let’s not focus on the issue of
optimization

33. 33
Property Category: Test Oracle

34. 34
Property Category: Test Oracle
result should stay the same: code after refactoring should give the same results as
the original one for the same input. This can be applied for any kind of code not only
examples like sorting or filtering. If we are reworking some UI elements to support
new cases it should behave the same like the old one for all old cases.
the new algorithm should not have worse performance then the old one: if the
refactored code should use less memory, use less hard drive space, perform less
operations, in other words just be better in some way or at least not worse, then
comparing old with new is the way to go.
when writing a property is not obvious: instead of finding a property of the code like
encode/decode or different paths same result, we can base our tests on the already
existing code and its results.

35. OK. Everything is nice but I am not writing
sorting for my day job!
35
Now off to the real world...
or reverse, or map, or json encoding for that matter

36. 36
1. open new database
2. put key1 and val1
3. close database
4. open database
5. delete key2
6. delete key1
7. close database
8. open database
9. delete key2
10. close database
11. open database
12. put key3 and val1
13. close database
14. open database
15. close database
16. open database
17. seek first
Expected output? key3
Actual output? key1
Reappearing "ghost" key after 17 steps: https://github.com/google/leveldb/issues/50
A ghost story ...
Would you come up with such an example?

37. 37
Reappearing "ghost" key after 17 steps: https://github.com/google/leveldb/issues/50
A ghost story - a rehaunting ...
After a patch that was supposedly fixing the issue a new example was found
this time made up from 33 steps. After almost a month it was finally fixed.
Fixed bug in picking inputs for a level-0 compaction.
When finding overlapping files, the covered range may expand as files are added
to the input set. We now correctly expand the range when this happens instead
of continuing to use the old range. This change fixes a bug related to deleted
keys showing up incorrectly after a compaction.

38. 38
Stateful testing...

39. 39
Stateful testing...
Model - describes the current expected state of the system. Often we know what to
expect from the code, it does not matter how the code gets to that result.
Commands - represent what the system under test should do, command generation
needs to take into account the current model state.
Validation - way to check the current model state and compare it with the system
under test state.
SUT - system under test, stateful in it’s nature

40. 40
A stateful representation of a shopping cart ...
public class ShoppingCart {
public Quantity get(Product product)
public List<Product> getAllProducts ()
public Quantity add(Product product, Quantity quantity)
public Quantity remove(Product product, Quantity quantity)
public void clear()
public void setDiscount(DiscountCode code)
public void clearDiscount()
public Price getTotalPrice()
public Quantity getTotalQuantity ()
}

41. 41
A simplified shopping cart model ...
public class ShoppingCartModel {
private List<ShoppingCartModelElement> products = new
LinkedList<>();
private int discount = 0; //%
private static class ShoppingCartModelElement {
public String product;
public int price;
public int quantity;
}
}

42. 42
A simplified representation of a command in our system ...
public interface Command<M, S> {
boolean preconditions(final M model);
void execute(final M model, final S sut);
void postconditions (final M model, final S sut);
}

43. 43
An implementation of one of the available commands ...
public class AddProductCommand implements Command<ShoppingCartModel , ShoppingCart >
{
public void execute(final ShoppingCartModel model,
final ShoppingCart sut) {
model.addProduct(
product .getName(),
quantity .getValue(),
product .getPrice().getValue()
);
sut.add( product, quantity);
}
public void postconditions (final ShoppingCartModel model,
final ShoppingCart sut) {
Assertions.assertEquals(
model.getQuantity( product.getName()),
sut.get( product).getValue()
);
}

44. 44
Available commands for our property test
- AddProductCommand
- RemoveProductCommand
- GetProductCommand
- GetAllProductsCommand
- ClearProductCommand
- ClearCommand
- ClearProductCommand
- SetDiscountCommand
- ClearDiscountCommand
- GetTotalPriceCommand
- GetTotalQuantityCommand
Now we need to create random sequences of commands that will stress the
implementation of our stateful component.

45. 45
Testing properties of a shopping cart ...
Let’s introduce a bug in the shopping cart:
- if the shopping carts has 3 or more elements
- it will NOT add any more elements
Run failed after following actions:
AddProductCommand{ product=Product{name='AAA', price=Price(3)},
quantity=Quantity(1)}
AddProductCommand{ product=Product{name='AAa', price=Price(3)},
quantity=Quantity(1)}
AddProductCommand{ product=Product{name='ABA', price=Price(3)},
quantity=Quantity(1)}
AddProductCommand{ product=Product{name='AAB', price=Price(3)},
quantity=Quantity(1)}

46. 46
Testing properties of a shopping cart ...
original-sample = [ActionSequence[FAILED]: [SetDiscountCommand{discount=Discount(58)},
GetTotalQuantityCommand{}, ClearCommand{}, SetDiscountCommand{discount=Discount(100)},
SetDiscountCommand{discount=Discount(58)}, GetTotalQuantityCommand{},
GetTotalPriceCommand{}, ClearDiscountCommand{}, SetDiscountCommand{discount=Discount(18)},
GetTotalQuantityCommand{}, GetTotalPriceCommand{}, ClearCommand{},
GetTotalQuantityCommand{}, AddProductCommand{ product=Product{name='ZzZaWB',
price=Price(6)}, quantity=Quantity(60)}, GetTotalPriceCommand{}, AddProductCommand{
product=Product{name='zEfaZZga', price=Price(8)}, quantity=Quantity(74)},
GetTotalPriceCommand{}, ClearDiscountCommand{}, AddProductCommand{
product=Product{name='faGZQAGAQ', price=Price(9)}, quantity=Quantity(154)},
GetTotalQuantityCommand{}, AddProductCommand{ product=Product{name='ZVdJaj',
price=Price(6)}, quantity=Quantity(66)}, SetDiscountCommand{discount=Discount(32)},
GetTotalQuantityCommand{}, GetTotalQuantityCommand{}, GetTotalQuantityCommand{},
AddProductCommand{ product=Product{name='AAzZza', price=Price(6)},
quantity=Quantity(5)}]]

47. 47
Thesis Defense: https://xkcd.com/1403/
Q&A time ...

48. 48
Further reading:
(book) PropEr Testing https://propertesting.com/toc.html
(documentation) QuickCheck: https://hackage.haskell.org/package/QuickCheck
(article) Introduction to PBT: https://fsharpforfunandprofit.com/posts/property-based-testing/
(documentation) jqwik: https://jqwik.net/
(presentation) Stateful PBT: https://www.youtube.com/watch?v=owHmYA52SIM
(presentation) Don’t Write Tests: https://www.youtube.com/watch?v=hXnS_Xjwk2Y
(presentation) Testing the Hard Stuff: https://www.youtube.com/watch?v=zi0rHwfiX1Q
(examples): https://github.com/artificialrevelations/property-based-testing-workshop