Property-based testing (PBT) focuses on testing specifications rather than implementations. It uses random testing against properties expressed as code to generate many test cases, reducing testing effort. PBT represents a system as states, commands to transition between states, and properties relating commands to expected states. This allows effective testing of stateful systems. PBT has been used successfully for concurrency, distributed systems, and finding bugs unit tests missed. Popular PBT libraries include Scalacheck, QuickCheck, and Hypothesis.

1. Property-based testing
Less is More
Ho Tien Vu @hotienvu
htvu@thoughtworks.com

2. Why we write tests
• Programs are implemented based on
specifications
• Tests are used to verify against specification
• Fail fast

3. Unit test
• First line of defend
• Test small modules in isolation
• Fast, repeatable
• Facilitates changes
• Documentation

4. Unit test - Example
• Testing sorted method on a List of Int
• empty list => []
• already-sorted list => [1,2,3,4,5,6,]
• reversed-sorted list => [6,5,4,3,2,1]
• arbitrary list => [1,4,3,6,5,2]

5. Problems
• Data is hard-coded
• Are we missing out edge cases?
- long list?
- What about boundaries? (-21474836478
2147483647)
• Are we biased towards the
implementation?
• Spend considerable amount of time writing/
fixing tests

6. The assertion hell

7. “Testing shows the presence, not the
absence of bugs” E.W. Dijkstra
http://googleresearch.blogspot.sg/2006/06/extra-extra-read-
all-about-it-nearly.html

9. Property-based test
• Testable Specification + Random Testing
• Focus on the specification
• Let the testing library take care of
generating test cases

10. Specification as code

11. Properties of sorted
• every element is less than or equal to the
next element in the sorted list
sorted[i] <= sorted[i+1] for i in 1..n-1

12. Properties of sorted
• should retain all elements of original list
sorted[i].size == original[i].size
sorted.contains(x) for x in original

13. Random Testing
• Random generator with emphasize on zero,
extremities, +/- unity, special values, etc
• Allow custom distribution

14. Generator
• DSL for generating random test data
- forAll, chooseFrom, oneOf, constant, etc
• Allow custom-type
• Composable - allow complex data
structure e.g. Tree

15. Test auto-shrink
• When encounter a failed test, the PBT
library will try to minimize it
• Reduce effort on debugging

16. Advantages of PBT
• Let developer focus on specifications
• Reduce testing effort
• More coverage
• Make testing FUN!

17. Testing stateful system
• Stateful system is hard to test
- testing 1 component O(n)
- testing 2 components O(n^2)…
• Often focus on how components behave
instead of the system as a whole
=> Need better abstraction

18. Testing stateful system
with PBT
• Abstraction in PBT
- State: information about the system
- Command: interface that changes system
from one state to another
- System Under Test(SUT) - the system we
want to test

19. Property = SUT.command(preState) == postState

20. Testing stateful system

21. Testing a Counter
class Counter(n: Int)
+ inc() = { n += 1 }
+ dec() = { n -= 1}
+ get() = n

22. Testing a Counter
• State := n
• SUT := Counter(0)
• Commands
- Inc => State = n + 1; SUT.Inc()
-Dec => State = n - 1; SUT.Dec()
-Get => State = n;
-postCondition := SUT.Get() == n

23. Example - Restful Web
Services
• 4 Commands: Create, Read, Update, Delete
• System under test: Resources
• State: Your Database Model

24. Example - Redis Client
• Testing a Redis-client
https://github.com/rickynils/scalacheck/tree/
master/examples/commands-redis

25. Example - LevelDb
• https://code.google.com/p/leveldb/issues/
detail?id=44
=> It is impossible for human to come up
with a test case of 17 steps

26. Concurrency/
Distributed System
• Have been used successfully in many
projects
- Riak
- Akka
• Often discover bugs that Unit tests couldn't

27. PBT library
• Haskell - QuickCheck
• JVM(Scala/Java/Clojure)
- Scalacheck (Java/Scala)
- test.check (Clojure)
• Python - hypothesis
• Erlang - QuickCheck
• Javascript, C++, etc

28. PBT is not a silver
bullet
• Writing PBT can be really hard if
- your program has side-effect
- it’s not trivial to verify the output
correctness (e.g. optimal function)
- there are lots of duplication between
specifications and implementations

29. Summary
• PBT let developer focus on and express
specification in a concise format
• PBT reduces testing effort
• PBT can be used to test stateful system
• PBT works well with functional
Programming Languages

30. Thank you!
https://github.com/hotienvu/pbt-scalacheck
@hotienvu
htvu@thoughtworks.com