Ruslan Shevchenko - Property based testing

1. — Ruslan Shevchenko [iov42; codespace]
// @rssh1, ruslan@shevchenko.kiev.ua
PROPERTY BASED TESTING
”
“

2. WHAT IS IT ?
PROPERTY BASED TESTING
Eng: property
властивість
свойство
Other : example
приклад
пример
for all x: Int x < (x+1)
3 < (3 + 1)

3. SHOW ME THE CODE/SCALA
import org.scalacheck.Properties
import org.scalacheck.Prop.forAll
object IntSpecification extends Properties("Int")
{
property("x < x+1") =
forAll{ (x:Int) => x < x+1 }
}

4. SHOW ME THE CODE/SCALA
import org.scalacheck.Properties
import org.scalacheck.Prop.forAll
object IntSpecification extends Properties("Int")
{
property(“x < x+1") =
forAll{ (x:Int) => x < x+1 }
}
[info] ! Int.x > x+1: Falsified after 3 passed tests.
[info] > ARG_0: 2147483647
[info] Failed: Total 1, Failed 1, Errors 0, Passed 0
[error] Failed tests:
[error] IntSpecification
[error] (test:test) sbt.TestsFailedException: Tests unsuccessfu

5. SHOW ME THE CODE/JAVA
…..
@RunWith(JUnitQuickcheck.class) public class IntPropertyTest {
@Property public void xLessIncr(int x)
{
assert(x < x+1);
}
}

6. SHOW ME THE CODE/JAVA
…..
@RunWith(JUnitQuickcheck.class) public class IntPropertyTest {
@Property public void xLessIncr(int x)
{
assert(x < x+1);
}
}
[info] Running com.github.rssh.IntPropertyTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,
Time elapsed: 0.75 sec
// Problem not found.
// Generation of random values are differ ;)

7. SHOW ME THE CODE/JAVASCRIPT
var jsc = require("jsverify")
describe('double ceiling', function() {
it ('x < x + 1e-15',function() {
expect(
jsc.checkForall(jsc.number,function(x){
return x < x+1e-15
})
).toBe(true)
});
});
// explore the limits of the floating point ceiling.

8. SHOW ME THE CODE/JAVASCRIPT
var jsc = require("jsverify")
describe('double ceiling', function() {
it ('x < x + 1e-15',function() {
expect(
jsc.checkForall(jsc.number,function(x){
return x < x+1e-15
})
).toBe(true)
});
});
// explore the limits of the floating point ceiling.
Failed after 8 tests and 1 shrinks. rngState:
0879d03f2e184b6c5a; Counterexample: 19.7693584933877;
[ 19.7693584933877 ]
F
Failures:
1) double ceiling x < x + 1e-15
1.1) Expected Object({ counterexample: [ 19.7693584933877 ],
counterexamplestr: '19.7693584933877', shrinks: 1, exc:
false, tests: 8, rngState: '0879d03f2e184b6c5a' }) to be
true.

9. MONKEY TESTING
PROPERTY-BASED TESTING
We generate random values.
Pass one to our function or service to test
Expect, that result of this function satisfy some properties
laws (classical property-based testing); crash
The Infinite Monkey Theorem

10. HISTORY
QuickCheck. Haskell. 1999
• https://en.wikipedia.org/wiki/QuickCheck
• http://www.cse.chalmers.se/~rjmh/QuickCheck/
John Hughes founded QuivQ
• commercial version of QuickCheck for Erlang
• extensions/interfaces for
• C/C++
• WebServices
• Industry specific standards … (appl. Volvo cars)

11. IMPLEMENTATIONS FOR POPULAR LANGUAGES
Scala: scalacheck
http://www.scalacheck.org
Java: junit-quickcheck
http://pholser.github.io/junit-quickcheck/
JavaScript: jsverify
http://jsverify.github.io/
Python: Hypothesis
http://hypothesis.works/
PHP: eris; phpquickchek
https://github.com/giorgiosironi/eris
https://github.com/steos/php-quickcheck
…………………… practically any language.

12. TYPICAL USAGE
Thinks and describe laws
(dependencies between input & output)
Tune input generators
Check such properties in output
Demos:
Programming tasks (sorting, serialization)
Form validation
State checks

13. LET’S CHECK SORTED ALGORITHM
def sort(x:IndexedSeq[T]):IndexedSeq[T]
invariant: length
definition of sorted:
idempotency:

14. LET’S CHECK SORTED ALGORITHM
def sort(x:Vector[T]): Vector[T]
invariant: length
property(“sorted array must the same size of origin”) =
forAll{ (x: Vector[Int]) => x.size == sort(x).size }

15. LET’S CHECK SORTED ALGORITHM
We need to generate:
• n - max number of entries
• sorted vector of the length n
• i < n
• j: i < j < n

16. CUSTOM GENERATOR
val orderedVectorGen = for {
n <- Gen.choose(2,100)
x <- Gen.containerOfN[Vector,Int]
i <- Gen.choose(0,n-2)
j <- Gen.choose(i,n-1)
} yield (n, sort(x), i, j)

17. CUSTOM GENERATOR
val orderedVectorGen = for {
n <- Gen.choose(2,100)
x <- Gen.containerOfN[Vector,Int]
i <- Gen.choose(0,n-2)
j <- Gen.choose(i,n-1)
} yield (n, sort(x), i, j)
property(“sorted array must reflect ordered indexes”) =
forAll(orderedVectorGen){ case (n, x, i, j) => x(i) <= x(j) }
property test can serve as documentation for input and output.

18. LAWS
Ordering[T]:
• reflexivity:
forall{ x:T => x <= x}
• anitsimmetry:
forall{ x:T, y:T =>
x <= y & y <= x ==> x==y }
• transitivity:
• forall{ x:T, y:T, z: T =>
x <= y & y <= z ==> x <= z }
// Laws = set of properties.
// Can be used as specification for type argument.

19. EXAMPLE: DATA VALIDATION
case class Subsriber(id:Long, // must be unique
firstName: String, // max 20 sym, no spaces.
lastName:String // max 20 sym, no space
)
val correctSubscriberGen = for {
id <- arbitrary[Long]
firstName <- Gen.alphaStr suchThat (_.length <= FIRST_NAME_LEN)
lastName <- Gen.alphaStr suchThat (_.length <= LAST_NAME_LEN)
} yield Subscriber(id,firstName, lastName)
val incorrectSubscriberGen = for {
id <- arbitrary[Long]
firstName <- incorrectNameGen(FIRST_NAME_LEN)
………

20. GENERAL QUESTIONS
Is property-based testing useful itself ?
— definitely yes.
— force to think hight-level
Are we steel need example-based testing ?
— probably yes.
• What to manual construct thing, which
• explore some type of bug.
• Dimension Problem.
• N probes can be very tiny part.

21. QUESTIONS ?
THANKS FOR LISTENING
Full example code is available on github:
https://github.com/rssh/property-based-testing-talk
@rssh1
Ruslan Shevchenko
<ruslan@shevchenko.kiev.ua>