What elevates software development into software engineering? In this talk, we’ll make the case that rigorous testing is one of the fundamental ways in which programmers can improve the predictability of the output we produce. We will examine some relatively unknown, or at least not widely adopted, software testing techniques that go beyond basic unit testing to help us deliver complex systems with confidence.

1. SOFTWARE ENGINEERING ⊇
SOFTWARE TESTING

8. "The application of a
systematic, disciplined,
quantifiable approach to the
development, operation, and
maintenance of software"

9. ⊆

10. ⊂

14. GENERIC INVARIANT OR POST
CONDITION OF YOUR CODE, GIVEN
SOME PRECONDITION
“Property-Based Testing in Java”, http://jqwik.net
PROPERTIES

15. DEFINED
PROPERTY-BASED TESTING
▸ It’s still testing
▸ Moves away from programmer-supplied examples
▸ Captures broader invariants ("properties") about programs
▸ Occupies lower part of test pyramid
▸ Why do we care about it?

16. REQUIREMENTS FOR PBT
WHAT DO WE NEED TO IMPLEMENT IT?
▸ Input generation
▸ Assertions about properties that holde
▸ Implies: identification of useful and desirable
properties!
▸ [Optional] Counter-example shrinking

17. EXAMPLE-BASED TESTING
INPUT DOMAIN MAPPED TO PASS/FAIL RESULT

18. PROPERTY-BASED TESTING
INPUT DOMAIN, VALIDITY TESTING FUNCTION PER PROPERTY
vs.

19. THE GRANDDADDY
QUICKCHECK
▸ Built in Haskell, released in 2000
▸ Strongly typed language enables elegant generator/
combinator DSL

20. THE IMITATORS
JVM-BASED LANGUAGE ALTERNATIVES
▸ ScalaCheck
▸ Clojure test.generative
▸ Frege QuickCheck
▸ Java
▸ Theories
▸ JUnit 4: junit-quickcheck
▸ JUnit 5: jqwik

21. class FizzBuzzes {
/**
* @param length desired size of sequence
* @return a FizzBuzz sequence with specified size
* starting from 1, e.g. for {@code length} == 15:
* <code>["1", "2", "Fizz", "4", "Buzz", "Fizz", "7",
* "8", "Fizz", "Buzz", "11", "Fizz", "13", "14",
* "FizzBuzz"]</code>
*/
static List<String> fizzBuzz(int length) {
return fizzBuzz().limit(length)
.collect(Collectors.toList());
}
}

22. @Test
public void produces_empty_sequence() {
assertThat(FizzBuzzes.fizzBuzz(0)).containsExactly();
}
@Test
public void produces_one_element() {
assertThat(FizzBuzzes.fizzBuzz(1)).containsExactly("1");
}
@Test
public void produces_fizzes_and_buzzes() {
assertThat(FizzBuzzes.fizzBuzz(15)).containsExactly(
"1", "2", "Fizz", "4", "Buzz", "Fizz", "7",
"8", "Fizz", "Buzz", "11", "Fizz", "13", "14",
"FizzBuzz");
}

23. @RunWith(JUnitQuickcheck.class)
public class PropertyTest {
}
http://pholser.github.io/junit-quickcheck/

24. class FizzBuzzGenerator extends Generator<List> {
public FizzBuzzGenerator() {
super(List.class);
}
@Override
public List<String> generate(SourceOfRandomness random, GenerationStatus status) {
return FizzBuzzes.fizzBuzz(status.size());
}
@Override
public List<List> doShrink(SourceOfRandomness random, List larger) {
return IntStream.rangeClosed(0, larger.size() - 1)
.mapToObj(maxIdx -> larger.subList(0, maxIdx))
.collect(Collectors.toList());
}
}

25. /**
* Checks that every third element is the literal
* <code>"Fizz"</code>.
*
* @param sequence valid FizzBuzz sequence
*/
@Property
public void fizzes(@From(FizzBuzzGenerator.class)
List<String> sequence) {
for (int i = 3; i <= sequence.size(); i += 3) {
assertThat(sequence.get(i - 1))
.contains(“Fizz");
}
}

26. Verifying property fizzes from demo.PropertyTest_2 with these values:
[demo.PropertyTest_2.fizzes:arg0 = [[]], seed = 7270277137548939525]
Verifying property fizzes from demo.PropertyTest_2 with these values:
[demo.PropertyTest_2.fizzes:arg0 = [[1]], seed = 7270277137548939525]
Verifying property fizzes from demo.PropertyTest_2 with these values:
[demo.PropertyTest_2.fizzes:arg0 = [[1, 2, Fizz, 4, Buzz, Fizz]], seed
= 7270277137548939525]
…
Verifying property fizzes from demo.PropertyTest_2 with these values:
[demo.PropertyTest_2.fizzes:arg0 = [[1, 2, Fizz, 4, Buzz, Fizz, 7, 8,
Fizz, Buzz, 11, Fizz, 13, 14, FizzBuzz, 16, 17, Fizz, 19, Buzz, Fizz,
22, 23, Fizz, Buzz, 26, Fizz, 28, 29, FizzBuzz, 31, 32, Fizz, 34, Buzz,
Fizz, 37, 38, Fizz, Buzz, 41, Fizz, 43, 44, FizzBuzz, 46, 47, Fizz, 49,
Buzz, Fizz, 52, 53, Fizz, Buzz, 56, Fizz, 58, 59, FizzBuzz, 61, 62,
Fizz, 64, Buzz, Fizz, 67, 68, Fizz, Buzz, 71, Fizz, 73, 74, FizzBuzz,
76, 77, Fizz, 79, Buzz, Fizz, 82, 83, Fizz, Buzz, 86]], seed =
7270277137548939525]

27. @RunWith(JUnitQuickcheck.class)
public class IntegerProperties {
@Property(shrink = true)
public void primality(@InRange(minInt = 2) int i) {
assertTrue(String.valueOf(i), IntMath.isPrime(i));
}
}
http://pholser.github.io/junit-quickcheck/

28. java.lang.AssertionError: Property primality falsified.
Original failure message: [422886279]
Original args: [422886279]
Args shrunken to: [4]
Seeds: [8922134899668966991]
http://pholser.github.io/junit-quickcheck/

29. THE DEVELOPER IS FORCED TO THINK ABOUT
WHAT THE CODE SHOULD DO AT A HIGH LEVEL
RATHER THAN GRUNT OUT A FEW UNMOTIVATED
TEST CASES.
Joe Nelson, “The Design and Use of QuickCheck”
THINKING ABOUT PROPERTIES

30. WHERE DOES IT FIT?
Red
✅ Green
Refactor

31. WHERE DOES IT FIT?
Feature
Spot check
Property? ✅

35. http://pitest.org

37. class BrokenReporter {
volatile boolean written = false;
void endReport() {
if (written) {
return;
}
written = true; // must only happen once
}
}

38. class BrokenReporter {
volatile boolean written = false;
AtomicInteger numWrites = new AtomicInteger();
void endReport() {
if (written) {
return;
}
written = true; // must only happen once
numWrites.incrementAndGet();
}
}

39. http://openjdk.java.net/projects/code-tools/jcstress/
@JCStressTest
@State
@Outcome(id = "1", expect = Expect.ACCEPTABLE)
class ReporterFixedStressTest {
private Reporter reporter = new Reporter();
private final Object lock = new Object();
@Actor
public void publisher1(IntResult1 result) {
reporter.endReport();
synchronized (lock) {
result.r1 = Math.max(
result.r1,
reporter.numWrites.get());
}
}
// additional actors omitted
}

40. *** FAILED tests
Strong asserts were violated. Correct implementations
should have no assert failures here.
1 matching test results.
[FAILED] ReporterBrokenStressTest
(JVM args: [])
Observed state Occurrences Expectation
1 42,206 ACCEPTABLE
2 174 FORBIDDEN
http://openjdk.java.net/projects/code-tools/jcstress/

41. class FixedReporter {
AtomicBoolean written = new AtomicBoolean(false);
void endReport() {
if (written.compareAndSet(false, true)) {
// happens only once, lock-free
}
}
}

43. http://jepsen.io

47. https://github.com/javapathfinder/

49. https://github.com/javapathfinder/

50. gov.nasa.jpf.listener.PreciseRaceDetector
race for field AtomicFail@15b.written
Thread-1 at AtomicFail.onCompletion(AtomicFail.java:23)
"written = true;" WRITE: putfield AtomicFail.written
Thread-2 at AtomicFail.onCompletion(AtomicFail.java:18)
"if (written) {" READ: getfield AtomicFail.written

51. thread java.lang.Thread:{id:0,name:main,status:WAITING,priority:5,isDaemon:false,lockCount:0,suspendCount:0}
waiting on: java.lang.Thread@15c
call stack:
at java.lang.Thread.join(Thread.java)
at AtomicFail.main(AtomicFail.java:13)
thread java.lang.Thread:{id:1,name:Thread-1,status:RUNNING,priority:5,isDaemon:false,lockCount:0,suspendCount:0}
call stack:
at AtomicFail.onCompletion(AtomicFail.java:26)
at AtomicFail$$Lambda$0.run(pc 5)
thread java.lang.Thread:{id:2,name:Thread-2,status:RUNNING,priority:5,isDaemon:false,lockCount:0,suspendCount:0}
call stack:
at AtomicFail.onCompletion(AtomicFail.java:18)
at AtomicFail$$Lambda$0.run(pc 5)

52. http://lamport.azurewebsites.net/tla/tla.html

53. ---- MODULE Transfer ----
EXTENDS Naturals, TLC
(* --algorithm transfer
variables alice_account = 10, bob_account = 10,
account_total = alice_account + bob_account;
process Transfer in 1..2
variable money in 1..20;
begin
Transfer:
if alice_account >= money then
A: alice_account := alice_account - money;
bob_account := bob_account + money;
end if;
C: assert alice_account >= 0;
end process
end algorithm *)
MoneyNotNegative == money >= 0
MoneyInvariant == alice_account + bob_account = account_total
====
https://learntla.com

54. https://learntla.com