Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/254wkpw. Aaron Bedra focuses on describing a system as a series of models that can be used to systematically and automatically generate input data and ensure that a code is behaving as expected. Bedra discusses property based testing and how it can help one build more resilient systems and even reduce the time needed to maintain a test suite. Filmed at qconlondon.com. Aaron Bedra is Chief Security Officer at eligible.com. He is the creator of Repsheet, an open source threat intelligence framework. Bedra is the co-author of Programming Clojure, 2nd Edition and a frequent contributor to open source software.

1. QCon London 2016
An Introduction to Property Based Testing
Aaron Bedra
Chief Security Officer, Eligible
@abedra

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/property-based-testing-2016

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon London
www.qconlondon.com

4. Why do we test?
• To better understand what we are building
• To help us think deeper about what we are building
• To ensure the correctness of what we are building
• To help us explore our design*
• To explain to others how our code should work

5. How do we test?
• With compilers (type systems, static analysis, etc)
• Manual testing
• X-Unit style tests
• Property/generative based tests
• Formal modeling

6. How do we test?
• With compilers (type systems, static analysis, etc)
• Manual testing
• X-Unit style tests
• Property/generative based tests
• Formal modeling

7. What is it?

8. An abstraction

9. Property based testing eliminates
the guess work on value and
order of operations testing

10. Magic numbers

11. Instead of specifying
how you specify what

12. Testing over time

13. When we start our test
suite, things are usually
easy to understand

14. public class Basic {
public static Integer calculate(Integer x, Integer y) {
return x + y;
}
}

15. public class BasicTest {
@Test
public void TestCalculate() {
assertEquals(Integer.valueOf(5), Basic.calculate(3, 2));
}
}

16. What other tests might
we write for this code?

17. Like all programs we
start simple

18. But over time things get
more complicated

19. What happens when our simple
calculate function grows to
include an entire domain?

20. Our test suite will undoubtedly
grow, but we have options to
control the growth

21. And also maintain
confidence in our tests

23. By changing our mental
model just a bit we can
cover much more ground

24. Let’s revisit our basic
example

25. public class Basic {
public static Integer calculate(Integer x, Integer y) {
return x + y;
}
}

26. But instead of a unit test,
let’s write a property

27. @RunWith(JUnitQuickcheck.class)
public class BasicProperties {
@Property public void calculateBaseAssumption(Integer x, Integer y) {
Integer expected = x + y;
assertEquals(expected, Basic.calculate(x, y));
}
}
public class BasicTest {
@Test
public void TestCalculate() {
assertEquals(Integer.valueOf(5), Basic.calculate(3, 2));
}
}

28. @RunWith(JUnitQuickcheck.class)
public class BasicProperties {
@Property(trials = 1000000) public void
calculateBaseAssumption(Integer x, Integer y) {
Integer expected = x + y;
assertEquals(expected, Basic.calculate(x, y));
}
}

29. This property isn’t much
different than the unit test
we had before it

30. It’s just one level of
abstraction higher

31. Let’s add a constraint to
our calculator

32. Let’s say that the output
cannot be negative

33. public class Basic {
public static Integer calculate(Integer x, Integer y) {
Integer total = x + y;
if (total < 0) {
return 0;
} else {
return total;
}
}
}
java.lang.AssertionError: Property calculateBaseAssumption falsified for args
shrunken to [0, -679447654]

34. Shrinking

35. @RunWith(JUnitQuickcheck.class)
public class BasicProperties {
@Property public void calculateBaseAssumption(Integer x, Integer y) {
Integer expected = x + y;
assertEquals(expected, Basic.calculate(x, y));
}
}
public class Basic {
public static Integer calculate(Integer x, Integer y) {
Integer total = x + y;
if (total < 0) {
return 0;
} else {
return total;
}
}
}

36. Now we can be more
specific with our property

37. @RunWith(JUnitQuickcheck.class)
public class BasicProperties {
@Property public void calculateBaseAssumption(Integer x, Integer y) {
assumeThat(x, greaterThan(0));
assumeThat(y, greaterThan(0));
assertThat(Basic.calculate(x, y), is(greaterThan(0)));
}
}
java.lang.AssertionError: Property calculateBaseAssumption falsified for args shrunken to [647853159,
1499681379]

38. We could keep going from
here but let’s dive into
some of the concepts

39. Refactoring

40. This is one of my favorite
use cases for invoking
property based testing

41. Legacy code becomes
the model

42. It’s incredibly powerful

43. It ensures you have
exact feature parity

44. Even for unintended
features!

45. Generators

46. You can use them for all
kinds of things

47. Scenario

48. Every route in your web
application

49. You could define
generators based on your
routes

50. And create valid and
invalid inputs for every
endpoint

51. You could run the
generators on every test

52. Or save the output of the
generation for faster
execution

53. Saved execution of
generators can even bring
you to simulation testing

54. There are tons of property
based testing libraries
available

55. But this is a talk in a
functional language track

56. So let’s have some fun

57. Let’s pretend we have
some legacy code

58. Written in C

59. And we want to test it to
make sure it actually
works

60. But there are no
quickcheck libraries
available*

61. Warning! The crypto you are
about to see should not be
attempted at work

62. Caesar’s Cipher

63. Let’s start with our
implementation

64. #include <stdlib.h>
#include <string.h>
#include <ctype.h>
char *caesar(int shift, char *input)
{
char *output = malloc(strlen(input));
memset(output, '0', strlen(input));
for (int x = 0; x < strlen(input); x++) {
if (isalpha(input[x])) {
int c = toupper(input[x]);
c = (((c - 65) + shift) % 26) + 65;
output[x] = c;
} else {
output[x] = input[x];
}
}
return output;
}

65. Next we create a new
implementation to test
against

66. caesar :: Int -> String -> String
caesar k = map f
where
f c
| inRange ('A', 'Z') c = chr $ ord 'A' +
(ord c - ord 'A' + k) `mod` 26
| otherwise = c

67. We now have two
functions that “should” do
the same thing

68. But they aren’t in the
same language

69. Thankfully Haskell has
good FFI support

70. foreign import ccall "caesar.h caesar"
c_caesar :: CInt -> CString -> CString
native_caesar :: Int -> String -> IO String
native_caesar shift input = withCString input $ c_str ->
peekCString(c_caesar (fromIntegral shift) c_str)

71. $ stack exec ghci caesar.hs caesar.so
GHCi, version 7.10.3: http://www.haskell.org/ghc/ :? for help
[1 of 1] Compiling Main ( caesar.hs, interpreted )
Ok, modules loaded: Main.
*Main> caesar 2 "ATTACKATDAWN"
"CVVCEMCVFCYP"
*Main> native_caesar 2 "ATTACKATDAWN"
"CVVCEMCVFCYP"

72. We can now execute our
C code from inside of
Haskell

73. We can use Haskell’s
quickcheck library to
verify our C code

74. First we need to write a
property

75. unsafeEq :: IO String -> String -> Bool
unsafeEq x y = unsafePerformIO(x) == y
genSafeChar :: Gen Char
genSafeChar = elements ['A' .. 'Z']
genSafeString :: Gen String
genSafeString = listOf genSafeChar
newtype SafeString = SafeString { unwrapSafeString :: String } deriving Show
instance Arbitrary SafeString where arbitrary = SafeString <$> genSafeString
equivalenceProperty = forAll genSafeString $ str ->
unsafeEq (native_caesar 2 str) (caesar 2 str)

76. unsafeEq :: IO String -> String -> Bool
unsafeEq x y = unsafePerformIO(x) == y
genSafeChar :: Gen Char
genSafeChar = elements ['A' .. 'Z']
genSafeString :: Gen String
genSafeString = listOf genSafeChar
newtype SafeString = SafeString { unwrapSafeString :: String } deriving Show
instance Arbitrary SafeString where arbitrary = SafeString <$> genSafeString
equivalenceProperty = forAll genSafeString $ str ->
unsafeEq (native_caesar 2 str) (caesar 2 str)

77. unsafeEq :: IO String -> String -> Bool
unsafeEq x y = unsafePerformIO(x) == y
genSafeChar :: Gen Char
genSafeChar = elements ['A' .. 'Z']
genSafeString :: Gen String
genSafeString = listOf genSafeChar
newtype SafeString = SafeString { unwrapSafeString :: String } deriving Show
instance Arbitrary SafeString where arbitrary = SafeString <$> genSafeString
equivalenceProperty = forAll genSafeString $ str ->
unsafeEq (native_caesar 2 str) (caesar 2 str)

78. unsafeEq :: IO String -> String -> Bool
unsafeEq x y = unsafePerformIO(x) == y
genSafeChar :: Gen Char
genSafeChar = elements ['A' .. 'Z']
genSafeString :: Gen String
genSafeString = listOf genSafeChar
newtype SafeString = SafeString { unwrapSafeString :: String } deriving Show
instance Arbitrary SafeString where arbitrary = SafeString <$> genSafeString
equivalenceProperty = forAll genSafeString $ str ->
unsafeEq (native_caesar 2 str) (caesar 2 str)

79. unsafeEq :: IO String -> String -> Bool
unsafeEq x y = unsafePerformIO(x) == y
genSafeChar :: Gen Char
genSafeChar = elements ['A' .. 'Z']
genSafeString :: Gen String
genSafeString = listOf genSafeChar
newtype SafeString = SafeString { unwrapSafeString :: String } deriving Show
instance Arbitrary SafeString where arbitrary = SafeString <$> genSafeString
equivalenceProperty = forAll genSafeString $ str ->
unsafeEq (native_caesar 2 str) (caesar 2 str)

80. unsafeEq :: IO String -> String -> Bool
unsafeEq x y = unsafePerformIO(x) == y
genSafeChar :: Gen Char
genSafeChar = elements ['A' .. 'Z']
genSafeString :: Gen String
genSafeString = listOf genSafeChar
newtype SafeString = SafeString { unwrapSafeString :: String } deriving Show
instance Arbitrary SafeString where arbitrary = SafeString <$> genSafeString
equivalenceProperty = forAll genSafeString $ str ->
unsafeEq (native_caesar 2 str) (caesar 2 str)

81. *Main> quickCheck equivalenceProperty
*** Failed! Falsifiable (after 20 tests):
"QYMSMCWTIXNDFDMLSL"
*Main> caesar 2 "QYMSMCWTIXNDFDMLSL"
"SAOUOEYVKZPFHFONUN"
*Main> native_caesar 2 "QYMSMCWTIXNDFDMLSL"
“SAOUOEYVKZPFHFONUN/Users/abedra/x“

82. #include <stdlib.h>
#include <string.h>
#include <ctype.h>
char *caesar(int shift, char *input)
{
char *output = malloc(strlen(input));
memset(output, '0', strlen(input));
for (int x = 0; x < strlen(input); x++) {
if (isalpha(input[x])) {
int c = toupper(input[x]);
c = (((c - 65) + shift) % 26) + 65;
output[x] = c;
} else {
output[x] = input[x];
}
}
return output;
}

83. We’ve found a memory
handling issue in our C
code!

84. In reality there are more
issues with this code, but our
issue was quickly exposed

85. And easily reproduced

86. Wrapping up

87. Not all testing is created
equal

88. You should use as many
different testing
techniques as you need

89. Remember to think about
the limits of your tools

90. And use tools that help
you achieve your results
more effectively

91. And more efficiently

92. Questions?

93. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/property-
based-testing-2016