Property-based Testing and Generators (Lua code samples)

1. Your systems. Working as one.
Property-based Testing and Generators
Sumant Tambe
7/29/2015

2. Agenda
• Property-based Testing
• The need for generators
• Design/Implementation of the generator library
• Using mathematics for API design
• Value constraints in type definitions
• Reactive generators
• Generators at compile-time (type generators)
8/25/2015 © 2015 RTI 2

3. Why should you care?
• If you write software for living and if you are
serious about any of the following
– Bug-free code
– Productivity
• Modularity
• Reusability
• Extensibility
• Maintainability
– Performance/latency
• Multi-core scalability
– and having fun while doing that!
8/25/2015 © 2015 RTI 3

4. 8/25/2015 © 2015 RTI 4
Source Google images

5. What’s a Property
• Reversing a linked-list twice has no effect
– reverse(reverse(list)) == list
• Compression can make things smaller
– compress(input).size <= input.size
• Round-trip serialization/deserialization has no
effect
– deserialize(serialize(input)) == input
8/25/2015 © 2015 RTI 5

6. Property-based Testing
• Complementary to traditional example-based testing
• Specify post-conditions that must hold no matter
what
• Encourages the programmer to think harder about
the code
• More declarative
• Need data generators to produce inputs
• Free reproducers!
– Good property-based testing frameworks shrink inputs to
minimal automatically
• Famous: Haskell’s QuickCheck, Scala’s ScalaTest
8/25/2015 © 2015 RTI 6

7. A property-test in RefleX
8/25/2015 © 2015 RTI 7
template <class T>
void test_roundtrip_property(const T & in)
{
T out; // empty
DDS::DynamicData dd(...); // empty
reflex::update_dynamicdata(dd, in);
reflex::read_dynamicdata(out, dd)
assert(in == out);
}
• What’s the input data?
• What are the input types?

8. Data Generators
8/25/2015 © 2015 RTI 8
Lua has less syntactic noise than C++. So, code
examples are in Lua

9. Data Generators
8/25/2015 © 2015 RTI 9
• Simplest data generator = math.random
math.randomseed(os.time())
local r = math.random(6)
print(r) -- one of 1, 2, 3, 4, 5, 6
r = math.random(20, 25)
print(r) -- one of 20, 21, 22, 23, 24, 25

10. Data Generators
8/25/2015 © 2015 RTI 10
• A boolean generator
function boolGen()
return math.random(2) > 1
end
• An uppercase character generator
function upperCaseGen() -- A=65, Z=90
return string.char(math.random(65, 90))
end
• An int16 generator
function int16Gen()
return math.random(MIN_INT16, MAX_INT16)
end

11. Data Generators
8/25/2015 © 2015 RTI 11
• A range generator
function rangeGen(lo, hi)
return math.random(lo, hi)
end
• Inconvenient because args must be passed every time
Use state!

12. Closure Recap
8/25/2015 © 2015 RTI 12
• A simple closure
local f = function ()
return boolGen()
end
print(f()) -- true or false
• Another closure
local a = 20
local f = function ()
print(a)
end
f() -- 20
a = 30
f() -- 30

13. Closures capture state
8/25/2015 © 2015 RTI 13
• Back to the range generator
function rangeGen(lo, hi)
return function()
return math.random(lo, hi)
end
end
local f = rangeGen(20, 25)
print(f()) -- one of 20, 21, 22, 23, 24, 25

14. Closures capture state
8/25/2015 © 2015 RTI 14
• The oneOf generator
function oneOfGen(array)
local len = #array –- get length of the array
return function()
return array[math.random(1, len)]
end
end
local days = { “Sun”, “Mon”, “Tue”, “Wed”, “Thu”, “Fri”, “Sat” }
local f = oneOfGen(days)
print(f()) -- one of Sun, Mon, Tue, Wed, Thu, Fri, Sat

15. 8/25/2015 © 2015 RTI 15
“Closures are poor man's
objects; Objects are poor
man's closure”
-- The venerable master Qc Na
Source Google images

16. Closure vs Object
• Is function a sufficient abstraction for any
generator?
function () {
return blah;
}
• Answer depends upon your language
– No. In Lua, C++, …
– Yes. In Haskell
8/25/2015 © 2015 RTI 16

17. The Generator Library
8/25/2015 17
local Gen = require(“generator”)
local rangeGen = Gen.rangeGen(20,25)
for i=1, 5 do
print(rangeGen:generate()) –- 5 values in range(20, 25)
end
Github: https://github.com/rticommunity/rticonnextdds-ddsl
local stepperGen = Gen.stepperGen(1, 10)
for i=1, 5 do
print(stepperGen:generate()) –- 1, 2, 3, 4, 5
end
local infiniteGen = Gen.stepperGen(1, 10, 1, true)
while true do
print(infiniteGen:generate()) –- 1, 2, 3, 4, ...
end

18. The Generator Library
8/25/2015 18
local Gen = require(“generator”)
local charGen = Gen.uppercaseGen()
local maxLen = 64
local strGen = Gen.stringGen(maxLen, charGen)
for i=1, 5 do
print(strGen:generate()) -- 5 uppercase strings upto size 64
end
Github: https://github.com/rticommunity/rticonnextdds-ddsl
sequenceGen is quite similar

19. The Generator library (batteries included)
8/25/2015 © 2015 RTI 19
boolGen posFloatGen emptyGen
charGen posDoubleGen singleGen
wcharGen floatGen constantGen
octetGen doubleGen oneOfGen
shortGen lowercaseGen inOrderGen
int16Gen uppercaseGen stepperGen
int32Gen alphaGen rangeGen
int64Gen alphaNumGen seqGen
uint16Gen printableGen arrayGen
uint32Gen stringGen aggregateGen
uint64Gen nonEmptyStringGen enumGen

20. The Generator “Class”
• Lua does not have “classes” but that’s not important
8/25/2015 © 2015 RTI 20
local Generator = {} –- an object that will serve as a class
function Generator:new(generateImpl)
local o = {}
o.genImpl = generateImpl -- A method assignment
-- some additional metatable manipulation here
return o
end
function Generator:generate()
return self.genImpl()
end
function rangeGen(lo, hi)
return Generator:new(function ()
return math.random(lo, hi)
end)
end

21. 8/25/2015 © 2015 RTI 21
Source Google images

22. Composing Generators
• Producing more complex Generators from one
or more simpler Generators
8/25/2015 © 2015 RTI 22
Generator
map
reduce
append
where
zipMany
concatMap
take
scan

23. Mapping over a generator
8/25/2015 © 2015 RTI 23
local days = { “Sun”, “Mon”, “Tue”, “Wed”, “Thu”, “Fri”, “Sat” }
local fiboGen = Gen.fibonacciGen()
-- 0 1 1 2 3 5 8 ...
local plus1Gen = fiboGen:map(function (i) return i+1 end)
-- 1 2 2 3 4 6 9 ...
local dayGen = plus1Gen:map(function (j) return days[j] end)
-- Sun, Mon, Mon, Tue, Wed, Fri, nil, ...
for i=1, 6 do
print(dayGen:generate()) -- Sun, Mon, Mon, Tue, Wed, Fri
end

24. Zipping generators
8/25/2015 © 2015 RTI 24
local xGen = Gen.stepperGen(0,100)
local yGen = Gen.stepperGen(0,200,2)
local pointGen =
Gen.zipMany(xGen, yGen, function (x, y)
return { x, y }
end)
for i=1, 5 do
print(pointGen:generate()) –- {0,0}, {1,2}, {3,4}, ...
end

25. Zipping generators
8/25/2015 © 2015 RTI 25
local xGen = Gen.stepperGen(0,100)
local yGen = Gen.stepperGen(0,200,2)
local colorGen = Gen.oneOfGen({ “RED”, “GREEN”, “BLUE” })
local sizeGen = Gen.constantGen(30)
local shapeGen =
Gen.zipMany(xGen, yGen, colorGen, sizeGen,
function (x, y, color, size)
return { x = x,
y = y,
color = color,
shapesize = size }
end)
• The mandatory Shapes example

26. Using Data Domain Specific Language (DDSL)
8/25/2015 © 2015 RTI 26
local ShapeType = xtypes.struct{
ShapeType = {
{ x = { xtypes.long } },
{ y = { xtypes.long } },
{ shapesize = { xtypes.long } },
{ color = { xtypes.string(128), xtypes.Key } },
}
}
local shapeGen = Gen.aggregateGen(ShapeType)
for i=1, 5 do
print(shapeGen:generate()) –- output next slide
end
• Use Lua DDSL to define types

27. Five random shapes
8/25/2015 © 2015 RTI 27
COLOR?!

28. Use a Collection of Generators
8/25/2015 © 2015 RTI 28
local ShapeType = xtypes.struct{
ShapeType = {
{ x = { xtypes.long } },
{ y = { xtypes.long } },
{ shapesize = { xtypes.long } },
{ color = { xtypes.string(128), xtypes.Key } },
}
}
local shapeGenLib = {}
shapeGenLib.x = Gen.stepperGen(0, 100)
shapeGenLib.y = Gen.stepperGen(0, 200, 2)
shapeGenLib.color = Gen.oneOf({ "RED", "GREEN", "BLUE" })
shapeGenLib.shapesize = Gen.rangeGen(20, 30)
local shapeGen = Gen.aggregateGen(ShapeType, shapeGenLib)
for i=1, 5 do
print(shapeGen:generate()) –- output next slide
end

29. Five sensible shapes
8/25/2015 © 2015 RTI 29

30. In not so distant future…
8/25/2015 © 2015 RTI 30
local ShapeType = xtypes.struct{
ShapeType = {
{ x = { xtypes.long,
xtypes.constraint{ “[ x | x <- stepperGen(0,100) ]” }}},
{ y = { xtypes.long,
xtypes.constraint{ “[ x*x | x <- stepperGen(0,200,2) ]” }}},
{ size = { xtypes.long,
xtypes.constraint{ “[ x | x <- rangeGen(20,30) ]” }}},
{ color= { xtypes.string(128),
xtypes.constraint{ “[ x | x <- constantGen('BLUE') ]” }}}
}
}
• Direct applicability in
• MEP  Message Emulation Processes (i.e., generate data)
• MCT  Generator expressions may be specified directly in the UI

31. 8/25/2015 © 2015 RTI 31
The Mathematics of the Generator API

32. Generators
• General design principal
– Create a language to solve a problem
– API is the language
– Reuse parsing/compilation of the host language (duh!)
• The “language” of Generators
– Primitive values are basic generators
– Complex values are composite generators
– All APIs result in some generator
• Especially, map, zip, reduce, etc.
– I.e., generators form an algebra
8/25/2015 © 2015 RTI 32

33. Algebra
A first-class abstraction
+
Compositional* API
8/25/2015 © 2015 RTI 33
*Composition based on mathematical concepts

34. 8/25/2015 © 2015 RTI 34
What Math Concepts?
• Algebraic Structures from Category Theory
– Monoid
– Monad
– Functor
– Applicative
– Group
– Endofunctor
• Where to begin?
– Haskell
– But innocent should start with Scala perhaps

35. 8/25/2015 © 2015 RTI 35
Does it have anything to do with DDS?
Yes!
Composable API for DDS exists. It’s Rx4DDS
Also, RPC-over-DDS (futures). Both are humble
beginnings
And remember, sky is the limit

36. Next Items
• Value constraints in type definitions
• Reactive generators
• Generators at compile-time (type generators)
• (Back to) Testing RefleX API properties
8/25/2015 © 2015 RTI 36

37. Constraints in Data Models
• Database Schemas
• XSD Restrictions
8/25/2015 © 2015 RTI 37
CREATE TABLE products (
product_no integer UNIQUE NOT NULL,
name text,
price numeric CHECK (price > 0)
);
<xs:element name="age">
<xs:simpleType>
<xs:restriction base="xs:integer">
<xs:minInclusive value="0"/>
<xs:maxInclusive value="120"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element name="prodid">
<xs:simpleType>
<xs:restriction base="xs:integer">
<xs:pattern value="[0-9][0-9][0-9]"/>
</xs:restriction>
</xs:simpleType>
</xs:element>

38. Constraints in Data Models
• STANAG 4607
– Ground Moving Target Indicator
• Object Constraint Language
– Defined by OMG: part of the UML standard
– Huge!
8/25/2015 © 2015 RTI 38

39. In not so distant future…
8/25/2015 © 2015 RTI 39
local ShapeType = xtypes.struct{
ShapeType = {
{ x = { xtypes.long,
xtypes.constraint{ “[ x | x <- stepperGen(0,100) ]” }}},
{ y = { xtypes.long,
xtypes.constraint{ “[ x*x | x <- stepperGen(0,200,2) ]” }}},
{ size = { xtypes.long,
xtypes.constraint{ “[ x | x <- rangeGen(20,30) ]” }}},
{ color= { xtypes.string(128),
xtypes.constraint{ “[ x | x <- constantGen('BLUE') ]” }}}
}
}
• Direct applicability in
• MEP  Message Emulation Processes (i.e., generate data)
• MCT  Generator expressions may be specified directly in the UI

40. General Syntax for Data Generation
• Mathematical Set Builder Notation
• Suggested Generator Syntax
– [ x | x <- stepperGen() ]
– [ 2*x | x <- fibonacciGen() ]
– [ “ID” .. i | i <- stepperGen(1, 256) ]
• Translated mechanically to the Generator API
– map, flatMap, etc.
– List Comprehension in disguise
• Python, Haskell has it. Lua does not.
8/25/2015 © 2015 RTI 40

41. Value Dependencies
• List Comprehension alone is not sufficient
• What if data members have value dependencies
– o.y = o.x * o.x
– o.len = length(o.somearray)
– o.min = minimum(o.somearray)
– o.max = maximum(o.somearray)
8/25/2015 © 2015 RTI 41

42. Syntax for Value Dependencies
• Generator Syntax seems to work for value
dependencies too
• Just refer to the generator of the parent attribute
– [ a*a | a <- $(x) ]
– [ len(a) | a <- $(somearray) ]
– [ min(point.x) | point <- t,
t <- $(trajectory) ]
(Assuming trajectory is an array of points )
8/25/2015 © 2015 RTI 42

43. Pull Generators aren’t enough
• Pull Generators don’t handle dependencies
well
– At least not without state
– Requires multiple passes through state for every
generated sample
• Seems expensive
– Dependencies must be “interpreted” for every
sample
• I.e., needs to refer type description to do its job
• Dependencies can’t be “compiled” for performance
– A relatively simple and powerful alternative exists
8/25/2015 © 2015 RTI 43

44. How dependencies break
• The ShapeType generator
asks leaf generators to
produce a new value.
• Dependencies are
immediately lost
8/25/2015 © 2015 RTI 44
X
Y
ShapeType
Y = X2
Circle represents a generator not state

45. Reactive Generators
• Work “just like DDS!”
• Propagate values “downstream”
– Push-based instead of pull-based
• Subject-Observer (pub/sub) pattern
– A single value is propagated to one or more dependent
attributes
• Dependent generators take the “right” action
– Such as doubling the value, counting min, etc.
• map, flatmap, reduce etc. work exactly like
pull generators
• Composable
– Generator expressions can be mechanically translated to
reactive generator API
8/25/2015 © 2015 RTI 45

46. Basics of Reactive Generators
8/25/2015 © 2015 RTI 46
class Generator<T>
{
void listen(callback);
};
Where callback is a closure
callback = function (t) {
// provided by the user
}
You can register multiple callbacks

47. Create Listen Push Dispose
Lifecycle of a reactive (push) generator
n n
Create a
Pull
Generator
Subject
transform,
…
Optional

48. Composing Reactive Generators
• Producing more complex Generators from one
or more simpler Generators
8/25/2015 © 2015 RTI 48
ReactGen
map
reduce
append
where
zipMany
concatMap
take
scan

49. Architecture
8/25/2015 © 2015 RTI 49

50. Back to Property-based Testing in
RefleX
8/25/2015 © 2015 RTI 50

51. Round-trip property-test for ShapeType
8/25/2015 © 2015 RTI 51
void test_roundtrip_shape(const ShapeType & in)
{
ShapeType out; // empty
DDS::DynamicData dd(...); // empty
reflex::write_dynamicdata(dd, in);
reflex::read_dynamicdata(out, dd)
assert(in == out);
}
• The assertion must hold for all instances.
void test_many_shapes()
{
auto generator = gen::make_aggregate_gen<ShapeType>();
for (int i = 0;i < 100; ++i) {
test_roundtrip_shapetype(gen.generate());
}
}

52. A generic property-test in RefleX
8/25/2015 © 2015 RTI 52
template <class T>
void test_roundtrip_property(const T & in)
{
T out; // empty
DDS::DynamicData dd(...); // empty
reflex::update_dynamicdata(dd, in);
reflex::read_dynamicdata(out, dd)
assert(in == out);
}
• What are the input types?
• Given a type, how to produce data?

53. Random Types at
Compile-time
8/25/2015 © 2015 RTI 53
Random Numbers at
Compile-time
if we can generate

54. Random Numbers at Compile-time
• Demo
– In C
– Then in C++
8/25/2015 © 2015 RTI 54
Live code

55. Linear Feedback Shift Register (LFSR)
• Simple random number generator
• 4 bit Fibonacci LFSR
• Feedback polynomial = X4 + X3 + 1
• Taps 4 and 3
8/25/2015 © 2015 RTI 55

56. Linear Feedback Shift Register (LFSR)
• Live code feedback polynomial (16 bit)
– X16 + X14 + X13 + X11 + 1
• Taps: 16, 14, 13, and 11
8/25/2015 © 2015 RTI 56

57. Type Generators Demystified
• Random seed as command line #define
• Compile-time random numbers using LFSR
• C++ meta-programming for type synthesis
• std::string generator for type names and
member names
• RefleX to map types to typecode and dynamic data
8/25/2015 © 2015 RTI 57

58. RefleX Property-test Type Generator
• “Good” Numbers
– 31415 (3 nested structs)
– 32415 (51 nested structs)
– 2626 (12 nested structs)
– 10295 (15 nested structs)
– 21525 (41 nested structs)
– 7937 ( 5 nested structs)
– ...
• “Bad” Numbers
– 2152 (test seg-faults) sizeof(tuple) > 2750 KB!
8/25/2015 © 2015 RTI 58
Github: https://github.com/rticommunity/rticonnextdds-reflex/blob/develop/test/generator

59. Thank You!
8/25/2015 © 2015 RTI 59