Quick dive into property-based testing and FsCheck library. Presented in 2018 on SoftServe .NET Community.

1. Increase the quality of your code
with property-based testing and FsCheck
Oleksandr Valetskyy
Technical Leader
SoftServe

2. About me

3. Introduction
Property-based testing
101
FsCheck testing tool
Properties patterns

4. Introduction
Agile manifesto emphasizes
- Frequent delivery
- Consistent development
pace
- Accommodating
requirements change
- Working software
- Customer satisfaction

5. Introduction
Agile manifesto emphasizes
- Frequent delivery
- Consistent development
pace
- Accommodating
requirements change
- Working software
- Customer satisfaction

6. Software bugs cost the U.S.
economy $59.5 billion
annually. More than a third
of this cost could be
avoided, if better software
testing was performed
(NIST, 2002)
The flipside

7. Software Fail Watch: 2016 in Review
https://www.tricentis.com/resource-assets/software-fail-
watch-2016/

8. During 2016, Raygun’s crash and error
reporting software ingested 7.7 billion
errors in total, which averages ~21
million per day.
Some of larger customers sent 50 million
crashes per month.

9. Software quality
Agile Adoption: Improving Software Quality
https://dzone.com/refcardz/agile-adoption-improving

10. Software testing verifies SUT (system-under-test)
• meets the requirements that guided its design
and development
• responds correctly to all kinds of inputs
• performs its functions within an acceptable
time,
• is sufficiently usable
• can be installed and run in its intended
environments, and
• achieves the general result its stakeholders
desire.

11. Automated tests
• By far the most commonly
used approach to ensure
software quality
• Easily cover base flows and
edge cases in requirements
• Can be run unattended and
on schedule, can verify
something manual testing
can’t (e.g. load test)
• Easy to maintain* and
expand

12. Automated testing drawbacks
• Initial investment - can add up
to 50% of development time
• Writing tests upfront as best
practices suggest is not
possible in most of the cases
• Tests failures do not always
give 100% confidence there are
problems with BL
• Implementing tests may be not
as exciting as other activities^^

13. Not us, definitely?

14. Example: shopping with discounts

17. Covering functionality with unit tests

19. Plot Twist

20. Yellow Sticker Discount

21. ┬──┬ ノ( ゜-゜ノ)
Property-based testing to the rescue!
Instead of defining actual VALUES in tests
and compare them we can verify
PROPERTIES of the rules
Arguments can be generated automatically
and be completely random

22. Yellow Sticker Discount Property-Based Tests

23. History
QuickCheck is a combinator
library originally written in Haskell,
designed to assist in software
testing by generating test cases for test
suites.
In QuickCheck the programmer writes
assertions about logical properties that
a function should fulfill. Then
QuickCheck attempts to generate a test
case that falsifies these assertions.
Once such a test case is found,
QuickCheck tries to reduce it to a
minimal failing subset by removing or
simplifying input data that are not
needed to make the test fail.
The project was started in 1999.
Currently there exist implementations
of QuickCheck for 30+ modernhttp://www.eecs.northwestern.edu/~robby/courses/395-495-2009-fall/quick.pdf

24. Property-based testing 101
Property-based testing (PBT) is a testing technique
that
tries to falsify a given property by generating
random input data and verifying the expected
behavior.
Properties can range from simple algebraic
equations to complex state machine models.
Like in all model-based testing techniques the
properties serve as a source for test-case
generation as well

25. Value-based vs. property-based testing
Assign inputs for case #1
Do the processing
Assert result
Assign inputs for case #2
Do the processing
Assert result
Assign inputs for case #3
Do the processing
Assert result
Describe the inputs
Describe the result
Profit!

26. FsCheck
.NET implementation of QuickCheck
written in F#
Works with specifications that are
expressed in F#, C# or VB.
FsCheck provides combinators to define
properties, observe the distribution of test
data, and define test data generators.
Used by:
FSharpx.Extras LinqOptimizer Neutronium
https://fscheck.github.io

27. FsCheck Generators
Generator – FsCheck.Gen<T> - tries to
create example inputs that tend
to grow larger. It begins by
generating small test cases, and
gradually increases the size as testing
progresses.
There are numerous built-in
generators for typical input types
(ints, floats, strings, lists, etc.) and
also possibility to generate custom
generators.

28. Built-in Generators
• Gen.Constant – always returns the same value
• Gen.Choose – generates singular integer values between a
minimum and maximum value
• Gen.Elements – generates singular values drawn from a
collection of possible values
• Gen.Map – gives ability to produce generator of one type
from another type
• Gen.ListOfLength – generate lists from individual value
generators
• Gen.Shuffle – a generator that generates a random
permutation of a given finite sequence.
• Gen.Two, Gen.Three, Gen.Four – generating tuples
• Gen.Filter – gives ability to apply custom rules for
generating values

29. Custom recursive tree generator

30. FsCheck Shrinkers
When a property fails, FsCheck
automatically displays a minimal
counter example.
This is done with the help of
Shrinker T ->
IEnumerable<T> function that
works hand-in-hand with the
generator such that when a
failure is found, the input that
caused the failure is then
'shrunk' trying to find a smaller
input that also causes a failure.

31. FsCheck defines default test data generators and
shrinkers for some often used types, for example
unit, bool, byte, int, float, char, string, DateTime,
lists, array 1D and 2D, Set, Map, objects and
functions from and to any of the above.
Furthermore, by using reflection, FsCheck can derive
default implementations of record types,
discriminated unions, tuples, enums and basic classes
in terms of any primitive types that are defined (either
in FsCheck or by you).
FsCheck Shrinkers

32. 3D vector shrinker example

33. Examples from OSS: Faker
faker-csharp/blob/master/tests/Faker.Models.Tests/ArrayHelperSpecs.cs
[TestFixture(Description = "Validates our extension methods for working with arrays")]
public class ArrayHelperSpecs
{
[Test(Description = "Ensure that our shuffle function works over a range of
intervals")]
public void Shuffled_lists_should_never_match_original()
{
Prop.ForAll<int[]>(original =>
{
var shuffle = original.Shuffle().ToArray();
return (!original.SequenceEqual(shuffle))
.When(original.Length > 1 && original.Distinct().Count() > 1)
.Label($"Expected shuffle({string.Join(",", shuffle)}) to be "
+ $"different than original({string.Join(",", original)})")
.And(original.All(x => shuffle.Contains(x))
.Label($"Expected shuffle({string.Join(",", shuffle)}) to contain"
+ $" same items as original({string.Join(",", original)})"));
}).QuickCheckThrowOnFailure();
}
}

34. protected JavascriptFactoryBulk_Tests(IBasicWindowLessHTMLEngineProvider testEnvironment,
ITestOutputHelper output) : base(testEnvironment, output)
{
Arb.Register<TestDataGenerator>();
}
public class TestDataGenerator
{
private static Gen<DateTime> DateTimeGenerator => Gen.zip3(Gen.Choose(1800, 2020),
Gen.Choose(0, 365), Gen.zip(Gen.Choose(0, 24), Gen.Choose(0, 3600)))
.Select(t => new DateTime(t.Item1, 1, 1, 0, 0, 0, DateTimeKind.Utc)
.AddDays(t.Item2).AddHours(t.Item3.Item1).AddSeconds(t.Item3.Item2));
private static Gen<T> DefaultGenerator<T>() => Arb.From<T>().Generator;
private static Gen<object> DefaultObjectGenerator<T>() =>
DefaultGenerator<T>().Select(t => (object)t);
private static Gen<object> ObjectGenerator =>
Gen.OneOf(DefaultObjectGenerator<Boolean>(), DefaultObjectGenerator<int>(),
DefaultObjectGenerator<double>(), DefaultGenerator<string>()
.Where(s => !string.IsNullOrEmpty(s))
.Select(t => (object)t), DateTimeGenerator.Select(t => (object)t));
public static Arbitrary<DateTime> ArbitraryDateTime() => Arb.From(DateTimeGenerator);
public static Arbitrary<object> ArbitraryObject() => Arb.From(ObjectGenerator);
}
Examples from OSS: Neutronium
NeutroniumCore/Neutronium/Tests/UniversalTests/Tests.Universal.WebBrowserEngineTests/JavascriptFa

35. [Theory]
[InlineData(false)]
[InlineData(true)]
[InlineData(10)]
[InlineData(0.5D)]
[InlineData(-1)]
[InlineData(99999.95)]
[InlineData(2.333333333)]
[InlineData(-0.66666666666666663)]
[InlineData(1.3333333333333333)]
[InlineData(double.NegativeInfinity)]
[InlineData("tititi")]
[InlineData("")]
[InlineData("n")]
[InlineData("rn")]
[InlineData("r")]
[InlineData("'")]
public void CreateBasics_BasicTypes_Returns_Correct_Value(object value)
{
TestConversion(value);
}
[Property]
public Property CreateBasics_Int_Create_Correct_Objects()
{
return CreateBasics_Returns_Correct_Value<int>();
}
[Property(Skip = "Decimal precision is not supported")]
public Property CreateBasics_Decimal_Create_Correct_Objects()
{
return CreateBasics_Returns_Correct_Value<decimal>();
}
[Property]
public Property CreateBasics_Double_Create_Correct_Objects()
{
return CreateBasics_Returns_Correct_Value<double>();
}
[Property]
public Property CreateBasics_DateTime_Create_Correct_Objects()
{
return CreateBasics_Returns_Correct_Value<DateTime>();
}
[Property]
public Property CreateBasics_String_Create_Correct_Objects()
{
Func<string, bool> when = Always<string>;
if (!SupportStringEmpty) when = StringIsNotEmpty;
return CreateBasics_Returns_Correct_Value(when);
}
Neutronium (cont’d)

36. Examples from OSS: LinqOptimizer
LinqOptimizer/tests/LinqOptimizer.Tests.CSharp/QueryTests.cs
[TestFixture]
public class QueryTests
{
[Test]
public void Select()
{
Func<IEnumerable<object>, bool> f = xs =>
{
var x = xs.AsQueryExpr().Select(n => n.ToString()).Run();
var y = xs.Select(n => n.ToString());
return Enumerable.SequenceEqual(x, y); };
Spec.ForAny<TestInput<object>>(xs =>
TestInput<object>.RunTestFunc<object>(f, xs))
.QuickCheckThrowOnFailure(); }
[Test]
public void Pipelined()
{
Spec.ForAny<int[]>(xs =>
{
var x = xs.AsQueryExpr().Where(n => n % 2 == 0)
.Select(n => n * 2).Select(n => n.ToString()).Select(n => n + "!").Run();
var y = xs .Where(n => n % 2 == 0)
.Select(n => n * 2) .Select(n => n.ToString()).Select(n => n + "!");
return Enumerable.SequenceEqual(x, y);
}).QuickCheckThrowOnFailure();
}
}

37. .ToArbitrary() – constructs an arbitrary instance
from generator
.Classify() – classifies test cases. Test cases
satisfying the condition are assigned the
classification given.
.Label () – adds the given label to the property.
The labels of a failing sub-property are displayed
when it fails.
.Sample (int size) – Generates n values of the
given size.
.VerboseCheckThrowOnFailure() – throws an
exception with the error message if the test fails or
is exhausted
Other points of interest

38. Caveat

39. [Property(QuietOnSuccess = false, MaxTest = 1000)]
public bool Test_To_Be_Run_Thousand_Times()
{
return true;
}
Configuring FsCheck

40. public class CapturingOutput
{
private readonly ITestOutputHelper _testOutputHelper;
public CapturingOutput(ITestOutputHelper testOutputHelper)
{
_testOutputHelper = testOutputHelper;
}
[Fact]
public void CapturingOutput_Example()
{
Prop.ForAll(Arb.Default.Bool(), i => i)
.VerboseCheckThrowOnFailure(_testOutputHelper);
}
}
Capturing output

41. public class ReplayExample
{
[Fact]
public void Replay_Example()
{
Prop.ForAll(Arb.Default.Bool(), i => i)
.Check(new Configuration
{
Replay = Random.StdGen.NewStdGen(487723102, 296400248)
});
}
}
Replaying failed test

42. Quick summary
Not a silver bullet!
Property-based testing is yet another approach to
software testing
It may look as pure magic at first, but can help in
cases when conventional tests are hard to
implement or not efficient
Does not substitute value-based testing, but
complements it nicely
Gives another perspective and mindset during
coding and verification

43. Philosophical detour
There are two ways of constructing a software design:
One way is to make it so simple that there are obviously no
deficiencies, and the other way is to make it so complicated that
there are no obvious deficiencies.
The first method is far more difficult. It demands the same skill,
devotion, insight, and even inspiration as the discovery of the
simple physical laws which underlie the complex phenomena of
nature.
Charles Antony Richard Hoare (Tony Hoare)
1980 Turing Award Lecture

44. Properties patterns
“Different paths, same destination”
Combining operations in different
orders, but getting the same result
Examples:
Addition commutativity
Sorting lists
List reversal

45. Properties patterns
“There and back again”
Combining some operation with its
inverse should produce the source
input
Examples:
Any inverse operation
Addition/substraction
Write/read
Serialize/deserialize
Encipher/decipher
Set/get
Insert/contains
Create/exists

46. Properties patterns
“Some things never change”
Checking some invariant that is
preserved during transformation
Examples:
Size of collection
Contents of a collection
Measures proportion during scaling

47. Properties patterns
“The more things change, the more
they stay the same”
Idempotence property – doing the
same operation twice should be the
same as doing it once
Examples:
Database filtering
Database updates
Message processing
SOA?

48. Properties patterns
“Solve a smaller problem first”
Structural induction – if something
can be broken into smaller parts
and we prove some property is
true for all of them, we can prove
the property is true for the whole
structure
Examples:
Operations with recursive
structures such as lists and trees

49. Properties patterns
“Hard to prove, easy to verify”
Verifying results of some algorithm
is way easier than finding the
solution
Examples:
Shortest path in maze vs. valid path
Sort list/verify it is sorted
Prime number factorization
String tokenizer

50. Properties patterns
“The test oracle” aka “Two heads are
better than one”
Using alternate (simplified*) version
of an algorithm or process (a “test
oracle”) to check results
Examples:
Optimized high-performance
algorithm vs. brute force version
Parallel/concurrent algorithm vs.
linear/single-thread version
Data generators

52. An introduction to property-based testing Or, why you should be using FsCheck and QuickCheck
https://fsharpforfunandprofit.com/posts/property-based-testing/
The lazy programmer's guide to writing 1000's of tests: An introduction to property based testing
https://skillsmatter.com/skillscasts/6432-the-lazy-programmers-guide-to-writing-1000s-of-
tests-an-introduction-to-property-based-testing
FsCheck
https://fscheck.github.io/FsCheck/index.html
QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs by Koen Claessen and John
Hughes
http://www.eecs.northwestern.edu/~robby/courses/395-495-2009-fall/quick.pdf
Software Fail Watch: 2016 in Review
https://www.tricentis.com/resource-assets/software-fail-watch-2016/
Property-based Testing with FsCheck by Deriving Properties from Business Rule Models. IEEE, 2016.
http://truconf.ist.tugraz.at/index.php/2016/03/04/publication-at-a-most-2016/
Agile Adoption: Improving Software Quality. Reducing Bugs and Improving Design
https://dzone.com/refcardz/agile-adoption-improving
John Hughes: Race Conditions, Distribution, Interactions — Testing the Hard Stuff and Staying Sane
https://vimeo.com/68383317
Discounts and allowances wiki article
https://en.wikipedia.org/wiki/Discounts_and_allowances#Dealing_with_quantity
Neutronium Github
https://github.com/NeutroniumCore/Neutronium
Real World Haskell by Bryan O'Sullivan, Don Stewart, and John Goerzen
http://book.realworldhaskell.org/read/testing-and-quality-assurance.html
Exploring Test-Driven Development with QuickCheck
http://www.natpryce.com/articles/000795.html
Unit Test Boilerplate Generator
References:

54. Thank you!