QuickCheck is a lightweight tool for randomly testing Haskell programs. It uses type classes like Arbitrary and CoArbitrary to generate random test cases for data types and functions. Properties to check are specified using a domain specific language embedded in Haskell. Case studies show it can find errors in programs like unification and pretty printing. While lightweight, QuickCheck has limitations like difficulty testing non-terminating programs and formal specifications are still needed to fully check programs.

1. QuickCheck:
A Lightweight Tool for Random Testing
of Haskell Programs
Koen Claessen, John Hughes ICFP ‘00
CMSC 737 Software Testing
Fang Cheng javran@cs.umd.edu

2. Prelude - Haskell
● Functional Programming
○ Functions as First-class Citizens
○ More Controls over Side-effects
○ Implicit Control Flow
● Strong Static Typing
○ Reasoning about Types
○ Rule out “bad programs” at compile time
○ If it compiles, it works!

3. Prelude - Haskell
● Functional Programming
○ First-class citizen
■ g . f = λx -> g (f x)
○ Side effects
■ nextRandom :: () -> Int
■ nextRandom :: Seed -> (Int, Seed)
○ Implicit control flow
■ sum . map (^2) . filter (< 4) $ [1..10]

4. Prelude - Haskell
● Strong Static Typing
○ Reasoning about Types
■ f1 :: ∀a . a -> a
■ f2 :: ∀a b . a -> b
■ f3 :: ∀a . [a] -> [a]

5. Prelude - Haskell
● Strong Static Typing
○ Reasoning about Types
■ f1 :: ∀a . a -> a
● Can only be the identity function
■ f2 :: ∀a b . a -> b
● Impossible
■ f3 :: ∀a . [a] -> [a]
● Theorems for free! [Wadler 1989]
● map f . f3 == f3 . map f

6. Prelude - Haskell
● Strong Static Typing
○ f: plus 2
○ f3: swap the first two element if possible
map f
[1,2,3,4] [3,4,5,6]
f3 f3
map f
[2,1,3,4] [4,3,5,6]

7. Overview - QuickCheck
● Test Case Generation
○ Value Generation
○ Function Generation
● Output Verification
○ Describing Desired Properties
● Case studies
● Discussion

8. Test Case Generation - Type Class
● Type class: ad hoc polymorphism support
○ in Prolog:
eq(bool).
eq(char).
…
ord(X) :- eq(X).

9. Test Case Generation - Arbitrary
● newtype Gen a = Gen (Rand -> a)
○ reads “some computation(Gen) that will give you a”
class Arbitrary a where
arbitrary :: Gen a
instance Arbitrary Int where
arbitrary = choose (-20,20)
instance (Arbitrary a, Arbitrary b) => Arbitrary (a,b) where
arbitrary = do {v1 <- arbitrary; v2 <- arbitrary; return (v1,v2)}
We split a random seed (using a primitive function) into two independent
seeds, notation “<-” and “return” will take care of that.

10. Test Case Generation - Arbitrary
● newtype Gen a = Gen (Rand -> a)
○ reads “some computation(Gen) that will give you a”
instance (Arbitrary a, Arbitrary b) => Arbitrary (a,b) where
arbitrary = liftM2 (,) arbitrary arbitrary
-- shorthand:
-- liftM f m1 = do { v1 <- m1; return (f v1) }
-- liftM2 g m1 m2 = do { v1 <- m1; v2 <- m2; return (g v1 v2) }

11. Test Case Generation - Arbitrary
● Generate Recursive Data Structure
○ data Tree a = Branch (Tree a) (Tree a)
| Leaf a
deriving (Show)
instance (Arbitrary a) => Arbitrary (Tree a) where
arbitrary = frequency
[ (1, liftM Leaf arbitrary)
, (2, liftM2 Branch arbitrary arbitrary)
]

12. Test Case Generation - Arbitrary
● Generate Recursive Data Structure
○ data Tree a = Branch (Tree a) (Tree a)
| Leaf a
deriving (Show)
instance (Arbitrary a) => Arbitrary (Tree a) where
arbitrary = frequency
[ (1, liftM Leaf arbitrary)
, (2, liftM2 Branch arbitrary arbitrary)
]
● The generator might not terminate
● Likely to produce huge trees

13. Test Case Generation - Arbitrary
● Generate Recursive Data Structure
○ Limit the size
○ The notion of “size” is hard to define, leave it to
programmers
○ newtype Gen a = Gen (Int -> Rand -> a)
○ sized :: (Int -> Gen a) -> Gen a

14. Test Case Generation - Arbitrary
● Generate Recursive Data Structure
○ data Tree a = Branch (Tree a) (Tree a)
| Leaf a
deriving (Show)
instance (Arbitrary a) => Arbitrary (Tree a) where
arbitrary = sized arbTree
arbTree :: Int -> Gen a
arbTree 0 = liftM Leaf arbitrary
arbTree n = frequency
[ (1, liftM Leaf arbitrary)
, (4, liftM2 Branch (arbTree (n `div` 2) )
(arbTree (n `div` 2) )
]

15. Test Case Generation - CoArbitrary
● Generate Functions
○ Use input value to perturb the random generator Gen b
■ class CoArbitrary a where
■ coarbitrary :: a -> Gen b -> Gen b
○ We should be able to generate an arbitrary b using Gen b
○ Use input value a to perturb Gen b so that we will have a modified
Gen b
○ This is a pure function because given the same generator, the output
depends merely on input value.

16. Output Verification - DSL
● a Domain Specific Language (DSL)
embedded in Haskell
○ Property
○ Function Equality
○ Conditional Laws
○ Classify Input
○ Custom Data Generator

17. Output Verification - DSL
prop_RevUnit x =
reverse [x] == [x]
prop_RevApp xs ys =
reverse (xs++ys) == reverse ys++reverse xs
prop_RevRev xs =
reverse (reverse xs) == xs
● Property
● Function Equality (===)
type Endo a = a -> a
prop_CompAssoc :: Endo Int -> Endo Int -> Endo Int -> Int -> Bool
prop_CompAssoc f g h = f . (g . h) === (f . g) . h

18. Output Verification - DSL
● Conditional Laws (==>)
○ “A ==> B” is different from “not A || B”
○ try checking it for 100 test cases satisfying the
condition
○ generate only a limited number of test cases
prop_MaxLe :: Int -> Int -> Property
prop_MaxLe x y = x <= y ==> max x y == y

19. Output Verification - DSL
● Classify Input: classify, collect
● Custom Data Generator: forAll
prop_Insert :: Int -> Property
prop_Insert x =
forAll orderedList $ xs ->
classify (null xs) “trivial” $
ordered (insert x xs)

20. Case studies
● Unification
○ Random generated terms are unlikely to be unified
○ New types are introduced to produce a different input distribution
● Lava
○ Provide symbolic input and use external theorem prover
● Pretty Printing
○ Extending Arbitrary type class to include:
○ shrink :: a -> [a]

21. Discussion
● On Random Testing
○ a Haskell test framework written in Haskell
■ Lightweight
■ Doesn’t tie to a particular implementation
○ Many criteria need reinterpretation
■ Reachable statement?
■ Constraint solving in Haskell program is hard

22. Discussion
● Correctness Criteria
○ a Haskell test framework written in Haskell
■ Property Language is much more general
● Function Equality (e.g. current vs. known correct version)
● Result Checking (e.g. mathematical properties)
● Conditional Properties (e.g. insert on sorted list)
● Testing High-Order Functions (e.g. function composition)
○ No published work on automatic testing of functional
programs against specification

23. Discussion
● Test Data Generation
○ a Haskell test framework written in Haskell
■ another DSL for describing test data generation
● Grammar inherited from Haskell
● Minimum learning effort
○ Controlling sizes to guarantee termination
■ Need developer to
● Interpret the meaning of “size”
● Specify generators for his/her own new types

24. Discussion
● Some Reflections
○ Formulating formal specification improves our
understanding of our programs
○ Three types of errors divided evenly
■ Errors in test generators
■ Errors in the specification
■ Errors in the program

25. References
● Claessen, Koen, and John Hughes. "QuickCheck: a lightweight tool for
random testing of Haskell programs." Acm sigplan notices 46.4 (2011): 53-
64.
● Wadler, Philip. "Theorems for free!." Proceedings of the fourth international
conference on Functional programming languages and computer
architecture. ACM, 1989.
● Weyuker, Elaine J. "On testing non-testable programs." The Computer
Journal 25.4 (1982): 465-470.