Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
QuickCheck: 
A Lightweight Tool for Random Testing 
of Haskell Programs 
Koen Claessen, John Hughes ICFP ‘00 
CMSC 737 Sof...
Prelude - Haskell 
● Functional Programming 
○ Functions as First-class Citizens 
○ More Controls over Side-effects 
○ Imp...
Prelude - Haskell 
● Functional Programming 
○ First-class citizen 
■ g . f = λx -> g (f x) 
○ Side effects 
■ nextRandom ...
Prelude - Haskell 
● Strong Static Typing 
○ Reasoning about Types 
■ f1 :: ∀a . a -> a 
■ f2 :: ∀a b . a -> b 
■ f3 :: ∀a...
Prelude - Haskell 
● Strong Static Typing 
○ Reasoning about Types 
■ f1 :: ∀a . a -> a 
● Can only be the identity functi...
Prelude - Haskell 
● Strong Static Typing 
○ f: plus 2 
○ f3: swap the first two element if possible 
map f 
[1,2,3,4] [3,...
Overview - QuickCheck 
● Test Case Generation 
○ Value Generation 
○ Function Generation 
● Output Verification 
○ Describ...
Test Case Generation - Type Class 
● Type class: ad hoc polymorphism support 
○ in Prolog: 
eq(bool). 
eq(char). 
… 
ord(X...
Test Case Generation - Arbitrary 
● newtype Gen a = Gen (Rand -> a) 
○ reads “some computation(Gen) that will give you a” ...
Test Case Generation - Arbitrary 
● newtype Gen a = Gen (Rand -> a) 
○ reads “some computation(Gen) that will give you a” ...
Test Case Generation - Arbitrary 
● Generate Recursive Data Structure 
○ data Tree a = Branch (Tree a) (Tree a) 
| Leaf a ...
Test Case Generation - Arbitrary 
● Generate Recursive Data Structure 
○ data Tree a = Branch (Tree a) (Tree a) 
| Leaf a ...
Test Case Generation - Arbitrary 
● Generate Recursive Data Structure 
○ Limit the size 
○ The notion of “size” is hard to...
Test Case Generation - Arbitrary 
● Generate Recursive Data Structure 
○ data Tree a = Branch (Tree a) (Tree a) 
| Leaf a ...
Test Case Generation - CoArbitrary 
● Generate Functions 
○ Use input value to perturb the random generator Gen b 
■ class...
Output Verification - DSL 
● a Domain Specific Language (DSL) 
embedded in Haskell 
○ Property 
○ Function Equality 
○ Con...
Output Verification - DSL 
prop_RevUnit x = 
reverse [x] == [x] 
prop_RevApp xs ys = 
reverse (xs++ys) == reverse ys++reve...
Output Verification - DSL 
● Conditional Laws (==>) 
○ “A ==> B” is different from “not A || B” 
○ try checking it for 100...
Output Verification - DSL 
● Classify Input: classify, collect 
● Custom Data Generator: forAll 
prop_Insert :: Int -> Pro...
Case studies 
● Unification 
○ Random generated terms are unlikely to be unified 
○ New types are introduced to produce a ...
Discussion 
● On Random Testing 
○ a Haskell test framework written in Haskell 
■ Lightweight 
■ Doesn’t tie to a particul...
Discussion 
● Correctness Criteria 
○ a Haskell test framework written in Haskell 
■ Property Language is much more genera...
Discussion 
● Test Data Generation 
○ a Haskell test framework written in Haskell 
■ another DSL for describing test data ...
Discussion 
● Some Reflections 
○ Formulating formal specification improves our 
understanding of our programs 
○ Three ty...
References 
● Claessen, Koen, and John Hughes. "QuickCheck: a lightweight tool for 
random testing of Haskell programs." A...
Upcoming SlideShare
Loading in …5
×

QuickCheck - Software Testing

Presenting QuickCheck in a software testing class

  • Be the first to comment

  • Be the first to like this

QuickCheck - Software Testing

  1. 1. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs Koen Claessen, John Hughes ICFP ‘00 CMSC 737 Software Testing Fang Cheng javran@cs.umd.edu
  2. 2. Prelude - Haskell ● Functional Programming ○ Functions as First-class Citizens ○ More Controls over Side-effects ○ Implicit Control Flow ● Strong Static Typing ○ Reasoning about Types ○ Rule out “bad programs” at compile time ○ If it compiles, it works!
  3. 3. Prelude - Haskell ● Functional Programming ○ First-class citizen ■ g . f = λx -> g (f x) ○ Side effects ■ nextRandom :: () -> Int ■ nextRandom :: Seed -> (Int, Seed) ○ Implicit control flow ■ sum . map (^2) . filter (< 4) $ [1..10]
  4. 4. Prelude - Haskell ● Strong Static Typing ○ Reasoning about Types ■ f1 :: ∀a . a -> a ■ f2 :: ∀a b . a -> b ■ f3 :: ∀a . [a] -> [a]
  5. 5. Prelude - Haskell ● Strong Static Typing ○ Reasoning about Types ■ f1 :: ∀a . a -> a ● Can only be the identity function ■ f2 :: ∀a b . a -> b ● Impossible ■ f3 :: ∀a . [a] -> [a] ● Theorems for free! [Wadler 1989] ● map f . f3 == f3 . map f
  6. 6. Prelude - Haskell ● Strong Static Typing ○ f: plus 2 ○ f3: swap the first two element if possible map f [1,2,3,4] [3,4,5,6] f3 f3 map f [2,1,3,4] [4,3,5,6]
  7. 7. Overview - QuickCheck ● Test Case Generation ○ Value Generation ○ Function Generation ● Output Verification ○ Describing Desired Properties ● Case studies ● Discussion
  8. 8. Test Case Generation - Type Class ● Type class: ad hoc polymorphism support ○ in Prolog: eq(bool). eq(char). … ord(X) :- eq(X).
  9. 9. Test Case Generation - Arbitrary ● newtype Gen a = Gen (Rand -> a) ○ reads “some computation(Gen) that will give you a” class Arbitrary a where arbitrary :: Gen a instance Arbitrary Int where arbitrary = choose (-20,20) instance (Arbitrary a, Arbitrary b) => Arbitrary (a,b) where arbitrary = do {v1 <- arbitrary; v2 <- arbitrary; return (v1,v2)} We split a random seed (using a primitive function) into two independent seeds, notation “<-” and “return” will take care of that.
  10. 10. Test Case Generation - Arbitrary ● newtype Gen a = Gen (Rand -> a) ○ reads “some computation(Gen) that will give you a” instance (Arbitrary a, Arbitrary b) => Arbitrary (a,b) where arbitrary = liftM2 (,) arbitrary arbitrary -- shorthand: -- liftM f m1 = do { v1 <- m1; return (f v1) } -- liftM2 g m1 m2 = do { v1 <- m1; v2 <- m2; return (g v1 v2) }
  11. 11. Test Case Generation - Arbitrary ● Generate Recursive Data Structure ○ data Tree a = Branch (Tree a) (Tree a) | Leaf a deriving (Show) instance (Arbitrary a) => Arbitrary (Tree a) where arbitrary = frequency [ (1, liftM Leaf arbitrary) , (2, liftM2 Branch arbitrary arbitrary) ]
  12. 12. Test Case Generation - Arbitrary ● Generate Recursive Data Structure ○ data Tree a = Branch (Tree a) (Tree a) | Leaf a deriving (Show) instance (Arbitrary a) => Arbitrary (Tree a) where arbitrary = frequency [ (1, liftM Leaf arbitrary) , (2, liftM2 Branch arbitrary arbitrary) ] ● The generator might not terminate ● Likely to produce huge trees
  13. 13. Test Case Generation - Arbitrary ● Generate Recursive Data Structure ○ Limit the size ○ The notion of “size” is hard to define, leave it to programmers ○ newtype Gen a = Gen (Int -> Rand -> a) ○ sized :: (Int -> Gen a) -> Gen a
  14. 14. Test Case Generation - Arbitrary ● Generate Recursive Data Structure ○ data Tree a = Branch (Tree a) (Tree a) | Leaf a deriving (Show) instance (Arbitrary a) => Arbitrary (Tree a) where arbitrary = sized arbTree arbTree :: Int -> Gen a arbTree 0 = liftM Leaf arbitrary arbTree n = frequency [ (1, liftM Leaf arbitrary) , (4, liftM2 Branch (arbTree (n `div` 2) ) (arbTree (n `div` 2) ) ]
  15. 15. Test Case Generation - CoArbitrary ● Generate Functions ○ Use input value to perturb the random generator Gen b ■ class CoArbitrary a where ■ coarbitrary :: a -> Gen b -> Gen b ○ We should be able to generate an arbitrary b using Gen b ○ Use input value a to perturb Gen b so that we will have a modified Gen b ○ This is a pure function because given the same generator, the output depends merely on input value.
  16. 16. Output Verification - DSL ● a Domain Specific Language (DSL) embedded in Haskell ○ Property ○ Function Equality ○ Conditional Laws ○ Classify Input ○ Custom Data Generator
  17. 17. Output Verification - DSL prop_RevUnit x = reverse [x] == [x] prop_RevApp xs ys = reverse (xs++ys) == reverse ys++reverse xs prop_RevRev xs = reverse (reverse xs) == xs ● Property ● Function Equality (===) type Endo a = a -> a prop_CompAssoc :: Endo Int -> Endo Int -> Endo Int -> Int -> Bool prop_CompAssoc f g h = f . (g . h) === (f . g) . h
  18. 18. Output Verification - DSL ● Conditional Laws (==>) ○ “A ==> B” is different from “not A || B” ○ try checking it for 100 test cases satisfying the condition ○ generate only a limited number of test cases prop_MaxLe :: Int -> Int -> Property prop_MaxLe x y = x <= y ==> max x y == y
  19. 19. Output Verification - DSL ● Classify Input: classify, collect ● Custom Data Generator: forAll prop_Insert :: Int -> Property prop_Insert x = forAll orderedList $ xs -> classify (null xs) “trivial” $ ordered (insert x xs)
  20. 20. Case studies ● Unification ○ Random generated terms are unlikely to be unified ○ New types are introduced to produce a different input distribution ● Lava ○ Provide symbolic input and use external theorem prover ● Pretty Printing ○ Extending Arbitrary type class to include: ○ shrink :: a -> [a]
  21. 21. Discussion ● On Random Testing ○ a Haskell test framework written in Haskell ■ Lightweight ■ Doesn’t tie to a particular implementation ○ Many criteria need reinterpretation ■ Reachable statement? ■ Constraint solving in Haskell program is hard
  22. 22. Discussion ● Correctness Criteria ○ a Haskell test framework written in Haskell ■ Property Language is much more general ● Function Equality (e.g. current vs. known correct version) ● Result Checking (e.g. mathematical properties) ● Conditional Properties (e.g. insert on sorted list) ● Testing High-Order Functions (e.g. function composition) ○ No published work on automatic testing of functional programs against specification
  23. 23. Discussion ● Test Data Generation ○ a Haskell test framework written in Haskell ■ another DSL for describing test data generation ● Grammar inherited from Haskell ● Minimum learning effort ○ Controlling sizes to guarantee termination ■ Need developer to ● Interpret the meaning of “size” ● Specify generators for his/her own new types
  24. 24. Discussion ● Some Reflections ○ Formulating formal specification improves our understanding of our programs ○ Three types of errors divided evenly ■ Errors in test generators ■ Errors in the specification ■ Errors in the program
  25. 25. References ● Claessen, Koen, and John Hughes. "QuickCheck: a lightweight tool for random testing of Haskell programs." Acm sigplan notices 46.4 (2011): 53- 64. ● Wadler, Philip. "Theorems for free!." Proceedings of the fourth international conference on Functional programming languages and computer architecture. ACM, 1989. ● Weyuker, Elaine J. "On testing non-testable programs." The Computer Journal 25.4 (1982): 465-470.

×