Successfully reported this slideshow.
Upcoming SlideShare
×

# QuickCheck - Software Testing

Presenting QuickCheck in a software testing class

• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

• Be the first to like this

### QuickCheck - Software Testing

1. 1. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs Koen Claessen, John Hughes ICFP ‘00 CMSC 737 Software Testing Fang Cheng javran@cs.umd.edu
2. 2. Prelude - Haskell ● Functional Programming ○ Functions as First-class Citizens ○ More Controls over Side-effects ○ Implicit Control Flow ● Strong Static Typing ○ Reasoning about Types ○ Rule out “bad programs” at compile time ○ If it compiles, it works!
3. 3. Prelude - Haskell ● Functional Programming ○ First-class citizen ■ g . f = λx -> g (f x) ○ Side effects ■ nextRandom :: () -> Int ■ nextRandom :: Seed -> (Int, Seed) ○ Implicit control flow ■ sum . map (^2) . filter (< 4) \$ [1..10]
4. 4. Prelude - Haskell ● Strong Static Typing ○ Reasoning about Types ■ f1 :: ∀a . a -> a ■ f2 :: ∀a b . a -> b ■ f3 :: ∀a . [a] -> [a]
5. 5. Prelude - Haskell ● Strong Static Typing ○ Reasoning about Types ■ f1 :: ∀a . a -> a ● Can only be the identity function ■ f2 :: ∀a b . a -> b ● Impossible ■ f3 :: ∀a . [a] -> [a] ● Theorems for free! [Wadler 1989] ● map f . f3 == f3 . map f
6. 6. Prelude - Haskell ● Strong Static Typing ○ f: plus 2 ○ f3: swap the first two element if possible map f [1,2,3,4] [3,4,5,6] f3 f3 map f [2,1,3,4] [4,3,5,6]
7. 7. Overview - QuickCheck ● Test Case Generation ○ Value Generation ○ Function Generation ● Output Verification ○ Describing Desired Properties ● Case studies ● Discussion
8. 8. Test Case Generation - Type Class ● Type class: ad hoc polymorphism support ○ in Prolog: eq(bool). eq(char). … ord(X) :- eq(X).
9. 9. Test Case Generation - Arbitrary ● newtype Gen a = Gen (Rand -> a) ○ reads “some computation(Gen) that will give you a” class Arbitrary a where arbitrary :: Gen a instance Arbitrary Int where arbitrary = choose (-20,20) instance (Arbitrary a, Arbitrary b) => Arbitrary (a,b) where arbitrary = do {v1 <- arbitrary; v2 <- arbitrary; return (v1,v2)} We split a random seed (using a primitive function) into two independent seeds, notation “<-” and “return” will take care of that.
10. 10. Test Case Generation - Arbitrary ● newtype Gen a = Gen (Rand -> a) ○ reads “some computation(Gen) that will give you a” instance (Arbitrary a, Arbitrary b) => Arbitrary (a,b) where arbitrary = liftM2 (,) arbitrary arbitrary -- shorthand: -- liftM f m1 = do { v1 <- m1; return (f v1) } -- liftM2 g m1 m2 = do { v1 <- m1; v2 <- m2; return (g v1 v2) }
11. 11. Test Case Generation - Arbitrary ● Generate Recursive Data Structure ○ data Tree a = Branch (Tree a) (Tree a) | Leaf a deriving (Show) instance (Arbitrary a) => Arbitrary (Tree a) where arbitrary = frequency [ (1, liftM Leaf arbitrary) , (2, liftM2 Branch arbitrary arbitrary) ]
12. 12. Test Case Generation - Arbitrary ● Generate Recursive Data Structure ○ data Tree a = Branch (Tree a) (Tree a) | Leaf a deriving (Show) instance (Arbitrary a) => Arbitrary (Tree a) where arbitrary = frequency [ (1, liftM Leaf arbitrary) , (2, liftM2 Branch arbitrary arbitrary) ] ● The generator might not terminate ● Likely to produce huge trees
13. 13. Test Case Generation - Arbitrary ● Generate Recursive Data Structure ○ Limit the size ○ The notion of “size” is hard to define, leave it to programmers ○ newtype Gen a = Gen (Int -> Rand -> a) ○ sized :: (Int -> Gen a) -> Gen a
14. 14. Test Case Generation - Arbitrary ● Generate Recursive Data Structure ○ data Tree a = Branch (Tree a) (Tree a) | Leaf a deriving (Show) instance (Arbitrary a) => Arbitrary (Tree a) where arbitrary = sized arbTree arbTree :: Int -> Gen a arbTree 0 = liftM Leaf arbitrary arbTree n = frequency [ (1, liftM Leaf arbitrary) , (4, liftM2 Branch (arbTree (n `div` 2) ) (arbTree (n `div` 2) ) ]
15. 15. Test Case Generation - CoArbitrary ● Generate Functions ○ Use input value to perturb the random generator Gen b ■ class CoArbitrary a where ■ coarbitrary :: a -> Gen b -> Gen b ○ We should be able to generate an arbitrary b using Gen b ○ Use input value a to perturb Gen b so that we will have a modified Gen b ○ This is a pure function because given the same generator, the output depends merely on input value.
16. 16. Output Verification - DSL ● a Domain Specific Language (DSL) embedded in Haskell ○ Property ○ Function Equality ○ Conditional Laws ○ Classify Input ○ Custom Data Generator
17. 17. Output Verification - DSL prop_RevUnit x = reverse [x] == [x] prop_RevApp xs ys = reverse (xs++ys) == reverse ys++reverse xs prop_RevRev xs = reverse (reverse xs) == xs ● Property ● Function Equality (===) type Endo a = a -> a prop_CompAssoc :: Endo Int -> Endo Int -> Endo Int -> Int -> Bool prop_CompAssoc f g h = f . (g . h) === (f . g) . h
18. 18. Output Verification - DSL ● Conditional Laws (==>) ○ “A ==> B” is different from “not A || B” ○ try checking it for 100 test cases satisfying the condition ○ generate only a limited number of test cases prop_MaxLe :: Int -> Int -> Property prop_MaxLe x y = x <= y ==> max x y == y
19. 19. Output Verification - DSL ● Classify Input: classify, collect ● Custom Data Generator: forAll prop_Insert :: Int -> Property prop_Insert x = forAll orderedList \$ xs -> classify (null xs) “trivial” \$ ordered (insert x xs)
20. 20. Case studies ● Unification ○ Random generated terms are unlikely to be unified ○ New types are introduced to produce a different input distribution ● Lava ○ Provide symbolic input and use external theorem prover ● Pretty Printing ○ Extending Arbitrary type class to include: ○ shrink :: a -> [a]
21. 21. Discussion ● On Random Testing ○ a Haskell test framework written in Haskell ■ Lightweight ■ Doesn’t tie to a particular implementation ○ Many criteria need reinterpretation ■ Reachable statement? ■ Constraint solving in Haskell program is hard
22. 22. Discussion ● Correctness Criteria ○ a Haskell test framework written in Haskell ■ Property Language is much more general ● Function Equality (e.g. current vs. known correct version) ● Result Checking (e.g. mathematical properties) ● Conditional Properties (e.g. insert on sorted list) ● Testing High-Order Functions (e.g. function composition) ○ No published work on automatic testing of functional programs against specification
23. 23. Discussion ● Test Data Generation ○ a Haskell test framework written in Haskell ■ another DSL for describing test data generation ● Grammar inherited from Haskell ● Minimum learning effort ○ Controlling sizes to guarantee termination ■ Need developer to ● Interpret the meaning of “size” ● Specify generators for his/her own new types
24. 24. Discussion ● Some Reflections ○ Formulating formal specification improves our understanding of our programs ○ Three types of errors divided evenly ■ Errors in test generators ■ Errors in the specification ■ Errors in the program
25. 25. References ● Claessen, Koen, and John Hughes. "QuickCheck: a lightweight tool for random testing of Haskell programs." Acm sigplan notices 46.4 (2011): 53- 64. ● Wadler, Philip. "Theorems for free!." Proceedings of the fourth international conference on Functional programming languages and computer architecture. ACM, 1989. ● Weyuker, Elaine J. "On testing non-testable programs." The Computer Journal 25.4 (1982): 465-470.