Thinking in Properties

1. Thinking in Properties And Beyond (Testing) Susan Potter 2020-08-01

2. Intro

3. Intro finger $(whoami) • Introduced to QuickCheck in Erlang ~2010 • Adopted Haskell’s QuickCheck, Hedgehog, and ScalaCheck at work • ”Testing” in production, thinking in properties, 4 years Susan Potter Thinking in Properties 2020-08-01 1 / 41

4. Intro Agenda • An Origin Story (with code) • Mental Models (above the code) • Beyond Testing (illustrations) Susan Potter Thinking in Properties 2020-08-01 2 / 41

5. An Origin Story

6. An Origin Story Discovering Superpowers Susan Potter Thinking in Properties 2020-08-01 3 / 41

7. An Origin Story Discovering Superpowers Explore domain with types data List a = EmptyList | Cons a (List a) type Pred a = (a -> Bool) type Comp a = (a -> a -> Ordering) sortBy :: Comp a -> List a -> List a filter :: Pred a -> List a -> List a reverse :: List a -> List a last, first :: List a -> Maybe a Susan Potter Thinking in Properties 2020-08-01 3 / 41

8. An Origin Story Discovering Superpowers Explore domain with usage examples -- | Reverse the elements of a list -- >>> reverse (Cons 1 (Cons 2 (Cons 3 EmptyList)))→ -- Cons 3 (Cons 2 (Cons 1 EmptyList)) -- -- >>> reverse EmptyList -- EmptyList reverse :: List a -> List a Susan Potter Thinking in Properties 2020-08-01 4 / 41

9. An Origin Story Discovering Superpowers Encode examples as re-runnable tests describe "Lib.reverse" $ do it "returns [5,4,3,2,1] given [1,2,3,4,5]" $ do reverse [1,2,3,4,5] `shouldBe` [5,4,3,2,1] it "returns empty list given empty list" $ do reverse [] `shouldBe` [] Susan Potter Thinking in Properties 2020-08-01 5 / 41

10. An Origin Story Discovering Superpowers Rinse & Repeat Susan Potter Thinking in Properties 2020-08-01 6 / 41

11. An Origin Story Discovering Superpowers Continuous Learning Figure 1: Schedule for Erlang Factory SF 2011 where my mind was blown Susan Potter Thinking in Properties 2020-08-01 7 / 41

12. An Origin Story Harnessing Newly Found Superpowers Characteristics of property-based testing Where we have: • generators that produce random ”arbitrary” values for inputs • general rules that hold without knowing inputs upfront • shrinking of failed values • test runs assert rule multiple times using new generated values Related terms: generative testing, fuzz testing (or ”fuzzing”) Susan Potter Thinking in Properties 2020-08-01 8 / 41

16. An Origin Story Harnessing Newly Found Superpowers One General Rule: Round-tripping (two rides gets us back home) Examples: -- Assumes x is encodable and decodable roundtrip0 = x -> decode (encode x) == Just x roundtrip1 = x -> decodeBase64 (encodeBase64 x) == Right x Counter-examples: • sha256 is one way! Susan Potter Thinking in Properties 2020-08-01 9 / 41

17. An Origin Story Harnessing Newly Found Superpowers Introducing generators To make these ideas concrete we will be using hedgehog with these imports: import Hedgehog import qualified Hedgehog.Gen as Gen import qualified Hedgehog.Range as Range import Control.Monad (replicateM) import Data.Function (($), (.)) Hedgehog integrates shrinking with generation. We will not discuss this difference to QuickCheck but read Well Typed’s blog post about this here. Susan Potter Thinking in Properties 2020-08-01 10 / 41

18. An Origin Story Harnessing Newly Found Superpowers Primitive generators by example >>> replicateM 25 $ Gen.sample Gen.lower "okohcpxrkfunkmwnqujnnhxkg" >>> let currencies = [ "USD", "JPY", "EUR", "CHF", "CNY" ] >>> replicateM 5 $ Gen.sample $ Gen.element currencies ["USD","CNY","USD","JPY","USD"] >>> replicateM 5 $ Gen.sample $ Gen.choice [ Gen.ascii, Gen.unicode ]→ ['f', 'c', 'j', '1068213', '<'] Susan Potter Thinking in Properties 2020-08-01 11 / 41

19. An Origin Story Harnessing Newly Found Superpowers Generating your domain’s data, 1/2 Suppose our domain looks like this: import Data.Word (Word8, Word16) type W16 = Word16 data IP = IPv4 Word8 Word8 Word8 Word8 | IPv6 W16 W16 W16 W16 W16 W16 W16 W16 Susan Potter Thinking in Properties 2020-08-01 12 / 41

20. An Origin Story Harnessing Newly Found Superpowers Generating your domain’s data, 2/2 genW8 = Gen.word8 Range.constantBounded genW16 = Gen.word16 Range.constantBounded genIPv4 = IPv4 <$> genW8 <*> genW8 <*> genW8 <*> genW8 genIPv6 = IPv6 <$> genW16 <*> genW16 <*> genW16 <*> genW16 <*> genW16 <*> genW16 <*> genW16 <*> genW16 genAnyIP = Gen.choice [ genIPv4, genIPv6 ] sampleIPs n = replicateM n (Gen.sample genAnyIP) Susan Potter Thinking in Properties 2020-08-01 13 / 41

21. An Origin Story Harnessing Newly Found Superpowers Sampling generated domain data >>> sampleIPs 3 [ "136.59.149.200" , "338d:2397:f612:e036:b27c:2298:4db8:b933" , "5.38.65.204" ] Susan Potter Thinking in Properties 2020-08-01 14 / 41

22. An Origin Story Harnessing Newly Found Superpowers Writing Our First Property! genList :: MonadGen m => m a -> m [a] genList = Gen.list (Range.linear 0 1000) genInt = Gen.int (Range.linear 0 100000) -- "round-tripping" property prop_reverse_reverse = property $ do xs <- forAll $ genList genInt Lib.reverse (Lib.reverse xs) === xs Susan Potter Thinking in Properties 2020-08-01 15 / 41

23. An Origin Story Harnessing Newly Found Superpowers Reviewing Our First Property! Questions about prop_reverse_reverse: • Does it assert anything about reverse ’s speciﬁcation? • Do callers of reverse need to exploit ”round-tripping”? • Does an implementation exist that typechecks yet fails this property? • Are we generating interesting data given the operation’s type? • Are we resigned to function-level property testing? Susan Potter Thinking in Properties 2020-08-01 16 / 41

28. An Origin Story Harnessing Newly Found Superpowers Quick Review: Example-based tests over time t ~ 0 → t → ∞ Quick ⌣ →  / ⌢ Coverage ? →  / ⌢ Repeatable ⌣ →  / ⌢ Documents usage  →  / ⌢ Documents contract ⌢ → ⌢ Effort  →  / ⌢ • can measure coverage • ﬁxtures provide test data • interesting ﬁxtures brittle • over time tends to ⌢ Susan Potter Thinking in Properties 2020-08-01 17 / 41

33. An Origin Story Harnessing Newly Found Superpowers Quick Review: Property-based tests initially t ~ 0 Quick ⌣ Coverage ⌣ Repeatable ⌣ Documents usage  Documents contract ⌣ /  Effort ? • Can we measure coverage ? • We need to maintain generators instead of ﬁxtures! • Not constrained by imagination! ⌣ • Am I smart enough to think up relevant and meaningful properties ? Susan Potter Thinking in Properties 2020-08-01 18 / 41

38. Mental Models

39. Mental Models Deriving Properties: Algebraic laws Law What might it be good for Idempotency e.g. event log effects, REST APIs, conﬁg effects Associativity e.g. map/reduce distribution Commutativity e.g. map/reduce local parallelism Distributivity e.g. stable or performant rewrite rules Identity element e.g. for invariance at that value for operation Round-tripping e.g. encoding/decoding Absorption e.g. boolean algebra rewrites Transitivity e.g. dependency closures Susan Potter Thinking in Properties 2020-08-01 19 / 41

40. Mental Models Algebraic laws: Idempotency (running 1+ times yields same result) Examples: idem0 = x -> abs x == abs (abs x) idem2 = x -> toUpper s == toUpper (toUpper s) • curl -XPUT https://foo.bar/resource/123 -d baz=qux Counter-example: • curl -XPOST https://foo.bar/resource -d baz=qux Algebra: Given x ∈ A and f ∈ (A → A) and f(x) = f(f(x)) then f is idempotent. Susan Potter Thinking in Properties 2020-08-01 20 / 41

41. Mental Models Algebraic laws: Associativity (brackets to the left or right) Examples: assoc0 = x y z -> (x + y) + z == x + (y + z) assoc1 = x y z -> (x ++ y) ++ z == x ++ (y ++ z) Counter-example: • (x − y) − z = x − (y − z) Algebra: Given x, y, z ∈ A, ⊕ ∈ (A → A → A) and (x ⊕ y) ⊕ z = x ⊕ (y ⊕ z) then ⊕ is associative. Susan Potter Thinking in Properties 2020-08-01 21 / 41

42. Mental Models Algebraic laws: Commutativity (any order will do) Examples: comm0 = x y -> x + y == y + x comm1 = x y -> x * y == y * x Counter-example: • x + +y = y + +x • x − y = y − x Algebra: Given x, y ∈ A, ⊕ ∈ (A → A → A) and $x ⊕ y = y ⊕x then ⊕ is commutative. Susan Potter Thinking in Properties 2020-08-01 22 / 41

43. Mental Models Algebraic laws: Distributivity (one operation over another) Examples: dist0 = x y z -> x*(y + z) == x*y + x*z Counter-example: • x + (y ∗ z) = x + y ∗ x + z where x = 1, y = 2, z = 3 1 + (2 ∗ 3) = 1 + 2 ∗ 1 + 3 ⇒ 7 = 6 Algebra: Given x, y, z ∈ A and ⊕, ⊗ ∈ (A → A → A) then ⊗ is distributive over ⊕ when x ⊗ (y ⊕ z) = x ⊗ y ⊕ x ⊗ z Susan Potter Thinking in Properties 2020-08-01 23 / 41

44. Mental Models Algebraic laws: Identity element Examples: identity0 = x -> 0 + x == x identity1 = x -> False || x == x identity2 = s -> "" ++ s == s Counter-example: -NonEmpty does not have an identity element Algebra: ∃e ∈ A, ∀a ∈ A, e ⊕ a = a then e is the identity element in A. Susan Potter Thinking in Properties 2020-08-01 24 / 41

45. Mental Models Algebraic laws: Absorption Examples: absorption0 = a b -> (a || (a && b)) == a absorption1 = a b -> (a && (a || b)) == a Algebra: Given ∧, ∨ ∈ (A → A → A) and a, b ∈ A then when a ∧ (a ∨ b) = a = a ∨ (a ∧ b) Susan Potter Thinking in Properties 2020-08-01 25 / 41

46. Mental Models Deriving Properties: Relational laws -- implicitly expect sort and last to be correct prop_max_is_last_of_sort = property $ do xs <- forAll $ genList Gen.ascii Just (max xs) === last (sort xs) prop_last_is_first_of_reversed = property $ do xs <- forAll $ genList Gen.unicode last xs === first (reverse xs) Susan Potter Thinking in Properties 2020-08-01 26 / 41

47. Mental Models Deriving Properties: Abstraction laws Using hedgehog-classes package we can check our typeclass instances according to the abstraction laws: import Hedgehog import Hedgehog.Classes import qualified Hedgehog.Gen as Gen import qualified Hedgehog.Range as Range investmentPortfolioSemigroup = lawsCheck (semigroupLaws genInvestmentPortfolio) portfolioFoldable = lawsCheck (foldableLaws genPortfolio) Susan Potter Thinking in Properties 2020-08-01 27 / 41

48. Mental Models Deriving Properties: Reﬂections, Rotations, Distortions prop_rotated_colors_same = property $ do img <- forAll $ genImage colors (rotate90 img) === colors img • normalizing audio shouldn’t change time length • reversing a list shouldn’t change length Susan Potter Thinking in Properties 2020-08-01 28 / 41

49. Mental Models Deriving Properties: Informal model checking Sometimes you can model the basic state machine of a system simply: • model of interesting parts of stateful system • not exhaustive • thinking in state machine models • generate sequence or call graph of commands • assert pre- and post-conditions or invariants • careful you don’t write a second implementation of the SUT just to test it! Susan Potter Thinking in Properties 2020-08-01 29 / 41

50. Mental Models Deriving Properties: Legacy oracles Replacing legacy systems: • bind to old lib as oracle • assert new rewritten library matches oracle for same inputs • good for e.g. calculation engines or data pipelines • might need large build engineering effort Susan Potter Thinking in Properties 2020-08-01 30 / 41

51. Mental Models Deriving Properties: Does not barf Wrapping lower-level code via FFI: • gaps between foreign input or output types and native types • runtime exceptions thrown for some input values (inform design) • sanity checking FFI wrapping Susan Potter Thinking in Properties 2020-08-01 31 / 41

52. Mental Models Deriving Properties: Metamorphic relations • Running against SUT twice with possibly different inputs • A relation exists between those inputs • Assert a relation exists between the outputs of those system runs An example across inputs and outputs, but the relation between inputs and outputs can be different: x, y ∈ Inputs, x ≤ y, x′ = SUT(x), y′ = SUT(y) then x′ ≤ y′ Susan Potter Thinking in Properties 2020-08-01 32 / 41

53. Mental Models Deriving Properties: Metamorphic relation patterns • Input equivalence • Shufﬂing • Conjunctive conditions • Disjunctive conditions • Disjoint partitions • Complete partitions • Partition difference Susan Potter Thinking in Properties 2020-08-01 33 / 41

54. Mental Models Deriving Properties: Heckle Yourself! • mutation testing • alter your code until your tests fail • if no tests fail, throw your tests out (curation) • question your assumptions Susan Potter Thinking in Properties 2020-08-01 34 / 41

55. Beyond testing

56. Beyond testing Properties of Delivery Pipelines Property: Source consistency Ensuring fast-forward only ”merges”: main() { local -r mergeBase="$(git merge-base HEAD origin/deploy)" local -r deployHead="$(git rev-parse origin/deploy)" test "${mergeBase}" = "${deployHead}" } set -e main # should exit with 0 for success Susan Potter Thinking in Properties 2020-08-01 35 / 41

57. Beyond testing Stateful Migrations (in production) Properties: pre and post conditions and invariants between migration phases • moving a stateful cluster from one datacenter to another • upgrading Elastic search into a new cluster • online schema migrations of large tables with binlog syncing and atomic rename (e.g. MySQL) Susan Potter Thinking in Properties 2020-08-01 36 / 41

60. Beyond testing System Monitoring Property: Connectedness! Given public name for service name: • resolve name to A records (IPs) • ∀ IPs should negotiate TLS handshake • ∀ IPs should make HTTP request with Host Susan Potter Thinking in Properties 2020-08-01 37 / 41

63. Beyond testing Production Data Checks Sometimes your generators don’t generate data you see in production! Legacy systems exist with no property-based testing toolchain! • Structured logging can record inputs and results; validate OOB • Run property checks against production inputs and outputs in Haskell :) Susan Potter Thinking in Properties 2020-08-01 38 / 41

64. Wrapping Up

65. Wrapping Up In Closing • Not all properties are useful • Initially hard to think up useful properties genMentalModels = Gen.choice [ genAlgebraicLaws, genRelationalLaws, genAbstrationLaws, genStateMachines, genMetamorphicRelations, genHeckleYourCode, genTestingInProduction ] Susan Potter Thinking in Properties 2020-08-01 39 / 41

66. Wrapping Up In Closing • Not all properties are useful • Initially hard to think up useful properties genMentalModels = Gen.choice [ genAlgebraicLaws, genRelationalLaws, genAbstrationLaws, genStateMachines, genMetamorphicRelations, genHeckleYourCode, genTestingInProduction ] Susan Potter Thinking in Properties 2020-08-01 39 / 41

67. Wrapping Up Questions? GitHub @mbbx6spp LinkedIn /in/susanpotter Twitter @SusanPotter Web Personal site Consulting Training Thank you for listening! Susan Potter Thinking in Properties 2020-08-01 40 / 41

68. Wrapping Up Credits • Photo by Elias Castillo on Unsplash • Photo by Juan Rumimpunu on Unsplash • Photo by LinkedIn Sales Navigator on Unsplash • Photo by Leonardo Sanches on Unsplash • Photo by Mélissa Jeanty on Unsplash • Photo by Chris Liverani on Unsplash • Photo by <a href=”https: //unsplash.com/@spanic?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText”>Damir Spanic</a> on <a href=”https://unsplash.com/s/photos/baseball-bat?utm_source=unsplash&utm_medium=referral&utm_content= creditCopyText”>Unsplash</a></span> • Photo by <a href=”https: //unsplash.com/@serrah?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText”>Serrah Galos</a> on <a href=”https://unsplash.com/s/photos/reflection?utm_source=unsplash&utm_medium=referral&utm_content= creditCopyText”>Unsplash</a></span> • Photo by Sergey Zolkin on Unsplash • Photo by Roman Mager on Unsplash • Photo by Miguel Ibáñez on Unsplash • Photo by Science in HD on Unsplash • Photo by Steve Douglas on Unsplash • Photo by Natalie Parham on Unsplash Susan Potter Thinking in Properties 2020-08-01 41 / 41

Thinking in Properties

Thinking in Properties

Recommended

More Related Content

Similar to Thinking in Properties

Similar to Thinking in Properties(20)

More from Susan Potter

More from Susan Potter(17)

Recently uploaded

Recently uploaded(20)

Thinking in Properties