Thinking in Properties

1.
Thinking in Properties AndBeyond (Testing) Susan Potter 2020-08-01

2.
Intro

3.
Intro finger $(whoami) • Introducedto QuickCheck in Erlang ~2010 • Adopted Haskell’s QuickCheck, Hedgehog, and ScalaCheck at work • ”Testing” in production, thinking in properties, 4 years Susan Potter Thinking in Properties 2020-08-01 1 / 41

4.
Intro Agenda • An OriginStory (with code) • Mental Models (above the code) • Beyond Testing (illustrations) Susan Potter Thinking in Properties 2020-08-01 2 / 41

5.
An Origin Story

6.
An Origin StoryDiscovering Superpowers Susan Potter Thinking in Properties 2020-08-01 3 / 41

7.
An Origin StoryDiscovering Superpowers Explore domain with types data List a = EmptyList | Cons a (List a) type Pred a = (a -> Bool) type Comp a = (a -> a -> Ordering) sortBy :: Comp a -> List a -> List a filter :: Pred a -> List a -> List a reverse :: List a -> List a last, first :: List a -> Maybe a Susan Potter Thinking in Properties 2020-08-01 3 / 41

8.
An Origin StoryDiscovering Superpowers Explore domain with usage examples -- | Reverse the elements of a list -- >>> reverse (Cons 1 (Cons 2 (Cons 3 EmptyList)))→ -- Cons 3 (Cons 2 (Cons 1 EmptyList)) -- -- >>> reverse EmptyList -- EmptyList reverse :: List a -> List a Susan Potter Thinking in Properties 2020-08-01 4 / 41

9.
An Origin StoryDiscovering Superpowers Encode examples as re-runnable tests describe "Lib.reverse" $ do it "returns [5,4,3,2,1] given [1,2,3,4,5]" $ do reverse [1,2,3,4,5] `shouldBe` [5,4,3,2,1] it "returns empty list given empty list" $ do reverse [] `shouldBe` [] Susan Potter Thinking in Properties 2020-08-01 5 / 41

10.
An Origin StoryDiscovering Superpowers Rinse & Repeat Susan Potter Thinking in Properties 2020-08-01 6 / 41

11.
An Origin StoryDiscovering Superpowers Continuous Learning Figure 1: Schedule for Erlang Factory SF 2011 where my mind was blown Susan Potter Thinking in Properties 2020-08-01 7 / 41

12.
An Origin StoryHarnessing Newly Found Superpowers Characteristics of property-based testing Where we have: • generators that produce random ”arbitrary” values for inputs • general rules that hold without knowing inputs upfront • shrinking of failed values • test runs assert rule multiple times using new generated values Related terms: generative testing, fuzz testing (or ”fuzzing”) Susan Potter Thinking in Properties 2020-08-01 8 / 41

13.

14.

15.

16.
An Origin StoryHarnessing Newly Found Superpowers One General Rule: Round-tripping (two rides gets us back home) Examples: -- Assumes x is encodable and decodable roundtrip0 = x -> decode (encode x) == Just x roundtrip1 = x -> decodeBase64 (encodeBase64 x) == Right x Counter-examples: • sha256 is one way! Susan Potter Thinking in Properties 2020-08-01 9 / 41

17.
An Origin StoryHarnessing Newly Found Superpowers Introducing generators To make these ideas concrete we will be using hedgehog with these imports: import Hedgehog import qualified Hedgehog.Gen as Gen import qualified Hedgehog.Range as Range import Control.Monad (replicateM) import Data.Function (($), (.)) Hedgehog integrates shrinking with generation. We will not discuss this difference to QuickCheck but read Well Typed’s blog post about this here. Susan Potter Thinking in Properties 2020-08-01 10 / 41

18.
An Origin StoryHarnessing Newly Found Superpowers Primitive generators by example >>> replicateM 25 $ Gen.sample Gen.lower "okohcpxrkfunkmwnqujnnhxkg" >>> let currencies = [ "USD", "JPY", "EUR", "CHF", "CNY" ] >>> replicateM 5 $ Gen.sample $ Gen.element currencies ["USD","CNY","USD","JPY","USD"] >>> replicateM 5 $ Gen.sample $ Gen.choice [ Gen.ascii, Gen.unicode ]→ ['f', 'c', 'j', '1068213', '<'] Susan Potter Thinking in Properties 2020-08-01 11 / 41

19.
An Origin StoryHarnessing Newly Found Superpowers Generating your domain’s data, 1/2 Suppose our domain looks like this: import Data.Word (Word8, Word16) type W16 = Word16 data IP = IPv4 Word8 Word8 Word8 Word8 | IPv6 W16 W16 W16 W16 W16 W16 W16 W16 Susan Potter Thinking in Properties 2020-08-01 12 / 41

20.
An Origin StoryHarnessing Newly Found Superpowers Generating your domain’s data, 2/2 genW8 = Gen.word8 Range.constantBounded genW16 = Gen.word16 Range.constantBounded genIPv4 = IPv4 <$> genW8 <*> genW8 <*> genW8 <*> genW8 genIPv6 = IPv6 <$> genW16 <*> genW16 <*> genW16 <*> genW16 <*> genW16 <*> genW16 <*> genW16 <*> genW16 genAnyIP = Gen.choice [ genIPv4, genIPv6 ] sampleIPs n = replicateM n (Gen.sample genAnyIP) Susan Potter Thinking in Properties 2020-08-01 13 / 41

21.
An Origin StoryHarnessing Newly Found Superpowers Sampling generated domain data >>> sampleIPs 3 [ "136.59.149.200" , "338d:2397:f612:e036:b27c:2298:4db8:b933" , "5.38.65.204" ] Susan Potter Thinking in Properties 2020-08-01 14 / 41

22.
An Origin StoryHarnessing Newly Found Superpowers Writing Our First Property! genList :: MonadGen m => m a -> m [a] genList = Gen.list (Range.linear 0 1000) genInt = Gen.int (Range.linear 0 100000) -- "round-tripping" property prop_reverse_reverse = property $ do xs <- forAll $ genList genInt Lib.reverse (Lib.reverse xs) === xs Susan Potter Thinking in Properties 2020-08-01 15 / 41

23.
An Origin StoryHarnessing Newly Found Superpowers Reviewing Our First Property! Questions about prop_reverse_reverse: • Does it assert anything about reverse ’s speciﬁcation? • Do callers of reverse need to exploit ”round-tripping”? • Does an implementation exist that typechecks yet fails this property? • Are we generating interesting data given the operation’s type? • Are we resigned to function-level property testing? Susan Potter Thinking in Properties 2020-08-01 16 / 41

24.

25.

26.

27.

28.
An Origin StoryHarnessing Newly Found Superpowers Quick Review: Example-based tests over time t ~ 0 → t → ∞ Quick ⌣ →  / ⌢ Coverage ? →  / ⌢ Repeatable ⌣ →  / ⌢ Documents usage  →  / ⌢ Documents contract ⌢ → ⌢ Effort  →  / ⌢ • can measure coverage • ﬁxtures provide test data • interesting ﬁxtures brittle • over time tends to ⌢ Susan Potter Thinking in Properties 2020-08-01 17 / 41

29.

30.

31.

32.

33.
An Origin StoryHarnessing Newly Found Superpowers Quick Review: Property-based tests initially t ~ 0 Quick ⌣ Coverage ⌣ Repeatable ⌣ Documents usage  Documents contract ⌣ /  Effort ? • Can we measure coverage ? • We need to maintain generators instead of ﬁxtures! • Not constrained by imagination! ⌣ • Am I smart enough to think up relevant and meaningful properties ? Susan Potter Thinking in Properties 2020-08-01 18 / 41

34.

35.

36.

37.

38.
Mental Models

39.
Mental Models Deriving Properties:Algebraic laws Law What might it be good for Idempotency e.g. event log effects, REST APIs, conﬁg effects Associativity e.g. map/reduce distribution Commutativity e.g. map/reduce local parallelism Distributivity e.g. stable or performant rewrite rules Identity element e.g. for invariance at that value for operation Round-tripping e.g. encoding/decoding Absorption e.g. boolean algebra rewrites Transitivity e.g. dependency closures Susan Potter Thinking in Properties 2020-08-01 19 / 41

40.
Mental Models Algebraic laws:Idempotency (running 1+ times yields same result) Examples: idem0 = x -> abs x == abs (abs x) idem2 = x -> toUpper s == toUpper (toUpper s) • curl -XPUT https://foo.bar/resource/123 -d baz=qux Counter-example: • curl -XPOST https://foo.bar/resource -d baz=qux Algebra: Given x ∈ A and f ∈ (A → A) and f(x) = f(f(x)) then f is idempotent. Susan Potter Thinking in Properties 2020-08-01 20 / 41

41.
Mental Models Algebraic laws:Associativity (brackets to the left or right) Examples: assoc0 = x y z -> (x + y) + z == x + (y + z) assoc1 = x y z -> (x ++ y) ++ z == x ++ (y ++ z) Counter-example: • (x − y) − z = x − (y − z) Algebra: Given x, y, z ∈ A, ⊕ ∈ (A → A → A) and (x ⊕ y) ⊕ z = x ⊕ (y ⊕ z) then ⊕ is associative. Susan Potter Thinking in Properties 2020-08-01 21 / 41

42.
Mental Models Algebraic laws:Commutativity (any order will do) Examples: comm0 = x y -> x + y == y + x comm1 = x y -> x * y == y * x Counter-example: • x + +y = y + +x • x − y = y − x Algebra: Given x, y ∈ A, ⊕ ∈ (A → A → A) and $x ⊕ y = y ⊕x then ⊕ is commutative. Susan Potter Thinking in Properties 2020-08-01 22 / 41

43.
Mental Models Algebraic laws:Distributivity (one operation over another) Examples: dist0 = x y z -> x*(y + z) == x*y + x*z Counter-example: • x + (y ∗ z) = x + y ∗ x + z where x = 1, y = 2, z = 3 1 + (2 ∗ 3) = 1 + 2 ∗ 1 + 3 ⇒ 7 = 6 Algebra: Given x, y, z ∈ A and ⊕, ⊗ ∈ (A → A → A) then ⊗ is distributive over ⊕ when x ⊗ (y ⊕ z) = x ⊗ y ⊕ x ⊗ z Susan Potter Thinking in Properties 2020-08-01 23 / 41

44.
Mental Models Algebraic laws:Identity element Examples: identity0 = x -> 0 + x == x identity1 = x -> False || x == x identity2 = s -> "" ++ s == s Counter-example: -NonEmpty does not have an identity element Algebra: ∃e ∈ A, ∀a ∈ A, e ⊕ a = a then e is the identity element in A. Susan Potter Thinking in Properties 2020-08-01 24 / 41

45.
Mental Models Algebraic laws:Absorption Examples: absorption0 = a b -> (a || (a && b)) == a absorption1 = a b -> (a && (a || b)) == a Algebra: Given ∧, ∨ ∈ (A → A → A) and a, b ∈ A then when a ∧ (a ∨ b) = a = a ∨ (a ∧ b) Susan Potter Thinking in Properties 2020-08-01 25 / 41

46.
Mental Models Deriving Properties:Relational laws -- implicitly expect sort and last to be correct prop_max_is_last_of_sort = property $ do xs <- forAll $ genList Gen.ascii Just (max xs) === last (sort xs) prop_last_is_first_of_reversed = property $ do xs <- forAll $ genList Gen.unicode last xs === first (reverse xs) Susan Potter Thinking in Properties 2020-08-01 26 / 41

47.
Mental Models Deriving Properties:Abstraction laws Using hedgehog-classes package we can check our typeclass instances according to the abstraction laws: import Hedgehog import Hedgehog.Classes import qualified Hedgehog.Gen as Gen import qualified Hedgehog.Range as Range investmentPortfolioSemigroup = lawsCheck (semigroupLaws genInvestmentPortfolio) portfolioFoldable = lawsCheck (foldableLaws genPortfolio) Susan Potter Thinking in Properties 2020-08-01 27 / 41

48.
Mental Models Deriving Properties:Reﬂections, Rotations, Distortions prop_rotated_colors_same = property $ do img <- forAll $ genImage colors (rotate90 img) === colors img • normalizing audio shouldn’t change time length • reversing a list shouldn’t change length Susan Potter Thinking in Properties 2020-08-01 28 / 41

49.
Mental Models Deriving Properties:Informal model checking Sometimes you can model the basic state machine of a system simply: • model of interesting parts of stateful system • not exhaustive • thinking in state machine models • generate sequence or call graph of commands • assert pre- and post-conditions or invariants • careful you don’t write a second implementation of the SUT just to test it! Susan Potter Thinking in Properties 2020-08-01 29 / 41

50.
Mental Models Deriving Properties:Legacy oracles Replacing legacy systems: • bind to old lib as oracle • assert new rewritten library matches oracle for same inputs • good for e.g. calculation engines or data pipelines • might need large build engineering effort Susan Potter Thinking in Properties 2020-08-01 30 / 41

51.
Mental Models Deriving Properties:Does not barf Wrapping lower-level code via FFI: • gaps between foreign input or output types and native types • runtime exceptions thrown for some input values (inform design) • sanity checking FFI wrapping Susan Potter Thinking in Properties 2020-08-01 31 / 41

52.
Mental Models Deriving Properties:Metamorphic relations • Running against SUT twice with possibly different inputs • A relation exists between those inputs • Assert a relation exists between the outputs of those system runs An example across inputs and outputs, but the relation between inputs and outputs can be different: x, y ∈ Inputs, x ≤ y, x′ = SUT(x), y′ = SUT(y) then x′ ≤ y′ Susan Potter Thinking in Properties 2020-08-01 32 / 41

53.
Mental Models Deriving Properties:Metamorphic relation patterns • Input equivalence • Shufﬂing • Conjunctive conditions • Disjunctive conditions • Disjoint partitions • Complete partitions • Partition difference Susan Potter Thinking in Properties 2020-08-01 33 / 41

54.
Mental Models Deriving Properties:Heckle Yourself! • mutation testing • alter your code until your tests fail • if no tests fail, throw your tests out (curation) • question your assumptions Susan Potter Thinking in Properties 2020-08-01 34 / 41

55.
Beyond testing

56.
Beyond testing Properties ofDelivery Pipelines Property: Source consistency Ensuring fast-forward only ”merges”: main() { local -r mergeBase="$(git merge-base HEAD origin/deploy)" local -r deployHead="$(git rev-parse origin/deploy)" test "${mergeBase}" = "${deployHead}" } set -e main # should exit with 0 for success Susan Potter Thinking in Properties 2020-08-01 35 / 41

57.
Beyond testing Stateful Migrations(in production) Properties: pre and post conditions and invariants between migration phases • moving a stateful cluster from one datacenter to another • upgrading Elastic search into a new cluster • online schema migrations of large tables with binlog syncing and atomic rename (e.g. MySQL) Susan Potter Thinking in Properties 2020-08-01 36 / 41

58.

59.

60.
Beyond testing System Monitoring Property:Connectedness! Given public name for service name: • resolve name to A records (IPs) • ∀ IPs should negotiate TLS handshake • ∀ IPs should make HTTP request with Host Susan Potter Thinking in Properties 2020-08-01 37 / 41

61.

62.

63.
Beyond testing Production DataChecks Sometimes your generators don’t generate data you see in production! Legacy systems exist with no property-based testing toolchain! • Structured logging can record inputs and results; validate OOB • Run property checks against production inputs and outputs in Haskell :) Susan Potter Thinking in Properties 2020-08-01 38 / 41

64.
Wrapping Up

65.
Wrapping Up In Closing •Not all properties are useful • Initially hard to think up useful properties genMentalModels = Gen.choice [ genAlgebraicLaws, genRelationalLaws, genAbstrationLaws, genStateMachines, genMetamorphicRelations, genHeckleYourCode, genTestingInProduction ] Susan Potter Thinking in Properties 2020-08-01 39 / 41

66.
Wrapping Up In Closing •Not all properties are useful • Initially hard to think up useful properties genMentalModels = Gen.choice [ genAlgebraicLaws, genRelationalLaws, genAbstrationLaws, genStateMachines, genMetamorphicRelations, genHeckleYourCode, genTestingInProduction ] Susan Potter Thinking in Properties 2020-08-01 39 / 41

67.
Wrapping Up Questions? GitHub @mbbx6spp LinkedIn/in/susanpotter Twitter @SusanPotter Web Personal site Consulting Training Thank you for listening! Susan Potter Thinking in Properties 2020-08-01 40 / 41

68.
Wrapping Up Credits • Photoby Elias Castillo on Unsplash • Photo by Juan Rumimpunu on Unsplash • Photo by LinkedIn Sales Navigator on Unsplash • Photo by Leonardo Sanches on Unsplash • Photo by Mélissa Jeanty on Unsplash • Photo by Chris Liverani on Unsplash • Photo by <a href=”https: //unsplash.com/@spanic?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText”>Damir Spanic</a> on <a href=”https://unsplash.com/s/photos/baseball-bat?utm_source=unsplash&utm_medium=referral&utm_content= creditCopyText”>Unsplash</a></span> • Photo by <a href=”https: //unsplash.com/@serrah?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText”>Serrah Galos</a> on <a href=”https://unsplash.com/s/photos/reflection?utm_source=unsplash&utm_medium=referral&utm_content= creditCopyText”>Unsplash</a></span> • Photo by Sergey Zolkin on Unsplash • Photo by Roman Mager on Unsplash • Photo by Miguel Ibáñez on Unsplash • Photo by Science in HD on Unsplash • Photo by Steve Douglas on Unsplash • Photo by Natalie Parham on Unsplash Susan Potter Thinking in Properties 2020-08-01 41 / 41

Thinking in Properties

More Related Content

Similar to Thinking in Properties

More from Susan Potter

Recently uploaded

Thinking in Properties