Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
9p p 2   Pontificating
Programming is Hard
Who watches the watchers?
Let’s talk about  verification
XUnit
Standard Practice• We all use it• Most of us rely on it day to day• We seem to trust it implicitly
def test_example  # create something to test  obj = MyExampleObject.new(arguments)  # program statement  result = obj.exam...
XUnit is Complected• It’s doing too much
XUnit is Complected• It’s doing too much• BDD is just XUnit with lots of bad English
XUnit is Complected• It’s doing too much• BDD is just XUnit with lots of bad English• Poor separation of the tester and te...
XUnit is Complected• It’s doing too much• BDD is just XUnit with lots of bad English• Poor separation of the tester and te...
XUnit is Complected• It’s doing too much• BDD is just XUnit with lots of bad English• Poor separation of the tester and te...
Your mock object is a joke; that object is mocking you.          For needing it. – Rich Hickey
Coverage is a Lie• Too many false indicators of comprehensive  verification• We work really hard at the wrong things to  ac...
Always in Manual Drive• Devise, write, and maintain all cases• Edge cases are a (big) problem • Especially with simple fun...
Generative Testing
Logic #fail 9x f (x) 6= expect(x) ) bugs9x f (x) = expect(x) ) ¬bugs
Logic #fail 9x f (x) 6= expect(x) ) bugs9x f (x) = expect(x) ) ¬bugs
Logic #fail 9x f (x) 6= expect(x) ) bugs9x f (x) = expect(x) ) ¬bugs
Generative Testing• Developing test cases is hard• We have computers!• Define your domain, auto-generate inputs• Write less...
Generative Testing• Developing test cases is hard• We have computers!• Define your domain, auto-generate inputs• Write less...
(defspec integer-commutative-laws  (partial map identity)  [^{:tag `integer} a ^{:tag `integer} b]  (if (longable? (+ a b)...
"integer commutative law" in {  check { (x: Int, y: Int) =>    (x + y) mustEqual (y + x)    (x * y) mustEqual (y * x)  }}
Generative Testing• Real world cases?
Generative Testing• Real world cases? • Sparse trees
Generative Testing• Real world cases? • Sparse trees • Complex pre-conditions
Generative Testing• Real world cases? • Sparse trees • Complex pre-conditions• How many iterations?
Rice’s Theorem
[…] there exists no automatic method thatdecides with generality non-trivial questionson the black-box behavior of compute...
Rice’s Theorem• You can never have enough assertions
Rice’s Theorem• You can never have enough assertions• All you can do is improve “confidence” • (for some naïve definition th...
Rice’s Theorem• You can never have enough assertions• All you can do is improve “confidence” • (for some naïve definition th...
Rice’s Theorem• You can never have enough assertions• All you can do is improve “confidence” • (for some naïve definition th...
def addGood(x: Int, y: Int) = x + y// egad!def addBad(x: Int, y: Int) = (x, y) match {  case (0, 0) => 0  case (0, 1) => 1...
Black Box Testing• Do we just…enumerate cases?
Black Box Testing• Do we just…enumerate cases?• When do we stop? (hint: never!)
Black Box Testing• Do we just…enumerate cases?• When do we stop? (hint: never!)• Can we assume edge cases are predictable?
Black Box Testing• Do we just…enumerate cases?• When do we stop? (hint: never!)• Can we assume edge cases are predictable?...
Black Box Testing• Do we just…enumerate cases?• When do we stop? (hint: never!)• Can we assume edge cases are predictable?...
In the beginning…
An Axiomatic Basis for     Computer    Programming
Partial Correctness   P {Q} R
Built on two ideas• Axioms • The foundation of proofs in our system    (assignment, precendence, identity, etc)• Inference...
It drove us to thinkabout consequence in program execution
Be Careful!¬9x 8y (y  x)
Computers are finite!  8x (x  max)
Real Example        Commutative law x+y =y+x x⇥y =y⇥x
Rules of Consequence(` P {Q} R) ^ (` R   S) ) (` P {Q} S)(` P {Q} R) ^ (` S   P ) ) (` S {Q} R)
Our problems are solved!
Se       e G are solved!Our problems ö               del
Type Systems
Type Systems• Mildly controversial…
Type Systems• Mildly controversial…• Formal verification…that you can’t disable!
Type Systems• Mildly controversial…• Formal verification…that you can’t disable!• Part of the language (inextricably) • (ye...
Type Systems• Mildly controversial…• Formal verification…that you can’t disable!• Part of the language (inextricably) • (ye...
G`t :T      0                   P ROGRESS t ) t _ t is valueG`t :T        t )t   0                         P RESERVATION  ...
case class Miles(value: Int)case class Kilometers(value: Int)def launchMarsMission(distance: Miles) = {  val kph = distanc...
trait Forall[CC[_]] {  def apply[A]: CC[A]}trait ST[S, A] {  def flatMap[B](f: A => ST[S, B]): ST[S, B]}object ST {  def r...
Correct by Construction• Incorrect algorithms fail to compile                               Call me Ishmael…
Correct by Construction• Incorrect algorithms fail to compile• Bugs are a thing of the past                               ...
Correct by Construction• Incorrect algorithms fail to compile• Bugs are a thing of the past• Programmers herald new era   ...
Correct by Construction• Incorrect algorithms fail to compile• Bugs are a thing of the past• Programmers herald new era• P...
Curry-Howard• Types are logical propositions• Values are proofs of propositions• Type checking is testing logical validity...
-- Thm. Integer implies Walrustheorem :: Integer -> Walrustheorem a = theorem a
-- Thm. everything implies Walrustheorem :: forall a . a -> Walrustheorem a = theorem a
-- Thm. everything is true!theorem :: forall a . atheorem = theorem
Gödel• Self-referentiality is a problem
Gödel• Self-referentiality is a problem • Hello, Strange Loop!
Gödel• Self-referentiality is a problem • Hello, Strange Loop!• Incomplete or inconsistent
Gödel• Self-referentiality is a problem • Hello, Strange Loop!• Incomplete or inconsistent• Some valid programs cannot be ...
Gödel• Self-referentiality is a problem • Hello, Strange Loop!• Incomplete or inconsistent• Some valid programs cannot be ...
System F• Strongly normalizing λ calculus
System F• Strongly normalizing λ calculus• Statically typed, and complete!
System F• Strongly normalizing λ calculus• Statically typed, and complete! • All valid programs can be typed
System F• Strongly normalizing λ calculus• Statically typed, and complete! • All valid programs can be typed • No invalid ...
System F• Strongly normalizing λ calculus• Statically typed, and complete! • All valid programs can be typed • No invalid ...
System F
System F• Not expressive enough
System F• Not expressive enough• Cannot bootstrap its own compiler
System F• Not expressive enough• Cannot bootstrap its own compiler• Many valid programs have no encoding
System F• Not expressive enough• Cannot bootstrap its own compiler• Many valid programs have no encoding• Not Turing-compl...
Static Typing• Can’t even answer some basic questions
Static Typing• Can’t even answer some basic questions• Fails to definitively answer any questions • …for Turing-complete ca...
Static Typing• Can’t even answer some basic questions• Fails to definitively answer any questions • …for Turing-complete ca...
Static Typing• Can’t even answer some basic questions• Fails to definitively answer any questions • …for Turing-complete ca...
add :: Integer -> Integer -> Integeradd x y = x + y
add :: Integer -> Integer -> Integeradd x y = x + yadd :: Prime -> Prime -> Integeradd x y = x + y
(define (add x y)  (+ x y))
(define (add x y)  (+ x y))(module strangeloop racket  (provide   (contract-out    [add (->* (prime? prime?)              ...
(define (divisible? i m)  (cond   [(= i 1) true]   [else (cond          [(= (remainder m i) 0) false]          [else (divi...
(require strangeloop)(add 7 7);; > 14
(add   4 7);; >   add: contract violation;;     expected: prime?, given: 4;;     in: the 1st argument of;;          (-> pr...
But like everythingelse, it suffers from   deficiencies...
Before Contract Application(time (add 492876847 492876847));; > cpu time: 0 real time: 0 gc time: 0;; 985753694
Before Contract Application(time (add 492876847 492876847));; > cpu time: 0 real time: 0 gc time: 0;; 985753694           ...
Runtime issues canmake contracts a non-starter for production         code
It’s All Bunk• Black-Box verification is insufficient
It’s All Bunk• Black-Box verification is insufficient• White-Box verification is undecidable
It’s All Bunk• Black-Box verification is insufficient• White-Box verification is undecidable • …in general!
It’s All Bunk• Black-Box verification is insufficient• White-Box verification is undecidable • …in general!• Constrain the do...
Functional Style• Write your code in a functional style• It is constraining…that’s the point!• FP is easier to reason abou...
Programming: The Good Parts
References•   From System F to Typed Assembly Language•   An Axiomatic Basis for Computer Programming•   Hoare Logic and A...
Pontificating quantification
Pontificating quantification
Upcoming SlideShare
Loading in …5
×

Pontificating quantification

1,381 views

Published on

Strangeloop 2012 presentation with Daniel Spiewak

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

Pontificating quantification

  1. 1. 9p p 2 Pontificating
  2. 2. Programming is Hard
  3. 3. Who watches the watchers?
  4. 4. Let’s talk about verification
  5. 5. XUnit
  6. 6. Standard Practice• We all use it• Most of us rely on it day to day• We seem to trust it implicitly
  7. 7. def test_example # create something to test obj = MyExampleObject.new(arguments) # program statement result = obj.example_method(arguments) # verification assert_equal("fact", result) #implicit teardownendit "must be an example" do obj = MyExampleObject.new(arguments) result = obj.example_method(arguments) result.should == factend
  8. 8. XUnit is Complected• It’s doing too much
  9. 9. XUnit is Complected• It’s doing too much• BDD is just XUnit with lots of bad English
  10. 10. XUnit is Complected• It’s doing too much• BDD is just XUnit with lots of bad English• Poor separation of the tester and testee
  11. 11. XUnit is Complected• It’s doing too much• BDD is just XUnit with lots of bad English• Poor separation of the tester and testee• What are you testing?
  12. 12. XUnit is Complected• It’s doing too much• BDD is just XUnit with lots of bad English• Poor separation of the tester and testee• What are you testing? • Are you sure?!
  13. 13. Your mock object is a joke; that object is mocking you. For needing it. – Rich Hickey
  14. 14. Coverage is a Lie• Too many false indicators of comprehensive verification• We work really hard at the wrong things to achieve good coverage• “The Emperors new suit”
  15. 15. Always in Manual Drive• Devise, write, and maintain all cases• Edge cases are a (big) problem • Especially with simple functions• Example: Integer commutative law
  16. 16. Generative Testing
  17. 17. Logic #fail 9x f (x) 6= expect(x) ) bugs9x f (x) = expect(x) ) ¬bugs
  18. 18. Logic #fail 9x f (x) 6= expect(x) ) bugs9x f (x) = expect(x) ) ¬bugs
  19. 19. Logic #fail 9x f (x) 6= expect(x) ) bugs9x f (x) = expect(x) ) ¬bugs
  20. 20. Generative Testing• Developing test cases is hard• We have computers!• Define your domain, auto-generate inputs• Write less, test more• Find edge cases automatically!
  21. 21. Generative Testing• Developing test cases is hard• We have computers!• Define your domain, auto-generate inputs• Write less, test more• Find edge cases automatically! • (money back guarantee)
  22. 22. (defspec integer-commutative-laws (partial map identity) [^{:tag `integer} a ^{:tag `integer} b] (if (longable? (+ a b)) (assert (= (+ a b) (+ b a) (+ a b) (+ b a) (unchecked-add a b) (unchecked-add b a))) (assert (= (+ a b) (+ b a)))) (if (longable? (* a b)) (assert (= (* a b) (* b a) (* a b) (* b a) (unchecked-multiply a b) (unchecked-multiply b a))) (assert (= (* a b) (* b a)))))
  23. 23. "integer commutative law" in { check { (x: Int, y: Int) => (x + y) mustEqual (y + x) (x * y) mustEqual (y * x) }}
  24. 24. Generative Testing• Real world cases?
  25. 25. Generative Testing• Real world cases? • Sparse trees
  26. 26. Generative Testing• Real world cases? • Sparse trees • Complex pre-conditions
  27. 27. Generative Testing• Real world cases? • Sparse trees • Complex pre-conditions• How many iterations?
  28. 28. Rice’s Theorem
  29. 29. […] there exists no automatic method thatdecides with generality non-trivial questionson the black-box behavior of computerprograms. – Wikipedia
  30. 30. Rice’s Theorem• You can never have enough assertions
  31. 31. Rice’s Theorem• You can never have enough assertions• All you can do is improve “confidence” • (for some naïve definition thereof)
  32. 32. Rice’s Theorem• You can never have enough assertions• All you can do is improve “confidence” • (for some naïve definition thereof)• Black box testing is a farce
  33. 33. Rice’s Theorem• You can never have enough assertions• All you can do is improve “confidence” • (for some naïve definition thereof)• Black box testing is a farce• Partial function indistinguishable from total
  34. 34. def addGood(x: Int, y: Int) = x + y// egad!def addBad(x: Int, y: Int) = (x, y) match { case (0, 0) => 0 case (0, 1) => 1 case (1, 0) => 1 case (2, 3) => 5 case (100, 500) => 600}
  35. 35. Black Box Testing• Do we just…enumerate cases?
  36. 36. Black Box Testing• Do we just…enumerate cases?• When do we stop? (hint: never!)
  37. 37. Black Box Testing• Do we just…enumerate cases?• When do we stop? (hint: never!)• Can we assume edge cases are predictable?
  38. 38. Black Box Testing• Do we just…enumerate cases?• When do we stop? (hint: never!)• Can we assume edge cases are predictable?• What can we assume?
  39. 39. Black Box Testing• Do we just…enumerate cases?• When do we stop? (hint: never!)• Can we assume edge cases are predictable?• What can we assume?• It’s all undecidable, let’s go shopping
  40. 40. In the beginning…
  41. 41. An Axiomatic Basis for Computer Programming
  42. 42. Partial Correctness P {Q} R
  43. 43. Built on two ideas• Axioms • The foundation of proofs in our system (assignment, precendence, identity, etc)• Inference • Higher order ideas that use combinations of axioms to draw conclusions
  44. 44. It drove us to thinkabout consequence in program execution
  45. 45. Be Careful!¬9x 8y (y  x)
  46. 46. Computers are finite! 8x (x  max)
  47. 47. Real Example Commutative law x+y =y+x x⇥y =y⇥x
  48. 48. Rules of Consequence(` P {Q} R) ^ (` R S) ) (` P {Q} S)(` P {Q} R) ^ (` S P ) ) (` S {Q} R)
  49. 49. Our problems are solved!
  50. 50. Se e G are solved!Our problems ö del
  51. 51. Type Systems
  52. 52. Type Systems• Mildly controversial…
  53. 53. Type Systems• Mildly controversial…• Formal verification…that you can’t disable!
  54. 54. Type Systems• Mildly controversial…• Formal verification…that you can’t disable!• Part of the language (inextricably) • (yes, that is a definitional requirement)
  55. 55. Type Systems• Mildly controversial…• Formal verification…that you can’t disable!• Part of the language (inextricably) • (yes, that is a definitional requirement)• Model correctness as consistency
  56. 56. G`t :T 0 P ROGRESS t ) t _ t is valueG`t :T t )t 0 P RESERVATION G`t :T 0
  57. 57. case class Miles(value: Int)case class Kilometers(value: Int)def launchMarsMission(distance: Miles) = { val kph = distance.value / (6 * 30 * 24) increaseSpeedToKPH(kph)}
  58. 58. trait Forall[CC[_]] { def apply[A]: CC[A]}trait ST[S, A] { def flatMap[B](f: A => ST[S, B]): ST[S, B]}object ST { def run[A]( f: Forall[({ type λ[S] = ST[S, A] => A })#λ]): A}
  59. 59. Correct by Construction• Incorrect algorithms fail to compile Call me Ishmael…
  60. 60. Correct by Construction• Incorrect algorithms fail to compile• Bugs are a thing of the past Call me Ishmael…
  61. 61. Correct by Construction• Incorrect algorithms fail to compile• Bugs are a thing of the past• Programmers herald new era Call me Ishmael…
  62. 62. Correct by Construction• Incorrect algorithms fail to compile• Bugs are a thing of the past• Programmers herald new era• Profit? Call me Ishmael…
  63. 63. Curry-Howard• Types are logical propositions• Values are proofs of propositions• Type checking is testing logical validity• A system’s consistency is what is checked
  64. 64. -- Thm. Integer implies Walrustheorem :: Integer -> Walrustheorem a = theorem a
  65. 65. -- Thm. everything implies Walrustheorem :: forall a . a -> Walrustheorem a = theorem a
  66. 66. -- Thm. everything is true!theorem :: forall a . atheorem = theorem
  67. 67. Gödel• Self-referentiality is a problem
  68. 68. Gödel• Self-referentiality is a problem • Hello, Strange Loop!
  69. 69. Gödel• Self-referentiality is a problem • Hello, Strange Loop!• Incomplete or inconsistent
  70. 70. Gödel• Self-referentiality is a problem • Hello, Strange Loop!• Incomplete or inconsistent• Some valid programs cannot be typed
  71. 71. Gödel• Self-referentiality is a problem • Hello, Strange Loop!• Incomplete or inconsistent• Some valid programs cannot be typed• Strong normalization helps • (avoids the need for recursive types!)
  72. 72. System F• Strongly normalizing λ calculus
  73. 73. System F• Strongly normalizing λ calculus• Statically typed, and complete!
  74. 74. System F• Strongly normalizing λ calculus• Statically typed, and complete! • All valid programs can be typed
  75. 75. System F• Strongly normalizing λ calculus• Statically typed, and complete! • All valid programs can be typed • No invalid programs can be typed
  76. 76. System F• Strongly normalizing λ calculus• Statically typed, and complete! • All valid programs can be typed • No invalid programs can be typed• Immensely expressive (lists, numbers, etc)
  77. 77. System F
  78. 78. System F• Not expressive enough
  79. 79. System F• Not expressive enough• Cannot bootstrap its own compiler
  80. 80. System F• Not expressive enough• Cannot bootstrap its own compiler• Many valid programs have no encoding
  81. 81. System F• Not expressive enough• Cannot bootstrap its own compiler• Many valid programs have no encoding• Not Turing-complete! (trivially so)
  82. 82. Static Typing• Can’t even answer some basic questions
  83. 83. Static Typing• Can’t even answer some basic questions• Fails to definitively answer any questions • …for Turing-complete calculi
  84. 84. Static Typing• Can’t even answer some basic questions• Fails to definitively answer any questions • …for Turing-complete calculi• Arms race of inconvenience
  85. 85. Static Typing• Can’t even answer some basic questions• Fails to definitively answer any questions • …for Turing-complete calculi• Arms race of inconvenience• Guardrails on a sidewalk
  86. 86. add :: Integer -> Integer -> Integeradd x y = x + y
  87. 87. add :: Integer -> Integer -> Integeradd x y = x + yadd :: Prime -> Prime -> Integeradd x y = x + y
  88. 88. (define (add x y) (+ x y))
  89. 89. (define (add x y) (+ x y))(module strangeloop racket (provide (contract-out [add (->* (prime? prime?) () integer?)])))
  90. 90. (define (divisible? i m) (cond [(= i 1) true] [else (cond [(= (remainder m i) 0) false] [else (divisible? (sub1 i) m)])]))(define (prime? n) (divisible? (sub1 n) n))
  91. 91. (require strangeloop)(add 7 7);; > 14
  92. 92. (add 4 7);; > add: contract violation;; expected: prime?, given: 4;; in: the 1st argument of;; (-> prime? prime?;; integer?);; contract from: strangeloop;; blaming: top-level;; at: stdin::79-82;; context...:
  93. 93. But like everythingelse, it suffers from deficiencies...
  94. 94. Before Contract Application(time (add 492876847 492876847));; > cpu time: 0 real time: 0 gc time: 0;; 985753694
  95. 95. Before Contract Application(time (add 492876847 492876847));; > cpu time: 0 real time: 0 gc time: 0;; 985753694 After Contract Application(time (add 492876847 492876847));; > cpu time: 15963 real time: 15995 gc time: 0;; 985753694
  96. 96. Runtime issues canmake contracts a non-starter for production code
  97. 97. It’s All Bunk• Black-Box verification is insufficient
  98. 98. It’s All Bunk• Black-Box verification is insufficient• White-Box verification is undecidable
  99. 99. It’s All Bunk• Black-Box verification is insufficient• White-Box verification is undecidable • …in general!
  100. 100. It’s All Bunk• Black-Box verification is insufficient• White-Box verification is undecidable • …in general!• Constrain the domain
  101. 101. Functional Style• Write your code in a functional style• It is constraining…that’s the point!• FP is easier to reason about • (both formally and informally)• Easier to test, and easier to verify
  102. 102. Programming: The Good Parts
  103. 103. References• From System F to Typed Assembly Language• An Axiomatic Basis for Computer Programming• Hoare Logic and Auxiliary Variables• The Underlying Logic of Hoare Logic• Applying Design by Contract• Proof Carrying Code• Uniform Proofs as a Foundation for Logic Programming• Safe Kernel Extensions Without Run-Time Checking

×