property-based testing (FrOsCon 9, 2014, August 23)


Published on

Published in: Software
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

property-based testing (FrOsCon 9, 2014, August 23)

  1. 1. testing with propertiesChristoph Neuroth ( )@c089
  2. 2. [horribly predictable quote] “Testing shows the presence, not the absence of bugs.” -- Edsger W. Dijkstra
  3. 3. Hoare: “ The role of testing, in theory, is to establish the base propositions of an inductive proof.[...] At present, this is mainly theory [but] seems to show a possibility of practical results, though proving correctness is a laborious and expensive process.” Perlis: “ Much of program complexity is spurious and a number of test cases properly studied will exhaust the testing problem. The problem is to isolate the right test cases, not to prove the algorithm, for that follows after the choice of the proper test cases.” Diijkstra: “Testing shows the presence, not the absence of bugs”
  4. 4. Testing with examples specify correct output for chosen inputs relatively easy to come up with good communication tool straightforward to implement straightforward to overlook important ones usually only cover a very small portion of your domain
  5. 5. yay, all tests passing! //obviously expect(Math.abs(0)).to.equal(0); //negativevaluesbecomepositive expect(Math.abs(-1)).to.equal(1); expect(Math.abs(-5)).to.equal(5); //whilepositivevaluesstayspositive expect(Math.abs(1)).to.equal(1); //shouldalsoworkwithfloatingpointnumbers expect(Math.abs(-1.23)).to.equal(1.23); expect(Math.abs(1.23)).to.equal(1.23); //oh,andjavascriptknowsaboutnegative0 expect(Math.abs(-0)).to.equal(0);
  6. 6. Problem -- ECMAScript® Language Specification “The Number type has exactly 18437736874454810627 (that is, 264−253+3) values” -- me “The box above has 90000 pixels”
  7. 7. remember when you started learning TDD? Math.abs=function(x){ if(x===0)return0; if(x===-1||x===1)return1; if((x>1&&x<2)||(x>-2&&x<-1))return1.23; return5; }
  8. 8. Testing with properties not to be confused with properties on JavaScript objects specify a property that holds for all / specified inputs instead of finding a proof, test random inputs similar to contracts examples use jsverify, other implementations available
  9. 9. Describing a property Humans: “whatever number we pass to abs, it should return a number greater or equal than 0.” Mathematicians: ∀ x ∈ ℝ: abs(x) ≥ 0 Programmers: varp=require('jsverify'); p.check(p.forall(p.number(),function(x){ varactual=Math.abs(x); returntypeofactual==='number'&&actual>=0; })); OK,passed100tests
  10. 10. Examples / Property / Proof
  11. 11. ∀ x ∈ ℝ: abs(x) ≥ x p.check(p.forall(p.number(),function(x){ returnMath.abs(x)>=x; })); Error:Failedafter3testsand0shrinks.rngState:8c0a43ef85c761db92;Coun terexample:1.5644732909277081;
  12. 12. More, um, well, examples
  13. 13. Defining the domain using filters varslowSum=function(n){ varsum=0,i; for(i=1;i<=n;i++){sum+=i;} returnsum; }, gaussSum=function(n){return(n/2)*(n+1);}, positiveNumber=p.suchthat(p.integer(),function(i){returni>0;}); p.assert(p.forall(positiveNumber,function(i){ returnslowSum(i)===gaussSum(i); }));
  14. 14. Defining the domain using custom generators oddNumber=function(){ return{ arbitrary:function(r){ varn=p.integer().arbitrary(r); return2*n+1; }, shrink:shrink.noop, show:show.def }; }; describe('odd',function(){ //varodd=function(n){returnn%2===1;}; varodd=function(n){returnMath.abs(n)%2===1;}; it('returnstrueforalloddnumbers',function(){ p.assert(p.forall(oddNumber(),odd)); });
  15. 15. idempotence: ∀ x: f(x) === f(f(x)) Reusable properties varis_idempotent=function(generator,fn){ returnp.forall(generator,function(x){ return_.isEqual(fn(fn(s)),fn(s)); }); }; p.assert(is_idempotent(p.array(),_.compact)); p.assert(is_idempotent(p.array(),_.uniq));
  16. 16. Shrinking Counterexamples p.assert(p.forall(p.array(p.integer()),function(l){ return_.all(l,function(n){returnn!=3;}); })); Error:Failedafter27testsand4shrinks.rngState:88361ed782a9c0b45f;Cou nterexample:[3];
  17. 17. Common use cases f(x)>=y //assertingthefunction'srange f(x)===f(f(x));//idempotence a(x)===b(x);//regressiontestforreimplementation newC1().f(x)===newC2().f(x);//closelyrelated newC(c1).f(x)===newC(c2).f(x);//closelyrelated max(a,b)===max(b,a));//commutativity zoom(zoom(img,n),-n)===img);//invertibility
  18. 18. Bonus crazieness What about higher-order functions? “If we are to check properties involving function valued variables, then we must be able to generate arbitrary functions. Rather surprisingly, we are able to do so.” p.assert(p.forall(p.array(),p.fn(),p.value(),function(a,f,c){,f,c).length===a.length; }));
  19. 19. Conclusion finding good properties makes you think harder very functional style, but should work on testing OO code not a replacement for TDD using examples but can be used to help finding missed edge cases best for unit testing (because of high number of test cases) also good for verifying assumptions on 3dparty code