Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Lies We Tell About Software Testing

2,791 views

Published on

This talk is more about half-truths than lies, but the concepts are important. There are several areas in which we think we have more confidence in our information than we actually do. This presentation covers both areas in which the evidence is against our beliefs and areas in which we simply need more information.

Published in: Technology
  • Be the first to comment

The Lies We Tell About Software Testing

  1. 1. Tes$ng  Lies   (The  &tle  is  the  first  lie)   Hannover,  Germany,  2014   Cur$s  “Ovid”  Poe   h5p://www.allaroundtheworld.fr/   Photo  CC  license:  h5p://commons.wikimedia.org/wiki/File:Hannover_Altstadt_128-­‐h.jpg   Copyright  2014,  All  Around  The  World  March  26,  2014  
  2. 2. But  first  …   March  26,  2014   Copyright  2014,  All  Around  The  World   h5p://www.slideshare.net/gvwilson/bits-­‐of-­‐evidence-­‐2338367  
  3. 3. Bits  of  Evidence  by  Dr.  Greg  Wilson   March  26,  2014   Copyright  2014,  All  Around  The  World   •  Overwhelming  evidence  that  evidence  works   •  Science  demands  evidence   •  Programmers  provide  anecdotes   •  (…  and  defend  them  vigorously)  
  4. 4. Evidence   •  Absence  of  evidence  is  not  evidence   •  …  but  evidence  of  absence  is   March  26,  2014   Copyright  2014,  All  Around  The  World  
  5. 5. Absence  of  Evidence   •  There  is  no  evidence  that  gods  exist   •  There  is  no  evidence  that  gods  don’t  exist   March  26,  2014   Copyright  2014,  All  Around  The  World  
  6. 6. Evidence  of  Absence   •  Belief:  small  subs  more  maintainable/correct   •  Reality:  these  proper$es  are  absent   March  26,  2014   Copyright  2014,  All  Around  The  World  
  7. 7. Evidence  of  Absence   •  Belief:  small  subrou$nes/methods  “be5er”   •  Inverse  correla$on  (Bell,  Perricone  1984)   •  No  correla$on  (Shen,  et  al.  1985)   •  No  correla$on  and  larger  is  cheaper  (two  studies)   •  Small  rou$nes  had  more  errors  (Selbi,  Basil  1991)   •  Less  maintenance  for  larger  rou$nes  (Lind,   Vairavan  1989)   •  500+  lines  had  more  errors  (Jones  1986)   (cita$ons  via:  Code  Complete,  Steve  McConnell,  Microsoh  Press,  2nd  Edi$on)   March  26,  2014   Copyright  2014,  All  Around  The  World  
  8. 8. P.O.P.  Copyright  2013,  Cur$s  "Ovid"  Poe   Reality  trumps  opinion   3/26/14  
  9. 9. Yeah,  yeah,  let’s  get  started   March  26,  2014   Copyright  2014,  All  Around  The  World  
  10. 10. Caveats   March  26,  2014   Copyright  2014,  All  Around  The  World   •  Some  points  self-­‐evident   •  Some  points  maddening   •  Not  many  studies   •  Always  be  skep$cal   Photo  CC  BY-­‐SA  2.0  h5p://www.flickr.com/photos/busbeytheelder/515542705/  
  11. 11. Code  Coverage   March  26,  2014   Copyright  2014,  All  Around  The  World  
  12. 12. 100%  Code  Coverage   March  26,  2014   Copyright  2014,  All  Around  The  World  
  13. 13. 100%  Code  Coverage   •  What  if  we  pass  zero?   •  …  or  a  string?   •  …  or  no  argument?   •  …  or  a  filehandle?   •  …  or  type  when  reciprocals  don't  make  sense?   •  …  or,  or,  or  …   March  26,  2014   Copyright  2014,  All  Around  The  World  
  14. 14. Expecta$ons   •  Tes$ng  should  be  an  act  of  aggression   •  Programmers  tend  to  test  against  expecta$ons   •  False  confidence   March  26,  2014   Copyright  2014,  All  Around  The  World  
  15. 15. Code  Coverage  of  Dead  Code   March  26,  2014   Copyright  2014,  All  Around  The  World  
  16. 16. Unit  Versus  Integra$on  Tests   •  Unit  tested  subs  don’t  talk  to  other  subs   •  But  your  code  certainly  does!   •  Dead  code  is  cogni$ve  overhead   •  Dead  code  can’t  be  tested  via  your  interface   March  26,  2014   Copyright  2014,  All  Around  The  World  
  17. 17. Code  Coverage  of  Dead  Code   March  26,  2014   Copyright  2014,  All  Around  The  World  
  18. 18. March  26,  2014   Copyright  2014,  All  Around  The  World  
  19. 19. March  26,  2014   Copyright  2014,  All  Around  The  World  
  20. 20. Covered  but  not  tested   March  26,  2014   Copyright  2014,  All  Around  The  World  
  21. 21. Devel::Cover  Recap   •  100%  code  coverage  doesn’t  handle  data   •  Don’t  cover  dead  code   •  Covered  but  not  tested   •  Don't  blindly  trust  your  tools   March  26,  2014   Copyright  2014,  All  Around  The  World  
  22. 22. Kwalitee   •  Lie:  Perl  community  is  the  most  test-­‐infected   •  Truth:  Quan$ty  ain’t  quality   March  26,  2014   Copyright  2014,  All  Around  The  World  
  23. 23. CPAN  Testers  (March  6,  2014)   March  26,  2014   Copyright  2014,  All  Around  The  World  
  24. 24. Versions  and  Plauorms   March  26,  2014   Copyright  2014,  All  Around  The  World  
  25. 25. CPAN  Testers   March  26,  2014   Copyright  2014,  All  Around  The  World  
  26. 26. Perl  Isn’t  Stealing  Enough   We  need  quality  to  go  with  quan$ty   March  26,  2014   Copyright  2014,  All  Around  The  World  
  27. 27. All  Pairs  Tes$ng   March  26,  2014   Copyright  2014,  All  Around  The  World  
  28. 28. The  Coupling  Effect   •  Inves$ga$ons  of  the  Sohware  Tes$ng  Coupling   Effect  —  A.  Jefferson  Offu5   •  Competent  programmers  write  nearly  correct   programs   •  Most  faults  are  simple  faults   •  Simple  faults  can  be  easily  detected   March  26,  2014   Copyright  2014,  All  Around  The  World  
  29. 29. All-­‐Pairs  Tes$ng   •  10  arguments  ==  10  billion  combina$ons   •  Combined  pairs  ==  54  combina$ons   •  Extend  each  argument  with  reasonable  values   March  26,  2014   Copyright  2014,  All  Around  The  World  
  30. 30. All-­‐Pairs  Tes$ng   First  pairs   Second  Pairs   Third  Pairs   Fourth  Pairs   And  so  on   arg1,  arg2   —   —   —   …   arg1,  arg3   arg2,  arg3   —   —   …   arg1,  arg4   arg2,  arg4   arg3,  arg4   —   …   …   …   …   arg4,  arg5   …   arg1,  arg0   arg2,  arg0   arg3,  arg0   …   …   March  26,  2014   Copyright  2014,  All  Around  The  World  
  31. 31. All-­‐Pairs  Tes$ng   March  26,  2014   Copyright  2014,  All  Around  The  World   •  Fast  genera$on  of  tests   •  Likely  to  catch  many  more  bugs   •  Built  in  to  many  tools  (for  sta$c  languages)   •  For  Perl:  ALLPAIRS  Test  Case  Genera$on  Tool  
  32. 32. Documenta$on   •  Lie:  Tests  are  documenta$on   •  Truth:  I  wish   March  26,  2014   Copyright  2014,  All  Around  The  World  
  33. 33. March  26,  2014   Copyright  2014,  All  Around  The  World  
  34. 34. March  26,  2014   Copyright  2014,  All  Around  The  World  
  35. 35. March  26,  2014   Copyright  2014,  All  Around  The  World  
  36. 36. Tests  As  Documenta$on   •  Tests  are  ohen  obfuscated   •  Ohen  wri5en  as  an  aherthought   •  Missing  or  non-­‐descrip$ve  test  names   •  Describe  code  logic  instead  of  business  logic   March  26,  2014   Copyright  2014,  All  Around  The  World  
  37. 37. Making  Tests  Documenta$on   •  Write  them  to  be  read   •  Describe  use  cases  with:   – test  fixtures  names   – subtest  names   – test  method  names   March  26,  2014   Copyright  2014,  All  Around  The  World  
  38. 38. March  26,  2014   Copyright  2014,  All  Around  The  World  
  39. 39. March  26,  2014   Copyright  2014,  All  Around  The  World  
  40. 40. March  26,  2014   Copyright  2014,  All  Around  The  World  
  41. 41. BDD  for  Test  Documentaiton   •  Test::BDD::Cucumber   •  Perl  Behavior  Driven  Development  (slideshare)   March  26,  2014   Copyright  2014,  All  Around  The  World   h5p://commons.wikimedia.org/wiki/File:Og%C3%B3rki...jpg  
  42. 42. Test  Driven  Development   •  Lie:  TDD  is  a  far  superior  tes$ng  methodology   •  Truth:  Prove  it.   March  26,  2014   Copyright  2014,  All  Around  The  World  
  43. 43. A  Typical  TDD  Study   •  Evalua$ng  the  Efficacy  of  Test-­‐Driven   Development:  Industrial  Case  Studies   •  Compared  to  non-­‐TDD   •  TDD  increased  development  $me  ~15%   •  TDD  had  far  fewer  defects  than  non-­‐TDD   March  26,  2014   Copyright  2014,  All  Around  The  World  
  44. 44. What  is  “non-­‐TDD”?   •  No  tes$ng?   •  Test  last?   •  Test  as  you  go?   •  QA-­‐only  tes$ng?   •  It  doesn’t  say!   March  26,  2014   Copyright  2014,  All  Around  The  World  
  45. 45. Same  Study   “An  (sic)  year-­‐long  empirical  study  performed  at   IBM  using  professional  programmers  found  that   the  TDD  prac?ce  helps  programmers  produce   higher  quality  code.”   March  26,  2014   Copyright  2014,  All  Around  The  World  
  46. 46. Same  Study   “An  (sic)  year-­‐long  empirical  study  performed  at   IBM  using  professional  programmers  found  that   the  TDD  prac?ce  helps  programmers  produce   higher  quality  code.”   Higher  quality  code  than  what?   And  what  does  “quality”  mean?   March  26,  2014   Copyright  2014,  All  Around  The  World  
  47. 47. Pro-­‐TDD  Studies   •  Comparisons  against  straw  men  common   •  Ohen  use  students   •  Some$mes  use  single  developers   •  Some$mes  aware  they’re  being  studied   •  Or  new  to  TDD  (may  be  more  mo$vated)   March  26,  2014   Copyright  2014,  All  Around  The  World  
  48. 48. A  TDD  Enthusiast  Looks  At  Evidence   •  The  benefits  of  TDD  are  neither  clear  nor  are   they  apparent  —  Kane  Mar   – Part  1   – Part  2   – Part  3   March  26,  2014   Copyright  2014,  All  Around  The  World  
  49. 49. Kane  Mar  –  Aher  Reading  Studies   “[Claims  about  TDD]  such  as  beHer  design,   beHer  APIs,  simpler  design,  lower  complexity,   increased  produc?vity,  more  maintainable  code   etc.,  are  simply  not  supported.”1   1.  A  cherry-­‐picked,  out-­‐of-­‐context  quote.   March  26,  2014   Copyright  2014,  All  Around  The  World  
  50. 50. The  Reality  (vis-­‐à-­‐vis  other  tes$ng  strategies)   •  TDD  tends  to  generate  more  tests   •  Higher  quality  not  proven   •  Absence  of  evidence,  not  evidence  of  absence   •  Don’t  stress  about  it   March  26,  2014   Copyright  2014,  All  Around  The  World  
  51. 51. Summary   •  Evidence  trumps  anecdotes   •  We  rely  on  anecdotes   •  That’s  sort  of  OK  if  we  don’t  have  evidence˜   •  Code  coverage  is  only  part  of  the  solu$on   •  We  need  to  escape  our  bubble   •  Be  skep$cal  of  hype   •  Be  skep$cal  of  me   March  26,  2014   Copyright  2014,  All  Around  The  World  
  52. 52. Summary  for  TDD  Fana$cs   •  Don't  sweat  anecdotes  for  casual  ideas   •  Don’t  accept  anecdotes  from  dogma$sts   March  26,  2014   Copyright  2014,  All  Around  The  World  
  53. 53. March  26,  2014   Copyright  2014,  All  Around  The  World   Image  by  Hay  Kranen  
  54. 54. Bonus  Slides!   March  26,  2014   Copyright  2014,  All  Around  The  World  
  55. 55. Basis  Path  Tes$ng   March  26,  2014   Copyright  2014,  All  Around  The  World  
  56. 56. Cycloma$c  Complexity   March  26,  2014   Copyright  2014,  All  Around  The  World   Cycloma$c  Complexity  (McCabe)   =  (edges  –  nodes)  +  (  2  *  exit  nodes  )   =  (  19            –  14            )  +    (  2  *  1  )   =  7   =  minimum  number  tests  
  57. 57. Calcula$ng  Cycloma$c  Complexity   March  26,  2014   Copyright  2014,  All  Around  The  World  
  58. 58. Cycloma$c  Complexity  and  Bugs   •  Some  studies  show  a  posi$ve  correla$on   •  Other  studies  are  not  conclusive   •  May  make  bugs  harder  to  fix   •  May  lead  to  side  effects  when  making  changes   •  A  Complexity  Measure  —  McCabe   •  A  Cri$que  of  Sohware  Defect  Predic$on  Models  —  Fenton,  Neil     •  Cycloma$c  Complexity  Metrics  Revisited  —  Gill,  Kemerer   March  26,  2014   Copyright  2014,  All  Around  The  World  
  59. 59. Perl   March  26,  2014   Copyright  2014,  All  Around  The  World  

×