The Lies We Tell About Software Testing
Upcoming SlideShare
Loading in...5
×
 

The Lies We Tell About Software Testing

on

  • 848 views

This talk is more about half-truths than lies, but the concepts are important. There are several areas in which we think we have more confidence in our information than we actually do. This ...

This talk is more about half-truths than lies, but the concepts are important. There are several areas in which we think we have more confidence in our information than we actually do. This presentation covers both areas in which the evidence is against our beliefs and areas in which we simply need more information.

Statistics

Views

Total Views
848
Views on SlideShare
547
Embed Views
301

Actions

Likes
2
Downloads
4
Comments
0

2 Embeds 301

http://www.allaroundtheworld.fr 290
https://twitter.com 11

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • No evidence for or against the existence of a god or godsStrong evidence of absence of effect from astrology
  • LorenzoTorrentino 1550 1st edition. Ptolemaic view of the solar system.http://en.wikipedia.org/wiki/File:1550_SACROBOSCO_Tractatus_de_Sphaera_-_(16)_Ex_Libris_rare_-_Mario_Taddei_.JPG
  • Caveat: integration tests may have several pieces of dead code talking to each other!
  • Simple faults can be corrected by a single change

The Lies We Tell About Software Testing The Lies We Tell About Software Testing Presentation Transcript

  • Tes$ng  Lies   (The  &tle  is  the  first  lie)   Hannover,  Germany,  2014   Cur$s  “Ovid”  Poe   h5p://www.allaroundtheworld.fr/   Photo  CC  license:  h5p://commons.wikimedia.org/wiki/File:Hannover_Altstadt_128-­‐h.jpg   Copyright  2014,  All  Around  The  World  March  26,  2014  
  • But  first  …   March  26,  2014   Copyright  2014,  All  Around  The  World   h5p://www.slideshare.net/gvwilson/bits-­‐of-­‐evidence-­‐2338367  
  • Bits  of  Evidence  by  Dr.  Greg  Wilson   March  26,  2014   Copyright  2014,  All  Around  The  World   •  Overwhelming  evidence  that  evidence  works   •  Science  demands  evidence   •  Programmers  provide  anecdotes   •  (…  and  defend  them  vigorously)  
  • Evidence   •  Absence  of  evidence  is  not  evidence   •  …  but  evidence  of  absence  is   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Absence  of  Evidence   •  There  is  no  evidence  that  gods  exist   •  There  is  no  evidence  that  gods  don’t  exist   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Evidence  of  Absence   •  Belief:  small  subs  more  maintainable/correct   •  Reality:  these  proper$es  are  absent   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Evidence  of  Absence   •  Belief:  small  subrou$nes/methods  “be5er”   •  Inverse  correla$on  (Bell,  Perricone  1984)   •  No  correla$on  (Shen,  et  al.  1985)   •  No  correla$on  and  larger  is  cheaper  (two  studies)   •  Small  rou$nes  had  more  errors  (Selbi,  Basil  1991)   •  Less  maintenance  for  larger  rou$nes  (Lind,   Vairavan  1989)   •  500+  lines  had  more  errors  (Jones  1986)   (cita$ons  via:  Code  Complete,  Steve  McConnell,  Microsoh  Press,  2nd  Edi$on)   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • P.O.P.  Copyright  2013,  Cur$s  "Ovid"  Poe   Reality  trumps  opinion   3/26/14  
  • Yeah,  yeah,  let’s  get  started   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Caveats   March  26,  2014   Copyright  2014,  All  Around  The  World   •  Some  points  self-­‐evident   •  Some  points  maddening   •  Not  many  studies   •  Always  be  skep$cal   Photo  CC  BY-­‐SA  2.0  h5p://www.flickr.com/photos/busbeytheelder/515542705/  
  • Code  Coverage   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 100%  Code  Coverage   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 100%  Code  Coverage   •  What  if  we  pass  zero?   •  …  or  a  string?   •  …  or  no  argument?   •  …  or  a  filehandle?   •  …  or  type  when  reciprocals  don't  make  sense?   •  …  or,  or,  or  …   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Expecta$ons   •  Tes$ng  should  be  an  act  of  aggression   •  Programmers  tend  to  test  against  expecta$ons   •  False  confidence   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Code  Coverage  of  Dead  Code   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Unit  Versus  Integra$on  Tests   •  Unit  tested  subs  don’t  talk  to  other  subs   •  But  your  code  certainly  does!   •  Dead  code  is  cogni$ve  overhead   •  Dead  code  can’t  be  tested  via  your  interface   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Code  Coverage  of  Dead  Code   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • March  26,  2014   Copyright  2014,  All  Around  The  World  
  • March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Covered  but  not  tested   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Devel::Cover  Recap   •  100%  code  coverage  doesn’t  handle  data   •  Don’t  cover  dead  code   •  Covered  but  not  tested   •  Don't  blindly  trust  your  tools   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Kwalitee   •  Lie:  Perl  community  is  the  most  test-­‐infected   •  Truth:  Quan$ty  ain’t  quality   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • CPAN  Testers  (March  6,  2014)   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Versions  and  Plauorms   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • CPAN  Testers   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Perl  Isn’t  Stealing  Enough   We  need  quality  to  go  with  quan$ty   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • All  Pairs  Tes$ng   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • The  Coupling  Effect   •  Inves$ga$ons  of  the  Sohware  Tes$ng  Coupling   Effect  —  A.  Jefferson  Offu5   •  Competent  programmers  write  nearly  correct   programs   •  Most  faults  are  simple  faults   •  Simple  faults  can  be  easily  detected   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • All-­‐Pairs  Tes$ng   •  10  arguments  ==  10  billion  combina$ons   •  Combined  pairs  ==  54  combina$ons   •  Extend  each  argument  with  reasonable  values   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • All-­‐Pairs  Tes$ng   First  pairs   Second  Pairs   Third  Pairs   Fourth  Pairs   And  so  on   arg1,  arg2   —   —   —   …   arg1,  arg3   arg2,  arg3   —   —   …   arg1,  arg4   arg2,  arg4   arg3,  arg4   —   …   …   …   …   arg4,  arg5   …   arg1,  arg0   arg2,  arg0   arg3,  arg0   …   …   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • All-­‐Pairs  Tes$ng   March  26,  2014   Copyright  2014,  All  Around  The  World   •  Fast  genera$on  of  tests   •  Likely  to  catch  many  more  bugs   •  Built  in  to  many  tools  (for  sta$c  languages)   •  For  Perl:  ALLPAIRS  Test  Case  Genera$on  Tool  
  • Documenta$on   •  Lie:  Tests  are  documenta$on   •  Truth:  I  wish   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • March  26,  2014   Copyright  2014,  All  Around  The  World  
  • March  26,  2014   Copyright  2014,  All  Around  The  World  
  • March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Tests  As  Documenta$on   •  Tests  are  ohen  obfuscated   •  Ohen  wri5en  as  an  aherthought   •  Missing  or  non-­‐descrip$ve  test  names   •  Describe  code  logic  instead  of  business  logic   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Making  Tests  Documenta$on   •  Write  them  to  be  read   •  Describe  use  cases  with:   – test  fixtures  names   – subtest  names   – test  method  names   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • March  26,  2014   Copyright  2014,  All  Around  The  World  
  • March  26,  2014   Copyright  2014,  All  Around  The  World  
  • March  26,  2014   Copyright  2014,  All  Around  The  World  
  • BDD  for  Test  Documentaiton   •  Test::BDD::Cucumber   •  Perl  Behavior  Driven  Development  (slideshare)   March  26,  2014   Copyright  2014,  All  Around  The  World   h5p://commons.wikimedia.org/wiki/File:Og%C3%B3rki...jpg  
  • Test  Driven  Development   •  Lie:  TDD  is  a  far  superior  tes$ng  methodology   •  Truth:  Prove  it.   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • A  Typical  TDD  Study   •  Evalua$ng  the  Efficacy  of  Test-­‐Driven   Development:  Industrial  Case  Studies   •  Compared  to  non-­‐TDD   •  TDD  increased  development  $me  ~15%   •  TDD  had  far  fewer  defects  than  non-­‐TDD   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • What  is  “non-­‐TDD”?   •  No  tes$ng?   •  Test  last?   •  Test  as  you  go?   •  QA-­‐only  tes$ng?   •  It  doesn’t  say!   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Same  Study   “An  (sic)  year-­‐long  empirical  study  performed  at   IBM  using  professional  programmers  found  that   the  TDD  prac?ce  helps  programmers  produce   higher  quality  code.”   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Same  Study   “An  (sic)  year-­‐long  empirical  study  performed  at   IBM  using  professional  programmers  found  that   the  TDD  prac?ce  helps  programmers  produce   higher  quality  code.”   Higher  quality  code  than  what?   And  what  does  “quality”  mean?   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Pro-­‐TDD  Studies   •  Comparisons  against  straw  men  common   •  Ohen  use  students   •  Some$mes  use  single  developers   •  Some$mes  aware  they’re  being  studied   •  Or  new  to  TDD  (may  be  more  mo$vated)   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • A  TDD  Enthusiast  Looks  At  Evidence   •  The  benefits  of  TDD  are  neither  clear  nor  are   they  apparent  —  Kane  Mar   – Part  1   – Part  2   – Part  3   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Kane  Mar  –  Aher  Reading  Studies   “[Claims  about  TDD]  such  as  beHer  design,   beHer  APIs,  simpler  design,  lower  complexity,   increased  produc?vity,  more  maintainable  code   etc.,  are  simply  not  supported.”1   1.  A  cherry-­‐picked,  out-­‐of-­‐context  quote.   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • The  Reality  (vis-­‐à-­‐vis  other  tes$ng  strategies)   •  TDD  tends  to  generate  more  tests   •  Higher  quality  not  proven   •  Absence  of  evidence,  not  evidence  of  absence   •  Don’t  stress  about  it   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Summary   •  Evidence  trumps  anecdotes   •  We  rely  on  anecdotes   •  That’s  sort  of  OK  if  we  don’t  have  evidence˜   •  Code  coverage  is  only  part  of  the  solu$on   •  We  need  to  escape  our  bubble   •  Be  skep$cal  of  hype   •  Be  skep$cal  of  me   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Summary  for  TDD  Fana$cs   •  Don't  sweat  anecdotes  for  casual  ideas   •  Don’t  accept  anecdotes  from  dogma$sts   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • March  26,  2014   Copyright  2014,  All  Around  The  World   Image  by  Hay  Kranen  
  • Bonus  Slides!   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Basis  Path  Tes$ng   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Cycloma$c  Complexity   March  26,  2014   Copyright  2014,  All  Around  The  World   Cycloma$c  Complexity  (McCabe)   =  (edges  –  nodes)  +  (  2  *  exit  nodes  )   =  (  19            –  14            )  +    (  2  *  1  )   =  7   =  minimum  number  tests  
  • Calcula$ng  Cycloma$c  Complexity   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Cycloma$c  Complexity  and  Bugs   •  Some  studies  show  a  posi$ve  correla$on   •  Other  studies  are  not  conclusive   •  May  make  bugs  harder  to  fix   •  May  lead  to  side  effects  when  making  changes   •  A  Complexity  Measure  —  McCabe   •  A  Cri$que  of  Sohware  Defect  Predic$on  Models  —  Fenton,  Neil     •  Cycloma$c  Complexity  Metrics  Revisited  —  Gill,  Kemerer   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • Perl   March  26,  2014   Copyright  2014,  All  Around  The  World