SlideShare a Scribd company logo
Tes$ng	
  Lies	
  
(The	
  &tle	
  is	
  the	
  first	
  lie)	
  
Hannover,	
  Germany,	
  2014	
  
Cur$s	
  “Ovid”	
  Poe	
  
h5p://www.allaroundtheworld.fr/	
  
Photo	
  CC	
  license:	
  h5p://commons.wikimedia.org/wiki/File:Hannover_Altstadt_128-­‐h.jpg	
  
Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  March	
  26,	
  2014	
  
But	
  first	
  …	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
h5p://www.slideshare.net/gvwilson/bits-­‐of-­‐evidence-­‐2338367	
  
Bits	
  of	
  Evidence	
  by	
  Dr.	
  Greg	
  Wilson	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
•  Overwhelming	
  evidence	
  that	
  evidence	
  works	
  
•  Science	
  demands	
  evidence	
  
•  Programmers	
  provide	
  anecdotes	
  
•  (…	
  and	
  defend	
  them	
  vigorously)	
  
Evidence	
  
•  Absence	
  of	
  evidence	
  is	
  not	
  evidence	
  
•  …	
  but	
  evidence	
  of	
  absence	
  is	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Absence	
  of	
  Evidence	
  
•  There	
  is	
  no	
  evidence	
  that	
  gods	
  exist	
  
•  There	
  is	
  no	
  evidence	
  that	
  gods	
  don’t	
  exist	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Evidence	
  of	
  Absence	
  
•  Belief:	
  small	
  subs	
  more	
  maintainable/correct	
  
•  Reality:	
  these	
  proper$es	
  are	
  absent	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Evidence	
  of	
  Absence	
  
•  Belief:	
  small	
  subrou$nes/methods	
  “be5er”	
  
•  Inverse	
  correla$on	
  (Bell,	
  Perricone	
  1984)	
  
•  No	
  correla$on	
  (Shen,	
  et	
  al.	
  1985)	
  
•  No	
  correla$on	
  and	
  larger	
  is	
  cheaper	
  (two	
  studies)	
  
•  Small	
  rou$nes	
  had	
  more	
  errors	
  (Selbi,	
  Basil	
  1991)	
  
•  Less	
  maintenance	
  for	
  larger	
  rou$nes	
  (Lind,	
  
Vairavan	
  1989)	
  
•  500+	
  lines	
  had	
  more	
  errors	
  (Jones	
  1986)	
  
(cita$ons	
  via:	
  Code	
  Complete,	
  Steve	
  McConnell,	
  Microsoh	
  Press,	
  2nd	
  Edi$on)	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
P.O.P.	
  Copyright	
  2013,	
  Cur$s	
  "Ovid"	
  Poe	
  
Reality	
  trumps	
  opinion	
  
3/26/14	
  
Yeah,	
  yeah,	
  let’s	
  get	
  started	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Caveats	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
•  Some	
  points	
  self-­‐evident	
  
•  Some	
  points	
  maddening	
  
•  Not	
  many	
  studies	
  
•  Always	
  be	
  skep$cal	
  
Photo	
  CC	
  BY-­‐SA	
  2.0	
  h5p://www.flickr.com/photos/busbeytheelder/515542705/	
  
Code	
  Coverage	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
100%	
  Code	
  Coverage	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
100%	
  Code	
  Coverage	
  
•  What	
  if	
  we	
  pass	
  zero?	
  
•  …	
  or	
  a	
  string?	
  
•  …	
  or	
  no	
  argument?	
  
•  …	
  or	
  a	
  filehandle?	
  
•  …	
  or	
  type	
  when	
  reciprocals	
  don't	
  make	
  sense?	
  
•  …	
  or,	
  or,	
  or	
  …	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Expecta$ons	
  
•  Tes$ng	
  should	
  be	
  an	
  act	
  of	
  aggression	
  
•  Programmers	
  tend	
  to	
  test	
  against	
  expecta$ons	
  
•  False	
  confidence	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Code	
  Coverage	
  of	
  Dead	
  Code	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Unit	
  Versus	
  Integra$on	
  Tests	
  
•  Unit	
  tested	
  subs	
  don’t	
  talk	
  to	
  other	
  subs	
  
•  But	
  your	
  code	
  certainly	
  does!	
  
•  Dead	
  code	
  is	
  cogni$ve	
  overhead	
  
•  Dead	
  code	
  can’t	
  be	
  tested	
  via	
  your	
  interface	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Code	
  Coverage	
  of	
  Dead	
  Code	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Covered	
  but	
  not	
  tested	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Devel::Cover	
  Recap	
  
•  100%	
  code	
  coverage	
  doesn’t	
  handle	
  data	
  
•  Don’t	
  cover	
  dead	
  code	
  
•  Covered	
  but	
  not	
  tested	
  
•  Don't	
  blindly	
  trust	
  your	
  tools	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Kwalitee	
  
•  Lie:	
  Perl	
  community	
  is	
  the	
  most	
  test-­‐infected	
  
•  Truth:	
  Quan$ty	
  ain’t	
  quality	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
CPAN	
  Testers	
  (March	
  6,	
  2014)	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Versions	
  and	
  Plauorms	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
CPAN	
  Testers	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Perl	
  Isn’t	
  Stealing	
  Enough	
  
We	
  need	
  quality	
  to	
  go	
  with	
  quan$ty	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
All	
  Pairs	
  Tes$ng	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
The	
  Coupling	
  Effect	
  
•  Inves$ga$ons	
  of	
  the	
  Sohware	
  Tes$ng	
  Coupling	
  
Effect	
  —	
  A.	
  Jefferson	
  Offu5	
  
•  Competent	
  programmers	
  write	
  nearly	
  correct	
  
programs	
  
•  Most	
  faults	
  are	
  simple	
  faults	
  
•  Simple	
  faults	
  can	
  be	
  easily	
  detected	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
All-­‐Pairs	
  Tes$ng	
  
•  10	
  arguments	
  ==	
  10	
  billion	
  combina$ons	
  
•  Combined	
  pairs	
  ==	
  54	
  combina$ons	
  
•  Extend	
  each	
  argument	
  with	
  reasonable	
  values	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
All-­‐Pairs	
  Tes$ng	
  
First	
  pairs	
   Second	
  Pairs	
   Third	
  Pairs	
   Fourth	
  Pairs	
   And	
  so	
  on	
  
arg1,	
  arg2	
   —	
   —	
   —	
   …	
  
arg1,	
  arg3	
   arg2,	
  arg3	
   —	
   —	
   …	
  
arg1,	
  arg4	
   arg2,	
  arg4	
   arg3,	
  arg4	
   —	
   …	
  
…	
   …	
   …	
   arg4,	
  arg5	
   …	
  
arg1,	
  arg0	
   arg2,	
  arg0	
   arg3,	
  arg0	
   …	
   …	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
All-­‐Pairs	
  Tes$ng	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
•  Fast	
  genera$on	
  of	
  tests	
  
•  Likely	
  to	
  catch	
  many	
  more	
  bugs	
  
•  Built	
  in	
  to	
  many	
  tools	
  (for	
  sta$c	
  languages)	
  
•  For	
  Perl:	
  ALLPAIRS	
  Test	
  Case	
  Genera$on	
  Tool	
  
Documenta$on	
  
•  Lie:	
  Tests	
  are	
  documenta$on	
  
•  Truth:	
  I	
  wish	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Tests	
  As	
  Documenta$on	
  
•  Tests	
  are	
  ohen	
  obfuscated	
  
•  Ohen	
  wri5en	
  as	
  an	
  aherthought	
  
•  Missing	
  or	
  non-­‐descrip$ve	
  test	
  names	
  
•  Describe	
  code	
  logic	
  instead	
  of	
  business	
  logic	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Making	
  Tests	
  Documenta$on	
  
•  Write	
  them	
  to	
  be	
  read	
  
•  Describe	
  use	
  cases	
  with:	
  
– test	
  fixtures	
  names	
  
– subtest	
  names	
  
– test	
  method	
  names	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
BDD	
  for	
  Test	
  Documentaiton	
  
•  Test::BDD::Cucumber	
  
•  Perl	
  Behavior	
  Driven	
  Development	
  (slideshare)	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
h5p://commons.wikimedia.org/wiki/File:Og%C3%B3rki...jpg	
  
Test	
  Driven	
  Development	
  
•  Lie:	
  TDD	
  is	
  a	
  far	
  superior	
  tes$ng	
  methodology	
  
•  Truth:	
  Prove	
  it.	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
A	
  Typical	
  TDD	
  Study	
  
•  Evalua$ng	
  the	
  Efficacy	
  of	
  Test-­‐Driven	
  
Development:	
  Industrial	
  Case	
  Studies	
  
•  Compared	
  to	
  non-­‐TDD	
  
•  TDD	
  increased	
  development	
  $me	
  ~15%	
  
•  TDD	
  had	
  far	
  fewer	
  defects	
  than	
  non-­‐TDD	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
What	
  is	
  “non-­‐TDD”?	
  
•  No	
  tes$ng?	
  
•  Test	
  last?	
  
•  Test	
  as	
  you	
  go?	
  
•  QA-­‐only	
  tes$ng?	
  
•  It	
  doesn’t	
  say!	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Same	
  Study	
  
“An	
  (sic)	
  year-­‐long	
  empirical	
  study	
  performed	
  at	
  
IBM	
  using	
  professional	
  programmers	
  found	
  that	
  
the	
  TDD	
  prac?ce	
  helps	
  programmers	
  produce	
  
higher	
  quality	
  code.”	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Same	
  Study	
  
“An	
  (sic)	
  year-­‐long	
  empirical	
  study	
  performed	
  at	
  
IBM	
  using	
  professional	
  programmers	
  found	
  that	
  
the	
  TDD	
  prac?ce	
  helps	
  programmers	
  produce	
  
higher	
  quality	
  code.”	
  
Higher	
  quality	
  code	
  than	
  what?	
  
And	
  what	
  does	
  “quality”	
  mean?	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Pro-­‐TDD	
  Studies	
  
•  Comparisons	
  against	
  straw	
  men	
  common	
  
•  Ohen	
  use	
  students	
  
•  Some$mes	
  use	
  single	
  developers	
  
•  Some$mes	
  aware	
  they’re	
  being	
  studied	
  
•  Or	
  new	
  to	
  TDD	
  (may	
  be	
  more	
  mo$vated)	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
A	
  TDD	
  Enthusiast	
  Looks	
  At	
  Evidence	
  
•  The	
  benefits	
  of	
  TDD	
  are	
  neither	
  clear	
  nor	
  are	
  
they	
  apparent	
  —	
  Kane	
  Mar	
  
– Part	
  1	
  
– Part	
  2	
  
– Part	
  3	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Kane	
  Mar	
  –	
  Aher	
  Reading	
  Studies	
  
“[Claims	
  about	
  TDD]	
  such	
  as	
  beHer	
  design,	
  
beHer	
  APIs,	
  simpler	
  design,	
  lower	
  complexity,	
  
increased	
  produc?vity,	
  more	
  maintainable	
  code	
  
etc.,	
  are	
  simply	
  not	
  supported.”1	
  
1.	
  A	
  cherry-­‐picked,	
  out-­‐of-­‐context	
  quote.	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
The	
  Reality	
  (vis-­‐à-­‐vis	
  other	
  tes$ng	
  strategies)	
  
•  TDD	
  tends	
  to	
  generate	
  more	
  tests	
  
•  Higher	
  quality	
  not	
  proven	
  
•  Absence	
  of	
  evidence,	
  not	
  evidence	
  of	
  absence	
  
•  Don’t	
  stress	
  about	
  it	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Summary	
  
•  Evidence	
  trumps	
  anecdotes	
  
•  We	
  rely	
  on	
  anecdotes	
  
•  That’s	
  sort	
  of	
  OK	
  if	
  we	
  don’t	
  have	
  evidence˜	
  
•  Code	
  coverage	
  is	
  only	
  part	
  of	
  the	
  solu$on	
  
•  We	
  need	
  to	
  escape	
  our	
  bubble	
  
•  Be	
  skep$cal	
  of	
  hype	
  
•  Be	
  skep$cal	
  of	
  me	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Summary	
  for	
  TDD	
  Fana$cs	
  
•  Don't	
  sweat	
  anecdotes	
  for	
  casual	
  ideas	
  
•  Don’t	
  accept	
  anecdotes	
  from	
  dogma$sts	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Image	
  by	
  Hay	
  Kranen	
  
Bonus	
  Slides!	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Basis	
  Path	
  Tes$ng	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Cycloma$c	
  Complexity	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Cycloma$c	
  Complexity	
  (McCabe)	
  
=	
  (edges	
  –	
  nodes)	
  +	
  (	
  2	
  *	
  exit	
  nodes	
  )	
  
=	
  (	
  19	
  	
  	
  	
  	
  	
  –	
  14	
  	
  	
  	
  	
  	
  )	
  +	
  	
  (	
  2	
  *	
  1	
  )	
  
=	
  7	
  
=	
  minimum	
  number	
  tests	
  
Calcula$ng	
  Cycloma$c	
  Complexity	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Cycloma$c	
  Complexity	
  and	
  Bugs	
  
•  Some	
  studies	
  show	
  a	
  posi$ve	
  correla$on	
  
•  Other	
  studies	
  are	
  not	
  conclusive	
  
•  May	
  make	
  bugs	
  harder	
  to	
  fix	
  
•  May	
  lead	
  to	
  side	
  effects	
  when	
  making	
  changes	
  
•  A	
  Complexity	
  Measure	
  —	
  McCabe	
  
•  A	
  Cri$que	
  of	
  Sohware	
  Defect	
  Predic$on	
  Models	
  —	
  Fenton,	
  Neil	
  	
  
•  Cycloma$c	
  Complexity	
  Metrics	
  Revisited	
  —	
  Gill,	
  Kemerer	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  
Perl	
  
March	
  26,	
  2014	
   Copyright	
  2014,	
  All	
  Around	
  The	
  World	
  

More Related Content

Viewers also liked

Dark Matter, Public Health, and Scientific Computing
Dark Matter, Public Health, and Scientific ComputingDark Matter, Public Health, and Scientific Computing
Dark Matter, Public Health, and Scientific Computing
Greg Wilson
 
What We've Learned From Building Basie
What We've Learned From Building BasieWhat We've Learned From Building Basie
What We've Learned From Building Basie
Greg Wilson
 
Bits of Evidence
Bits of EvidenceBits of Evidence
Bits of Evidence
Greg Wilson
 
Cómo evitar la corrosión
Cómo evitar la corrosión Cómo evitar la corrosión
Cómo evitar la corrosión
Pamela Carrillo
 
Market Research Report: Courier Market in India 2012
Market Research Report: Courier Market in India 2012Market Research Report: Courier Market in India 2012
Market Research Report: Courier Market in India 2012
Netscribes, Inc.
 
Reunión padres 6º 2014 15
Reunión padres 6º 2014 15Reunión padres 6º 2014 15
Reunión padres 6º 2014 15
josetiradorguez
 
Karst in Southern Ontario and the Niagara Escarpment
Karst in Southern Ontario and the Niagara EscarpmentKarst in Southern Ontario and the Niagara Escarpment
Karst in Southern Ontario and the Niagara Escarpment
Skyler MacGowan
 
Software para integración de aplicaciones, paquetes u otras técnicas en bases...
Software para integración de aplicaciones, paquetes u otras técnicas en bases...Software para integración de aplicaciones, paquetes u otras técnicas en bases...
Software para integración de aplicaciones, paquetes u otras técnicas en bases...
Roy Baltodano
 
juklak-penyusunan-proyeksi-kebutuhan-kepala-sekolah-madra
juklak-penyusunan-proyeksi-kebutuhan-kepala-sekolah-madrajuklak-penyusunan-proyeksi-kebutuhan-kepala-sekolah-madra
juklak-penyusunan-proyeksi-kebutuhan-kepala-sekolah-madra
largono drs
 
Mª Teresa García Silva - 'Aunando esfuerzos para el futuro de la investigació...
Mª Teresa García Silva - 'Aunando esfuerzos para el futuro de la investigació...Mª Teresa García Silva - 'Aunando esfuerzos para el futuro de la investigació...
Mª Teresa García Silva - 'Aunando esfuerzos para el futuro de la investigació...
Fundación Ramón Areces
 
Decoesfera.11.09
Decoesfera.11.09Decoesfera.11.09
Vol03 01 jett_gripenberg_lizarte finlandia
Vol03 01 jett_gripenberg_lizarte finlandiaVol03 01 jett_gripenberg_lizarte finlandia
Vol03 01 jett_gripenberg_lizarte finlandia
Juan Becerra López
 
Letras contemporáneas
Letras contemporáneasLetras contemporáneas
Letras contemporáneas
Dara Iris
 
El Mercantil Agosto 2010
El Mercantil Agosto 2010El Mercantil Agosto 2010
El Mercantil Agosto 2010
Absolutgest s.l
 
HTW2011: Thai Tran - Android: Opportunities and challenges
HTW2011: Thai Tran - Android: Opportunities and challengesHTW2011: Thai Tran - Android: Opportunities and challenges
HTW2011: Thai Tran - Android: Opportunities and challenges
How to Web
 
Vancouver1
Vancouver1Vancouver1
Vancouver1
Forensic Pathology
 
Monitoring and Managing Computer Resource Usage on OSGi Frameworks - Ikuo Yam...
Monitoring and Managing Computer Resource Usage on OSGi Frameworks - Ikuo Yam...Monitoring and Managing Computer Resource Usage on OSGi Frameworks - Ikuo Yam...
Monitoring and Managing Computer Resource Usage on OSGi Frameworks - Ikuo Yam...
mfrancis
 
Fundamental Elements, upcoming opportunities - Blasi, Hriscu
Fundamental Elements, upcoming opportunities - Blasi, HriscuFundamental Elements, upcoming opportunities - Blasi, Hriscu
Fundamental Elements, upcoming opportunities - Blasi, Hriscu
The European GNSS Agency (GSA)
 
Belgian eID cards - technicalities
Belgian eID cards - technicalitiesBelgian eID cards - technicalities
Belgian eID cards - technicalities
beires
 
Bollettino Rurale 18: Incubatore di rete
Bollettino Rurale 18: Incubatore di reteBollettino Rurale 18: Incubatore di rete
Bollettino Rurale 18: Incubatore di rete
GAL Sole Grano Terra
 

Viewers also liked (20)

Dark Matter, Public Health, and Scientific Computing
Dark Matter, Public Health, and Scientific ComputingDark Matter, Public Health, and Scientific Computing
Dark Matter, Public Health, and Scientific Computing
 
What We've Learned From Building Basie
What We've Learned From Building BasieWhat We've Learned From Building Basie
What We've Learned From Building Basie
 
Bits of Evidence
Bits of EvidenceBits of Evidence
Bits of Evidence
 
Cómo evitar la corrosión
Cómo evitar la corrosión Cómo evitar la corrosión
Cómo evitar la corrosión
 
Market Research Report: Courier Market in India 2012
Market Research Report: Courier Market in India 2012Market Research Report: Courier Market in India 2012
Market Research Report: Courier Market in India 2012
 
Reunión padres 6º 2014 15
Reunión padres 6º 2014 15Reunión padres 6º 2014 15
Reunión padres 6º 2014 15
 
Karst in Southern Ontario and the Niagara Escarpment
Karst in Southern Ontario and the Niagara EscarpmentKarst in Southern Ontario and the Niagara Escarpment
Karst in Southern Ontario and the Niagara Escarpment
 
Software para integración de aplicaciones, paquetes u otras técnicas en bases...
Software para integración de aplicaciones, paquetes u otras técnicas en bases...Software para integración de aplicaciones, paquetes u otras técnicas en bases...
Software para integración de aplicaciones, paquetes u otras técnicas en bases...
 
juklak-penyusunan-proyeksi-kebutuhan-kepala-sekolah-madra
juklak-penyusunan-proyeksi-kebutuhan-kepala-sekolah-madrajuklak-penyusunan-proyeksi-kebutuhan-kepala-sekolah-madra
juklak-penyusunan-proyeksi-kebutuhan-kepala-sekolah-madra
 
Mª Teresa García Silva - 'Aunando esfuerzos para el futuro de la investigació...
Mª Teresa García Silva - 'Aunando esfuerzos para el futuro de la investigació...Mª Teresa García Silva - 'Aunando esfuerzos para el futuro de la investigació...
Mª Teresa García Silva - 'Aunando esfuerzos para el futuro de la investigació...
 
Decoesfera.11.09
Decoesfera.11.09Decoesfera.11.09
Decoesfera.11.09
 
Vol03 01 jett_gripenberg_lizarte finlandia
Vol03 01 jett_gripenberg_lizarte finlandiaVol03 01 jett_gripenberg_lizarte finlandia
Vol03 01 jett_gripenberg_lizarte finlandia
 
Letras contemporáneas
Letras contemporáneasLetras contemporáneas
Letras contemporáneas
 
El Mercantil Agosto 2010
El Mercantil Agosto 2010El Mercantil Agosto 2010
El Mercantil Agosto 2010
 
HTW2011: Thai Tran - Android: Opportunities and challenges
HTW2011: Thai Tran - Android: Opportunities and challengesHTW2011: Thai Tran - Android: Opportunities and challenges
HTW2011: Thai Tran - Android: Opportunities and challenges
 
Vancouver1
Vancouver1Vancouver1
Vancouver1
 
Monitoring and Managing Computer Resource Usage on OSGi Frameworks - Ikuo Yam...
Monitoring and Managing Computer Resource Usage on OSGi Frameworks - Ikuo Yam...Monitoring and Managing Computer Resource Usage on OSGi Frameworks - Ikuo Yam...
Monitoring and Managing Computer Resource Usage on OSGi Frameworks - Ikuo Yam...
 
Fundamental Elements, upcoming opportunities - Blasi, Hriscu
Fundamental Elements, upcoming opportunities - Blasi, HriscuFundamental Elements, upcoming opportunities - Blasi, Hriscu
Fundamental Elements, upcoming opportunities - Blasi, Hriscu
 
Belgian eID cards - technicalities
Belgian eID cards - technicalitiesBelgian eID cards - technicalities
Belgian eID cards - technicalities
 
Bollettino Rurale 18: Incubatore di rete
Bollettino Rurale 18: Incubatore di reteBollettino Rurale 18: Incubatore di rete
Bollettino Rurale 18: Incubatore di rete
 

Similar to The Lies We Tell About Software Testing

PureApplication for testers
PureApplication for testersPureApplication for testers
PureApplication for testers
John Hawkins
 
Aten ntc-stories
Aten ntc-storiesAten ntc-stories
Aten ntc-stories
J Matthew Saunders
 
Recommendation Techn
Recommendation TechnRecommendation Techn
Recommendation Techn
Ted Dunning
 
Semantic Integration with Apache Jena and Stanbol
Semantic Integration with Apache Jena and StanbolSemantic Integration with Apache Jena and Stanbol
Semantic Integration with Apache Jena and Stanbol
All Things Open
 
Your PhD and You
Your PhD and YouYour PhD and You
Your PhD and You
Julián Urbano
 
Moving to Continuous Delivery without breaking everything
Moving to Continuous Delivery without breaking everythingMoving to Continuous Delivery without breaking everything
Moving to Continuous Delivery without breaking everything
XebiaLabs
 
Linked Data Snowball, or Why We Need Reconciliation
Linked Data Snowball, or Why We Need ReconciliationLinked Data Snowball, or Why We Need Reconciliation
Linked Data Snowball, or Why We Need Reconciliation
Robert Sanderson
 
Introduction to Semantics for Digital Surrey
Introduction to Semantics for Digital SurreyIntroduction to Semantics for Digital Surrey
Introduction to Semantics for Digital Surrey
logomachy
 

Similar to The Lies We Tell About Software Testing (8)

PureApplication for testers
PureApplication for testersPureApplication for testers
PureApplication for testers
 
Aten ntc-stories
Aten ntc-storiesAten ntc-stories
Aten ntc-stories
 
Recommendation Techn
Recommendation TechnRecommendation Techn
Recommendation Techn
 
Semantic Integration with Apache Jena and Stanbol
Semantic Integration with Apache Jena and StanbolSemantic Integration with Apache Jena and Stanbol
Semantic Integration with Apache Jena and Stanbol
 
Your PhD and You
Your PhD and YouYour PhD and You
Your PhD and You
 
Moving to Continuous Delivery without breaking everything
Moving to Continuous Delivery without breaking everythingMoving to Continuous Delivery without breaking everything
Moving to Continuous Delivery without breaking everything
 
Linked Data Snowball, or Why We Need Reconciliation
Linked Data Snowball, or Why We Need ReconciliationLinked Data Snowball, or Why We Need Reconciliation
Linked Data Snowball, or Why We Need Reconciliation
 
Introduction to Semantics for Digital Surrey
Introduction to Semantics for Digital SurreyIntroduction to Semantics for Digital Surrey
Introduction to Semantics for Digital Surrey
 

More from Curtis Poe

How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
Curtis Poe
 
life-off-earth.pptx
life-off-earth.pptxlife-off-earth.pptx
life-off-earth.pptx
Curtis Poe
 
Corinna-2023.pptx
Corinna-2023.pptxCorinna-2023.pptx
Corinna-2023.pptx
Curtis Poe
 
Corinna Status 2022.pptx
Corinna Status 2022.pptxCorinna Status 2022.pptx
Corinna Status 2022.pptx
Curtis Poe
 
Rummaging in the clOOset
Rummaging in the clOOsetRummaging in the clOOset
Rummaging in the clOOset
Curtis Poe
 
Rescuing a-legacy-codebase
Rescuing a-legacy-codebaseRescuing a-legacy-codebase
Rescuing a-legacy-codebase
Curtis Poe
 
Perl 6 For Mere Mortals
Perl 6 For Mere MortalsPerl 6 For Mere Mortals
Perl 6 For Mere Mortals
Curtis Poe
 
A Whirlwind Tour of Test::Class
A Whirlwind Tour of Test::ClassA Whirlwind Tour of Test::Class
A Whirlwind Tour of Test::Class
Curtis Poe
 
Agile Companies Go P.O.P.
Agile Companies Go P.O.P.Agile Companies Go P.O.P.
Agile Companies Go P.O.P.
Curtis Poe
 
Econ101
Econ101Econ101
Econ101
Curtis Poe
 
Testing With Test::Class
Testing With Test::ClassTesting With Test::Class
Testing With Test::Class
Curtis Poe
 
Logic Progamming in Perl
Logic Progamming in PerlLogic Progamming in Perl
Logic Progamming in Perl
Curtis Poe
 
Turbo Charged Test Suites
Turbo Charged Test SuitesTurbo Charged Test Suites
Turbo Charged Test Suites
Curtis Poe
 

More from Curtis Poe (13)

How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
 
life-off-earth.pptx
life-off-earth.pptxlife-off-earth.pptx
life-off-earth.pptx
 
Corinna-2023.pptx
Corinna-2023.pptxCorinna-2023.pptx
Corinna-2023.pptx
 
Corinna Status 2022.pptx
Corinna Status 2022.pptxCorinna Status 2022.pptx
Corinna Status 2022.pptx
 
Rummaging in the clOOset
Rummaging in the clOOsetRummaging in the clOOset
Rummaging in the clOOset
 
Rescuing a-legacy-codebase
Rescuing a-legacy-codebaseRescuing a-legacy-codebase
Rescuing a-legacy-codebase
 
Perl 6 For Mere Mortals
Perl 6 For Mere MortalsPerl 6 For Mere Mortals
Perl 6 For Mere Mortals
 
A Whirlwind Tour of Test::Class
A Whirlwind Tour of Test::ClassA Whirlwind Tour of Test::Class
A Whirlwind Tour of Test::Class
 
Agile Companies Go P.O.P.
Agile Companies Go P.O.P.Agile Companies Go P.O.P.
Agile Companies Go P.O.P.
 
Econ101
Econ101Econ101
Econ101
 
Testing With Test::Class
Testing With Test::ClassTesting With Test::Class
Testing With Test::Class
 
Logic Progamming in Perl
Logic Progamming in PerlLogic Progamming in Perl
Logic Progamming in Perl
 
Turbo Charged Test Suites
Turbo Charged Test SuitesTurbo Charged Test Suites
Turbo Charged Test Suites
 

Recently uploaded

Tailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer InsightsTailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer Insights
SynapseIndia
 
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
SynapseIndia
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
Steven Carlson
 
Mastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for SuccessMastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for Success
David Wilson
 
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
Neo4j
 
Semantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software DevelopmentSemantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software Development
Baishakhi Ray
 
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
Priyanka Aash
 
(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf
(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf
(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf
Priyanka Aash
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
Ivanti
 
Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024
aakash malhotra
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
Shiv Technolabs
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
maigasapphire
 
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision MakingConnector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
DianaGray10
 
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdfAcumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
BrainSell Technologies
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
Priyanka Aash
 
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
SAI KAILASH R
 
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
aslasdfmkhan4750
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
Priyanka Aash
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
Safe Software
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Networks
 

Recently uploaded (20)

Tailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer InsightsTailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer Insights
 
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
 
Mastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for SuccessMastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for Success
 
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
 
Semantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software DevelopmentSemantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software Development
 
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
(CISOPlatform Summit & SACON 2024) Gen AI & Deepfake In Overall Security.pdf
 
(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf
(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf
(CISOPlatform Summit & SACON 2024) Cyber Insurance & Risk Quantification.pdf
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
 
Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
 
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision MakingConnector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
 
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdfAcumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
 
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
 
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
High Profile Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class ...
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
 

The Lies We Tell About Software Testing

  • 1. Tes$ng  Lies   (The  &tle  is  the  first  lie)   Hannover,  Germany,  2014   Cur$s  “Ovid”  Poe   h5p://www.allaroundtheworld.fr/   Photo  CC  license:  h5p://commons.wikimedia.org/wiki/File:Hannover_Altstadt_128-­‐h.jpg   Copyright  2014,  All  Around  The  World  March  26,  2014  
  • 2. But  first  …   March  26,  2014   Copyright  2014,  All  Around  The  World   h5p://www.slideshare.net/gvwilson/bits-­‐of-­‐evidence-­‐2338367  
  • 3. Bits  of  Evidence  by  Dr.  Greg  Wilson   March  26,  2014   Copyright  2014,  All  Around  The  World   •  Overwhelming  evidence  that  evidence  works   •  Science  demands  evidence   •  Programmers  provide  anecdotes   •  (…  and  defend  them  vigorously)  
  • 4. Evidence   •  Absence  of  evidence  is  not  evidence   •  …  but  evidence  of  absence  is   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 5. Absence  of  Evidence   •  There  is  no  evidence  that  gods  exist   •  There  is  no  evidence  that  gods  don’t  exist   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 6. Evidence  of  Absence   •  Belief:  small  subs  more  maintainable/correct   •  Reality:  these  proper$es  are  absent   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 7. Evidence  of  Absence   •  Belief:  small  subrou$nes/methods  “be5er”   •  Inverse  correla$on  (Bell,  Perricone  1984)   •  No  correla$on  (Shen,  et  al.  1985)   •  No  correla$on  and  larger  is  cheaper  (two  studies)   •  Small  rou$nes  had  more  errors  (Selbi,  Basil  1991)   •  Less  maintenance  for  larger  rou$nes  (Lind,   Vairavan  1989)   •  500+  lines  had  more  errors  (Jones  1986)   (cita$ons  via:  Code  Complete,  Steve  McConnell,  Microsoh  Press,  2nd  Edi$on)   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 8. P.O.P.  Copyright  2013,  Cur$s  "Ovid"  Poe   Reality  trumps  opinion   3/26/14  
  • 9. Yeah,  yeah,  let’s  get  started   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 10. Caveats   March  26,  2014   Copyright  2014,  All  Around  The  World   •  Some  points  self-­‐evident   •  Some  points  maddening   •  Not  many  studies   •  Always  be  skep$cal   Photo  CC  BY-­‐SA  2.0  h5p://www.flickr.com/photos/busbeytheelder/515542705/  
  • 11. Code  Coverage   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 12. 100%  Code  Coverage   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 13. 100%  Code  Coverage   •  What  if  we  pass  zero?   •  …  or  a  string?   •  …  or  no  argument?   •  …  or  a  filehandle?   •  …  or  type  when  reciprocals  don't  make  sense?   •  …  or,  or,  or  …   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 14. Expecta$ons   •  Tes$ng  should  be  an  act  of  aggression   •  Programmers  tend  to  test  against  expecta$ons   •  False  confidence   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 15. Code  Coverage  of  Dead  Code   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 16. Unit  Versus  Integra$on  Tests   •  Unit  tested  subs  don’t  talk  to  other  subs   •  But  your  code  certainly  does!   •  Dead  code  is  cogni$ve  overhead   •  Dead  code  can’t  be  tested  via  your  interface   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 17. Code  Coverage  of  Dead  Code   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 18. March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 19. March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 20. Covered  but  not  tested   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 21. Devel::Cover  Recap   •  100%  code  coverage  doesn’t  handle  data   •  Don’t  cover  dead  code   •  Covered  but  not  tested   •  Don't  blindly  trust  your  tools   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 22. Kwalitee   •  Lie:  Perl  community  is  the  most  test-­‐infected   •  Truth:  Quan$ty  ain’t  quality   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 23. CPAN  Testers  (March  6,  2014)   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 24. Versions  and  Plauorms   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 25. CPAN  Testers   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 26. Perl  Isn’t  Stealing  Enough   We  need  quality  to  go  with  quan$ty   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 27. All  Pairs  Tes$ng   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 28. The  Coupling  Effect   •  Inves$ga$ons  of  the  Sohware  Tes$ng  Coupling   Effect  —  A.  Jefferson  Offu5   •  Competent  programmers  write  nearly  correct   programs   •  Most  faults  are  simple  faults   •  Simple  faults  can  be  easily  detected   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 29. All-­‐Pairs  Tes$ng   •  10  arguments  ==  10  billion  combina$ons   •  Combined  pairs  ==  54  combina$ons   •  Extend  each  argument  with  reasonable  values   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 30. All-­‐Pairs  Tes$ng   First  pairs   Second  Pairs   Third  Pairs   Fourth  Pairs   And  so  on   arg1,  arg2   —   —   —   …   arg1,  arg3   arg2,  arg3   —   —   …   arg1,  arg4   arg2,  arg4   arg3,  arg4   —   …   …   …   …   arg4,  arg5   …   arg1,  arg0   arg2,  arg0   arg3,  arg0   …   …   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 31. All-­‐Pairs  Tes$ng   March  26,  2014   Copyright  2014,  All  Around  The  World   •  Fast  genera$on  of  tests   •  Likely  to  catch  many  more  bugs   •  Built  in  to  many  tools  (for  sta$c  languages)   •  For  Perl:  ALLPAIRS  Test  Case  Genera$on  Tool  
  • 32. Documenta$on   •  Lie:  Tests  are  documenta$on   •  Truth:  I  wish   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 33. March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 34. March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 35. March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 36. Tests  As  Documenta$on   •  Tests  are  ohen  obfuscated   •  Ohen  wri5en  as  an  aherthought   •  Missing  or  non-­‐descrip$ve  test  names   •  Describe  code  logic  instead  of  business  logic   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 37. Making  Tests  Documenta$on   •  Write  them  to  be  read   •  Describe  use  cases  with:   – test  fixtures  names   – subtest  names   – test  method  names   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 38. March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 39. March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 40. March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 41. BDD  for  Test  Documentaiton   •  Test::BDD::Cucumber   •  Perl  Behavior  Driven  Development  (slideshare)   March  26,  2014   Copyright  2014,  All  Around  The  World   h5p://commons.wikimedia.org/wiki/File:Og%C3%B3rki...jpg  
  • 42. Test  Driven  Development   •  Lie:  TDD  is  a  far  superior  tes$ng  methodology   •  Truth:  Prove  it.   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 43. A  Typical  TDD  Study   •  Evalua$ng  the  Efficacy  of  Test-­‐Driven   Development:  Industrial  Case  Studies   •  Compared  to  non-­‐TDD   •  TDD  increased  development  $me  ~15%   •  TDD  had  far  fewer  defects  than  non-­‐TDD   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 44. What  is  “non-­‐TDD”?   •  No  tes$ng?   •  Test  last?   •  Test  as  you  go?   •  QA-­‐only  tes$ng?   •  It  doesn’t  say!   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 45. Same  Study   “An  (sic)  year-­‐long  empirical  study  performed  at   IBM  using  professional  programmers  found  that   the  TDD  prac?ce  helps  programmers  produce   higher  quality  code.”   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 46. Same  Study   “An  (sic)  year-­‐long  empirical  study  performed  at   IBM  using  professional  programmers  found  that   the  TDD  prac?ce  helps  programmers  produce   higher  quality  code.”   Higher  quality  code  than  what?   And  what  does  “quality”  mean?   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 47. Pro-­‐TDD  Studies   •  Comparisons  against  straw  men  common   •  Ohen  use  students   •  Some$mes  use  single  developers   •  Some$mes  aware  they’re  being  studied   •  Or  new  to  TDD  (may  be  more  mo$vated)   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 48. A  TDD  Enthusiast  Looks  At  Evidence   •  The  benefits  of  TDD  are  neither  clear  nor  are   they  apparent  —  Kane  Mar   – Part  1   – Part  2   – Part  3   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 49. Kane  Mar  –  Aher  Reading  Studies   “[Claims  about  TDD]  such  as  beHer  design,   beHer  APIs,  simpler  design,  lower  complexity,   increased  produc?vity,  more  maintainable  code   etc.,  are  simply  not  supported.”1   1.  A  cherry-­‐picked,  out-­‐of-­‐context  quote.   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 50. The  Reality  (vis-­‐à-­‐vis  other  tes$ng  strategies)   •  TDD  tends  to  generate  more  tests   •  Higher  quality  not  proven   •  Absence  of  evidence,  not  evidence  of  absence   •  Don’t  stress  about  it   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 51. Summary   •  Evidence  trumps  anecdotes   •  We  rely  on  anecdotes   •  That’s  sort  of  OK  if  we  don’t  have  evidence˜   •  Code  coverage  is  only  part  of  the  solu$on   •  We  need  to  escape  our  bubble   •  Be  skep$cal  of  hype   •  Be  skep$cal  of  me   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 52. Summary  for  TDD  Fana$cs   •  Don't  sweat  anecdotes  for  casual  ideas   •  Don’t  accept  anecdotes  from  dogma$sts   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 53. March  26,  2014   Copyright  2014,  All  Around  The  World   Image  by  Hay  Kranen  
  • 54. Bonus  Slides!   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 55. Basis  Path  Tes$ng   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 56. Cycloma$c  Complexity   March  26,  2014   Copyright  2014,  All  Around  The  World   Cycloma$c  Complexity  (McCabe)   =  (edges  –  nodes)  +  (  2  *  exit  nodes  )   =  (  19            –  14            )  +    (  2  *  1  )   =  7   =  minimum  number  tests  
  • 57. Calcula$ng  Cycloma$c  Complexity   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 58. Cycloma$c  Complexity  and  Bugs   •  Some  studies  show  a  posi$ve  correla$on   •  Other  studies  are  not  conclusive   •  May  make  bugs  harder  to  fix   •  May  lead  to  side  effects  when  making  changes   •  A  Complexity  Measure  —  McCabe   •  A  Cri$que  of  Sohware  Defect  Predic$on  Models  —  Fenton,  Neil     •  Cycloma$c  Complexity  Metrics  Revisited  —  Gill,  Kemerer   March  26,  2014   Copyright  2014,  All  Around  The  World  
  • 59. Perl   March  26,  2014   Copyright  2014,  All  Around  The  World  

Editor's Notes

  1. No evidence for or against the existence of a god or godsStrong evidence of absence of effect from astrology
  2. LorenzoTorrentino 1550 1st edition. Ptolemaic view of the solar system.http://en.wikipedia.org/wiki/File:1550_SACROBOSCO_Tractatus_de_Sphaera_-_(16)_Ex_Libris_rare_-_Mario_Taddei_.JPG
  3. Caveat: integration tests may have several pieces of dead code talking to each other!
  4. Simple faults can be corrected by a single change