© Akshay Mathur Page 1 	
  
Releasing	
  Software	
  without	
  Testing	
  Team	
  
Akshay	
  Mathur	
  
	
  
Introducti...
 
© Akshay Mathur Page 2 	
  
must	
  follow.	
  In	
  many	
  cases,	
  customers	
  are	
  themselves	
  not	
  aware	
 ...
 
© Akshay Mathur Page 3 	
  
As	
  customer-­‐facing	
  teams	
  learn	
  more	
  about	
  the	
  market	
  and	
  make	
...
 
© Akshay Mathur Page 4 	
  
to	
  simply	
  get	
  past	
  the	
  test	
  team.	
  As	
  someone	
  else	
  is	
  respon...
 
© Akshay Mathur Page 5 	
  
to	
  the	
  delivery	
  needs	
  to	
  be	
  aligned	
  with	
  this	
  idea	
  and	
  ever...
 
© Akshay Mathur Page 6 	
  
Limitations	
  
While	
  the	
  approach	
  described	
  here	
  produces	
  good	
  quality...
 
© Akshay Mathur Page 7 	
  
ShopSocially	
  implemented	
  this	
  approach	
  right	
  from	
  its	
  incorporation.	
 ...
 
© Akshay Mathur Page 8 	
  
o Having	
  the	
  testers	
  always	
  present	
  with	
  them,	
  developers	
  never	
  
...
Upcoming SlideShare
Loading in …5
×

Releasing Software Without Testing Team

319 views

Published on

This article is prepared for presenting in Software Systems Best Practice Conference, Anahiem, California

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
319
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Releasing Software Without Testing Team

  1. 1.   © Akshay Mathur Page 1   Releasing  Software  without  Testing  Team   Akshay  Mathur     Introduction   We  can’t  imagine  software  development  without  testing.  Programmers  are   human  and  they’re  bound  to  make  mistakes.  We  use  testing  as  a  tool  to  ensure   that  defects  are  caught  before  we  ship.     Good  software  engineering  practice  requires  us  to  isolate  development  and   testing  efforts.  This  is  commonly  implemented  by  assigning  each  of  these   responsibilities  to  different  teams.  Along  these  lines,  the  idea  of  the  magical  ratio   of  testers  to  developers  was  conceived.       In  order  to  improve  quality,  we  intuitively  increase  the  ratio  of  testers  to   developers.  As  a  direct  consequence  of  increasing  the  testing  effort  in  this  way,   budgets  and  time  to  market  are  directly  affected.  There  is  also  a  hidden  cost  at   work  that  results  from  the  interpersonal  issues  when  two  teams  with  different   objectives  try  to  work  together.     Development  teams  that  are  directly  responsible  for  the  quality  of  the  code  they   ship  reveal  several  desirable  advantages.  By  internalizing  quality  goals,   developers  are  forced  to  adopt  a  ‘test  first’  attitude.  This  approach  allows  them   to  get  better  over  time  at  managing  the  expected  standards  required  to  ship   software  to  customers.  By  eliminating  the  overhead  of  inter-­‐team  co-­‐ordination,   single  teams  find  agility  and  are  more  responsive.  Turnaround  time  for  bug   resolution  is  also  reduced  to  it’s  minimum.  These  are  great  qualities  to  achieve.   Software  Development  Life  Cycle   Before  we  can  review  the  role  of  the  software  development  team,  we  will  need  to   revisit  the  traditional  software  development  waterfall  model  and  the  role  the   team  plays  in  this  model.  In  this  model  requirements  analysis,  system  and   technical  design,  development,  testing  and  release  are  each  discrete  steps.  In   practice,  management  and  development  teams  rarely  have  the  freedom  to   implement  this  model  in  this  way.  Let’s  take  a  closer  look  at  what  the  underlying   forces  are  that  disrupt  the  simple  flow  of  this  model.     Requirements   If  we  think  of  any  software  project  to  be  governed  by  technical  and  non-­‐technical   requirements,  then  the  key  underlying  constraints  are  time  and  money.       At  the  start  of  the  cycle,  we  describe  the  functional  requirements  or  what  the   software  must  deliver.  These  are  created  on  basis  of  our  current  knowledge  of   what  our  customers  want.  As  market  demands  change,  functional  requirements  
  2. 2.   © Akshay Mathur Page 2   must  follow.  In  many  cases,  customers  are  themselves  not  aware  of  what  it  is   they  want.     Further,  written  functional  requirements  are  at  best  an  abstraction  of  what  is   really  desired.  It  is  left  to  development  team  to  fill  in  the  details  and  plug  the   gaps.  Thus,  after  having  created  the  functional  requirements,  we  are  still  left  with   lingering  questions.  How  do  we  know  for  sure  that  what  the  development  team   delivers  will  meet  the  requirements?  Can  all  the  requirements  be  met  within  the   budget  defined?       Considering  these  factors,  we  allow  ourselves  to  revisit  and  change  the   requirements.   System  Design   Since  requirements  are  a  prerequisite  to  the  design  stage,  many  of  the  challenges   we  found  in  the  first  stage  also  affect  this  stage.  You  could  require  that  designs   eliminate  the  need  to  revisit  development  even  if  the  requirements  were  to   change.  However,  this  approach  risks  over-­‐engineering  the  software  and   increases  the  development  complexity.  It  will  result  in  significant  waste  when   major  changes  are  made  to  the  original  requirements  in  order  to  keep  pace  with   the  market.     Development  and  Testing   Even  with  the  backdrop  of  the  flawed  waterfall  model,  the  implicit  expectation  is   that  the  software  delivered  has  to  be  of  excellent  quality.  In  a  nutshell,  it  must   meet  the  original  ever-­‐changing  requirements,  work  in  common  and  extreme   scenarios  that  can  be  imagined,  or  at  least  automatically  degrade  gracefully  in   the  remaining  scenarios  that  could  not  be  feasibly  accounted  for  or  imagined  in   the  first  place.     Delivering  this  magical  software  is  the  joint  responsibility  of  development  and   testing  team.  As  we’re  already  aware,  joint  responsibilities  lead  to  imperfect   outcomes.  Thus  optimizing  the  tradeoff  between  our  expectations  and  the  ability   for  this  joint  team  to  deliver  is  of  great  importance  to  delivering  great  software.     Let's  dig  a  little  deeper  and  see  how  development  and  testing  teams  work.  For   the  sake  of  simplicity,  we  will  assume  that  testing  is  given  the  same  importance   as  development.  For  instance,  requirements  are  discussed  with  both   development  and  testing  teams,  and  both  the  teams  are  able  to  begin  their  work   together.     The  problem   Communication  Issues   While  development  team  does  low  level  technical  design  and  starts  coding,   testing  team  starts  with  writing  test  plan  and  test  cases.  At  this  point,  only  black-­‐ box  testing  stuff  can  be  planned  because  actual  code  is  not  present  for  any  white-­‐ box  testing  approaches.    
  3. 3.   © Akshay Mathur Page 3   As  customer-­‐facing  teams  learn  more  about  the  market  and  make  changes  in  the   original  requirements,  these  changes  need  to  be  communicated  to  both  teams,  a   costly  and  imperfect  exercise.  Further,  as  the  development  team  makes  progress   and  learns  more  about  implementation  limitations,  requirements  and  designs   will  change  to  keep  to  delivery  schedules  and  budget  limits.  In  many  cases,  these   changes  remain  implicit  within  the  development  team  and  their  importance  is   realized  only  later.     Thus,  the  development  and  testing  teams  understanding  of  the  software  in   development  begins  to  diverge.  As  anyone  with  experience  in  testing  will  tell   you,  delays  and  imperfect  communication  results  in  unexpected  pauses,  rework   and  a  great  deal  of  frustration.  Features  that  commonly  go  ‘missing’  or  are  found   to  be  unexplainably  ‘broken’  in  the  build  routinely  halt  testing  efforts.   Coordination  Issues   The  perspectives  that  the  development  and  testing  teams  adopt  are  also  at  odds.   Testers  need  to  break  the  software  whereas  developers  adopt  an   implementation  perspective.  As  the  perspectives  are  different,  the  order  in   which  they  want  to  approach  the  software  and  it’s  sub-­‐components  will  be   different.  However,  in  order  to  write  comprehensive  test  cases,  testing  team   requires  implementation  details  of  features  that  the  development  team  will   simply  not  be  prepared  for.       The  testing  teams’  workload  depends  on  what  the  development  team  can  deliver.   Development  milestones  must  keep  an  optimal  workload  for  both  teams  in  mind.   As  a  result,  both  developers  and  testers  have  to  compromise  on  their  natural   order  of  work  and  come  up  with  a  mid-­‐way  build  plan.  Both  teams  are  now   constrained  by  this  plan  and  must  ignore  any  creative  optimization  of  the  plan  to   suit  their  individual  objectives.  Failing  to  meet  the  build  plan  affects  both  teams   and  raises  more  coordination  issues.     Mindset  Issues   Duality  in  rewards  and  holding  the  two  teams  accountable  also  plays  a   significant  role  in  further  dividing  the  two  teams.  For  instance,  on  uncovering   embarrassing  or  severe  defects  –  the  testing  team  is  pushed  to  tighten  their   processes.  On  the  other  hand,  a  successful  release  invites  praise  for  the   developers.         The  testing  team  essentially  isolates  developers  from  the  how  the  software   performs  with  end-­‐users.  If  the  two  teams  work  in  a  staggered  mode,  developers   simply  throw  code  over  the  wall  to  the  testers  to  test  and  move  on  to  their  next   responsibility.  As  a  result,  developers  miss  out  on  the  opportunity  of  processing   valuable  feedback  from  the  field  in  context  of  the  current  release  effort.  Now  bug   resolution  cycles  are  also  longer  as  developers  have  to  pre-­‐empt  their  current   responsibilities,  switch  context  to  fix  bugs.     Isolation  from  the  end-­‐user  also  has  a  subtle  effect  on  individuals  of  allowing   overall  quality  to  slip.  Developers  begin  to  believe  that  they  must  now  write  code  
  4. 4.   © Akshay Mathur Page 4   to  simply  get  past  the  test  team.  As  someone  else  is  responsible  for  delivering   quality,  now  in  the  back  of  developer's  mind,  it  is  okay  to  write  code  that  does   not  take  care  of  all  cases  or  does  not  deliver  the  complete  functionality.  Also,  it  is   fine  not  to  do  the  impact  analysis  upfront  because  if  the  new  code  breaks  some   other  functionality,  testing  team  will  report  that  as  well  and  that  will  go  through   the  bug  fix  cycle.     As  testers  are  rewarded  for  the  non-­‐obvious  defects  find  before  release.  This   encourages  attachment  with  bug  reports.  If  for  the  testing  team,  the  gross   number  of  bug  reports  accepted  for  fixing  is  the  only  measuring  benchmark  it   can  be  disappointing  for  testers  and  the  team  at  large  to  have  bugs  rejected  by   developers.       The  Problem  in  a  nutshell   In  a  practical  software  development  project,  the  company  needs  to  facilitate  a   great  deal  of  coordination  and  communication  infrastructure.  In  spite  of  that,  a   divide  gets  created  between  development  and  testing  team  members.  Overall,   these  things  not  only  add  to  the  cost  but  also  ruin  environment  of  the  workplace.   The  Solution   Simplest  solution  to  eliminate  the  divide  is  to  merge  the  teams.  However,  it  is   observed  that  superficial  merging  does  not  work.     Superficial  Merging   In  an  experiment  to  reduce  this  divide  the  two  teams  were  merged  into  one   'Engineering  Team'  with  single  reporting  lead.       This  is  a  cosmetic  change  and  resulted  into  negative  impact.  Processes  stayed  the   same  and  since  the  engineering  manager,  being  from  development  stream,  failed   to  understand  the  issues  of  testers.     In  a  different  experiment,  developers  were  asked  to  alternate  their  primary   development  roles  with  testing  responsibilities.  This  also  failed  for  following   reasons:   • Developers  were  pre-­‐empted  in  the  midst  of  their  testing  assignments  as   and  when  they  need  to  address  urgent  issues  from  previous  releases.   • Some  developers  refused  to  be  rotated  to  testing  team.   • Testing  team  felt  that  developers  coming  on  rotation  need  a  lot  of  training   for  adopting  formal  testing  and  were  not  being  helpful.     True  Merging   True  merging  requires  merging  the  objectives  and  the  functions  of  both  teams.   This  implies  that  there  are  no  specialist  positions.  Each  individual  must  do  both   development  and  testing.  Thus,  merging  teams  is  difficult  to  implement  as  an   afterthought.  The  complete  process  right  from  planning,  recruitment,  execution  
  5. 5.   © Akshay Mathur Page 5   to  the  delivery  needs  to  be  aligned  with  this  idea  and  everybody  has  to  work   with  this  approach.     The  company  management  needs  to  make  way  for  new  processes.  For  example,   when  recruiting,  the  job  profile  needs  to  communicate  to  candidates  that  they   are  expected  to  wear  both  testing  and  development  hats.  Context  switches   between  development  and  testing  must  be  instantaneous.  Interestingly,  no   formal  testing  documentation  is  needed  –  eliminating  written  test  cases,  test   plans  and  test  reports.  The  number  and  quality  of  bug  reports  also  goes  down.       The  above  appears  difficult  to  implement  as  it  requires  a  change  in  culture  but  it   works  in  favor  of  quality.  In  the  remaining  document,  we  shall  discuss  that  in   length.     How  it  works   As  said  earlier,  there  is  no  dedicated  testing  team.  Everybody  involved  in  the   project  knows  how  the  software  works  or  what  is  expected  from  it.  The  waterfall   model  implementation  remains  the  same.  Just  the  development  and  testing  is   carried  out  differently  from  how  two  teams  would  approach  it.  Individuals  are   better  aware  of  changing  requirements  and  are  team  is  co-­‐ordination  is  better   synchronized.       When  work  begins  they  are  free  to  work  in  the  sequence  they  prefer  and  make   optimizations  and  adjustments  in  the  schedule  as  needed.  While  estimating  time   for  a  feature,  members  add  additional  time  necessary  to  smoke-­‐test  the  overall   build  after  integration.       Developers  are  skilled  in  basic  testing  techniques.  For  doing  a  better  job  in   testing,  the  developer  does  a  thorough  impact  analysis  of  any  change  he  makes.   Access  and  understanding  of  the  details  in  code  can  increase  the  accuracy  of   impact  analysis  and  including  testability  hooks.  Knowledge  of  the  code  also  helps   in  running  better  test  cases  and  choosing  better  test  data.  In  this  way  trivial  bugs   are  resolved  in  the  development  environment  itself.       When  a  developer  encounters  a  bug  in  the  code  written  by  someone  else,  he   discusses  the  bug  with  original  writer  and  together  they  come  up  with  a  plan  for   resolution.  Bug  tracking  system  is  mostly  used  to  serve  reminders  and  that  is   why  the  quality  of  bug  reports  is  poor  as  per  standards.  Now  bug  reports  contain   the  investigation  and  resolution  notes  rather  than  steps  to  reproduce  test  data   etc.     In  this  system,  both  praise  and  fire  come  to  the  same  person.  This  makes  the   developer  completely  responsible  for  the  product  quality  and  encourages  better   quality.    
  6. 6.   © Akshay Mathur Page 6   Limitations   While  the  approach  described  here  produces  good  quality  software,  the  formal   documentation  of  test  cases,  test  data  and  test  reports  etc.  becomes  unavailable.   Also  there  are  cases  where  this  methodology  cannot  directly  apply  or  will  need   to  be  adapted.       For  instance,  if  a  large  software  product  is  now  simply  being  maintained,  the   development  and  testing  workloads  are  no  longer  evenly  matched.  Thus,  the   testing  workload  is  significantly  higher  and  calls  for  specialization.       In  other  cases  such  as  mission-­‐critical  software,  the  cost  of  managing  multiple   teams  plus  all  the  overheads  may  still  be  significantly  less  than  the  cost  of  a   defect  going  out  in  the  field.     Another  case  is  of  software  being  shipped  embedded  on  a  device  where  no   matter  how  you  test  otherwise;  test  cycle  on  actual  devices  is  mandatory.     One  more  case  is  of  WIPS  (Wireless  Intrusion  Prevention  Systems)  where  so   many  environmental  factors  are  at  play  that  a  radio  frequency  isolated  place  is   needed  for  testing.   Scaling   The  approach  described  in  this  paper  was  applied  successfully  with  small  team   sizes,  or  in  cases  where  the  batch  size  of  features  released  is  small.  However,  the   idea  behind  the  proposal  is  to  make  the  developers  write  good  code  at  the  first   place  by  holding  them  responsible  for  the  quality  and  making  them  think  about   the  potential  failure  points  ahead  of  time.     At  large  scale,  if  the  testing  team  can’t  be  completely  eliminated,  it  should  be   introduced  very  late  in  the  process.  For  instance,  if  the  testing  team  comes  in  the   picture  at  customer  acceptance  test  level  when  for  the  purpose  of  development   team,  the  software  is  considered  ‘shipped’;  the  development  team  members  have   to  upgrade  themselves  by  all  means  for  delivering  high  quality  software.     Case  Studies   ShopSocially   ShopSocially  provides  embed-­‐able  apps  to  online  retailers.  The  retailers  add  a   code  snippet  on  the  pages  of  the  online  store  and  based  on  the  rules  configured   in  the  console,  different  apps  trigger  on  different  pages.  Some  apps  are  run  off   retailer’s  website  as  well.     In  a  nutshell,  ShopSocially  has  a  web  application  that  is  distributed  on  the   websites  of  500+  retailers.  Hence,  ShopSocially  serves  multi  million  requests  per   day.    
  7. 7.   © Akshay Mathur Page 7   ShopSocially  implemented  this  approach  right  from  its  incorporation.  The   processes  work  exactly  in  the  way  described  in  the  ‘How  it  works’  section  above.     We  at  ShopSocially  get  many  requirements  and  enhancement  requests  as  bugs.   Let’s  leave  them  apart  and  talk  about  the  software  bugs  testing  team  could  have   caught.  In  last  3  years,  only  17  software  bugs  were  reported  from  the  field,  while   in  last  6  months  the  number  is  only  2.  None  of  these  bugs  was  critical.       Another  category  of  bugs  got  reported  is  browser  compatibility  issues  where   some  portion  of  the  enterprise  console  does  not  render  properly  in  Internet   Explorer  7.0.     In  the  beginning,  a  lot  of  cosmetic  issues  also  got  reported.  As  a  result,  on  one   hand,  developers  upgraded  themselves  for  having  a  better  eye  for  cosmetic   details,  on  other  hand,  they  adapted  better  user  interface  development  tools  and   frameworks  and  the  problem  is  now  reduced  to  its  minimum.     GS  Lab   GS  Lab  provides  software  development  and  testing  services.  Many  of  its  clients   are  early  stage  start-­‐ups.  By  adopting  these  techniques,  GS  lab  delivers  very  good   quality  code.  This  saves  a  lot  of  money  for  client  as  the  cost  of  testing  team  gets   eliminated.     AirTight  Networks   AirTight  Networks  is  the  leader  in  Wireless  Intrusion  Prevention  System  (WIPS).   The  product  is  for  everyone  that  is  from  SOHO  to  large  enterprise.  However,   most  of  the  customers  are  large  enterprises  having  deployed  the  system  across   the  globe.  The  product  includes  server  appliances  and  a  number  of  sensor  boxes   running  proprietary  code.     AirTight  Networks  can’t  completely  eliminate  the  testing  team  because  of   following  reasons:   • The  testing  requires  specialized  skills  and  environment.   • The  software  needs  to  run  on  different  devices  and  testing  is  required  on   all  the  devices  to  eliminate  any  device  specific  issues.   • Many  releases  of  the  product  are  deployed  in  the  field  and  customers   upgrade  at  their  convenience.  The  company  has  to  support  all  of  them.   • It  is  a  security  product  and  a  security  breach  may  cost  a  lot  to  the   customer.   However,  after  doing  many  experiments,  AirTight  Networks  has  employed  many   processes  for  bridging  the  gap  between  development  and  testing  teams.  Here  are   a  few  of  them:   • Developers  and  testers  sit  next  to  each  other.     o This  eliminates  the  requirement  of  formal  communication  in  some   cases.  Testers  get  automatically  involved  in  the  discussions   happening  on  the  developers’  desk  during  the  course  of   development.  
  8. 8.   © Akshay Mathur Page 8   o Having  the  testers  always  present  with  them,  developers  never   forget  to  include  them  in  formal  communication.   • Complete  access  of  source  code  is  made  available  to  testing  team.  Testers   are  also  trained  for  navigating  the  code  and  doing  better  impact  analysis.   • Cross-­‐referencing  is  made  between  source  control  system  and  bug   tracking  system.   • Commit  log  of  source  control  system  is  made  available  via  a  web  interface   for  eliminating  push-­‐based  formal  communicating  from  development   team.   • While  committing  code,  the  developers  write  detailed  commit  log  having   bug  report  or  enhancement  request  number  and  details  of  the  change   being  committed.   • Developers  also  need  to  provide  details  of  tests  they  ran  so  that  testing   team  rests  assured  that  trivial  stuff  is  working  fine.   • Scope  of  testing  by  testing  team  for  intermediate  builds  is  made  limited   only  to  new  and  impacted  functionality  and  developers  need  to   communicate  the  scope  after  doing  their  impact  analysis.   • As  far  as  development  team  is  concerned,  the  software  is  considered   shipped  after  integration  test  cycle.  Any  bug  in  previously  released   features  discovered  during  regression  test  cycle,  is  considered  as  the  bug   from  the  field  and  developers  are  held  responsible  for  it.     Summary   Academically,  the  software  development  life  cycle  looks  very  simple  but  it   carries  a  lot  of  challenges  for  practical  scenarios.  Making  developers  responsible   for  quality  is  the  key  of  producing  good  quality  software.     In  many  cases,  this  may  be  achieved  by  eliminating  testing  team  completely.  In   some  other  cases,  late  introduction  of  testing  team  in  the  process  may  help.  

×