Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

#LFMF: Tales of Test Automation Gone Wrong


Published on

This is a "Best Of" list of my (and others) failures over the years in attempting to adopt test automation. In it, you'll find 13 categories of What Not to Do. Presented at Turku Agile Day 2012 (#tad012).

Published in: Technology, Education

#LFMF: Tales of Test Automation Gone Wrong

  1. 1. # L F M F   (and others who shall r anonymous emain even if they deserve it)! don’t Learn  from  My  Fail:    Tales  of  Test  Automa6on  Gone  Wrong     (Plus  a  Li*le  Bit  about  How  to  Do  It  Right)   Elisabeth  Hendrickson   Quality  Tree  So7ware,  Inc.   @testobsessed  
  2. 2. Fail  Pa;ern  #0:    “No  6me  to  automate”   Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.   No  automaFon  means   rapid  growth  in  the  manual  tesFng  burden.   Or  untested  so7ware.  
  3. 3. Velocity!Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.  
  4. 4. Fail  Pa;ern  #1:    Automa6on  for  the  sake   Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.   of  automa6on   If  you  really  want  a  mess,   use  a  count  of  automated   tests  as  a  measure  of   progress.  
  5. 5. Image  of  prinFng  office  from  hJp://  Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.  
  6. 6. Fail  Pa;ern  #2:    All  automa6on  is  end-­‐to-­‐ Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.   end  through  the  GUI   End-­‐to-­‐end  tests  are  slow  and  fragile.  Some  are  good.   More  are  not  beJer.  
  7. 7. Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.  Flooded  basement  image  courtesy  of  carlpenergy  Showerhead  image  courtesy  of  deusx  Images  from  Flickr  licensed  through  CreaFve  Commons  AJribuFon  
  8. 8. Fail  Pa;ern  #3:     Dispropor6onate   Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.  Amount  of  Time  Spent   Fixing  Broken  Tests   Just  when  you  get  one   working  again,  another   breaks.  
  9. 9. Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.  Image  courtesy  of  cdrussorusso  From  hJp://    Licensed  Under  CreaFve  Commons  AJribuFon  
  10. 10. Fail  Pa;ern  #4:     Fragile  Tests   Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.   They  pass,  they  fail,  but   there  seems  to  be  no  correlaFon  between  the  test   results  and  the  actual   so7ware  behavior.  
  11. 11. Hey, the build is red.! year.! full moon this Tuesday with a Yeah, it’s the 2ndCopyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.  
  12. 12. Fail  Pa;ern  #5:    Inter-­‐dependent  Tests   Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.   When  one  fails,     they  all  fail  together.  
  13. 13. Run #1! Run #3! Run #2! Run #4!Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.  
  14. 14. Fail  Pa;ern  #6:     Useless  Tests   Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.  The  tests  can  never  fail,   so  they  yield  no   informaFon.  
  15. 15. Yes, I have actually… seen this in Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.   tests.!assert True But rare ly. More… common: tes rets that ify the s oftwa ver does w hat the softwar e does.!
  16. 16. Fail  Pa;ern  #7:     Bad  Tests   Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.   Obfuscated  intent,  massive  duplicaFon,  and  Fght  coupling  between  the  interface  and  the   tests  makes  for  an   unmaintainable  mess.  
  17. 17. Test Login Input Text first_name fred Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.   Input Text last_name flintstone Input Text user_name fflint Input Text password p@ssw0rd Click Submit Click Logout Click Login Input Text username fflint WAT?!?! Input Text password badpassword Click Submit Input Text ...
  18. 18. Fail  Pa;ern  #8:     Bad  Code   Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.  Smelly  test  automaFon  code   that  violates  SOLID   principles  is  as  much  of  a   problem  as  smelly   producFon  code.  
  19. 19. Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.  
  20. 20. Fail  Pa;ern  #9:     Only  specialists   Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.   automate   They  might  be  good  at  automaFon,  but  they’re  fighFng  an  uphill  baJle.  
  21. 21. Fail  Pa;ern  #10:     Forcing  devout  non-­‐ Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.   coders  to  write   automa6on  If  they  lack  either  interest  or   apFtude  to  automate,  the   results  will  be  bad.  
  22. 22. Fail  Pa;ern  #11:    Tests  That  Don’t  Get  Run   Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.   Someone  writes  them,  no   one  runs  them.  So  when   they  suffer  bitrot,  no  one   can  hear  them  scream.  
  23. 23. Unrun ! tests! Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.   become ! creaky,! dusty,! sad.!Photo  by  crabchick  hJp://  Licensed  under  CreaFve  Commons  AJribuFon  
  24. 24. Fail  Pa;ern  #12:    Failing  Tests  Ignored   Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.   They  run.     They  fail.     No  one  cares.  
  25. 25. Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.  Image  of  the  1906  San  Francisco  Earthquake  a7ermath  from  hJp://  
  26. 26. #LFMF   Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.  Automate important expectations!Treat test code with the same level ofcare as production code!Run all the automated tests in the CIbuild (no manual kickoff)!Fix failing tests immediately!Foster a “whole team tests” culture!
  27. 27. Teema Frisk! @hackyourtv! Juha Rantanen! Jyrki Pulliainen! Jarkko Kailanto! Many thanks to …!Copyright  ©  2012  Elisabeth  Hendrickson,  Quality  Tree  So7ware,  Inc.