The Pathologies of Failed Test Automation Projects

449 views

Published on

Most test automation projects never die—they just become a mess and are redone. Initial solutions that start well and are full of promise often end up as brittle and unmaintainable monsters consuming more effort than they save. Political feuds can flourish as different automation solutions compete for attention and dominance. Tests become inefficient in both execution time and resource usage. Disillusionment ensues, projects are redefined, and the cycle begins again. Surely we can learn how to avoid such trouble on the next project. Michael Stahl has analyzed automation projects and identified recognizable failure patterns—mushrooming, duplication, going for the numbers, and others. Michael describes these patterns, suggests how to detect them early, and shares ways to avoid or mitigate them. Whether your team is just starting on test automation—or is already in full flight—you’ll take back ideas to improve the chances of achieving success in your test automation efforts.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
449
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

The Pathologies of Failed Test Automation Projects

  1. 1. W3 Test Automation 5/1/2013 11:30:00 AM The Pathologies of Failed Test Automation Projects Presented by: Michael Stahl Intel Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
  2. 2. Michael Stahl Michael Stahl is a software validation architect at Intel, working with a team that validates Intel's graphics hardware drivers. In this role, Michael defines testing strategies and work methodologies for test teams, and tests part of the product himself - the work he enjoys most. As a twenty-two year veteran and senior engineer, he has been witness to what works - and what doesn’t - in test strategies, communications, and automation. An avid teacher, Michael enjoys sharing his observations with others. Contact Michael at michael.stahl@intel.com and review previous conference presentations on his website testprincipia.com.
  3. 3. The Pathologies of failed Test Automation Projects Michael Stahl, Intel Apr 2013 Based, in part, on work done with Alon Linetzki, Best Testing (www.best-testing.com)
  4. 4. 2 On the Menu… • • • Automation failure patterns What can we do? Summary Based, in part, on work done with Alon Linetzki, Best Testing (www.best-testing.com) © Michael Stahl, 2013, all rights reserved
  5. 5. Disclaimers    Names and brands referenced herein may be claimed as the property of third parties The views expressed in this presentation are solely my own, and do not in any manner represent the views of my employer Information in this presentation is provided “AS IS” without any warranties or representations of any kind © Michael Stahl, 2013, all rights reserved
  6. 6. A common automation story… 4  Once upon a time… I can automate my work!... Mr. Otto Mate Cool! Can you make it… Otto’s Team © Michael Stahl, 2013, all rights reserved
  7. 7. A common automation story… 5 Piece of cake! Perfect! Can you add… Mr. Otto Mate What a wonderful world!... Otto’s Team Mr. Mann a. Ger © Michael Stahl, 2013, all rights reserved
  8. 8. A common automation story… 6 Arggghhh! Fix Fix Fix Otto! The last release fails!... Otto’s Team Mr. Otto Mate … and mates Sir! I need time! I need help! Makes sense… Mr. Mann a. Ger © Michael Stahl, 2013, all rights reserved
  9. 9. A common automation story… 7 Mr. Otto Mate … and mates … and mates … and mates © Michael Stahl, 2013, all rights reserved
  10. 10. A common automation story… 8 Let’s REDESIGN!!! Mr. Oto Mate … and mates #$&@***!!! … and mates Mr. Mann a. Ger © Michael Stahl, 2013, all rights reserved … and mates
  11. 11. 9 A pattern emerges… © Michael Stahl, 2013, all rights reserved
  12. 12. Pattern #1: Mushrooming © Michael Stahl, 2013, all rights reserved
  13. 13. A pattern… 11 Single User / Developer Simple Tool Stage 1 – Small & Local © Michael Stahl, 2013, all rights reserved
  14. 14. A pattern… 12 Multiple Users / Single Developer Enhanced Tool Stage 2 – Generalization © Michael Stahl, 2013, all rights reserved
  15. 15. A pattern… 13 Multiple Users & Developers Complicated Tool Stage 3 – Staffing © Michael Stahl, 2013, all rights reserved
  16. 16. A pattern… 14 Multiple Users & Developers Test Case Management Stage 4 – Non-core features © Michael Stahl, 2013, all rights reserved
  17. 17. A pattern… 15 Stage 5 – Overload © Michael Stahl, 2013, all rights reserved Arghhh!
  18. 18. Pattern #2: The Competition 16 © Michael Stahl, 2013, all rights reserved
  19. 19. Pattern #2 – The Competition (ver. 1) 17 Team A Team B © Michael Stahl, 2013, all rights reserved
  20. 20. Pattern #2 – The Competition (ver. 2) 18 Team A Team B Team C !!! Team D © Michael Stahl, 2013, all rights reserved
  21. 21. Pattern #2 – The Competition (ver. 3) 19 Team A Team B !!! Team C !!! … … !!! … Team D © Michael Stahl, 2013, all rights reserved
  22. 22. Pattern #3: The Night Run Fallacy © Michael Stahl, 2013, all rights reserved
  23. 23. Pattern #3 – The Night Run Fallacy 21 © Michael Stahl, 2013, all rights reserved
  24. 24. Pattern #3 – The Night Run Fallacy 22 Night Time Test Time © Michael Stahl, 2013, all rights reserved
  25. 25. Pattern #3 – The Night Run Fallacy 23 Add a snapshot – or a movie snippet – of PAVE Corporate Truism: It’s easier to get budget for machines than for more testers © Michael Stahl, 2013, all rights reserved
  26. 26. Pattern #3 – The Night Run Fallacy 24 Add a snapshot – or a movie snippet – of PAVE © Michael Stahl, 2013, all rights reserved
  27. 27. Pattern #3 – The Night Run Fallacy 25 Test Automation Truism: Machines create work for more testers © Michael Stahl, 2013, all rights reserved
  28. 28. The Tragedy of the Commons multiple individuals… will ultimately deplete a shared limited resource… even when it is not in their long-term interest Garrett Hardin, Science, 1968 © Michael Stahl, 2013, all rights reserved
  29. 29. Pattern #4: Going for the Numbers © Michael Stahl, 2013, all rights reserved
  30. 30. 28 © Michael Stahl, 2013, all rights reserved
  31. 31. Pattern #5 – Going for the Numbers 29 © Michael Stahl, 2013, all rights reserved
  32. 32. Pattern #4 – Going for the Numbers 30 Robustness is Invisible © Michael Stahl, 2013, all rights reserved
  33. 33. Pattern #5: The Magician Stahl, 2013, all rights reserved Apprentice Syndrome © Michael
  34. 34. Pattern# 5 – The Magician Apprentice 32 © Michael Stahl, 2013, all rights reserved
  35. 35. Recap: The Patterns Mushrooming 3 3 The Competition The Night time Fallacy Going for the Numbers The Magician Apprentice © Michael Stahl, 2013, all rights reserved
  36. 36. 34 So... What can we do? © Michael Stahl, 2013, all rights reserved
  37. 37. 35 Are the patterns… Unavoidable ? © Michael Stahl, 2013, all rights reserved ?
  38. 38. 36 Counter measures © Michael Stahl, 2013, all rights reserved
  39. 39. 37 Mushrooming © Michael Stahl, 2013, all rights reserved
  40. 40. Alert Signals & Counter Measures 38 How to use this information?  GPS  Locate the stage you are at  Get directions for the way out  Map  Start right and avoid the wrong turns © Michael Stahl, 2013, all rights reserved
  41. 41. Alert Signals 39 © Michael Stahl, 2013, all rights reserved
  42. 42. Alert Signals 40 © Michael Stahl, 2013, all rights reserved
  43. 43. Alert Signals 41 Failing the project Failing the project Failing the project Failing the project Failing the project © Michael Stahl, 2013, all rights reserved
  44. 44. Alert Signals: Stage 1 Small, local, feature-centered 42     Single feature test tool? The creator is the user? “Skunk works”? Key words:   “Tool” “Utility” © Michael Stahl, 2013, all rights reserved
  45. 45. Counter Measures: Stage 1 Small, local, feature-centered 43 Ensure the following… and relax:  Code control  Documentation    User Manual (Usage line…) “Green” in the code High level design © Michael Stahl, 2013, all rights reserved
  46. 46. Alert Signals: Stage 2 Generalization 44        Additional features? Multiple users? Automation web site? >25% of the tester’s time? Key words: “Use by other testers” “Common Libraries” © Michael Stahl, 2013, all rights reserved
  47. 47. Counter Measures: Stage 2 Generalization 45 “The hardest part of building a software system is deciding precisely what to build... No other part of the work so cripples the resulting system if done wrong. No other part is more difficult to rectify later” - Fred P. Brooks (author of “The Mythical Man-Month”) © Michael Stahl, 2013, all rights reserved
  48. 48. Counter Measures: Stage 2 Generalization 46     Stage 1 measures Strategy Architecture Lightweight PM    Version control Scope control Bugs & Requests database © Michael Stahl, 2013, all rights reserved
  49. 49. Alert Signals: Stage 3 Institutionalization and staffing 47         Requests for additional heads? Automation F2F? Tool-related delays in execution? Key words: “Tool Owner”; “Automation team” “Framework”; “Infrastructure” “Roll back” “Bug fix release” © Michael Stahl, 2013, all rights reserved
  50. 50. 48 © Michael Stahl, 2013, all rights reserved
  51. 51. Counter Measures: Stage 3 Institutionalization and staffing 49 Stage 1, 2 counter measures Management level decision time         Programming language Framework focus Code and release management Acceptance tests Skillset development Metrics © Michael Stahl, 2013, all rights reserved
  52. 52. Counter Measures: Stage 3 Institutionalization and staffing 50 Metrics suggestions  Automation framework quality       Number of false fails Framework’s test results; Bug trends ROI Number of runs Invested effort by type (new, maintenance, rewrite) Number of bugs found by Automation (?) © Michael Stahl, 2013, all rights reserved
  53. 53. Alert Signals: Stage 4 Change of focus: Technology  Management 51        Generic features? Test log wading? False fails? Key words: “test suite / cycle generation” “robustness enhancement” “setup issues” © Michael Stahl, 2013, all rights reserved
  54. 54. Counter Measures: Stage 4 Change of focus: Technology  Management 52 Stage 1, 2, 3 counter measures Build VS Buy Re-architect      Core / Non-Core Solid infrastructure Influence upstream    Testability hooks Design for Test (automation) © Michael Stahl, 2013, all rights reserved
  55. 55. Counter Measures: Stage 4  Build VS Buy? (hint: Buy) See: http://www.stickyminds.com/s.asp?F=S17601_COL_2 Stage 4 – Non-core features © Michael Stahl, 2013, all rights reserved
  56. 56. Counter Measures: Stage 4 Change of focus: Technology  Management 54 Build (VS Buy) if…        Competitive edge Existing expertize Core competency Cheaper; Faster Good use of resources Acceptable risk Long term support Main source: Allen Eskelin http://www.informit.com/articles/article.aspx?p=21775 © Michael Stahl, 2013, all rights reserved
  57. 57. Alert Signals: Stage 5 Maintenance overload; Re-design 55 Maintenance & logistics overload? Limitations overplay? Loss of credibility? Stage 1 initiatives? Key words          “Did it fail in manual test?” “Architecture limitation” “refactoring”; “redesign” “…I can write a small program…” © Michael Stahl, 2013, all rights reserved
  58. 58. Alert Signals: Stage 5 Maintenance overload; Re-design 56 Loss of credibility © Michael Stahl, 2013, all rights reserved
  59. 59. Counter Measures: Stage 5 Maintenance overload; Re-design 57 Options:      Continue…  Give up problematic areas Partial return to Stage 1 “We value Robustness over New Features” Prepare for re-design – with a new map... © Michael Stahl, 2013, all rights reserved
  60. 60. How to Use this information? 58  GPS   Locate the stage you are at Get directions for the way out Be alert for Alerts Implement Counter Measures Identify your stage Analyze your situation © Michael Stahl, 2013, all rights reserved
  61. 61. How to Use this information? 59  Map  Start right and avoid the wrong turns Plan your trip Implement Counter Measures Be alert for Alerts Analyze your situation © Michael Stahl, 2013, all rights reserved
  62. 62. 60 The Competition © Michael Stahl, 2013, all rights reserved
  63. 63. Pattern #2 – The Competition (ver. 1) 61   Salvageable up to stage 2 Stage 3 and up:    Merge the teams EOL both tools Start a 3rd © Michael Stahl, 2013, all rights reserved
  64. 64. Pattern #2 – The Competition (ver. 2) 62  Plugin Architecture © Michael Stahl, 2013, all rights reserved
  65. 65. Pattern #2 – The Competition (ver. 3) 63  Fix the main problem Accept, encourage Stage 1 stuff…  Maintaining vigil:   Avoid ver.1 pattern © Michael Stahl, 2013, all rights reserved
  66. 66. 64 The Night Run Fallacy © Michael Stahl, 2013, all rights reserved
  67. 67. Pattern #3 – The Night Run Fallacy 65 6 5    6 5 Never forget test time, test efficiency Balance test skills VS automation skills Allocate machines or machine time © Michael Stahl, 2013, all rights reserved
  68. 68. 66 Going for the Numbers © Michael Stahl, 2013, all rights reserved
  69. 69. Pattern #4 – Going for the Numbers 67  Change how you count “Done” © Michael Stahl, 2013, all rights reserved
  70. 70. 68 The Magician Apprentice Syndrome © Michael Stahl, 2013, all rights reserved
  71. 71. Pattern# 6 – The Magician Apprentice 69 http://www.youtube.com/watch?v=zlnz1rSJj7Y © Michael Stahl, 2013, all rights reserved
  72. 72. Pattern# 6 – The Magician Apprentice 70 Automation should not necessarily mimic humans… … it should get the job done © Michael Stahl, 2013, all rights reserved
  73. 73. Pattern# 6 – The Magician Apprentice 71 http://www.youtube.com/watch?v=qDVYIT85ntg © Michael Stahl, 2013, all rights reserved
  74. 74. Pattern# 5 – The Magician Apprentice 72    Holistic approach VS “test after test” Re-think, re-strategize, re-evaluate before automating Identify Actions, Keywords (KDT) © Michael Stahl, 2013, all rights reserved
  75. 75. 73 Summary © Michael Stahl, 2013, all rights reserved
  76. 76. The Patterns Mushrooming 7 4 The Competition The Night time Fallacy Going for the Numbers The Magician Apprentice © Michael Stahl, 2013, all rights reserved
  77. 77. Summary 75 The patterns are pervasive  Almost inevitable  Awareness is key (engineers; managers)  Driven by organizational and human nature  The solution is only partially technical  © Michael Stahl, 2013, all rights reserved
  78. 78. Thank You! Questions time… michael.stahl@intel.com www.testprincipia.com © Michael Stahl, 2013, all rights reserved

×