Your SlideShare is downloading. ×
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
The Pathologies of Failed Test Automation Projects
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

The Pathologies of Failed Test Automation Projects

328

Published on

Most test automation projects never die—they just become a mess and are redone. Initial solutions that start well and are full of promise often end up as brittle and unmaintainable monsters consuming …

Most test automation projects never die—they just become a mess and are redone. Initial solutions that start well and are full of promise often end up as brittle and unmaintainable monsters consuming more effort than they save. Political feuds can flourish as different automation solutions compete for attention and dominance. Tests become inefficient in both execution time and resource usage. Disillusionment ensues, projects are redefined, and the cycle begins again. Surely we can learn how to avoid such trouble on the next project. Michael Stahl has analyzed automation projects and identified recognizable failure patterns—mushrooming, duplication, going for the numbers, and others. Michael describes these patterns, suggests how to detect them early, and shares ways to avoid or mitigate them. Whether your team is just starting on test automation—or is already in full flight—you’ll take back ideas to improve the chances of achieving success in your test automation efforts.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
328
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. W3 Test Automation 5/1/2013 11:30:00 AM The Pathologies of Failed Test Automation Projects Presented by: Michael Stahl Intel Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
  • 2. Michael Stahl Michael Stahl is a software validation architect at Intel, working with a team that validates Intel's graphics hardware drivers. In this role, Michael defines testing strategies and work methodologies for test teams, and tests part of the product himself - the work he enjoys most. As a twenty-two year veteran and senior engineer, he has been witness to what works - and what doesn’t - in test strategies, communications, and automation. An avid teacher, Michael enjoys sharing his observations with others. Contact Michael at michael.stahl@intel.com and review previous conference presentations on his website testprincipia.com.
  • 3. The Pathologies of failed Test Automation Projects Michael Stahl, Intel Apr 2013 Based, in part, on work done with Alon Linetzki, Best Testing (www.best-testing.com)
  • 4. 2 On the Menu… • • • Automation failure patterns What can we do? Summary Based, in part, on work done with Alon Linetzki, Best Testing (www.best-testing.com) © Michael Stahl, 2013, all rights reserved
  • 5. Disclaimers    Names and brands referenced herein may be claimed as the property of third parties The views expressed in this presentation are solely my own, and do not in any manner represent the views of my employer Information in this presentation is provided “AS IS” without any warranties or representations of any kind © Michael Stahl, 2013, all rights reserved
  • 6. A common automation story… 4  Once upon a time… I can automate my work!... Mr. Otto Mate Cool! Can you make it… Otto’s Team © Michael Stahl, 2013, all rights reserved
  • 7. A common automation story… 5 Piece of cake! Perfect! Can you add… Mr. Otto Mate What a wonderful world!... Otto’s Team Mr. Mann a. Ger © Michael Stahl, 2013, all rights reserved
  • 8. A common automation story… 6 Arggghhh! Fix Fix Fix Otto! The last release fails!... Otto’s Team Mr. Otto Mate … and mates Sir! I need time! I need help! Makes sense… Mr. Mann a. Ger © Michael Stahl, 2013, all rights reserved
  • 9. A common automation story… 7 Mr. Otto Mate … and mates … and mates … and mates © Michael Stahl, 2013, all rights reserved
  • 10. A common automation story… 8 Let’s REDESIGN!!! Mr. Oto Mate … and mates #$&@***!!! … and mates Mr. Mann a. Ger © Michael Stahl, 2013, all rights reserved … and mates
  • 11. 9 A pattern emerges… © Michael Stahl, 2013, all rights reserved
  • 12. Pattern #1: Mushrooming © Michael Stahl, 2013, all rights reserved
  • 13. A pattern… 11 Single User / Developer Simple Tool Stage 1 – Small & Local © Michael Stahl, 2013, all rights reserved
  • 14. A pattern… 12 Multiple Users / Single Developer Enhanced Tool Stage 2 – Generalization © Michael Stahl, 2013, all rights reserved
  • 15. A pattern… 13 Multiple Users & Developers Complicated Tool Stage 3 – Staffing © Michael Stahl, 2013, all rights reserved
  • 16. A pattern… 14 Multiple Users & Developers Test Case Management Stage 4 – Non-core features © Michael Stahl, 2013, all rights reserved
  • 17. A pattern… 15 Stage 5 – Overload © Michael Stahl, 2013, all rights reserved Arghhh!
  • 18. Pattern #2: The Competition 16 © Michael Stahl, 2013, all rights reserved
  • 19. Pattern #2 – The Competition (ver. 1) 17 Team A Team B © Michael Stahl, 2013, all rights reserved
  • 20. Pattern #2 – The Competition (ver. 2) 18 Team A Team B Team C !!! Team D © Michael Stahl, 2013, all rights reserved
  • 21. Pattern #2 – The Competition (ver. 3) 19 Team A Team B !!! Team C !!! … … !!! … Team D © Michael Stahl, 2013, all rights reserved
  • 22. Pattern #3: The Night Run Fallacy © Michael Stahl, 2013, all rights reserved
  • 23. Pattern #3 – The Night Run Fallacy 21 © Michael Stahl, 2013, all rights reserved
  • 24. Pattern #3 – The Night Run Fallacy 22 Night Time Test Time © Michael Stahl, 2013, all rights reserved
  • 25. Pattern #3 – The Night Run Fallacy 23 Add a snapshot – or a movie snippet – of PAVE Corporate Truism: It’s easier to get budget for machines than for more testers © Michael Stahl, 2013, all rights reserved
  • 26. Pattern #3 – The Night Run Fallacy 24 Add a snapshot – or a movie snippet – of PAVE © Michael Stahl, 2013, all rights reserved
  • 27. Pattern #3 – The Night Run Fallacy 25 Test Automation Truism: Machines create work for more testers © Michael Stahl, 2013, all rights reserved
  • 28. The Tragedy of the Commons multiple individuals… will ultimately deplete a shared limited resource… even when it is not in their long-term interest Garrett Hardin, Science, 1968 © Michael Stahl, 2013, all rights reserved
  • 29. Pattern #4: Going for the Numbers © Michael Stahl, 2013, all rights reserved
  • 30. 28 © Michael Stahl, 2013, all rights reserved
  • 31. Pattern #5 – Going for the Numbers 29 © Michael Stahl, 2013, all rights reserved
  • 32. Pattern #4 – Going for the Numbers 30 Robustness is Invisible © Michael Stahl, 2013, all rights reserved
  • 33. Pattern #5: The Magician Stahl, 2013, all rights reserved Apprentice Syndrome © Michael
  • 34. Pattern# 5 – The Magician Apprentice 32 © Michael Stahl, 2013, all rights reserved
  • 35. Recap: The Patterns Mushrooming 3 3 The Competition The Night time Fallacy Going for the Numbers The Magician Apprentice © Michael Stahl, 2013, all rights reserved
  • 36. 34 So... What can we do? © Michael Stahl, 2013, all rights reserved
  • 37. 35 Are the patterns… Unavoidable ? © Michael Stahl, 2013, all rights reserved ?
  • 38. 36 Counter measures © Michael Stahl, 2013, all rights reserved
  • 39. 37 Mushrooming © Michael Stahl, 2013, all rights reserved
  • 40. Alert Signals & Counter Measures 38 How to use this information?  GPS  Locate the stage you are at  Get directions for the way out  Map  Start right and avoid the wrong turns © Michael Stahl, 2013, all rights reserved
  • 41. Alert Signals 39 © Michael Stahl, 2013, all rights reserved
  • 42. Alert Signals 40 © Michael Stahl, 2013, all rights reserved
  • 43. Alert Signals 41 Failing the project Failing the project Failing the project Failing the project Failing the project © Michael Stahl, 2013, all rights reserved
  • 44. Alert Signals: Stage 1 Small, local, feature-centered 42     Single feature test tool? The creator is the user? “Skunk works”? Key words:   “Tool” “Utility” © Michael Stahl, 2013, all rights reserved
  • 45. Counter Measures: Stage 1 Small, local, feature-centered 43 Ensure the following… and relax:  Code control  Documentation    User Manual (Usage line…) “Green” in the code High level design © Michael Stahl, 2013, all rights reserved
  • 46. Alert Signals: Stage 2 Generalization 44        Additional features? Multiple users? Automation web site? >25% of the tester’s time? Key words: “Use by other testers” “Common Libraries” © Michael Stahl, 2013, all rights reserved
  • 47. Counter Measures: Stage 2 Generalization 45 “The hardest part of building a software system is deciding precisely what to build... No other part of the work so cripples the resulting system if done wrong. No other part is more difficult to rectify later” - Fred P. Brooks (author of “The Mythical Man-Month”) © Michael Stahl, 2013, all rights reserved
  • 48. Counter Measures: Stage 2 Generalization 46     Stage 1 measures Strategy Architecture Lightweight PM    Version control Scope control Bugs & Requests database © Michael Stahl, 2013, all rights reserved
  • 49. Alert Signals: Stage 3 Institutionalization and staffing 47         Requests for additional heads? Automation F2F? Tool-related delays in execution? Key words: “Tool Owner”; “Automation team” “Framework”; “Infrastructure” “Roll back” “Bug fix release” © Michael Stahl, 2013, all rights reserved
  • 50. 48 © Michael Stahl, 2013, all rights reserved
  • 51. Counter Measures: Stage 3 Institutionalization and staffing 49 Stage 1, 2 counter measures Management level decision time         Programming language Framework focus Code and release management Acceptance tests Skillset development Metrics © Michael Stahl, 2013, all rights reserved
  • 52. Counter Measures: Stage 3 Institutionalization and staffing 50 Metrics suggestions  Automation framework quality       Number of false fails Framework’s test results; Bug trends ROI Number of runs Invested effort by type (new, maintenance, rewrite) Number of bugs found by Automation (?) © Michael Stahl, 2013, all rights reserved
  • 53. Alert Signals: Stage 4 Change of focus: Technology  Management 51        Generic features? Test log wading? False fails? Key words: “test suite / cycle generation” “robustness enhancement” “setup issues” © Michael Stahl, 2013, all rights reserved
  • 54. Counter Measures: Stage 4 Change of focus: Technology  Management 52 Stage 1, 2, 3 counter measures Build VS Buy Re-architect      Core / Non-Core Solid infrastructure Influence upstream    Testability hooks Design for Test (automation) © Michael Stahl, 2013, all rights reserved
  • 55. Counter Measures: Stage 4  Build VS Buy? (hint: Buy) See: http://www.stickyminds.com/s.asp?F=S17601_COL_2 Stage 4 – Non-core features © Michael Stahl, 2013, all rights reserved
  • 56. Counter Measures: Stage 4 Change of focus: Technology  Management 54 Build (VS Buy) if…        Competitive edge Existing expertize Core competency Cheaper; Faster Good use of resources Acceptable risk Long term support Main source: Allen Eskelin http://www.informit.com/articles/article.aspx?p=21775 © Michael Stahl, 2013, all rights reserved
  • 57. Alert Signals: Stage 5 Maintenance overload; Re-design 55 Maintenance & logistics overload? Limitations overplay? Loss of credibility? Stage 1 initiatives? Key words          “Did it fail in manual test?” “Architecture limitation” “refactoring”; “redesign” “…I can write a small program…” © Michael Stahl, 2013, all rights reserved
  • 58. Alert Signals: Stage 5 Maintenance overload; Re-design 56 Loss of credibility © Michael Stahl, 2013, all rights reserved
  • 59. Counter Measures: Stage 5 Maintenance overload; Re-design 57 Options:      Continue…  Give up problematic areas Partial return to Stage 1 “We value Robustness over New Features” Prepare for re-design – with a new map... © Michael Stahl, 2013, all rights reserved
  • 60. How to Use this information? 58  GPS   Locate the stage you are at Get directions for the way out Be alert for Alerts Implement Counter Measures Identify your stage Analyze your situation © Michael Stahl, 2013, all rights reserved
  • 61. How to Use this information? 59  Map  Start right and avoid the wrong turns Plan your trip Implement Counter Measures Be alert for Alerts Analyze your situation © Michael Stahl, 2013, all rights reserved
  • 62. 60 The Competition © Michael Stahl, 2013, all rights reserved
  • 63. Pattern #2 – The Competition (ver. 1) 61   Salvageable up to stage 2 Stage 3 and up:    Merge the teams EOL both tools Start a 3rd © Michael Stahl, 2013, all rights reserved
  • 64. Pattern #2 – The Competition (ver. 2) 62  Plugin Architecture © Michael Stahl, 2013, all rights reserved
  • 65. Pattern #2 – The Competition (ver. 3) 63  Fix the main problem Accept, encourage Stage 1 stuff…  Maintaining vigil:   Avoid ver.1 pattern © Michael Stahl, 2013, all rights reserved
  • 66. 64 The Night Run Fallacy © Michael Stahl, 2013, all rights reserved
  • 67. Pattern #3 – The Night Run Fallacy 65 6 5    6 5 Never forget test time, test efficiency Balance test skills VS automation skills Allocate machines or machine time © Michael Stahl, 2013, all rights reserved
  • 68. 66 Going for the Numbers © Michael Stahl, 2013, all rights reserved
  • 69. Pattern #4 – Going for the Numbers 67  Change how you count “Done” © Michael Stahl, 2013, all rights reserved
  • 70. 68 The Magician Apprentice Syndrome © Michael Stahl, 2013, all rights reserved
  • 71. Pattern# 6 – The Magician Apprentice 69 http://www.youtube.com/watch?v=zlnz1rSJj7Y © Michael Stahl, 2013, all rights reserved
  • 72. Pattern# 6 – The Magician Apprentice 70 Automation should not necessarily mimic humans… … it should get the job done © Michael Stahl, 2013, all rights reserved
  • 73. Pattern# 6 – The Magician Apprentice 71 http://www.youtube.com/watch?v=qDVYIT85ntg © Michael Stahl, 2013, all rights reserved
  • 74. Pattern# 5 – The Magician Apprentice 72    Holistic approach VS “test after test” Re-think, re-strategize, re-evaluate before automating Identify Actions, Keywords (KDT) © Michael Stahl, 2013, all rights reserved
  • 75. 73 Summary © Michael Stahl, 2013, all rights reserved
  • 76. The Patterns Mushrooming 7 4 The Competition The Night time Fallacy Going for the Numbers The Magician Apprentice © Michael Stahl, 2013, all rights reserved
  • 77. Summary 75 The patterns are pervasive  Almost inevitable  Awareness is key (engineers; managers)  Driven by organizational and human nature  The solution is only partially technical  © Michael Stahl, 2013, all rights reserved
  • 78. Thank You! Questions time… michael.stahl@intel.com www.testprincipia.com © Michael Stahl, 2013, all rights reserved

×