Why test automation projects are failing


Published on

Summarizing the experience of dozens successful and unsuccessful projects, the author of the presentation explains the most common reasons of test automation projects failures. Also the speech will cover some best practices and the ways to make UI test automation beneficial. From the presentation you'll find the answers to the following questions:
- Why automated testing is expensive?
- What are the reasons of long test executions?
- What makes automated tests fail unpredictably?
You'll also be able to find solutions to get rid of typical failures on your test automation project.

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Why test automation projects are failing

  1. 1. Why test automation projectsare failing?Igor Khrolhttp://www.khroliz.com
  2. 2. About myself• Igor Khrol• More than 7 years in automatedtesting• Consultant, trainer, manager,engineer, architect• Selenium, HP QTP, TestCompete,Jmeter• Speaker at different conferences
  3. 3. Agenda• What is wrong with test automation?• Five typical mistakes in UI automated testing• Possible solutions to make successfulautomated testing:– Technical– Organizational• Q&A
  4. 4. Successful test automationproject: what’s this?
  5. 5. • What is “success” for testautomation?• The same as for any other project– Customer is happy– Outcome is more than investments• Typical expectations from testautomation– Amount of manual testing is reduced– Regression testing takes less time thanpreviously– More defects are foundTest Automation Success
  6. 6. What is the change to get “success”?BeneficialNo result
  7. 7. What is the change to get “success”?BeneficialNo result80% 20%
  8. 8. Typical Reasons
  9. 9. Reason #1Recording
  10. 10. • The first feature from commercial tools• Looks promising in marketing demosReasons: Recording
  11. 11. • Why failure?– Recorded scripts don’t work for a bit morecomplex apps than demo– Impossible to maintainReasons: Recording
  12. 12. Recording: how to fix
  13. 13. • Don’t use recoding• Write the code manuallyRecording: how to fix
  14. 14. Recording: how to fix• Ensure that recoding is working for yourapplication– Tune the application– Tune the tool• Don’t try to maintain the scripts– In case of failure re-record the script
  15. 15. Recording: how to fix• In case of failure re-record the scriptRecord Run Result FailManualCheckRe-recordPass
  16. 16. Reason #2UI Automation is slow
  17. 17. • UI Automation includes the full end-to-endsolution– Browser (or client app)– Network– Application server (or server app)– Database• UI automation works with browser: not withyour codeReasons: UI Automation is slow
  18. 18. UI Automation is slow:how to fix
  19. 19. • Don’t use “Sleeps” for synchronizationUI Automation is slow: how to fix
  20. 20. • Run test scripts in parallel– Design test architecture for parallelization– Design your test scenarios and test data forparallelization– Use Clouds to run test scriptsUI Automation is slow: how to fix
  21. 21. • Focus on other Test Automation possibilitiesUI Automation is slow: how to fixUIAPI TestingUnit Testing
  22. 22. Reason #3UI Automation is unstable
  23. 23. • The nature of UI Automation• 2-3% of failed test scripts in averageReasons: UI Automation is unstable
  24. 24. UI Automation is unstable:how to fix
  25. 25. • Focus on other Test Automation possibilitiesUI Automation is unstable: how to fixUIAPI TestingUnit Testing
  26. 26. • Re-run failed test scripts before result analysis• Analyze tests that failed twice• Total – 1000• Defects – 5• Failure percentage – 3%• 1 min to execute 1 test• 10 min to analyze 1 failureUI Automation is unstable: how to fix1 runFailed cases(1000 – 5 ) * 3% + 5 = 35Execution time1000 min = 16.7 hTime to analyze35 * 10 min = 5.8 h1 run + 1 re-runFailed cases(1000 – 5) * 3% + 5 = 35(35 - 5) * 3% + 5 = 6Execution time1000 min + 35 min = 17.3 hTime to analyze6 * 10 min = 1 h
  27. 27. Reason #4Too costly to develop andmaintain
  28. 28. • Too slow– Test script should be run 10-15 times in order tobe developed– 2-5 times to update due to changesReasons: too costly to develop andmaintain
  29. 29. • Wrong tool selectionReasons: too costly to develop andmaintainCode with mistake Fixed codeWhy???
  30. 30. • Not enough skills in the teamReasons: too costly to develop andmaintainDevelopment Testing
  31. 31. Too costly to develop andmaintain: how to fix
  32. 32. • Work closely with development team to madeapplication UI automatable– Provide static Id’s for controls– Implement strict procedure for synchronizationToo costly to develop and maintain:how to fix
  33. 33. • “Not enough skills to create the code bymanual testing team” – how to handle?• Separate test automation framework from testscripts creationToo costly to develop and maintain:how to fixAutomation Tool VendorFramework Automation TeamTest Scripts Manual Team
  34. 34. Reason #5Automation is not used
  35. 35. • “We do not trust testautomation – it’s better tore-check manually”• It’s so complex to run andanalyze the scriptsReasons: Automation is not used
  36. 36. Automation is not used:how to fix
  37. 37. • Integrate test automation with existing testsystem• Ease the process of test executionAutomation is not used: how to fix
  38. 38. • Establish clear result analysis procedureAutomation is not used: how to fixRunIf Fail – Do ManuallyPost BugFail – To SUT, Pass – To Autotest
  39. 39. To summarize…
  40. 40. • What to do to get success in test automation– Select right tool, framework and technicalapproach– Distribute carefully the responsibilities– Collaborate closely between the teams:development, QA and test automationSummary
  41. 41. Thank you for attention!Igor Khrol• E-mail: khroliz@gmail.com• Skype: igor.khrol• LinkedIn: http://www.linkedin.com/in/khroliz• Blog: http://en.khroliz.com• Facebook: https://www.facebook.com/khroliz