Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Decreasing false positives in automated testing


Published on

False positives: automated testing’s arch nemesis.

When automated tests are written well, they are part of a healthy CI/CD process that can save developer time and company money. But when a team gets false positives from unreliable tests, the entire build can get derailed. What’s worse, too many false positives can erode an organization's belief in the value of using a test automation framework at all.

Published in: Engineering

Decreasing false positives in automated testing

  1. 1. Reducing False Positives & Presents
  2. 2. QASource Confidential QASource & automation  100,000+ automated test cases  100+ Selenium engineers  500+ years collective automation experience  100+ automation frameworks
  3. 3. QASource Confidential Question #1 How many years of experience do you have with Selenium? < 2 years 2-4 years 5+ years What is the size of your automation suite? < 100 test cases 100-250 test cases 250-500 test cases 500+ test cases What is the most important reason why you invest in automation? Save Money Save Time/ Release Faster More QA Coverage Question #2 Question #3 Attendees poll
  4. 4. QASource Confidential Anand Ramakrishnan QA Director, QASource • 15+ years of automation experience • Directing a team of 150 engineers • M.S. in Computer Applications • M.S. in Cyber Law & Security
  5. 5. QASource Confidential Key learnings & takeaways What are false positives and why do they occur Common causes and the challenges How to implement key strategies to reduce them Case studies and real world examples FREE step by step guide to creating CSS locators
  6. 6. QASource Confidential What are false positives? Tests that are marked as failed, when in reality they should have passed or should functionally work.
  7. 7. QASource Confidential Why do false positives occur? Automation Approach Implemented Framework Written Test
  8. 8. QASource Confidential Relying on UI Relying on sequence of execution Manual intervention Slow performance Reference data change Shared environment Change in UI element properties Potential causes for false positives
  9. 9. QASource Confidential Impacts on automation Ignoring failures Wasting time investigating failures Decrease in productivity Frustration for engineering team Risk of missing potential bug Increasing cost of automation Uncertainty of application health
  10. 10. QASource Confidential Attendees poll What is the rate of false positives in your automation? 5% or less 5% to 15% 15% to 20% 20% or more
  11. 11. QASource Confidential  Complex workflow application  Rich interactive UI  2500+ automated test cases  Page object pattern approach  Multiple OS, browsers and environments Case study
  12. 12. QASource Confidential  Frequent UI Changes  Synchronization issues  Memory leak issue Challenges  Elements identified using Xpath  Hardcoded waits  No tear down ROOT CAUSE CHALLENGES
  13. 13. QASource Confidential Keys to reducing false positives Deploy application on optimal configurations
  14. 14. QASource Confidential Keys to reducing false positives Controlled environment for automation
  15. 15. QASource Confidential Keys to reducing false positives Keeping tests short
  16. 16. QASource Confidential Keys to reducing false positives Keeping tests independent TEST SCRIPT 1 TEST SCRIPT 2 TEST SCRIPT 3 TEST SCRIPT 4 Create User Create Group Add User to Group Search User
  17. 17. QASource Confidential Using right locators for objects identification Keys to reducing false positives IDs Name CSS locators
  18. 18. QASource Confidential Bad Practice Object identification strategy Good Practice
  19. 19. QASource Confidential Keys to reducing false positives Tear-down approach
  20. 20. QASource Confidential Bad Practice Tear down approach
  21. 21. QASource Confidential Good Practice Tear down approach
  22. 22. QASource Confidential Keys to reducing false positives Dynamic object synchronization
  23. 23. QASource Confidential Bad Practice Object synchronization
  24. 24. QASource Confidential Good Practice Object synchronization
  25. 25. QASource Confidential Keys to reducing false positives Re-execution capability RE-EXECUTION LOGIC FLAG TEST SUITE TEST CASE 1 TEST CASE 2 ————— ————— ————— TEST CASE N Pass Failed RE-EXECUTION OF FAILED TESTS Test case name Status Re- executed TEST CASE 1 PASS NO TEST CASE 2 PASS YES TEST CASE 3 FAIL YES _____ _____ _____ TEST CASE N PASS YES EXECUTION EXECUTION REPORT
  26. 26. QASource Confidential Conclusion Using right locators for objects identification Dynamic object synchronization Re-execution capability Tear-down approach Keeping tests short Keeping tests independent Controlled environment for automation Deploy application on optimal configuration
  27. 27. QASource Confidential Benefits of eliminating false positives Certainty of application health Increase in productivity Save time not investigating false positives Will not miss potential bugs Decrease cost of automation
  28. 28. QASource Confidential Get your FREE Step by Step Guide to Create CSS Locators
  29. 29. QASource Confidential Q&A
  30. 30. QASource Confidential