WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

WSO2Con Asia 2014 - Effective Test Automation in an Agile Environment

  1. 1. Effective Test Automation in an Agile Environment Krishantha Samaraweera Senior Technical Lead – Test Automation WSO2
  2. 2. Agenda ● Automated testing ● Success stories ● Automation Concept – Test Pyramid ● WSO2 Test Automation Framework ● WSO2 Platform Automated Test Suite ● Lessons learned ● Automation Best Practices ● Continuous Integration and Delivery
  3. 3. What is Test Automation ● Automation testing – Testing which can be done programmatically ● Compare result – Actual and Expected ● Generate reports ● Goal is to reduce the number of test cases to run manually ● Not to eliminate Manual testing all together
  4. 4. Why Automated Testing ● To get early and instant feedback ● Safety net to save time and which is reinvented in to manual testing. ● Doesn't require human intervention. ● Unattended (Overnight) ● Manual testing has high risk of mission out on something
  5. 5. Success Stories ● Google runs 100M+ automated test cases every day ● Google customer surveys team can deliver a change to production 8 mins after code is committed. ● Ebay runs 35000 test cases in 5 hours ● At Facebook, each of 5,000 engineers commits to trunk HEAD at least once a day and the code at trunk HEAD is pushed to production once daily. ● Atlassian JIRA is protected by over 14,000 unit tests and 19,000 functional tests from regressions
  6. 6. Success Stories Cont'd.. ● Etsy used to take 14 hours and an army of people to deploy one release into production before CD kicked in. Now it takes 15 mins, 1 person. They did 729 deployments to production in Nov 2010, only 6 deployment related incidents. ● At Amazon, new code is deployed to production at a staggering rate of once every 11.6 seconds during a normal business day. ● That’s 3,000 production deployments per day.
  7. 7. Test Automation Pyramid
  8. 8. Test Automation Pyramid Cont'd..
  9. 9. WSO2 Test Automation Framework ● Why our own framework – Open source test scripts – Stay lean – Single framework for all products – Facilitate backend testing + UI tests
  10. 10. Technology outlineTechnology outline
  11. 11. Test Automation Framework Architecture
  12. 12. Glimpse of Test Automation Framework ● Simple automation context API. ● Platform wide test execution support. ● Annotation based test execution management support. ● Test reports and coverage report generation. ● Test tooling integration (Jmeter). ● Selenium WebDriver integration and ability to run UI tests on cross browser environments. ● Framework extensibility capability through pluggable modules. ● Easy retrieval of new configurations.
  13. 13. WSO2 Platform Automated Test Suite ● Package tests for other people to use ● Distribution contain all product integration tests and platform tests jars. ● Test script management tool ● Ant based test executor ● Test execution on distributed product clustered or cloud deployment. ● Cross platform test execution
  14. 14. Lesson Learned ● Once you abandon the maintenance of the automation, it is likely to die. For a better chance of success, choose an automation approach that will require the least maintenance. ● Automation requiring multiple cycles and project releases for it to become fully effective and provide an acceptable ROI. ● Start by automating the most valuable tests.
  15. 15. Lesson Learned Cont'd ● Plan your test automation at the beginning of the project. ● Unit Testing FIRST – Fix your test Pyramid. ● Staff should know how to diagnose failures. Minimize calling automators for every issue
  16. 16. Lesson Learned Cont'd.. ● Automation development requires the same discipline as software development. ● Organization wide commitment is required for successful test automation methodology ● Everyone should know what is automated.
  17. 17. Test Automation Benefits – Efficiency ROI Time savings of test execution with automation vs test execution without automation Product # of Tests Manual time per test case Iterations Manual test execution time Automated execution test time Saved time ESB 922 8 min 5 384 hours 10 hours 77 man days G-Reg 1557 8 min 5 1038 hours 15 hours 130 man days
  18. 18. Continuous Integration ● Continuous Integration is a software development practice ● Team integrate their work frequently ● Each commit is verified by automated build ● Rapid feedback ● Ships better code faster ● Fixing build breaks ASAP important. ● Should have a notification mechanism
  19. 19. Continuous Integration Cont'd.. Source :
  20. 20. Continuous Delivery How long would it take your organization to deploy a change [to production] that involves just one single line of code? Do you do this on a repeatable, reliable basis?” - Mary and Tom Poppendieck, Implementing Lean Software Development
  21. 21. Continuous Delivery Build software in such a way that it ready to be released at any time – Continuous feedback – Reduce risk of failures in production system – Rapid, reliable and repeatable automated processes
  22. 22. Test Cases to be Automated ● HIGH RISK – Business critical test cases. ● Features tend to break frequently ● Repetitive tasks are good candidates for automation ● Test cases that are tedious or difficult to perform manually ● Test cases which are time consuming. ● Tests that use multiple data values for the same actions ● Smoke Tests
  23. 23. Test Cases not suitable for Automation ● Test cases that are newly designed – Not executed manually at least once. ● Test cases for which the requirements are changing frequently. ● Usability testing – “How easy is the application to use?” ● One-time testing
  24. 24. Testing Philosophy ● You don't have to write 1,000 tests ● The benefit of the tests scales very nicely. – 1,000 tests is good – 100 tests is also good – 10 tests is also good ● And even one test is better than no tests at all
  25. 25. Testing Philosophy ● So don't start out by saying "Oh, I have to write 1,000 tests for this." ● Then you might not write any at all. ● Start by saying "I'll write one test". ● Then write it ● Then maybe write another one ● You will immediately see the benefit of having one test
  26. 26. Conclusion ● Fix your test pyramid right ● Test maintenance is the king ● Run you tests frequently ● Importance players of successful test automation are right selection of tools, testing process and team. ● Manual testing is also important
  27. 27. References [1]“GTAC 2013: Presentations - Google Test Automation Conference — Google Developers.” [Online]. Available: [Accessed: 17-Mar-2014]. [2]“Continuous Integration for Agile Project Managers (Part 3) | Mike CI blogs.” [Online]. Available: [Accessed: 17-Mar-2014]. [3]“All About Automated Testing.” [Online]. Available: [Accessed: 17-Mar-2014]. [4]“Writing Automated Tests.” [Online]. Available: [Accessed: 17-Mar-2014]. [5]“Continuous Integration.” [Online]. Available: [Accessed: 17-Mar-2014].
  28. 28. Thank You