End-to-End Test Automation for Both Horizontal and Vertical Scale
•
7 likes•5,013 views
Slides from my talk at Selenium Camp Test Automation Conference - 2017 https://seleniumcamp.com/talk/end-to-end-test-automation-for-both-horizontal-and-vertical-scale/ Test automation (TA) activity has become a key critical work to guarantee the quality of system under test (SUT) by driving test and also development effort effectively. To bring this efficiency to projects, companies are investing on TA projects in a more motivated way. The question here is how we should design the automation strategy to handle complex TA projects together effectively. It can be done by automating test scenarios as E2E (end to end). Vertical E2E TA consists of; automating Test Data Preparation Phase and Unit, Integration and UI tests. For horizontal E2E TA; UI and Integration test cases, which are automated, designed as integrated real user scenarios. I will tell about the prerequisites, principles and key factors to have E2E automated tests. And also I will share hands on experienced E2E test automation projects that Selenium was the key tool.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to End-to-End Test Automation for Both Horizontal and Vertical Scale
Similar to End-to-End Test Automation for Both Horizontal and Vertical Scale (20)
Recently uploaded
Recently uploaded (20)
End-to-End Test Automation for Both Horizontal and Vertical Scale
- 1. End to End Test Automation for Both Horizontal and Vertical Scale Erdem YILDIRIM
- 2. 2000 – Software Developer 2006 – Discovered Test Automation & Decided to be Test Engineer 2010 – Test Lead at Defence Technologies and Engineering (STM) 2013 – Test Lead at İnnova (Telecommunication) About Me Linkedin: https://tr.linkedin.com/in/erdem-yildirim-0b5ab466
- 3. E2E Test Automation (Definition, Improvements, Vertical & Horizontal Scale, Prerequisites, Advantages of Each Test Levels & Both Scales) Test Automation Strategy (Approaches, How to Form, Criteria/Principles, Key Factors For a Successful Strategy, Gains) E2E Test Automation Experiences (ROI, The Problems Faced In the Process) Need for Both Manual and Automated Testing (Is Manual Tests Still Needed? Cross Tests) Agenda
- 4. Agenda E2E Test Automation (Definition, Improvements, Vertical & Horizontal Scale, Prerequisites, Advantages of Each Test Levels & Both Scales) Test Automation Strategy (Approaches, How to Form, Criteria/Principles, Key Factors For a Successful Strategy, Gains) E2E Test Automation Experiences (ROI, The Problems Faced In the Process) Need for Both Manual and Automated Testing (Is Manual Tests Still Needed? Cross Tests)
- 5. Machines are great for running repetetive / straightforward scenarios that don’t require any intelligence or experience. Test Automation
- 6. End to end testing is verifying that all units of an application and all subsystems of a system interact as expected with each other that the system as a whole works as intended. E2E Test Automation
- 7. Ideal Software Test Automation Pyramid UI Tests Integration Tests Unit Tests Manual (Scripted) Test
- 8. It is OK that automating whole Unit, Inregration, UI levels is great. Can we improve it?
- 10. An Extra Level: Test Data Preparation How to prepare test dataset and bring database to known state for test cases before every test run? Is Teardown Method OK?
- 11. Test Data Preparation Automation Teardown Method with Automation Setup Exercise Verify Teardown SUT Install Create Return ValuesIndirect Inputs Record Mock Object
- 12. Advantages of Automating Test Data Preparation Phase Faster coding, Faster test executions, Clean test environment, Testing with real and more complex test data, Loose coupling between data and code layers Facilitates testing for multiple customers (each with a DB)
- 13. So, we were able to automate the entire vertical E2E testing: Unit, Inregration, UI +Test Data Preparation levels Can we improve it further?
- 14. E2E Test Automation Horizontal & Vertical Scale UI Tests Integration Tests Unit Tests Test Data Preparation CI–ContinuousIntegration Horizontal E2E (UI & Integration Layer) Login / Add to Chart / Checkout / Pay Vertical E2E Implementing tests on all level of the pyramid
- 15. E2E Test Automation Horizontal & Vertical Scale UI Tests Integration Tests Unit Tests Test Data Preparation CI–ContinuousIntegration Horizontal E2E (UI & Integration Layer) Login / Add to Chart / Checkout / Pay Vertical E2E Implementing tests on all level of the pyramid
- 16. Prerequisites of Horizontal E2E Tests All system and subsystems’ test environments should be ready. If they are not ready, mocking work should be performed.
- 17. Prerequisites of Vertical E2E Tests Having a supporting test/development strategy like TDD, BDD, Continuous Testing It should be driven by all stakeholders (dev, tester, pm, tl) in Project.
- 18. Advantages of Vertical and Horizontal E2E Tests Advantages of Horizontal E2E Tests: High business logic coverage, Testing from the user perspective, Prevents production incidents, Gives the confidence that everything is OK Advantages of Vertical E2E Tests: High code coverage, Fast execution, More determined and focussed tests, Useful for especially safety critical softwares
- 20. E2E Test Automation (Definition, Improvements, Vertical & Horizontal Scale, Prerequisites, Advantages of Each Test Levels & Both Scales Test Automation Strategy (Approaches, How to Form, Criteria/Principles, Key Factors For a Successful Strategy, Prerequisites, Gains) E2E Test Automation Experiences (ROI, The Problems Faced In the Process) Need for Both Manual and Automated Testing (Is Manual Tests Still Needed? Cross Tests) Agenda
- 21. E2E Test Strategy Which projects’ tests strategy should be designed as E2E? Can all tests be written as E2E ?
- 22. Test Strategy I Put That Sh t on Everthing*
- 23. Alternate Test Automation Approach Continuous Testing To develop software within shorter delivery cycles; Agile, DevOps and Continuous Delivery approaches are on the forefront. Continuous Delivery supports Continuous Testing which is a testing strategy that consists of a large number of automated unit and acceptance tests but a small number of automated end-to- end tests. Continuous Testing claims that E2E tests are inefficient and they should be limited. It is a means of using Vertical E2E Test Automation effectively but avoiding Horizontal E2E Test Automation
- 24. Which Test Automation Strategy is the Best? There isn’t one absolute strategy that is valid for every SUT
- 25. Criteria / Principles To Form Test Strategy Test Types / Methodologies that your staff is strong, Coverage Strategy: Business Logic Coverage or Code Coverage Execution Time Priority Cost These Criteria should be considered with the dynamics and needs of Project, Staff, Company, Customer.
- 26. It is not an obligation to select & implement a strategy strictly. You can mix the approaches and form your own strategy based on your needs. It’s about finding the right balance. Criteria / Principles To Form Test Strategy
- 27. Key Factors To Form & Implement a Successful Test Automation Strategy Manage test releases effectively Establish test measurement strategy & metrics Establish test management strategy Choose your test delivery model Determine & implement test tools strategy Assess your test & process maturity
- 28. Feasibility Analysis Evaluation of Tools / Methodologies Forming Automation Strategy (Plan, Design) Implementation Test & Maintain Key Factors To Form & Implement a Successful Test Automation Strategy
- 29. Based on Delivery Strategy Automated test executions should be planned and clear CI roadmap should be outlined based on needs.
- 30. An Example Automated Test Execution/CI Strategy Based on Below Criteria Delivery Velocity; should be fast so smoke Unit test first, smoke UI test second, full test sets last-after deploy in order. Test Risk Level: Medium; smoke tests for all test types runs before deployment. But it is risky to run full sets after deployment, since bug could stay undetected.
- 31. Dev Team Version Control Build & Unit Tests UI Test Automation (Smoke) Deploy (Prod) UI Test Automation (Smoke) UI Test Automation (Full Set) Monitoring Dev CI Pre-Production Production Check in Trigger Trigger Approval Approval Approval Approval Feedback Feedback Feedback Feedback Feedback Automated Test Runs After Nightly Build [1] • Unit Test Automation • Web Service Test Automation • UI Test Automation (Full Set) • Cross Browser & Platform Tests Automated Test Runs After Weekly Build [2] • Automated Non Functional Tests (Performance & Load Tests) • Static Code Quality Analysis Feedback Feedback Feedback
- 32. Communication Collaboration Automation Living Documentation Drives whole project team Healthy test basis / requirements Replaces scripted test with automation scripts To Strengthen the Automation Process: BDD – Behaviour Driven Development
- 33. Benefits of E2E Test Automation Velocity Stability Efficiency Money
- 34. E2E Test Automation (Definition, Improvements, Vertical & Horizontal Scale, Prerequisites, Advantages of Each Test Levels & Both Scales Test Automation Strategy (Approaches, How to Form, Criteria/Principles, Key Factors For a Successful Strategy, Prerequisites, Gains) E2E Test Automation Experiences (ROI, The Problems Faced In the Process) Need for Both Manual and Automated Testing (Is Manual Tests Still Needed? Cross Tests) Agenda
- 35. E2E Test Automation Experiences 7 active projects’ tests that are being automated as E2E
- 36. ROI Project Name # of E2E Test Scenarios Manual Test Execution Time Automation Execution Time Ratio of Saving Legalite 235 6 Days (48 hrs) 1,5 Hour 32 times HOPE 85 2 Days (16 hrs) 1 Hour 16 times PROTTON 1200 5 Days (40 hrs) 2 Hours 20 times TTSis 2000 8 Days (64 hrs) 3 Hours 21 times Planor 110 3 Days (24 hrs) 2 Hours 12 times TTS 120 2 Days (16 hrs) 1 Hour 16 times ATS 30 1 Day (8 hrs) 1 Hour 8 times
- 37. Challenges / Problems Faced In the Process Technical Staff Training and Motivation Choice of Tool and Methodology Planning Doing Feasibility Analysis Unproperly. Wrongly designed test automation strategy. Managerial Doing incrementally and showing the work & results.
- 38. Test Team (Agile) Organizational Structure PM PM PM PM Team Team Team Team T L Test Engineers TL: Test Lead PM: Project Manager Me!
- 39. E2E Test Automation (Definition, Improvements, Vertical & Horizontal Scale, Prerequisites, Advantages of Each Test Levels & Both Scales Test Automation Strategy (Approaches, How to Form, Criteria/Principles, Key Factors For a Successful Strategy, Prerequisites, Gains) E2E Test Automation Experiences (ROI, The Problems Faced In the Process) Need for Both Manual and Automated Testing (Is Manual Tests Still Needed? Cross Tests) Agenda
- 40. Testing cost: testing accounts for roughly 26% of IT budgets *According to a 2014 industrial survey of 1543 executives from 25 countries Lack of testing is even more costly; the global cost of locating and removing bugs from software has risen to $312 billion annually and it makes up half of the development time of the average project. * A 2013 study by the Cambridge University Testing Cost vs Lack of Testing Cost
- 41. Trying to find defects on System Under Test using human intelligence and experience. Manual testing
- 42. Is Manual Test Still Needed? Mostly: Yes! Avoid Scripted Testing; Automate it instead Concantrate on Exploratory Testing; Machines are great for running straigtforward tests Humans are great finding new defects
- 43. Support your team by moving testers across teams and a little bit of fun
- 44. At the End of Bug Fest Team Facility; Playing Tabu, Surprise Gifts, Coffee, Free Time
- 46. • V. Garousi and M. V. Mäntylä, "When and what to automate in software testing? A multi-vocal literature review," Information and Software Technology, vol. 76, pp. 92-117, 2016. • Testistanbul 2016 - Keynote: "Why Automated Verification Matters" by Kristian Karl • V. Garousi and D. Pfahl, "When to Automate Software Testing? A Decision-Support Approach Based on Process Simulation," Wiley Journal of Software: Evolution and Process, vol. 28, pp. 272-285, 2016. Effective Software Testing: 50 Ways to Improve Your Software Testing, Addison-Wesley Longman Publishing Co., Inc. Boston, MA, USA ©2002 End-to-End Integration Testing Design, Computer Software and Applications Conference, 2001. COMPSAC 2001. 25th Annual International Distributed End-to-End Testing Management, Enterprise Distributed Object Computing Conference, 2001 HP Capgemini. Sogetti, World quality report 2014-2015. www.sogeti.com/ solutions/testing/wqr/, Last accessed: Sept. 2015. [2] T. Britton, L. Jeng, G. Carver, P. Cheak, and T. Katzenellenbogen, “Reversible debugging software,” University of Cambridge, Judge Business School, Tehnical Report, 2013 How Google Tests Software, James A. Whittaker Behavior Driven Development Webapps mit Groovy Spock und Geb http://www.alwaysagileconsulting.com/articles/end-to-end-testing-considered-harmful/ http://techarcis.com/services/end-to-end-test-strategy/ References
Editor's Notes
- ...with subtopics of...
- ... so we are tring to take advantage of this oppurtunity by automating our tests.
- ...automating tests in this style is called E2E Test Automation in grey literature, especially it is called Horizontal E2E Test Automation.
- Unit Test; a unit is defined as the smallest testable part of an application. each test case is independent from the others. Integration testing tests integration or interfaces between components, system or hardware. UI Test is also called as acceptance test means testing from the user perspective and validating requirements or user stories. At the top; some limited mauel testing activity.
- 1. We can replace manuel test with exploratory test which is an approach not a technique and more productive than scripted testing more traceable than ad-hoc testing. 2. We can add one more layer to the bottom; test data preparation automation layer. 3. We can add Continuous integration that we can execute test automation projects automatically.
- Why are we adding test data preparation layer as an extra layer: answer is again a question. ... instead of doing it in extra level, you can also do it using teardown method in coding phase. Sometimes applying Teardown Method as hardcoded in your test automation code is OK. But we can ask the question that is it the best approach all the time? A good automation architect ensures that scripts are reusable. A great automation architect ensures that scripts and test data, both are reusable.
- Have a look at the four steps of test automation. Note that teardown is the 4.th step. Setup phase of the test, establishes the prior state of the test. The exercise phase is where we execute our test actions. In the Verify phase; we specify and verify the expected outcome. The final phase teardown, is all about housekeeping, we clean up the test environment and put the world back into the state in which we found it. İf we want to syncronize with BDD given/when/then logic; setup means(given) exercise means(when) action, verify mean(then). This is the ideal scenario that integrated systems are up and ready for testing. When they are not; we should prepare mock objects to emulate real systems. So we can test our system and integration in earlier phase without waiting other systems to be ready. When we use mock objects; Setup Works on mock object, mock objects feeds SUT and verification is done over mock object.
- .... it is useful for especially SUT is a Midddleware or you have different test environments that you should run your tests. For example; if you have 5 customers and these customers have different production environments. ------ ------ How can we do this; First extract data from DB, put your DB into that known state DbUnit is a JUnit extension, puts your database into a known state between test runs. Jailer simplifies the extraction of referentially intact data. Once you have defined an extraction model, it can be used to extract data from the production database fast and easy whenever up-to-date test data is required.
- ... ... there are two different views on this topic. One says writing tests independent and loosing test cases from each other is better. And the other says writting dependent scenarios from the end user perspective is better. Both of them have some advantages & disadvantages that we will have a look this again on following slides.
- Tools can be changed based on the needs, these are the ones we prefer.
- ... if they are not even mockable, you can not perform horizontal E2E tests.
- 1. Let’s have a look at the advantages & disadvantages based on the test types. 2. If you want to cover business logic with your tests; UI test automation is the right choice. 3. If code coverage is more important then you should perform unit test automation in first order. 4. And also for the short execution time and low costs, you can prefer unit tests.
- It can not be used in all projects. It should be applied only for the proper ones... We will have a look how can we know and decide our project is proper for E2E
- ...it claims that E2E GUI automation is always tougher than other types of automated tests. So if there is a situation when you can achieve your target by not automating the GUI, but by some other levels like unit and integration.
- You should select the test type that your staff is strong or you should be confident to be able to train them. For the coverage strategy: You should make a choice between Business Logic Coverage or Code Coverage For the Execution Time: You should determine your priorities and limits. Cost: you should evaluate your human resources situation that will work for the automation.
- These are the steps you should consider to form a automation strategy
- In a nutshell, we should work on following topics; Feasibility Analysis Eveluate Project, Staff, Company Culture. There is an important and also trendy topic in test automation World is that you should consider: two W’s of automated testing: when and what Many organizations see software test automation as a solution to decrease testing costs and to reduce cycle time in software development. However, establishment of automated testing may fail if test automation is not applied in the right time, right context and with the appropriate approach. Eveluate Tools / Methodologies [Consider Project Dynamics, third party components that are being used by development team], BDD, TDD Form the Automation Strategy (Plan-Design) “Automating without good test design may result in a lot of activity, but little value.” [Consider reusability & modularity] Test & Maintain [Eveluate defect cause analysis with cause effect analysis]
- Lets design a CI-delivery strategy for executions of automated tests. Firstly, we should work on our criteria; the values written next critearia are complete fictitious values. Normally we should form up them based on your Project dynamics and needs.
- And we formed up our test execution and delivery strategy. Automation means running fewer tests more often. You have to start small by attacking your build acceptance tests first. Then cover your smoke tests. Then move onto your time taking fulls set of tests. Here in this design the critical question is where you put these automated tests? Before or after deployment? But make sure every test you automate, it saves time for a manual tester to focus on more important things. Automate few tests that are valuable and time savers or difficult to do for manual testers. If you did that, the task of automation is done. http://www.softwaretestinghelp.com/bvt-build-verification-testing-process/
- The given part describes the state of the world before you begin the behavior you're specifying in this scenario. You can think of it as the pre-conditions to the test. The when section is that behavior that you're specifying. Finally the then section describes the changes you expect due to the specified behavior
- Velocity: faster bug finding & fixation process. It means reducing cycle time for software development. Stability: We have more stable and robust products. You became more comfident before deployment to production environment. Efficiency: We can test more with less effort. And these three bullets means Saving Money
- We have 7 active projects whose tests are being automated as E2E
- Legalite: Low automation Project works on cloud. It is being automated from the starting point of Project to now with %80 of coverage. In TTSis project: we experienced different kind of FAT (Factory accaptance tests). In this Project test automation coverage is 100 percent. FAT would normally be performed in two weeks with the customer by running all scanarios manually. But we showed them our automated tests and then we executed automated tests instead of manuel tests for FAT. FAT was done in 3 Hours instead of 2 weeks. At the end; customer appraciated us and we had the reputation of doing the product right and our approach of running the automated tests with less effort in less time.
- Technical, Staff: You should prepare and trainn your staff for the tools you select. It sometimes means training them but also sometimes mentoring and motivating them and taking care of their problems. Technical, Tools: Sometimes it is hard to find or buy the proper tool for the Project. You might select hybrid solution and use multiple tools like using SeleniumWebDriver as main tool and sometimes Sikuli to automate the components/objects hard to catch with object based test atumation tools like Selenium.
- Organizational structure of test team and development teams working for projects using agile methodology. Our test team consists of 14 test engineers working in 8 projects. They work generally with the PM. Daily asignment are made by PM’s. I determine the test strategy for the technical test works (test automation, performance, ui test, BDD, ALM) with the test team. Test engineers do the Daily tasks; write test automation-performance test code. Big thanks to them for their great effort. I work on strategy, roadmap, training and career paths of the team members and things that increase the efficiency of the test activities and test team effort.
- Sometimes it is not easy to understand the impact of the testing over product quality and SDLC According to a 2014 industrial survey of 1543 executives from 25 countries, testing and quality assurance of software-intensive systems accounts for roughly 26% of IT budgets [1], but lack of testing is even more costly. A 2013 study by the Cambridge University [2] states that the global cost of locating and removing bugs from software has risen to $312 billion annually and it makes up half of the development time of the average project.
- If your project is not suitable for automating all tests with full coverage then somewhere you will need the experience and intelligence of the test engineer. But avoid manually testing of straightforward scenarios that machines are great for running straightforward scenarios as we mentioned in first slide. Take care of choosing the test cases that can not be automated.
- We perfromed BugFest testing activity. Here our aim was testing the application from end-to-end maunally across different projects. Normally testers works on their daily tasks generally testing a specific development or bugfix. But after I investigated that when ı put a different tester on a Project, new tester could some bugs be fould?
- If there is any questions I would like to answer. You can also ask questions by email or linkedin. Thank you. If we have time I want to give brief information about GEB.