This document discusses test automation, including the purpose of test automation, the test automation process, and the test automation pyramid. The key points are: 1. Test automation aims to improve test efficiency, provide wider test coverage, reduce costs, and speed up testing. 2. The test automation process involves defining the test scope, designing tests, coding tests, setting up the test environment, running tests, and maintaining automation over time. 3. The test automation pyramid illustrates that unit tests should form the base, as they are quick to write and run, while user interface tests are at the top as they are more complex and time-consuming.

1. Test Automation Lesson
Version 1.0 (9th July 2019)
Sadaaki Emura

2. Agenda
1. Purpose of Test Automation
2. Test automation tool
3. Test automation process (End to End test)
4. Test automation pyramid

3. Scope
 System under test is Web Application (Browser , Native app)
 Target test level is System Test (End to End)

4. Purpose of test automation

5. Purpose of Test Automation
What test automation does
• Set up test preconditions (build server, docker, set up app, create test data etc)
• Execute tests
• Validation (compare actual outcomes to expected outcomes)
@いらすとや

6. Purpose of Test Automation
What effect comes
• Improve test efficiency
• Provide wider function coverage (improve test coverage)
• Reduce the total test cost
• Speed up test execution , quick feedback , increase the test cycle
@いらすとや

7. Advantages & Disadvantage Test Automation
Advantage
• Run test repeatedly, faster
• Can do test that cannot be done by manual (complex, wide coverage, repeat)
• Reduce human operation error, improve consistency of tests
• Effective and efficient use of testing resource
Disadvantage
• Additional cost for scripting and maintenance
• Require additional technologies , skills
• Need maintenance ( update script, run stable )
• Test can be more complex
• Additional investigation for errors by automation
Limitation
• Not all manual tests can be automated
• Can only check machine-understandable results
• can only check actual results with test oracle prepared (compare)
• Cannot do exploratory testing

8. Test automation tool

9. Tool depend on Test level
@wikipedia
• Requirements and architecture phase
• Check documentation format, proofreading
• Ex. RedPen, ClearDoc
• Implementation
• Statistic analysis
• check coding rule, bug, security issue
• Ex. Coverity , Adlint
• Dynamic analysis (Unit test)
• Check program specification by unit level, white-box
• Ex. Junit, PHPinit
• Integration test and verification
• Check interface level (API)
• Ex. SoapUI
• System Verification and Validation
• Check End to End level, load test, security test
• Ex. Selenium, Appium, Ranorex, UFT

10. End to End test automation tool
How to
recognize object
License Technical skill
Ranorex Id, XPath Not free Middle level
UFT Id, XPath Not free Middle level
Selenium IDE Id, XPath Free Low level
Eggplant Image Not free Middle level
Selenium +
Webdriver
Id, XPath Free High level

11. How to recognize objects
Class , Id Xpath

12. Test automation process (End to End test)

13. Test Automation process
1. Test scope
2. Test design
3. Coding design
4. Setup scripting environment (System Under Test)
5. Script code
6. Check test automation
7. Run test automation, evolve into Continuous Integration
8. Maintain test automation

14. Test Automation process
1.Test scope
Decide which area , function should be automated
• Collaborate with manual testing
• Consider cost and building time
Sample standard
• No scope automation cannot do test
• No scope the functions scripting is complex
• No scope scenario reproduce is difficult
• No scope test is seldom used
• No scope the functions specification is changed frequently

15. Test Automation process
2.Test design
Test automation is same as app.
That’s why test automation specification is needed.
Test design is same as this specification.
• Basically same as manual test design / scenario
• Clarify step, what is validated and what is expectation for machine to understand
• Clarify what is not automated (constraint) in this step to cover it by manual testing

16. Test Automation process
3.Coding design
Consider architecture same as app programing
• Simple code, coding rule
• Maintainability (library , object oriented re-usable)
• Performance ( minimum steps )
• Data driven ( data and script are separated )
• Keyword driven ( encapsulation steps )
• Consistency ( initialize, tear down )
• Sensitive and robust

17. Data driven
Capture
1. Input ‘testuser001’ in userid
2. Input ‘abcdefg’ in password
3. Click ‘login’
Data driven
1. Input $userid in userid
2. Input $password in password
3. Click ‘login’
Data source
userid , password
testuser001 , abcdefg
testuser002 , hijklmn
Separate step and data
Reusability
Maintainability

18. Sensitive and robust
How volume do test automation validate?
Sensitive
• Compare many information to validate
• Coding cost increase
• Easy to fail by little change UI
Robust
• Limit little information to compare
• Coding cost is a small
• Difficult to fail by little change UI
• Miss bug often

19. Test Automation process
4.Setup scripting environment (System Under Test)
build test environment
• Script test code based on this environment
• If this system does not work, it’s difficult to script code
• If system UI include DOM level was changed , code might be remade
5.Script code
• Recognize object by id, XPath, image etc.
• Build step
• Set validation that compare actual with expectation

20. When can we start scripting ?
Ex. Waterfall model
requireme
nt
Design coding test release
Class, id 設計する？
Build environment
When is system stable ?
Actually we can
start scripting

21. Test Automation process
6.Check test automation
• Run script and check if test automation is correct
• Basically if app has bug, test automation might not be correct even if it works successfully
• Need to do failed test pattern
• Check if report is enough
7.Run test automation, evolve into Continuous Integration
• Do test
• Use for contentious integration

22. Test Automation process
8.Maintain test automation
• Increase coverage
• Up-to-date app specification
• Up-to-date test automation tool version, test device version
• Monitor test results from CI
• Monitor performance , tune up script
• Check flaky test script, investigate and fix it
• Delete unused , un-useful test script
• Reduce task dependency in CI
• Visualize metrics ( success/failed , performance )
• Keep good data ( delete unnecessary data )

23. Test automation pyramid

24. Test Automation pyramid
(c) OnTestAutomation
Top Layer : User interface tests
• Scripting test and execution takes long time
• Environment is unstable
Base Layer : Unit tests
• Script test and execute quickly
• Difficult to find integration bugs
• Analyze and fix Bug are easy