My experiment with existing UI automation tests and reshaping the testing pyramid

1. 1
Reshaping The Testing
Pyramid
Kapil Sethi
kapil.sethi9@gmail.com

2. 2
About Me…
• Quality Analyst
• Automation enthusiast
• Worked with

3. 3
Product Background
• Desktop application
• Offline, Online and Network modes of operation
• Has multiple versions in production
• Multiple teams committing to the same repository

4. 4
As a team

5. 5
As a team

6. 6
As a team

7. 7
Observations
• Road blocks
• 2 different UI test automation approaches (White and
TestComplete)
• Loads of UI automation tests
• Some of the UI automation tests were not part of the
build pipelines
• Flaky builds
• 2 weeks of manual regression testing before each release

8. 8
Road Blocks
• In Detail
• Number of UI automation tests: ~450
• Running on 2 different modes: ~900 UI automation tests
• Parallel execution using 8 VMS
• Average time taken to execute a test suite: ~35-40 mins
• More than ~400 TestComplete UI automation tests

9. 9
Observations

10. 10
Goal

11. 11
The Headache

12. 12
Experiment
• Analysis/Flow
• Analyse existing UI tests
• Check the integration tests coverage
• Identify integration tests candidate and approach
• Share the outcome with the team and finalize the
approach

13. 13
Guinea Pig for the Experiment
• One of the most flaky UI automation test suite:
Reports
• Number of UI tests: 93
• Time taken to complete execution: ~65 mins
• Going to add 11 new features in Reports

14. 14
Result of the Experiment
• Added 5 integration tests in BDD format

15. 15
Outcome of the Experiment
• Added 145 integration tests in BDD format
• Execution time: ~ 3 mins
• Deleted 50 UI automation tests
• Execution time for UI automation time: ~25 mins

16. 16
Going Forward…
• More collaborative effort
• Review unit tests coverage
• Add BDD format integration tests
• Add UI automation tests only if needed

17. 17
Going Forward…
• New Feature: Reports
• Reviewed unit tests coverage
• Added BDD format integration tests: 72 (~2 mins)
• Added UI automation tests: 11 (~4 mins)
• No manual regression testing needed

18. 18
Key Learning
• How to conduct an experiment
• Placing the right tests at right level
• Team Buy-In
• Balancing the existing work with rewrite of
tests/improvement approach

19. 19
Questions