Slides from SmartBear Software's webinar on automated UI testing: How to Make Your UI Tests Stable, Scalable, and Maintainable Watch the full webinar here: http://www2.smartbear.com/outbound-TestComplete-Stable-UI-Tests-Webinar-Replay.html

1. How to Make Your UI Tests
Stable, Scalable, and
Maintainable

2. 2
SmartBear’s technology stack

3. 3
Quick Intro
Nikhil Kaul Jeffrey Martin

4. 4
On our plate today
1
Why you shouldn't tolerate flakiness?
2
Common reasons for flaky UI tests
4
What to do with flaky UI tests?
5
How to separate signal from noise?
3
How to overcome these challenges?

5. 5
Flaky tests shouldn't be tolerated
• 300 tests
• Each test has .5% failure rate
• Pass rate for each tests: 99.5%
• Pass rate for the suite: (99.5%)^300 = 22.23%

6. 6
Understand the nature of flakiness
Tech ComplexitiesTest EnviromentTest DataTest Design

7. 7
Understand the nature of flakiness
Tech ComplexitiesTest EnviromentTest DataTest Design

8. 8
Don’t interact with UI elements directly
Test services seperately from the UI
Use page objects
Automate at three levels
‒ Unit
‒ Service
‒ UI
Use UI encapsulation Build modular test design
‒ Identify user journeys
‒ Unit
‒ Service
‒ UI
Automate at three levels
Use UI encapsulation
‒ Page Objects
UIServiceUnit

9. 9
How to implement in real world
Add Item
Click on an item with name XYZ
Clickitem()
Set element with name ABC to quantity Z
Set element()
Click on button labeled place in cart
clickbutton()
What? How?

10. 10
Understand the nature of flakiness
Tech ComplexitiesTest EnviromentTest DataTest Design

11. 11
Separate the test data from test design
Test services seperately from the UI
Use page objects
Automate at three levels
‒ Unit
‒ Service
‒ UI
Use page objects Build modular test design
– Test specific data
– Reference dataLimit data
dependency
between tests
Tests should run
independently
Using databases
‒ Conduct tests
inside a transaction

12. 12
Understand the nature of flakiness
Tech ComplexitiesTest EnviromentTest DataTest Design

13. 13
Limit your dependency on environments
Test services seperately from the UI
Use page objects
Automate at three levels
‒ Unit
‒ Service
‒ UI
Use page objects Build modular test design
‒ Application API
‒ HTML API
‒ Identify user journeys
Limit calls to external systems
Parameterize connections
Hermetic Servers

14. 14
Understand the nature of flakiness
Tech ComplexitiesTest EnviromentTest DataTest Design

15. 15
Automate at three levels
‒ Unit
‒ Service
‒ UI
Use page objects
Build modular test design‒ Application API
‒ HTML API
Limit calls to external systems
Parameterize connections
Hermetic Servers
Technical complexities can be overcome
AJAX Calls
‒ Callback
‒ Polling Interval
‒ WaitUntil
No Response Calls
‒ Fake objects
‒ Mock
‒ Virtual Service

16. 16
Acceptance test builds should never be red
Separate flaky
from healthy tests
Limit the number of tests as well
as the time

17. 17
Use statistics for measuring test stability
Sample Size (N)
• 50
Desired confidence (α)
• 95% confidence
P
• 29/50 = .58
1-p
• 21/50
𝐵𝑖𝑛𝑜𝑚𝑖𝑎𝑙 𝐶𝐼 = 𝑝 ± z1−α 2
𝑝(1 − p)
n
𝐶𝐼 = .58 ± 1.96
.58(1 − .58)
50
𝐶𝐼 = .58 ± .14
𝐶𝐼 = between 44% and 72%.
±z1−α 2 = . ±z.95 = 1.96

18. 18
Takeaways
1
Flaky tests shouldn't be tolerated
2
Understand the nature of flakiness
4
Use statistics for measuring test stability
3
Acceptance test builds should never be red

19. Questions?