Engineering Trust in Your Automated Tests

1. Engineering Trust in Your Automated Tests
Jyoti Mittal
VML

2. Every business wants to go fast…

3. But speed without
quality kills!

4. http://www.computing.co.uk/digital_assets/634fe325-aa28-41d5-8676-855b06567fe2/CTG-DevOps-Review-2017.pdf
“Where are the main hold-ups in the software production process?”
4
…of delays were occurring at the
Test/QA stage of the cycle.63%A 2017 survey of DevOps
practitioners found that

5. QA is the biggest DevOps bottleneck

6. Manual Testing:
Evolve or Die

7. AUTOMATED TESTING
How do you build better apps, faster?

9. Technology Stack
• Saucelabs
• Jenkins driven test execution
• New Relic Test Monitors
• JMeter Performance testing
• Totalvalidator accessibility testing
• BurpSuite Security Testing
UI
Web Services
Unit

10. Why AUTOMATED TESTING is so hard
to achieve?

11. PROBLEM
Test Flakiness is a big

12. Build #1032
Build #1031

13. The boy who cried wolf

14. How bad is the test flakiness?

15. Flaky Tests should NOT be tolerated

16. Google has around 4.2 million
tests that run on continuous
integration system. Of these,
around 63,000 (2%) have a
flaky run over a week.
Tests
https://testing.googleblog.com/2017/04/where-do-our-flaky-tests-come-from.html

17. How to build the Trust?
Create Reliable Tests

18. 1. Quarantine the Flaky Tests

19. Facebook: Automated Bots to manage flaky tests
New

20. 2. Consider flaky tests as a
friend
They’re telling you something.
Be sure to listen.

21. Asynchronous Behavior
Third Party Web Service calls
Concurrency/Race conditions
Test Order Dependency
3. Understand the nature of Flakiness

22. Use Intelligent WaitsSleep is your enemy
Asynchronous Behavior
//pseudo-code
makeAsyncCall;
Thread.sleep(5000);
readResponse;
makeAsyncCall;
startTime = Time.now;
while(! responseReceived) {
if (Time.now - startTime > waitLimit)
throw new TestTimeoutException;
sleep (pollingInterval);
}
readResponse;

23. Concurrency
Thread-safe the test code
Tests should be Atomic
Avoid shared resources

24. 4. Love your Test Code!
SOLID Software Design Principles, DRY principle
Design Patterns like Page Object Model, Factory pattern
Package reusable page object APIs and encourage reusability
Run Static code analysis and set quality gates on the codebase

25. 5. Its okay to start over
Rewrite

26. Quality is a Team Sport

27. Key Takeaways
Test Automation is a MUST to achieve Speed with Quality.
Trust is crucial. Flaky tests should not be tolerated.
Everyone participates in Quality.

28. Thank you!
Linkedin linkedin.com/in/jyoti-mittal/
Twitter @Jyoti_developer