This document summarizes Vishal Banthia's presentation on achieving reliable mobile test automation. It discusses the difficulties of flaky tests and inconsistent test execution environments. It then presents solutions like capturing comprehensive test reports with screenshots, videos, and logs to better diagnose flaky tests. It also recommends leveraging cloud services for scalable and accessible test execution. The presentation demonstrates these concepts through tools like Bitrise, OpenSTF, and CircleCI used in Mercari's mobile test automation pipeline.

1. Vishal
SRE-SET Automation Night #1
Reliable Mobile Test
Automation

2. Software Engineer in Test
Case Study (Mobile Automation)
SWET (past)
Core Member
Vishal Banthia @vbanthia @vbanthia_

3. Some Previous Talks, Blogs & Tutorials
• Android E2E Testing at Mercari @ Android Test Night
• Android Device Farm at Mercari
• Appium Docker Demo
• STF Appium Example

4. • Test Automation Life Cycle
• How to achieve Reliable Automation
• Demo
Agenda

5. Test Automation Life Cycle

6. • Have you ever wrote any kind of UI Tests? Browser,
Mobile, Desktop, etc
• Do you think reliable mobile test automation is possible?
Questionnaire!

7. Test Automation Life Cycle ...
Product Manager
QA Team
Engineering Team

8. ➢ Product manager comes up with new features
➢ Developers implement them
➢ QA team test new features and do regression testing
➢ Release
Development flow

9. Whats up?
Development flow ...
Product ManagerQA Team Engineering Team
Where is the
feature?
Code is done.
Waiting for QAToo many features
Too many test cases
Too many devices
No Time
I am dying ...

10. Retrospective Meeting ...
Product Manager
QA Team Engineering Team

11. We Need Automation!
Retrospective Result ...

12. Creation of Test Automation Team
Engineering Team
Test
AutomationTeam
Engineering Team

13. Test Automation Team
• SET @google, @mercari
• SWET @DeNA
• SDET @microsoft

14. Journey of Automation Team

15. Start looking for automation tools
detox

16. Initial Learning
• Initial learning such as setup, writing simple scenario is not
too difficult.
• With some programming experience one can start writing
mobile e2e tests.

17. General Approach
Try to automate tests as much as possible from manual QA
test cases list.

18. Writing tests is the easiest part in whole journey!

19. Difficult part is to actually use them in RELEASE flow!

20. What are the difficulties?
• Flaky tests
• Test execution environment

21. Because of these problems, people start losing
trust in it!

22. And this happens…
Sad Reality :(

23. So, what’s next!

24. Let’s go back to difficulties
• Flaky tests
• Test execution environment

25. Flaky Tests

26. ● User Interface issues:
○ UI is not working as per feature specification
○ Test code is not handling UI automation logic properly for scenarios
involving scroll, swipe etc and test is flaky
● Backend issues:
○ Backend is temporary down
○ API is not working as per specification
When can a test fail?

27. When can a test fail? ...
● 3rd party service issues:
○ Service is temporary down and error handling is not being done properly
in test automation logic.
● Hardware issues:
○ Test is failing on specific device because of hardware limitation such as
Memory, Processor.
○ Device wifi is unstable.

28. Problem is not the flaky test!
Problem is we don’t know what is happening
during test execution

29. So, what’s solution?

30. Record Everything!

31. Record ...
• Screenshots
• Screenrecord (Video)
• ADB logs
• Timestamp of each UI action and test assertion
• Test script logs
• Appium logs

32. And put them in a nice report!

33. rspec_html_reporter
• https://github.com/vbanthia/rspec_html_reporter
• Sample Report
• Android Example
http://tech.mercari.com/entry/2017/12/12/103741

34. rspec_html_reporter ...
• Extension of RSpec HTML Formatter with feature such as:
• Pretty report
• Can embed metadata such as screenshot and screenrecord
• ADB Logcat & other metadata are WIP.

35. This report will not solve flaky test itself, but it will help in
distinguishing between a bug and flaky test!
Flaky test solution

36. Let’s go back to difficulties
• Flaky tests
• Test execution environment

37. Test Execution Environment

38. Test Execution Environment
In general, try to use Cloud Services as much as possible.
This provides quick feedback and they are easily scalable.
And anyone should be able to execute test cases anytime
easily
Android E2E Testing at Mercari
Android Device Farm at Mercari

39. Mercari Android CI (Tools & Service)
• Bitrise: To build application
• Google Cloud Storage: To store built apks
• Google Container Registry: To store docker images
• OpenSTF: For Android Device Farm
• CircleCI 2.0: Build docker image & run tests in parallel containers
• CircleCI Artifacts: To store test reports
• AWS Lambda: To trigger nightly builds
• Slack: Notification & manual triggers

40. Demo Video

41. • Mobile test automation is challenging but not impossible.
• With correct approach and planning it is possible to create
reliable test automation and actually improve product
quality and release efficiency
Conclusion

42. Thank you for coming!