This document discusses test automation strategies for mobile applications. It begins with an introduction of the presenter, Kazuaki Matsuo. It then covers various aspects to consider when designing test automation, including scope, phase, level, size and type. For each aspect, it provides examples from Cookpad's test automation for their Android and iOS applications in both Japan and globally. It discusses adapting concepts to real products and challenges integrating performance testing. Finally, it lists some common tools used for mobile test automation.

1. Kazuaki Matsuo
International Business Development
Product/Engineering Group
What/How To Design
Test Automation
for Mobile
Cookpad Inc.Feb 10th, 2018

2. Who I am
2
Kazuaki Matsuo
• International Business Development Product/Engineering Group
• Software Engineer in Quality
• Social Media
• Twitter: @Kazu_cocoa, GitHub: KazuCocoa
• OSS: Appium Project

3. Why do you evolve
test automation?
3

4. Okay, let’s go
4
Motivations
• Build fast feedback cycle
• Make test deterministic
• Design and build product lines
• Build “Culture of automation” to be autonomous organisation

5. By the way,
5
Can you work with the same context?
A: What about this integration test?
B: Well… we should use mock to make them stable.
A: I won’t use mocking since it hides real world.
B: I see, but without mock, the test may be flaky.
A: This is integration and my understanding of the type is less
mocking and close to in the wild, though…

6. In this session, you can learn
6
Concepts and Ways you go
• Concepts for automated test design / plan
• How adapt them to real projects/products
• Cookpad International applications
• Cookpad Japan applications

7. In this session, you can learn
7
Concepts and Ways you go
• Concepts for automated test design / plan
• How adapt them to real projects/products
• Cookpad International applications
• Cookpad Japan applications
Won’t talk about
business/humans/
techniques/tools

8. Scope Phase Level sIze Type

9. Scope
9
What is your test target?

10. Entire system?
10

11. Entire system?
11

12. Entire system?
12

13. Entire system?
13

14. Focus on the mobile
14

15. Focus on the mobile
15

16. The app has some UI flows
16

17. The app has some UI flows
17

18. Focus on UI flow?
18

19. Focus on UI flow?
19

20. Focus on UI flow?
20

21. Focus on UI flow?
21

22. More internal process?
22

23. More internal process?
23

24. More internal process?
24

25. Thus,
25

26. Thus,
26

27. Thus,
27

28. Scope Phase Level sIze Type

29. 29
Phase
What phase do you focus?

30. What phase would you like to automate?
30

31. What phase would you like to automate?
31
Timeline for release

32. What phase would you like to automate?
32
release

33. What phase would you like to automate?
33
unit test
component
test
coding
low-level
design
integration
test
high-level
design
system
test
acceptance
test
specification
analyse
requirement

34. What phase would you like to automate?
34
unit test
component
test
coding
low-level
design
integration
test
high-level
design
system
test
acceptance
test
specification
analyse
requirement

35. Phase for development or production
35

36. Phase for development or production
36
release

37. What phase would you automate?
37
In Development

38. What phase would you automate?
38
In Production

39. What phase would you automate?
39
In Development
In Production

40. Scope Phase Level sIze Type

41. 41
Level
How deep do you dive to
automation world?

42. What activity is the target?
42
Plan/Design Implement Run
Analyse
The Results

43. How deep would you like to drill down?
43
https://blog.appdiff.com/test-autonomy-levels-7de7967d030e
0. Manual Testing
1. Scripted Testing
2. Exploratory Bots
3. Human-Directed
4. Generative
5. Fully Autonomous

44. Scope Phase Level sIze Type

45. 45
Size
What size to you test?

46. Test Pyramid
46
UI Tests
Integration Tests
Unit Tests

47. 47
UI Tests
Unit Tests
manual testing
Integration Tests
Test Pyramid

48. Test Size
48http://techlife.cookpad.com/entry/2016/08/13/test-size-for-mobile
https://testing.googleblog.com/2010/12/test-sizes.html
feature Small Medium Large Enormous
network no localhost localhost/yes yes
system access no partial/yes yes yes
OS(APIs) no yes yes yes
View no partial yes yes
external system no no no yes
time per a test < 100ms < 2s < 120s < 500s

49. Scope Phase Level sIze Type

50. 50
Type
What type do you test?

51. Agile Testing Quadrants
51
Business Facing
Technology Facing
GuideDevelopment
CritiquetheProduct
http://www.informit.com/articles/article.aspx?p=2253544&ranMID=24808

52. Agile Testing Quadrants
52
Unit Tests
Component Tests (code level)
Testing Connectivity
Business Facing
Technology Facing
GuideDevelopment
CritiquetheProduct
http://www.informit.com/articles/article.aspx?p=2253544&ranMID=24808

53. Agile Testing Quadrants
53
Example
A/B Tests
Story Tests (Written First)
UX (User Experience) Testing
Prototypes (paper/wireframes)
Simulations
Unit Tests
Component Tests (code level)
Testing Connectivity
Business Facing
Technology Facing
GuideDevelopment
CritiquetheProduct
http://www.informit.com/articles/article.aspx?p=2253544&ranMID=24808

54. Agile Testing Quadrants
54
Example
A/B Tests
Story Tests (Written First)
UX (User Experience) Testing
Prototypes (paper/wireframes)
Simulations
Unit Tests
Component Tests (code level)
Testing Connectivity
Exploratory Testing
Workflows
System Integration
(business oriented)
Usability Testing
User Acceptance Testing
Business Facing
Technology Facing
GuideDevelopment
CritiquetheProduct
http://www.informit.com/articles/article.aspx?p=2253544&ranMID=24808

55. Agile Testing Quadrants
55
Example
A/B Tests
Story Tests (Written First)
UX (User Experience) Testing
Prototypes (paper/wireframes)
Simulations
Unit Tests
Component Tests (code level)
Testing Connectivity
Exploratory Testing
Workflows
System Integration
(business oriented)
Usability Testing
User Acceptance Testing
Performance Testing
Load Testing
Security Testing
Quality Attributes (…ilities)
Business Facing
Technology Facing
GuideDevelopment
CritiquetheProduct
http://www.informit.com/articles/article.aspx?p=2253544&ranMID=24808

56. Agile Testing Quadrants
56
Example
A/B Tests
Story Tests (Written First)
UX (User Experience) Testing
Prototypes (paper/wireframes)
Simulations
Unit Tests
Component Tests (code level)
Testing Connectivity
Exploratory Testing
Workflows
System Integration
(business oriented)
Usability Testing
User Acceptance Testing
Performance Testing
Load Testing
Security Testing
Quality Attributes (…ilities)
Outside-in
Barrier-free
Collaborative
Business Facing
Technology Facing
GuideDevelopment
CritiquetheProduct
http://www.informit.com/articles/article.aspx?p=2253544&ranMID=24808

57. Scope Phase Level sIze Type

58. Adapt the design to
real products
58

59. Cookpad Android
for Global
59

60. Before stepping into concrete design…
60
Define your ideal product line
• What is your team’s ideal product line?
• To achieve the line, what kind of activities would you automate?
• Planning/Designing test automation as well

61. Apply SPLIT
61
Scope Phase Level sIze Type

62. Apply SPLIT
62
Scope Phase Level sIze Type
Android provides concept
for “Test Size” officially

63. Split entire system to some parts
63

64. Scope
64

65. Scope 1 / 4
65

66. Scope 2 / 4
66

67. Scope 3 / 4
67

68. Scope 4 / 4
68

69. Phase
69
In Development
In Production

70. Phase
70
In Development

71. Size
71
UI Tests
Integration Tests
Unit Tests

72. Size
72
UI Tests
Integration Tests
Unit Tests

73. Size
73
UI Tests
Integration Tests
Unit Tests
New
New

74. Size
74

75. Size
75
L: UI Component based
M: Integration Tests
S: Unit Tests
E: User Scenario based

76. Size
76
L: UI Component based
M: Integration Tests
S: Unit Tests
E: User Scenario based

77. Size
77
L: UI Component based
M: Integration Tests
S: Unit Tests
E: User Scenario based

78. Size
78
L: UI Component based
M: Integration Tests
S: Unit Tests
E: User Scenario based

79. Size
79
L: UI Component based
M: Integration Tests
S: Unit Tests
E: User Scenario based

80. Type
80
Example
A/B Tests
Story Tests (Written First)
UX (User Experience) Testing
Prototypes (paper/wireframes)
Simulations
Unit Tests
Component Tests (code level)
Testing Connectivity
Exploratory Testing
Workflows
System Integration
(business oriented)
Usability Testing
User Acceptance Testing
Performance Testing
Load Testing
Security Testing
Quality Attributes (…ilities)
Outside-in
Barrier-free
Collaborative
Business Facing
Technology Facing
GuideDevelopment
CritiquetheProduct
http://www.informit.com/articles/article.aspx?p=2253544&ranMID=24808

81. Type
81
Example
Story Tests (Written First)
Unit Tests
Component Tests (code level)
Testing Connectivity
User Acceptance Testing
Business Facing
Technology Facing
GuideDevelopment
CritiquetheProduct
http://www.informit.com/articles/article.aspx?p=2253544&ranMID=24808

82. Convince other members with SPLIT
82

83. In the real world…
83

84. Current status ( Jan, 2018 ~ Feb, 2018)
84
L: UI Component based
M: Integration Tests
S: Unit Tests
E: User Scenario based

85. Scope Phase Level sIze Type

86. Cookpad Android/iOS
for Japan
86

87. History
87
Cookpad Android / iOS run long history
following with Android / iOS platforms evolution

88. History
88
Cookpad Android / iOS run long history
following with Android / iOS platforms evolution
https://academy.realm.io/posts/tasting-tests-at-cookpad-try-swift-2017/

89. Current status ( Sep?, 2014~ Feb, 2018)
89http://techlife.cookpad.com/entry/2016/08/13/test-size-for-mobile
L: UI Component based
M: Integration Tests
S: Unit Tests
E: User Scenario based

90. 90
L: UI Component based
M: Integration Tests
S: Unit Tests
E: User Scenario based
http://techlife.cookpad.com/entry/2016/08/13/test-size-for-mobile
some of manual test
for regressions
Current status ( Sep?, 2014~ Feb, 2018)

91. 91
Cookpad iOS
for Global

92. Any plan or design?
92
No particular entire plan / design
• but, one interesting automaton

93. Type
93
Example
A/B Tests
Story Tests (Written First)
UX (User Experience) Testing
Prototypes (paper/wireframes)
Simulations
Unit Tests
Component Tests (code level)
Testing Connectivity
Exploratory Testing
Workflows
System Integration
(business oriented)
Usability Testing
User Acceptance Testing
Performance Testing
Load Testing
Security Testing
Quality Attributes (…ilities)
Outside-in
Barrier-free
Collaborative
Business Facing
Technology Facing
GuideDevelopment
CritiquetheProduct
http://www.informit.com/articles/article.aspx?p=2253544&ranMID=24808

94. Type for Android
94
Example
Story Tests (Written First)
Unit Tests
Component Tests (code level)
Testing Connectivity
User Acceptance Testing
Business Facing
Technology Facing
GuideDevelopment
CritiquetheProduct
http://www.informit.com/articles/article.aspx?p=2253544&ranMID=24808

95. Automate performance testing
95
Performance Testing
Business Facing
Technology Facing
GuideDevelopment
CritiquetheProduct
http://www.informit.com/articles/article.aspx?p=2253544&ranMID=24808

96. Challenge to integrate performance testing in CI
96
before after
Measure performance for particular scenarios

97. Challenge to integrate performance testing in CI
97
before after
Measure performance for particular scenarios

98. Challenge to integrate performance testing in CI
98
before after
Measure performance for particular scenarios

99. resources/tools
99

100. Tools
100
• Common
• Appium, appium_lib
• fastlane, RSpec, Turnip, turnip_formatter
• KazuCocoa/http_proxy, tomakehurst/wiremock, Kobold
• Android
• JUnit 4, AssertJ, Truth, Espresso, FindBugs(SpotBugs), Composer
• Robolectric, Powermock, Mockito
• KazuCocoa/droid-monitor, KazuCocoa/droid_adbs, KazuCocoa/DroidTestHelper
• iOS
• XCTest(XCUITest), EarlGrey, wix/AppleSimulatorUtils
• OCMock, OHHTTPStubs

101. Scope Phase Level sIze Type

102. kazuaki-matsuo@cookpad.com
@Kazu_cocoa
Thank you
102