This document discusses improving automation testing speed at Rakuten. It describes how the author's team previously faced bottlenecks like long setup times and late test feedback. They created a "Mobile Labo" solution using Appium and Selenium to run tests concurrently on multiple devices from a single script. This sped up testing significantly. However, new issues emerged around device clashes and locating devices. The author aims to resolve such problems to further improve speed in the future.

1. Improve Automation Testing
~ speed up deployment ~
Nov. 30, 2018
Sadaaki Emura
Rakuten, Inc.

2. 2
Introduction

3. 3
Who am I
Name : Sadaaki Emura (nickname M)
join in Rakuten: 2007
Group: Service Quality Assurance in LPD
established in 2016
Role: Test Engineer Lead, automation
Hobby: jog , climbing , horse racing

4. 4
LPD organization
Product
Manager
Engineer QA
(Leisure Product Department)

5. 5
Our team (QA)
Scope : 5 services
Members : 9 + off site (if needed)
Skill : QA specialist (almost not programmer)
Mission : Quality Assurance
by manual & automation test

6. 6
Our Automation Testing activity

7. 7
Automation testing activity
2016
Proof
of
concept
2017
Expansion
2018
Improve
speed

8. 8
Why is this year theme “improve speed”?
Make impact
= release many apps frequently
photoAC

9. 9
What was bottleneck in development speed?
Specification
Design
Development Test (E2E) Development
Test Design
Test case / script
Implement
Test Execution
scope

10. 10
What was bottleneck in development speed?
1. Setup test environment takes time
2. Test feedback was late
3. Scripting for PC and SP takes time

11. 11
1. Setup test environment takes time
photoAC
What was bottleneck in development speed?
Example
Install apps (1st time) 120 min
Setup devices 5 min
Run apps 2 min
Closing 5 min
:

12. 12
2. Test feedback was late (E2E)
■One job has
Mouse click : 3 sec
Input form : 5 sec
Page transition : 10 sec
:
2-3 mins / page
10 mins / job
What was bottleneck in development speed?
■Single process worsens speed
Test
job
Test
job
Test
job
10 mins

13. 13
For PC For SP
What was bottleneck in development speed?
3. Scripting for PC and SP takes time
Apps support PC and SP
Need to create 2 script

14. 14
How did we resolve?

15. 15
Solution to resolve these bottleneck
1. Every time, Everywhere, Everyone can use
2. Many test run at same time
3. One script covers multiple devices
Features

16. 16
Test script
management
Mobile Labo Architecture
Chrom
e
FF
IE
iPhon
e
Andro
id
Edge
Safari
CI environment
AppiumAPI
SeleniumAPI
SeleniumAPI
Client side
Test
Environment
(Mobile Labo)
Clients
Windows server MAC server MAC server
Test script
management
Test script
management
https://jenkins.io/
Java

17. 17
Mobile Labo image
https://qiita.com/emurin
Document ▼
Mac
Windows
Smart phone
Appium/Selenium

18. 18
1. Every time, Everywhere, Everyone can use
per person every time
As Is
Install apps (1st time) 120 min
Setup devices 5 min
Run apps 2 min
Closing 5 min
:
To Be
Connect API 1 min

19. 19
2. Many test run at same time
FireFox Chrome
android iOS iOS
One JENKINS runs a couple of test script

20. 20
3. One script covers multiple devices
PC
SP
Able to run test in each device
※apps rule
- same function
- same class name (HTML)
:
example : responsive site

21. 21
What’s happened?

22. 22
Improvement effect 1
Example (keiba)
Betting Lite (投票Lite) in Horse Racing
PC & SP are same function
• Step is Choose race, Bet , Confirm ,
Complete
• 9 betting method / 4 type
• Selecting order of placing(Max 3rd)
⇒ 736 test case x 2 devices(PC , SP)
PC SP

23. 23
Improvement effect 1 (initial phase)
Manual testing
Previous automation
Mobile labo
120 hour
60 hour
Method Speed
30 hour
※1 include scripting , setup task
※2 include scripting task
※1
※2

24. 24
Improvement effect 1 (CI phase)
Manual testing
Previous automation
Mobile labo
120 hour
1.5 hour
Method Speed
0.5 hour
※1 include setup task
※1

25. 25
Improvement effect 2
Example (Wedding)
Wedding Service is responsive site
• Reserve wedding hall
• Review wedding hall
• Search wedding hall
:
⇒ 302 test case x 2 devices(PC , SP)
PC SP

26. 26
Improvement effect 2 (initial phase)
Manual testing
Previous automation
Mobile labo
50 hour
20 hour
Method Speed
11 hour
※1 include scripting , setup task
※2 include scripting task
※1
※2

27. 27
Improvement effect 2 (CI phase)
Manual testing
Previous automation
Mobile labo
50 hour
0.5 hour
Method Speed
0.2 hour
※1 include setup task
※1

28. 28
Improvement effect
Mobile Labo improves speed
FREEIMAGES/Geo Okretic

29. 29
Next challenge

30. 30
photoAC
Next challenge
encounter issues

31. 31
Issue 1 : clash with others
iPhon
e 1
Andro
id
Chro
me
iPhon
e 2
iPhon
e 1
Andro
id
Chro
me
iPhon
e 2
clash
Able to use devices
Not able to use same device
at same time

32. 32
Issue 2 : need to know device location
iPhon
e 1
Andro
id 1
Appium API
MAC server 1
iPhon
e 2
Andro
id 2
iPhon
e X
Andro
id X
Appium API
MAC server X
…
Clients set configuration
- Appium IP address & port
- Capabilities (UDID etc)
Where ?

33. 33
New problem and Next step
Improve speed more
1. Clash with others
2. Need to know device location
FREEIMAGES/Alexander

34. 34
In this year ?

35. 35
New problem and Next step (This was situation in Aug.)
Improve speed more
1. Clash with others
2. Need to know device location
FREEIMAGES/Alexander
Improved

36. 36
New problem was resolved in 2 months
Andro
id
Selenium hub
Appium node
Andro
id
Appium node
・・・・
• Handle node
(client does not need to know where appium)
• Block client if devices are busy