A presentation slide for a sponser session in selenium commiter day 2017. https://seleniumjp.connpass.com/event/55364/

1. Copyright © DeNA Co.,Ltd. All Rights Reserved. Copyright © DeNA Co.,Ltd. All Rights Reserved.
Methods to Sustain
Long Term Opera5ons of
E2E Automated Tests
Selenium Commi<er Day 2017
July 1st 2017
SWET Group
Toshiya Komoda
(@toshiya_komoda)

2. Copyright © DeNA Co.,Ltd. All Rights Reserved.
Bio
■ Toshiya Komoda
⁃ Server side engineer
⁃ Test Engineer
■ developed game pla?orms, using automated tests
■ currently working in SWET group
⁃ SWET - SoFware Engineer in Test
⁃ dedicated team for test engineering

3. Copyright © DeNA Co.,Ltd. All Rights Reserved.
mobage game plaMorm
■ Mainly a browser game pla?orms
⁃ tens of millions users play games on the pla?orm
■ A variety of services for 3rd party developers
⁃ OpenID Connect
⁃ payment service
⁃ Community service
• friend, chat
⁃ and more

4. Copyright © DeNA Co.,Ltd. All Rights Reserved.
History of mobage game plaMorms
■ Business trend shiFs to
⁃ browser to naNve
⁃ pla?orm to game publisher
■ SNll
⁃ a lot of users play games on our pla?orms
⁃ the profit is huge
■ Our goal
⁃ maintain high availability and strict backward compaNbility
⁃ with minimum maintenance cost
automated E2E test suites has been key to achieve this goal

5. Copyright © DeNA Co.,Ltd. All Rights Reserved.
Some sta5cs
■ 5
⁃ years operaNons
■ 22
⁃ test suites
⁃ grouped based on the funcNonality
■ 700
⁃ browser test cases
■ 60000
⁃ web API test cases
■ enable developers to release server codes without manual QA
⁃ reduce lead 5me
⁃ reduce QA Cost

6. Copyright © DeNA Co.,Ltd. All Rights Reserved.
Life cycle of test suites
■ Development Phase
⁃ build E2E tests as QA for service development
■ Healthy Phase
⁃ all test suites are stable enough
■ Broken Phase
⁃ environmental changes, test data corrupNon
⁃ increase of the raNo of unstable tests…
■ Dead Phase
⁃ no one can pass the E2E tests
⁃ QA cost and lead time increase a lot

7. Copyright © DeNA Co.,Ltd. All Rights Reserved.
Life cycle of test suites
■ Development Phase
⁃ build E2E tests as QA for service development
■ Healthy Phase
⁃ all test suites are stable enough
■ Broken Phase
⁃ environmental changes, test data corrupNon
⁃ increase of the raNo of unstable tests…
■ Dead Phase
⁃ no one can pass the E2E tests
⁃ QA cost and lead time increase a lot
Gain > maintenance cost

8. Copyright © DeNA Co.,Ltd. All Rights Reserved.
Methods to keep
tests healthy

9. Copyright © DeNA Co.,Ltd. All Rights Reserved.
1. Periodic execu5on of all test suites
■ Goal
⁃ improve stability of test suites
⁃ minimize cost to fix broken tests
■ execute all test suites in all environments
■ we executes tests once in a day
■ test stability is required
■ find broken tests as much as possible

10. Copyright © DeNA Co.,Ltd. All Rights Reserved.
2. Define clear process to exclude test cases
■ Goal
⁃ Control cost and coverage trade-off
■ Allow to exclude and reNre test cases
⁃ highly flaky test cases
⁃ broken test cases that are hard to fix
■ Example process
⁃ idenNfy the reason of the test failure
• off course, if there is a bug in producNon service, fix it.
⁃ if the reason is not a bug in producNon service
⁃ make JIRA Nckets and exclude the test cases

11. Copyright © DeNA Co.,Ltd. All Rights Reserved.
3. Improve test debuggability
■ Goal
⁃ minimize test maintenance cost
⁃ maximize the number of engineers who can see tests
■ Test language
⁃ should be familiar language to service developers
■ Test Infrastructure should provide
⁃ performance
⁃ retry mechanism
⁃ precise debug log

12. Copyright © DeNA Co.,Ltd. All Rights Reserved.
3. Improve test debuggability
■ Live preview and video records for UI tests
■ zalenium
⁃ h]ps://github.com/zalaxndo/zalenium
⁃ developed by zaland
⁃ A selenium grid extension

13. Copyright © DeNA Co.,Ltd. All Rights Reserved.
4. Make tests loosely coupled with SUT (system under test)
■ Goal
⁃ make tests hard to break
⁃ make tests robust to changes of SUT
■ Web API tests are relaNvely easy to maintain. Why ?
⁃ clear interface between tests and SUT
■ UI tests are not. Why ?
⁃ hard-coded page link informaNon
⁃ hard-coded DOM id or other a]ributes values
■ Can we make UI tests loosely coupled with SUT as API tests ?
⁃ 2 ongoing projects will be shown

14. Copyright © DeNA Co.,Ltd. All Rights Reserved.
4.a Visual regression tes5ng with site crawler
■ Goal
⁃ remove hard-coded link informaNon from UI test codes
■ use crawler techniques in regression tests
⁃ detect links in web pages automaNcally
⁃ take screenshots of all pages in the site
2-up swipe

15. Copyright © DeNA Co.,Ltd. All Rights Reserved.
4.b Input topic iden5fica5on with machine learning
■ Goal
⁃ remove hard-coded DOM a]ributes values from UI test codes
■ use natural language processing technique to handle input forms
⁃ based on a research paper
「Using SemanNc Similarity for Input Topic IdenNficaNon in Crawling-based Web ApplicaNon TesNng」,
Jun-Wei Lin, Farn Wang
• h]ps://arxiv.org/abs/1608.06549
■ our prototype shows promising results
⁃ can esNmate 80% of input forms
⁃ experiments for login and user register forms
■ can be used with the previous site crawler

16. Copyright © DeNA Co.,Ltd. All Rights Reserved.
Summary:
Methods to Sustain Long Term Opera5ons of E2E Automated Tests
1. Periodic execuNon of all test suites
2. Define clear process to exclude test cases
3. Improve test debuggability
4. make tests loosely coupled with SUT (system under test)