Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Copyright	©	DeNA	Co.,Ltd.	All	Rights	Reserved.	Copyright	©	DeNA	Co.,Ltd.	All	Rights	Reserved.	
Methods	to	Sustain	
Long	Te...
Copyright	©	DeNA	Co.,Ltd.	All	Rights	Reserved.	
Bio	
■  Toshiya	Komoda	
⁃  Server	side	engineer	
⁃  Test	Engineer	
■  deve...
Copyright	©	DeNA	Co.,Ltd.	All	Rights	Reserved.	
mobage	game	plaMorm	
■  Mainly	a	browser	game	pla?orms	
⁃  tens	of	million...
Copyright	©	DeNA	Co.,Ltd.	All	Rights	Reserved.	
History	of	mobage	game	plaMorms	
■  Business	trend	shiFs	to	
⁃  browser		t...
Copyright	©	DeNA	Co.,Ltd.	All	Rights	Reserved.	
Some	sta5cs	
■  5		
⁃  years	operaNons	
■  22	
⁃  test	suites	
⁃  grouped	...
Copyright	©	DeNA	Co.,Ltd.	All	Rights	Reserved.	
Life	cycle	of	test	suites	
■  Development	Phase	
⁃  build	E2E	tests	as	QA	...
Copyright	©	DeNA	Co.,Ltd.	All	Rights	Reserved.	
Life	cycle	of	test	suites	
■  Development	Phase	
⁃  build	E2E	tests	as	QA	...
Copyright	©	DeNA	Co.,Ltd.	All	Rights	Reserved.	
Methods to keep
tests healthy
Copyright	©	DeNA	Co.,Ltd.	All	Rights	Reserved.	
1.	Periodic	execu5on	of	all	test	suites	
■ 	Goal		
⁃ improve	stability	of	...
Copyright	©	DeNA	Co.,Ltd.	All	Rights	Reserved.	
2.	Define	clear	process	to	exclude	test	cases	
■  Goal	
⁃ Control		cost	and...
Copyright	©	DeNA	Co.,Ltd.	All	Rights	Reserved.	
3.	Improve	test	debuggability	
■  Goal	
⁃  minimize	test	maintenance	cost	...
Copyright	©	DeNA	Co.,Ltd.	All	Rights	Reserved.	
3.	Improve	test	debuggability	
■  Live	preview	and	video	records	for	UI	te...
Copyright	©	DeNA	Co.,Ltd.	All	Rights	Reserved.	
4.	Make	tests	loosely	coupled	with	SUT	(system	under	test)	
■  Goal	
⁃  ma...
Copyright	©	DeNA	Co.,Ltd.	All	Rights	Reserved.	
4.a	Visual	regression	tes5ng	with	site	crawler	
■  Goal		
⁃  remove	hard-c...
Copyright	©	DeNA	Co.,Ltd.	All	Rights	Reserved.	
4.b	Input	topic	iden5fica5on	with	machine	learning	
■  Goal	
⁃  remove	hard...
Copyright	©	DeNA	Co.,Ltd.	All	Rights	Reserved.	
Summary:	
Methods	to	Sustain	Long	Term	Opera5ons	of	E2E	Automated	Tests			...
Upcoming SlideShare
Loading in …5
×

selenimu commiter day 2017: Methods to Sustain Long Term Operations of E2E Automated Tests

3,296 views

Published on

A presentation slide for a sponser session in selenium commiter day 2017.

https://seleniumjp.connpass.com/event/55364/

Published in: Technology
  • Download this 3-step guide to generating insane amounts of media coverage for your from LinkedIn: http://bit.ly/linkedin3stepguide
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

selenimu commiter day 2017: Methods to Sustain Long Term Operations of E2E Automated Tests

  1. 1. Copyright © DeNA Co.,Ltd. All Rights Reserved. Copyright © DeNA Co.,Ltd. All Rights Reserved. Methods to Sustain Long Term Opera5ons of E2E Automated Tests Selenium Commi<er Day 2017 July 1st 2017 SWET Group Toshiya Komoda (@toshiya_komoda)
  2. 2. Copyright © DeNA Co.,Ltd. All Rights Reserved. Bio ■  Toshiya Komoda ⁃  Server side engineer ⁃  Test Engineer ■  developed game pla?orms, using automated tests ■  currently working in SWET group ⁃  SWET - SoFware Engineer in Test ⁃  dedicated team for test engineering
  3. 3. Copyright © DeNA Co.,Ltd. All Rights Reserved. mobage game plaMorm ■  Mainly a browser game pla?orms ⁃  tens of millions users play games on the pla?orm ■  A variety of services for 3rd party developers ⁃  OpenID Connect ⁃  payment service ⁃  Community service •  friend, chat ⁃  and more
  4. 4. Copyright © DeNA Co.,Ltd. All Rights Reserved. History of mobage game plaMorms ■  Business trend shiFs to ⁃  browser to naNve ⁃  pla?orm to game publisher ■  SNll ⁃  a lot of users play games on our pla?orms ⁃  the profit is huge ■  Our goal ⁃  maintain high availability and strict backward compaNbility ⁃  with minimum maintenance cost automated E2E test suites has been key to achieve this goal
  5. 5. Copyright © DeNA Co.,Ltd. All Rights Reserved. Some sta5cs ■  5 ⁃  years operaNons ■  22 ⁃  test suites ⁃  grouped based on the funcNonality ■  700 ⁃ browser test cases ■  60000 ⁃ web API test cases ■  enable developers to release server codes without manual QA ⁃ reduce lead 5me ⁃ reduce QA Cost
  6. 6. Copyright © DeNA Co.,Ltd. All Rights Reserved. Life cycle of test suites ■  Development Phase ⁃  build E2E tests as QA for service development ■  Healthy Phase ⁃  all test suites are stable enough ■  Broken Phase ⁃  environmental changes, test data corrupNon ⁃  increase of the raNo of unstable tests… ■  Dead Phase ⁃  no one can pass the E2E tests ⁃  QA cost and lead time increase a lot
  7. 7. Copyright © DeNA Co.,Ltd. All Rights Reserved. Life cycle of test suites ■  Development Phase ⁃  build E2E tests as QA for service development ■  Healthy Phase ⁃  all test suites are stable enough ■  Broken Phase ⁃  environmental changes, test data corrupNon ⁃  increase of the raNo of unstable tests… ■  Dead Phase ⁃  no one can pass the E2E tests ⁃  QA cost and lead time increase a lot Gain > maintenance cost
  8. 8. Copyright © DeNA Co.,Ltd. All Rights Reserved. Methods to keep tests healthy
  9. 9. Copyright © DeNA Co.,Ltd. All Rights Reserved. 1. Periodic execu5on of all test suites ■  Goal ⁃ improve stability of test suites ⁃ minimize cost to fix broken tests ■  execute all test suites in all environments ■  we executes tests once in a day ■  test stability is required ■  find broken tests as much as possible
  10. 10. Copyright © DeNA Co.,Ltd. All Rights Reserved. 2. Define clear process to exclude test cases ■  Goal ⁃ Control cost and coverage trade-off ■  Allow to exclude and reNre test cases ⁃ highly flaky test cases ⁃ broken test cases that are hard to fix ■  Example process ⁃ idenNfy the reason of the test failure •  off course, if there is a bug in producNon service, fix it. ⁃ if the reason is not a bug in producNon service ⁃ make JIRA Nckets and exclude the test cases
  11. 11. Copyright © DeNA Co.,Ltd. All Rights Reserved. 3. Improve test debuggability ■  Goal ⁃  minimize test maintenance cost ⁃  maximize the number of engineers who can see tests ■  Test language ⁃  should be familiar language to service developers ■  Test Infrastructure should provide ⁃  performance ⁃  retry mechanism ⁃  precise debug log
  12. 12. Copyright © DeNA Co.,Ltd. All Rights Reserved. 3. Improve test debuggability ■  Live preview and video records for UI tests ■  zalenium ⁃  h]ps://github.com/zalaxndo/zalenium ⁃  developed by zaland ⁃  A selenium grid extension
  13. 13. Copyright © DeNA Co.,Ltd. All Rights Reserved. 4. Make tests loosely coupled with SUT (system under test) ■  Goal ⁃  make tests hard to break ⁃  make tests robust to changes of SUT ■  Web API tests are relaNvely easy to maintain. Why ? ⁃  clear interface between tests and SUT ■  UI tests are not. Why ? ⁃  hard-coded page link informaNon ⁃  hard-coded DOM id or other a]ributes values ■  Can we make UI tests loosely coupled with SUT as API tests ? ⁃  2 ongoing projects will be shown
  14. 14. Copyright © DeNA Co.,Ltd. All Rights Reserved. 4.a Visual regression tes5ng with site crawler ■  Goal ⁃  remove hard-coded link informaNon from UI test codes ■  use crawler techniques in regression tests ⁃  detect links in web pages automaNcally ⁃  take screenshots of all pages in the site 2-up swipe
  15. 15. Copyright © DeNA Co.,Ltd. All Rights Reserved. 4.b Input topic iden5fica5on with machine learning ■  Goal ⁃  remove hard-coded DOM a]ributes values from UI test codes ■  use natural language processing technique to handle input forms ⁃  based on a research paper 「Using SemanNc Similarity for Input Topic IdenNficaNon in Crawling-based Web ApplicaNon TesNng」, Jun-Wei Lin, Farn Wang •  h]ps://arxiv.org/abs/1608.06549 ■  our prototype shows promising results ⁃  can esNmate 80% of input forms ⁃  experiments for login and user register forms ■  can be used with the previous site crawler
  16. 16. Copyright © DeNA Co.,Ltd. All Rights Reserved. Summary: Methods to Sustain Long Term Opera5ons of E2E Automated Tests 1.  Periodic execuNon of all test suites 2.  Define clear process to exclude test cases 3.  Improve test debuggability 4.  make tests loosely coupled with SUT (system under test)

×