SlideShare a Scribd company logo
All is thanks to you, Selenium
ありがとう、セレニウム
April 18, 2019
Selenium Conf Tokyo 2019
Woohyeok Aaron Kim (@woosyume)
EC Incubation Department
Rakuten, Inc.
2
Woohyeok Kim
# Software engineer at Rakuten
# Worked in Japan since 2016
# Ex-Captain at Korea Army
キム・ウヒョク
3
Profile
ment
4
Woohyeok Kim
Working at Ranking Frontend Team
Experience of frontend project release.
Driving Test Automation project
# Software engineer at Rakuten
# Worked in Japan since 2016
# Ex-Captain at Korea Army
Enjoying to improve our product
Google AMP Adoption
Validating Module for Half-term Ranking
Some kaizen of modules to improve the current things.
Always pursue something
キム・ウヒョク
5
Overview
Ranking Service Intro ランキングサービスのご紹介
Background 背景
Motivation きっかけ
What we’ve done やったこと
Achievement 成果
Synergy 相乗効果
Conclusion まとめ
6
Ranking Service Intro
ランキングサービスのご紹介
7
Aggregate sales data from Rakuten Ichiba.
Calculate score for each item.
Provide bestsellers information.
What is Ranking Service ランキングサービスとは
1) Irastoya, 2019/04/03, https://www.irasutoya.com/
2) Ranking Top, 2019/03/07, https://ranking.rakuten.co.jp
1) 1)
2)
8
Ranking Service System Structure
Platform
provides score data
API
provides fixed rank data
Frontend
provides web UISupports Other services
ランキングサービスの仕組み
9
Background
背景
10
Our issues about UI Test
Manual UI Test takes TOO MUCH time
Bottle Neck
UIテストについての悩み
11
それって
めっちゃ
ストレス溜まるやん!
METTCHA STRESS
12
Concern if we can implement FULLY-AUTOMATED test or not
We need to check actually UNLIMITED cases
Our issues about UI Test
FASHION JACKET COLOR
PANTS SIZE
Duration Top Genre Genre
REALTIME
DAILY
WEEKLY
MONTHLY
FOOD
…
…
…
4 39 13,000
Tag
130,000
UIテストについての悩み
13
Concern about HIGH Cost for development
Our issues about UI Test
Frontend Application Test Application
UIテストについての悩み
14
METTCHA STRESS
助けてください
SOMEBODY HELP ME
15
Motivation
きっかけ
16
Opportunity for us to start test automation project テスト自動化のきっかけ
1)
1) Selenium HP, 2019/04/03, https://www.seleniumhq.org/projects/
2) Ranking Top, 2019/03/07, https://ranking.rakuten.co.jp
2)
17
Expected ROI(Return On Investment)
I proposed the following ROI and it was approved.
Investment Return
4 MONTHS / PPL 1MONTH / PPL
期待した効果
1) pngtree, 2019/04/03, https://pngtree.com/so/approved
1)
18
What we’ve done
実装した内容
19
User Behavior Check
Click function
Move to expected URL
ユーザ行動の確認
1)
1) Ranking Top, 2019/04/03, https://ranking.rakuten.co.jp
20
Banner / Advertisement Check
Contents itself
Click function
バナー・広告の確認
1)
1) Ranking Top, 2019/04/03, https://ranking.rakuten.co.jp
21
Data Integrity Check
Ranking Result
- Rank
- Item Name
- Shop Name
- Price
- Review#
- Particular condition
Update DateGenre Name
ランキングデータ整合性の確認
1)
1) Ranking Top, 2019/04/03, https://ranking.rakuten.co.jp
22
Others
External Resource Errors, AMP Syntax Errors
その他の確認
23
Scope
Additionally we have 2 more products in Scope.
Smartphone Web Smartphone Web with AMP
実装スコープ
1) 2)
1) Ranking Top, 2019/03/07, https://ranking.rakuten.co.jp
2) Ranking Event, 2019/03/07, https://ranking.rakuten.co.jp/event/supersale
24
Create expected value dynamically
For decreasing maintenance cost
(automatic request creation for API)
DatabaseRanking API
Test Automation
Application
Ranking Front
入力値・期待値の自動生成
1)
1) Ranking Top, 2019/04/03, https://ranking.rakuten.co.jp
25
Image Comparison
Increase test accuracy by checking if image is broken or not.
Calculates difference with the previous execution
Before After
TEST FAILS
画像比較による文字化け・画面崩れの検知
1)
1) Ranking Event, 2019/04/03, https://ranking.rakuten.co.jp/event/anpanman/
26
Achievement
成果
27
Progress per Number of Tests
Device Total Automated Coverage
PC 180 153 85%
Smartphone 180 72 40%
Smartphone with AMP 90 27 30%
Total 450 252 60%
自動化したテストケースの数
28
Actual ROI(Return On Investment) on current phase
Investment Return
Expected 4 MONTHS / PPL 1MONTH / PPL
Actual
1.3MONTH / PPL
So far
0.6MONTH+ / PPL
期待した効果と実際の効果
29
Cost Improvement per execution
Device Before After Reduction
PC 8hours 3hours 60%
Smartphone 8hours 3hours 60%
Smartphone with AMP 3hours 2hours 40%
Total 19hours 8hours 59%
テスト実行のコスト削減
30
Enhancing test accuracy : Assertion#
5000
0
Before
機械の力でテストの精度を高める
After
6200+
500
31
Is that all?
32
Synergy
相乗効果
33
Kaizen initiatives rise
5
0
10
2017.09 ~ 2018.09 2018.10 ~ 2019.04
Increase
chance
4x
to provide user
with
GOOD SERVICE
改善チケットの消化率が高まる
12
6
34
Getting knowledge for product
Need to understand current behavior
Document Code Test Case
プロダクトについての理解が深まる
35
Getting knowledge for product
if ( genre has child ) ?
Application
プロダクトについての理解が深まる
1)
1) Ranking Top, 2019/04/03, https://ranking.rakuten.co.jp
36
Test Cost Reduction for the other frontend projects
Myth : Test Automation Project IS JUST COSTS
AMP Project jQuery Update
Event Ranking
for Smartphone
Bookmark
Improvement
他フロントリリースのコスト削減
Saved
37/ 76hours
37
Tips
役立つ情報
38
Consider Future Cost
Minimize future maintenance cost by creating expected value dynamically.
Properties Input file Database API
将来の管理コストの考慮
39
Consider Future Cost
Minimize future maintenance cost by creating expected value dynamically.
Properties Input file Database API
将来の管理コストの考慮
40
Consider execution time
It results from slow-test issue
Optimizing processing time with @Category
Be able to execute necessary test cases only beyond packages.
Package A
Package B
▶ Test A
▶ Test B
▶ Test C
▶ Test D
実行時間の考慮
41
Manage version matching with the object
Update features continuously
By inputting test automation application into project scope
Estimation
Consider if creating
new test feature is
necessary or not
Development
Implement new test feature at the same
time of implementing unit test
Regression Test
Test newly
updated module
with the latest
version of
automation app
Release
Merge it to
master branch
after confirming
the test works on
PROD also
テスト対象とバージョンを合わせる
42
Let it be easy to analyze the root cause when test fails
Let’s think about the reason why we executes test
Provide user good product by finding and fixing bugs
Suite
Class
Method
Fail Message / Debug Log
Fail Message
Debug Log
失敗の原因を明確にする
43
Blueprint
未来図
44
Blueprint
Execute Automatically
at every commit
For zero cost
未来図
45
Conclusive Thoughts
提案
Conclusive Thoughts
46
Conclusive Thoughts
TRY Selenium
Prioritize the most valuable parts for your business
Selenium can change more than we thought
Foster Team Culture for service improvement
提案
Sales
Resource
* Pareto Rule
20% INPUT accounts for
80% OUTPUT Feature A
20%
Test Cases
Feature A
70%+B
6.47%
C
7.17%
D
9.50%
Sales
Advertisement
[SeleniumConf2019 Tokyo] All is thanks to you, selenium

More Related Content

What's hot

API Testing – Keeping a Check on Agile Software Development
API Testing – Keeping a Check on Agile Software DevelopmentAPI Testing – Keeping a Check on Agile Software Development
API Testing – Keeping a Check on Agile Software Development
Software Testing Solution
 
API Fortress - Evolving from API Testing to API Monitoring
API Fortress - Evolving from API Testing to API MonitoringAPI Fortress - Evolving from API Testing to API Monitoring
API Fortress - Evolving from API Testing to API Monitoring
Patrick Poulin
 
Pom unit 1
Pom unit 1Pom unit 1
Pom unit 1
Shivani Sharma
 
TaraVault Overview Presentation (2019)
TaraVault Overview Presentation (2019)TaraVault Overview Presentation (2019)
TaraVault Overview Presentation (2019)
Inflectra
 
Testing
TestingTesting
Testing
Gaurav Nigam
 
From One Test To Test Framework With Rapise
From One Test To Test Framework With Rapise From One Test To Test Framework With Rapise
From One Test To Test Framework With Rapise
Inflectra
 
Test Automation Frameworks: Assumptions, Concepts & Tools
Test Automation Frameworks: Assumptions, Concepts & ToolsTest Automation Frameworks: Assumptions, Concepts & Tools
Test Automation Frameworks: Assumptions, Concepts & Tools
Amit Rawat
 
Methodologies 3: Using Spira for Waterfall
Methodologies 3: Using Spira for WaterfallMethodologies 3: Using Spira for Waterfall
Methodologies 3: Using Spira for Waterfall
Inflectra
 
Preparing for Holiday Performance, API edition
Preparing for Holiday Performance, API editionPreparing for Holiday Performance, API edition
Preparing for Holiday Performance, API edition
SmartBear
 
Api testing and steps to do it
Api testing and steps to do itApi testing and steps to do it
Api testing and steps to do it
Zoe Gilbert
 
Enforcing Your Organization's API Design Standards with SwaggerHub
Enforcing Your Organization's API Design Standards with SwaggerHubEnforcing Your Organization's API Design Standards with SwaggerHub
Enforcing Your Organization's API Design Standards with SwaggerHub
SmartBear
 

What's hot (11)

API Testing – Keeping a Check on Agile Software Development
API Testing – Keeping a Check on Agile Software DevelopmentAPI Testing – Keeping a Check on Agile Software Development
API Testing – Keeping a Check on Agile Software Development
 
API Fortress - Evolving from API Testing to API Monitoring
API Fortress - Evolving from API Testing to API MonitoringAPI Fortress - Evolving from API Testing to API Monitoring
API Fortress - Evolving from API Testing to API Monitoring
 
Pom unit 1
Pom unit 1Pom unit 1
Pom unit 1
 
TaraVault Overview Presentation (2019)
TaraVault Overview Presentation (2019)TaraVault Overview Presentation (2019)
TaraVault Overview Presentation (2019)
 
Testing
TestingTesting
Testing
 
From One Test To Test Framework With Rapise
From One Test To Test Framework With Rapise From One Test To Test Framework With Rapise
From One Test To Test Framework With Rapise
 
Test Automation Frameworks: Assumptions, Concepts & Tools
Test Automation Frameworks: Assumptions, Concepts & ToolsTest Automation Frameworks: Assumptions, Concepts & Tools
Test Automation Frameworks: Assumptions, Concepts & Tools
 
Methodologies 3: Using Spira for Waterfall
Methodologies 3: Using Spira for WaterfallMethodologies 3: Using Spira for Waterfall
Methodologies 3: Using Spira for Waterfall
 
Preparing for Holiday Performance, API edition
Preparing for Holiday Performance, API editionPreparing for Holiday Performance, API edition
Preparing for Holiday Performance, API edition
 
Api testing and steps to do it
Api testing and steps to do itApi testing and steps to do it
Api testing and steps to do it
 
Enforcing Your Organization's API Design Standards with SwaggerHub
Enforcing Your Organization's API Design Standards with SwaggerHubEnforcing Your Organization's API Design Standards with SwaggerHub
Enforcing Your Organization's API Design Standards with SwaggerHub
 

Similar to [SeleniumConf2019 Tokyo] All is thanks to you, selenium

Agile methodology
Agile methodologyAgile methodology
Agile methodology
Dipanway Bhabuk
 
Building Performance Testing into the Development Process
Building Performance Testing into the Development ProcessBuilding Performance Testing into the Development Process
Building Performance Testing into the Development Process
ba66e77
 
ENT206 Product Development in the Cloud
ENT206 Product Development in the CloudENT206 Product Development in the Cloud
ENT206 Product Development in the Cloud
Amazon Web Services
 
Product Development in the Cloud
Product Development in the Cloud Product Development in the Cloud
Product Development in the Cloud
Amazon Web Services
 
QUES#19 Automation and Quality 2022.pdf
QUES#19 Automation and Quality 2022.pdfQUES#19 Automation and Quality 2022.pdf
QUES#19 Automation and Quality 2022.pdf
sonalsingh547884
 
Amazon CI/CD Practices for Software Development Teams - SRV320 - Anaheim AWS ...
Amazon CI/CD Practices for Software Development Teams - SRV320 - Anaheim AWS ...Amazon CI/CD Practices for Software Development Teams - SRV320 - Anaheim AWS ...
Amazon CI/CD Practices for Software Development Teams - SRV320 - Anaheim AWS ...
Amazon Web Services
 
Agile-plus-DevOps Testing for Packaged Applications
Agile-plus-DevOps Testing for Packaged ApplicationsAgile-plus-DevOps Testing for Packaged Applications
Agile-plus-DevOps Testing for Packaged Applications
Worksoft
 
Amazon CI/CD Practices for Software Development Teams - SRV320 - Atlanta AWS ...
Amazon CI/CD Practices for Software Development Teams - SRV320 - Atlanta AWS ...Amazon CI/CD Practices for Software Development Teams - SRV320 - Atlanta AWS ...
Amazon CI/CD Practices for Software Development Teams - SRV320 - Atlanta AWS ...
Amazon Web Services
 
implementing_ai_for_improved_performance_testing_the_key_to_success.pdf
implementing_ai_for_improved_performance_testing_the_key_to_success.pdfimplementing_ai_for_improved_performance_testing_the_key_to_success.pdf
implementing_ai_for_improved_performance_testing_the_key_to_success.pdf
sarah david
 
Amazon CI/CD Practices for Software Development Teams - SRV320 - Chicago AWS ...
Amazon CI/CD Practices for Software Development Teams - SRV320 - Chicago AWS ...Amazon CI/CD Practices for Software Development Teams - SRV320 - Chicago AWS ...
Amazon CI/CD Practices for Software Development Teams - SRV320 - Chicago AWS ...
Amazon Web Services
 
Automate you Appium test like a pro!
Automate you Appium test like a pro!Automate you Appium test like a pro!
Automate you Appium test like a pro!
TestObject - Mobile Testing
 
1. Watch the video a. Episode 9 Munchausen by Proxy Dying fo
1. Watch the video a. Episode 9 Munchausen by Proxy Dying fo1. Watch the video a. Episode 9 Munchausen by Proxy Dying fo
1. Watch the video a. Episode 9 Munchausen by Proxy Dying fo
BenitoSumpter862
 
1. Watch the video a. Episode 9 Munchausen by Proxy Dying fo
1. Watch the video a. Episode 9 Munchausen by Proxy Dying fo1. Watch the video a. Episode 9 Munchausen by Proxy Dying fo
1. Watch the video a. Episode 9 Munchausen by Proxy Dying fo
SantosConleyha
 
Process Flow and Narrative for Agile
Process Flow and Narrative for AgileProcess Flow and Narrative for Agile
Process Flow and Narrative for Agile
Glen Alleman
 
Can you trust your APIs - White Paper on API Testing
Can you trust your APIs - White Paper on API TestingCan you trust your APIs - White Paper on API Testing
Can you trust your APIs - White Paper on API Testing
Swetha Sridharan
 
Thailand SPIN Day 2014: มิตร ศัตรู หรือความไม่รู้ต่างหากที่หลอกเรา (29/5/2557...
Thailand SPIN Day 2014: มิตร ศัตรู หรือความไม่รู้ต่างหากที่หลอกเรา (29/5/2557...Thailand SPIN Day 2014: มิตร ศัตรู หรือความไม่รู้ต่างหากที่หลอกเรา (29/5/2557...
Thailand SPIN Day 2014: มิตร ศัตรู หรือความไม่รู้ต่างหากที่หลอกเรา (29/5/2557...
Apple Taton
 
How To Review The Sprints Efficiently
How To Review The Sprints EfficientlyHow To Review The Sprints Efficiently
How To Review The Sprints Efficiently
Lemi Orhan Ergin
 
20200205 DHS Agile Center of Excellence Agile Governance Ariel Partners
20200205 DHS Agile Center of Excellence Agile Governance Ariel Partners20200205 DHS Agile Center of Excellence Agile Governance Ariel Partners
20200205 DHS Agile Center of Excellence Agile Governance Ariel Partners
Craeg Strong
 
Setting up the program for EVM Compliant Validation
Setting up the program for EVM Compliant ValidationSetting up the program for EVM Compliant Validation
Setting up the program for EVM Compliant Validation
Glen Alleman
 
Accelerating Your Digital Agenda with Continuous Testing by Diego Lo Giudice
Accelerating Your Digital Agenda with Continuous Testing by Diego Lo GiudiceAccelerating Your Digital Agenda with Continuous Testing by Diego Lo Giudice
Accelerating Your Digital Agenda with Continuous Testing by Diego Lo Giudice
Sauce Labs
 

Similar to [SeleniumConf2019 Tokyo] All is thanks to you, selenium (20)

Agile methodology
Agile methodologyAgile methodology
Agile methodology
 
Building Performance Testing into the Development Process
Building Performance Testing into the Development ProcessBuilding Performance Testing into the Development Process
Building Performance Testing into the Development Process
 
ENT206 Product Development in the Cloud
ENT206 Product Development in the CloudENT206 Product Development in the Cloud
ENT206 Product Development in the Cloud
 
Product Development in the Cloud
Product Development in the Cloud Product Development in the Cloud
Product Development in the Cloud
 
QUES#19 Automation and Quality 2022.pdf
QUES#19 Automation and Quality 2022.pdfQUES#19 Automation and Quality 2022.pdf
QUES#19 Automation and Quality 2022.pdf
 
Amazon CI/CD Practices for Software Development Teams - SRV320 - Anaheim AWS ...
Amazon CI/CD Practices for Software Development Teams - SRV320 - Anaheim AWS ...Amazon CI/CD Practices for Software Development Teams - SRV320 - Anaheim AWS ...
Amazon CI/CD Practices for Software Development Teams - SRV320 - Anaheim AWS ...
 
Agile-plus-DevOps Testing for Packaged Applications
Agile-plus-DevOps Testing for Packaged ApplicationsAgile-plus-DevOps Testing for Packaged Applications
Agile-plus-DevOps Testing for Packaged Applications
 
Amazon CI/CD Practices for Software Development Teams - SRV320 - Atlanta AWS ...
Amazon CI/CD Practices for Software Development Teams - SRV320 - Atlanta AWS ...Amazon CI/CD Practices for Software Development Teams - SRV320 - Atlanta AWS ...
Amazon CI/CD Practices for Software Development Teams - SRV320 - Atlanta AWS ...
 
implementing_ai_for_improved_performance_testing_the_key_to_success.pdf
implementing_ai_for_improved_performance_testing_the_key_to_success.pdfimplementing_ai_for_improved_performance_testing_the_key_to_success.pdf
implementing_ai_for_improved_performance_testing_the_key_to_success.pdf
 
Amazon CI/CD Practices for Software Development Teams - SRV320 - Chicago AWS ...
Amazon CI/CD Practices for Software Development Teams - SRV320 - Chicago AWS ...Amazon CI/CD Practices for Software Development Teams - SRV320 - Chicago AWS ...
Amazon CI/CD Practices for Software Development Teams - SRV320 - Chicago AWS ...
 
Automate you Appium test like a pro!
Automate you Appium test like a pro!Automate you Appium test like a pro!
Automate you Appium test like a pro!
 
1. Watch the video a. Episode 9 Munchausen by Proxy Dying fo
1. Watch the video a. Episode 9 Munchausen by Proxy Dying fo1. Watch the video a. Episode 9 Munchausen by Proxy Dying fo
1. Watch the video a. Episode 9 Munchausen by Proxy Dying fo
 
1. Watch the video a. Episode 9 Munchausen by Proxy Dying fo
1. Watch the video a. Episode 9 Munchausen by Proxy Dying fo1. Watch the video a. Episode 9 Munchausen by Proxy Dying fo
1. Watch the video a. Episode 9 Munchausen by Proxy Dying fo
 
Process Flow and Narrative for Agile
Process Flow and Narrative for AgileProcess Flow and Narrative for Agile
Process Flow and Narrative for Agile
 
Can you trust your APIs - White Paper on API Testing
Can you trust your APIs - White Paper on API TestingCan you trust your APIs - White Paper on API Testing
Can you trust your APIs - White Paper on API Testing
 
Thailand SPIN Day 2014: มิตร ศัตรู หรือความไม่รู้ต่างหากที่หลอกเรา (29/5/2557...
Thailand SPIN Day 2014: มิตร ศัตรู หรือความไม่รู้ต่างหากที่หลอกเรา (29/5/2557...Thailand SPIN Day 2014: มิตร ศัตรู หรือความไม่รู้ต่างหากที่หลอกเรา (29/5/2557...
Thailand SPIN Day 2014: มิตร ศัตรู หรือความไม่รู้ต่างหากที่หลอกเรา (29/5/2557...
 
How To Review The Sprints Efficiently
How To Review The Sprints EfficientlyHow To Review The Sprints Efficiently
How To Review The Sprints Efficiently
 
20200205 DHS Agile Center of Excellence Agile Governance Ariel Partners
20200205 DHS Agile Center of Excellence Agile Governance Ariel Partners20200205 DHS Agile Center of Excellence Agile Governance Ariel Partners
20200205 DHS Agile Center of Excellence Agile Governance Ariel Partners
 
Setting up the program for EVM Compliant Validation
Setting up the program for EVM Compliant ValidationSetting up the program for EVM Compliant Validation
Setting up the program for EVM Compliant Validation
 
Accelerating Your Digital Agenda with Continuous Testing by Diego Lo Giudice
Accelerating Your Digital Agenda with Continuous Testing by Diego Lo GiudiceAccelerating Your Digital Agenda with Continuous Testing by Diego Lo Giudice
Accelerating Your Digital Agenda with Continuous Testing by Diego Lo Giudice
 

More from Woohyeok Kim

[Devopsdays2021] Roll Your Product with Kaizen Culture
[Devopsdays2021] Roll Your Product with Kaizen Culture[Devopsdays2021] Roll Your Product with Kaizen Culture
[Devopsdays2021] Roll Your Product with Kaizen Culture
Woohyeok Kim
 
[Regional Scrum Gathering Tokyo 2021] Scrum with OODA loop
[Regional Scrum Gathering Tokyo 2021] Scrum with OODA loop[Regional Scrum Gathering Tokyo 2021] Scrum with OODA loop
[Regional Scrum Gathering Tokyo 2021] Scrum with OODA loop
Woohyeok Kim
 
[Scrum Fest Mikawa] Successful Scrum Migration
[Scrum Fest Mikawa] Successful Scrum Migration[Scrum Fest Mikawa] Successful Scrum Migration
[Scrum Fest Mikawa] Successful Scrum Migration
Woohyeok Kim
 
Developers Summit 2020 Kansai : Post Corona
Developers Summit 2020 Kansai : Post CoronaDevelopers Summit 2020 Kansai : Post Corona
Developers Summit 2020 Kansai : Post Corona
Woohyeok Kim
 
[DevRelCon Earth 2020] Developers, be the evangelist
[DevRelCon Earth 2020] Developers, be the evangelist[DevRelCon Earth 2020] Developers, be the evangelist
[DevRelCon Earth 2020] Developers, be the evangelist
Woohyeok Kim
 
[Japan Selenium User Community in Remote] Test Automation Journey
[Japan Selenium User Community in Remote] Test Automation Journey[Japan Selenium User Community in Remote] Test Automation Journey
[Japan Selenium User Community in Remote] Test Automation Journey
Woohyeok Kim
 
[Japan Selenium User Community in Remote] Intro
[Japan Selenium User Community in Remote] Intro[Japan Selenium User Community in Remote] Intro
[Japan Selenium User Community in Remote] Intro
Woohyeok Kim
 
[#RSGT2020] SCRUM X ARMY
[#RSGT2020] SCRUM X ARMY[#RSGT2020] SCRUM X ARMY
[#RSGT2020] SCRUM X ARMY
Woohyeok Kim
 
[Selenium kansai] Intro
[Selenium kansai] Intro[Selenium kansai] Intro
[Selenium kansai] Intro
Woohyeok Kim
 
[Selenium] Effective ROI for test automation
[Selenium] Effective ROI for test automation[Selenium] Effective ROI for test automation
[Selenium] Effective ROI for test automation
Woohyeok Kim
 
[Rakuten Technology Conference 2019] Be the central on your field
[Rakuten Technology Conference 2019] Be the central on your field[Rakuten Technology Conference 2019] Be the central on your field
[Rakuten Technology Conference 2019] Be the central on your field
Woohyeok Kim
 
[DevRel/Japan Conf] Empower brand power of your community
[DevRel/Japan Conf] Empower brand power of your community[DevRel/Japan Conf] Empower brand power of your community
[DevRel/Japan Conf] Empower brand power of your community
Woohyeok Kim
 
UX with google amp for WebDesigner
UX with google amp for WebDesignerUX with google amp for WebDesigner
UX with google amp for WebDesigner
Woohyeok Kim
 
[E2E Test AutomationDay2019] intro
[E2E Test AutomationDay2019] intro[E2E Test AutomationDay2019] intro
[E2E Test AutomationDay2019] intro
Woohyeok Kim
 
[KaizenDiary] Minimize processing time on Automated Test
[KaizenDiary] Minimize processing time on Automated Test[KaizenDiary] Minimize processing time on Automated Test
[KaizenDiary] Minimize processing time on Automated Test
Woohyeok Kim
 

More from Woohyeok Kim (15)

[Devopsdays2021] Roll Your Product with Kaizen Culture
[Devopsdays2021] Roll Your Product with Kaizen Culture[Devopsdays2021] Roll Your Product with Kaizen Culture
[Devopsdays2021] Roll Your Product with Kaizen Culture
 
[Regional Scrum Gathering Tokyo 2021] Scrum with OODA loop
[Regional Scrum Gathering Tokyo 2021] Scrum with OODA loop[Regional Scrum Gathering Tokyo 2021] Scrum with OODA loop
[Regional Scrum Gathering Tokyo 2021] Scrum with OODA loop
 
[Scrum Fest Mikawa] Successful Scrum Migration
[Scrum Fest Mikawa] Successful Scrum Migration[Scrum Fest Mikawa] Successful Scrum Migration
[Scrum Fest Mikawa] Successful Scrum Migration
 
Developers Summit 2020 Kansai : Post Corona
Developers Summit 2020 Kansai : Post CoronaDevelopers Summit 2020 Kansai : Post Corona
Developers Summit 2020 Kansai : Post Corona
 
[DevRelCon Earth 2020] Developers, be the evangelist
[DevRelCon Earth 2020] Developers, be the evangelist[DevRelCon Earth 2020] Developers, be the evangelist
[DevRelCon Earth 2020] Developers, be the evangelist
 
[Japan Selenium User Community in Remote] Test Automation Journey
[Japan Selenium User Community in Remote] Test Automation Journey[Japan Selenium User Community in Remote] Test Automation Journey
[Japan Selenium User Community in Remote] Test Automation Journey
 
[Japan Selenium User Community in Remote] Intro
[Japan Selenium User Community in Remote] Intro[Japan Selenium User Community in Remote] Intro
[Japan Selenium User Community in Remote] Intro
 
[#RSGT2020] SCRUM X ARMY
[#RSGT2020] SCRUM X ARMY[#RSGT2020] SCRUM X ARMY
[#RSGT2020] SCRUM X ARMY
 
[Selenium kansai] Intro
[Selenium kansai] Intro[Selenium kansai] Intro
[Selenium kansai] Intro
 
[Selenium] Effective ROI for test automation
[Selenium] Effective ROI for test automation[Selenium] Effective ROI for test automation
[Selenium] Effective ROI for test automation
 
[Rakuten Technology Conference 2019] Be the central on your field
[Rakuten Technology Conference 2019] Be the central on your field[Rakuten Technology Conference 2019] Be the central on your field
[Rakuten Technology Conference 2019] Be the central on your field
 
[DevRel/Japan Conf] Empower brand power of your community
[DevRel/Japan Conf] Empower brand power of your community[DevRel/Japan Conf] Empower brand power of your community
[DevRel/Japan Conf] Empower brand power of your community
 
UX with google amp for WebDesigner
UX with google amp for WebDesignerUX with google amp for WebDesigner
UX with google amp for WebDesigner
 
[E2E Test AutomationDay2019] intro
[E2E Test AutomationDay2019] intro[E2E Test AutomationDay2019] intro
[E2E Test AutomationDay2019] intro
 
[KaizenDiary] Minimize processing time on Automated Test
[KaizenDiary] Minimize processing time on Automated Test[KaizenDiary] Minimize processing time on Automated Test
[KaizenDiary] Minimize processing time on Automated Test
 

Recently uploaded

LIGA(E)11111111111111111111111111111111111111111.ppt
LIGA(E)11111111111111111111111111111111111111111.pptLIGA(E)11111111111111111111111111111111111111111.ppt
LIGA(E)11111111111111111111111111111111111111111.ppt
ssuser9bd3ba
 
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...
Dr.Costas Sachpazis
 
Democratizing Fuzzing at Scale by Abhishek Arya
Democratizing Fuzzing at Scale by Abhishek AryaDemocratizing Fuzzing at Scale by Abhishek Arya
Democratizing Fuzzing at Scale by Abhishek Arya
abh.arya
 
CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptx
CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptxCFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptx
CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptx
R&R Consult
 
Top 10 Oil and Gas Projects in Saudi Arabia 2024.pdf
Top 10 Oil and Gas Projects in Saudi Arabia 2024.pdfTop 10 Oil and Gas Projects in Saudi Arabia 2024.pdf
Top 10 Oil and Gas Projects in Saudi Arabia 2024.pdf
Teleport Manpower Consultant
 
The Benefits and Techniques of Trenchless Pipe Repair.pdf
The Benefits and Techniques of Trenchless Pipe Repair.pdfThe Benefits and Techniques of Trenchless Pipe Repair.pdf
The Benefits and Techniques of Trenchless Pipe Repair.pdf
Pipe Restoration Solutions
 
Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...
Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...
Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...
AJAYKUMARPUND1
 
Student information management system project report ii.pdf
Student information management system project report ii.pdfStudent information management system project report ii.pdf
Student information management system project report ii.pdf
Kamal Acharya
 
Cosmetic shop management system project report.pdf
Cosmetic shop management system project report.pdfCosmetic shop management system project report.pdf
Cosmetic shop management system project report.pdf
Kamal Acharya
 
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdfHybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
fxintegritypublishin
 
COLLEGE BUS MANAGEMENT SYSTEM PROJECT REPORT.pdf
COLLEGE BUS MANAGEMENT SYSTEM PROJECT REPORT.pdfCOLLEGE BUS MANAGEMENT SYSTEM PROJECT REPORT.pdf
COLLEGE BUS MANAGEMENT SYSTEM PROJECT REPORT.pdf
Kamal Acharya
 
Immunizing Image Classifiers Against Localized Adversary Attacks
Immunizing Image Classifiers Against Localized Adversary AttacksImmunizing Image Classifiers Against Localized Adversary Attacks
Immunizing Image Classifiers Against Localized Adversary Attacks
gerogepatton
 
power quality voltage fluctuation UNIT - I.pptx
power quality voltage fluctuation UNIT - I.pptxpower quality voltage fluctuation UNIT - I.pptx
power quality voltage fluctuation UNIT - I.pptx
ViniHema
 
在线办理(ANU毕业证书)澳洲国立大学毕业证录取通知书一模一样
在线办理(ANU毕业证书)澳洲国立大学毕业证录取通知书一模一样在线办理(ANU毕业证书)澳洲国立大学毕业证录取通知书一模一样
在线办理(ANU毕业证书)澳洲国立大学毕业证录取通知书一模一样
obonagu
 
Vaccine management system project report documentation..pdf
Vaccine management system project report documentation..pdfVaccine management system project report documentation..pdf
Vaccine management system project report documentation..pdf
Kamal Acharya
 
Courier management system project report.pdf
Courier management system project report.pdfCourier management system project report.pdf
Courier management system project report.pdf
Kamal Acharya
 
Automobile Management System Project Report.pdf
Automobile Management System Project Report.pdfAutomobile Management System Project Report.pdf
Automobile Management System Project Report.pdf
Kamal Acharya
 
Event Management System Vb Net Project Report.pdf
Event Management System Vb Net  Project Report.pdfEvent Management System Vb Net  Project Report.pdf
Event Management System Vb Net Project Report.pdf
Kamal Acharya
 
The role of big data in decision making.
The role of big data in decision making.The role of big data in decision making.
The role of big data in decision making.
ankuprajapati0525
 
TECHNICAL TRAINING MANUAL GENERAL FAMILIARIZATION COURSE
TECHNICAL TRAINING MANUAL   GENERAL FAMILIARIZATION COURSETECHNICAL TRAINING MANUAL   GENERAL FAMILIARIZATION COURSE
TECHNICAL TRAINING MANUAL GENERAL FAMILIARIZATION COURSE
DuvanRamosGarzon1
 

Recently uploaded (20)

LIGA(E)11111111111111111111111111111111111111111.ppt
LIGA(E)11111111111111111111111111111111111111111.pptLIGA(E)11111111111111111111111111111111111111111.ppt
LIGA(E)11111111111111111111111111111111111111111.ppt
 
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...
 
Democratizing Fuzzing at Scale by Abhishek Arya
Democratizing Fuzzing at Scale by Abhishek AryaDemocratizing Fuzzing at Scale by Abhishek Arya
Democratizing Fuzzing at Scale by Abhishek Arya
 
CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptx
CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptxCFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptx
CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptx
 
Top 10 Oil and Gas Projects in Saudi Arabia 2024.pdf
Top 10 Oil and Gas Projects in Saudi Arabia 2024.pdfTop 10 Oil and Gas Projects in Saudi Arabia 2024.pdf
Top 10 Oil and Gas Projects in Saudi Arabia 2024.pdf
 
The Benefits and Techniques of Trenchless Pipe Repair.pdf
The Benefits and Techniques of Trenchless Pipe Repair.pdfThe Benefits and Techniques of Trenchless Pipe Repair.pdf
The Benefits and Techniques of Trenchless Pipe Repair.pdf
 
Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...
Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...
Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...
 
Student information management system project report ii.pdf
Student information management system project report ii.pdfStudent information management system project report ii.pdf
Student information management system project report ii.pdf
 
Cosmetic shop management system project report.pdf
Cosmetic shop management system project report.pdfCosmetic shop management system project report.pdf
Cosmetic shop management system project report.pdf
 
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdfHybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
 
COLLEGE BUS MANAGEMENT SYSTEM PROJECT REPORT.pdf
COLLEGE BUS MANAGEMENT SYSTEM PROJECT REPORT.pdfCOLLEGE BUS MANAGEMENT SYSTEM PROJECT REPORT.pdf
COLLEGE BUS MANAGEMENT SYSTEM PROJECT REPORT.pdf
 
Immunizing Image Classifiers Against Localized Adversary Attacks
Immunizing Image Classifiers Against Localized Adversary AttacksImmunizing Image Classifiers Against Localized Adversary Attacks
Immunizing Image Classifiers Against Localized Adversary Attacks
 
power quality voltage fluctuation UNIT - I.pptx
power quality voltage fluctuation UNIT - I.pptxpower quality voltage fluctuation UNIT - I.pptx
power quality voltage fluctuation UNIT - I.pptx
 
在线办理(ANU毕业证书)澳洲国立大学毕业证录取通知书一模一样
在线办理(ANU毕业证书)澳洲国立大学毕业证录取通知书一模一样在线办理(ANU毕业证书)澳洲国立大学毕业证录取通知书一模一样
在线办理(ANU毕业证书)澳洲国立大学毕业证录取通知书一模一样
 
Vaccine management system project report documentation..pdf
Vaccine management system project report documentation..pdfVaccine management system project report documentation..pdf
Vaccine management system project report documentation..pdf
 
Courier management system project report.pdf
Courier management system project report.pdfCourier management system project report.pdf
Courier management system project report.pdf
 
Automobile Management System Project Report.pdf
Automobile Management System Project Report.pdfAutomobile Management System Project Report.pdf
Automobile Management System Project Report.pdf
 
Event Management System Vb Net Project Report.pdf
Event Management System Vb Net  Project Report.pdfEvent Management System Vb Net  Project Report.pdf
Event Management System Vb Net Project Report.pdf
 
The role of big data in decision making.
The role of big data in decision making.The role of big data in decision making.
The role of big data in decision making.
 
TECHNICAL TRAINING MANUAL GENERAL FAMILIARIZATION COURSE
TECHNICAL TRAINING MANUAL   GENERAL FAMILIARIZATION COURSETECHNICAL TRAINING MANUAL   GENERAL FAMILIARIZATION COURSE
TECHNICAL TRAINING MANUAL GENERAL FAMILIARIZATION COURSE
 

[SeleniumConf2019 Tokyo] All is thanks to you, selenium

  • 1. All is thanks to you, Selenium ありがとう、セレニウム April 18, 2019 Selenium Conf Tokyo 2019 Woohyeok Aaron Kim (@woosyume) EC Incubation Department Rakuten, Inc.
  • 2. 2 Woohyeok Kim # Software engineer at Rakuten # Worked in Japan since 2016 # Ex-Captain at Korea Army キム・ウヒョク
  • 4. 4 Woohyeok Kim Working at Ranking Frontend Team Experience of frontend project release. Driving Test Automation project # Software engineer at Rakuten # Worked in Japan since 2016 # Ex-Captain at Korea Army Enjoying to improve our product Google AMP Adoption Validating Module for Half-term Ranking Some kaizen of modules to improve the current things. Always pursue something キム・ウヒョク
  • 5. 5 Overview Ranking Service Intro ランキングサービスのご紹介 Background 背景 Motivation きっかけ What we’ve done やったこと Achievement 成果 Synergy 相乗効果 Conclusion まとめ
  • 7. 7 Aggregate sales data from Rakuten Ichiba. Calculate score for each item. Provide bestsellers information. What is Ranking Service ランキングサービスとは 1) Irastoya, 2019/04/03, https://www.irasutoya.com/ 2) Ranking Top, 2019/03/07, https://ranking.rakuten.co.jp 1) 1) 2)
  • 8. 8 Ranking Service System Structure Platform provides score data API provides fixed rank data Frontend provides web UISupports Other services ランキングサービスの仕組み
  • 10. 10 Our issues about UI Test Manual UI Test takes TOO MUCH time Bottle Neck UIテストについての悩み
  • 12. 12 Concern if we can implement FULLY-AUTOMATED test or not We need to check actually UNLIMITED cases Our issues about UI Test FASHION JACKET COLOR PANTS SIZE Duration Top Genre Genre REALTIME DAILY WEEKLY MONTHLY FOOD … … … 4 39 13,000 Tag 130,000 UIテストについての悩み
  • 13. 13 Concern about HIGH Cost for development Our issues about UI Test Frontend Application Test Application UIテストについての悩み
  • 16. 16 Opportunity for us to start test automation project テスト自動化のきっかけ 1) 1) Selenium HP, 2019/04/03, https://www.seleniumhq.org/projects/ 2) Ranking Top, 2019/03/07, https://ranking.rakuten.co.jp 2)
  • 17. 17 Expected ROI(Return On Investment) I proposed the following ROI and it was approved. Investment Return 4 MONTHS / PPL 1MONTH / PPL 期待した効果 1) pngtree, 2019/04/03, https://pngtree.com/so/approved 1)
  • 19. 19 User Behavior Check Click function Move to expected URL ユーザ行動の確認 1) 1) Ranking Top, 2019/04/03, https://ranking.rakuten.co.jp
  • 20. 20 Banner / Advertisement Check Contents itself Click function バナー・広告の確認 1) 1) Ranking Top, 2019/04/03, https://ranking.rakuten.co.jp
  • 21. 21 Data Integrity Check Ranking Result - Rank - Item Name - Shop Name - Price - Review# - Particular condition Update DateGenre Name ランキングデータ整合性の確認 1) 1) Ranking Top, 2019/04/03, https://ranking.rakuten.co.jp
  • 22. 22 Others External Resource Errors, AMP Syntax Errors その他の確認
  • 23. 23 Scope Additionally we have 2 more products in Scope. Smartphone Web Smartphone Web with AMP 実装スコープ 1) 2) 1) Ranking Top, 2019/03/07, https://ranking.rakuten.co.jp 2) Ranking Event, 2019/03/07, https://ranking.rakuten.co.jp/event/supersale
  • 24. 24 Create expected value dynamically For decreasing maintenance cost (automatic request creation for API) DatabaseRanking API Test Automation Application Ranking Front 入力値・期待値の自動生成 1) 1) Ranking Top, 2019/04/03, https://ranking.rakuten.co.jp
  • 25. 25 Image Comparison Increase test accuracy by checking if image is broken or not. Calculates difference with the previous execution Before After TEST FAILS 画像比較による文字化け・画面崩れの検知 1) 1) Ranking Event, 2019/04/03, https://ranking.rakuten.co.jp/event/anpanman/
  • 27. 27 Progress per Number of Tests Device Total Automated Coverage PC 180 153 85% Smartphone 180 72 40% Smartphone with AMP 90 27 30% Total 450 252 60% 自動化したテストケースの数
  • 28. 28 Actual ROI(Return On Investment) on current phase Investment Return Expected 4 MONTHS / PPL 1MONTH / PPL Actual 1.3MONTH / PPL So far 0.6MONTH+ / PPL 期待した効果と実際の効果
  • 29. 29 Cost Improvement per execution Device Before After Reduction PC 8hours 3hours 60% Smartphone 8hours 3hours 60% Smartphone with AMP 3hours 2hours 40% Total 19hours 8hours 59% テスト実行のコスト削減
  • 30. 30 Enhancing test accuracy : Assertion# 5000 0 Before 機械の力でテストの精度を高める After 6200+ 500
  • 33. 33 Kaizen initiatives rise 5 0 10 2017.09 ~ 2018.09 2018.10 ~ 2019.04 Increase chance 4x to provide user with GOOD SERVICE 改善チケットの消化率が高まる 12 6
  • 34. 34 Getting knowledge for product Need to understand current behavior Document Code Test Case プロダクトについての理解が深まる
  • 35. 35 Getting knowledge for product if ( genre has child ) ? Application プロダクトについての理解が深まる 1) 1) Ranking Top, 2019/04/03, https://ranking.rakuten.co.jp
  • 36. 36 Test Cost Reduction for the other frontend projects Myth : Test Automation Project IS JUST COSTS AMP Project jQuery Update Event Ranking for Smartphone Bookmark Improvement 他フロントリリースのコスト削減 Saved 37/ 76hours
  • 38. 38 Consider Future Cost Minimize future maintenance cost by creating expected value dynamically. Properties Input file Database API 将来の管理コストの考慮
  • 39. 39 Consider Future Cost Minimize future maintenance cost by creating expected value dynamically. Properties Input file Database API 将来の管理コストの考慮
  • 40. 40 Consider execution time It results from slow-test issue Optimizing processing time with @Category Be able to execute necessary test cases only beyond packages. Package A Package B ▶ Test A ▶ Test B ▶ Test C ▶ Test D 実行時間の考慮
  • 41. 41 Manage version matching with the object Update features continuously By inputting test automation application into project scope Estimation Consider if creating new test feature is necessary or not Development Implement new test feature at the same time of implementing unit test Regression Test Test newly updated module with the latest version of automation app Release Merge it to master branch after confirming the test works on PROD also テスト対象とバージョンを合わせる
  • 42. 42 Let it be easy to analyze the root cause when test fails Let’s think about the reason why we executes test Provide user good product by finding and fixing bugs Suite Class Method Fail Message / Debug Log Fail Message Debug Log 失敗の原因を明確にする
  • 44. 44 Blueprint Execute Automatically at every commit For zero cost 未来図
  • 46. 46 Conclusive Thoughts TRY Selenium Prioritize the most valuable parts for your business Selenium can change more than we thought Foster Team Culture for service improvement 提案 Sales Resource * Pareto Rule 20% INPUT accounts for 80% OUTPUT Feature A 20% Test Cases Feature A 70%+B 6.47% C 7.17% D 9.50% Sales

Editor's Notes

  1. 3분 주로 비용 대 코스트, 시너지
  2. 30초 3:30
  3. 1분 30초 5:00 생명이 걸린 걸 경험했기 때문에 모든 리스크는 일단 피하고 보자
  4. 40초 5:40 1) 현재 라쿠텐 랭킹서비스의 프론트엔드 팀에서 테스트 자동화 프로젝트를 드라이브 2) 구글 AMP기술을 이용한 페이지 로드 개선과 같이 프로덕트 개선으로 이어지는 무언가에 항상 관심을 가지고 있습니다.
  5. 1분 30초 7:10 등록상품 수 퀴즈 표시 페이지, 표시 값
  6. 1분 8:10 랭킹 서비스의 목적은 현재 트렌드를 분석하고 데이터를 제공함으로써 고객분들이 라쿠텐 이치바에서 보다 쇼핑을 즐길 수 있도록 공헌. 2억개가 넘은 상품정보와 같은 빅데이터를 통해 랭킹을 계산하여 실시간, 일간, 주간, 월간 그리고 반기, 연간 랭킹 등을 제공하고 있습니다.
  7. 1분 9:10 랭킹을 산정하는 기본 플랫폼이 있고, 그 정보들을 내부 또는 외부에 제공하기 위한 API 그리고 자동화 테스트의 대상인 프론트엔드 어플리케이션이 있습니다. 데이터량이 많기 때문에 hbase와 같은 대규모 분산형 데이터베이스나 redis 등의 오픈소스를 사용하고 있습니다 랭킹 인트로까지를 10분으로 하자.
  8. 배경이 총 3분 30초 12:40
  9. 도입하기까지 몇 가지 고민이 있었다. 시간이 너무 걸린다는 것 테스트할 대상은 많은데 엔지니어 리소스는 한정 테스트가 19시간 걸린다는 것도 문제이지만 더 문제인건 그것으로 인해 …今回はやめよう
  10. 또한 자동화를 한다고 해도 그것을 어떻게 완벽하게 자동화 할 수 있을지가 문제였습니다. 체크할 케이스가 너무나도 많았습니다. 서비스 개념 설명 몇천만 정도 될 줄 알았는데 모지바케가 있어도 캐치하지 못하는 경우가 발생.
  11. 개발기간에 대한 코스트도 문제였습니다. 프론트엔드 어플리케이션의 모든 behavior를 커버해야하는 만큼 테스트 어플리케이션도 규모가 커지게 되는데 그에 대한 코스트를 얼마나 감당할 수 있을지.
  12. 1:30 14:10 いつまでもこう言うボトルネックを追うとくのはできませんでした。
  13. https://www.seleniumhq.org/projects/ 저희 메인 어플리케이션이 자바이기 때문에 자바 언어로의 개발을 서포트하고 유저 behavior에 기반한 테스트를 서포트하는 셀레늄 그 중에서도 웹드라이버를 선정해서 HTTP Status 코드나 페이지별 스크린샷을 찍는 프로토타입을 만들어 봤습니다. 해보고 이거라면 자동화가 가능할 수도 있겠다 라는 생각을 했습니다.
  14. 1분 30초 모티베이션까지 10분 컷! https://pngtree.com/so/approved 프로토 타입으로 쇼케이스를 한 후 ROI를 제시하여 정식으로 프로젝트를 진행하게 되었f습니다. 그렇게 하면 2년 정도 지나면 제대로 퍼포먼스를 낼 수 있겠다 싶었습니다.
  15. 3:00 17:10 5:00 22:10 2:00 24:10
  16. 첫 번째는 유저행동 확인
  17. 두 번째는 배너 광고 확인
  18. 데이터 정합성 확인
  19. 그 외
  20. 짓소 내용 3분 기타 스코프
  21. 5분 테스트의 기본은 기대치가 실제 값을 비교하는 것이죠. 테스트 자동화에서 가장 고민을 많이 했던 부분이 바로 이 부분인데요 두 가지의 고민을 했습니다. 1. 앞서 말씀드린 테스트 범위가 방대하다는 점 이것이 무엇을 의미하냐면 표시할 페이지가 많다는 건 즉, 들어가서 확인할 URL이 많다. 그만큼 표시할 값이 많다. 그건 또 기대치가 많다. 2. 유지보수 코스트 테스트 자동화 어플리케이션이 초기 1, 2년에는 잘 사용되다가 결국에 버려지는 이유가 1) 복잡한 구조 2) 기대되는 값의 변경 에 따른 유지보스 코스트가 어마어마하게 발생할 수 있기 때문. 이것들을 해결하기 위해 어떻게 하면 한 번만 만들면 그 뒤에 손을 대지 않아도 될까. 해답은 모든 값을 동적으로 생성하는 것이었습니다. Genre id, test url, api query 등이 그 예. 그래서 테스트 실행 시 모든 입력치가 자동으로 생성되는데 이 데이터들은 실행마다 변하게 됩니다. 앞서 프론트엔드 어플리케이션과 거동을 맞출 필요가 있다고 했는데 그에 따라 어떤 값이 표시되는지, 또 어떤 조건 하에 그 값이 표시되는지 등을 재현하여 기대치를 다이나믹해서 재구성합니다. 이 때 프론트어플리케이션과 동일한 값을 얻을 수 있도록 하기 위해 실제 사용하는 API, Database와의 연결 그래서 STG, PROD 어디서든 동적인 기대치로 정해진 테스트를 수행할 수 있게 했습니다. UI리뉴얼 등의 경우에 대해서도 웹 엘리먼트만의 수정을 통해 최대한 낮은 코스트로 대응할 수 있게 했습니다.
  22. 2분 또 하나는 화면 비교를 통한 1위부터 80위까지 표시 만약 밑에서 모지바케가 발생한 경우 이를 찾아내기가 쉽지 않다. 1프로라도 오차가 발생할 시 %는 허용범위 일상적으로 엔지니어가 발견하기 어려운 것들을 대신 발견할 수 있도록. 엔지니어의 시력, 노안, 컨디션
  23. 성과 4분 컷 27
  24. 지금 이걸로 뭐하자는거야? 완성도 안됐잖아?
  25. 다른 안건들이 있기 때문에 이 프로젝트를 빈 시간 등을 활용해 진행해 나갔습니다만 절반도 안되는 투자에도 지금 시점에 이미 절반 이상의 리턴
  26. 또 단순히 기존 테스트케이스를 대체한 것 뿐만 아니라 더 많은 assertion으로 인간의 힘으로는 어려운 더 깊고 코마카이한 테스트가 가능하게 되었다. 예를 들어 지금까지 랭킹 데이터 정합성 체크를 한다고 하면 1위 21위 80 이런 식으로 확인을 했다면 기계의 힘을 빌려 1~80위까지 상품 이름이나 점포명 등을 모두 확인 현 시점에 약 이 정도이고, 10000
  27. 재미있는 건 이 뿐만이 아니라는 겁니다. 지금까지 말한 것은 사실은 예상이 되었던 결과입니다. 프로덕트를 확인하거나, 실패를 빨리 검지하거나 하는 것은 테스트 자동화의 일반적인 메리트로서 일컬어지고 있는 부분이기 때문입니다.
  28. 사실은 지금까지 말한 것, 숫자 같은 것 모두 잊으셔도 됩니다. 제가 말 하고 싶은 내용은 다 지금부터입니다.
  29. 라쿠텐, 특히 엔지니어들은 두 가지 업무 경로를 가지고 있습니다. 하나는 일반적으로 비즈니스 리퀘스트로서 정해진 안건, 예를 들어 UI리뉴얼, 챗봇 서비스 등 유저에게 직접적으로 영향을 미치는 프로젝트가 있고 또 하나는 우리 엔지니어가 직접 기획하는 릴리즈인데 그것이 유저에게 직접 영향을 미치지 않더라도 어쨌든 서비스를 개선하는 프로젝트, 예를 들면 jQuery 업데이트라든가, 당장 표시되는 값이 바뀌지 않는다 하더라도 하드코딩 되어있는 값을 동적으로 바꾼다든가 이 후자에 대해서 항상 테스트가 걸림돌이 되어 실행하지 못하는 경우가 많았는데 지금은 그렇지가 않습니다. 전에는 아 테스트 때문에 이번에는 조금 여유가 없을 것 같다 하던 생각이 테스트는 뭐 자동화 어플리케이션 있으니까 괜찮지 않나 하는 그런 이야기를 들으면 개인적으로 굉장히 기쁩니다. 소화한 프로젝트 수 2배로 보이는데 사실은 4배. 팀의 문화가 바뀌는거에요. 단순히 테스트를 자동화하는 것 뿐만 아니라 서비스 개선을 위한 팀원의 마인드가 마에므키하게 변했다는 것이 굉장히 의의를 가지고 있다고 생각합니다. 이게 서비스를 운영하는 여러분들이 가장 가지고싶어하는 가치 아닐까요.
  30. 테스트를 하기 위해서 또 기대치를 동적으로 생성하기 위해서는 테스트 대상과 완전히 거동을 일치시켜야 하기 때문에 조사에 꽤 많은 시간을 할애했다. ‘프로그래밍 할 땐 스스로 컴파일러가 되어라.’ 어떤 조건 아래에 어떤 값이 표시되는지, 외부 리소스를 어떻게 참조하고 있는지 등. Document, 코드, 기존 테스트케이스 내가 웹 어플리케이션이 된 줄 알았다.
  31. 하면서 어떻게 하면 가장 효율적인 짓소가 될 지 팀원과 상의를 하는 과정에서 서로 아 ~ 이런 게 있었어? 하며 프로덕트에 대한 지식이 성장. 일반적으로 신입이 들어오면 도큐먼트 작업을 부탁하는 일이 많은데 제가 매니저가 되면 테스트 자동화를 부탁해볼까 하고도 생각합니다.
  32. 35 ~ 38 : 10분 이런 부분이 실제 ROI에 반영되어있지 않기 때문에 상승효과로서 우리는 셀레늄 테스트 자동화를 통해 더 많은 것을 기대할 수 있다는 것입니다.
  33. Dynamic Data source データ元 홈페이지 자체가 리뉴얼 되는 것은 어쩔 수 없지만 라벨의 변화, 동적 데이터의 변화
  34. Dynamic Data source データ元 홈페이지 자체가 리뉴얼 되는 것은 어쩔 수 없지만 라벨의 변화, 동적 데이터의 변화
  35. 동작을 자동화 할 뿐 페이지 로드, 네트워크 등의 속도를 개선해주지는 않기 때문에 수백개의 케이스를 실행하면 반드시 슬로우 테스트 이슈가 발생합니다. 패키지의 경계를 넘어서 필요한 테스트만을 실행할 수 있도록 단지 지금 단계에서 이렇게 하고 있을 뿐 지금은 단일 서버에서 실행하고 있기 때문에 카테고리를 사용하고 있지만 카테고리별로 각 서버로 실행을 분산시키는 것이 목표
  36. 피쳐를 지속적으로 업데이트 해서 테스트 대상의 어플리케이션과 버전을 揃えること 그렇게 하지 않으면 이전 버전을 테스트하거나 할 때 쓸데 없는(余計な) 테스트 실패가 발생할 수 있다. 미츠모리 단계에서부터.
  37. 테스트의 궁극적인 목적 실패의 발견이 아니라 실패의 원인을 규명하고 해결하는 것 이 메소드가 실패는 해 있는데 어디가 문제인지 모르겠어 이렇게 되면 또 조사에 코스트가 발생하게 됩니다. 조사에 드는 코스트도 생각해서 로그를 알아보기 쉽게 쓰거나 혹은 디버그 로그를 활용하는 방법을 추천.
  38. Tips : 3분 분산 서버별 이미지 비교만을 위한 서버 유저 거동 확인을 위한 서버 등을 나누어 최종적으로는 실행시간으로부터 완전히 자유로울 수 있도록 하는 것이 목표.
  39. 결론 : 3분 셀레늄은 많은 것을 바꾸어 줍니다. 테스트를 자동화해주며, 안정적인 서비스 운영에 기여하며 무엇보다 서비스 개선을 위한 팀 문화가 정착이 됩니다. 일단은 트라이 해봅시다. 파레토 법칙 20%의 인기상품이 80%의 이익을 창출한다. 다음은 당신의 순서입니다. 좋은 이야기 들려주세요 제안