SlideShare a Scribd company logo
1 of 29
Download to read offline
Journey of CI/CD Pipeline
Improvement in Yahoo! JAPAN
Yahoo Japan Corporation
04/19/2019 @ Selenium Conf Tokyo
Teppei Yamaguchi
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
• Challenges we have faced in adaptation of company-wide
pipeline and automated tests in many people / multi-product
company
• 多人数・多プロダクトのある企業における、全社的なパイプラ
インの整備および自動テスト導入・普及の課題やその克服の流
れを紹介します。
Abstraction
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
Responsibility
• Transforming the whole company into “Agile” and “Test automation”
Career
• Programmer
• Software Engineering Researcher/Consultant
• Coach
Test Coach, Agile Coach
Yahoo Japan Corporation
About me
Teppei Yamaguchi
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
• Programmers & Designers: 3,000+
• Services: 100+ (Portal, News, Ad, Shopping, Auction, Mail, Weather, …)
• Service quality is the responsibility of the service department
• No company-wide QA process
• No QA department
• Members change teams often
Yahoo! JAPAN
13
Service A
Division
Product C
Division
Business/Designer/
Programmer etc.
Platform Service
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
• Total number of CI/CD pipelines
• Executions per day(all the pipelines in total)
The Current CI/CD Pipelines of Yahoo! JAPAN
40,000+
15,400+ [times]
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
The Current CI/CD Pipelines of Yahoo! JAPAN
Github
Enterprise
SCM Execution Platform
Screwdriver.cd
CI/CD Tools
Clone Deploy
Publish
Artifactory
Enterprise
OpenWhisk Kubernetes
Pivotal Application Service
Package Repository
npm
Enterprise
Docker
Registry
Install
Concourse E2E Test Tool
Applicat Selenium
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
Journey of CI/CD Pipeline Improvement
2013
1st Improvement
Project
2nd Improvement
Project
3rd Improvement
Project
20192014 2015 2016 2017 2018
Automated E2E Testing Support
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
1st Improvement Project
- Preparation for CI/CD implementation -
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Situation
• Manual testing, deployment etc. was standard for many teams
• Only a few advanced teams did CI/CD with Jenkins
▼Issue
• We couldn’t provide services and products continuously on the latest OS and
middleware
• Spend lots of time to support older OS and middleware
• Security risk
1st Improvement Project -Preparation for CI/CD implementation-
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Top-down: establish new strategy
• Clarify responsible groups of each module
• Latest OS
▼Bottom-up: support individuals / teams
• Jenkins support
1st Improvement Project -Preparation for CI/CD implementation-
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Problems in project
• High business goals, making it difficult to get resources for this work
▼Action for project problem solution
• Explain individually to team’s managers, over and over again
1st Improvement Project -Preparation for CI/CD implementation-
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Result
• Update to latest OS and clarify responsible groups was completed
• CI implementation itself hardly progressed
1st Improvement Project -Preparation for CI/CD implementation-
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
2nd Improvement Project
- CI/CD implementation -
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Situation
• Some teams used CI/CD with Jenkins
• The importance of CI/CD was recognized but many teams didn’t use CI/CD
▼Issue
• Cost to start using CI/CD was too high
• Fixing security issues in CI/CD environment was high cost
• Costs to move members between teams were high because all teams
customized their own CI/CD in silos and requires relearning
2nd Improvement Project -CI/CD implementation-
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
2nd Improvement Project -CI/CD implementation-
▼Top-down: establish new strategy
• Use of centralized pipeline tools(Screwdriver.cd, Concourse, Approduce)
• Write test code
• Explain to business managers for reserving CI/CD implementation resources
▼Bottom-up: support individuals / teams
• Provide Pipeline tool support
• Provide testing support with test automation
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Problems in project
• High preparation cost for automated E2E testing
• Difficult to automate testing for legacy systems
▼Action for project problem solution
• Started new project to support automated E2E testing
• Clarified the strategy so that automating tests for legacy can be postponed
2nd Improvement Project -CI/CD implementation-
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Result
• Automated build and automated deployment with pipeline tools became
common
• Couldn’t write test code for some legacy systems
2nd Improvement Project -CI/CD implementation-
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
Automated E2E Testing Support
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Situation
• Builds & deployments were executed with the pipelines, but automated E2E tests
were few
▼Issue
• No E2E test environment in the pipeline and members didn’t know how to build it
• Automated E2E tests of mobile app was not stable
• Maintaining mobile devices required for man-powered E2E testing was too effort
intensive
Automated E2E Testing Support
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Bottom-up: support individuals / teams
• Provide centralized E2E test environment & tool(based on Appium)

for mobile app using the pipeline
• Provide Selenium container
• Test automation support
• Pair test programming, Pair E2E test environment development
• Support incremental implementation of automated E2E test
Automated E2E Testing Support
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Result
• Execute automated E2E test in the pipeline
• Programmers got motivated for test automation
Still improving…
Automated E2E Testing Support
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
3rd Improvement Project
- CI/CD implement to legacy system -
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Situation
• Many teams did CI/CD with pipeline tools
• CI/CD couldn’t be done for legacy systems
▼Issue
• Cost to automate testing for legacy systems was too high
3rd Improvement Project -CI/CD implement to legacy system-
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Top-down: establish new strategy
• Refactoring from legacy systems to new testable systems
• Explain to executives for assign CI/CD implementation resources
▼Bottom-up: support individuals / teams
• Provide Pipeline tool support
• Provide testing support
3rd Improvement Project -CI/CD implement to legacy system-
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Result
• Ongoing
3rd Improvement Project -CI/CD implement to legacy system-
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
Conclusion
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
Journey of CI/CD Pipeline Improvement
Preparation
2013
1st Improvement
Project
CI/CD
implementation
2nd Improvement
Project
CI/CD implement to
legacy system
3rd Improvement
Project
20192014 2015 2016 2017 2018
Automated E2E Testing Support
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
• Implementation of company-wide CI/CD takes a long time
• There are many problems that can not be discovered without trying, 

because every teams are in various situations.
• Need continuous improvement
• Especially if you do something that does not go well
Points
"Journey of CI/CD Pipeline Improvement in Yahoo! JAPAN" @ #SeConfTokyo

More Related Content

What's hot

Vincent Spena - Agile and Lean Methods for Hardware Product Development
Vincent Spena - Agile and Lean Methods for Hardware Product DevelopmentVincent Spena - Agile and Lean Methods for Hardware Product Development
Vincent Spena - Agile and Lean Methods for Hardware Product Development
Vincent Spena
 
Introducing Jetbox
Introducing JetboxIntroducing Jetbox
Introducing Jetbox
lcraig333
 
IBM UrbanCode Deploy Quick Start Service Offering
IBM UrbanCode Deploy Quick Start Service OfferingIBM UrbanCode Deploy Quick Start Service Offering
IBM UrbanCode Deploy Quick Start Service Offering
IBM Rational software
 

What's hot (20)

DevOps, Databases and The Phoenix Project UGF4042 from OOW14
DevOps, Databases and The Phoenix Project UGF4042 from OOW14DevOps, Databases and The Phoenix Project UGF4042 from OOW14
DevOps, Databases and The Phoenix Project UGF4042 from OOW14
 
Adopting DevOps for 2-Speed IT
Adopting DevOps for 2-Speed ITAdopting DevOps for 2-Speed IT
Adopting DevOps for 2-Speed IT
 
Adapting Deployment Pipelines for Complex Applications
Adapting Deployment Pipelines for Complex ApplicationsAdapting Deployment Pipelines for Complex Applications
Adapting Deployment Pipelines for Complex Applications
 
A Continuous Delivery Safety Net for Databases
A Continuous Delivery Safety Net for DatabasesA Continuous Delivery Safety Net for Databases
A Continuous Delivery Safety Net for Databases
 
Platform Health Assessment at Department of Homeland Security Citizenship and...
Platform Health Assessment at Department of Homeland Security Citizenship and...Platform Health Assessment at Department of Homeland Security Citizenship and...
Platform Health Assessment at Department of Homeland Security Citizenship and...
 
Unicorns on an Aircraft Carrier: CDSummit London and Stockholm Keynote
Unicorns on an Aircraft Carrier: CDSummit London and Stockholm KeynoteUnicorns on an Aircraft Carrier: CDSummit London and Stockholm Keynote
Unicorns on an Aircraft Carrier: CDSummit London and Stockholm Keynote
 
Should You Break Up With Your Monolith?
Should You Break Up With Your Monolith?Should You Break Up With Your Monolith?
Should You Break Up With Your Monolith?
 
Death to Manual Deployments
Death to Manual DeploymentsDeath to Manual Deployments
Death to Manual Deployments
 
Vincent Spena - Agile and Lean Methods for Hardware Product Development
Vincent Spena - Agile and Lean Methods for Hardware Product DevelopmentVincent Spena - Agile and Lean Methods for Hardware Product Development
Vincent Spena - Agile and Lean Methods for Hardware Product Development
 
Improving Software Delivery with DevOps & Software Defined Environments | The...
Improving Software Delivery with DevOps & Software Defined Environments | The...Improving Software Delivery with DevOps & Software Defined Environments | The...
Improving Software Delivery with DevOps & Software Defined Environments | The...
 
Continous integration-leon-kehl-2010
Continous integration-leon-kehl-2010Continous integration-leon-kehl-2010
Continous integration-leon-kehl-2010
 
How ancestry used ai and ml for continuous, autonomous cloud optimization a...
How ancestry used ai and ml for continuous, autonomous cloud optimization   a...How ancestry used ai and ml for continuous, autonomous cloud optimization   a...
How ancestry used ai and ml for continuous, autonomous cloud optimization a...
 
Introducing Jetbox
Introducing JetboxIntroducing Jetbox
Introducing Jetbox
 
IBM UrbanCode Deploy Quick Start Service Offering
IBM UrbanCode Deploy Quick Start Service OfferingIBM UrbanCode Deploy Quick Start Service Offering
IBM UrbanCode Deploy Quick Start Service Offering
 
Continuous Delivery Maturity Model
Continuous Delivery Maturity ModelContinuous Delivery Maturity Model
Continuous Delivery Maturity Model
 
Accelerating the Developers' Road to the Cloud at Enterprise Scale
Accelerating the Developers' Road to the Cloud at Enterprise ScaleAccelerating the Developers' Road to the Cloud at Enterprise Scale
Accelerating the Developers' Road to the Cloud at Enterprise Scale
 
DevOps Thinking for the Line of Business
DevOps Thinking for the Line of BusinessDevOps Thinking for the Line of Business
DevOps Thinking for the Line of Business
 
IBM InterConnect 2016: Security for DevOps in an Enterprise
IBM InterConnect 2016: Security for DevOps in an Enterprise IBM InterConnect 2016: Security for DevOps in an Enterprise
IBM InterConnect 2016: Security for DevOps in an Enterprise
 
dev@InterConnect workshop - Lean and DevOps
dev@InterConnect workshop - Lean and DevOpsdev@InterConnect workshop - Lean and DevOps
dev@InterConnect workshop - Lean and DevOps
 
Path to Production: Value Stream Mapping in a DevOps World
Path to Production: Value Stream Mapping in a DevOps WorldPath to Production: Value Stream Mapping in a DevOps World
Path to Production: Value Stream Mapping in a DevOps World
 

Similar to "Journey of CI/CD Pipeline Improvement in Yahoo! JAPAN" @ #SeConfTokyo

Improving Quality through Continuous Integration - A case study of CollabNet
Improving Quality through Continuous Integration - A case study of CollabNetImproving Quality through Continuous Integration - A case study of CollabNet
Improving Quality through Continuous Integration - A case study of CollabNet
Venkat Janardhanam, MS, MBA
 

Similar to "Journey of CI/CD Pipeline Improvement in Yahoo! JAPAN" @ #SeConfTokyo (20)

Implementing Continuous Integration to Improve Software Quality
Implementing Continuous Integration to Improve Software QualityImplementing Continuous Integration to Improve Software Quality
Implementing Continuous Integration to Improve Software Quality
 
Linking Upstream and Downstream Agile
Linking Upstream and Downstream AgileLinking Upstream and Downstream Agile
Linking Upstream and Downstream Agile
 
Enterprise CI as-a-Service using Jenkins
Enterprise CI as-a-Service using JenkinsEnterprise CI as-a-Service using Jenkins
Enterprise CI as-a-Service using Jenkins
 
Neil Potter Presentation
Neil Potter Presentation Neil Potter Presentation
Neil Potter Presentation
 
Extend Agile and DevOps Practices Across Hybrid IT
Extend Agile and DevOps Practices Across Hybrid ITExtend Agile and DevOps Practices Across Hybrid IT
Extend Agile and DevOps Practices Across Hybrid IT
 
Harman deepak v - agile on steriod - dev ops led transformation
Harman  deepak v - agile on steriod - dev ops led transformationHarman  deepak v - agile on steriod - dev ops led transformation
Harman deepak v - agile on steriod - dev ops led transformation
 
CI/CD Overview
CI/CD OverviewCI/CD Overview
CI/CD Overview
 
Improving Quality through Continuous Integration - A case study of CollabNet
Improving Quality through Continuous Integration - A case study of CollabNetImproving Quality through Continuous Integration - A case study of CollabNet
Improving Quality through Continuous Integration - A case study of CollabNet
 
Alexyj Kovaliov "Waterfalling to Agile"
Alexyj Kovaliov "Waterfalling to Agile" Alexyj Kovaliov "Waterfalling to Agile"
Alexyj Kovaliov "Waterfalling to Agile"
 
Extending Jenkins to the Mainframe. A Simpler Approach.
Extending Jenkins to the Mainframe.  A Simpler Approach.Extending Jenkins to the Mainframe.  A Simpler Approach.
Extending Jenkins to the Mainframe. A Simpler Approach.
 
Advanced Continuous Delivery Best Practices (DEV317-R1) - AWS re:Invent 2018
Advanced Continuous Delivery Best Practices (DEV317-R1) - AWS re:Invent 2018Advanced Continuous Delivery Best Practices (DEV317-R1) - AWS re:Invent 2018
Advanced Continuous Delivery Best Practices (DEV317-R1) - AWS re:Invent 2018
 
Enterprise Agile at Lockheed Martin - 4th February 2014
Enterprise Agile at Lockheed Martin - 4th February 2014Enterprise Agile at Lockheed Martin - 4th February 2014
Enterprise Agile at Lockheed Martin - 4th February 2014
 
What We're Learning Adopting Spring Boot and PCF for Dell.com's eCommerce
What We're Learning Adopting Spring Boot and PCF for Dell.com's eCommerceWhat We're Learning Adopting Spring Boot and PCF for Dell.com's eCommerce
What We're Learning Adopting Spring Boot and PCF for Dell.com's eCommerce
 
S106 using ibm urban code deploy to deliver your apps to cics
S106 using ibm urban code deploy to deliver your apps to cicsS106 using ibm urban code deploy to deliver your apps to cics
S106 using ibm urban code deploy to deliver your apps to cics
 
MARISA SAWATPHADUNGKIJ
MARISA SAWATPHADUNGKIJMARISA SAWATPHADUNGKIJ
MARISA SAWATPHADUNGKIJ
 
SB Support System
SB Support SystemSB Support System
SB Support System
 
Enterprise DevOps Transformation
Enterprise DevOps TransformationEnterprise DevOps Transformation
Enterprise DevOps Transformation
 
DevOps is a ReOrg
DevOps is a ReOrgDevOps is a ReOrg
DevOps is a ReOrg
 
How Oracle Integrates Accessibility into the Development Process
How Oracle Integrates Accessibility into the Development ProcessHow Oracle Integrates Accessibility into the Development Process
How Oracle Integrates Accessibility into the Development Process
 
Starting the DevOps Train
Starting the DevOps TrainStarting the DevOps Train
Starting the DevOps Train
 

More from teyamagu

More from teyamagu (9)

Conference Organize Tips at 2019/06/22 #devlove #devlovex
Conference Organize Tips at 2019/06/22 #devlove #devlovexConference Organize Tips at 2019/06/22 #devlove #devlovex
Conference Organize Tips at 2019/06/22 #devlove #devlovex
 
Conference Summary Report: STARWEST 2017, Agile Testing Days 2017 ( #JaSST )
Conference Summary Report: STARWEST 2017, Agile Testing Days 2017 ( #JaSST )Conference Summary Report: STARWEST 2017, Agile Testing Days 2017 ( #JaSST )
Conference Summary Report: STARWEST 2017, Agile Testing Days 2017 ( #JaSST )
 
テスト自動化 はじめの一歩 (Test Automation -First Step-) in アジャイルひよこクラブ #agile_hiyoko
テスト自動化 はじめの一歩 (Test Automation -First Step-) in アジャイルひよこクラブ #agile_hiyokoテスト自動化 はじめの一歩 (Test Automation -First Step-) in アジャイルひよこクラブ #agile_hiyoko
テスト自動化 はじめの一歩 (Test Automation -First Step-) in アジャイルひよこクラブ #agile_hiyoko
 
Yahoo! JAPANにおけるアジャイル開発の普及戦略 #SPIJapan2016
Yahoo! JAPANにおけるアジャイル開発の普及戦略 #SPIJapan2016Yahoo! JAPANにおけるアジャイル開発の普及戦略 #SPIJapan2016
Yahoo! JAPANにおけるアジャイル開発の普及戦略 #SPIJapan2016
 
良きモノの提供に向けた協働 - 開発とテストが一体となったソフトウェア開発 -
良きモノの提供に向けた協働 - 開発とテストが一体となったソフトウェア開発 - 良きモノの提供に向けた協働 - 開発とテストが一体となったソフトウェア開発 -
良きモノの提供に向けた協働 - 開発とテストが一体となったソフトウェア開発 -
 
Web企業における大規模組織での品質の取り組み
Web企業における大規模組織での品質の取り組みWeb企業における大規模組織での品質の取り組み
Web企業における大規模組織での品質の取り組み
 
テスト自動化への1エンジニアとしての期待
テスト自動化への1エンジニアとしての期待テスト自動化への1エンジニアとしての期待
テスト自動化への1エンジニアとしての期待
 
アジャイル開発への組織の理解を得るために
アジャイル開発への組織の理解を得るために アジャイル開発への組織の理解を得るために
アジャイル開発への組織の理解を得るために
 
2014's Plan of Community Support Division, Agile Team Supporters in Japan
2014's Plan of Community Support Division, Agile Team Supporters in Japan2014's Plan of Community Support Division, Agile Team Supporters in Japan
2014's Plan of Community Support Division, Agile Team Supporters in Japan
 

Recently uploaded

Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdfMastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
mbmh111980
 

Recently uploaded (20)

Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
 
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdfMastering Windows 7 A Comprehensive Guide for Power Users .pdf
Mastering Windows 7 A Comprehensive Guide for Power Users .pdf
 
Breaking the Code : A Guide to WhatsApp Business API.pdf
Breaking the Code : A Guide to WhatsApp Business API.pdfBreaking the Code : A Guide to WhatsApp Business API.pdf
Breaking the Code : A Guide to WhatsApp Business API.pdf
 
Agnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in KrakówAgnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in Kraków
 
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
 
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital TransformationWSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
 
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdfA Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
 
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdfMicrosoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
 
How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?
How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?
How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?
 
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with StrimziStrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi
 
AI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning FrameworkAI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning Framework
 
Abortion ^Clinic ^%[+971588192166''] Abortion Pill Al Ain (?@?) Abortion Pill...
Abortion ^Clinic ^%[+971588192166''] Abortion Pill Al Ain (?@?) Abortion Pill...Abortion ^Clinic ^%[+971588192166''] Abortion Pill Al Ain (?@?) Abortion Pill...
Abortion ^Clinic ^%[+971588192166''] Abortion Pill Al Ain (?@?) Abortion Pill...
 
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
 
AI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in MichelangeloAI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in Michelangelo
 
Workforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfWorkforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdf
 
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product UpdatesGraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
 
Top Mobile App Development Companies 2024
Top Mobile App Development Companies 2024Top Mobile App Development Companies 2024
Top Mobile App Development Companies 2024
 
Secure Software Ecosystem Teqnation 2024
Secure Software Ecosystem Teqnation 2024Secure Software Ecosystem Teqnation 2024
Secure Software Ecosystem Teqnation 2024
 
CompTIA Security+ (Study Notes) for cs.pdf
CompTIA Security+ (Study Notes) for cs.pdfCompTIA Security+ (Study Notes) for cs.pdf
CompTIA Security+ (Study Notes) for cs.pdf
 
The Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion ProductionThe Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion Production
 

"Journey of CI/CD Pipeline Improvement in Yahoo! JAPAN" @ #SeConfTokyo

  • 1. Journey of CI/CD Pipeline Improvement in Yahoo! JAPAN Yahoo Japan Corporation 04/19/2019 @ Selenium Conf Tokyo Teppei Yamaguchi
  • 2. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 • Challenges we have faced in adaptation of company-wide pipeline and automated tests in many people / multi-product company • 多人数・多プロダクトのある企業における、全社的なパイプラ インの整備および自動テスト導入・普及の課題やその克服の流 れを紹介します。 Abstraction
  • 3. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 Responsibility • Transforming the whole company into “Agile” and “Test automation” Career • Programmer • Software Engineering Researcher/Consultant • Coach Test Coach, Agile Coach Yahoo Japan Corporation About me Teppei Yamaguchi
  • 4. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 • Programmers & Designers: 3,000+ • Services: 100+ (Portal, News, Ad, Shopping, Auction, Mail, Weather, …) • Service quality is the responsibility of the service department • No company-wide QA process • No QA department • Members change teams often Yahoo! JAPAN 13 Service A Division Product C Division Business/Designer/ Programmer etc. Platform Service
  • 5. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 • Total number of CI/CD pipelines • Executions per day(all the pipelines in total) The Current CI/CD Pipelines of Yahoo! JAPAN 40,000+ 15,400+ [times]
  • 6. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 The Current CI/CD Pipelines of Yahoo! JAPAN Github Enterprise SCM Execution Platform Screwdriver.cd CI/CD Tools Clone Deploy Publish Artifactory Enterprise OpenWhisk Kubernetes Pivotal Application Service Package Repository npm Enterprise Docker Registry Install Concourse E2E Test Tool Applicat Selenium
  • 7. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 Journey of CI/CD Pipeline Improvement 2013 1st Improvement Project 2nd Improvement Project 3rd Improvement Project 20192014 2015 2016 2017 2018 Automated E2E Testing Support
  • 8. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 1st Improvement Project - Preparation for CI/CD implementation -
  • 9. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Situation • Manual testing, deployment etc. was standard for many teams • Only a few advanced teams did CI/CD with Jenkins ▼Issue • We couldn’t provide services and products continuously on the latest OS and middleware • Spend lots of time to support older OS and middleware • Security risk 1st Improvement Project -Preparation for CI/CD implementation-
  • 10. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Top-down: establish new strategy • Clarify responsible groups of each module • Latest OS ▼Bottom-up: support individuals / teams • Jenkins support 1st Improvement Project -Preparation for CI/CD implementation-
  • 11. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Problems in project • High business goals, making it difficult to get resources for this work ▼Action for project problem solution • Explain individually to team’s managers, over and over again 1st Improvement Project -Preparation for CI/CD implementation-
  • 12. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Result • Update to latest OS and clarify responsible groups was completed • CI implementation itself hardly progressed 1st Improvement Project -Preparation for CI/CD implementation-
  • 13. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 2nd Improvement Project - CI/CD implementation -
  • 14. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Situation • Some teams used CI/CD with Jenkins • The importance of CI/CD was recognized but many teams didn’t use CI/CD ▼Issue • Cost to start using CI/CD was too high • Fixing security issues in CI/CD environment was high cost • Costs to move members between teams were high because all teams customized their own CI/CD in silos and requires relearning 2nd Improvement Project -CI/CD implementation-
  • 15. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 2nd Improvement Project -CI/CD implementation- ▼Top-down: establish new strategy • Use of centralized pipeline tools(Screwdriver.cd, Concourse, Approduce) • Write test code • Explain to business managers for reserving CI/CD implementation resources ▼Bottom-up: support individuals / teams • Provide Pipeline tool support • Provide testing support with test automation
  • 16. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Problems in project • High preparation cost for automated E2E testing • Difficult to automate testing for legacy systems ▼Action for project problem solution • Started new project to support automated E2E testing • Clarified the strategy so that automating tests for legacy can be postponed 2nd Improvement Project -CI/CD implementation-
  • 17. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Result • Automated build and automated deployment with pipeline tools became common • Couldn’t write test code for some legacy systems 2nd Improvement Project -CI/CD implementation-
  • 18. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 Automated E2E Testing Support
  • 19. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Situation • Builds & deployments were executed with the pipelines, but automated E2E tests were few ▼Issue • No E2E test environment in the pipeline and members didn’t know how to build it • Automated E2E tests of mobile app was not stable • Maintaining mobile devices required for man-powered E2E testing was too effort intensive Automated E2E Testing Support
  • 20. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Bottom-up: support individuals / teams • Provide centralized E2E test environment & tool(based on Appium)
 for mobile app using the pipeline • Provide Selenium container • Test automation support • Pair test programming, Pair E2E test environment development • Support incremental implementation of automated E2E test Automated E2E Testing Support
  • 21. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Result • Execute automated E2E test in the pipeline • Programmers got motivated for test automation Still improving… Automated E2E Testing Support
  • 22. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 3rd Improvement Project - CI/CD implement to legacy system -
  • 23. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Situation • Many teams did CI/CD with pipeline tools • CI/CD couldn’t be done for legacy systems ▼Issue • Cost to automate testing for legacy systems was too high 3rd Improvement Project -CI/CD implement to legacy system-
  • 24. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Top-down: establish new strategy • Refactoring from legacy systems to new testable systems • Explain to executives for assign CI/CD implementation resources ▼Bottom-up: support individuals / teams • Provide Pipeline tool support • Provide testing support 3rd Improvement Project -CI/CD implement to legacy system-
  • 25. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Result • Ongoing 3rd Improvement Project -CI/CD implement to legacy system-
  • 26. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 Conclusion
  • 27. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 Journey of CI/CD Pipeline Improvement Preparation 2013 1st Improvement Project CI/CD implementation 2nd Improvement Project CI/CD implement to legacy system 3rd Improvement Project 20192014 2015 2016 2017 2018 Automated E2E Testing Support
  • 28. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 • Implementation of company-wide CI/CD takes a long time • There are many problems that can not be discovered without trying, 
 because every teams are in various situations. • Need continuous improvement • Especially if you do something that does not go well Points