Yahoo Japan has been working to implement CI/CD pipelines and automated testing across its large organization with many products and teams over several years through a series of improvement projects. The first project focused on preparing infrastructure for CI/CD. The second project focused on implementing CI/CD tools and pipelines but faced challenges with automating tests for legacy systems. A subsequent project aimed to support automated end-to-end testing. The current third project is working to implement CI/CD for legacy systems through refactoring and providing support. The presentation emphasizes that transforming a large company to adopt new practices like CI/CD and testing is an ongoing process that requires continuous improvement over many years.

1. Journey of CI/CD Pipeline
Improvement in Yahoo! JAPAN
Yahoo Japan Corporation
04/19/2019 @ Selenium Conf Tokyo
Teppei Yamaguchi

2. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
• Challenges we have faced in adaptation of company-wide
pipeline and automated tests in many people / multi-product
company
• 多人数・多プロダクトのある企業における、全社的なパイプラ
インの整備および自動テスト導入・普及の課題やその克服の流
れを紹介します。
Abstraction

3. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
Responsibility
• Transforming the whole company into “Agile” and “Test automation”
Career
• Programmer
• Software Engineering Researcher/Consultant
• Coach
Test Coach, Agile Coach
Yahoo Japan Corporation
About me
Teppei Yamaguchi

4. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
• Programmers & Designers: 3,000+
• Services: 100+ (Portal, News, Ad, Shopping, Auction, Mail, Weather, …)
• Service quality is the responsibility of the service department
• No company-wide QA process
• No QA department
• Members change teams often
Yahoo! JAPAN
13
Service A
Division
Product C
Division
Business/Designer/
Programmer etc.
Platform Service

5. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
• Total number of CI/CD pipelines
• Executions per day(all the pipelines in total)
The Current CI/CD Pipelines of Yahoo! JAPAN
40,000+
15,400+ [times]

6. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
The Current CI/CD Pipelines of Yahoo! JAPAN
Github
Enterprise
SCM Execution Platform
Screwdriver.cd
CI/CD Tools
Clone Deploy
Publish
Artifactory
Enterprise
OpenWhisk Kubernetes
Pivotal Application Service
Package Repository
npm
Enterprise
Docker
Registry
Install
Concourse E2E Test Tool
Applicat Selenium

7. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
Journey of CI/CD Pipeline Improvement
2013
1st Improvement
Project
2nd Improvement
Project
3rd Improvement
Project
20192014 2015 2016 2017 2018
Automated E2E Testing Support

8. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
1st Improvement Project
- Preparation for CI/CD implementation -

9. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Situation
• Manual testing, deployment etc. was standard for many teams
• Only a few advanced teams did CI/CD with Jenkins
▼Issue
• We couldn’t provide services and products continuously on the latest OS and
middleware
• Spend lots of time to support older OS and middleware
• Security risk
1st Improvement Project -Preparation for CI/CD implementation-

10. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Top-down: establish new strategy
• Clarify responsible groups of each module
• Latest OS
▼Bottom-up: support individuals / teams
• Jenkins support
1st Improvement Project -Preparation for CI/CD implementation-

11. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Problems in project
• High business goals, making it difficult to get resources for this work
▼Action for project problem solution
• Explain individually to team’s managers, over and over again
1st Improvement Project -Preparation for CI/CD implementation-

12. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Result
• Update to latest OS and clarify responsible groups was completed
• CI implementation itself hardly progressed
1st Improvement Project -Preparation for CI/CD implementation-

13. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
2nd Improvement Project
- CI/CD implementation -

14. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Situation
• Some teams used CI/CD with Jenkins
• The importance of CI/CD was recognized but many teams didn’t use CI/CD
▼Issue
• Cost to start using CI/CD was too high
• Fixing security issues in CI/CD environment was high cost
• Costs to move members between teams were high because all teams
customized their own CI/CD in silos and requires relearning
2nd Improvement Project -CI/CD implementation-

15. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
2nd Improvement Project -CI/CD implementation-
▼Top-down: establish new strategy
• Use of centralized pipeline tools(Screwdriver.cd, Concourse, Approduce)
• Write test code
• Explain to business managers for reserving CI/CD implementation resources
▼Bottom-up: support individuals / teams
• Provide Pipeline tool support
• Provide testing support with test automation

16. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Problems in project
• High preparation cost for automated E2E testing
• Difficult to automate testing for legacy systems
▼Action for project problem solution
• Started new project to support automated E2E testing
• Clarified the strategy so that automating tests for legacy can be postponed
2nd Improvement Project -CI/CD implementation-

17. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Result
• Automated build and automated deployment with pipeline tools became
common
• Couldn’t write test code for some legacy systems
2nd Improvement Project -CI/CD implementation-

18. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
Automated E2E Testing Support

19. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Situation
• Builds & deployments were executed with the pipelines, but automated E2E tests
were few
▼Issue
• No E2E test environment in the pipeline and members didn’t know how to build it
• Automated E2E tests of mobile app was not stable
• Maintaining mobile devices required for man-powered E2E testing was too effort
intensive
Automated E2E Testing Support

20. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Bottom-up: support individuals / teams
• Provide centralized E2E test environment & tool(based on Appium)
for mobile app using the pipeline
• Provide Selenium container
• Test automation support
• Pair test programming, Pair E2E test environment development
• Support incremental implementation of automated E2E test
Automated E2E Testing Support

21. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Result
• Execute automated E2E test in the pipeline
• Programmers got motivated for test automation
Still improving…
Automated E2E Testing Support

22. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
3rd Improvement Project
- CI/CD implement to legacy system -

23. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Situation
• Many teams did CI/CD with pipeline tools
• CI/CD couldn’t be done for legacy systems
▼Issue
• Cost to automate testing for legacy systems was too high
3rd Improvement Project -CI/CD implement to legacy system-

24. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Top-down: establish new strategy
• Refactoring from legacy systems to new testable systems
• Explain to executives for assign CI/CD implementation resources
▼Bottom-up: support individuals / teams
• Provide Pipeline tool support
• Provide testing support
3rd Improvement Project -CI/CD implement to legacy system-

25. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
▼Result
• Ongoing
3rd Improvement Project -CI/CD implement to legacy system-

26. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
Conclusion

27. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
Journey of CI/CD Pipeline Improvement
Preparation
2013
1st Improvement
Project
CI/CD
implementation
2nd Improvement
Project
CI/CD implement to
legacy system
3rd Improvement
Project
20192014 2015 2016 2017 2018
Automated E2E Testing Support

28. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止
• Implementation of company-wide CI/CD takes a long time
• There are many problems that can not be discovered without trying,
because every teams are in various situations.
• Need continuous improvement
• Especially if you do something that does not go well
Points