Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

"Journey of CI/CD Pipeline Improvement in Yahoo! JAPAN" @ #SeConfTokyo

256 views

Published on

Presentation slides "Journey of CI/CD Pipeline Improvement in Yahoo! Japan" at Selenium Conf Tokyo 2019( #SeConfTokyo )

Published in: Software
  • Be the first to comment

"Journey of CI/CD Pipeline Improvement in Yahoo! JAPAN" @ #SeConfTokyo

  1. 1. Journey of CI/CD Pipeline Improvement in Yahoo! JAPAN Yahoo Japan Corporation 04/19/2019 @ Selenium Conf Tokyo Teppei Yamaguchi
  2. 2. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 • Challenges we have faced in adaptation of company-wide pipeline and automated tests in many people / multi-product company • 多人数・多プロダクトのある企業における、全社的なパイプラ インの整備および自動テスト導入・普及の課題やその克服の流 れを紹介します。 Abstraction
  3. 3. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 Responsibility • Transforming the whole company into “Agile” and “Test automation” Career • Programmer • Software Engineering Researcher/Consultant • Coach Test Coach, Agile Coach Yahoo Japan Corporation About me Teppei Yamaguchi
  4. 4. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 • Programmers & Designers: 3,000+ • Services: 100+ (Portal, News, Ad, Shopping, Auction, Mail, Weather, …) • Service quality is the responsibility of the service department • No company-wide QA process • No QA department • Members change teams often Yahoo! JAPAN 13 Service A Division Product C Division Business/Designer/ Programmer etc. Platform Service
  5. 5. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 • Total number of CI/CD pipelines • Executions per day(all the pipelines in total) The Current CI/CD Pipelines of Yahoo! JAPAN 40,000+ 15,400+ [times]
  6. 6. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 The Current CI/CD Pipelines of Yahoo! JAPAN Github Enterprise SCM Execution Platform Screwdriver.cd CI/CD Tools Clone Deploy Publish Artifactory Enterprise OpenWhisk Kubernetes Pivotal Application Service Package Repository npm Enterprise Docker Registry Install Concourse E2E Test Tool Applicat Selenium
  7. 7. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 Journey of CI/CD Pipeline Improvement 2013 1st Improvement Project 2nd Improvement Project 3rd Improvement Project 20192014 2015 2016 2017 2018 Automated E2E Testing Support
  8. 8. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 1st Improvement Project - Preparation for CI/CD implementation -
  9. 9. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Situation • Manual testing, deployment etc. was standard for many teams • Only a few advanced teams did CI/CD with Jenkins ▼Issue • We couldn’t provide services and products continuously on the latest OS and middleware • Spend lots of time to support older OS and middleware • Security risk 1st Improvement Project -Preparation for CI/CD implementation-
  10. 10. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Top-down: establish new strategy • Clarify responsible groups of each module • Latest OS ▼Bottom-up: support individuals / teams • Jenkins support 1st Improvement Project -Preparation for CI/CD implementation-
  11. 11. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Problems in project • High business goals, making it difficult to get resources for this work ▼Action for project problem solution • Explain individually to team’s managers, over and over again 1st Improvement Project -Preparation for CI/CD implementation-
  12. 12. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Result • Update to latest OS and clarify responsible groups was completed • CI implementation itself hardly progressed 1st Improvement Project -Preparation for CI/CD implementation-
  13. 13. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 2nd Improvement Project - CI/CD implementation -
  14. 14. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Situation • Some teams used CI/CD with Jenkins • The importance of CI/CD was recognized but many teams didn’t use CI/CD ▼Issue • Cost to start using CI/CD was too high • Fixing security issues in CI/CD environment was high cost • Costs to move members between teams were high because all teams customized their own CI/CD in silos and requires relearning 2nd Improvement Project -CI/CD implementation-
  15. 15. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 2nd Improvement Project -CI/CD implementation- ▼Top-down: establish new strategy • Use of centralized pipeline tools(Screwdriver.cd, Concourse, Approduce) • Write test code • Explain to business managers for reserving CI/CD implementation resources ▼Bottom-up: support individuals / teams • Provide Pipeline tool support • Provide testing support with test automation
  16. 16. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Problems in project • High preparation cost for automated E2E testing • Difficult to automate testing for legacy systems ▼Action for project problem solution • Started new project to support automated E2E testing • Clarified the strategy so that automating tests for legacy can be postponed 2nd Improvement Project -CI/CD implementation-
  17. 17. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Result • Automated build and automated deployment with pipeline tools became common • Couldn’t write test code for some legacy systems 2nd Improvement Project -CI/CD implementation-
  18. 18. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 Automated E2E Testing Support
  19. 19. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Situation • Builds & deployments were executed with the pipelines, but automated E2E tests were few ▼Issue • No E2E test environment in the pipeline and members didn’t know how to build it • Automated E2E tests of mobile app was not stable • Maintaining mobile devices required for man-powered E2E testing was too effort intensive Automated E2E Testing Support
  20. 20. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Bottom-up: support individuals / teams • Provide centralized E2E test environment & tool(based on Appium)
 for mobile app using the pipeline • Provide Selenium container • Test automation support • Pair test programming, Pair E2E test environment development • Support incremental implementation of automated E2E test Automated E2E Testing Support
  21. 21. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Result • Execute automated E2E test in the pipeline • Programmers got motivated for test automation Still improving… Automated E2E Testing Support
  22. 22. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 3rd Improvement Project - CI/CD implement to legacy system -
  23. 23. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Situation • Many teams did CI/CD with pipeline tools • CI/CD couldn’t be done for legacy systems ▼Issue • Cost to automate testing for legacy systems was too high 3rd Improvement Project -CI/CD implement to legacy system-
  24. 24. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Top-down: establish new strategy • Refactoring from legacy systems to new testable systems • Explain to executives for assign CI/CD implementation resources ▼Bottom-up: support individuals / teams • Provide Pipeline tool support • Provide testing support 3rd Improvement Project -CI/CD implement to legacy system-
  25. 25. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 ▼Result • Ongoing 3rd Improvement Project -CI/CD implement to legacy system-
  26. 26. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 Conclusion
  27. 27. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 Journey of CI/CD Pipeline Improvement Preparation 2013 1st Improvement Project CI/CD implementation 2nd Improvement Project CI/CD implement to legacy system 3rd Improvement Project 20192014 2015 2016 2017 2018 Automated E2E Testing Support
  28. 28. Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. 無断引用・転載禁止 • Implementation of company-wide CI/CD takes a long time • There are many problems that can not be discovered without trying, 
 because every teams are in various situations. • Need continuous improvement • Especially if you do something that does not go well Points

×