SlideShare a Scribd company logo

Configuration As Code - Adoption of the Job DSL Plugin at Netflix

Justin Ryan
Justin Ryan
Justin RyanSenior Software Engineer at Netflix

The Jenkins Job DSL plugin allows programmers to express job configurations as code. Learn about the benefits, from the obvious (store your configurations in the SCM of your choice) to the not-so-obvious (focus on intent, instead of succumbing to the distraction of multiple, complex job configuration options). We will share our experience adopting the plugin over the past year to create and maintain more complex job pipelines at Netflix.

Configuration As Code - Adoption of the Job DSL Plugin at Netflix

1 of 29
Download to read offline
Configuration as Code
Adoption of the Job DSL Plugin at Netflix

Justin Ryan <jryan@netflix.com>
@quidryan
http://www.slideshare.net/quidryan
Who Are We
• Engineering Tools
• Justin Ryan / jryan@netflix.com / @quidryan
• Curt Patrick / cpatrick@netflix.com
• Freedom & Responsibility
Situation
• No single flow that builds everything
• Every team has their own jobs
• Each branch needs multiple jobs
Problem
• Lots of copying of jobs
• Subtle fields hidden in Advanced button
• No ability to mimic another team
• Change in UI can be slow
Alternatives
• Template Project Plugin from Cloudbees
• Job Generator Plugin
• Literate Plugin
• etc...
Infrastructure As Code
“The end goal of infrastructure as code is to
perform as many infrastructure tasks as possible
programmatically. Key word is automation."

(http://www.somic.org/2012/09/28/concise-introduction-to-infrastructure-as-code/)
Ad

Recommended

10+ Deploys Per Day: Dev and Ops Cooperation at Flickr
10+ Deploys Per Day: Dev and Ops Cooperation at Flickr10+ Deploys Per Day: Dev and Ops Cooperation at Flickr
10+ Deploys Per Day: Dev and Ops Cooperation at FlickrJohn Allspaw
 
DevJam 2019 - Introduction to Kubernetes
DevJam 2019 - Introduction to KubernetesDevJam 2019 - Introduction to Kubernetes
DevJam 2019 - Introduction to KubernetesRonny Trommer
 
클라우드 여정의 시작 - 클라우드 전문가 조직의 프랙티컬 가이드-김학민, AWS SA Manager::AWS 마이그레이션 A to Z 웨비나
클라우드 여정의 시작 - 클라우드 전문가 조직의 프랙티컬 가이드-김학민, AWS SA Manager::AWS 마이그레이션 A to Z 웨비나클라우드 여정의 시작 - 클라우드 전문가 조직의 프랙티컬 가이드-김학민, AWS SA Manager::AWS 마이그레이션 A to Z 웨비나
클라우드 여정의 시작 - 클라우드 전문가 조직의 프랙티컬 가이드-김학민, AWS SA Manager::AWS 마이그레이션 A to Z 웨비나Amazon Web Services Korea
 
DevOps concepts, tools, and technologies v1.0
DevOps concepts, tools, and technologies v1.0DevOps concepts, tools, and technologies v1.0
DevOps concepts, tools, and technologies v1.0Mohamed Taman
 
쿠알못이 Amazon EKS로 안정적인 서비스 운영하기 - 최용호(넥슨코리아) :: AWS Community Day 2020
쿠알못이 Amazon EKS로 안정적인 서비스 운영하기 - 최용호(넥슨코리아) :: AWS Community Day 2020쿠알못이 Amazon EKS로 안정적인 서비스 운영하기 - 최용호(넥슨코리아) :: AWS Community Day 2020
쿠알못이 Amazon EKS로 안정적인 서비스 운영하기 - 최용호(넥슨코리아) :: AWS Community Day 2020AWSKRUG - AWS한국사용자모임
 
Learning Docker from Square One
Learning Docker from Square OneLearning Docker from Square One
Learning Docker from Square OneDocker, Inc.
 

More Related Content

What's hot

What is Cloud Native Explained?
What is Cloud Native Explained?What is Cloud Native Explained?
What is Cloud Native Explained?jeetendra mandal
 
An Introduction to Vuejs
An Introduction to VuejsAn Introduction to Vuejs
An Introduction to VuejsPaddy Lock
 
Introduction to DevOps | Edureka
Introduction to DevOps | EdurekaIntroduction to DevOps | Edureka
Introduction to DevOps | EdurekaEdureka!
 
DevOps Powerpoint Presentation Slides
DevOps Powerpoint Presentation SlidesDevOps Powerpoint Presentation Slides
DevOps Powerpoint Presentation SlidesSlideTeam
 
Engineering Velocity: Shifting the Curve at Netflix
Engineering Velocity: Shifting the Curve at NetflixEngineering Velocity: Shifting the Curve at Netflix
Engineering Velocity: Shifting the Curve at NetflixDianne Marsh
 
Amazon SageMaker를 통한 대용량 모델 훈련 방법 살펴보기 - 김대근 AWS AI/ML 스페셜리스트 솔루션즈 아키텍트 / 최영준...
Amazon SageMaker를 통한 대용량 모델 훈련 방법 살펴보기 - 김대근 AWS AI/ML 스페셜리스트 솔루션즈 아키텍트 / 최영준...Amazon SageMaker를 통한 대용량 모델 훈련 방법 살펴보기 - 김대근 AWS AI/ML 스페셜리스트 솔루션즈 아키텍트 / 최영준...
Amazon SageMaker를 통한 대용량 모델 훈련 방법 살펴보기 - 김대근 AWS AI/ML 스페셜리스트 솔루션즈 아키텍트 / 최영준...Amazon Web Services Korea
 
Crossplane @ Mastering GitOps.pdf
Crossplane @ Mastering GitOps.pdfCrossplane @ Mastering GitOps.pdf
Crossplane @ Mastering GitOps.pdfQAware GmbH
 
Docker로 서버 개발 편하게 하기
Docker로 서버 개발 편하게 하기Docker로 서버 개발 편하게 하기
Docker로 서버 개발 편하게 하기Dronix
 
[AWS Dev Day] 앱 현대화 | DevOps 개발자가 되기 위한 쿠버네티스 핵심 활용 예제 알아보기 - 정영준 AWS 솔루션즈 아키...
[AWS Dev Day] 앱 현대화 | DevOps 개발자가 되기 위한 쿠버네티스 핵심 활용 예제 알아보기 - 정영준 AWS 솔루션즈 아키...[AWS Dev Day] 앱 현대화 | DevOps 개발자가 되기 위한 쿠버네티스 핵심 활용 예제 알아보기 - 정영준 AWS 솔루션즈 아키...
[AWS Dev Day] 앱 현대화 | DevOps 개발자가 되기 위한 쿠버네티스 핵심 활용 예제 알아보기 - 정영준 AWS 솔루션즈 아키...Amazon Web Services Korea
 
DevOps Culture
DevOps CultureDevOps Culture
DevOps Culturerouanw
 
DevOps with GitHub Actions
DevOps with GitHub ActionsDevOps with GitHub Actions
DevOps with GitHub ActionsNilesh Gule
 
Introduction to Docker Compose
Introduction to Docker ComposeIntroduction to Docker Compose
Introduction to Docker ComposeAjeet Singh Raina
 
What is DevOps? | DevOps Introduction | DevOps Tools | DevOps Tutorial For Be...
What is DevOps? | DevOps Introduction | DevOps Tools | DevOps Tutorial For Be...What is DevOps? | DevOps Introduction | DevOps Tools | DevOps Tutorial For Be...
What is DevOps? | DevOps Introduction | DevOps Tools | DevOps Tutorial For Be...Simplilearn
 
DevOps Tutorial For Beginners | DevOps Tutorial | DevOps Tools | DevOps Train...
DevOps Tutorial For Beginners | DevOps Tutorial | DevOps Tools | DevOps Train...DevOps Tutorial For Beginners | DevOps Tutorial | DevOps Tools | DevOps Train...
DevOps Tutorial For Beginners | DevOps Tutorial | DevOps Tools | DevOps Train...Simplilearn
 
ISO (and other standard) Management Systems with OpenERP
ISO (and other standard) Management Systems with OpenERPISO (and other standard) Management Systems with OpenERP
ISO (and other standard) Management Systems with OpenERPMaxime Chambreuil
 

What's hot (20)

DevOps 101
DevOps 101DevOps 101
DevOps 101
 
What is Cloud Native Explained?
What is Cloud Native Explained?What is Cloud Native Explained?
What is Cloud Native Explained?
 
An Introduction to Vuejs
An Introduction to VuejsAn Introduction to Vuejs
An Introduction to Vuejs
 
Introduction to DevOps | Edureka
Introduction to DevOps | EdurekaIntroduction to DevOps | Edureka
Introduction to DevOps | Edureka
 
DevOps Powerpoint Presentation Slides
DevOps Powerpoint Presentation SlidesDevOps Powerpoint Presentation Slides
DevOps Powerpoint Presentation Slides
 
Meetup 23 - 03 - Application Delivery on K8S with GitOps
Meetup 23 - 03 - Application Delivery on K8S with GitOpsMeetup 23 - 03 - Application Delivery on K8S with GitOps
Meetup 23 - 03 - Application Delivery on K8S with GitOps
 
Engineering Velocity: Shifting the Curve at Netflix
Engineering Velocity: Shifting the Curve at NetflixEngineering Velocity: Shifting the Curve at Netflix
Engineering Velocity: Shifting the Curve at Netflix
 
Amazon SageMaker를 통한 대용량 모델 훈련 방법 살펴보기 - 김대근 AWS AI/ML 스페셜리스트 솔루션즈 아키텍트 / 최영준...
Amazon SageMaker를 통한 대용량 모델 훈련 방법 살펴보기 - 김대근 AWS AI/ML 스페셜리스트 솔루션즈 아키텍트 / 최영준...Amazon SageMaker를 통한 대용량 모델 훈련 방법 살펴보기 - 김대근 AWS AI/ML 스페셜리스트 솔루션즈 아키텍트 / 최영준...
Amazon SageMaker를 통한 대용량 모델 훈련 방법 살펴보기 - 김대근 AWS AI/ML 스페셜리스트 솔루션즈 아키텍트 / 최영준...
 
Crossplane @ Mastering GitOps.pdf
Crossplane @ Mastering GitOps.pdfCrossplane @ Mastering GitOps.pdf
Crossplane @ Mastering GitOps.pdf
 
Docker로 서버 개발 편하게 하기
Docker로 서버 개발 편하게 하기Docker로 서버 개발 편하게 하기
Docker로 서버 개발 편하게 하기
 
Kubernetes Basics
Kubernetes BasicsKubernetes Basics
Kubernetes Basics
 
[AWS Dev Day] 앱 현대화 | DevOps 개발자가 되기 위한 쿠버네티스 핵심 활용 예제 알아보기 - 정영준 AWS 솔루션즈 아키...
[AWS Dev Day] 앱 현대화 | DevOps 개발자가 되기 위한 쿠버네티스 핵심 활용 예제 알아보기 - 정영준 AWS 솔루션즈 아키...[AWS Dev Day] 앱 현대화 | DevOps 개발자가 되기 위한 쿠버네티스 핵심 활용 예제 알아보기 - 정영준 AWS 솔루션즈 아키...
[AWS Dev Day] 앱 현대화 | DevOps 개발자가 되기 위한 쿠버네티스 핵심 활용 예제 알아보기 - 정영준 AWS 솔루션즈 아키...
 
DevOps Culture
DevOps CultureDevOps Culture
DevOps Culture
 
Cloud Maturity Model: The Road to Adoption
Cloud Maturity Model: The Road to AdoptionCloud Maturity Model: The Road to Adoption
Cloud Maturity Model: The Road to Adoption
 
DevOps with GitHub Actions
DevOps with GitHub ActionsDevOps with GitHub Actions
DevOps with GitHub Actions
 
Introduction to Docker Compose
Introduction to Docker ComposeIntroduction to Docker Compose
Introduction to Docker Compose
 
Vue.js
Vue.jsVue.js
Vue.js
 
What is DevOps? | DevOps Introduction | DevOps Tools | DevOps Tutorial For Be...
What is DevOps? | DevOps Introduction | DevOps Tools | DevOps Tutorial For Be...What is DevOps? | DevOps Introduction | DevOps Tools | DevOps Tutorial For Be...
What is DevOps? | DevOps Introduction | DevOps Tools | DevOps Tutorial For Be...
 
DevOps Tutorial For Beginners | DevOps Tutorial | DevOps Tools | DevOps Train...
DevOps Tutorial For Beginners | DevOps Tutorial | DevOps Tools | DevOps Train...DevOps Tutorial For Beginners | DevOps Tutorial | DevOps Tools | DevOps Train...
DevOps Tutorial For Beginners | DevOps Tutorial | DevOps Tools | DevOps Train...
 
ISO (and other standard) Management Systems with OpenERP
ISO (and other standard) Management Systems with OpenERPISO (and other standard) Management Systems with OpenERP
ISO (and other standard) Management Systems with OpenERP
 

Viewers also liked

DevOps Practices: Configuration as Code
DevOps Practices:Configuration as CodeDevOps Practices:Configuration as Code
DevOps Practices: Configuration as CodeDoug Seven
 
Microsoft Azureの機械学習サービス (Azure Machine Learning/Microsoft Cognitive Services)
Microsoft Azureの機械学習サービス (Azure Machine Learning/Microsoft Cognitive Services)Microsoft Azureの機械学習サービス (Azure Machine Learning/Microsoft Cognitive Services)
Microsoft Azureの機械学習サービス (Azure Machine Learning/Microsoft Cognitive Services)Naoki (Neo) SATO
 
自動化を支えるCI/CDツールの私の選択 ~何をするためにCI/CDツールを選ぶか~
自動化を支えるCI/CDツールの私の選択 ~何をするためにCI/CDツールを選ぶか~自動化を支えるCI/CDツールの私の選択 ~何をするためにCI/CDツールを選ぶか~
自動化を支えるCI/CDツールの私の選択 ~何をするためにCI/CDツールを選ぶか~Recruit Lifestyle Co., Ltd.
 
クラウドを活用したシステム開発における、ネットワークのInfrastructure as Code
クラウドを活用したシステム開発における、ネットワークのInfrastructure as Codeクラウドを活用したシステム開発における、ネットワークのInfrastructure as Code
クラウドを活用したシステム開発における、ネットワークのInfrastructure as Codeskipping classes
 
なぜソフトウェアアーキテクトが必要なのか - Devlove 20110423
なぜソフトウェアアーキテクトが必要なのか - Devlove 20110423なぜソフトウェアアーキテクトが必要なのか - Devlove 20110423
なぜソフトウェアアーキテクトが必要なのか - Devlove 20110423Yusuke Suzuki
 
みんなのTerraformで AWSをテラフォーミングさせるぜ
みんなのTerraformで AWSをテラフォーミングさせるぜみんなのTerraformで AWSをテラフォーミングさせるぜ
みんなのTerraformで AWSをテラフォーミングさせるぜTakamasa Sakai
 
スマホ版ログレスにポストエフェクトシステムを導入した話
スマホ版ログレスにポストエフェクトシステムを導入した話スマホ版ログレスにポストエフェクトシステムを導入した話
スマホ版ログレスにポストエフェクトシステムを導入した話章暢 藤井
 
スマホ版ログレスでグローバル展開を想定したサーバ構築をAnsibleで試してみた話
スマホ版ログレスでグローバル展開を想定したサーバ構築をAnsibleで試してみた話スマホ版ログレスでグローバル展開を想定したサーバ構築をAnsibleで試してみた話
スマホ版ログレスでグローバル展開を想定したサーバ構築をAnsibleで試してみた話Akihiro Sugeno
 
新規事業が対峙する現実からエンジニアリングを俯瞰する #devsumiB #devsumi
新規事業が対峙する現実からエンジニアリングを俯瞰する #devsumiB #devsumi新規事業が対峙する現実からエンジニアリングを俯瞰する #devsumiB #devsumi
新規事業が対峙する現実からエンジニアリングを俯瞰する #devsumiB #devsumiItsuki Kuroda
 
インフラエンジニアの綺麗で優しい手順書の書き方
インフラエンジニアの綺麗で優しい手順書の書き方インフラエンジニアの綺麗で優しい手順書の書き方
インフラエンジニアの綺麗で優しい手順書の書き方Shohei Koyama
 

Viewers also liked (10)

DevOps Practices: Configuration as Code
DevOps Practices:Configuration as CodeDevOps Practices:Configuration as Code
DevOps Practices: Configuration as Code
 
Microsoft Azureの機械学習サービス (Azure Machine Learning/Microsoft Cognitive Services)
Microsoft Azureの機械学習サービス (Azure Machine Learning/Microsoft Cognitive Services)Microsoft Azureの機械学習サービス (Azure Machine Learning/Microsoft Cognitive Services)
Microsoft Azureの機械学習サービス (Azure Machine Learning/Microsoft Cognitive Services)
 
自動化を支えるCI/CDツールの私の選択 ~何をするためにCI/CDツールを選ぶか~
自動化を支えるCI/CDツールの私の選択 ~何をするためにCI/CDツールを選ぶか~自動化を支えるCI/CDツールの私の選択 ~何をするためにCI/CDツールを選ぶか~
自動化を支えるCI/CDツールの私の選択 ~何をするためにCI/CDツールを選ぶか~
 
クラウドを活用したシステム開発における、ネットワークのInfrastructure as Code
クラウドを活用したシステム開発における、ネットワークのInfrastructure as Codeクラウドを活用したシステム開発における、ネットワークのInfrastructure as Code
クラウドを活用したシステム開発における、ネットワークのInfrastructure as Code
 
なぜソフトウェアアーキテクトが必要なのか - Devlove 20110423
なぜソフトウェアアーキテクトが必要なのか - Devlove 20110423なぜソフトウェアアーキテクトが必要なのか - Devlove 20110423
なぜソフトウェアアーキテクトが必要なのか - Devlove 20110423
 
みんなのTerraformで AWSをテラフォーミングさせるぜ
みんなのTerraformで AWSをテラフォーミングさせるぜみんなのTerraformで AWSをテラフォーミングさせるぜ
みんなのTerraformで AWSをテラフォーミングさせるぜ
 
スマホ版ログレスにポストエフェクトシステムを導入した話
スマホ版ログレスにポストエフェクトシステムを導入した話スマホ版ログレスにポストエフェクトシステムを導入した話
スマホ版ログレスにポストエフェクトシステムを導入した話
 
スマホ版ログレスでグローバル展開を想定したサーバ構築をAnsibleで試してみた話
スマホ版ログレスでグローバル展開を想定したサーバ構築をAnsibleで試してみた話スマホ版ログレスでグローバル展開を想定したサーバ構築をAnsibleで試してみた話
スマホ版ログレスでグローバル展開を想定したサーバ構築をAnsibleで試してみた話
 
新規事業が対峙する現実からエンジニアリングを俯瞰する #devsumiB #devsumi
新規事業が対峙する現実からエンジニアリングを俯瞰する #devsumiB #devsumi新規事業が対峙する現実からエンジニアリングを俯瞰する #devsumiB #devsumi
新規事業が対峙する現実からエンジニアリングを俯瞰する #devsumiB #devsumi
 
インフラエンジニアの綺麗で優しい手順書の書き方
インフラエンジニアの綺麗で優しい手順書の書き方インフラエンジニアの綺麗で優しい手順書の書き方
インフラエンジニアの綺麗で優しい手順書の書き方
 

Similar to Configuration As Code - Adoption of the Job DSL Plugin at Netflix

Make Mobile Apps Quickly
Make Mobile Apps QuicklyMake Mobile Apps Quickly
Make Mobile Apps QuicklyGil Irizarry
 
IBM and Node.js - Old Doge, New Tricks
IBM and Node.js - Old Doge, New TricksIBM and Node.js - Old Doge, New Tricks
IBM and Node.js - Old Doge, New TricksDejan Glozic
 
Make Cross-platform Mobile Apps Quickly - SIGGRAPH 2014
Make Cross-platform Mobile Apps Quickly - SIGGRAPH 2014Make Cross-platform Mobile Apps Quickly - SIGGRAPH 2014
Make Cross-platform Mobile Apps Quickly - SIGGRAPH 2014Gil Irizarry
 
Exploring Next Generation Buildpacks - Anand Rao & Scott Deeg
Exploring Next Generation Buildpacks - Anand Rao & Scott DeegExploring Next Generation Buildpacks - Anand Rao & Scott Deeg
Exploring Next Generation Buildpacks - Anand Rao & Scott DeegVMware Tanzu
 
Untangling - fall2017 - week 9
Untangling - fall2017 - week 9Untangling - fall2017 - week 9
Untangling - fall2017 - week 9Derek Jacoby
 
Surrogate dependencies (in node js) v1.0
Surrogate dependencies  (in node js)  v1.0Surrogate dependencies  (in node js)  v1.0
Surrogate dependencies (in node js) v1.0Dinis Cruz
 
Web Components: The Future of Web Development is Here
Web Components: The Future of Web Development is HereWeb Components: The Future of Web Development is Here
Web Components: The Future of Web Development is HereJohn Riviello
 
Evolving Services Into A Cloud Native World
Evolving Services Into A Cloud Native WorldEvolving Services Into A Cloud Native World
Evolving Services Into A Cloud Native WorldIain Hull
 
Measure and Increase Developer Productivity with Help of Serverless at AWS Co...
Measure and Increase Developer Productivity with Help of Serverless at AWS Co...Measure and Increase Developer Productivity with Help of Serverless at AWS Co...
Measure and Increase Developer Productivity with Help of Serverless at AWS Co...Vadym Kazulkin
 
Angular Ivy- An Overview
Angular Ivy- An OverviewAngular Ivy- An Overview
Angular Ivy- An OverviewJalpesh Vadgama
 
Building production-quality apps with Node.js
Building production-quality apps with Node.jsBuilding production-quality apps with Node.js
Building production-quality apps with Node.jsmattpardee
 
Ensuring Design Standards with Web Components
Ensuring Design Standards with Web ComponentsEnsuring Design Standards with Web Components
Ensuring Design Standards with Web ComponentsJohn Riviello
 
Dev ops lessons learned - Michael Collins
Dev ops lessons learned  - Michael CollinsDev ops lessons learned  - Michael Collins
Dev ops lessons learned - Michael CollinsDevopsdays
 
Infrastructure automation-in-the-cloud-130613045624-phpapp02
Infrastructure automation-in-the-cloud-130613045624-phpapp02Infrastructure automation-in-the-cloud-130613045624-phpapp02
Infrastructure automation-in-the-cloud-130613045624-phpapp02Karim Labidi
 
ASP .Net Core SPA Templates
ASP .Net Core SPA TemplatesASP .Net Core SPA Templates
ASP .Net Core SPA TemplatesEamonn Boyle
 
Measure and Increase Developer Productivity with Help of Serverless at JCON 2...
Measure and Increase Developer Productivity with Help of Serverless at JCON 2...Measure and Increase Developer Productivity with Help of Serverless at JCON 2...
Measure and Increase Developer Productivity with Help of Serverless at JCON 2...Vadym Kazulkin
 

Similar to Configuration As Code - Adoption of the Job DSL Plugin at Netflix (20)

Make Mobile Apps Quickly
Make Mobile Apps QuicklyMake Mobile Apps Quickly
Make Mobile Apps Quickly
 
IBM and Node.js - Old Doge, New Tricks
IBM and Node.js - Old Doge, New TricksIBM and Node.js - Old Doge, New Tricks
IBM and Node.js - Old Doge, New Tricks
 
Make Cross-platform Mobile Apps Quickly - SIGGRAPH 2014
Make Cross-platform Mobile Apps Quickly - SIGGRAPH 2014Make Cross-platform Mobile Apps Quickly - SIGGRAPH 2014
Make Cross-platform Mobile Apps Quickly - SIGGRAPH 2014
 
Exploring Next Generation Buildpacks - Anand Rao & Scott Deeg
Exploring Next Generation Buildpacks - Anand Rao & Scott DeegExploring Next Generation Buildpacks - Anand Rao & Scott Deeg
Exploring Next Generation Buildpacks - Anand Rao & Scott Deeg
 
Untangling - fall2017 - week 9
Untangling - fall2017 - week 9Untangling - fall2017 - week 9
Untangling - fall2017 - week 9
 
Surrogate dependencies (in node js) v1.0
Surrogate dependencies  (in node js)  v1.0Surrogate dependencies  (in node js)  v1.0
Surrogate dependencies (in node js) v1.0
 
Web Components: The Future of Web Development is Here
Web Components: The Future of Web Development is HereWeb Components: The Future of Web Development is Here
Web Components: The Future of Web Development is Here
 
Evolving Services Into A Cloud Native World
Evolving Services Into A Cloud Native WorldEvolving Services Into A Cloud Native World
Evolving Services Into A Cloud Native World
 
Job DSL Plugin for Jenkins
Job DSL Plugin for JenkinsJob DSL Plugin for Jenkins
Job DSL Plugin for Jenkins
 
Measure and Increase Developer Productivity with Help of Serverless at AWS Co...
Measure and Increase Developer Productivity with Help of Serverless at AWS Co...Measure and Increase Developer Productivity with Help of Serverless at AWS Co...
Measure and Increase Developer Productivity with Help of Serverless at AWS Co...
 
Angular Ivy- An Overview
Angular Ivy- An OverviewAngular Ivy- An Overview
Angular Ivy- An Overview
 
Building production-quality apps with Node.js
Building production-quality apps with Node.jsBuilding production-quality apps with Node.js
Building production-quality apps with Node.js
 
Ensuring Design Standards with Web Components
Ensuring Design Standards with Web ComponentsEnsuring Design Standards with Web Components
Ensuring Design Standards with Web Components
 
Dev ops lessons learned - Michael Collins
Dev ops lessons learned  - Michael CollinsDev ops lessons learned  - Michael Collins
Dev ops lessons learned - Michael Collins
 
The Developers World
The Developers WorldThe Developers World
The Developers World
 
Infrastructure automation-in-the-cloud-130613045624-phpapp02
Infrastructure automation-in-the-cloud-130613045624-phpapp02Infrastructure automation-in-the-cloud-130613045624-phpapp02
Infrastructure automation-in-the-cloud-130613045624-phpapp02
 
Introduction to Node.js
Introduction to Node.jsIntroduction to Node.js
Introduction to Node.js
 
React.js at Cortex
React.js at CortexReact.js at Cortex
React.js at Cortex
 
ASP .Net Core SPA Templates
ASP .Net Core SPA TemplatesASP .Net Core SPA Templates
ASP .Net Core SPA Templates
 
Measure and Increase Developer Productivity with Help of Serverless at JCON 2...
Measure and Increase Developer Productivity with Help of Serverless at JCON 2...Measure and Increase Developer Productivity with Help of Serverless at JCON 2...
Measure and Increase Developer Productivity with Help of Serverless at JCON 2...
 

Recently uploaded

Azure Migration Guide for IT Professionals
Azure Migration Guide for IT ProfessionalsAzure Migration Guide for IT Professionals
Azure Migration Guide for IT ProfessionalsChristine Shepherd
 
Curtain Module Manual Zigbee Neo CS01-1C.pdf
Curtain Module Manual Zigbee Neo CS01-1C.pdfCurtain Module Manual Zigbee Neo CS01-1C.pdf
Curtain Module Manual Zigbee Neo CS01-1C.pdfDomotica daVinci
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologySafe Software
 
Manual sensor Zigbee 3.0 MOES ZSS-X-PIRL-C
Manual  sensor Zigbee 3.0 MOES ZSS-X-PIRL-CManual  sensor Zigbee 3.0 MOES ZSS-X-PIRL-C
Manual sensor Zigbee 3.0 MOES ZSS-X-PIRL-CDomotica daVinci
 
GDSC MMCOE - ML Campaign
GDSC MMCOE - ML CampaignGDSC MMCOE - ML Campaign
GDSC MMCOE - ML CampaignLavesh Akhadkar
 
My sample product research idea for you!
My sample product research idea for you!My sample product research idea for you!
My sample product research idea for you!KivenRaySarsaba
 
Introduction to Serverless with AWS Lambda in C#.pptx
Introduction to Serverless with AWS Lambda in C#.pptxIntroduction to Serverless with AWS Lambda in C#.pptx
Introduction to Serverless with AWS Lambda in C#.pptxBrandon Minnick, MBA
 
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre..."Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...shaiyuvasv
 
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdf
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdfLLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdf
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdfThomas Poetter
 
Manual Eurotronic Thermostatic Valve Comry Z-Wave
Manual Eurotronic Thermostatic Valve Comry Z-WaveManual Eurotronic Thermostatic Valve Comry Z-Wave
Manual Eurotronic Thermostatic Valve Comry Z-WaveDomotica daVinci
 
Zi-Stick UBS Dongle ZIgbee from Aeotec manual
Zi-Stick UBS Dongle ZIgbee from  Aeotec manualZi-Stick UBS Dongle ZIgbee from  Aeotec manual
Zi-Stick UBS Dongle ZIgbee from Aeotec manualDomotica daVinci
 
AWS reInvent 2023 recaps from Chicago AWS user group
AWS reInvent 2023 recaps from Chicago AWS user groupAWS reInvent 2023 recaps from Chicago AWS user group
AWS reInvent 2023 recaps from Chicago AWS user groupAWS Chicago
 
Semiconductor Review Magazine Feature.pdf
Semiconductor Review Magazine Feature.pdfSemiconductor Review Magazine Feature.pdf
Semiconductor Review Magazine Feature.pdfkeyaramicrochipusa
 
Traffic Signboard Classification with Voice alert to the driver.pptx
Traffic Signboard Classification with Voice alert to the driver.pptxTraffic Signboard Classification with Voice alert to the driver.pptx
Traffic Signboard Classification with Voice alert to the driver.pptxharimaxwell0712
 
2024 February Patch Tuesday
2024 February Patch Tuesday2024 February Patch Tuesday
2024 February Patch TuesdayIvanti
 
Bluetooth Low Energy(BLE) and beacons working
Bluetooth Low Energy(BLE) and beacons workingBluetooth Low Energy(BLE) and beacons working
Bluetooth Low Energy(BLE) and beacons workingshrey Ansh
 
AWS for the beginning is cloud computing
AWS for the beginning  is  cloud computingAWS for the beginning  is  cloud computing
AWS for the beginning is cloud computingkajalghule1
 
My self introduction to know others abut me
My self  introduction to know others abut meMy self  introduction to know others abut me
My self introduction to know others abut meManoj Prabakar B
 

Recently uploaded (20)

Azure Migration Guide for IT Professionals
Azure Migration Guide for IT ProfessionalsAzure Migration Guide for IT Professionals
Azure Migration Guide for IT Professionals
 
Curtain Module Manual Zigbee Neo CS01-1C.pdf
Curtain Module Manual Zigbee Neo CS01-1C.pdfCurtain Module Manual Zigbee Neo CS01-1C.pdf
Curtain Module Manual Zigbee Neo CS01-1C.pdf
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI Technology
 
Manual sensor Zigbee 3.0 MOES ZSS-X-PIRL-C
Manual  sensor Zigbee 3.0 MOES ZSS-X-PIRL-CManual  sensor Zigbee 3.0 MOES ZSS-X-PIRL-C
Manual sensor Zigbee 3.0 MOES ZSS-X-PIRL-C
 
GDSC MMCOE - ML Campaign
GDSC MMCOE - ML CampaignGDSC MMCOE - ML Campaign
GDSC MMCOE - ML Campaign
 
My Country Mobile
My Country MobileMy Country Mobile
My Country Mobile
 
My sample product research idea for you!
My sample product research idea for you!My sample product research idea for you!
My sample product research idea for you!
 
Introduction to Serverless with AWS Lambda in C#.pptx
Introduction to Serverless with AWS Lambda in C#.pptxIntroduction to Serverless with AWS Lambda in C#.pptx
Introduction to Serverless with AWS Lambda in C#.pptx
 
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre..."Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...
"Journey of Aspiration: Unveiling the Path to Becoming a Technocrat and Entre...
 
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdf
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdfLLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdf
LLMs, LMMs, their Improvement Suggestions and the Path towards AGI.pdf
 
Manual Eurotronic Thermostatic Valve Comry Z-Wave
Manual Eurotronic Thermostatic Valve Comry Z-WaveManual Eurotronic Thermostatic Valve Comry Z-Wave
Manual Eurotronic Thermostatic Valve Comry Z-Wave
 
Zi-Stick UBS Dongle ZIgbee from Aeotec manual
Zi-Stick UBS Dongle ZIgbee from  Aeotec manualZi-Stick UBS Dongle ZIgbee from  Aeotec manual
Zi-Stick UBS Dongle ZIgbee from Aeotec manual
 
AWS reInvent 2023 recaps from Chicago AWS user group
AWS reInvent 2023 recaps from Chicago AWS user groupAWS reInvent 2023 recaps from Chicago AWS user group
AWS reInvent 2023 recaps from Chicago AWS user group
 
Semiconductor Review Magazine Feature.pdf
Semiconductor Review Magazine Feature.pdfSemiconductor Review Magazine Feature.pdf
Semiconductor Review Magazine Feature.pdf
 
5 Tech Trend to Notice in ESG Landscape- 47Billion
5 Tech Trend to Notice in ESG Landscape- 47Billion5 Tech Trend to Notice in ESG Landscape- 47Billion
5 Tech Trend to Notice in ESG Landscape- 47Billion
 
Traffic Signboard Classification with Voice alert to the driver.pptx
Traffic Signboard Classification with Voice alert to the driver.pptxTraffic Signboard Classification with Voice alert to the driver.pptx
Traffic Signboard Classification with Voice alert to the driver.pptx
 
2024 February Patch Tuesday
2024 February Patch Tuesday2024 February Patch Tuesday
2024 February Patch Tuesday
 
Bluetooth Low Energy(BLE) and beacons working
Bluetooth Low Energy(BLE) and beacons workingBluetooth Low Energy(BLE) and beacons working
Bluetooth Low Energy(BLE) and beacons working
 
AWS for the beginning is cloud computing
AWS for the beginning  is  cloud computingAWS for the beginning  is  cloud computing
AWS for the beginning is cloud computing
 
My self introduction to know others abut me
My self  introduction to know others abut meMy self  introduction to know others abut me
My self introduction to know others abut me
 

Configuration As Code - Adoption of the Job DSL Plugin at Netflix

  • 1. Configuration as Code Adoption of the Job DSL Plugin at Netflix Justin Ryan <jryan@netflix.com> @quidryan http://www.slideshare.net/quidryan
  • 2. Who Are We • Engineering Tools • Justin Ryan / jryan@netflix.com / @quidryan • Curt Patrick / cpatrick@netflix.com • Freedom & Responsibility
  • 3. Situation • No single flow that builds everything • Every team has their own jobs • Each branch needs multiple jobs
  • 4. Problem • Lots of copying of jobs • Subtle fields hidden in Advanced button • No ability to mimic another team • Change in UI can be slow
  • 5. Alternatives • Template Project Plugin from Cloudbees • Job Generator Plugin • Literate Plugin • etc...
  • 6. Infrastructure As Code “The end goal of infrastructure as code is to perform as many infrastructure tasks as possible programmatically. Key word is automation." (http://www.somic.org/2012/09/28/concise-introduction-to-infrastructure-as-code/)
  • 7. Configuration As Code • Netflix - We understand the value of scaling and automation • Jobs should be responsible citizens • Team can own their scripts
  • 8. Job DSL Plugin • Create a Free-form project • Add a “Process Job DSL” build step • Enter Groovy script using a Jenkins DSL • Run job
  • 9. Step 2. Add Build Step
  • 10. Step 3. Specify Script
  • 12. Job DSL Language • Groovy • One additional method, “job” • Documentation in Github Wiki • https://github.com/jenkinsci/job-dsl-plugin/wiki • Job DSL Commands • Job Reference Page
  • 13. Single Job from Github job { name "RxJava-Test”  scm {   github(“git@github.com:Netflix/RxJava.git”)  }  steps {   gradle("test")  } }
  • 14. Leveraging Groovy import groovy.json.JsonSlurper def project = 'quidryan/aws-sdk-test' def github = 'https://api.github.com' def api = new URL("${github}/repos/${project}/branches") def branches = new JsonSlurper().parse(api.newReader()) branches.each {    def branchName = it.name    job {        name "${project}-${branchName}".replaceAll('/','-')        scm {            git("git://github.com/${project}.git", branchName)        }        steps {            maven("test -Dproject.name=${project}/${branchName}")        }    } }
  • 15. Disadvantages • Learning Curve • Can get complicated • Not all plugins exist • Possible problems with plugin version skew
  • 16. The Project • 408 installs at last check • Active Google Group, with 78 members • Regular Google Hangout • Constant stream of contributors • Core contributors - myself, Daniel Spilker, Stefan Wolf, Andrew Harmel-Law
  • 17. Use Case - Opower “Grab json blobs, which define pipeline, from an internal Rails app then loop through with Job DSL. I now have only one job to set up manually. So, in theory, I set up one job and I end up with 600 jobs. Throw each set into a folder and you've got yourself a red/green dashboard.” James Levinson <james.levinson@opower.com>
  • 19. Use Case - Chicago Trading “I use the Job DSL to create gerrit build and release build jobs, parameterized on the  git repository. Then I use the Build Flow plugin to orchestrate those jobs” Alan Beale <alan.beale@chicagotrading.com>
  • 20. Netflix Conventions • Utilities methods to create common jobs • Groovy Category to enhance “job” object • Published as a jar
  • 21. Merge Jobs import netflix.Merge Merge.nfMergeJobs( [jobFactory: this, securityGroup: ‘nbs’,    base: '//depot/IT_Engineering/NBS/nbsrules',    projectPrefix: 'BILLING-rules'] ) Merge.nfMergeJobs( [jobFactory: this, securityGroup: ‘nbs’,    base: '//depot/IT_Engineering/NBS/Phoenix',    projectPrefix: 'BILLING-phoenix'] )
  • 22. Merge Jobs • BILLING-rules-dev-force-to-test • BILLING-rules-test-force-to-prod • BILLING-rules-prod-merge-to-test • BILLING-rules-test-force-to-dev • BILLING-phoenix-dev-force-to-test • BILLING-phoenix-test-force-to-prod • BILLING-phoenix-prod-merge-to-test • BILLING-phoenix-test-force-to-dev
  • 23. Jobs as Objects import netflix.* def defaults = [...] def jobs = [‘dev’,’test’,’prod’].collectMany {[ CBFQuickBuild(defaults + [branch:branch]), CBFQualityBuild(defaults + [branch:branch]), nfBakeJob(defaults + [branch:branch]) ]} jobs.each {    it.publishers {        extendedEmail('jryan@netflix.com', 'Oops') {            trigger(triggerName: 'StillUnstable', subject: 'Subject', recipientList:'RecipientList',                 sendToDevelopers: true, sendToRequester: true, includeCulprits: true, sendToRecipientList: false)    } }
  • 24. Extra features • Environment variables • Queue jobs to run • Read files workspace • Configure blocks • @Grab • @Category
  • 25. @Category use(netflix.Conventions) { def myjob = job { name ‘BuildFromP4’ }  myjob.addPerforce("//depot/webapps/astrid/...") }
  • 26. Good Practice • Respect “source of truth” • All scripts go in version control • Use configure block if needed • Trigger “seed” job on SCM changes
  • 27. Future of project • Plugins provide DSL methods • Produce DSL from real job • Support for utility classes • Produce jobs from build, e.g. Gradle or Maven
  • 28. Jenkins User Conference Palo Alto , Oct 23 2013 Thank You To Our Sponsors Platinum Gold Silver #jenkinsconf
  • 29. Thank You We’re hiring. Justin Ryan <jryan@netflix.com> @quidryan http://www.slideshare.net/quidryan

Editor's Notes

  1. Send feedback to @quidryan, presentation will be up later on slideshare, if you miss some link.
  2. I’d like to start by thanking my co-worker Curt Patrick. While I mostly work on writing the Job DSL Plugin, he has the spent the most time using it and integrating it into teams. Engineering Tools Is part of a larger organization who makes developing for the cloud palatable. Tasked with: Building and Deploying to the cloud Try to: Keep simple, simple. Keep hard, possible. Coupled with Freedom &amp; Responsibility See the Netflix culture slide deck. Or listen to episode 28 of theshipshow.com’s podcast to hear two members of the team explain how F&amp;R work in the context of our team. In general, we can’t necessarily tell people what to do
  3. We use a Service Orientated Architecture, with lots of little services. There is no one monolithic app to zip together. Each team has their own release schedule, some are short, some are long. Meaning there is not a set of jobs that all pushes go through. The jobs used between teams are “similar&quot;, and varying across branches and potentially environments. Think of a matrix of flows, measuring hundreds of projects by three branches. We have 3200 active jobs on our primary master.
  4. The problem then becomes the complexity. We provide some canonical jobs, and teams will copy among themselves. But in the end these are copies, and we don’t have a way to keep them up to date or effectively share them. We have a lot of plugins on our master, which makes the rendering of the configure screen slow.
  5. Caveat: Our problems are not necessarily your problems. We are not the teams making jobs for people, but we’re the ones who try to curate and streamline the usage of Jenkins. This gives us a unique prospective. So if I say something disparaging, it might just be because our situation is different than yours. It’s not giving anything away to say that we ended up using Job DSL. But it’s worth pointing out the alternatives at this stage. Since it was what we had to think about without this plugin. I’ll be the first to admit that I’m lazy when evaluating plugins, unless they have lots of screenshots. Template Project Plugin - UI Based, which is hard to scale out to many dozens of teams which all have their own preferences for jobs. The variations we have don’t map to a system being driven by parameters. Their docs state that it’s best when “only differ by a few attributes”, that’s not us. Job Generator Plugin - Good integration aligning input/outputs of jobs. Great for standardized flows, but that’s not us. Literate Plugin - Very new. Solves one of our use-cases which is to tie job creation to the source code. But we’re learned that the minutia of each plugin extremely subtle and important to get right. The idea of checking in specific settings in a subdirectory isn’t very scalable either across projects. Speaking of templates, we found that we don’t use them.
  6. Recognizing that the problem is about complexity, we can look at other areas which have had to tackle complexity recently. There have been plenty of attempts to wrap complex systems in pretty UIs. UML for coding. There are plenty of “Enterprise” solutions for configuration management, but it’s Chef/Puppet that’s winning that war. In Operations, their world is being up-ended by using Chef/Puppet. It’s making the problem of configuring hundreds if not thousands of systems manageable. Our problem is the same.
  7. Proper automation has to deal with all the quirks and customizations that real world requires. And to tackle those customizations we write code. Code is code. Build is Code (Ant/Gradle). Infrastructure is Code (Chef). The only missing piece is our build jobs, why are they special? In this model, Jobs are just artifacts of the DSL scripts. We expect any piece of our infrastructure to be a responsible citizen. They should be in SCM, they should get code reviews, re-usability through higher-level methods. Teams write and own the scripts. What they write is isolated from the work of others. There are no global constructs like templates. Meaning that changes are managed by them on their schedule, i.e. they don’t have the take a template change in the middle of a prod push. I’m sold!
  8. This is accomplished using the Job DSL Plugin. Let me explain the how it’s used. Create a “Seed” job, which will create the other jobs. Groovy runtime’s been instrumented to expose certain domain specific language features while maintaining the language. The specific build step will execute the script and collect a list of jobs to maintain. The general model is you create the ideal situation and the plugin take care of create/update/delete.
  9. We have 7 SCMs, 6 triggers, 15 build steps, 28 publishers, ~20 wrappers.
  10. It can be difficult to get started. I find that once there’s a “seed” job to modify, it’s easy to tweak it.Ideally, you could make the job the way you like it, then translate it back to dsl code, but that’s a future feature.Mental mind-shift, given that SEED jobs are jobs themselves.Plugin itself is dependent on contributors to fill in unimplemented plugins and we have a problem with version skew.
  11. Friendly mailing list and a monthly Google Hangout to let people ask questions. Constant stream of contributors adding miscellaneous plugins as they find a need. I’m always amazed how many plugins are out there. A few contributors stand out as contributing to the core
  12. sic.
  13. sic. Showing that the job dsl is good at creating jobs, while other plugins are better at executing them.
  14. Back to Netflix. We want to be a launching off point for teams, where EngTools can add value and not be gating them. We do that producing a .jar with utility methods and enhancements. We end up with our own language above the job dsl. Users are able to use the DSL directly or leverage what we offer on top of it. Focus on making most complicated problems easier (e.g. nftaskbuilder), while also addressing common problems (branching and merging).
  15. There’s some internals showing up here, around jobFactory, just ignore them for now. These each create 4 merge jobs, 2 to clobber up from dev to test, test to prod. And 2 merge jobs to merge back from prod to test and test to dev. Making this code even cleaner is left up to the reader.
  16. Creating 8 standardized jobs, which if changed by EngTools would get updated. We can abstract out the appropriate methods for our use-case.
  17. Some caveats, we’re using Groovy.Using groovy variables like properties, or variables defined earlier. Groovy strings, aka GStrings, have token replacement.
  18. addPerforce is a Netflix specific function that suites our needs and environment.
  19. E.g. Create a builds for every public Netflix project on GitHub. The Github API is the source of truth. Or Store model of your builds somewhere, like a web-app, so that users have a nice UI.Put scripts in VCS, we put them in a shared repo, to encourage teams learning from each other.
  20. Leave here, just Google “job dsl”, it’ll come right up. This was maybe Job DSL 101, for 102 tune into the Google Hangout to answer all new questions. As you might have guessed, we’re not just your standard Jenkins shop. We’re always looking out for someone who deeply understands Jenkins and can contribute back to the community, as part of their job.