Distributed Docker Pipeline Architecture with CloudBees Jenkins Enterprise

© 2017 CloudBees, Inc. All Rights Reserved. 1
Distributed Pipeline Architecture
with CloudBees Jenkins Enterprise
and Docker
WORKSHOP

Prerequisites
● A GitHub Account - github.com
○ with a personal access token -
https://github.com/settings/tokens
● A Docker Hub Account - hub.docker.com
● General understanding of Jenkins Pipeline

DevOps Toolkit 2.1
REVIEW

CD enables speed
A single CD pipeline provides your
team speed with little instability...

But as CD scales business risk increases
As other teams adopt CD instability
increases and speed decreases

With CloudBees, continuous scaling just works
DPA enables scale with isolated,
on-demand containerized CD servers
for each of your team
...in fact you can move faster

CloudBees Distributed Pipeline Architecture
CloudBees Distributed Pipeline Architecture reduces business
risk through:
• Project Isolation
• DevOps project teams get their own Jenkins Master
• Distributes workload across masters
• Cross project contamination of workspaces and data is
eliminated
• Scalable Architecture
• Scaling and elasticity achieved through use of cluster
managed containers
• Data Isolation
• Data contamination from previous executions are easily
eliminated

CloudBees Jenkins Enterprise Cluster
Automated Cluster Install
• Robust CLI for managing cluster
• Automated infrastructure
provisioning for AWS and
OpenStack
Elastic/Ephemeral Jenkins Agents
• Dynamically spins up Docker
images as agents, stops and
removes on job completion
Built-in Fault Tolerance
• JENKINS_HOME is externally
managed to provide HA for
CJOC & Jenkins Masters
• Unhealthy CJOC & Jenkins
Masters are automatically
restarted on a healthy node
Straightforward, Fast, Dynamic
Provisioning of Jenkins Masters
• Easy & quick to create Masters
of any size
• Entire Jenkins Master lifecycle is
managed by CJOC

CloudBees Jenkins Enterprise Installation
Standing up a new CloudBees Jenkins Enterprise Cluster is straightforward
and fast:
• Download CJE install/CLI binary to a dedicated admin workstation
• Create a CJE project:
• $ cje init-project cje-project aws
• From the CJE project directory, initialize the new cluster:
• $ cje prepare cluster-init
• Configure common settings in cluster-init.config
• Verify and Install
• $ cje verify
• $ cje apply
• In 30 minutes or so, you will have a new CJE cluster

HANDS-ON TIME
CREATE A CJE USER ACCOUNT
● Open https://goo.gl/3tRlSQ
● Click on the Create an account link
● Enter information for all fields for the Sign up form
○ Username - needs to match your GitHub
username
○ Password - no restrictions, keep it simple
● Click the Sign Up button
● Success ...

Fast Provisioning of Jenkins Masters
DEMO

Jenkins Masters as Code
DEMO

FROM cloudbees/cje-mm:2.46.3.2
#skip setup wizard and disable CLI
ENV JVM_OPTS -Djenkins.install.runSetupWizard=false -Djenkins.CLI.disabled=true -server
#jenkins master configuration (groovy scripts)
COPY ./init.groovy.d/* /usr/share/jenkins/ref/init.groovy.d/
COPY ./license-activated/*
/usr/share/jenkins/ref/license-activated-or-renewed-after-expiration.groovy.d/
COPY ./quickstart/* /usr/share/jenkins/ref/quickstart.groovy.d/
#install additional plugins
ENV JENKINS_UC http://jenkins-updates.cloudbees.com
COPY plugins.txt plugins.txt
COPY jenkins-support /usr/local/bin/jenkins-support
COPY install-plugins.sh /usr/local/bin/install-plugins.sh
RUN /usr/local/bin/install-plugins.sh $(cat plugins.txt)
1
2
3
4
5
6

Configure CJOC to Use Custom Master Image
beedemo/cje-mm:2.46.3.2
DEMO

HANDS-ON TIME
PROVISION A MANAGED MASTER
● Open {CJE CJOC URL}
● Select New Item from the left menu
● Enter your first initial and last name as the item
name -> kmadel
● Select Managed Master as the item type
● Click OK
● On the Managed Master configuration page, under
Provisioning, click the Advanced button and
update the Disk value to 10
● Click the Save button at the bottom

HANDS-ON TIME
FORK GITHUB REPOS
● Login to github.com
● Open github.com/beedemo/go-demo
● Click on the Fork button on the top-right of the page
● Select the GitHub account/Organization your are using for
this workshop
● Repeat for:
○ github.com/beedemo/maven-project
○ github.com/beedemo/jenkins-dind-agent

Explore the Config-as-Code Managed Master
DEMO

Credentials Management
DEMO

HANDS-ON TIME
CONFIGURE DOCKER HUB CREDENTIALS
● Open {YOUR MASTER URL} and click the Credentials link
in the left menu
● Under Stores scoped to Jenkins, click on the (global) link
● Click the Add Credentials link in the left menu
● The Kind should default to Username with password - if
not select that from the dropdown
● Enter your Docker Hub Username and Password
● Enter an easy to remember ID such as docker-hub-kmadel
● Click the OK button at the bottom

Blue Ocean and Declarative Syntax for Pipeline
Blue Ocean is the new user interface that rethinks the user experience of
Jenkins with a focus on Pipeline.
Declarative syntax for Jenkins Pipelines designed to…
• Have smarter default behavior for most use cases
• Make Pipelines more structured
• Provide better error reporting and handling

Explore the go-demo Jenkinsfile
DEMO

HANDS-ON TIME
UPDATE THE GO-DEMO JENKINSFILE
● Login to github.com and open github.com/{user/org}/go-demo
● Switch to the build-cache-image branch
● Open the Jenkinsfile in edit mode
● Update the DOCKER_HUB_USER to match your Docker Hub
username
● Update the DOCKER_CREDENTIAL_ID to match the ID of the
Docker Hub credential you created earlier

HANDS-ON TIME
UPDATE THE GO-DEMO JENKINSFILE
● Go to the bottom to the go-demo Jenkinsfile - to the post section
● Add a success condition block, and within that block add the following step
to send a Slack notification:
○ slackSend(color: "good", message: "${env.JOB_NAME} completed
successfully, details at ${env.RUN_DISPLAY_URL}")
● Now we will add a failure block with a slight modified Slack step:
○ slackSend(color: "danger", message: "${env.JOB_NAME} failed,
details at ${env.RUN_DISPLAY_URL}")
● Add whatever Commit changes description you like and then click the
Commit changes button to commit to the build-cache-image branch

Palace and Docker Agent Templates
• Palace is a built-in Shared Agent Cloud for instant access to dynamically
provisioned Jenkins Agents
• It provides Dynamic, Ephemeral and Elastic Docker Image based agents
• Easy to configure Docker Agent Templates - individual items, not part of
one massive Jenkins Cloud configuration file
• Allows defining agent templates as separate items - no other Jenkins Cloud
plugin has this level of self-service capability
• The Palace service has built-in fault tolerance identical to what is provided
for CJOC and Managed Masters
• If the cluster node running the Palace service goes down, Marathon will
automatically restart it on another healthy node

HANDS-ON TIME
CREATE A DOCKER AGENT TEMPLATE
● Open {YOUR MASTER URL} and click the New Item link in the left menu
● Enter dind-compose for the item name
● Next, scroll down and select Docker Agent Template as the item type and click
the OK button
● Update the CPU shares to 0.2 and the Memory (MB) to 512
● For the Definition Image enter localhost:31095/beedemo/dind-compose-agent:go-demo
● Next, under Options, click the Add button and select Launch in privileged
mode - this is required to run Docker in a Docker container
● Click Add again, select Use a custom docker shell command and enter
wrapper.sh as the Custom shell command
● Finally, click the Save button

HANDS-ON TIME
CREATE A GITHUB ORG PROJECT WITH BLUE OCEAN
● Open {YOUR MASTER URL} and click on the Blue Ocean link at the top of the page
● Next, click on the New Pipeline button on the left side of the screen
● Click on the Github button
● Enter your GitHub personal access token and click Connect
● Next select your GitHub user or a GitHub Organization you are using for this
workshop
● Select Auto-discover Jenkinsfiles to scan all of the repos in your select
account/Organization
● Click on the Create Pipelines button and watch Blue Ocean discover the repos that
have branches containing a Jenkinsfile and then click on the build-cache-image
branch

HANDS-ON TIME
CREATE A PULL REQUEST TO THE GO-DEMO MASTER BRANCH
● Exit to the Jenkins classic UI by clicking on the Go to classic button
● Next, click on the Organization Folder project, click the Configure link, then click the
Save button and then switch back to the Blue Ocean UI
● Now login to github.com and open github.com/{user/org}/settings/hooks
● Next go to github.com/{user/org}/go-demo and switch to the build-cache-image
branch and click the New pull request button
● GitHub will switch you to the beedemo Org, select the base fork drop-down and
select your account/org, then select the base drop-down and select master
● Next click the Create pull request button and you should see that All checks have
passed already, so go ahead and click on Merge pull request and then Confirm
merge
● Switch back over to your Managed Master and the Blue Ocean UI

Pipeline Project Recognizers and Custom Marker Files
• By default, the Pipeline Organization and Multi-branch scans will only
include repositories that have at least one branch with a file named
Jenkinsfile
• CloudBees has a feature that allows you to specify arbitrary file names,
such as pom.xml, to use instead of Jenkinsfile when scanning branches
for Pipeline Organization Folder and Multi-branch projects
• This feature allows you to automatically provide standard automation for
teams that aren’t ready to start using Jenkins Pipeline and Jenkinsfiles
quite yet

HANDS-ON TIME
SET UP CUSTOM MARKER FILES
● Exit to the Jenkins classic UI by clicking on the Go to classic button
● Next, click on the Organization Folder project, click the Configure link
● Scroll down to the Project Recognizers section and click the Add button
underneath the default Pipeline Jenkinsfile project recognizer and select
Custom script
● Enter pom.xml for the Marker file, then under the Pipeline section, select
Pipeline script from SCM as the Definition and Git as the SCM
● Enter https://github.com/beedemo/custom-marker-pipelines for the
Repository URL and then enter pom-Jenkinsfile as the Script Path
● Now repeat, only specifying Dockerfile as the Marker file and
Dockerfile-Jenkinsfile as the Script Path, the click the Save button

Explore the beedemo/custom-marker-pipelines Repo
DEMO

UNUSED SLIDES BELOW

HANDS-ON TIME
CONFIGURE GITHUB CREDENTIALS
● Click the Global credentials (unrestricted) link at the top of the
page
● Click the Add Credentials link in the left menu
● The Kind should default to Username with password - if not
select that from the dropdown
● Enter your GitHub account id as the Username value
● Enter your GitHub access token as the Password value
● Enter an easy to remember ID such as github-token-kmadel
● Click the OK button at the bottom

Distributed Docker Pipeline Architecture with CloudBees Jenkins Enterprise

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Distributed Docker Pipeline Architecture with CloudBees Jenkins Enterprise

Similar to Distributed Docker Pipeline Architecture with CloudBees Jenkins Enterprise (20)

Recently uploaded

Recently uploaded (20)

Distributed Docker Pipeline Architecture with CloudBees Jenkins Enterprise