The document discusses using Mesos and Mesosphere DCOS to elastically scale Jenkins. It provides background on Mesos, DCOS, and Jenkins. Conventional methods for scaling Jenkins like using a single or multiple static Jenkins masters are described. A demo is then shown of running Jenkins on Mesos which allows Jenkins agents to dynamically register based on available resources. Future work discussed includes tighter integration of Jenkins for continuous integration and continuous delivery (CI/CD) workflows within DCOS.

1. © 2016 Mesosphere, Inc. All Rights Reserved.
Roger Ignazio | January 20, 2016
ELASTIC JENKINS
WITH MESOS
AND DCOS

2. © 2016 Mesosphere, Inc. All Rights Reserved.
Roger Ignazio
QE Tech Lead @ Mesosphere
@rogerignazio
ABOUT ME

3. © 2016 Mesosphere, Inc. All Rights Reserved.
mesosinaction.com
MESOS IN ACTION

4. © 2016 Mesosphere, Inc. All Rights Reserved.
● Conventional methods for scaling Jenkins
● Elastic Jenkins on Mesos and Mesosphere DCOS
● Demo
● Future work
AGENDA

5. © 2016 Mesosphere, Inc. All Rights Reserved.
ALLOW ME TO
INTRODUCE…

6. © 2016 Mesosphere, Inc. All Rights Reserved.
● General-purpose cluster resource manager
● Represents many machines as a single entity
● Advertises resources directly to applications
● Works at scale: Apple, Twitter, Netflix, Airbnb, …
A BRIEF INTRO TO MESOS

7. © 2016 Mesosphere, Inc. All Rights Reserved.
A BRIEF INTRO TO MESOS

8. © 2015 Mesosphere, Inc. All Rights Reserved. 8
A BRIEF INTRO
TO DCOS
● Enterprise-grade
distributed
operating system
● Mesos as a
distributed systems
kernel
● Built using open
source and
commercial tools
Existing
Infrastructure
Mesosphere
DCOS
Services &
Containers

9. © 2016 Mesosphere, Inc. All Rights Reserved.
● Open source, distributed CI tool
● Repeatable build/test of software projects
● Large community, rich plugin ecosystem
A BRIEF INTRO TO JENKINS

10. © 2016 Mesosphere, Inc. All Rights Reserved.
SCALING
JENKINS

11. © 2016 Mesosphere, Inc. All Rights Reserved.
SINGLE JENKINS MASTER
Team A . . .
Jenkins Agent 1
Jenkins Agent 4
Jenkins Agent N
Team B
Jenkins Agent 2
Jenkins Agent 5
Team X
Jenkins Agent 3
Jenkins Agent 6
. . .
Jenkins Master

12. © 2016 Mesosphere, Inc. All Rights Reserved.
SINGLE JENKINS MASTER → FAILED!
Team A . . .
Jenkins Agent 1
Jenkins Agent 4
Jenkins Agent N
Team B
Jenkins Agent 2
Jenkins Agent 5
Team X
Jenkins Agent 3
Jenkins Agent 6
. . .
Jenkins Master

13. © 2016 Mesosphere, Inc. All Rights Reserved.
MULTIPLE JENKINS MASTERS
Team A Jenkins
. . .
Jenkins Agent 1
Jenkins Agent 2
. . .
Jenkins Agent N
Team B Jenkins
Jenkins Agent 1
Jenkins Agent 2
. . .
Jenkins Agent N
Team X Jenkins
Jenkins Agent 1
Jenkins Agent 2
. . .
Jenkins Agent N

14. © 2016 Mesosphere, Inc. All Rights Reserved.
MULTIPLE JENKINS MASTERS
Team A Jenkins
. . .
Jenkins Agent 1
Jenkins Agent 2
. . .
Jenkins Agent N
Team B Jenkins
Jenkins Agent 1
Jenkins Agent 2
. . .
Jenkins Agent N
Team X Jenkins
Jenkins Agent 1
Jenkins Agent 2
. . .
Jenkins Agent N

15. © 2016 Mesosphere, Inc. All Rights Reserved.
MULTIPLE JENKINS MASTERS → STATIC PARTITIONING
Team A Jenkins
. . .
Jenkins Agent 1
Jenkins Agent 2
. . .
Jenkins Agent N
Team B Jenkins
Jenkins Agent 1
Jenkins Agent 2
. . .
Jenkins Agent N
Team X Jenkins
Jenkins Agent 1
Jenkins Agent 2
. . .
Jenkins Agent N
90% Utilized
(Normal)
140% Utilized
(40 builds in queue)
0% Utilized
(Idle)

16. © 2016 Mesosphere, Inc. All Rights Reserved.
JENKINS ON MESOS
Team A Jenkins . . .
Mesos Agent 1
Mesos Agent 4
Mesos Agent N
Team B Jenkins
Mesos Agent 2
Mesos Agent 5
Team X Jenkins
Mesos Agent 3
Mesos Agent 6
. . .
Mesos Master

17. © 2016 Mesosphere, Inc. All Rights Reserved.
DEMO

18. © 2015 Mesosphere, Inc. All Rights Reserved. 18
FUTURE
WORK
Tighter CI/CD integration
with DCOS
● App deployment
● Service discovery

19. © 2016 Mesosphere, Inc. All Rights Reserved.
Q & A
@rogerignazio
roger@mesosphere.com
mesosphere.com/product
mesos.apache.org
github.com/jenkinsci/mesos-plugin