This document discusses eBay's use of Apache Mesos, Docker, and Jenkins to improve their continuous integration (CI) solution. They were facing issues with thousands of underutilized VMs for running builds and managing different OSes and versions. Their new solution uses Mesos for cluster management, Marathon to launch Jenkins masters, and the Jenkins Mesos plugin to launch Dockerized build slaves. This allows them to standardize on Docker containers to isolate dependencies and OS flavors, improving utilization and maintenance while enabling polyglot builds.

1. Delivering eBay's CI Solution
with Apache Mesos & Docker
Mohit Soni
Software Engineer
Ashish Hunnargikar
Software Engineer

2. Why we did it....
• Thousands of VMs dedicated to run Jenkins!
• Utilization is less than 5% 
• VMs idle most of time (no one runs CI builds frequently)
VM sprawl and poor resource utilization
• Polyglot application stacks (Java, C++, Node.js, Python, Scala)
• Different OS flavors (Ubuntu & RHEL)
• Software version conflicts
• Special hardware requirements
Build slave management and maintenance is a nightmare
1

3. And here’s what we did....
•Mesos for cluster management
•Marathon to launch Jenkins masters
•Jenkins Mesos-plugin to launch build slaves
•Build slaves Dockerized as well as regular
2

4. 3

5. 4

6. Why use Docker containers?
• Managing varied slave flavors in a large cluster is hard!
• Docker effectively isolates dependencies inside the container
• Docker allows host OS to remain independent of workload
dependencies
Why build Docker-in-Docker?
• Run Docker build/push/pull inside a Docker container
• Eliminates redundant CI build dependency downloads
• Helps to enforce a heterogeneous Mesos slave cluster
5

7. Basic Use Case
Build C++ & Java applications on both RHEL & Ubuntu
Jenkins
Slave Job
Package
Repository
Jenkins
Master
docker run Upload
Ubuntu Mesos Slave
/bin/bash C++/Java
Build
Scripts
RHEL
Docker
Container
Initialize
• Mesos slave host server is running Ubuntu
• Jenkins job is running builds in a RHEL or Ubuntu container
• Application dependencies can be preinstalled in base images OR can be
downloaded during build time
6

8. Advanced Use Case
Build RHEL & Ubuntu Docker images
• Mesos slave host server is running Ubuntu
• Jenkins job is running inside a Docker container
• Jenkins job is building a RHEL/Ubuntu Docker image from an application
Dockerfile
7

9. Enabling the Docker CI workflow using Jenkins & Mesos
8

10. Initial Build Strategy
Individual Docker-in-Docker Setup
Complete isolation
Docker daemon running inside the Docker container requires privileged mode
I/O overhead of downloading and duplicating all the AUFS layers
9

11. Optimized Build Strategy
Shared Docker-in-Docker Setup
Docker daemon not required inside the container so no privileged mode
Single reusable AUFS image layer cache
Redundant network and disk I/O eliminated for layer downloads
Lack of isolation for layers
10

12. For more information visit:
1. Apache Mesos: http://mesos.apache.org
2. Jenkins Mesos Plugin: https://github.com/jenkinsci/mesos-plugin
3. Docker: https://www.docker.io
4. Marathon Framework: https://github.com/mesosphere/marathon
Q&A
http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-
mesos-part-i/
http://www.ebaytechblog.com/2014/05/12/delivering-ebays-ci-solution-with-apache-
mesos-part-ii/
www.ebaytechblog.com
11