The document discusses continuous integration (CI) practices at eBay and PayPal. It describes their previous model of having thousands of individual Jenkins instances running on separate VMs, which led to poor resource utilization and high costs. Their new architecture leverages Apache Mesos and containers to run lightweight Jenkins masters and slaves across a cluster, improving efficiency while maintaining user experience. Key contributions to the Jenkins Mesos plugin from eBay are also outlined.

1. Jenkins User Conference San Francisco #jenkinsconf
How to train your
Jenkins? … to Use Your Resources
Effectively and Get More Value for Each Buck Spent on
Resources
Manivannan Selvaraj & Soma Oruganti
eBay Inc.
ww.ebay.com
October 23, 2014
#jenkinsconf

2. Jenkins User Conference San Francisco #jenkinsconf
At eBay Inc. we take CI very seriously

3. Jenkins User Conference San Francisco #jenkinsconf
Background - CI in PayPal
• One Ginormous Jenkins Instance, customized for PayPal
• 40,000 runnable jobs
• 12,000 builds are run each day
• Java, Node, Python, and C++ stacks, Fortify and Freestyle jobs too
• Unit test and deploy flows
• One Jenkins master with a pool of a hundred Jenkins slaves
• Specialized “Monster Machine” stages build 40,000,000 lines of code in
a half hour

4. Jenkins User Conference San Francisco #jenkinsconf
Background - CI in eBay Marketplaces
Jenkins CI
Wo
rks
pac
e
Jenkins CI
Wo
rks
pac
e
Jenkins CI
Wo
rks
pac
e
Jenkins CI
Wo
rks
pac
e
Jenkins CI
Wo
rks
pac
e
Jenkins CI
Wo
rks
pac
e
Jenkins CI
Wo
rks
pac
e
Jenkins CI
Wo
rks
pac
e
Jenkins CI
Wo
rks
pac
e
Jenkins CI
Wo
rks
pac
e
Jenkins CI
Wo
rks
pac
e
Jenkins CI
Wo
rks
pac
e
Jenkins CI
Wo
rks
pac
e
Jenkins CI
Wo
rks
pac
e
Jenkins CI
Wo
rks
pac
e
Jenkins CI
Wo
rks
pac
e
Jenkins CI
Wo
rks
pac
e
Jenkins CI
Wo
rks
pac
e
Jenkins CI
Wo
rks
pac
e
Jenkins CI
Wo
rks
pac
e
Jenkins CI
Jenkins CI
Wo
rks
pac
e
Jenkins CI
Wo
rks
pac
e
Jenkins CI
Wo
rks
pac
e
• Individual VMs running Jenkins for each Application
• 2500+ VMs in use
Wo
rks
pac
e
• Each Jenkins would have 2 executor on master to perform builds

5. Jenkins User Conference San Francisco #jenkinsconf
Individual Jenkins VMs
Pros
Can add their own plug-ins
Can upgrade to various versions
of Jenkins
Issues are isolated to one
instance
Cons
Poor resource utilization - < 10%
Cost of maintaining 2500+ VMs –
Millions of $ every year
Provisioning VMs takes several
minutes
Individual instances have local
storage of limited size
CI’s can disappear, losing all
data and build history

6. Jenkins User Conference San Francisco #jenkinsconf
New CI Architecture - Challenges
• Put a smile on Dev-Ops
• Optimal utilization of resources
• Preserve user experience
• High availability
• Self healing infrastructure

7. Jenkins User Conference San Francisco #jenkinsconf
Light weight Jenkins
Apache Mesos Cloud
…
Light weight Jenkins for each user
Developer 1 Developer 2 Developer n
0.4 CPU
512 MB mem

8. Jenkins User Conference San Francisco #jenkinsconf
Mesos Architecture

9. Jenkins User Conference San Francisco #jenkinsconf
Build slaves - Jenkins Mesos Plugin
• Jenkins Mesos plugin community is very active
• We have contributed enhancements/patches to the plugin

10. Jenkins User Conference San Francisco #jenkinsconf
CI Application life cycle
CIAAS
Marathon
Mesos Master
Mesos Cluster
NFS
Request for CI
Zookeeper
Jenkins Master Jenkins Slave

11. Jenkins User Conference San Francisco #jenkinsconf
Jenkins Mesos Plugin - our Contributions
• Custom workspace
• Multiple flavors of slaves
• Access to workspace
• Enable Maven jobs
• Disconnect of slaves after idle timeout

12. Jenkins User Conference San Francisco #jenkinsconf

13. Jenkins User Conference San Francisco #jenkinsconf
Thank You To Our Sponsors
Platinum Gold
Silver Corporate