Building Cloud Tools for Netflix with Jenkins

Building and Deploying Netﬂix in the Cloud @bmoyles @garethbowles #netﬂixcloud

Who Are These Guys?Brian Moyles Gareth Bowles

What We BuildLarge number of loosely-coupled Java Web ServicesCommon code in libraries that can be shared acrossappsEach service is “baked” - installed onto a base AmazonMachine Image and then created as a new AMI ...... and then deployed into a Service Cluster (a set ofAuto Scaling Groups running a particular service)

Getting Built

Build Pipeline Artifactory yum libraries Jenkins CBF steps resolve compile publish report sync check build test sourcePerforce GitHub

build.xml<project name="helloworld"> <import file="../../../Tools/build/webapplication.xml"/></project>ivy.xml<info organisation="netflix" module="helloworld"> <publications> <artifact name="helloworld" type="package" e:classifier="package" ext="tgz"/> <artifact name="helloworld" type="javadoc" e:classifier="javadoc" ext="jar"/> </publications> <dependencies> <dependency org="netflix" name="resourceregistry" rev="latest.${input.status}" conf="compile"/> <dependency org="netflix" name="platform" rev="latest.${input.status}" conf="compile" /> ...

Jenkins at Netﬂix

Jenkins Statistics1600 job deﬁnitions, 50% SCM triggered2000 builds per dayCommon Build Framework updates trigger 800 rebuilds;by scaling up to 20 cloud slaves we can complete theﬂood of new builds in 30 minutes2TB of build data

Jenkins Architecturex86_64 slave 11 x86_64 slave 1 x86_64 slave buildnode01 1 x86_64 slave Standard buildnode01 custom slaves buildnode01 buildnode01 custom slaves custom slaves slave group misc. architecture custom slaves misc. architecture misc. architecture custom slaves Amazon Linux Single Master misc. architecture m1.xlarge misc. architecture Ad-hoc slaves Red Hat Linux 2x quad core x86_64 misc. O/S & architectures 26G RAMx86_64 slave 11 x86_64Custom x86_64slave 1 slave buildnode01 ~40 custom slaves buildnode01 slave group buildnode01 maintained by product Amazon Linux teams various us-west-1 VPC Netflix data center Netflix data center and office

Other Uses of JenkinsMonitoring of our test and production Cassandra clustersAutomated integration tests, including bake and deployProduction bake and deploymentHousekeeping of the build / deploy infrastructure: Reap unreferenced artifacts in Artifactory Disable Jenkins jobs with no recent successful builds Mark Jenkins builds as permanent if they are used by an active deployment in prod or test Alert owners when slaves get disconnected

Jenkins Scaling ChallengesFlood of simultaneous builds can quickly exhaust all buildexecutors and clog the pipelineFlood of simultaneous builds can hammer rest of theinfrastructure (especially Artifactory)Making global changes to all jobsSome plugins don’t scale to our number of jobs / buildsHard to test every job before upgrading master or pluginsLarge amount of state encapsulated in build data makesrestoring from backup time consuming

Netﬂix Extensions to Jenkins Job DSL plugin: allow jobs to be set up with minimal deﬁnition, using templates and a Groovy-based DSL. Housekeeping and maintenance processes implemented as Jenkins jobs, system Groovy scripts

TheDynaSlavePluginOur cloud-basedarmy of build nodes

The DynaSlave PluginGenesisOriginal build ﬂeet: 15 VMs on datacenter hardware, 8GRAM, single vCPU, 2 executors per nodeMany jobs build on SCM change. Changes to ourcommon build framework create massive thunderingherd since everything depends on it.Ask for more VMs? Modify CBF less frequently?

The DynaSlave PluginWhat We WantedLeverage our extensive AWS infrastructure, tooling, andexperienceNo manual ﬁddling with machines once they launchQuick and easy to maintain a ﬁxed pool of slave nodesthat can grow/shrink to meet build demand

The DynaSlave PluginWhat We HaveExposes a new endpoint in Jenkins that EC2 instancesin VPC use for registrationAllows a slave to name itself, label itself, tell Jenkinshow many executors it can supportEC2 == Ephemeral. Disconnected nodes that are gonefor > 30 mins are reapedSizing handled by EC2 ASGs, tweaks passed throughvia user data (labels, names, etc)

The DynaSlave PluginWhat’s NextDynamic resource management: have Jenkins respondto build demand and manage its own slave poolsSlave groups: Allows us to create specialized (andisolated from the genpop) pools of build nodesRefresh mechanism for slave tools (JDKs, Ant versions,etc)Enhanced security/registration of nodesGive it back to the community (watchtechblog.netﬂix.com!)

Further Readinghttp://techblog.netflix.comhttp://www.slideshare.net/adrianco @netflixosshttps://github.com/netflixhttp://jobs.netflix.com

Thank you @bmoyles @garethbowles

Thank youQuestions? @bmoyles @garethbowles

http://www.scoop.it	957
https://twitter.com	17
https://opsthink.atlassian.net	5
http://martin-reviews.blogspot.com	2
http://us-w1.rockmelt.com	2
http://tweetedtimes.com	1
http://www.plurk.com	1

Building Cloud Tools for Netflix with Jenkins

by Gareth Bowles on May 18, 2012

Accessibility

Categories

Upload Details

Usage Rights

7 Embeds 985

Statistics

Building Cloud Tools for Netflix with Jenkins Presentation Transcript