Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How Jenkins Builds the Netflix Global Streaming Service


Published on

Jenkins User Conference SF 2012 presentation "How Jenkins Builds the Netflix Global Streaming Service"

Published in: Technology

How Jenkins Builds the Netflix Global Streaming Service

  1. How Jenkins Builds the Netflix Global Streaming Service @bmoyles @garethbowles #netflixcloud
  2. The Engineering Tools TeamMake build, test and deploy simple and pain-freeVarious skills - developers, build/release engineers,devops guysCI infrastructureCommon Build FrameworkBakeryAsgard
  3. Jenkins History Jenkins (and H***** before it) in use at Netflix since 2009 Rapid acceptance and increase in usage, thanks to: Jenkins’ flexibility and ease of use Our Common Build Framework Quantity, size and variety of jobs means we need automation and extensions
  4. What We BuildLarge number of loosely-coupled Java Web ServicesCommon code in libraries that can be shared acrossappsEach service is “baked” - installed onto a base AmazonMachine Image and then created as a new AMI ...... and then deployed into a Service Cluster (a set ofAuto Scaling Groups running a particular service)So our unit of deployment is the AMI
  5. Getting Built
  6. Build Pipeline ASGs Artifactory yum Bakery Asgard Jenkins Jars/WARs RPMs AMIs sync resolve compile publish bake deploy Common Build Framework sourcePerforce GitHub
  7. Jenkins Setup for a New Build Job
  8. Ant Setup for a New Build Jobbuild.xml<project name="helloworld"> <import file="../../../Tools/build/webapplication.xml"/></project>ivy.xml<info organisation="netflix" module="helloworld"> <publications> <artifact name="helloworld" type="package" e:classifier="package" ext="tgz"/> <artifact name="helloworld" type="javadoc" e:classifier="javadoc" ext="jar"/> </publications> <dependencies> <dependency org="netflix" name="resourceregistry" rev="latest.${input.status}" conf="compile"/> <dependency org="netflix" name="platform" rev="latest.${input.status}" conf="compile" /> ...
  9. Jenkins at Netflix
  10. Not Just a Build ServerMonitoring of our Cassandra clustersAutomated integration tests, including bakeand deployHousekeeping and monitoring of the build / deploy infrastructure our AWS environment
  11. Jenkins Statistics1600 job definitions, 50% SCM triggered2000 builds per dayCommon Build Framework changes trigger 800 builds2TB of build dataOver 100 plugins
  12. Current Jenkins Architecture (One Master to Build Them All)x86_64 slave 11 x86_64 slave 1 x86_64 slave buildnode01 1 x86_64 slave Standard buildnode01 custom slaves buildnode01 buildnode01 custom slaves custom slaves slave group misc. architecture custom slaves misc. architecture misc. architecture custom slaves Amazon Linux Single Master misc. architecture m1.xlarge misc. architecture Ad-hoc slaves Red Hat Linux 2x quad core x86_64 misc. O/S & architectures 26G RAMx86_64 slave 11 x86_64Custom x86_64slave 1 slave buildnode01 ~40 custom slaves buildnode01 slave group buildnode01 maintained by product Amazon Linux teams variousAWS us-west-1 VPC Netflix data center Netflix DC and office
  13. Scaling Challenges -and How We Met Them
  14. Jenkins Scaling ChallengesThundering herds clogging Jenkins, downstreamsystemsMaking global changes across jobsMaking sure plugins can handle 1600 jobsTesting new Jenkins and plugin versionsBackups
  15. New Architecture (Cloudy with a Chance of Builds) x86_64 x86_64 slave slave 1 Build custom slave1 buildnode01 group Build Master custom slaves Amazon Linux Ad-hoc slaves slaves CentOS or Ubuntu misc. m1.xlarge misc. O/S & misc. m2.2xlarge architectures (4 core / 32GB) x86_64x86_64 slave Custom slave 1 1 slave groups custom buildnode01 Amazon Linux custom Test Master slaves various Ad-hoc slaves slaves CentOS or Ubuntu misc. misc. O/S & misc. m2.2xlarge architectures (4 core / 32GB) x86_64x86_64 slave slave 1 Test slave 1buildnode01 groupAmazon Linux m1.xlarge AWS us-west-1 VPC AWS and Netflix office
  16. Deploying a New Master EBS Live Test EBS master master ENI Standby 1 EBS Standby 2EBS Active ASG Test ASG Jenkins Master cluster
  17. Building a New MasterJenkins build watches and downloads the new JenkinsWAR fileDownload latest versions of our installed plugins (from alist stored externally)Create an RPM containing the Jenkins webappBake an AMI with the RPM installed under NetflixTomcat
  18. Job Configuration ManagementConfiguration Slicing plugin Great for the subset of config options that it handles Don’t want to extend it for each new optionJob DSL plugin Set up new jobs with minimal definition, using templates and a Groovy-based DSL Change the template once to update all the jobs that use it
  19. Old and busted... The new hotness... etc.
  20. More MaintenanceSystem Groovy scripts to clean up old job configs andassociated artifacts: Disable jobs that haven’t built successfully for 60 days Set # of builds to keep if unset Mark builds permanent if installed on active instances Delete unreferenced JARs from Artifactory
  21. Keeping Slaves BusySystem Groovy scripts to monitor slave status andreport on problemsThe Dynaslave plugin: slave self-registration
  22. Further Reading @netflixoss
  23. Thank You To Our Sponsors
  24. Thank you @bmoyles @garethbowles
  25. Thank youQuestions? @bmoyles @garethbowles