An almost complete continuous delivery pipeline including configuration management
Upcoming SlideShare
Loading in...5

An almost complete continuous delivery pipeline including configuration management



How we have created a build pipeline for continous delivery at Recorded Future. This includes also test of Chef cookbooks and configuration.

How we have created a build pipeline for continous delivery at Recorded Future. This includes also test of Chef cookbooks and configuration.
Presentation at
Config Management Camp 2014



Total Views
Views on SlideShare
Embed Views



3 Embeds 59 42
http://bde-confluence 14 3



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.


13 of 3 Post a comment

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    An almost complete continuous delivery pipeline including configuration management An almost complete continuous delivery pipeline including configuration management Presentation Transcript

    • An Almost Complete Continuous Delivery  Pipeline Including Configuration Management Ulf Månsson @ulfmansson Recorded Future AB
    • Why almost complete? You never complete It's not easy
    • Recorded Future
    • We wanted ● Quick feedback ● Build all artifacts ● Upload all build artifacts to s3 ● Trust in deploying every successful build ● Test everything
    • What do we mean test everything? ● Unit tests ● Integration tests ● End-to-end test ● UI test ● Performance test ● Cookbook tests ● Deployment tests ● Configuration tests
    • No champagne
    • Long journey ● The “build master” left the company ● Lacking focus ● Very long build times ● Use the build pipe line plugin
    • Continues delivery embrace devops ● Everyone should dare to touch stuff ● Developers touch ops code ● Operation touch application code ● Everyone should be able to deploy
    • operations@recordedfuture ● Quite big team ● Infra code ● Build pipe line ● Service and on premise installations ● End to end tests ● Performance tests ● Amazon operation costs
    • Our system ● ● ● ● ● A number of major infrastructure components – ElasticSearch, RabbitMQ etc About 45 different processes Development in Java, Python, Scala and Ruby External API and web servers Developed by 5 different teams in Sweden, US and Ukraine
    • Subversion For good and bad All code is code Keep the code in the same repository Configurations also in subversion Treat all code in the same way, it doesn’t matter if it’s application code, infra code, test code or whatever
    • Feature toggling to our help ● No branching ● One trunk ● Took some time train developers ● ● Now we try to train infra coders as well – Feature toggling in Chef Read more by Jez Humble uous-integration-and-feature-branches/
    • Jenkins and his slaves...
    • Build Flow Plugin ­ the thing! out.println "Start"["environment"]["SVN_REVISION"] out.print "Original revision for this build is " out.println revision parallel ( {build("master-build-flow",SVN_REVISION: revision)}, { ignore(FAILURE) {build("master-infrastructure_test",SVN_REVISION: revision)}}, {build("master-fast_track_upload_cookbooks-flow",SVN_REVISION: revision)}, { ignore(FAILURE){build("master-infrastructure_lint",SVN_REVISION: revision)}} ) build( "master-fast_track_deploy-flow", SVN_REVISION: revision) build( "master-fast_track_integration_test_harvesting-flow", SVN_REVISION: revision) parallel ( { build("master-fast_track_test_ff-flow", SVN_REVISION: revision) }, { build("master-fast_track_test_ie-flow", SVN_REVISION: revision) } )
    • Build Graph View Plugin
    • Jenkins text finder puts "The build is UNSTABLE" if failed
    • S3 ● Store artifacts ● Simple ● Control ● Reliable ● Accessible
    • Test tools ● Test kitchen ● Server spec ● Maven ● Cucumber ● Rspec ● Load impact ● Selenium
    • Build tools Then: Only Maven Now: Mix of Rake and Maven
    • Environment cookbook # Cookbook Name:: env # Recipe:: fast_track_1  (test environment) include_recipe 'env::prod' node.default[:rabbitmq][:version] = "3.1.5" # Interactive node.default['rfwebapp']['interactive']['java_opts'] = '­Xms1G  ­Xmx1G' In env::default include_recipe "env::#{node.chef_environment}"
    • Why environment cookbook ● ● ● ● We can version handle it We are sure that the test environment will get the same settings as production environment On premise installation use chef-solo When we upload the cookbooks the environment settings will be uploaded as well
    • Our master build pipe line
    • Master­flow ● Only the build-flow plugin, describes the build ● Triggered by subversion polls ● Co-ordinate the sub build processes ● Only one at a time running
    • Master build flow ● Compiles all application code ● Run unit tests ● Builds application artifacts ● Uploads artifacts to S3
    • Upload cookbooks ● Upload cookbooks to chef server for the build test environment - fast_track
    • Infrastructure lint ● Food critic of course ● Tailor – nope ● Rubocop – probably as next step
    • Fast track... ● All-in-one box ● Will deploy,configure and launch all processes ● Configure infrastructure components ● Spin-up EC2 instance
    • Fast track cont... ● Use Chef server ● Deploy all processes with Chef ● Instance used for following tests ● Perfect case for Docker
    • Infrastructure test ● Test-kitchen tests ● Mainly server-spec ● Run in lxc with vagrant-lxc ● Run our “default” suite in test-kitchen ● Disabled in the Master-flow at the moment
    • End­to­end test ● Cucumber tests ● Populate system with data ● Tests via our API ● Check process log database to ensure all steps has been run
    • Browser tests IE and Firefox ● Run via Selenium ● Test browser functionality ● Upload screen snapshots from the tests to s3
    • Supporting tools
    • Bygge – an overview
    • Logstash ● Collect all log from the fast track run ● Used for problem solving ● Easy to access with a link from Bygge ● We also use it in production
    • Sensu ­ monitoring ● Almost same monitoring as in production of each fast truck run ● Easy to access monitoring dashboard and see errors ● Will fail builds when we have checks that fails
    • Lessons learned
    • Opscode jenkins cookbook ● Really good ● Providers for slaves, jobs, plugins ● Updated
    • Run parallel Many and smaller build steps
    • From 150 minutes to 45 minutes in 3 days ● SSD instances for Jenkins ● SSD test instances ● Build slaves ● Parallel execution ● Pre upload
    • Infrastructure Logstash Jenkins Sensu Fast track instances Fast track instances Fast track instances
    • I would like to share experience