An almost complete continuous delivery pipeline including configuration management

2,703 views

Published on

How we have created a build pipeline for continous delivery at Recorded Future. This includes also test of Chef cookbooks and configuration.
Presentation at
Config Management Camp 2014

Published in: Technology
3 Comments
7 Likes
Statistics
Notes
No Downloads
Views
Total views
2,703
On SlideShare
0
From Embeds
0
Number of Embeds
92
Actions
Shares
0
Downloads
54
Comments
3
Likes
7
Embeds 0
No embeds

No notes for slide

An almost complete continuous delivery pipeline including configuration management

  1. 1. An Almost Complete Continuous Delivery  Pipeline Including Configuration Management Ulf Månsson @ulfmansson Recorded Future AB
  2. 2. Why almost complete? You never complete It's not easy
  3. 3. Recorded Future
  4. 4. We wanted ● Quick feedback ● Build all artifacts ● Upload all build artifacts to s3 ● Trust in deploying every successful build ● Test everything
  5. 5. What do we mean test everything? ● Unit tests ● Integration tests ● End-to-end test ● UI test ● Performance test ● Cookbook tests ● Deployment tests ● Configuration tests
  6. 6. No champagne
  7. 7. Long journey ● The “build master” left the company ● Lacking focus ● Very long build times ● Use the build pipe line plugin
  8. 8. Continues delivery embrace devops ● Everyone should dare to touch stuff ● Developers touch ops code ● Operation touch application code ● Everyone should be able to deploy
  9. 9. operations@recordedfuture ● Quite big team ● Infra code ● Build pipe line ● Service and on premise installations ● End to end tests ● Performance tests ● Amazon operation costs
  10. 10. Our system ● ● ● ● ● A number of major infrastructure components – ElasticSearch, RabbitMQ etc About 45 different processes Development in Java, Python, Scala and Ruby External API and web servers Developed by 5 different teams in Sweden, US and Ukraine
  11. 11. Subversion For good and bad All code is code Keep the code in the same repository Configurations also in subversion Treat all code in the same way, it doesn’t matter if it’s application code, infra code, test code or whatever
  12. 12. Feature toggling to our help ● No branching ● One trunk ● Took some time train developers ● ● Now we try to train infra coders as well – Feature toggling in Chef Read more by Jez Humble http://continuousdelivery.com/2011/07/on-dvcs-contin uous-integration-and-feature-branches/
  13. 13. Jenkins and his slaves...
  14. 14. Build Flow Plugin ­ the thing! out.println "Start" revision=build.properties["environment"]["SVN_REVISION"] out.print "Original revision for this build is " out.println revision parallel ( {build("master-build-flow",SVN_REVISION: revision)}, { ignore(FAILURE) {build("master-infrastructure_test",SVN_REVISION: revision)}}, {build("master-fast_track_upload_cookbooks-flow",SVN_REVISION: revision)}, { ignore(FAILURE){build("master-infrastructure_lint",SVN_REVISION: revision)}} ) build( "master-fast_track_deploy-flow", SVN_REVISION: revision) build( "master-fast_track_integration_test_harvesting-flow", SVN_REVISION: revision) parallel ( { build("master-fast_track_test_ff-flow", SVN_REVISION: revision) }, { build("master-fast_track_test_ie-flow", SVN_REVISION: revision) } )
  15. 15. Build Graph View Plugin
  16. 16. Jenkins text finder puts "The build is UNSTABLE" if failed
  17. 17. S3 ● Store artifacts ● Simple ● Control ● Reliable ● Accessible
  18. 18. Test tools ● Test kitchen ● Server spec ● Maven ● Cucumber ● Rspec ● Load impact ● Selenium
  19. 19. Build tools Then: Only Maven Now: Mix of Rake and Maven
  20. 20. Environment cookbook # Cookbook Name:: env # Recipe:: fast_track_1  (test environment) include_recipe 'env::prod' node.default[:rabbitmq][:version] = "3.1.5" # Interactive node.default['rfwebapp']['interactive']['java_opts'] = '­Xms1G  ­Xmx1G' In env::default include_recipe "env::#{node.chef_environment}"
  21. 21. Why environment cookbook ● ● ● ● We can version handle it We are sure that the test environment will get the same settings as production environment On premise installation use chef-solo When we upload the cookbooks the environment settings will be uploaded as well
  22. 22. Our master build pipe line
  23. 23. Master­flow ● Only the build-flow plugin, describes the build ● Triggered by subversion polls ● Co-ordinate the sub build processes ● Only one at a time running
  24. 24. Master build flow ● Compiles all application code ● Run unit tests ● Builds application artifacts ● Uploads artifacts to S3
  25. 25. Upload cookbooks ● Upload cookbooks to chef server for the build test environment - fast_track
  26. 26. Infrastructure lint ● Food critic of course ● Tailor – nope ● Rubocop – probably as next step
  27. 27. Fast track... ● All-in-one box ● Will deploy,configure and launch all processes ● Configure infrastructure components ● Spin-up EC2 instance
  28. 28. Fast track cont... ● Use Chef server ● Deploy all processes with Chef ● Instance used for following tests ● Perfect case for Docker
  29. 29. Infrastructure test ● Test-kitchen tests ● Mainly server-spec ● Run in lxc with vagrant-lxc ● Run our “default” suite in test-kitchen ● Disabled in the Master-flow at the moment
  30. 30. End­to­end test ● Cucumber tests ● Populate system with data ● Tests via our API ● Check process log database to ensure all steps has been run
  31. 31. Browser tests IE and Firefox ● Run via Selenium ● Test browser functionality ● Upload screen snapshots from the tests to s3
  32. 32. Supporting tools
  33. 33. Bygge – an overview
  34. 34. Logstash ● Collect all log from the fast track run ● Used for problem solving ● Easy to access with a link from Bygge ● We also use it in production
  35. 35. Sensu ­ monitoring ● Almost same monitoring as in production of each fast truck run ● Easy to access monitoring dashboard and see errors ● Will fail builds when we have checks that fails
  36. 36. Lessons learned
  37. 37. Opscode jenkins cookbook ● Really good ● Providers for slaves, jobs, plugins ● Updated
  38. 38. Run parallel Many and smaller build steps
  39. 39. From 150 minutes to 45 minutes in 3 days ● SSD instances for Jenkins ● SSD test instances ● Build slaves ● Parallel execution ● Pre upload
  40. 40. Infrastructure Logstash Jenkins Sensu Fast track instances Fast track instances Fast track instances
  41. 41. I would like to share experience

×