1. An Almost Complete Continuous Delivery
Pipeline Including Configuration Management
Ulf Månsson @ulfmansson
Recorded Future AB

2. Why almost complete?
You never complete
It's not easy

4. Recorded Future

6. We wanted
●
Quick feedback
●
Build all artifacts
●
Upload all build artifacts to s3
●
Trust in deploying every successful build
●
Test everything

7. What do we mean test everything?
●
Unit tests
●
Integration tests
●
End-to-end test
●
UI test
●
Performance test
●
Cookbook tests
●
Deployment tests
●
Configuration tests

11. No champagne

14. Long journey
●
The “build master” left the company
●
Lacking focus
●
Very long build times
●
Use the build pipe line plugin

15. Continues delivery embrace devops
●
Everyone should dare to touch stuff
●
Developers touch ops code
●
Operation touch application code
●
Everyone should be able to deploy

16. operations@recordedfuture
●
Quite big team
●
Infra code
●
Build pipe line
●
Service and on premise installations
●
End to end tests
●
Performance tests
●
Amazon operation costs

17. Our system
●
●
●
●
●
A number of major infrastructure
components – ElasticSearch,
RabbitMQ etc
About 45 different processes
Development in Java, Python, Scala
and Ruby
External API and web servers
Developed by 5 different teams in
Sweden, US and Ukraine

18. Subversion
For good and bad
All code is code
Keep the code in the same repository
Configurations also in subversion
Treat all code in the same way, it doesn’t matter if it’s application
code, infra code, test code or whatever

21. Feature toggling to our help
●
No branching
●
One trunk
●
Took some time train developers
●
●
Now we try to train infra coders as well – Feature
toggling in Chef
Read more by Jez Humble
http://continuousdelivery.com/2011/07/on-dvcs-contin
uous-integration-and-feature-branches/

22. Jenkins
and his slaves...

23. Build Flow Plugin ­ the thing!
out.println "Start"
revision=build.properties["environment"]["SVN_REVISION"]
out.print "Original revision for this build is "
out.println revision
parallel (
{build("master-build-flow",SVN_REVISION: revision)},
{ ignore(FAILURE) {build("master-infrastructure_test",SVN_REVISION: revision)}},
{build("master-fast_track_upload_cookbooks-flow",SVN_REVISION: revision)},
{ ignore(FAILURE){build("master-infrastructure_lint",SVN_REVISION: revision)}}
)
build( "master-fast_track_deploy-flow", SVN_REVISION: revision)
build( "master-fast_track_integration_test_harvesting-flow", SVN_REVISION: revision)
parallel (
{ build("master-fast_track_test_ff-flow", SVN_REVISION: revision) },
{ build("master-fast_track_test_ie-flow", SVN_REVISION: revision) }
)

24. Build Graph View Plugin

25. Jenkins text finder
puts "The build is UNSTABLE" if failed

26. S3
●
Store artifacts
●
Simple
●
Control
●
Reliable
●
Accessible

29. Test tools
●
Test kitchen
●
Server spec
●
Maven
●
Cucumber
●
Rspec
●
Load impact
●
Selenium

30. Build tools
Then: Only Maven
Now: Mix of Rake and Maven

31. Environment cookbook
# Cookbook Name:: env
# Recipe:: fast_track_1 (test environment)
include_recipe 'env::prod'
node.default[:rabbitmq][:version] = "3.1.5"
# Interactive
node.default['rfwebapp']['interactive']['java_opts'] = '­Xms1G
­Xmx1G'
In env::default include_recipe "env::#{node.chef_environment}"

32. Why environment cookbook
●
●
●
●
We can version handle it
We are sure that the test environment will get the
same settings as production environment
On premise installation use chef-solo
When we upload the cookbooks the environment
settings will be uploaded as well

33. Our master build pipe line

34. Master­flow
●
Only the build-flow plugin, describes the build
●
Triggered by subversion polls
●
Co-ordinate the sub build processes
●
Only one at a time running

35. Master build flow
●
Compiles all application code
●
Run unit tests
●
Builds application artifacts
●
Uploads artifacts to S3

36. Upload cookbooks
●
Upload cookbooks to chef server for the build test
environment - fast_track

37. Infrastructure lint
●
Food critic of course
●
Tailor – nope
●
Rubocop – probably as next step

38. Fast track...
●
All-in-one box
●
Will deploy,configure and launch all processes
●
Configure infrastructure components
●
Spin-up EC2 instance

39. Fast track cont...
●
Use Chef server
●
Deploy all processes with Chef
●
Instance used for following tests
●
Perfect case for Docker

40. Infrastructure test
●
Test-kitchen tests
●
Mainly server-spec
●
Run in lxc with vagrant-lxc
●
Run our “default” suite in test-kitchen
●
Disabled in the Master-flow at the moment

41. End­to­end test
●
Cucumber tests
●
Populate system with data
●
Tests via our API
●
Check process log database to ensure all steps has
been run

42. Browser tests IE and Firefox
●
Run via Selenium
●
Test browser functionality
●
Upload screen snapshots from the tests to s3

43. Supporting tools

44. Bygge – an overview

45. Logstash
●
Collect all log from the fast track run
●
Used for problem solving
●
Easy to access with a link from Bygge
●
We also use it in production

46. Sensu ­ monitoring
●
Almost same monitoring as in production of each fast
truck run
●
Easy to access monitoring dashboard and see errors
●
Will fail builds when we have checks that fails

47. Lessons learned

48. Opscode jenkins cookbook
●
Really good
●
Providers for slaves, jobs, plugins
●
Updated

49. Run parallel
Many and smaller build steps

51. From 150 minutes to 45 minutes in 3 days
●
SSD instances for Jenkins
●
SSD test instances
●
Build slaves
●
Parallel execution
●
Pre upload

53. Infrastructure
Logstash
Jenkins
Sensu
Fast track instances
Fast track instances
Fast track instances

54. I would like to share experience