Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Automated deployments 
for mission-critical 
financial services 
Jeremy Alons 
Systems Engineer, Spot Trading LLC 
Views a...
A case study in 
continuous delivery
Why? 
• Previously, deployments were time consuming 
• Took hours of developer time, sometimes spread 
over weeks 
• Manua...
Our goals 
• Rapid feedback 
• Tie configuration to deployment 
• Be highly parallelized, if possible 
• Infrastructure as...
Challenges 
• Lots of unique software 
• Differences force complexity 
• Misused the tools we had originally 
• Required a...
Steps: 
Build 
Deploy 
Config 
Run
Steps: 
Build 
Deploy! 
Config 
Run
Steps: 
Build 
Deploy 
Config! 
Run
Build: Bamboo (or Jenkins, travis-ci, etc) executes 
commands 
i.e. gcc -o foo foo.c 
sh tests.sh 
etc
Source code in. 
Artifacts out.
Now what? 
We need to get artifacts 
to where they need to go
This poses problems. 
• What hosts? 
• How do we get correct versions to those hosts? 
• Are we sure those hosts are suita...
SaltStack to the 
rescue!
Fast, scalable and flexible systems 
management software for data center 
automation, cloud orchestration, server 
provisi...
Fast, scalable and flexible systems 
management software for data center 
automation, cloud orchestration, server 
provisi...
Fast, scalable and flexible systems 
management software for data center 
automation, cloud orchestration, server 
provisi...
Fast, scalable and flexible systems 
management software for data center 
automation, cloud orchestration, server 
provisi...
Fast, scalable and flexible systems 
management software for data center 
automation, cloud orchestration, server 
provisi...
Salt is extremely extensible 
• External pillars 
* Just python 
• Interface with ZooKeeper for host discovery 
• ZooKeepe...
Grains 
Interface to derive information about the underlying system 
• Static information about 
systems 
• Loaded at mini...
Pillar 
Pillar is an interface for Salt designed to offer global values 
that can be distributed to all minions. 
• Someth...
Grain: Ask 
Pillar: Tell
Build triggered 
server1 
server2 
Salt Master 
ZooKeeper 
Stash 
Bamboo 
Zoom 
Selects build to deploy 
Tells salt to fin...
Bamboo talks to salt via Salt’s 
RESTful api 
Salt talks to ZooKeeper via an 
external pillar
Salt asks ZooKeeper 
“What hosts are currently 
responsible for app Foo?”
ZooKeeper responds, we 
run a series of state files 
on the targeted servers
States 
• Check stats (memory, disk, etc) 
• - cmd.run salt://deploy/check_host.py 
• Apply system settings (buffers, os m...
After build, package into 
rpm 
• Use the tools available 
• Versioning 
• Well understood 
• Reporting baked in
RPM 
• Specify version 
• Allows control over latest build, n-1, etc 
• Passed through via pillar at install time, fed by ...
API wrapper 
• Home grown deployment software 
• Command line utility that converts bamboo 
variables to salt states 
• De...
deploy state 
# Run the installation on the targeted server 
deploy_{{ application_name }}: 
pkg.installed: 
- name: {{ ap...
Things we’ve learned 
• In the end, it’s all just bits on disk 
• All software is just software, and (should be) equal 
• ...
Highlights 
• Implemented deployment pipeline 
• Artifact of the build 
• Bringing components together 
• Repeatable, rapi...
Thank you. 
For more about some of the interesting work Spot is doing, 
check our Engineering Blog at 
http://www.spottrad...
Upcoming SlideShare
Loading in …5
×

Spot Trading - A case study in continuous delivery for mission critical financial services

3,421 views

Published on

This is a presentation given by Jeremy Alons, Spot Trading, at the DevOps Summit Chicago in August 2014. Jeremy shares how Spot Trading does automated deployments for mission-critical financial services with a case study in continuous delivery.

Published in: Technology

Spot Trading - A case study in continuous delivery for mission critical financial services

  1. 1. Automated deployments for mission-critical financial services Jeremy Alons Systems Engineer, Spot Trading LLC Views and statements are my own and may not reflect those of my employer v.8
  2. 2. A case study in continuous delivery
  3. 3. Why? • Previously, deployments were time consuming • Took hours of developer time, sometimes spread over weeks • Manual process meant many problems at many points
  4. 4. Our goals • Rapid feedback • Tie configuration to deployment • Be highly parallelized, if possible • Infrastructure as code • Repeatable
  5. 5. Challenges • Lots of unique software • Differences force complexity • Misused the tools we had originally • Required additional tooling (that continues today)
  6. 6. Steps: Build Deploy Config Run
  7. 7. Steps: Build Deploy! Config Run
  8. 8. Steps: Build Deploy Config! Run
  9. 9. Build: Bamboo (or Jenkins, travis-ci, etc) executes commands i.e. gcc -o foo foo.c sh tests.sh etc
  10. 10. Source code in. Artifacts out.
  11. 11. Now what? We need to get artifacts to where they need to go
  12. 12. This poses problems. • What hosts? • How do we get correct versions to those hosts? • Are we sure those hosts are suitable for the application?
  13. 13. SaltStack to the rescue!
  14. 14. Fast, scalable and flexible systems management software for data center automation, cloud orchestration, server provisioning, configuration management and more - www.saltstack.com
  15. 15. Fast, scalable and flexible systems management software for data center automation, cloud orchestration, server provisioning, configuration management and more - www.saltstack.com
  16. 16. Fast, scalable and flexible systems management software for data center automation, cloud orchestration, server provisioning, configuration management and more - www.saltstack.com
  17. 17. Fast, scalable and flexible systems management software for data center automation, cloud orchestration, server provisioning, configuration management and more - www.saltstack.com
  18. 18. Fast, scalable and flexible systems management software for data center automation, cloud orchestration, server provisioning, configuration management and more - www.saltstack.com
  19. 19. Salt is extremely extensible • External pillars * Just python • Interface with ZooKeeper for host discovery • ZooKeeper maintains source of truth • But how does Bamboo talk to ZooKeeper? • Facilitate the communication through Salt
  20. 20. Grains Interface to derive information about the underlying system • Static information about systems • Loaded at minion start time • OS (major, minor), ipv[4|6], etc • Something you ask the server
  21. 21. Pillar Pillar is an interface for Salt designed to offer global values that can be distributed to all minions. • Something you tell a server • “pillar” is a dictionary of items sent to minions • ext_pillar are derived by the master by executing python • Still sent to minions
  22. 22. Grain: Ask Pillar: Tell
  23. 23. Build triggered server1 server2 Salt Master ZooKeeper Stash Bamboo Zoom Selects build to deploy Tells salt to find matching daemons Returns results Runs deploy state Asks for hosts 8 Return hosts to target Displays real time status and dependency map Developer 1 Pushes commit 3 2 4 5 6 7 0
  24. 24. Bamboo talks to salt via Salt’s RESTful api Salt talks to ZooKeeper via an external pillar
  25. 25. Salt asks ZooKeeper “What hosts are currently responsible for app Foo?”
  26. 26. ZooKeeper responds, we run a series of state files on the targeted servers
  27. 27. States • Check stats (memory, disk, etc) • - cmd.run salt://deploy/check_host.py • Apply system settings (buffers, os monitoring, etc) • Install software • yum install Foo-release-2.3-10
  28. 28. After build, package into rpm • Use the tools available • Versioning • Well understood • Reporting baked in
  29. 29. RPM • Specify version • Allows control over latest build, n-1, etc • Passed through via pillar at install time, fed by Bamboo deployment plan • Allows Bamboo to drive version released
  30. 30. API wrapper • Home grown deployment software • Command line utility that converts bamboo variables to salt states • Deploys itself via the deployment process - Dog fooding the deployment process
  31. 31. deploy state # Run the installation on the targeted server deploy_{{ application_name }}: pkg.installed: - name: {{ application_name }} - version: {{ pillar[‘deploy_pillar’][‘version’] }} - refresh: True - enablerepo: spot_applications - require: - sls: spot_applications_repo
  32. 32. Things we’ve learned • In the end, it’s all just bits on disk • All software is just software, and (should be) equal • Be specific and exact • Catch issues soon (test early, test often) • Being clever will come back to haunt you (after all, this is just software development)
  33. 33. Highlights • Implemented deployment pipeline • Artifact of the build • Bringing components together • Repeatable, rapid feedback
  34. 34. Thank you. For more about some of the interesting work Spot is doing, check our Engineering Blog at http://www.spottradingllc.com/technology/ email: jeremy.alons@spottradingllc.com Questions?

×