This is a presentation given by Jeremy Alons, Spot Trading, at the DevOps Summit Chicago in August 2014. Jeremy shares how Spot Trading does automated deployments for mission-critical financial services with a case study in continuous delivery.

1. Automated deployments
for mission-critical
financial services
Jeremy Alons
Systems Engineer, Spot Trading LLC
Views and statements are my own and may not reflect those of my employer
v.8

2. A case study in
continuous delivery

3. Why?
• Previously, deployments were time consuming
• Took hours of developer time, sometimes spread
over weeks
• Manual process meant many problems at many
points

4. Our goals
• Rapid feedback
• Tie configuration to deployment
• Be highly parallelized, if possible
• Infrastructure as code
• Repeatable

5. Challenges
• Lots of unique software
• Differences force complexity
• Misused the tools we had originally
• Required additional tooling (that continues today)

6. Steps:
Build
Deploy
Config
Run

7. Steps:
Build
Deploy!
Config
Run

8. Steps:
Build
Deploy
Config!
Run

9. Build: Bamboo (or Jenkins, travis-ci, etc) executes
commands
i.e. gcc -o foo foo.c
sh tests.sh
etc

10. Source code in.
Artifacts out.

11. Now what?
We need to get artifacts
to where they need to go

12. This poses problems.
• What hosts?
• How do we get correct versions to those hosts?
• Are we sure those hosts are suitable for the
application?

13. SaltStack to the
rescue!

14. Fast, scalable and flexible systems
management software for data center
automation, cloud orchestration, server
provisioning, configuration management
and more
- www.saltstack.com

15. Fast, scalable and flexible systems
management software for data center
automation, cloud orchestration, server
provisioning, configuration management
and more
- www.saltstack.com

16. Fast, scalable and flexible systems
management software for data center
automation, cloud orchestration, server
provisioning, configuration management
and more
- www.saltstack.com

17. Fast, scalable and flexible systems
management software for data center
automation, cloud orchestration, server
provisioning, configuration management
and more
- www.saltstack.com

18. Fast, scalable and flexible systems
management software for data center
automation, cloud orchestration, server
provisioning, configuration management
and more
- www.saltstack.com

19. Salt is extremely extensible
• External pillars
* Just python
• Interface with ZooKeeper for host discovery
• ZooKeeper maintains source of truth
• But how does Bamboo talk to ZooKeeper?
• Facilitate the communication through Salt

20. Grains
Interface to derive information about the underlying system
• Static information about
systems
• Loaded at minion start time
• OS (major, minor), ipv[4|6], etc
• Something you ask the server

21. Pillar
Pillar is an interface for Salt designed to offer global values
that can be distributed to all minions.
• Something you tell a server
• “pillar” is a dictionary of items
sent to minions
• ext_pillar are derived by the
master by executing python
• Still sent to minions

22. Grain: Ask
Pillar: Tell

23. Build triggered
server1
server2
Salt Master
ZooKeeper
Stash
Bamboo
Zoom
Selects build to deploy
Tells salt to find
matching daemons
Returns results
Runs deploy state
Asks for
hosts
8
Return hosts
to target
Displays real time
status and
dependency map
Developer
1 Pushes commit
3 2
4
5
6
7
0

24. Bamboo talks to salt via Salt’s
RESTful api
Salt talks to ZooKeeper via an
external pillar

25. Salt asks ZooKeeper
“What hosts are currently
responsible for app Foo?”

26. ZooKeeper responds, we
run a series of state files
on the targeted servers

27. States
• Check stats (memory, disk, etc)
• - cmd.run salt://deploy/check_host.py
• Apply system settings (buffers, os monitoring, etc)
• Install software
• yum install Foo-release-2.3-10

28. After build, package into
rpm
• Use the tools available
• Versioning
• Well understood
• Reporting baked in

29. RPM
• Specify version
• Allows control over latest build, n-1, etc
• Passed through via pillar at install time, fed by
Bamboo deployment plan
• Allows Bamboo to drive version released

30. API wrapper
• Home grown deployment software
• Command line utility that converts bamboo
variables to salt states
• Deploys itself via the deployment process
- Dog fooding the deployment process

31. deploy state
# Run the installation on the targeted server
deploy_{{ application_name }}:
pkg.installed:
- name: {{ application_name }}
- version: {{ pillar[‘deploy_pillar’][‘version’] }}
- refresh: True
- enablerepo: spot_applications
- require:
- sls: spot_applications_repo

32. Things we’ve learned
• In the end, it’s all just bits on disk
• All software is just software, and (should be) equal
• Be specific and exact
• Catch issues soon (test early, test often)
• Being clever will come back to haunt you (after all,
this is just software development)

33. Highlights
• Implemented deployment pipeline
• Artifact of the build
• Bringing components together
• Repeatable, rapid feedback

34. Thank you.
For more about some of the interesting work Spot is doing,
check our Engineering Blog at
http://www.spottradingllc.com/technology/
email: jeremy.alons@spottradingllc.com
Questions?