The document discusses the use of SaltStack for orchestrating event-driven infrastructure, highlighting its scalability and speed for automating server provisioning and application deployments. It details various components, such as states, pillars, grains, events, and reactors, with practical examples of commands and configurations used in a real-world scenario at Bloglovin. Additionally, it includes information on the Salt API for integrating external applications and services.

1. Orchestrate Event-­‐Driven
Infrastructure with SaltStack
Love Nyberg
Site reliability & Automa@on Engineer

2. SaltStack
SaltStack is extremely fast and scalable systems and
configura@on management soDware for predic@ve
orchestra@on, cloud and data center automa@on, server
provisioning, applica@on deployment and more.

3. Agenda
salt @ bloglovin
Terminology
Orchestrate
Reactor
Beacons

4. salt @ bloglovin
tens of millions of monthly unique visitors
hundred of millions requests per day
couple of million stories parsed per day
hybrid bare metal servers/cloud vm’s
2 salt masters
~90 salt minions

5. salt @ bloglovin
Parts we use!
States/Pillars/Grains
Orchestrate
Reactor
Salt Api
Salt Cloud

6. Terminology
Configura)on
Master/Minion
Masterless
Transport
ZeroMQ
Raet
SSH
Parts
States
Modules
Grains
Pillars

7. Execu@on
salt '<target>' <function> [arguments]

8. Execu@on modules
root@master:/# salt minion_love network.interface eth0
minion_love:
|_
----------
address:
172.17.0.9
broadcast:
None
label:
eth0
netmask:
255.255.0.0

9. Execu@on modules
root@master:~# salt minion1 cmd.run "ip addr show eth0"
minion1:
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc
pfifo_fast state UP group default qlen 1000
link/ether 12:4b:af:05:be:cd brd ff:ff:ff:ff:ff:ff
inet 10.0.0.206/24 brd 10.0.0.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::104b:afff:fe05:becd/64 scope link
valid_lft forever preferred_lft forever

10. State
/srv/salt/state/sshd/init.sls
ssh:
pkg.latest
- name: openssh-server
salt test_minion state.sls ssh

11. State/Execu@on
/srv/salt/state/sshd/init.sls
ssh:
pkg.latest
- name: openssh-server
root@master:/# salt test_minion pkg.latest_version openssh-server

12. {%- set ips = salt['network.ip_addrs'](None, False, '172.17.0.0/24')
%}
ssh:
pkg.latest
- name: openssh-server
service.running:
- enable: True
- watch:
- file: ssh
- pkg: ssh
file.managed:
- name: /etc/ssh/sshd_config
- source: salt://sshd/files/sshd_config
- template: jinja
- context:
ips: {{ ips }}

13. Grains
via state
root@mon1:/# salt minion_love grains.get ip_interfaces:eth0
minion_love:
- 172.17.0.9
within a state file
{%- set eth0 = salt['grains.get'](ip_interfaces:eth0, '') %}

14. Pillar
root@master:/# salt minion_love pillar.item secrets
minion_love:
----------
secrets:
----------
foo:
bar

15. Apply state
root@master:/# salt minion_love state.highstate
salt '<target>' <function> [arguments]

16. Event system
Event system is a local ZeroMQ PUB interface which fires salt events
Every event has a tag and a data structure

17. Event chain `test.ping`
Event fired at Wed May 27 22:00:14 2015
*************************
Tag: new_job
Data:
{'_stamp': '2015-05-27T20:00:14.774445',
'arg': [],
'fun': 'test.ping',
'jid': '20150527220014773556',
'minions': ['minion_love'],
'tgt': 'minion_love',
'tgt_type': 'glob',
'user': 'sudo_love'}

18. Event chain `test.ping`
Event fired at Wed May 27 22:00:14 2015
*************************
Tag: salt/job/20150527220014773556/ret/mon1
Data:
{'_stamp': '2015-05-27T20:00:14.981640',
'cmd': '_return',
'fun': 'test.ping',
'fun_args': [],
'id': ‘minion_love',
'jid': '20150527220014773556',
'retcode': 0,
'return': True,
'success': True}

19. Reactor
Ability to execute commands
Allow events to trigger ac@ons

20. # This script is to be called from the reactor system
{%- set application = data['data']['application'] %}
{%- set container = data['data']['container'] %}
{%- set node = data['data']['salt-node'] %}
{%- set tag = data['data']['tag'] %}
sync-states:
cmd.saltutil.sync_states:
- tgt: {{ node }}
deploy-containers:
cmd.state.sls:
- tgt: {{ node }}
- arg:
- containers.{{ container }}
- kwarg:
pillar:
containers:
imagetag:
{{ application }}: {{ tag }}

21. Salt Api
Yes! There’s an rest-­‐api
curl -sS salt.jacksoncage.se/run
-H 'Accept: application/x-yaml'
-d client='local'
-d tgt='*'
-d fun='test.ping'
-d username='saltdev'
-d password='saltdev'
-d eauth='pam'

22. Salt Api
# Example start creation dev box via curl:
curl -H 'X-Salt-Key: Secret-Key-Here'
-H 'Content-Type: application/json'
-d '{"username":"love"}'
https://salt.jacksoncage.se/hook/api/devbox/create

23. Orchestrate
Configura@on order
Sequenced applied states
Orchestrate deployment in order on mul@ple systems

24. # Create AWS EC2 instance
create-instance:
salt.state:
- tgt: {{ master }}
- sls: devlovin.cloud.create
- concurrent: True
- pillar:
devlovinreact:
instancename: {{ basename }}{{ username }}
# Run highstate
highstate-run:
salt.state:
- tgt: {{ basename }}{{ username }}
- highstate: True
- pillar:
devlovinreact:
username: {{ username }}
- require:
- salt: create-instance

25. Beacons
Hook into system processes
Translate external events into the salt event bus
Events can then be responded by Reactor

26. Beacons
DEMO

27. Ques@ons?
Love Nyberg
@jacksoncage
love@bloglovin.com

28. Thanks!
Love Nyberg
@jacksoncage
love@bloglovin.com