Event > React > Orchestrate

1. Intelligent Infrastructure
with SaltStack
Love Billingskog Nyberg
Infrastructure and Opera=ons Team Lead
Event > React > Orchestrate

2. Hi!
Love
@jacksoncage

3. bloglovin
tens of millions of monthly unique visitors
hundred of millions requests per day
couple of million stories parsed per day
hybrid bare-metal servers/cloud vm’s
4 salt masters
~90 salt minions

4. salt @ bloglovin
Parts we use!
States/Pillars/Grains
Pillarstack*
Orchestrate
Reactor
Beacons
Engines

5. Parallel execu=on
to run all the resources concurrently
Event driven
to monitor and react dynamically only to changes
Next genera=on

6. Next genera=on

7. Parallel execu=on
root@salt-dev:~# salt '*' test.ping
web101:
True
db101:
True
ship101:
True
master:
True
Easy to target remote execu5on against mul5ple targets

8. Parallel execu=on
Run commands via execu5on modules
root@salt-dev:~# salt web* service.stop nginx
web101:
True

9. Tradi=onal configura=on
Se;ng state on interval or on run

10. Event driven configura=on
Se;ng state when needed or change has happened

11. Goals
Monitoring - How can salt react to monitoring events and adjust
system without any manual interference
Minions - When a new device is connected get it’s data and no=fy
me, chat Vw!
Containers - When a new container starts, update load balancer
for that cluster and start sending traffic. Also when a container
stops, get logs and restart and send logs to team

12. Events
salt/job/20160413070418586332/new {
"_stamp": "2016-04-13T11:04:18.587129",
"arg": [],
"fun": "test.ping",
"jid": "20160413070418586332",
"minions": [
"web101"
],
"tgt": "web101",
"tgt_type": "glob",
"user": "sudo_love"
}
salt-run state.event pretty=True

13. Events
salt/job/20160413070418586332/ret/web101 {
"_stamp": "2016-04-13T11:04:18.789831",
"cmd": "_return",
"fun": "test.ping",
"fun_args": [],
"id": "web101",
"jid": "20160413070418586332",
"retcode": 0,
"return": true,
"success": true
}

14. State events
Enabling state events causes the Salt minion to send progress events as
each func5on in a state run completes execu5on.
state_events: True
minion

15. State events
Which can become an over load, let’s fire only on specific states
installed/vim:
pkg.installed:
- name: vim
- fire_event: installed/vim
minion

16. Custom event
Enabling state events causes the Salt minion to send progress events as
each func5on in a state run completes execu5on.
salt-call event.send /deploy/test/case '{"data": "Ready"}'
/deploy/test/case {
"_stamp": "2016-04-13T11:01:24.553140",
"cmd": "_minion_event",
"data": {
"__pub_fun": "event.send",
"__pub_jid": "20160413070124500641",
"__pub_pid": 17046,
"__pub_tgt": "salt-call",
"data": "Ready"
…
minion

17. Beacons
beacons:
service:
nginx:
onchangeonly: True
Hook into system processes. Translate external events into the salt event bus
minion

18. Engines
engines:
docker_events:
docker_url: unix://var/run/docker.sock
Salt Engines are long-running, external system processes that leverage Salt.
minion

19. Great!
Events for everything!

20. Reactor
Salt’s Reactor system gives you the ability to trigger ac5ons in response to any event.
reactor: # Master config section "reactor"
- 'salt/minion/*/start': # Match tag "salt/minion/*/start"
- /srv/reactor/start.sls # Things to do when a minion starts
- 'deploy/containers': # Match tag "deploy/containers"
- /srv/reactor/deploy-containers.sls # Things to do when a container is deployed
- 'salt/netapi/hook/api/devlovin/create': # Listing on salt api for /devlovin/create
- /srv/reactor/devlovin-create.sls # Things to do when a create request is sent
master

21. Reactor
Salt’s Reactor system gives you the ability to trigger ac5ons in response to any event.
Remote execu=on: run an execu=on module on the targeted
minions.
Runners: These are tasks you would start using salt-run.
Wheel: Wheel commands manage your Salt environment.
master

22. Reactor
Salt’s Reactor system gives you the ability to trigger ac5ons in response to any event.
<section id>:
local.<function>:
- tgt: <target>
- arg:
<arguments>
master

23. # This script is to be called from the reactor system
{%- set application = data['data']['application'] %}
{%- set container = data['data']['container'] %}
{%- set target = data['data']['target'] %}
{%- set tag = data['data']['tag'] %}
sync-states:
local.saltutil.sync_states:
- tgt: {{ target }}
deploy-containers:
local.state.sls:
- tgt: {{ target }}
- arg:
- containers.{{ container }}
- kwarg:
pillar:
containers:
imagetag:
{{ application }}: {{ tag }}
master

24. highstate_run:
local.state.highstate:
- tgt: db101
call_some_orchestrate_file:
runner.state.orchestrate:
- mods: some_orchestrate_file
- pillar:
stuff: things
Highstate run
Star5ng a orchestrator runner
master

25. Orchestrate
The orchestrate runner generalizes the Salt state system to a Salt master context.
master

26. # Create AWS EC2 instance
create-instance:
salt.state:
- tgt: {{ master }}
- sls: devlovin.cloud.create
- concurrent: True
- pillar:
devlovinreact:
instancename: {{ basename }}{{ username }}
# Run highstate
highstate-run:
salt.state:
- tgt: {{ basename }}{{ username }}
- highstate: True
- pillar:
devlovinreact:
username: {{ username }}
- require:
- salt: create-instance
master

27. Demo
Monitoring - How can salt react to monitoring events and adjust
system without any manual interference

28. Demo
Minions - When a new device is connected get it’s data and no=fy
me, chat Vw!

29. Demo
Containers - When a new container starts, update load balancer
for that cluster and start sending traffic. Also when a container
stops, get logs and restart and send logs to team

30. Ques=ons?
Love Billingskog Nyberg
@jacksoncage
love@bloglovin.com

31. Thanks!
Love Billingskog Nyberg
@jacksoncage
love@bloglovin.com
Use the SaltConf16 event app to provide feedback