SaltStack is used at Bloglovin to manage tens of millions of monthly website visitors across hundreds of servers. SaltStack allows them to execute commands concurrently across servers, react dynamically to changes through events, and manage infrastructure in a next generation way. Key SaltStack components used include states, pillars, grains, reactors, beacons, engines and orchestration.

1. Intelligent Infrastructure
with SaltStack
Love Billingskog Nyberg
Infrastructure and Opera=ons Team Lead
Event > React > Orchestrate

2. Hi!
Love
@jacksoncage

3. bloglovin
tens of millions of monthly unique visitors
hundred of millions requests per day
couple of million stories parsed per day
hybrid bare-metal servers/cloud vm’s
4 salt masters
~90 salt minions

4. salt @ bloglovin
Parts we use!
States/Pillars/Grains
Pillarstack*
Orchestrate
Reactor
Beacons
Engines

5. Parallel execu=on
to run all the resources concurrently
Event driven
to monitor and react dynamically only to changes
Next genera=on

6. Next genera=on

7. Parallel execu=on
root@salt-dev:~# salt '*' test.ping
web101:
True
db101:
True
ship101:
True
master:
True
Easy to target remote execu5on against mul5ple targets

8. Parallel execu=on
Run commands via execu5on modules
root@salt-dev:~# salt web* service.stop nginx
web101:
True

9. Tradi=onal configura=on
Se;ng state on interval or on run

10. Event driven configura=on
Se;ng state when needed or change has happened

11. Goals
Monitoring - How can salt react to monitoring events and adjust
system without any manual interference
Minions - When a new device is connected get it’s data and no=fy
me, chat Vw!
Containers - When a new container starts, update load balancer
for that cluster and start sending traffic. Also when a container
stops, get logs and restart and send logs to team

12. Events
salt/job/20160413070418586332/new {
"_stamp": "2016-04-13T11:04:18.587129",
"arg": [],
"fun": "test.ping",
"jid": "20160413070418586332",
"minions": [
"web101"
],
"tgt": "web101",
"tgt_type": "glob",
"user": "sudo_love"
}
salt-run state.event pretty=True

13. Events
salt/job/20160413070418586332/ret/web101 {
"_stamp": "2016-04-13T11:04:18.789831",
"cmd": "_return",
"fun": "test.ping",
"fun_args": [],
"id": "web101",
"jid": "20160413070418586332",
"retcode": 0,
"return": true,
"success": true
}

14. State events
Enabling state events causes the Salt minion to send progress events as
each func5on in a state run completes execu5on.
state_events: True
minion

15. State events
Which can become an over load, let’s fire only on specific states
installed/vim:
pkg.installed:
- name: vim
- fire_event: installed/vim
minion

16. Custom event
Enabling state events causes the Salt minion to send progress events as
each func5on in a state run completes execu5on.
salt-call event.send /deploy/test/case '{"data": "Ready"}'
/deploy/test/case {
"_stamp": "2016-04-13T11:01:24.553140",
"cmd": "_minion_event",
"data": {
"__pub_fun": "event.send",
"__pub_jid": "20160413070124500641",
"__pub_pid": 17046,
"__pub_tgt": "salt-call",
"data": "Ready"
…
minion

17. Beacons
beacons:
service:
nginx:
onchangeonly: True
Hook into system processes. Translate external events into the salt event bus
minion

18. Engines
engines:
docker_events:
docker_url: unix://var/run/docker.sock
Salt Engines are long-running, external system processes that leverage Salt.
minion

19. Great!
Events for everything!

20. Reactor
Salt’s Reactor system gives you the ability to trigger ac5ons in response to any event.
reactor: # Master config section "reactor"
- 'salt/minion/*/start': # Match tag "salt/minion/*/start"
- /srv/reactor/start.sls # Things to do when a minion starts
- 'deploy/containers': # Match tag "deploy/containers"
- /srv/reactor/deploy-containers.sls # Things to do when a container is deployed
- 'salt/netapi/hook/api/devlovin/create': # Listing on salt api for /devlovin/create
- /srv/reactor/devlovin-create.sls # Things to do when a create request is sent
master

21. Reactor
Salt’s Reactor system gives you the ability to trigger ac5ons in response to any event.
Remote execu=on: run an execu=on module on the targeted
minions.
Runners: These are tasks you would start using salt-run.
Wheel: Wheel commands manage your Salt environment.
master

22. Reactor
Salt’s Reactor system gives you the ability to trigger ac5ons in response to any event.
<section id>:
local.<function>:
- tgt: <target>
- arg:
<arguments>
master

23. # This script is to be called from the reactor system
{%- set application = data['data']['application'] %}
{%- set container = data['data']['container'] %}
{%- set target = data['data']['target'] %}
{%- set tag = data['data']['tag'] %}
sync-states:
local.saltutil.sync_states:
- tgt: {{ target }}
deploy-containers:
local.state.sls:
- tgt: {{ target }}
- arg:
- containers.{{ container }}
- kwarg:
pillar:
containers:
imagetag:
{{ application }}: {{ tag }}
master

24. highstate_run:
local.state.highstate:
- tgt: db101
call_some_orchestrate_file:
runner.state.orchestrate:
- mods: some_orchestrate_file
- pillar:
stuff: things
Highstate run
Star5ng a orchestrator runner
master

25. Orchestrate
The orchestrate runner generalizes the Salt state system to a Salt master context.
master

26. # Create AWS EC2 instance
create-instance:
salt.state:
- tgt: {{ master }}
- sls: devlovin.cloud.create
- concurrent: True
- pillar:
devlovinreact:
instancename: {{ basename }}{{ username }}
# Run highstate
highstate-run:
salt.state:
- tgt: {{ basename }}{{ username }}
- highstate: True
- pillar:
devlovinreact:
username: {{ username }}
- require:
- salt: create-instance
master

27. Demo
Monitoring - How can salt react to monitoring events and adjust
system without any manual interference

28. Demo
Minions - When a new device is connected get it’s data and no=fy
me, chat Vw!

29. Demo
Containers - When a new container starts, update load balancer
for that cluster and start sending traffic. Also when a container
stops, get logs and restart and send logs to team

30. Ques=ons?
Love Billingskog Nyberg
@jacksoncage
love@bloglovin.com

31. Thanks!
Love Billingskog Nyberg
@jacksoncage
love@bloglovin.com
Use the SaltConf16 event app to provide feedback