Example of how you can leverage the salt event bus to support your infrastructure life-cycle for monitoring with Zabbix. Enable workflows like when adding salt states to a minion automatically apply associated monitoring templates. or when decommissioning hosts, automatically remove them from Zabbix.

1. Salt with Zabbix

2. Saltstack Configuration Management
Salt can do much more than configure a single machine
It is a state engine which allows for complicated automation workflows

3. Monitoring SDLC
As capabilities are added to the environment they must be monitored
Zabbix provides capabilities and APIs for monitoring component, architecture, and
service level monitoring
Concerned with complexity, scale, and maintainability. Create patterns allowing
distributed teams to interface with monitoring, without needing to know all the
technical details.

4. Zabbix Server
Zabbix Introduction
host1
Nginx Service
Web Service: myapp
Host Group
Host: host1
Templates
OS Linux Nginx myapp
Item: CPU Item:
Connection
status
Web Scenario:
Site availability
Trigger: High
CPU
Graph:
Connections
SLA:
Availability
Inventory Data

5. Salt States: myapp service
base:
'salt-client1.station':
...
- openssh.config
- zabbix.agent.conf
- nginx
- example_app

6. Salt States: myapp service
base:
'salt-client1.station':
...
- openssh.config
- zabbix.agent.conf
- nginx
- example_app
Mapping
Linux Template
Zabbix Template
Nginx Template
Myapp Template

7. state.highstate
Salt-client1.station:
...
Name: /etc/ssh/sshd_config - Function: file.managed - Result: Clean Started: - 22:10:19.597471 Duration: 110.485 ms
Name: zabbix-agent - Function: pkg.installed - Result: Clean Started: - 22:10:19.790945 Duration: 11.088 ms
Name: nginx - Function: pkg.installed - Result: Clean Started: - 22:10:20.002695 Duration: 10.612 ms
Name: /etc/nginx/nginx.conf - Function: file.managed - Result: Clean Started: - 22:10:20.013534 Duration: 32.175 ms
Name: /etc/nginx/sites-available/mysite - Function: file.managed - Result: Clean Started: - 22:10:20.046610 Duration: 27.779 ms
Name: /var/www/html/mysite/index.html - Function: file.managed - Result: Clean Started: - 22:10:20.119449 Duration: 17.067 ms
Summary for salt-client1.station
-------------
Succeeded: 34
Failed: 0
-------------
Total states run: 34
Total run time: 548.998 ms

8. state.highstate

9. state.highstate
Yay!
But what about monitoring?

10. Salt States: Notify Zabbix
base:
'salt-client1.station':
...
- openssh.config
- zabbix.agent.conf
- nginx
- example_app
- monitoring

11. Monitoring State Example
create_host_group:
zabbix_hostgroup.present:
- name: {{ zab_hostgroup }}
create_host_{{ zab_host }}:
zabbix_host.present:
- host: {{ zab_host }}
- groups:
- {{ zab_hostgroup }}
- interfaces:
- {{ zab_fqdn }}:
- ip: {{ zab_host_ip }}
- type: 'Agent'
- port: 10050
- inventory: {{ zab_inv }}
add_zabbix_templates_to_host:
zabbix_host.assign_templates:
- host: {{ zab_host }}
- templates: {{ zab_templates }}
Add Zabbix host group
Add host to Zabbix
Assign templates to host in
Zabbix

12. Bonus: Zabbix Inventory

13. Monitoring State Example
Sounds simple enough. But how do we associate salt states with Zabbix templates?
Goal: “Create patterns allowing distributed teams to interface with monitoring, without
needing to know all the technical details.”
add_zabbix_templates_to_host:
zabbix_host.assign_templates:
- host: {{ zab_host }}
- templates: {{ zab_templates }}

14. Monitoring State Example
Sounds simple enough. But how do we associate salt states with Zabbix templates?
Goal: “Create patterns allowing distributed teams to interface with monitoring, without
needing to know all the technical details.”
add_zabbix_templates_to_host:
zabbix_host.assign_templates:
- host: {{ zab_host }}
- templates: {{ zab_templates }}
sshd_banner:
file.managed:
- name: {{ openssh.banner }}
...
enable_linux_template:
grains.list_present:
- name: monitoring_templates
- value: 'Template OS Linux by Zabbix agent
active'

15. Monitoring State Example
# salt 'salt-client1.station' grains.get monitoring_templates
salt-client1.station:
- Template OS Linux by Zabbix agent active
- Template App Nginx by Zabbix agent
- myapp

16. Monitoring State Example
# salt 'salt-client1.station' grains.get monitoring_templates
salt-client1.station:
- Template OS Linux by Zabbix agent active
- Template App Nginx by Zabbix agent
- myapp

17. Monitoring State Example
Cool! But what about the life cycle?
No, the machine life cycle: deleting/updating

18. Phase 2: Salt Event Bus
Up until now, we kept things direct by having the salt minion add itself to Zabbix
But this has drawbacks in that the minion needs to have the Zabbix password, and
cannot notify Zabbix it has been deleted.

19. Phase 2: Salt Event Bus
Salt Minion Salt Master Zabbix Server
Event Bus
Highstate
update
Highstate
update
Orch:
Gather
data
Add/update host
in Zabbix
Decom Process
Minion
destroyed
Minion
destroyed
Remove host
from Zabbix

20. Notify Salt Event Bus
Update_monitor_templates:
event.send:
- name: custom/monitoring/update
- data:
ip_address: {{ salt['network.ip_addrs']() | first | default([]) }}
default_group: {{ salt['grains.get']('ec2_tags:SupportGroup', 'GenericSupport')
}}/{{ salt['grains.get']('ec2_tags:Environment', 'Tagless') }}
fqdn: {{ hostfqdn }}
inventory: [
"os": {{ salt['grains.get']('osrelease', 'UnknownOS') }},
]
- onchanges:
- grains: monitoring_templates
mineupdate:
module.wait:
- mine.update:
- name: mine.update
- watch:
- grains: monitoring_templates

21. Event Reactor
zabbix_update_host_templates:
runner.state.orchestrate:
- args:
- mods: orchestrations.monitoring_update_template
- pillar:
event_data: {{ data | json() }}

22. Event Orchestration
{%- set data = salt.pillar.get('event_data') %}
{# Queries salt mine for monitoring_groups values for specified host #}
{%- set monitoring_templates = salt.saltutil.runner('mine.get',
tgt=data['id'],
fun='monitoring_templates',
tgt_type='list')[data['id']] %}
ensure_in_zabbix:
salt.state:
- tgt: "salt-master1*"
- tgt_type: glob
- concurrent: True
- sls: monitoring.orch.ensure_zabbixed
- pillar:
groups: {{ data['data']['default_group'] }}
host: {{ data['id'] }}
ip: {{ data['data']['ip_address'] }}
templates: {{ monitoring_templates }}
inventory: {{ data['data']['inventory'] }}
fqdn: {{ data['data']['fqdn'] }}

23. Salt Event Bus Summary
By using the salt event bus, we can tie the salt master’s state engine into any part of our
provisioning process.
Also means that the Zabbix password only needs to be granted to the salt master itself
(or wherever you run the orchestration job (Zabbix server?))

24. Workflow Integration Summary
Salt states simply need to add the Zabbix template name to a grain list.
The monitoring team owns the monitoring integration state and orchestration

25. Questions?
Thanks for attending!