Salt – A Scalable Systems
Management Solution for Datacenters
FrOSCon 2016, St. Augustin August 21, 2016
Sebastian Meyer
Linux Consultant & Trainer
B1 Systems GmbH
meyer@b1-systems.de
B1 Systems GmbH - Linux/Open Source Consulting,Training, Support & Development
Introducing B1 Systems
founded in 2004
operating both nationally and internationally
nearly 100 employees
provider for IBM, SUSE, Oracle & HP
vendor-independent (hardware and software)
focus:
consulting
support
development
training
operations
solutions
B1 Systems GmbH Salt – Scalable Systems Management 2/47
Areas of Expertise
B1 Systems GmbH Salt – Scalable Systems Management 3/47
Salt – Introduction
B1 Systems GmbH Salt – Scalable Systems Management 4/47
Yet Another Systems Management Solution?
takes inspiration from Puppet, Chef or Ansible
focuses on the entire system life cycle
easily scalable to a few thousand systems
convenient and easy to learn
configuration management and remote execution
B1 Systems GmbH Salt – Scalable Systems Management 5/47
Salt – Concept
B1 Systems GmbH Salt – Scalable Systems Management 6/47
Master & Minions
B1 Systems GmbH Salt – Scalable Systems Management 7/47
Scalability: Masters, Syndics & Minions
B1 Systems GmbH Salt – Scalable Systems Management 8/47
High Availability: Multiple Masters& Minions
B1 Systems GmbH Salt – Scalable Systems Management 9/47
Salt Modes
minions pull from master
master pushes to Minions
minions apply states locally
master applies states on minions via SSH
B1 Systems GmbH Salt – Scalable Systems Management 10/47
Remote Execution System
B1 Systems GmbH Salt – Scalable Systems Management 11/47
Salt Command
B1 Systems GmbH Salt – Scalable Systems Management 12/47
Grains
B1 Systems GmbH Salt – Scalable Systems Management 13/47
Configuration Management
B1 Systems GmbH Salt – Scalable Systems Management 14/47
States
ID:
module.function:
- name: name
- argument1: value
- argument2:
- value1
- value2
B1 Systems GmbH Salt – Scalable Systems Management 15/47
Top File
base:
’*’:
- monitoring
- ssh
- syslog
’*lan*’:
- ntp.lan
’*dmz*’:
- ntp.dmz
- firewall
all servers:
monitoring
ssh config
syslog
servers in LAN:
ntp config
servers in DMZ:
ntp config
firewall
B1 Systems GmbH Salt – Scalable Systems Management 16/47
Pillars
B1 Systems GmbH Salt – Scalable Systems Management 17/47
Pillar Data
Pillar Example
ntp:
{% if grains[’id’].startswith(’myntpserver’) %}
ntpservers: ["0.us.pool.ntp.org","1.us.pool.ntp.org"]
comment: ’’
{% else %}
ntpservers: ["10.1.1.20","10.1.1.21"]
comment: ’myinternalservers’
{% endif %}
Source: https://github.com/saltstack-formulas/ntp-formula/blob/master/pillar.example
B1 Systems GmbH Salt – Scalable Systems Management 18/47
Pillars and States
States top.sls
base:
’*’:
- monitoring
- ssh
- syslog
- ntp
’*dmz*’:
- firewall
Pillar top.sls
base:
’*’:
- monitoring
- ssh
- syslog
’*lan*’:
- ntp.lan
’*dmz*’:
- ntp.dmz
- firewall
B1 Systems GmbH Salt – Scalable Systems Management 19/47
Deploying the State
Master pushes to minions
salt ’*’ state.highstate
salt ’*’ state.sls mystate
Minions pull from master
salt-call state.highstate
salt-call state.sls mystate
B1 Systems GmbH Salt – Scalable Systems Management 20/47
Reusing States: Formulas
reusing existing code
roughly the same as Puppet modules/Ansible roles
collection of States and files
github.com/saltstack-formulas/ for "official" formulas
B1 Systems GmbH Salt – Scalable Systems Management 21/47
Using Formulas
directly from VCS or local
extendable via include
configurable via Pillar data
variables mapped via Jinja map
requirements across Formulas possible
B1 Systems GmbH Salt – Scalable Systems Management 22/47
Demo
B1 Systems GmbH Salt – Scalable Systems Management 23/47
Returners
salt ’*’ disk.usage --return redis_return
B1 Systems GmbH Salt – Scalable Systems Management 24/47
Salts Event Driven Infrastructure
B1 Systems GmbH Salt – Scalable Systems Management 25/47
Overview
actions trigger events
events are communicated via the event bus
reactors execute trigger actions responding to events
B1 Systems GmbH Salt – Scalable Systems Management 26/47
Event Bus
B1 Systems GmbH Salt – Scalable Systems Management 27/47
Actions & Events
master# salt ’salt-minion-01’ disk.percent /srv
salt-minion-01:
11%
B1 Systems GmbH Salt – Scalable Systems Management 28/47
Actions & Events
20160422163250339970 {
[...]
}
salt/job/20160422163250339970/new {
"_stamp": "2016-04-22T14:32:50.340357",
"arg": [ "/srv" ],
"fun": "disk.percent",
"jid": "20160422163250339970",
"minions": [ "salt-minion-01" ],
"tgt": "salt-minion-01",
"tgt_type": "glob",
"user": "root"
}
B1 Systems GmbH Salt – Scalable Systems Management 29/47
Actions & Events
salt/job/20160422163250339970/ret/salt-minion-01 {
"_stamp": "2016-04-22T14:32:50.536877",
"cmd": "_return",
"fun": "disk.percent",
"fun_args": [ "/srv" ],
"id": "salt-minion-01",
"jid": "20160422163250339970",
"retcode": 0,
"return": "11%",
"success": true
}
B1 Systems GmbH Salt – Scalable Systems Management 30/47
Events in a State
b1/mystate/status/update:
event.send:
- data:
status: "Installation done!"
B1 Systems GmbH Salt – Scalable Systems Management 31/47
Beacons
hook into system on minion
create events
inotify, diskusage, load, journald ...
B1 Systems GmbH Salt – Scalable Systems Management 32/47
Beacons - Example
inotify Beacon
beacons:
inotify:
/etc/motd:
mask:
- modify
B1 Systems GmbH Salt – Scalable Systems Management 33/47
Reactors
B1 Systems GmbH Salt – Scalable Systems Management 34/47
Calling Reactors on Events
Reactor Example
reactor:
- ’salt/minion/*/start’:
- /srv/reactor/start.sls
- ’b1/mystate/status/*’:
- salt://reactor/status.sls
B1 Systems GmbH Salt – Scalable Systems Management 35/47
Demo
B1 Systems GmbH Salt – Scalable Systems Management 36/47
Use Cases?
load-balancing
job automation
alerting
B1 Systems GmbH Salt – Scalable Systems Management 37/47
Salt Cloud
B1 Systems GmbH Salt – Scalable Systems Management 38/47
Overview
B1 Systems GmbH Salt – Scalable Systems Management 39/47
Providers
Amazon EC2 Provider Example
my-ec2:
driver: ec2
id: ’MYEC2ID’
key: ’adsfrf453fMYKEYasdsadg43’
private_key: /etc/salt/my_key.pem
keyname: my_key
securitygroup: default
minion:
master: saltmaster.example.com
B1 Systems GmbH Salt – Scalable Systems Management 40/47
Profiles
profile name
provider
image or template
options for the instance
minion options
B1 Systems GmbH Salt – Scalable Systems Management 41/47
Profiles
LXC Profile Example
myfancyprofile:
provider: lxc-host01
lxc_profile:
template: ubuntu
options:
release: trusty
password: test123
B1 Systems GmbH Salt – Scalable Systems Management 42/47
Maps
Mapfile
profile1:
- instance_name_1
- instance_name_2
profile2:
- instance_name_3:
grains:
mykey: myvalue
- instance_name_4
Execute Mapfile
salt-cloud -m /path/to/mapfile
B1 Systems GmbH Salt – Scalable Systems Management 43/47
Bootstrapping a New Salt Environment
Mapfile
profile1:
- instance_name_1:
make_master: True
minion:
master: myoldmaster
local_master: True
- instance_name_2
- instance_name_3
- instance_name_4
...
B1 Systems GmbH Salt – Scalable Systems Management 44/47
Saltify Existing Machines 1/2
Saltify Provider
saltify-all-machines:
driver: saltify
minion:
master: mysaltmaster
Saltify Profile
salt-machine:
provider: saltify-all-machines
ssh_username: root
key_filename: ’/etc/salt/pki/master/ssh/salt-ssh.rsa’
B1 Systems GmbH Salt – Scalable Systems Management 45/47
Saltify Existing Machines 2/2
Mapfile
salt-machine:
- first-machine:
ssh_host: 1.2.3.4
- second-machine:
ssh_host: 1.2.3.5
- third-machine:
ssh_host: 1.2.3.6
B1 Systems GmbH Salt – Scalable Systems Management 46/47
Thank You!
For more information, refer to info@b1-systems.de
or +49 (0)8457 - 931096
B1 Systems GmbH - Linux/Open Source Consulting,Training, Support & Development

Salt - A Scalable Systems Management Solution for Datacenters

  • 1.
    Salt – AScalable Systems Management Solution for Datacenters FrOSCon 2016, St. Augustin August 21, 2016 Sebastian Meyer Linux Consultant & Trainer B1 Systems GmbH meyer@b1-systems.de B1 Systems GmbH - Linux/Open Source Consulting,Training, Support & Development
  • 2.
    Introducing B1 Systems foundedin 2004 operating both nationally and internationally nearly 100 employees provider for IBM, SUSE, Oracle & HP vendor-independent (hardware and software) focus: consulting support development training operations solutions B1 Systems GmbH Salt – Scalable Systems Management 2/47
  • 3.
    Areas of Expertise B1Systems GmbH Salt – Scalable Systems Management 3/47
  • 4.
    Salt – Introduction B1Systems GmbH Salt – Scalable Systems Management 4/47
  • 5.
    Yet Another SystemsManagement Solution? takes inspiration from Puppet, Chef or Ansible focuses on the entire system life cycle easily scalable to a few thousand systems convenient and easy to learn configuration management and remote execution B1 Systems GmbH Salt – Scalable Systems Management 5/47
  • 6.
    Salt – Concept B1Systems GmbH Salt – Scalable Systems Management 6/47
  • 7.
    Master & Minions B1Systems GmbH Salt – Scalable Systems Management 7/47
  • 8.
    Scalability: Masters, Syndics& Minions B1 Systems GmbH Salt – Scalable Systems Management 8/47
  • 9.
    High Availability: MultipleMasters& Minions B1 Systems GmbH Salt – Scalable Systems Management 9/47
  • 10.
    Salt Modes minions pullfrom master master pushes to Minions minions apply states locally master applies states on minions via SSH B1 Systems GmbH Salt – Scalable Systems Management 10/47
  • 11.
    Remote Execution System B1Systems GmbH Salt – Scalable Systems Management 11/47
  • 12.
    Salt Command B1 SystemsGmbH Salt – Scalable Systems Management 12/47
  • 13.
    Grains B1 Systems GmbHSalt – Scalable Systems Management 13/47
  • 14.
    Configuration Management B1 SystemsGmbH Salt – Scalable Systems Management 14/47
  • 15.
    States ID: module.function: - name: name -argument1: value - argument2: - value1 - value2 B1 Systems GmbH Salt – Scalable Systems Management 15/47
  • 16.
    Top File base: ’*’: - monitoring -ssh - syslog ’*lan*’: - ntp.lan ’*dmz*’: - ntp.dmz - firewall all servers: monitoring ssh config syslog servers in LAN: ntp config servers in DMZ: ntp config firewall B1 Systems GmbH Salt – Scalable Systems Management 16/47
  • 17.
    Pillars B1 Systems GmbHSalt – Scalable Systems Management 17/47
  • 18.
    Pillar Data Pillar Example ntp: {%if grains[’id’].startswith(’myntpserver’) %} ntpservers: ["0.us.pool.ntp.org","1.us.pool.ntp.org"] comment: ’’ {% else %} ntpservers: ["10.1.1.20","10.1.1.21"] comment: ’myinternalservers’ {% endif %} Source: https://github.com/saltstack-formulas/ntp-formula/blob/master/pillar.example B1 Systems GmbH Salt – Scalable Systems Management 18/47
  • 19.
    Pillars and States Statestop.sls base: ’*’: - monitoring - ssh - syslog - ntp ’*dmz*’: - firewall Pillar top.sls base: ’*’: - monitoring - ssh - syslog ’*lan*’: - ntp.lan ’*dmz*’: - ntp.dmz - firewall B1 Systems GmbH Salt – Scalable Systems Management 19/47
  • 20.
    Deploying the State Masterpushes to minions salt ’*’ state.highstate salt ’*’ state.sls mystate Minions pull from master salt-call state.highstate salt-call state.sls mystate B1 Systems GmbH Salt – Scalable Systems Management 20/47
  • 21.
    Reusing States: Formulas reusingexisting code roughly the same as Puppet modules/Ansible roles collection of States and files github.com/saltstack-formulas/ for "official" formulas B1 Systems GmbH Salt – Scalable Systems Management 21/47
  • 22.
    Using Formulas directly fromVCS or local extendable via include configurable via Pillar data variables mapped via Jinja map requirements across Formulas possible B1 Systems GmbH Salt – Scalable Systems Management 22/47
  • 23.
    Demo B1 Systems GmbHSalt – Scalable Systems Management 23/47
  • 24.
    Returners salt ’*’ disk.usage--return redis_return B1 Systems GmbH Salt – Scalable Systems Management 24/47
  • 25.
    Salts Event DrivenInfrastructure B1 Systems GmbH Salt – Scalable Systems Management 25/47
  • 26.
    Overview actions trigger events eventsare communicated via the event bus reactors execute trigger actions responding to events B1 Systems GmbH Salt – Scalable Systems Management 26/47
  • 27.
    Event Bus B1 SystemsGmbH Salt – Scalable Systems Management 27/47
  • 28.
    Actions & Events master#salt ’salt-minion-01’ disk.percent /srv salt-minion-01: 11% B1 Systems GmbH Salt – Scalable Systems Management 28/47
  • 29.
    Actions & Events 20160422163250339970{ [...] } salt/job/20160422163250339970/new { "_stamp": "2016-04-22T14:32:50.340357", "arg": [ "/srv" ], "fun": "disk.percent", "jid": "20160422163250339970", "minions": [ "salt-minion-01" ], "tgt": "salt-minion-01", "tgt_type": "glob", "user": "root" } B1 Systems GmbH Salt – Scalable Systems Management 29/47
  • 30.
    Actions & Events salt/job/20160422163250339970/ret/salt-minion-01{ "_stamp": "2016-04-22T14:32:50.536877", "cmd": "_return", "fun": "disk.percent", "fun_args": [ "/srv" ], "id": "salt-minion-01", "jid": "20160422163250339970", "retcode": 0, "return": "11%", "success": true } B1 Systems GmbH Salt – Scalable Systems Management 30/47
  • 31.
    Events in aState b1/mystate/status/update: event.send: - data: status: "Installation done!" B1 Systems GmbH Salt – Scalable Systems Management 31/47
  • 32.
    Beacons hook into systemon minion create events inotify, diskusage, load, journald ... B1 Systems GmbH Salt – Scalable Systems Management 32/47
  • 33.
    Beacons - Example inotifyBeacon beacons: inotify: /etc/motd: mask: - modify B1 Systems GmbH Salt – Scalable Systems Management 33/47
  • 34.
    Reactors B1 Systems GmbHSalt – Scalable Systems Management 34/47
  • 35.
    Calling Reactors onEvents Reactor Example reactor: - ’salt/minion/*/start’: - /srv/reactor/start.sls - ’b1/mystate/status/*’: - salt://reactor/status.sls B1 Systems GmbH Salt – Scalable Systems Management 35/47
  • 36.
    Demo B1 Systems GmbHSalt – Scalable Systems Management 36/47
  • 37.
    Use Cases? load-balancing job automation alerting B1Systems GmbH Salt – Scalable Systems Management 37/47
  • 38.
    Salt Cloud B1 SystemsGmbH Salt – Scalable Systems Management 38/47
  • 39.
    Overview B1 Systems GmbHSalt – Scalable Systems Management 39/47
  • 40.
    Providers Amazon EC2 ProviderExample my-ec2: driver: ec2 id: ’MYEC2ID’ key: ’adsfrf453fMYKEYasdsadg43’ private_key: /etc/salt/my_key.pem keyname: my_key securitygroup: default minion: master: saltmaster.example.com B1 Systems GmbH Salt – Scalable Systems Management 40/47
  • 41.
    Profiles profile name provider image ortemplate options for the instance minion options B1 Systems GmbH Salt – Scalable Systems Management 41/47
  • 42.
    Profiles LXC Profile Example myfancyprofile: provider:lxc-host01 lxc_profile: template: ubuntu options: release: trusty password: test123 B1 Systems GmbH Salt – Scalable Systems Management 42/47
  • 43.
    Maps Mapfile profile1: - instance_name_1 - instance_name_2 profile2: -instance_name_3: grains: mykey: myvalue - instance_name_4 Execute Mapfile salt-cloud -m /path/to/mapfile B1 Systems GmbH Salt – Scalable Systems Management 43/47
  • 44.
    Bootstrapping a NewSalt Environment Mapfile profile1: - instance_name_1: make_master: True minion: master: myoldmaster local_master: True - instance_name_2 - instance_name_3 - instance_name_4 ... B1 Systems GmbH Salt – Scalable Systems Management 44/47
  • 45.
    Saltify Existing Machines1/2 Saltify Provider saltify-all-machines: driver: saltify minion: master: mysaltmaster Saltify Profile salt-machine: provider: saltify-all-machines ssh_username: root key_filename: ’/etc/salt/pki/master/ssh/salt-ssh.rsa’ B1 Systems GmbH Salt – Scalable Systems Management 45/47
  • 46.
    Saltify Existing Machines2/2 Mapfile salt-machine: - first-machine: ssh_host: 1.2.3.4 - second-machine: ssh_host: 1.2.3.5 - third-machine: ssh_host: 1.2.3.6 B1 Systems GmbH Salt – Scalable Systems Management 46/47
  • 47.
    Thank You! For moreinformation, refer to info@b1-systems.de or +49 (0)8457 - 931096 B1 Systems GmbH - Linux/Open Source Consulting,Training, Support & Development