Seth House, SaltStack senior engineer, presented at the first Rackspace Unlocked.io event in New York City the week of Cloud Expo. His presentation titled, "Real-time cloud management with SaltStack" is provided here.

1. Real-time cloud management with Salt
Unlocked.io Seth House <shouse@saltstack.com>
2013-06-14
1 / 47

2. Salt philosophy
1 Salt philosophy
Building blocks
What are our building blocks?
Building blocks and complexity
Success comes from simplicity
Salt is aptly named
Flow + state = EVERYTHING
Building on state and flow
Building blocks for infra
Building remote ex
Building states
Building cloud
Building monitoring
Conclusion
2 Salt internals (briefly)
2 / 47

3. Salt philosophy Building blocks
Building blocks
3 / 47

4. Salt philosophy Building blocks
Building blocks
Everything we have is built on earlier generations
Building blocks of history
Agriculture
Cities
Governments
Mathematics
Manufacturing
Language
4 / 47

5. Salt philosophy What are our building blocks?
What are our building blocks?
Computers Systems
Data centers are the modern platform of invention
Large scale computing drives virtually all high level human interactions
Data centers themselves are becoming the building blocks
5 / 47

6. Salt philosophy Building blocks and complexity
Building blocks and complexity
As systems grow they become more complex
Increased complexity makes using larger building blocks impossible
Building blocks used today are simple
Easy to use
Easy to understand
Yet powerful
6 / 47

7. Salt philosophy Success comes from simplicity
Success comes from simplicity
Viable building blocks are fundamentally simple
Things that are in themselves simple to understand and use
Concepts that can cross barriers and cleanly apply to many higher level
uses
YAML is a great example
Unix is a great example
7 / 47

8. Salt philosophy Salt is aptly named
Salt is aptly named
8 / 47

9. Salt philosophy Salt is aptly named
Salt is aptly named
The name is the vision
Salt is in nature, a building block
Salt is in cooking, a building block
Salt is in life, a building block
Yes, this is where the logo came from
Yes, this is why it is Salt STACK
9 / 47

10. Salt philosophy Flow + state = EVERYTHING
Flow + state = EVERYTHING
Core idea behind Salt
Flow = Access and control
State = The functional state of the system
Between them All things are controlled
10 / 47

11. Salt philosophy Building on state and flow
Building on state and flow
Basics
Remote execution built on Flow
Config Management built on State
Ramping Up
Monitoring build on Flow + State
Cloud management is Flow + State
Auto healing is Flow + Flow
Orchestration is Flow + State
Distributed Deployment is Flow + State
11 / 47

12. Salt philosophy Building blocks for infra
Building blocks for infra
Salt has “Configuration Management”
Salt has “Remote Execution”
12 / 47

13. Salt philosophy Building remote ex
Building remote ex
Async Message Bus
Micro publish commands
Minion Side Logic
Return to arbitrary location
13 / 47

14. Salt philosophy Building states
Building states
All about data
Low level data structure calls state functions
High level data structure defines state calls
Languages are now arbitrary
14 / 47

15. Salt philosophy Building cloud
Building cloud
Remote Ex controls hypers
Remote ex commands hypers
All runs on Live data
15 / 47

16. Salt philosophy Building monitoring
Building monitoring
Remote Ex build on system libs
Data gathering is already there!
States used to detect the state of a system
States can declare that the state of a system is out of bounds
Fires back to the event bus
16 / 47

17. Salt philosophy Conclusion
Conclusion
17 / 47

18. Salt philosophy Conclusion
Conclusion
Salt is comprised of many building blocks
The underlying components are available, but they do not need to be
interfaced with
Salt is made to morph into the needs of the infra
Made to make small infrastructures easy, and large infrastructures
powerful
18 / 47

19. Salt internals (briefly)
1 Salt philosophy
2 Salt internals (briefly)
Master
Minions
Execution modules
Execution example
State modules
State module example
3 Salt speed
4 Minion data
5 Events
19 / 47

20. Salt internals (briefly) Master
Master
salt-master -d
Open two ports (pub/sub & reply channel)
20 / 47

21. Salt internals (briefly) Minions
Minions
salt-minion -d
Connect to the master
No open ports required
Listens for pubs from the master
/etc/salt/minion:
#master: salt
21 / 47

22. Salt internals (briefly) Execution modules
Execution modules
Contain all the functionality
22 / 47

23. Salt internals (briefly) Execution example
Execution example
salt ’web-*’ network.interfaces
23 / 47

24. Salt internals (briefly) State modules
State modules
Wrap execution modules
Before-check
test=true
Call out to execution modules
After-check
24 / 47

25. Salt internals (briefly) State module example
State module example
top.sls:
base:
’web-*’:
- httpd
httpd.sls:
httpd:
pkg:
- installed
25 / 47

26. Salt speed
1 Salt philosophy
2 Salt internals (briefly)
3 Salt speed
Communication
pub/sub
4 Minion data
5 Events
6 Schedules
7 What’s coming
26 / 47

27. Salt speed Communication
Communication
ZeroMQ
msgpack
27 / 47

28. Salt speed pub/sub
pub/sub
Asynchronous
Minions determine targeting match
Minions do all the work
28 / 47

29. Minion data
1 Salt philosophy
2 Salt internals (briefly)
3 Salt speed
4 Minion data
Peer
Peer example
Salt Mine
Salt Mine example
Returners
Returner full-circle example
5 Events
29 / 47

30. Minion data
Live data; peer interface
Recent data; Salt mine
Historical data; returners
30 / 47

31. Minion data Peer
Peer
/etc/salt/master:
peer:
lb-.*:
- network.interfaces
Be mindful of data security
Communication still goes through the master
31 / 47

32. Minion data Peer example
Peer example
Configuring haproxy.cfg:
{% for server,ip in
salt[’publish.publish’](
’web*’,
’network.interfaces’,
[’eth0’]).items() %}
server {{ server }} {{ ip[0] }}:80 check
{% endfor %}
32 / 47

33. Minion data Salt Mine
Salt Mine
/etc/salt/{master,minion}:
mine_functions:
network.interfaces: [eth0]
mine_interval: 60
New in Salt v0.15
Either master or minion config
Be mindful of data security
33 / 47

34. Minion data Salt Mine example
Salt Mine example
Configuring haproxy.cfg:
{% for server,ip in
salt[’mine.get’](
’web-*’,
’network.interfaces’,
[’eth0’]).items() %}
server {{ server }} {{ ip[0] }}:80 check
{% endfor %}
34 / 47

35. Minion data Returners
Returners
/etc/salt/{master,minion}:
redis.db: 0
redis.host: myredis
redis.port: 6379
Minions write directly
Can be read into Pillar via ext_pillar
35 / 47

36. Minion data Returner full-circle example
Returner full-circle example
Collect the data:
salt ’web-*’ network.interfaces eth0
--return redis_return
Fetch the data via a custom ext_pillar module.
Use the data:
{% for server,ip in
salt[’pillar.get’](’web.ip_addrs’, {}).items()
server {{ server }} {{ ip[0] }}:80 check
{% endfor %}
36 / 47

37. Events
1 Salt philosophy
2 Salt internals (briefly)
3 Salt speed
4 Minion data
5 Events
Fire events
Watch for events (manually)
Reactor (react to events)
6 Schedules
7 What’s coming 37 / 47

38. Events Fire events
Fire events
salt ’lb-*’ event.fire_master
refresh_pool loadbalancer
38 / 47

39. Events Watch for events (manually)
Watch for events (manually)
Some assembly required
salt/tests/eventlisten.py
Coming soon to salt-api
39 / 47

40. Events Reactor (react to events)
Reactor (react to events)
/etc/salt/master:
reactor:
- loadbalancer:
- /src/reactor/refresh_pool.sls
/src/reactor/refresh_pool.sls:
{% if data[’type’] == ’refresh_pool’ %}
highstate_run:
cmd.state.highstate:
- tgt: lb-*
{% endif %}
40 / 47

41. Schedules
1 Salt philosophy
2 Salt internals (briefly)
3 Salt speed
4 Minion data
5 Events
6 Schedules
Add events
Stats gathering
7 What’s coming
41 / 47

42. Schedules Add events
Add events
/etc/salt/{master,minion} (or pillar):
schedule:
highstate:
function: state.highstate
minutes: 60
42 / 47

43. Schedules Stats gathering
Stats gathering
schedule:
uptime:
function: status.uptime
seconds: 60
returner: redis
meminfo:
function: status.meminfo
minutes: 5
returner: redis
43 / 47

44. What’s coming
1 Salt philosophy
2 Salt internals (briefly)
3 Salt speed
4 Minion data
5 Events
6 Schedules
7 What’s coming
Monitoring states
Data resolution
44 / 47

45. What’s coming
Salt v.0.next
45 / 47

46. What’s coming Monitoring states
Monitoring states
Configure inline with existing states
Individual components are in place
Needed: glue
Needed: alerting
46 / 47

47. What’s coming Data resolution
Data resolution
Time-series data
Thin and/or summarize older and older data
Free with some returners
47 / 47