Saltconf 2016: Salt stack transport and concurrency

Salt Transport Modularity and
Concurrency for Performance and
Scale
Thomas Jackson
Staff Site Reliability Engineer
LinkedIn

3
Agenda
• for item in (‘transport’, ‘concurrency’):
• History
• Problems
• Options
• Solution

Transport in Salt
4
Salt Transport: a history
• In the beginning Salt was primarily a remote execution engine
• Send jobs from Master to N minions (defined by some target)
• In the beginning there was

5
"ZeroMQ (also spelled ØMQ, 0MQ or ZMQ)
is a high-performance asynchronous
messaging library, aimed at use in
distributed or concurrent applications.”
- Wikipedia (https://en.wikipedia.org/wiki/ZeroMQ)

6
We took a normal TCP socket, injected it with a mix of radioactive
isotopes stolen from a secret Soviet atomic research project,
bombarded it with 1950-era cosmic rays, and put it into the hands
of a drug-addled comic book author with a badly-disguised fetish
for bulging muscles clad in spandex. Yes, ZeroMQ sockets are the
world-saving superheroes of the networking world.
- http://zguide.zeromq.org/page:all#How-It-Began

7
How ZMQ PUB/SUB looks
Server
context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.bind("tcp://*:12345")
socket.send(”Message")
Client
socket = context.socket(zmq.SUB)
socket.connect("tcp://localhost:12345")
print socket.recv()

8
How ZMQ REQ/REP looks
Server
socket = context.socket(zmq.REP)
socket.bind("tcp://*:12345")
message = socket.recv()
socket.send(“got message”)
Client
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:12345")
socket.send("Hello”)
message = socket.recv()

Request lifecycle
9
Master Minion
1. Job publish
2. Sign-in (optional – potentially reused or cached)
3. Pillar Fetch
4. SLS/file fetch (optional)
5. Return

Initial ZeroMQ implementation
10
• Master-initiated messages
• Using the pub/sub socket pair in zmq
• All broadcast messages from the master to the minion
• Minion-initiated messages
• Using the req/rep socket pair in zmq
• All messages initiated by the minion, such as:
• Sign-in
• Job return
• Module sync
• Pillar
• Etc.

Initial problems
11
• Message loss
• Broadcasts where filtered client side
• Added zmq filtering: https://github.com/saltstack/salt/pull/13285
• Etc.

Larger problems
13
• Huge ZMQ publisher memory leak (https://github.com/zeromq/libzmq/issues/954)
• Workaround: Process manager in salt
• No concept of client state
• When messages arrive, there is no way to see if the client is still connected– which leads to auth storms
• Workaround: Exponential backoff on the minion side
• No sync "connect" (https://github.com/saltstack/salt/pull/21570)
• Workaround: fire event and wait for it to return (or timeout to expire)
• Some users have issues with the LGPL license
• Workaround: n/a 

15
The Reliable Asynchronous Event Transport, or
RAET, is an alternative transport medium developed
specifically with Salt in mind. It has been developed to
allow queuing to happen up on the application layer
and comes with socket layer encryption. It also
abstracts a great deal of control over the socket layer
and makes it easy to bubble up errors and exceptions.
- docs.saltstack.com
Salt Transport: previous attempt

RAET
16
Salt Transport: previous attempt
• The good
• No ZMQ!
• The bad
• Effectively a re-implementation of the daemons (separate files, etc.)
• Unable to run zmq and RAET simultaneously (initially, hydra was added later – which just runs both daemons at once)
• The different
• Changed the model from “minions always connect” to “minions are listening”, meaning minions have a socket to
attack

What do we really need
18
Salt Transport: back to basics
• Salt is a platform, not a specific transport– we need transports to be modular
• Some requirements:
• Simple interface to implement (such that other modules can be written)
• Test coverage (including pre-canned tests for new modules)
• Support N transports simultaneously (for ramps, and complex infra)
• Clear contract of security/privacy requirements of various methods

• ReqChannel: minion to master messages
19
Salt Transport: Channels!
• Master
• pre_fork(self, process_manager)
• post_fork(self, payload_handler, io_loop)
• Minion
• send(self, load, tries=3, timeout=60)
• crypted_transfer_decode_dictentry(self, load, dictkey=None, tries=3, timeout=60)

• PubChannel: broadcasts to the appropriate minions
20
• Master
• pre_fork(self, process_manager)
• publish(self, load)
• Minion:
• on_recv(self, callback)

Responsibilities
21
• Serialization
• Encryption
• Targeting (pub channel only)

TCP channel
22
• Wire protocol: msgpack({'head': SOMEHEADER, 'body': SOMEBODY})
• Main advantages over ZMQ? better failure modes
• Faster failure detection (if minion isn’t connected to the master, you don’t have to wait for the timeouts)
• True link-status (no more auth storms!)
• Basically, we have sockets again! 
• https://docs.saltstack.com/en/develop/topics/transports/tcp.html

TCP: How does it look?
23
async_channel = salt.transport.client.AsyncReqChannel.factory(minion_opts)
ret = yield async_channel.send(msg)

TCP: How accurate?
24
• ZeroMQ
• Total jobs: 1000
• Completed jobs: 171
• Hit rate: 17.1%
• TCP
• Total jobs: 1000
• Completed jobs: 1000
• Hit rate: 100%

TCP: How does it perform
25
• 15 byte message
• ZeroMQ*
• Average time: 0.00295809405715
• QPS: 2246.952241147
• TCP
• Average time: 0.0023341544863
• QPS: 2580.04452801

TCP: How does it perform
26
• 1053 byte message
• ZeroMQ*
• Average time: 0.00278297542184
• QPS: 2489.300394919
• TCP
• Average time: 0.00251070397869
• QPS: 2602.4855051

Awesome!
27
• Definitely awesome!
• But async? What was that about?
• Before we get into specifics, lets talk about concurrency

The General Problem
28
Concurrency
We have lots of things to do, some of which are blocking calls to remote things which
are “slow”. It is more efficient (and overall “faster”) to work on something else while we
wait for that “slow” call.

Current state of concurrency in Salt
30
Concurrency
• Master-side: the master creates N Mworkers to process N requests in parallel
• N Mworkers to process N requests in parallel
• Interaces with non-blocking as well, using `while True:` loops to do timeouts etc.
• Minion-side:
• Threads used in MultiMaster for managing the multiple master connections

Problems
31
Concurrency
• No unified approach (multiprocessing, threading, nonblocking “loops” -- all in use)
• Slow and/or blocking operations hold process/thread while waiting
• No consistent use of non-blocking libraries, so the code is a mix of loops and
blocking calls
• Limited scalability (each approach scales differently)

Common solutions in Python
32
Concurrency
• Threading
• Multiprocessing
• User-space “threads”: Coroutines / stackless threads

33
Concurrency
Threading
• Some isolation between threads
• Pre-emptive scheduling
Import threading
def handle_request():
ret = requests.get(‘http://slowthing/’)
# do something else
threads = []
for x in xrange(0, NUM)REQUESTS):
t = threading.Thread(target=handle_request)
t.start()
threads.append(t)
for t in threads:
t.join()

34
Concurrency
Multiprocessing
• Complete isolation
• Pre-emptive scheduling
Import multiprocessing
def handle():
ret = requests.get(‘http://slowthing/’)
# do something else
Processes = []
for x in xrange(0, NUM)REQUESTS):
p = multiprocessing.Process(target=handle)
p.start()
processes.append(p)
For p in processes:
p.join()

• User-space “threads”: Coroutines / stackless threads
35
Concurrency
• Some libraries you may have heard of
• gevent
• Stackless python
• Greenlet
• Twisted
• Tornado
• How are these implemented
• Green threads
• callbacks
• coroutines

Why Coroutines?
36
Concurrency
• Coroutines have been in use in python for a while (tornado)
• The new asyncio in python3 (tulip) is coroutines
(https://docs.python.org/3/library/asyncio.html)

37
Coroutines are computer program components
that generalize subroutines for nonpreemptive
multitasking, by allowing multiple entry points
for suspending and resuming execution at
certain locations.
- https://en.wikipedia.org/wiki/Coroutine
Concurrency

38
Concurrency
Coroutines– what is this magic?
def item_of_work():
while True:
input = yield
yield do_something(input)

39
Concurrency
Coroutines– what is this magic?
def some_complex_handle():
while True:
input = yield
out1 = do_something(input)
yield None
out2 = do_something2(out1)
yield None
return do_something3(out2)

40
Concurrency
Tornado coroutines
• Some isolation between coroutines
• Explicit yield
• Light “threads”
Import threading
@tornado.gen.coroutine
def handle_request():
ret = yield requests.get(‘http://slow/’)
# do something else
loop = tornado.ioloop.IOLoop.current()
loop.spawn_callback(handle_request)
loop.start()

Coroutines– futures
41
Concurrency
• Futures are just objects that represent a thing that will complete in the future
• This allows methods to return immediately, but finish the task in the future
• This allows the callers to yield execution until the futures they depend on complete

42
Concurrency
Coroutines– with futures
• Yield execution, and get returns
• Method looks fairly normal
• Stack traces in here have context
• Easy chaining of futures
@tornado.gen.coroutine
def some_complex_handle(request):
a = yield is_authd(request)
if not a:
return False
ret = yield do_request(request)
yield save1(ret), save2(ret)
return ret

Tornado in Salt
43
Concurrency
• What is tornado?
• Python web framework and asynchronous networking library
• Why Tornado and not asyncio?
• Free python 2.x compatibility!
• A fairly comprehensive set of libraries for it (http, locks, queues, etc.)

Back to the transport interfaces
44
Concurrency
• AsyncReqChannel
• send: return a future
• crypted_transfer_decode_dictentry: return a future
ret = yield channel.send(load, timeout=timeout)

Now what?
45
Concurrency
• Now that we have a real concurrency model, what have we done with it?
• MultiMinion in a single process (coroutine per connection)
• Easily implement concurrent networking within Salt
• TCP transport
• IPC

Really? Problems?
47
Concurrency problems
• Most common pitfalls to concurrent programming
• race conditions and memory collisions
• deadlocks

Race conditions
48
• Weird data problems in the reactor: https://github.com/saltstack/salt/issues/23373
• The underlying problem: injected stuff in modules (__salt__ etc.) were just dicts—
which aren’t threadsafe (or coroutinesafe!)
• The solution? `ContextDict`

Copy-on-write thread/coroutine specific dict
49
ContextDict
• Works just like a dict
• Exposes a clone() method, which creates a `ChildContextDict` which is a
thread/coroutine local copy
• With tornado’s StackContext, we switch the backing dict of the parent with your
child using a context manager
cd = ContextDict(foo=bar)
print cd[‘foo’] # will be bar
with tornado.stack_context.StackContext(cd.clone):
cd[‘foo’] = ‘baz’
print cd[‘foo’] # will be baz
More examples: https://github.com/saltstack/salt/blob/develop/tests/unit/context_test.py

Deadlocks
50
• haven't seen any yet *knock on wood* -- in general we avoid these since each
coroutine is more-or-less independent of the others

Layers!
51
• Don’t forget, concurrency at all layers– including your DC-wide state execution
• For example: automated highstate enforcement of your whole DC
• Does it matter if all DB hosts update at once?
• Does it matter if all web servers update at once?
• Does it matter if all edge boxes update at once?

concurrency controls for state execution
52
zk_concurrency
acquire_lock:
zk_concurrency.lock:
- name: /trafficeserver
- zk_hosts: 'zookeeper:2181'
- max_concurrency: 4
- prereq:
- service: trafficserver
trafficserver:
service.running: []
release_lock:
zk_concurrency.unlock:
- name: /trafficserver
- require:
- service: trafficserver

Things on my “list”
53
Future Awesomeness
• Transport
• failover groups
• even better HA (https://github.com/saltstack/salt/issues/25700 -- get involved in the conversation)
• Concurrency
• async ext_pillar
• Partially concurrent state execution (prefetch, etc.)?
• Coroutine-based:
• Reactor
• Engines
• Beacons
• Thorium

Saltconf 2016: Salt stack transport and concurrency

More Related Content

What's hot

Viewers also liked

Similar to Saltconf 2016: Salt stack transport and concurrency

Recently uploaded

Saltconf 2016: Salt stack transport and concurrency

Editor's Notes