Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Understanding salt modular sub-systems and customization


Published on

Overview of the modular sub-systems in SaltStack, and basics for writing custom execution modules and state modules for Salt. Presented to the Portland SaltStack Users Group ( by me (Jason Denning) on 04/21/2015.

Published in: Software
  • Be the first to comment

Understanding salt modular sub-systems and customization

  1. 1. Understanding Salt Modular Sub-Systems and Customization
  2. 2. About Me (Jason Denning) • DevOps Engineer • SaltStack user for ~ 3 years • SaltStack Certified Engineer #2 • AWS Certified Architect • OSS Advocate (first Linux install ~ 1998) • Pythonista • jasondenning on GitHub / @jason_denning on Twitter
  3. 3. ● Provide analytics and marketing/segmentation tools for mobile app developers ● Hosted backend APIs available world- wide ● Lots of traffic ● Big fans of SaltStack ●
  4. 4. Frustratingly Flexible • Salt is extremely flexible and extensible • Highly modular / easy to customize • Can be difficult (at first) to know where your custom code should go • Occasionally confusing nomenclature • Pillar • Grains • Mine • Reactor • ...
  5. 5. But once you know your way around.. ...Salt’s pretty awesome • Easy to customize • Scalable • Secure • Doesn’t suffer from the Pareto Principal (80% rule)
  6. 6. This Talk You’ll (hopefully) understand: • When to use different functionality • How all the pieces fit together • How to get started customizing Salt • Where to go next
  7. 7. Salt’s not Configuration Management • Salt is a remote execution framework • Run arbitrary code on remote hosts • Tools to make this easy and flexible • Configuration Management is just a common use-case • Originally intended to be a tool for managing cloud-based infrastructure
  8. 8. Remote Execution 101 Basic remote execution: $ ssh jason@myhost "ls /home/jason" it_worked.txt Multiple hosts? $ for h in myhost1 myhost2 myhost3; > do ssh jason@myhost "ls /home/jason"; > done
  9. 9. Remote Execution 101 Ok, but what if: • I want to run it on 100 hosts? • My login credentials are different on some hosts? • I actually want to do something with the output? • I want to do this many times?
  10. 10. Remote Execution 102 How about a script? #!/bin/bash # Make sure your SSH config (~/.ssh/config) is setup! command=$1 host_list=$2 for h in host_list; do ssh "$h" "$command" done # FIXME: Add output handling # FIXME: Add error handling # FIXME: Add logging # TODO: FIND A BETTER WAY TO DO THIS!!!!
  11. 11. Looks like we need: A script that can handle: ● Executing arbitrary commands ● ...on one or more remote hosts ● …with a sane way to get the output (STDOUT? Write a file? Text output? JSON? YAML??) ● …and graceful error handling ● …and logging ● …and authentication ● …and everything else I haven't thought of yet!
  12. 12. And then we realize.. ● We only want to run a command if the host is in a particular state... ● the command is different for some hosts because they have a different OS... ● we need to configure host X before we configure host Y... ● we want to generate a file from a template, using data from an external database ● we need to keep that external data secret
  13. 13. You want me to maintain a 5,000 line BASH Script?
  14. 14. I’m convinced - let’s use a framework This is why Salt has so many modular subsystems: • Each type of component scratches a particular itch (there’s some overlap) • ⇒ Modular • Someone is inevitably going to want something slightly different • ⇒ Extensible • We’re almost always going to want things like error handling • ⇒ One framework instead of several mini-frameworks
  15. 15. Salt Basics • A host can be a “master”, “minion”, “masterless”, or any combination • a master can instruct minions to run code (via ‘salt’ cmd) • a minion can trigger actions on itself using code/configuration stored on its master (via ‘salt-call’ cmd) • a minion can run in isolation, called “masterless” (via ‘salt-call’) • a single machine can run both master and minion daemons The point: Some subsystems run on the master, some on the minion
  16. 16. The Event Sub-System Salt transfers commands and other data (including files)between hosts via the Event Sub-System. • Publish/Subscribe message queue (based on ZeroMQ) • Master can publish messages on minion queues, and vice-versa • Code execution on the minion is triggered via Event messages • Output is sent back to the master from the minion in the same fashion • Messages are encrypted, and authenticated with Salt’s Public Key Infrastructure (PKI) • Reactor: Watches the message queue on the master, and triggers code execution when it sees specific events
  17. 17. How Salt Works (simplified) $ salt minion1 Salt Master <do stuff> minion1 Master Message Bus (zmq / RAET) Message (Event) Message (Event) Minion Message Bus (zmq / RAET) Message (Event) Message (Event) <Encrypted>
  18. 18. Code Execution Sub-Systems • Execution Modules (a.k.a. “modules”): Code that executes on the minion • Runner Modules: Code that executes on the master • State Modules: Code that executes on the minion, depending on the current state of the host (typically utilize execution modules)
  19. 19. Code Execution: CLI • ‘salt’ command: (from master) instruct minion(s) to invoke an execution module, which executes code on the minion • ‘salt-call’ command: (from minion) invoke an execution module, which executes code on the minion • ‘salt-run’ command: (from master) invoke a runner module, which executes code on the master
  20. 20. Data Sub-Systems • Pillar: Data which is passed to a particular minion via the master • Pillar Modules: Code that runs on the master which generates Pillar data • Grains: Data which is set directly on the minion • Grains Modules: Code that runs on the minion which generates Grains data • Mine: Data collected from minion(s) which is stored on the master, and made available to all other minions
  21. 21. Major Subsystems: Output Handling • Output Modules: Format output from the CLI • e.g. text (human-readable), YAML, JSON • Returner Modules: Send output to external data-stores • e.g. syslog, MySQL, etcd • Note: Minions always return output to the master on the Event bus, regardless of whether Returner modules are used or not
  22. 22. SLS Files • Human-readable files, with the .sls extension • Processed by the Renderer subsystem to create data structures that are understood by other sub-systems • Most users interact with Salt via SLS files • Renderer Modules: Convert SLS files into usable data structures (execute on both master and minion) • Composable (e.g. Jinja + YAML) • Can access Pillar and Grains data
  23. 23. Targeting Minions • A salt-master can target minions based on the minions’ ids, Grains, or Pillar data • States and Pillar data are assigned to via the Master Tops sub- system (typically, this is just a file, called top.sls) • Salt-SSH targets hosts (which aren’t running the minion daemon) via the Roster sub-system
  24. 24. Other Modular Sub-Systems • Auth : Enable external authentication systems • Fileserver: File storage backends used by the master • Wheel: Used to manage the master’s configuration • Net-API: Access the master via the web New in 2015.2: • Engine: Long-running code on either master or minion • Beacons: Code running on minions, used to translate external events to the Event bus
  25. 25. More Stuff!?!? • Salt Cloud: Manage cloud (IaaS) infrastructure • Salt SSH: Apply salt states on hosts without a minion daemon, via SSH • Salt Virt: Manage virtual hosts and hypervisors • Proxy-Minions: Manage “dumb” hardware, such as network hardware
  26. 26. That’s a lot of sub-systems!
  27. 27. Let’s Get Customizin’ • First, make sure that you actually need to write code • • • … etc. • • Look at existing module code: • • • … etc.
  28. 28. Custom Execution Modules # _modules/ ‘’’ This comment will print if you run $ salt-call sys.doc hello ‘’’ __virtualname__ = ‘hello’ def world(): ‘’’ Prints “Hello World!” - this will print if you run $salt-call sys.doc’’’ print(“Hello World!”)
  29. 29. Custom Execution Modules • Need to determine if the module can run on this minion? • write a function called “__virtual__()” • if __virtual__() returns False, the module will not be available • Useful if your code has external dependencies • Need a “private” function? • Prefix the function name with an underscore • i.e. _this_is_invisible_to_the_user() • Any function that doesn’t begin with an underscore will be callable
  30. 30. Custom Execution Modules - Imports # A useful pattern if you need to import an external library try: import foomodule HAS_FOO = True except ImportError: HAS_FOO = False def __virtual__(): ‘’’Only load the execution module if we could import foomodule’’’ if HAS_FOO: return True else: return False
  31. 31. Accessing Other Modules • When Salt loads your module, it will create a couple of magic dicts, just like in SLS files • __salt__ : use to call other execution modules • e.g. __salt__[‘’](“ls /”) • __grains__ : use to access grains • e.g. minion_id = __grains__[‘id’]
  32. 32. Custom State Modules • Much like execution modules • Need to return a dict with this format: { ‘result’: True, ‘comment’: “Info about the state execution”, ‘changes’: { ‘old’: ‘the state before function was applied’, ‘new’: ‘state after the function was applied’ } } • ‘result’ should be True if the state is correct, False if there was an error, or None if this was a test run
  33. 33. Custom State Modules - Tips • State functions must accept a ‘name’ argument, at minimum • Enable test mode! ($ salt-call state.sls test=True) • Write a function called “mod_init(low)” which accepts the Low State Data Structure to do one-time setup and initialization • If you want to enable the ‘watch’ requisite, write a function called “mod_watch()” - see the service state for a good example
  34. 34. Example State From the documentation:
  35. 35. Recap • Salt makes it easy to run custom code on your infrastructure, with lots of icing • Lots of modular sub-systems, but they’re useful! • Write custom code once, run many times • No 5,000 line BASH scripts!
  36. 36. Now we can run whatever code we want!
  37. 37. Thanks! Questions? GitHub: jasondenning Twitter: @jason_denning