This document provides an introduction to testing Salt states. It discusses why testing is important for infrastructure code and outlines approaches for testing at different levels, from basic syntax checking to integration and acceptance tests. Methods covered include using salt-call to test individual states, writing tests that run in isolated Docker containers to prevent interference, and automating testing with scripts to run multiple permutations and aggregate results. Part 2 will provide code examples for automation and discuss additional testing techniques.

1. Testing Salt States: Part 1
The Basics

2. About Me (Jason Denning)
• DevOps Engineer
• SaltStack user for ~ 3 years
• SaltStack Certified Engineer #2
• AWS Certified Architect
• OSS Advocate (first Linux install ~ 1998)
• Pythonista
• jasondenning on GitHub / @jason_denning on Twitter

3. ● Provide analytics and
marketing/segmentation tools for
mobile app developers
● Hosted backend APIs available world-
wide
● Lots of traffic
● Big fans of SaltStack
● upsight.com

4. WARNING: Work In Progress
● Testing infrastructure is not a common practice
● These slides are rough
● Your Mileage May Vary!
● We should work (together) on improving this

5. Why Test?
● DevOps: Infrastructure is code
● Code should be tested
● ⇒ Infrastructure should be tested

6. Why Test? (continued)
● Find errors faster
● Prevent regressions
● Allow others to contribute
● Simplify Code Reviews (CI)
● Improve Documentation

7. What to Test
● Basic Syntax (linting)
● Individual States
● State Relationships
● Different Environments
● As much as possible (within reason)

8. Testing Basics
● Tests should run quickly (and automatically)
● Tests should be isolated - know what you’re testing!
● Unit Tests: Test a single ‘unit’ of code
● Integration Tests: Test interactions between units
● Acceptance Tests: Test that code does what we want

9. Minimal Testing
$ salt-call state.highstate test=True
● Better than blindly applying
states
● Still not what we’re looking for
○ Manual
○ Slow
○ No isolation
○ Error-prone (“Oops, I
didn’t see the red line!”)

10. First Improvement
Test states individually:
$ salt-call state.sls foo test=True
● Slight improvement
○ More isolation
○ Faster than highstate
● Still not what we want

11. Testing Basic Syntax of SLS Files
● First Attempt: Verify SLS files are valid YAML
○ Problem: Only works for very basic states - Jinja breaks the test!
○ Problem: What about other renderers?
●
● Improvement: Test that Salt can render the SLS file
○ “salt-call state.sls foo test=True” works for this
○ Alternative: “salt-call state.show_sls foo”

12. Testing Basic Syntax
$ salt-call state.sls foo test=True
(with syntax error)
$ salt-call state.show_sls foo
(with syntax error)
Note: Better error message!

13. Testing Basic Syntax
$ salt-call state.show_sls foo
(no syntax error)

14. Testing Basic Syntax - Notes
● Salt built-ins are useful!
● Better: A separate tool for validating syntax
(somebody should build one!)
● Don’t forget about outputters:
● salt-call state.show_sls foo --output=json

15. Testing in Isolation
● We need to be sure that test results are not influenced
by previous tests
● Unit testing frameworks handle this automatically
● Ideal: Create a new, clean infrastructure for each test
○ Problem: Tests must run quickly!
● Solutions:
○ Use VMs (still to slow if we’re running lots of tests)
○ Use Containers (e.g. LXC / Docker)

16. Docker Containers vs. Full VMs
Docker Pros:
● Easy
● Quick to build and destroy
Docker Cons:
● Missing Functionality (testing services will take
more work)
● Linux only
VM Pros:
● Full OS install - services work / any OS
● Easy to manage if using tools such as Vagrant
● Might be identical to your actual environment
VM Cons:
● Slow(er) to build and destroy
Best Practice: Use both! (For different types of test)

17. Docker: Quickstart
● Docker makes it easy to build & manage LXC containers
● Allows one to quickly customize existing container images
● Mount host directories in the container
Basic Dockerfile (~/saltpdx/Dockerfile):
FROM phusion/baseimage
RUN sudo add-apt-repository -y ppa:saltstack/salt
RUN sudo apt-get update
RUN sudo apt-get -y install salt-minion
VOLUME ["/etc/salt", "/salt/states", "/salt/pillar"]
CMD salt-call state.highstate
● Build an image (tagged salt/test, version 1):
$ docker build -t ‘salt/test:1’ .
● Start a container using the image:
$ docker run --rm -it -v ~/saltpdx/salt:/etc/salt -v ~/saltpdx/states:/salt/states -v /
~/saltpdx/pillar:/salt/pillar salt/test:1

18. Docker Quickstart
● Docker creates a container
based on the image
● Host directories are
specified using the -v flag
● Docker starts the container
and runs the CMD specified
in the Dockerfile
● --rm flag tells docker to
delete the container after
the command has run
● -it (or -i -t) for interactive
terminal mode (add
‘/bin/bash’ to end of
command to get a shell)

19. Docker Tips
● Mount host directories as volumes for:
○ Salt configuration
○ States
○ Pillar
○ Logs?
● Configure containers as salt-masterless to avoid networking & PKI
issues
● Use the --rm flag to automatically delete container instances after
they’ve run
● Tag & version images

20. Docker: Entrypoint
● When you run a Docker container, Docker executes a binary called ENTRYPOINT
● The CMD that you pass to Docker (via Dockerfile or the cli) is passed as an argument to
the ENTRYPOINT
● By default, Docker will use “/bin/sh -c” as the ENTRYPOINT - this can be overridden
● e.g.:
$ docker run --rm --entrypoint /bin/echo salt/test:1 foo
● We can exploit this to make a simple test runner
● Dockerfile:
FROM phusion/baseimage
RUN sudo add-apt-repository -y ppa:saltstack/salt
RUN sudo apt-get update
RUN sudo apt-get -y install salt-minion
VOLUME ["/etc/salt", "/salt/states", "/salt/pillar"]
ADD test_salt.sh /test_salt.sh
RUN chmod +x /test_salt.sh
ENTRYPOINT ["/bin/bash", "/test_salt.sh"]

21. Docker: Entrypoint (con’t)
● test_salt.sh:

22. Docker: Entrypoint (con’t)
After building a new image, we can
run:
$ docker run --rm -v / ~/saltpdx/salt:
/etc/salt -v / ~/saltpdx/states:
/salt/states -v / ~/saltpdx/pillar:
/salt/pillar salt/test:2 / highstate
Which will start a container, run “salt-
call state.highstate”, and exit.

23. Docker: Entrypoint (con’t)
Or, we can run:
$ docker run --rm -v / ~/saltpdx/salt:
/etc/salt -v / ~/saltpdx/states:
/salt/states -v / ~/saltpdx/pillar:
/salt/pillar salt/test:2 / foo
Which will run “salt-call state.
show_sls foo”, then “salt-call state.sls
foo”, and exit.

24. Next Step: Automation
● We’ve got the basic tools - the next step is to add some automation
● Host-side automation:
○ Starting and destroying containers
○ Determine what states to test
○ Collect & summarize test results
○ Test various permutations (different Linux version, different
grains, etc.)
● Container-side automation
○ Discover & test sub-components (e.g. file templates)
○ Integration/Acceptance/Behavioral Tests

25. Automation Overview
● Testing is initiated by running a test-controller script on the host
● The test-controller should:
○ Generate a list of states to test and environment permutations
○ Start a container and trigger the test runner
○ Collect test results
○ Destroy the container
○ Repeat for each state
● The Docker image should have a test-runner script that is used as
the ENTRYPOINT
● The test-runner script should:
○ Determine specific tests to run
○ Run the tests
○ Output Results

26. Next Time - Part 2
● Actual code for these automation scripts
● Behavioral Testing (for acceptance/integration tests)
● Collecting and Summarizing Results in a useful fashion
● Continuous Integration
● Adding Vagrant VMs for additional tests
● Testing multi-machine infrastructures
● …???

27. Thanks!
Discussion / Questions?
salt@jasondenning.com
GitHub: jasondenning
Twitter: @jason_denning