Sides accompanying the tale of using saltcheck from the salt project to validate OS, app deployment, users, AWS resources, and more for multiple hadoop clusters.

1. Saltcheck
Christian McHugh
Linkedin: https://www.linkedin.com/in/christian-mchugh-a9a79949/
Github: mchugh19

2. Agenda
Tool Selection
Saltstack Background/Architecture
Problem Statement
Saltcheck Solution

3. Tool Selection

4. Saltstack Architecture
Saltstack is a remote execution client/server system which also does configuration
management.
Primary usage is to install a minion agent on infrastructure, and connect them all to a central
saltmaster which issues commands.
Saltmaster
Minion 1
Minion 2

5. Saltstack Architecture
Proxy minions can be used to connect to infrastructure which does not allow agent installations (network
switches)
Saltmaster
Minion 1
Proxy
minion 1
Switch 1
(As an alternative to the minion install, you can also connect to hosts from the master over ssh. You can also
run without a master and just use the schedule the minion to run on its own to control the host
configuration)

6. Saltstack Architecture
Salt-minion has a very modular codebase:
● Execution Modules (531) - Contain functionality to list, add, modify, etc things.
Generally used through the command line and internally by salt states.
● State modules - Host logic to control idempotency. Manages state logic and leverages
execution module functionality
● Render modules - Write state instructions in (default) yaml/jinja, but also allows full
python if desired
● Auth, pillar (external data sources), grains (variables on minion), engines, output, etc

7. Saltstack Architecture: yaml/jinja
/etc/http/conf/http.conf:
file.managed:
- source: salt://apache/http.conf
- user: root
- group: root
- mode: 644
{% set motd = ['/etc/motd'] %}
{% if grains['os'] == 'Debian' %}
{% set motd = ['/etc/motd.tail', '/var/run/motd'] %}
{% endif %}
{% for motdfile in motd %}
{{ motdfile }}:
file.managed:
- source: salt://motd
{% endfor %}

8. Saltcheck
Validates output of salt execution modules
Uses salt rendering system, allowing for use of jina/yaml, grains, pillars, secrets, etc
Since saltcheck is itself a salt execution module, it can be targeted an run like any other salt
function.

9. Saltcheck
Salt execution module: salt ‘*’ saltcheck.run_highstate_tests
package_latest:
module_and_function: pkg.upgrade_available
args:
- apache2
assertion: assertFalse
{% for package in ["apache2", "openssh"] %}
test_{{ package }}_latest:
module_and_function: pkg.upgrade_available
args:
- {{ package }}
assertion: assertFalse
{% endfor %}

10. Problem description
Validation of large hadoop clusters
● Code deploy
● Config file
● User management
● AWS environment
● Data state
● Network connectivity

11. Saltcheck Example: code deploy
{% set refdata_version = salt['pillar.get']('Refdata_tag:version') %}
{% set oozie_version = salt['grains.get']('ec2_tags:OOZIE_VERSION') %}
{% set prime_version = salt['grains.get']('ec2_tags:PRIME_RELEASE') %}
validate_refdata_jars:
module_and_function: hadoop.dfs_present
args:
- /refData/oozie-automation/{{ refdata_version }}_{{ prime_version
}}/oozie/workflows/refdata-config-{{ prime_version }}.jar
assertion: assertTrue
…

12. Saltcheck Example: config files
validate_oozie_env_jobTracker:
module_and_function: hadoop.dfs
args:
- text
- /oozie-automation/{{ oozie_version }}_{{ prime_version
}}/oozie/common/env.properties
expected-return: jobTracker = yarnRM
assertion: assertNotIn
print_result: False

13. Saltcheck Example: user management
{% for usr,data in salt['pillar.get']('users').iteritems() %}
validate_user_{{ usr }}:
module_and_function: user.info
assertion_section: shell
args:
- {{ usr }}
assertion: assertEqual
expected-return: /bin/bash
check_ssh_key_{{ usr }}:
module_and_function: ssh.check_key
args:
- {{ usr }}
- {{ data.key }}
- ssh-rsa
- ''
- ''
assertion: assertEqual
expected-return: exists
{% endfor %}

14. Saltcheck Example: s3 & python boto3
validate_oozie_env_jobTracker:
module_and_function: s3.get
args:
- mybucket
- dataset/{{ prime_version }}/data
assertion: assertTrue
boto3-installed:
module_and_function: cmd.run
args:
- python -c 'import pkgutil; print(1 if pkgutil.find_loader("boto3") else 0)'
assertion: assertEqual
expected-return: 1

15. Jenkins & Saltstack
https://plugins.jenkins.io/saltstack/
import groovy.json.*
node() {
saltresult = salt authtype: 'pam', clientInterface: local(arguments: '"emr.deploy"',
function: 'state.apply', target: 'ec2:role:hadoop-dev', targettype: 'grain'),
credentialsId: 'a3d814c2-84ed-4752-94a8-271791bb5375', servername:
'http://localhost:8000'
saltresult = salt authtype: 'pam', clientInterface: local(arguments: '"emr.deploy"',
function: 'saltcheck.run_state_tests', target: 'ec2:role:hadoop-dev', targettype:
'grain'),
credentialsId: 'a3d814c2-84ed-4752-94a8-271791bb5375', servername:
'http://localhost:8000'
def prettyJson = JsonOutput.prettyPrint(saltresult)
println(prettyJson)
}

16. Tool Summary
Saltcheck is able to use Salt’s remote execution capabilities and modules to talk to a wide
variety of infrastructure. All while utilizing the same familiar markup, and data backends.
For teams using Salt, saltcheck allows for knowledge reuse and shareable patterns

17. Tool Sprawl
Teams have limited resources and capacity for yet another thing
IT trends toward specialization, be aware of the load put on your teams

18. Thank you