Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

SaltConf14 - Ben Cane - Using SaltStack in High Availability Environments

on

  • 892 views

An overview on the benefits and best practices of using SaltStack for consistency and automation in highly available enterprise environments such as financial services.

An overview on the benefits and best practices of using SaltStack for consistency and automation in highly available enterprise environments such as financial services.

Statistics

Views

Total Views
892
Views on SlideShare
887
Embed Views
5

Actions

Likes
5
Downloads
27
Comments
0

1 Embed 5

http://www.slideee.com 5

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

SaltConf14 - Ben Cane - Using SaltStack in High Availability Environments Presentation Transcript

  • 1. Using SaltStack in High Availability Environments
  • 2. Who is this guy?
  • 3. Who is this guy? My name is Benjamin Cane, I run a blog bencane.com focused on Linux Systems Administration. By day I am a Solutions Architect working in the financial services industry. My focus is solely on building mission critical environments that require 99.99% to 99.999% availability. In other words High and Continuous Availability environments. This presentation is going to cover a few best practices for using SaltStack in High Availability environments. Note: The views, opinions and recommendations in this presentation are entirely my own and do not represent the views of my employer.
  • 4. High Availability
  • 5. What is High Availability High availability is a system design approach and associated service implementation that ensures a prearranged level of operational performance will be met during a contractual measurement period. - Source: Wikipedia A Highly Available environment is a system that is designed to be available to the end user the majority of the time.
  • 6. High Availability In Numbers High Availability is usually measured by the downtime a system experiences in a year. What constitutes High Availability is subjective to the business and systems at hand. Availability % Downtime Per Year 90% 36.5 Days 99% 3.65 Days 99.9% 8.76 Hours 99.99% 52.56 Minutes 99.999% 5.26 Minutes 99.9999% 31.5 Seconds 99.99999% 3.15 Seconds
  • 7. What Causes Downtime Usually Humans
  • 8. Best Practices
  • 9. Replace manual processes
  • 10. Replacing Humans with SaltStack Since humans are one of the top causes of service unavailability, it only makes sense to replace them with automated processes.
  • 11. Building and Provisioning By using a configuration management tool such as SaltStack you can automate the installation and configuration of new systems. By automating this task you can ensure that a system is deployed the same way every time. Ensuring a system is deployed in the exact same method every time reduces configuration time bombs from impacting production traffic later.
  • 12. Automate Everything The goal should be to automate everything. Not only does this decrease provisioning time, but rather the more items that are automated the less likely those items are to be repeatedly misconfigured or forgotten.
  • 13. Automating Server Configuration One of the most opportune points for human error is in the server configurations. SaltStack has great functionality around configuring a server, and it should be used to automatically deploy server side configurations. In the next few slides I will show how I created a pillar file that has all of my host specific information inside. This pillar is then used to deploy a configuration file the same way every time using templates.
  • 14. Server Settings Pillar In this example you will see a pillar labeled systems that sets items such as the SSH Port, Node Groups, Active Interfaces and the IP information for those interfaces. systems: {% if grains['fqdn'] == 'hostname01.example.com' %}   sshport: 22   nodegroups:     - website_apps   interfaces:     - eth0   defaultgw: 10.0.0.1   ip:     eth0: 10.0.0.10   mask:     eth0: 255.255.255.0   mtu:     eth0: 1500 {% elif grains['fqdn'] == 'hostname02.example.com' %} systems: {% if grains['fqdn'] == 'hostname01.example.com' %}   sshport: 22   nodegroups:     - website_apps   interfaces:     - eth0   defaultgw: 10.0.0.1   ip:     eth0: 10.0.0.10   mask:     eth0: 255.255.255.0   mtu:     eth0: 1500 {% elif grains['fqdn'] == 'hostname02.example.com' %}
  • 15. System Settings State File Below is an example of deploying network interface files using the systems pillar values. {% for interface in pillar['systems']['interfaces'] %} /etc/sysconfig/network-scripts/ifcfg-{{ interface }}:   file.managed:     - source: salt://network/config/etc/sysconfig/network-scripts/ifcfg-interfaces     - user: root     - group: root     - mode: 644     - template: jinja     - context:       interface: {{ interface }}       ip: {{ pillar['systems']['ip'][interface] }}       mask: {{ pillar['systems']['mask'][interface] }}    mtu: {{ pillar['systems']['mtu'][interface] }} {% endfor %} {% for interface in pillar['systems']['interfaces'] %} /etc/sysconfig/network-scripts/ifcfg-{{ interface }}:   file.managed:     - source: salt://network/config/etc/sysconfig/network-scripts/ifcfg-interfaces     - user: root     - group: root     - mode: 644     - template: jinja     - context:       interface: {{ interface }}       ip: {{ pillar['systems']['ip'][interface] }}       mask: {{ pillar['systems']['mask'][interface] }}    mtu: {{ pillar['systems']['mtu'][interface] }} {% endfor %}
  • 16. Benefits of using pillars to define systems configuration • Reduces number of configuration files that need to be manually edited (only 1 file to peer review as well) • You can stage hosts configuration in advance allowing for rapid server provisioning and re-provisioning • Configurations are always consistent, no setting the default route in /etc/sysconfig/network on some servers and /etc/sysconfig/network-scripts/ifcfg-eth0 on others • Files are always deployed the same way every time
  • 17. Automating Application Installation SaltStack allows you to automate application installation easily. This should be one of the first items to automate. By automating the installation of applications you can ensure that the software is installed the same way each time, every time.
  • 18. Install a Package Installing a package via a package manager is pretty straight forward. mysql-server: pkg: - installed mysql-server: pkg: - installed The above configuration is used for package managers such as yum for Red Hat based distributions or apt-get for Debian based distributions.
  • 19. 3rd Party Vendor Applications Some 3rd party software companies don't always make it easy to automate a software's installation. You should do it regardless of the difficulty. You may need to create a wrapper script, tell SaltStack to run the 3rd party software vendors installation scripts, or become very familiar with expect. But all of this work up front will save countless hours over the years of installing and re-installing software.
  • 20. Installing Software with a Script SaltStack can be used to run any script, including scripts that are packages with software. /some/path/custom-installer.sh:   cmd.run:     - unless: test -f /some/path/bin/start.sh - order: last - stateful: True - require: - pkg: jre - user: someguy /some/path/custom-installer.sh:   cmd.run:     - unless: test -f /some/path/bin/start.sh - order: last - stateful: True - require: - pkg: jre - user: someguy
  • 21. Make all of your scripts stateful Saltstack can understand if the script was successful or not. To do this add the following to your .sls file. - stateful: True- stateful: True Then add the following to the bottom of your scripts output. echo  # an empty line here so the next line will be the last. echo "changed=yes comment='Job Accomplished'" echo  # an empty line here so the next line will be the last. echo "changed=yes comment='Job Accomplished'" echo  # an empty line here so the next line will be the last. echo "changed=no comment=’Oh Noes'" echo  # an empty line here so the next line will be the last. echo "changed=no comment=’Oh Noes'" Bad Results: Good Results:
  • 22. Installing tar packaged applications You can also use the archive module to extract software that is packaged and deployed via tar files. appinstaller:   module.run:     - name: archive.tar     - options: --owner username --group groupname xzf     - tarfile: /path/to/file/on/server/somefile.tar.gz     - cwd: /software/dir/     - require:       - user: username     - unless: test -f /software/dir/bin appinstaller:   module.run:     - name: archive.tar     - options: --owner username --group groupname xzf     - tarfile: /path/to/file/on/server/somefile.tar.gz     - cwd: /software/dir/     - require:       - user: username     - unless: test -f /software/dir/bin
  • 23. Configuring Applications Why stop at just automating the installation of the software? Why not automate the configuration of it as well?
  • 24. Using Templates SaltStack allows you to utilize templates to deploy consistent configuration files. With a combination of pillars and templates you can deploy custom configuration files easily. /etc/mysql/my.cnf:   file.managed:     - source: salt://mysql/config/etc/mysql/my.cnf     - user: root     - group: root     - mode: 644     - template: jinja     - context:       listenip: {{ salt['network.interfaces']()['eth0']['inet'][0]['address'] }} /etc/mysql/my.cnf:   file.managed:     - source: salt://mysql/config/etc/mysql/my.cnf     - user: root     - group: root     - mode: 644     - template: jinja     - context:       listenip: {{ salt['network.interfaces']()['eth0']['inet'][0]['address'] }}
  • 25. Running Custom Commands Some applications require a command to be run after installation. Just like installing software you can configure the software with cmd.run as well. emcpreg:   cmd.run:     - name: /sbin/emcpreg -add SOME-EMC-LICENSE-STRING     - onlyif: /sbin/emcpreg -list 2>&1 | grep -q "No such file"     - require:       - pkg: EMCpower.LINUX emcpreg:   cmd.run:     - name: /sbin/emcpreg -add SOME-EMC-LICENSE-STRING     - onlyif: /sbin/emcpreg -list 2>&1 | grep -q "No such file"     - require:       - pkg: EMCpower.LINUX
  • 26. Change execution The execution of changes is another human driven process that is prone to simply mistakes; mistakes that can affect availability. If you want to mitigate human errors during changes, it requires that all changes to testing and production environments be 100% deployed via configuration management tools. By using a configuration management tool you can avoid root causes such as: •/etc/hosts file was deployed with incorrect permissions •Required configuration file was not deployed with updated application
  • 27. Automated state runs With SaltStack you can have each server pull the desired states and update automatically by putting the following into a cronjob on each minion. # call a high-state every 5 minutes */5 0 0 0 0 salt-call state.highstate # call a high-state every 5 minutes */5 0 0 0 0 salt-call state.highstate This is great when you are running hundreds or thousands of hosts that all perform the same jobs and can take over for others. Or for systems that can be restarted quickly without human interaction or service disruption (i.e. Apache, NGINX). As of 0.12 SaltStack now has a scheduler function that allows you to define a state run on a defined schedule on either the master or minions. Protip: You should stagger the schedules (cron or salt scheduler) to ensure that every server doesn’t restart services at the same time.
  • 28. Automatic state runs is not appropriate for every situation While having every system run a high state every 2 minutes sounds great; it is not the best answer in every case. Knowing when to automate everything and when to keep some things semi-automated is key to building highly available systems. If the systems you are managing are extremely sensitive to changes and application restarts. It is best to establish a policy where the high state is run manually at a scheduled time of day or week that aligns with your change management procedures.
  • 29. Use test=True You can have SaltStack perform a dry-run state change by appending test=True on the end of the state run command. salt '*' state.highstate test=Truesalt '*' state.highstate test=True By using test=True you can see what is being changed and test your salt state and pillar configurations before executing. This is also a great way of auditing what on your systems needs to be changed to match your salt configuration.
  • 30. Use test=True by default in production In the minion configuration file you can set the default run to always use the test feature. # You can specify that all modules should run in test mode: test: True # You can specify that all modules should run in test mode: test: True This should be used on systems that are “semi-automated” and are incredibly sensitive to changes. If you want to run a highstate on these systems you will need to run a state run with test=False. # salt '*' state.highstate test=False# salt '*' state.highstate test=False
  • 31. Keep things well organized
  • 32. Using more than the base environment In large enterprise environments you will find various applications, some applications will only have a hand full of servers, others have thousands. Keeping track of what gets installed or configured where in these types of environments becomes unreasonable with basic hostname matching. This is especially true if the enterprise hostname convention does not allow for easy identification of server roles.
  • 33. Using additional environments SaltStack lets you define environments for both states and pillars. This allows you to segregate configurations for each application environment.
  • 34. Similar but different configuration files In the example below I show how you can use additional environments to have the same state file defined to deploy a unique hosts file to two different application environments. salt/states/base salt/states/app1-dev/hosts/hosts.file salt/states/app2-dev/hosts/hosts.file salt/states/base salt/states/app1-dev/hosts/hosts.file salt/states/app2-dev/hosts/hosts.file base: '*' - users.operations - screen app1-dev: 'app1*' - hosts app2-dev: 'app2*' - hosts base: '*' - users.operations - screen app1-dev: 'app1*' - hosts app2-dev: 'app2*' - hosts
  • 35. Defining a new environment To define and create new environments simply edit the /etc/salt/master configuration file. file_roots:   base:  - /salt/states/base file_roots:   base:  - /salt/states/base Find:   newenv:  - /salt/states/newenv   newenv:  - /salt/states/newenv Append:
  • 36. Using node groups In addition to custom environments you can also define node groups within SaltStack. Node groups are a grouping of minions that can be grouped by several parameters. When used with additional environments this allows you to perform salt runs on well defined groups of minions.
  • 37. Using node groups in the top.sls file Node groups allows you to define states based on a classification rather than a hostname. app1-dev:   'app1-webservers':     - match: nodegroup     - nginx - php - wordpress app2-dev: ‘app2-webservers’: - match: nodegroup - nginx - uwsgi - django app1-dev:   'app1-webservers':     - match: nodegroup     - nginx - php - wordpress app2-dev: ‘app2-webservers’: - match: nodegroup - nginx - uwsgi - django
  • 38. Using node groups for salt runs You can use node groups to coordinate which servers should perform a highstate or other tasks such as cmd.run # salt -N app1_webservers state.highstate# salt -N app1_webservers state.highstate # salt -N app1_webservers cmd.run “service nginx restart”# salt -N app1_webservers cmd.run “service nginx restart” High State: Command Run:
  • 39. Defining node groups Node Groups are defined in the /etc/salt/master configuration file. This example uses several compound matchers that can be used to define node groups. nodegroups:   app1_webservers: 'L@host001.example.com,host010.example.com or host308.example.com'   app1_appservers: 'G@nodegroup:app1_appservers'   app1_dbservers: 'P@nodegroup:app1_dbservers' app1_servers: 'S@192.168.1.0/24' nodegroups:   app1_webservers: 'L@host001.example.com,host010.example.com or host308.example.com'   app1_appservers: 'G@nodegroup:app1_appservers'   app1_dbservers: 'P@nodegroup:app1_dbservers' app1_servers: 'S@192.168.1.0/24'
  • 40. Architectural Considerations
  • 41. Architectural Considerations While using SaltStack appropriately can greatly increase your consistency and automation which brings with it higher availability. This only works while SaltStack is up and running.
  • 42. Spread the load In large environments it is advisable to setup a dedicated salt master server for different environment types. For example if you have a development, test and 2 production sites it would be advisable to have at least 1 salt master for each environment. Using 2 salt masters for the 2 production sites would allow you to setup a master server for each site giving you independence during data center outages..
  • 43. Setup multiple master servers As of version 0.16 SaltStack minions have the ability to synchronize with multiple masters. This allows you to survive a failure of a single SaltStack master server. master:   - saltmaster01.example.com   - saltmaster02.example.com master:   - saltmaster01.example.com   - saltmaster02.example.com All master servers must have the same: •/etc/salt/pki/master/master.pem •file_roots •pillar_roots
  • 44. Some other stuff
  • 45. Pro tips • Don't become so dependent on SaltStack that you can't perform remediation tasks without it. • Keep your implementation as simple to support as possible. • Keep your implementation well organized to ensure configurations only go where they are meant to go. • Use test=True like your life depends on it, because it might... • Use source control! • If you are going to use SaltStack, use it! Automate or Semi Automate everything. • Keep an eye on new features, SaltStack is evolving fast. • Check out the modules there is some cool stuff in there.
  • 46. EOF Presented by: Benjamin Cane – bencane.com Twitter: @madflojo