SaltConf14 - Saurabh Surana, HP Cloud - Automating operations and support with SaltStack


Published on

Using SaltStack to automate enterprise IT operations and support capabilities is not as well documented as the more traditional SaltStack use cases. This session will show how the HP Cloud team runs a secure and reliable SaltStack automation environment by writing Salt states and modules to simplify day-to-day operations and support while extending SaltStack capabilities through dynamic states and modules. The talk will also show how to protect sensitive information and safe guard against user errors.

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

SaltConf14 - Saurabh Surana, HP Cloud - Automating operations and support with SaltStack

  1. 1. Automating Operations and Support with Salt Stack Saurabh Surana
  2. 2. Agenda • OpenStack Trove – Quick Intro • Salt Environment Overview • Automation Overview • What changes for operations with salt stack? • Salt Modules • Salt States • Challenges we are facing
  3. 3. OpenStack - Trove • OpenStack’s Database as a Service Project • REST API for creating managed database instances • Provides mysql support out of the box • Actively being worked on by HP, Rackspace, Mirantis….
  4. 4. OpenStack – Trove TROVE API TROVE TASKMANAGER TROVE CONDUCTOR DBRabbitMQ MySQL Instance MySQL Instance OpenStackNova User1User2
  5. 5. Salt Environment Overview • Separate master setup per deployment • Multi master setup • Manage salt-master just like any other minion
  6. 6. Salt Environment Overview • States and Pillar in separate repository • States are common across all the deployments (Prod Env) • Pillar is maintained per environment
  7. 7. Salt Environment Overview • Every database instance that gets created is a minion • Automated (but secured) way of accepting and deleting those keys
  8. 8. Things to consider for Operations with Salt Stack
  9. 9. How is it different with Salt Stack? • Access to salt master is needed • Access to sensitive data (Pillar) • Access to infrastructure • Auditing individual user actions • Auditing Salt Command execution
  10. 10. Access to Sensitive data • Correct ACLs to protect the pillar data • Consider encrypting your pillar data • Protecting Salt master configurations and minion keys
  11. 11. Access to infrastructure • Since user is on salt master, he has access to every node in the infrastructure • That too, as root, salt minion is running as root • Protecting what type of actions can be performed by individual users, by controlling access to modules
  12. 12. Some modules worth blocking • cmd.* • file.* • cp.* • config.* • pillar.* • …… • (There are very few which are to be considered safe)
  13. 13. Auditing individual user actions • Make sure salt commands are executed from their own user account, this can be achieved with the help of correct configurations • Salt master logs every user action • user name, • salt command • salt minion
  14. 14. Auditing Salt Command Execution • Every salt command that was executed, get recorded in salt job management framework • Can query the details of a particular job? • Consider using a returner, simplify serach
  15. 15. Controlling salt master through salt • Controlling salt master like any other salt minion using salt framework • We do this by installing a minion on master • This allows use to treat the master as any other minion in the system, so we can write states, pillar and modules for it
  16. 16. Salt States
  17. 17. Salt States • Separating states to simplify day to day operations • Writing states to catch and possiblify mitigate user errors
  18. 18. Separating states • One big state files seems simple and state forward • Try separating those states into simple group of actions • This will allow you to run those specific actions when they are needed and will not affect anything else • This will be very good addition for operations
  19. 19. Separating states • For example separating following from each other: – Package Installation – User creation – Configuration parameter management – System level service/package/upgrade management • Note: Where ever required you need to maintain necessary dependencies even after separation
  20. 20. Separating states • If package installation is separate, at the time of upgrading packages, users will be unaffected • If user management is separate state file then, at the time of password rotation, nothing for packages or system level services will be affected
  21. 21. Writing states to catch user errors • Writing state with necessary checks to help catch user errors • E.g while writing states for salt-master, make sure that its getting executed on the master only. May be add a check on grains[id]
  22. 22. Salt Modules • Extending salt modules • Writing modules to catch user errors
  23. 23. Extending Salt Modules • Consider extending salt modules: • If existing salt module is possible to cause harms if misused • If for certain action user needs to run multiple salt commands • If completing an action, user will need values from pillar data
  24. 24. Thanks • Demo Scripts/files @