Writing SaltStack Modules - OpenWest 2013


Published on

A tutorial for systems administrators and DevOps teams on writing SaltStack modules for infrastructure management and application deployment and configuration management. Presented by Joseph Hall, SaltStack senior engineer, at the OpenWest 2013 conference.

Published in: Technology

Writing SaltStack Modules - OpenWest 2013

  1. 1. WritingSalt ModulesJoseph HallSenior Engineer at SaltStack, Inc.Presented at OpenWest 20131Friday, June 7, 13
  2. 2. The future of technology is dependent upon our ability to effectively utilize infrastructures at any scale.2Friday, June 7, 13
  3. 3. What Salt DoesSalt is a Remote Execution framework, which enables functionality such as:• Configuration Management• Cloud Management• Monitoring• Alerting• Responding to Alerts• etc.3Friday, June 7, 13
  4. 4. What do these things all have incommon?Every component on the previous slide is a function of distributed computing.So why not also use Salt to do the things that we think about, when we think about distributedcomputing?4Friday, June 7, 13
  5. 5. Salt Venn DiagramSaltChefPuppetCFEngine5Friday, June 7, 13
  6. 6. Not Playing the Same GameFlickr photo by kawwsu29 Flickr photo by ant.photos6Friday, June 7, 13
  7. 7. Things that Should Be SaltedFlickr photo by gdgt buzz Flickr photo by Toronto Home TheaterFlickr photo by IntelFreePress Photo by Joseph Hall7Friday, June 7, 13
  8. 8. A Basic Use CaseWriting a simple web crawler• Need to collect various web pages• Use multiple nodes to collect data• Return web content to salt-master for storage8Friday, June 7, 13
  9. 9. Using urllib2 for Our Crawler• Easy to use• Ships with Python• Negates need to install 3rd party dependencies• If a more complex spider is needed, a 3rd party library can be usedo If the 3rd party module is not installed, the module needs to know not to make itselfavailable, via the __virtual__() function9Friday, June 7, 13
  10. 10. Execution Module Basics• The __virtual__() functiono Not technically required, but is a best practiceo Executed when salt-minion is loadedo Identifies the module nameo Commonly used to identify whether a module should be made available• Private functionso Name starts with underscoreo Only used internally by the moduleo Will not show up in sys.doc• Public functionso Called directly by salt commandso Will show up in sys.doc10Friday, June 7, 13
  11. 11. A Few Basics• PEP-8 is your friend• The Salt Style Guide supersedes PEP-8• Always use docstrings• Avoid late imports• Use (proper) loggingo Do not use warn/error when info is correcto Do not use info when debug is correcto Do not add log messages to __virtual__()11Friday, June 7, 13
  12. 12. Using the Salt Logging Systemimport logginglog = logging.getLogger(__name__)log.info(Always useful to users)log.warn(Something isnt right, but I can stillcontinue)log.error(Something is definitely wrong, and shouldbe fixed)log.debug(Only developers care about me)12Friday, June 7, 13
  13. 13. Sample __virtual__() Functiontry: import urllib2 HAS_URLLIB2 = Trueexcept: HAS_URLLIB2 = Falsedef __virtual__(): if HAS_URLLIB2 is True: return crawler return False13Friday, June 7, 13
  14. 14. Sample Private Functiondef _query(url):result = urllib2.urlopen(url)return {url: result.url,code: result.code,msg: result.msg,headers: result.headers.dict,content: result.read(),}14Friday, June 7, 13
  15. 15. Sample Public Functiondef fetch(urls=None):ret = {}if type(urls) is str:ret[urls] = _query(urls)elif type(urls) is list:for url in urls:ret[url] = _query(url)return ret15Friday, June 7, 13
  16. 16. Using Docstringsdef fetch(urls=None):Fetch a URLCLI Example::salt myminion crawler.download http://www.mydomain.com/...SNIP...16Friday, June 7, 13
  17. 17. Using our New Module(command line)Testing/debugging locally:• salt-call --local -l debug crawler.fetch http://tinyurl.com/cnyypkFrom the salt-master:• salt myminion crawler.fetch http://tinyurl.com/cnyypk17Friday, June 7, 13
  18. 18. Ready for more?Lets write a salt runner!18Friday, June 7, 13
  19. 19. Bootstrapping a Runner• manage.py is a good example runner• cp manage.py yourrunnername.py• Make changes to suit your needso status() has the actual client in ito up() and down() use the data to form a report19Friday, June 7, 13
  20. 20. Using the LocalClientimport salt.clientclient = salt.client.LocalClient(__opts__[conf_file],)minions = client.cmd(*,test.ping,timeout=__opts__[timeout],)20Friday, June 7, 13
  21. 21. Returning Outputimport salt.outputdata = some_stuff()salt.output.display_output(data, , __opts__)21Friday, June 7, 13
  22. 22. Which Minions to Use?• Maybe we dont want to use all of our minions for crawling.• Just using the minions that have our crawler installed is sloppy and indeterminate• So lets explicitly define which minions to use, in a configurable location22Friday, June 7, 13
  23. 23. Setting Up a Pillar# cat /srv/pillar/top.slsbase:spider*:- crawler# cat /srv/pillar/crawler.slscrawler.enabled: True23Friday, June 7, 13
  24. 24. Targeting the Pillarminions = client.cmd(crawler.enabled:True,test.ping,timeout=__opts__[timeout],expr_form=pillar,)24Friday, June 7, 13
  25. 25. Fetching From a Miniondef fetch(urls=None):minions = up()client = salt.client.LocalClient(__opts__[conf_file])data = client.cmd(minions[0],crawler.fetch, arg=[urls],timeout=__opts__[timeout])for minion in data.keys():for url in data[minion]:data[minion][url][content] = salt.output.display_output(data, , __opts__)25Friday, June 7, 13
  26. 26. ResourcesSlides Available At:http://tinyurl.com/utos-salt-modules-2013Sources Available At:https://github.com/techhat/salt-crawler/Find me on Twitter:@techhat26Friday, June 7, 13