Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Jeff mc cune sf 2010

3,341 views

Published on

  • Be the first to comment

Jeff mc cune sf 2010

  1. 1. Are we compliant? Auditing Change Management Policies with Splunk and Puppet http://bit.ly/puppetsplunkslides Jeff McCune jeff@puppetlabs.com Monday, October 11, 2010 1
  2. 2. Jeff McCune • Joined Puppet Labs in May, 2010 • Former SA at Netsmart Technologies • Solaris / RedHat Web App Infrastructure • Human Health Information Systems • HIPPA, SAS 70 Type II Compliance Monday, October 11, 2010 2
  3. 3. What’s this all about? • Audits are a fact of life • Systems drift • Puppet Master manifests change • The logs provide no link • Puppet and Git in synchrony with Splunk Monday, October 11, 2010 3
  4. 4. Fun with Regulations • Increased focus on compliance • SAS 70 • HIPPA • IPA • PCI DSS • etc, etc... Monday, October 11, 2010 4
  5. 5. Compliance is Easy Clones Golden VM Monday, October 11, 2010 5
  6. 6. Drifting in and out of Compliance Follow procedures Justify the change Firefighting Inevitable Constant drift Monday, October 11, 2010 6
  7. 7. The Trouble with Time • Are we compliant? • right now? • last week? • last year? • Why weren’t we? • Why is this difficult? Monday, October 11, 2010 7
  8. 8. Advanced Management • We have next-generation tools • Puppet • Git • Subversion • Splunk • Redmine Monday, October 11, 2010 8
  9. 9. Two major issues • Propagation • Time Monday, October 11, 2010 9
  10. 10. Change Propagation Many Nodes a872b46 Larry’s commit Monday, October 11, 2010 10
  11. 11. Time “Why did that one thing happen that one time?” Monday, October 11, 2010 11
  12. 12. Bridge the Gap Events Commits Monday, October 11, 2010 12
  13. 13. The Missing Link • puppetmasterd  -­‐-­‐config-­‐version      /demo/get-­‐config-­‐version-­‐script • [root@puppet  ~]#  /demo/get-­‐config-­‐version   ref="refs/heads/jeff"  commit="b585f7fe" • Jeff’s processor, --reports=logversion Should ship with puppet “soon” Monday, October 11, 2010 13
  14. 14. get-config-version #!  /bin/bash set  -­‐u set  -­‐e cd  /demo/puppet-­‐demotools ref="$(git  symbolic-­‐ref  HEAD)" if  [[  -­‐f  .git/"${ref}"  ]];  then        commit="$(cat  .git/${ref})" else        commit="UNKNOWN" fi echo  "ref="${ref}"  commit="${commit}"" Monday, October 11, 2010 14
  15. 15. logversion.rb #  Create  logversion.rb  by  copying  log.rb def  process    self.logs.each  do  |log|        saved_message  =  "#{log.message}"        log.message  <<  "  "  <<  log.version        Puppet::Util::Log.newmessage(log)      log.message  =  saved_message    end end Monday, October 11, 2010 15
  16. 16. Untagged Events Monday, October 11, 2010 16
  17. 17. Tagged Events Monday, October 11, 2010 17
  18. 18. Who to blame? Blame this guy The commit proves it Monday, October 11, 2010 18
  19. 19. Putting it all together • Demo time! Monday, October 11, 2010 19
  20. 20. Steps to Reproduce • Fork and clone puppet-­‐demotools on github • logversion.rb goes into /usr/lib/ruby/site_ruby/1.8/puppet/reports • --config_version /path/to/your/script • --reports=logversion,store • Make sure syslog catches daemon.* and splunk is indexing syslog • Note: syslog outputs are off with -­‐-­‐verbose Monday, October 11, 2010 20
  21. 21. Future Work • Commit hooks into puppetmasterd activation and the ticketing system • Splunk URLs to redmine, trac, salesforce... • Closed loop from business case to system modification by puppet. Monday, October 11, 2010 21
  22. 22. Questions? • Google Moderator • http://bit.ly/arewecompliant? • http://bit.ly/puppetsplunkslides • Twitter: 0xEFF • Email: jeff@puppetlabs.com Monday, October 11, 2010 22

×