This document discusses using Puppet and Splunk to audit change management policies and ensure compliance. It outlines how Puppet manages system configurations but the logs provide no link between configuration changes and commits. The document proposes enhancing Puppet reports to tag events with the Git commit, allowing Splunk to search for the commit that caused a specific event. This would help determine if systems were compliant by tying configuration changes to commits and answering questions about what happened and why.

1. Are we compliant?
Auditing Change Management Policies
with Splunk and Puppet
http://bit.ly/puppetsplunkslides
Jeff McCune
jeff@puppetlabs.com
Monday, October 11, 2010 1

2. Jeff McCune
• Joined Puppet Labs in May, 2010
• Former SA at Netsmart Technologies
• Solaris / RedHat Web App Infrastructure
• Human Health Information Systems
• HIPPA, SAS 70 Type II Compliance
Monday, October 11, 2010 2

3. What’s this all about?
• Audits are a fact of life
• Systems drift
• Puppet Master manifests change
• The logs provide no link
• Puppet and Git in synchrony with Splunk
Monday, October 11, 2010 3

4. Fun with Regulations
• Increased focus on compliance
• SAS 70
• HIPPA
• IPA
• PCI DSS
• etc, etc...
Monday, October 11, 2010 4

5. Compliance is Easy
Clones
Golden VM
Monday, October 11, 2010 5

6. Drifting in and out of
Compliance
Follow procedures
Justify the change
Firefighting
Inevitable
Constant drift
Monday, October 11, 2010 6

7. The Trouble with Time
• Are we compliant?
• right now?
• last week?
• last year?
• Why weren’t we?
• Why is this difficult?
Monday, October 11, 2010 7

8. Advanced Management
• We have next-generation tools
• Puppet
• Git
• Subversion
• Splunk
• Redmine
Monday, October 11, 2010 8

9. Two major issues
• Propagation
• Time
Monday, October 11, 2010 9

10. Change Propagation
Many
Nodes
a872b46
Larry’s commit
Monday, October 11, 2010 10

11. Time
“Why did that one thing happen that one time?”
Monday, October 11, 2010 11

12. Bridge the Gap
Events Commits
Monday, October 11, 2010 12

13. The Missing Link
• puppetmasterd -­‐-­‐config-­‐version
/demo/get-­‐config-­‐version-­‐script
• [root@puppet ~]# /demo/get-­‐config-­‐version
ref="refs/heads/jeff" commit="b585f7fe"
• Jeff’s processor, --reports=logversion
Should ship with puppet “soon”
Monday, October 11, 2010 13

14. get-config-version
#! /bin/bash
set -­‐u
set -­‐e
cd /demo/puppet-­‐demotools
ref="$(git symbolic-­‐ref HEAD)"
if [[ -­‐f .git/"${ref}" ]]; then
commit="$(cat .git/${ref})"
else
commit="UNKNOWN"
fi
echo "ref="${ref}" commit="${commit}""
Monday, October 11, 2010 14

15. logversion.rb
# Create logversion.rb by copying log.rb
def process
self.logs.each do |log|
saved_message = "#{log.message}"
log.message << " " << log.version
Puppet::Util::Log.newmessage(log)
log.message = saved_message
end
end
Monday, October 11, 2010 15

16. Untagged Events
Monday, October 11, 2010 16

17. Tagged Events
Monday, October 11, 2010 17

18. Who to blame?
Blame this guy The commit proves it
Monday, October 11, 2010 18

19. Putting it all together
• Demo time!
Monday, October 11, 2010 19

20. Steps to Reproduce
• Fork and clone puppet-­‐demotools on github
• logversion.rb goes into
/usr/lib/ruby/site_ruby/1.8/puppet/reports
• --config_version /path/to/your/script
• --reports=logversion,store
• Make sure syslog catches daemon.* and splunk is
indexing syslog
• Note: syslog outputs are off with -­‐-­‐verbose
Monday, October 11, 2010 20

21. Future Work
• Commit hooks into puppetmasterd
activation and the ticketing system
• Splunk URLs to redmine, trac, salesforce...
• Closed loop from business case to system
modification by puppet.
Monday, October 11, 2010 21

22. Questions?
• Google Moderator
• http://bit.ly/arewecompliant?
• http://bit.ly/puppetsplunkslides
• Twitter: 0xEFF
• Email: jeff@puppetlabs.com
Monday, October 11, 2010 22