Splunk SF Bay Area Usergroup meeting December 8, 2021: Splunking config files, presentation by Daniel Wilson

1. Copyright © 2011, Splunk Inc. Listen to your data.
11/4/2021
Daniel Wilson
Senior Security Engineer
Splunk your Configs to
Improve Security Posture

2. Copyright © 2011, Splunk Inc. Listen to your data.
Agenda
2
• Introductions
• What is a Config file?
• Preparing Splunk
• Use Cases
• Gotcha’s
• Q&A

3. Copyright © 2011, Splunk Inc. Listen to your data.
Summary
3
A quick security talk to discuss how and why you would want to
index your config files.

4. Copyright © 2011, Splunk Inc. Listen to your data.
Introductions – Daniel Wilson
4
• Security? IT guy? Hey you?
• Security more or less 8 years now
• Selling computers in ’97
• Speaks randomly Splunk User Group
• Formal Career and Skills development coach at StubHub/eBay
through Leader a Coach program and Jr talent development
Splunk Blogged About these features in 2007, that’s where I learned these tricks
https://www.splunk.com/en_us/blog/tips-and-tricks/dont-forget-to-index-your-config-files.html

5. Copyright © 2011, Splunk Inc. Listen to your data.
Introductions – Establish Credibility
5
40 Certs over the years….no idea what is expired
• Splunk Arch level 2, Splunk Admin, Splunk Power User
• AWS Security Specialist, MCSE Security, CCNA Sec, Security+, CySA+
• RHCSA, Cloud+, Linux+, CCNP Routing/Switching
• MTA Software Development, MTA Python
Other Stuffz
• Active defense, Cloud Security, Network/Systems Security and
Automation
• CIS and MITRE
• GDPR, PCI and SOX
• DevOps Culture

6. Copyright © 2011, Splunk Inc. Listen to your data.
Introduction Audience
6
• Who’s in the audience?
– Splunk Admins
– Security Folk
– Auditors
– ComplianceAuditors, Compliance

7. Copyright © 2011, Splunk Inc. Listen to your data.
What is a Config File
7
• Flat file generally containing key values
• Read by apps when they start/stop
• Often contain critical settings
• Example of a SSHD Config
• “ini” files on Windows

8. Copyright © 2011, Splunk Inc. Listen to your data.
Use Cases – Why Splunk?
8
• Monitoring your configs critical part of your File Integrity
Monitoring Strategy (FIM)
• Tools like AIDE tell you something changed
• Auditd, tells you who changed and when
• Both AIDE and Auditd lack content
• GIT managed Configs are great… security professionals have been
burned with lack of enforcement though.

9. Copyright © 2011, Splunk Inc. Listen to your data.
Use Case – Looking at configs!
9
• Make easy to use dashboard for auditors and non-technical users
• Tip: Rmcomments macro included to ease reading

10. Copyright © 2011, Splunk Inc. Listen to your data.
Use Case – Comparing Config
10
• Compare files manually
• Enrich your alerts with just the details
index=configs source=/etc/ssh/sshd_config
| head 2
| diff pos1=1 pos2=2

11. Copyright © 2011, Splunk Inc. Listen to your data.
Use Case – Alert on Login Script Changes
11
• Actual control I implemented that later caught our internal
RedTeam after getting p0wned.
• Add a input for all your login scripts for your platform
• Run this job every 15 minutes
index=configs source="/home/*/.bash*"

12. Copyright © 2011, Splunk Inc. Listen to your data.
Use Cases – Clear Text in Database
12
• Example of detecting of clear text passwords in PostGres
• Note inline search extractions, will not extract by default
• Tip: Add CIM fields like App to your results to improve
searches
index=configs sourcetype=config_file source=*pg_hba* " password"
| dedup host, source
| rex field=_raw "host.*(?<insecure>password)"
| eval message = "Clear text passwords accepted by PostGres"
| eval app = "Postgres"
| table host, source, _raw, app

13. Copyright © 2011, Splunk Inc. Listen to your data.
Use Case – SSHD Empty Passwords?
13
• Great Compliance Search right here, tweakable
• Not you might want to script the input in here
• Tip: Enrich your alerts with MITRE details
index=configs sourcetype=config_file source=/etc/ssh/sshd_config
earliest=-48h latest=now
| dedup index, sourcetype, host, source
| rex mode=sed "s/#PasswordAuthentication yes//g"
| search "PermitEmptyPasswords yes" OR "PermitEmptyPasswords Yes"
| eval vrisk_score = 100, domain="Endpoint", dest=host, dest_dns=host
| eval reason="Endpoint - SSH PermitEmptyPasswords yes set"
| eval MITRE="T1110"
| eval _time = now()
| table dest, vrisk_score, domain, reason, MITRE

14. Copyright © 2011, Splunk Inc. Listen to your data.
Use Case – Config Drift
14
• By using md5 function we can see the drift
• Consider sorting by your data gardens for compliance reports

15. Copyright © 2011, Splunk Inc. Listen to your data.
Splunk Admin
15
• Splunk_TA_nix does all this
• Props
• Inputs
• Indexes
• I put all this in an app called TA-configsdemo on Splunkbase to help
you play with these settings without dealing with Splunk_TA_nix

16. Copyright © 2011, Splunk Inc. Listen to your data.
Splunk – Props.conf
16
• Rather than creating events, Splunk create one event per file
• You can and should review your settings with btool
• I noticed 4 settings in props.conf that are worthy conversations
• Btool on your indexer and search head
AUTO_KV_JSON = true
CHECK_METHOD = modtime
DATETIME_CONFIG = NONE
KV_MODE = none
$ ./splunk cmd btool props list config_file

17. Copyright © 2011, Splunk Inc. Listen to your data.
Props - AUTO_KV_JSON
17
AUTO_KV_JSON = <boolean>
* Used for search-time field extractions only.
* Specifies whether to try json extraction automatically.
* Default: true
• Meaning if your file is well structured JSON you will get field
extraction by default.
• I used another sourcetype for this

18. Copyright © 2011, Splunk Inc. Listen to your data.
Props - CHECK_METHOD = modtime
18
File checksum configuration
* Set CHECK_METHOD to "endpoint_md5" to have Splunk software perform a checksum of the
first and last 256 bytes of a file. When it finds matches, Splunk software lists the file as already
indexed and indexes only new data, or ignores it if there is no new data.
* Set CHECK_METHOD to "modtime" to check only the modification time of the file.
• Super helpful on config files that are really small and don’t have enough characters to be
checked with the first and last 256. Avoid the “too small problem” in Splunk sourcetypes.

19. Copyright © 2011, Splunk Inc. Listen to your data.
Props - DATETIME_CONFIG = NONE
19
"NONE" leaves the event time set to whatever time was selected by the input layer
* For data sent by Splunk forwarders over the Splunk-to-Splunk protocol, the input layer is the
time that was selected on the forwarder by its input behavior (as below).
* For file-based inputs (monitor, batch) the time chosen is the modification timestamp on the file
being read.
* For other inputs, the time chosen is the current system time when the event is read from the
pipe/socket/etc.
* Both "CURRENT" and "NONE" explicitly disable the per-text timestamp identification, so the
default event boun
• In this case a config_file time stamped by your operating system like in centOS might be dated
6-7 years ago. You need to consider this in your indexer retention strategy.

20. Copyright © 2011, Splunk Inc. Listen to your data.
Props - KV_MODE = none
20
* none: if you want no field/value extraction to take place.
• You will NOT get field extractions by default from your Config files
• While a lot of your config_files are going to be key value they are
going to be large and this is going to be expensive to turn on.

21. Copyright © 2011, Splunk Inc. Listen to your data.
inputs.conf
21
[monitor:///etc/ssh/sshd_conf*]
index=configs
sourcetype=config_file
If you have Splunk_TA_Nix installed or configured your props.conf as
we mentioned the source will work.

22. Copyright © 2011, Splunk Inc. Listen to your data.
inputs.conf, cont
22
• Trick to cat a file in
• Time will be NOW
• Saved our auditor days
• MD5 identical
• Make sure the file is there!
…
do-execcat() {
# display config
if [ -f "$strConfigLocation" ]; then
cat $strConfigLocation
fi
}
…
[script://./bin/cat_sshd_config.sh]
index=osnixvcustom
sourcetype=config_file
source=/etc/ssh/sshd_config
interval=86400

23. Copyright © 2011, Splunk Inc. Listen to your data.
Indexer Stuff
23
• Very low sourcetype uniformity
• Don’t recommend you mix it with other types for this reason
• Ideally not a default index you would search either due to a large
set of characters and strings vs a traditional log

24. Copyright © 2011, Splunk Inc. Listen to your data.
Gotcha’s
24
• Config files are cheap
• Watch out for shared file systems
• Ensure your index permissions are well managed
• Don’t index any secrets you don’t want collected

25. Copyright © 2011, Splunk Inc. Listen to your data.
Thank You :)