George Starcher presented on how the University of Alabama at Birmingham uses Splunk to analyze log data and secure the university. They have increased their log data ingestion from 150GB to 175GB per day. They developed a Common Information Model to standardize log data and simplify searches. They automated security responses by using Splunk searches to trigger scripts that control their intrusion prevention system to block abusive IP addresses in real time. This recovered an hour of daily work and improved security. Starcher discussed future plans to further automate systems and integrate additional external intelligence sources into Splunk.
3. About Me
George Starcher, Enterprise Information Security Engineer II,
CISSP
Splunk Certified Knowledge Manager and Splunk
Certified Administrator
Splunk IRC Channel; Birmingham, AL - Splunk User Group
Log all the things!
RaspberryPi + Splunk = Optimal Laundry Time
Yes, there is a Splunk Universal Forwarder now!
www.georgestarcher.com
3
4. One Year Ago
License Usage was 150GB/day
Ingesting normal log types
Base parsing of fields
We saw huge increase on speed for investigating issues
The honeymoon period with our data
5. Now
License Usage averaging 175GB/day
Added a lot of log metadata and simplifying searches
Common Information Model
Starting to add external Intelligence Sources
We were already doing geo lookups
Keeping the magic in the relationship
Automating Splunk control over other systems
6. Securing the University
6
Before:
• Lots of “typical” log mining
• Not as vibrant integration to ES App as wanted
• Manual Daily Operations Processes
After:
• Searches easier to understand and resilient to
new log sources
• ES App much better populated
• Alert Script Control of Other Systems
10. @SplunkDev Team - THANKS!!
@gblock - Glenn Block
@damiendallimore -Damien Dallimore
David Noble - Twitter App
11. Alert Scripts - IPS Control
Had manual process for blocking abusive scanners: SSH, RDP,
VNC, etc
– Consumed 30-45 minutes per day
– Permanent blacklist entries
Moved to automated process
– Scheduled Splunk Searches driven by any log source
– Greatly reduced time and static blacklist maintenance
– Plugged in Web Services (REST) calls to the IPS
12. Alert Scripts - How it Works
Intrusion Prevention ApplianceIntrusion Prevention Appliance
21. Phishing
Started Feb 10, 2014
• Blocked for any access from Nigeria every 5 minutes
Expanded Multi-Country Feb 15, 2014
• Blocked for combination from certain countries and a lookup
table of hosted providers
Feb 17, 2014
• Noticed unexpected Exchange from Nigeria
23. Splunk from Tool to Team Member
We recovered an hour of daily operations labor per day by automating
existing processes and some regular intelligence reports.
The automation provides the ability of our IPS to respond to data it could
never handle directly. Combining the automated response with different
quarantine policies in the IPS we change the ground under the attacker’s
feet.
Simplifying searches based on Common Information Model helps with
cross training staff and integration of new log sources.
24. What is Next
Update to Splunk v6
Update to Splunk App for
Enterprise Security
Application v. 3.0
Add automation to more of
our systems
Add Data exchange from/to
Intelligence sharing systems