This document discusses Jeff Campbell's role as the Information Security Architect at Penn State Hershey Medical Center and their use of Splunk. It describes how Penn State Hershey Medical Center has over 9,000 employees and a combined $1.5 billion budget across its institutes and hospitals. It outlines some of the challenges they faced with decentralized logging prior to Splunk, and how Splunk provided a centralized log repository allowing for faster searching and correlation across systems. It provides examples of how Penn State Hershey is using Splunk for security use cases, operational improvements, and additional sources. It also discusses their Splunk architecture and future plans to expand Splunk usage.

1. Copyright © 2013 Splunk Inc.
Penn State Hershey
Medical Center
Jeff Campbell
Information Security Architect

2. My Background and Role
Jeff Campbell, CISSP+ISSAP

Penn State Hershey Medical Center since 1997

Security practitioner for about 9 years

Splunk user since 2010
–
Architecture, Administration, Knowledge Expert
–
Building advanced searches, summary indexing, alerting, reporting
–
Splunk SME

4. 5 major institutes
24 academic departments
4 hospitals & 19 hospital affiliates
58 ambulatory care practices
> 9,000 employees
Combined $1.5 billion budget
Penn State Hershey Children's Hospital has been ranked in five specialties in U.S. News & World
Report's 2013-14 Best Children's Hospitals rankings.

5. Before Splunk
•
Most logs decentralized (firewall logs in SIEM system)
•
•
•
•
•
•
Limited retention
Searching over extended time ranges very slow
Lots of silos  limited visibility between systems
It could take hours to find a needle in the haystack
Manual correlation between different log files
Poor results
We needed a faster way to get answers

6. Choosing Flexibility and Fast Time to Value
Requirements
•Central repository
•Quickly search for needed data
•Correlate & report across systems
>
•
•
•
•
Splunk
Flexible framework
Fast time to value
Centralized real time data
Google for IT.. but more!

7. Splunk Use Cases at Penn State Hershey
Started as a security tool (firewall, Active Directory)
– Satisfy legal requests for data
Immediate “operational” improvements
– Account lockouts (major win)
– Associate users with Ips
– Performance monitoring
Added additional sources
– Dhcp, AV, network auth, Exchange, network infrastructure…

8. Splunk Architecture
splunk1
~70GB per day from 140 forwarders (~20%)
AD Security
Exchange
Windows
Linux
FTP
splunk2
Servers running dedicated lightweight Splunk
agents gather local system log data and
forward to one of the four Splunk indexers,
alternating on a predetermined time interval
splunk3
splunk4
Rsyslog
... and growing
Expand coverage
VMware
Citrix
rsyslog1
splunkutil
rsyslog2
Systems without Splunk Agents
may send syslog data. A Cisco
SLB IP address will load
balance between two
redundant Syslog servers. The
Syslog servers will run Splunk
agents and forward to the
indexers.
host
Firewalls
ACS
DHCP
Network infrastructure
Smtp
Web proxies
Some data useful to Splunk will
not come from log files which
can be monitored. The
SplunkUtil server runs scripts
to "scrape" data from various
data sources (databases, file
systems, directory services,
web services, etc) and forwards
the data onto Splunk indexers.

9. Splunk Applications
Windows
MS Exchange
Vulnerability Data

10. Vulnerability Overview

11. Return on Investment
Labor cost savings from Splunk easily offset the
purchase price

12. Future Plans
“Splunk for Everything” (infrastructure)
– VMware & Citrix end-to-end performance
Executive dashboards & reports
Key applications (financials, trouble ticketing, asset management, etc.)
– Better data aggregation
– Expand Splunk user base
Clinical Systems? (HIPAA/HITECH compliance, “meaningful use”)
12

13. Best Practices and Lessons Learned
Plan data intake well; Know what logs look like before they come in
Don’t be afraid to play; Fully explore the product & its capabilities
RTFM (try to Google “Splunk propsconf”)
If you’re stuck, go to Splunk Answers – great community there
People in your organization will see the value – help them find something in their data

14. Thank You