3. 33
athenahealth
Founded in 1997, provides cloud based services such as network-
enabled EHR, practice management and population health services
Connecting more than 72,000 providers and health systems nationwide
4,000+ employees
We were voted Forbes “Most Innovative Growth Company” and a
Deloitte “Fast 500 Company” in 2014 and have earned numerous
employer awards
Three InfoSec Towers
Risk, SIRT (Security Incident Response Team), and ITSec
4. 44
My Background and Role
Previous work experience includes sysadmin and network engineer
Manage the ITSec and Access Control teams at athenahealth
– Team of five helps run various security-related tools including server anti-virus,
web proxy, IDS, DLP, and e-mail sandboxing solutions
At company for just over a year now; I personally never used Splunk at my
other companies
athenahealth has been Splunk customer for 3 years
Favorite Splunk tagline: “Get drunk with Splunk”
Fun Fact: In college, I accidentally wrote an uncontrolled forking process
for a homework assignment that took down almost all of the Unix servers.
5. 55
What Was The Need?
InfoSec had many sources of information:
• When something is flagged, how do we prioritize and take appropriate action?
• Strategic approach where we need to understand:
• What is alerting?
• Why is it alerting?
• Is it a high value or high risk target?
• All of the necessary information to take action
• Easily correlate information from multiple tools in a “single pane of glass”.
6. 66
Enter Splunk
• Easily consume data from various sources (syslog, text files, etc.)
• Splunk Enterprise Security (ES) for the SIRT
• Crafted alerts and reporting to look for high value targets
• If we see a bad pattern within our network, we can quickly alert and take action
• We can tweak and tailor alerts and reports over time
• Official Splunk and 3rd party apps
• Our teams use Splunk every day
7. 77
Splunk At athenahealth
Over ten “power users”
Anti-malware, anti-virus, system data, system
logs, VPN/firewall/router logs, various other
unstructured data
400GB/day license
Goal: Retain two years of searchable data
Windows logs are the most verbose
7
100s of Forwarders
8 Indexers
3 Search Heads + 1 Deployment Server
WinEventLog:Security
5x any other sources
8. 88
Value to the Organization
• Immediate visibility
• Virtually any data, even mainframe and other legacy infrastructure
• Less “alert fatigue” via very granular control
• Ability to dig in and investigate, correlate (it’s not a proprietary black hole)
Alerts must be clear and actionable or they’re a waste!
https://www.pagerduty.com/blog/lets-talk-about-alert-fatigue/
• Better team efficiency - Reduce confusion and wasted time over where to look
for information
• Intuitive UI
• Distributed design allows for HA and a mix of Windows and Linux services
9. 99
Top Security Takeaways
Think about where all your disparate pieces of security information live
How do you handle unstructured data?
Control over how to consume data and alert on it
Some of the reasons we like Splunk:
– Intuitive user experience
– Flexibility and ability to bring in unstructured data
– Granular security within search apps
– Fast searches – Sub-minute versus couple of hours (use SSDs!)
– 3rd party app support
– Vendor and VAR support and knowledge base
11. 1111
Be Visible & Valuable
Champion the technology - Be a helping hand to other groups
Keep an eye on the infrastructure and offer help and feedback to groups
Have strong documentation that’s easy to find and search
Don’t make it painful to get access, grant it automatically
Conduct regular meetings with business users
– General user overview
– Specialized meetings for administrators, developers, etc.
– Lunch & Learn Sessions for informal training (food always helps!)
Vendor vetting process
– How logs are consumable (both on-prem and cloud)?
12. 1212
Example: Ease Of Use
Step 1: Start somewhere
Step 3: Find what you’re really looking for
Step 5: Setup an alert
Step 2: Tweak your search
Step 4: Finalize your search
172.16.2.3
Saved Alert Search Tips:
• Avoid NOT (Computationally expensive)
• Be specific (Exact text, hostnames, etc.)
• Think and plan for the unexpected
• Provide all of the necessary info to be actionable
13. 1313
Get People Hooked
Linux server logs
Windows server and domain controller logs (including account lockouts)
Virtual Server Infrastructure (ESXi, OpenStack, etc.)
DHCP and DNS logs
SSO logs (PingFed, Okta, Azure, etc.)
In-house developed application logs, SFTP server logs
VPN, firewall, and router logs
Two-factor, web proxy, and MDM logs
Endpoint logs (anti-virus, anti-malware, Bit9, Carbon Black, etc.)
14. 1414
Be Careful Of Your Own Success
Remind groups what Splunk is and is not to be used for
Be the gatekeeper. Keep it clean. Use permissions within the app.
Documentation, documentation, documentation
Use alerting to warn you of high amounts of logs and proactively get
ahead of it before you go over your daily license limit
Run the forwarders so you have the final say
Ask for other groups to help chip in towards more licenses
On prem vs. hosted Splunk...which one is better for you? Can you grow?
15. 1515
Where To Next?
Designed to scale
– Just got asked “Where can we store six years’ worth of log data?”
Emphasis on a ‘devops’ mentality across the org
– OpenStack and AWS
Improve the risk data around assets
Continue to develop our threat feed data
