This document provides an agenda for an Enterprise Security hands-on guided tour using Splunk software. The tour will demonstrate the Splunk App for Enterprise Security and cover topics including data ingestion, the common information model, risk analysis, threat intelligence, incident response exercises, and correlation searches. It encourages participants to bring a laptop and notes several break periods for providing feedback via text message or online survey for a chance to win gift cards.
3. 3
Agenda
What is the Splunk App for Enterprise Security?
Guided Tour
– General Overview
– Data Ingest and Common Information Model
– Risk and Threat Intel
– Incident Response Exercise
– Creating a Correlation Search
Wrap Up
– Two surveys, two gift cards!
5. Machine data contains a definitive record
of all interactions
Splunk is a very effective platform to collect,
store, and analyze all of that data
Human Machine
Machine Machine
6. Platform for Operational Intelligence
The Splunk Portfolio
Rich Ecosystem of
Apps & Add-Ons
Splunk Premium
Solutions
Mainframe
Data
Relational
Databases
MobileForwarders Syslog/TCP
IoT
Devices
Network
Wire Data
Hadoop
7. 7
Rapid Ascent in the Gartner SIEM Magic Quadrant*
*Gartner, Inc., SIEM Magic Quadrant 2011-2015. Gartner does not endorse any vendor, product or
service depicted in its research publication and not advise technology users to select only those
vendors with the highest ratings or other designation. Gartner research publications consist of the
opinions of Gartner’s research organization and should not be construed as statements of fact.
Gartner disclaims all warranties, express or implied, with respect to this research, including any
warranties of merchantability or fitness for a particular purpose.
2015 Leader and the only vendor to
improve its visionary position
2014 Leader
2013 Leader
2012 Challenger
2011 Niche Player
2015
8. 8
ES Fast Facts
● Current version: 4.0
● Two releases per year
● Content comes from industry experts, market analysis, but most
importantly YOU
● The best of Splunk carries through to ES – flexible, scalable, fast,
and customizable
● ES has its own development team, dedicated support, services
practice, and training courses
18. 18
Data Ingest + Common Information Model
18
● You’ve got a ton of systems
● How to bring in:
● Network AV
● Windows + OS X AV
● PCI-zone Linux AV
● Network Sandboxing
● APT Protection
● Splunk + CIM is Easy
19. 19 19
Click Add Data, under Settings
Settings, from any page in
Splunk
20. 20 20
Bringing Data into Splunk
is easy!
Data Normalized to Common
Information Model
Under Settings (upper right
side), Add Data
Click the Cisco app
icon
23. 23
Click Malware Attacks to PivotClick
From Search Nav Menu,
select Pivot then Malware
Nested Models – easily distinguish
subsets of data
24. 24
Filter Timeframe to Last 60 Minutes
Change
Total count of attacks
Change to Area Chart to show
Attacks over Time
Click
From Search Nav Menu, select Pivot,
then Malware, then Malware Attacks
25. 25
The time range we selected
Split out by Vendor with “Add
Color”
Click
SCROLL to
vendor_product
CIM has many
usable attributes
26. 26
For as many vendors as you have,
pivot and report across any field!
27. 27
How Does This Apply?
Let’s Open the Malware Center to See
Under Security Domains, under Endpoint,
open Malware Center
28. 28
Various ways to filter data
Malware-Specific KSIs and Reports
Security Domains ->
Endpoint -> Malware Center
29. 29
Searches that rely on
this data model
How Complete is my ES?
What else could I onboard?
Instructor Only
37. 37 37Attack Map
The Challenge:
• Industry says Threat Intel is
key to APT Protection
• Management wants all
threat intel checked against
every system, constantly
• Don’t forget to keep your
15+ threat feeds updated
The Solution:
46. 46
Auditors / Management / Compliance Says…
46
● Can you show me <Typical Report>?
● Reporting is easy in Splunk
● But we have more than
300 standard reports too
50. 50
What is Zeus?
• Zeus, ZeuS, or Zbot is arguably the deadliest and
longest persisting financial malware
• Steals banking information by man-in-the-browser
keystroke logging (Banks, Credit Cards, Paypal, etc.)
• Financial Damage: hundreds of millions of dollars
• $3M FBI bounty on 32 yr-old creator
Zeus Malware: Threat Banking Industry, White Paper by Unisys 2010
51. 51
Incident Response Scenario
51
http (web) session to
command & control
server
Remote control
Steal data
Persist in company
Rent as botnet
WEB
.pdf
.pdf executes & unpacks malware
overwriting and running “allowed” programs
Svchost.exeCalc.exe
Attacker creates
malware, embed in .pdf,
emails
to the target
MAIL
Read email, open attachment
52. 52 52
Go to Incident ReviewClick
Sort by UrgencyClick
Find Event with Your Persona
Finally, click the adjacent “>”
Status of All Tickets
Filter on owner, urgency,
status, tag, and more
Explore and Analyze Incidents
Incident Review
53. 53
View Raw Event
Data from asset framework
Incident Review, expand
incident with your persona
54. 54
Drill down on “115.29.46.99”
and select Domain Dossier
Click
Click
Pivot off of everything. Go
internal or external.
Customize.
Incident Review, expand
incident with your persona
55. 55
Oh look! China!
Incident Review Tab is still open.
Click back to it Incident Review
In your Incident, hit Drop
Down next to Destination
and then “Domain Dossier”
56. 56
Drill down on
“115.29.46.99” and select
“Web Search as destination”
Click
Click
Incident Review, expand
incident with your persona
57. 57
Only one internal address,
that’s good…
Change to 24 hours
if needed
Click back to
Incident Review
In your Incident, hit Drop Down
next to Destination and then “Web
Search (as Destination)”
58. 58
Drill down on the Source field
(192.168.56.102) and select
“Asset Investigator”
Click
Click
Incident Review, expand
incident with your persona
59. 59
Data from asset framework
Configurable Swimlanes
Darker=more events
All happened around same timeChange to
“Today” if needed
Asset Investigator, enter
“192.168.56.102”
60. 60
Change to
“Today” if needed
Select “Exec File
Activity” vertical bar
Asset Investigator, enter
“192.168.56.102”
61. 61
“calc.exe” running out of the
user profile? Hmmm….
Drill into the
raw events
Asset Investigator, enter
“192.168.56.102”
Analysts may wish to share
with each other. Collaboration!
62. 62
Raw events from Carbon
Black
Splunk automatic field
extraction
Click the “>” next to oldest suspicious
event, calc.exe, to see field mapping
From Asset Investigator, Click
“Exec File” and then click
“Open in Search” icon
63. 63
What else do we know
about this unique process?
Open in New Search
64. 64
Open Event Actions
Open Custom Email
Threat Investigator
Dashboard
• A weaponized PDF was sent
• “calc.exe” was dropped
• Activity with “known bad” IP
70. 70
Fill out Status: Pending. Urgency: Low.
Owner: <your persona>. Comment:
<whatever you want>.
Populate
Click
Incident Review, expand
incident with your persona
80. 80
They Got You Once, Never Again
80
● Chris opened PDF because it was
legitimate (before weaponizing)
● They brute forced portal to get PDF
● You successfully find the attack
● How do you alert moving forward?
83. 83
Returns data if we see a lot of
logon attempts and then access
to portal admin pages from a
single IP on a known threat list
In Find menu (upper right)
type “Portal Brute Force”
84. 84
We COULD select this text,
copy it, and use it in a
correlation search…but let’s
make it easy.
103. 10
3
We Want to Hear your Feedback!
After the Breakout Sessions conclude
Text Splunk to 20691
To be entered for a chance to win a $100 AMEX gift card!
Browse to https://davidveuve.com/slsc for a
SurveyMonkey, to be entered for a chance
to win a $50 Amazon gift card!
We don’t have a ton of time and ES is quite a feature-rich product. It would take many hours to go through everything the app can do. So we’ll spend only a few minutes on some intro slides, and then the great bulk of this session will be hands-on.
Now unfortunately, you do need a modern laptop with a modern browser to participate. You can probably get away with a Surface or something like that, but iPads, old browsers, and especially IBM PCjr’s will not work. (don’t laugh – I actually had one of those.)
Everything I’m going through up here has been pretty well documented in a word doc. You can use the link here to get that doc, or if you’re really interested in it later come see me. You won’t need it right now though.
Each of you has creds – there are 10 fairly large Amazon EC2 instances that have been provisioned for this exercise and if we’re at capacity there will be 12 of you on each. Now’s a good time to try hitting that URL and logging into Splunk.
Splunk excels at creating a data fabric
Machine data: Anything with a timestamp, regardless of incoming format.
Throw it all in there!
Collect it. Store it in one place. Make it accessible for search/analytics/reporting/alerting.
DETECTION NOT PREVENTION! ASSUME BREACH!
So we need a place we can go to DETECT attacks. DETECT breaches. DETECT the “weird.”
So if you had a place to see “everything” that happened…
….what would that mean for your SOC and IR teams?
The Splunk platform consists of multiple products and deployment models to fit your needs.
Splunk Enterprise – for on-premise deployment
Splunk Cloud – Fully managed service with 100% SLA and all the capabilities of Splunk Enterprise…in the Cloud
Hunk – for analytics on data in Hadoop
Splunk Mint – to get insights into data from Mobile devices
The products can pull in data from virtually any source to support multiple use cases.
Splunk Apps extend and simplify deployments by providing pre-packaged content designed for specific use cases and data types.
Our rapid ascent reflects the customer traction we have and value we deliver to customers – with thousands of security customers and 40% year-over-year growth, we are the fastest growing SIEM vendor in the market. 2011 was our first time in the MQ; In 2 short years we raced up to the top quadrant in the MQ.
3.3. 3.0 was the first release done against Splunk 6 and that was a huge step forward – mainly because of the use of CIM and accelerated data models.
Unlike other competitive solutions ES is constantly evolving – on average twice a year. Upgrades are pretty seamless.
Where does content come from? All of the typical places but most importantly it comes from YOU. We take the best ideas that you give us, and we productize them and make them scalable and supportable.
Splunk is more than a product – it is a wide open platform that inspires. None of this is lost in ES – splunk with ES is just as flexible and customizable. And it leverages technology in the core product like mapreduce and data models. You need ES to scale to the security intelligence needs of a huge enterprise? No problem.
ES has its own dev team and roadmap, dedicated support individuals, a services practice schooled in it and other complementary infosec. Also lots of training is available.
This should look familiar to you. What we’re doing here is giving a starting point for any Security Analyst to understand at a high level what’s going on in the environment. A single pane of glass, if you will, for all security data.
Everything we are seeing here is customizable – the panels, the indicators, via standard Splunk functionality.
Most of the data on this dashboard is centered on Notable Events. Notable Events are a concept unique to Splunk with ES – there’s an entire Notable Event framework that allows us to perform simple or complex correlations, and then create events by analyzing disparate events from disparate sources.
Notable Events in ES are categorized into various high-level security domains: access, audit, identity, network, and threat. We’ll see those categories throughout the app.
You can see Splunk Sparklines here – these little green lines. These are great for detecting quick trends in the security events – a continuous line means something constant, which could be a heartbeat or a scripted attack. A spike could be a single attack or maybe just someone fat-fingering their password a few times.
We’ll drill into some of these incidents in a few minutes, but let’s continue on with our tour. How does all this data get into Splunk?
Start the day like any analyst
Coffee time, or jump into incidents?
End the day like any board member
Are my security KPIs (KSIs) being met?
This should look familiar to you. What we’re doing here is giving a starting point for any Security Analyst to understand at a high level what’s going on in the environment. A single pane of glass, if you will, for all security data.
Everything we are seeing here is customizable – the panels, the indicators, via standard Splunk functionality.
Most of the data on this dashboard is centered on Notable Events. Notable Events are a concept unique to Splunk with ES – there’s an entire Notable Event framework that allows us to perform simple or complex correlations, and then create events by analyzing disparate events from disparate sources.
Notable Events in ES are categorized into various high-level security domains: access, audit, identity, network, and threat. We’ll see those categories throughout the app.
You can see Splunk Sparklines here – these little green lines. These are great for detecting quick trends in the security events – a continuous line means something constant, which could be a heartbeat or a scripted attack. A spike could be a single attack or maybe just someone fat-fingering their password a few times.
We’ll drill into some of these incidents in a few minutes, but let’s continue on with our tour. How does all this data get into Splunk?
So what does the data look like once it’s onboarded into Splunk in a CIM-compatible format?
Let’s look at one example in ES: Malware Center.
Let’s do a quick pivot to show what we can do with these fields. First we’ll load up the Malware Attacks data model and change the time to last 60 minutes. Then we’ll go to an area chart which by default shows us this time period stretched out on an X axis…
So these are overall malware attacks over the last 60 minutes in our environment. Let’s split out by the signatures….
So these are overall malware attacks over the last 60 minutes in our environment. Let’s split out by the signatures….
So these are overall malware attacks over the last 60 minutes in our environment. Let’s split out by the signatures….
Here we have a simple dashboard showing us all sorts of detail about recent malware activity in the environment. Like Security Posture, this is high level information, but more granular about a certain security domain (Malware, which is under Endpoint). We have these “centers” throughout ES for things like Access, Traffic, Intrusions, Updates, Vulnerabilities, and many other security-relevant areas, and you can investigate them later.
For now, let’s drill into two of the “top infections” to see CIM at work. Looking at this dashboard we can’t tell that we actually have at least two different endpoint protection systems feeding data into Splunk: Sophos, Trend Micro, and Symantec Endpoint Protection. Splunk normalizes the data on search time, according to CIM, to create this (and the other) dashboards.
Click on Mal/Packer, and you’ll see that this infection was detected by Sophos. The raw logs are literally a click away:
The more data you have flowing into Splunk and into ES, the more useful it becomes. And ES is self auditing to tell you which data sources you are missing:
In version 3.1 of Enterprise Security we introduced a full Risk Analysis framework. This is unique because we allow you to assign an arbitrary risk number, that means something to you, based on a notable event. You can assign risk to a user, or to a system, or to some other object that you see in the environment – perhaps a particular piece of malware is considered risky to you so you elevate the risk on the malware “object” itself.
Let’s bring up the Risk Analysis page associated with Advanced Threat:
The main reason why this risk framework is important is that it gets you away from writing specific rules for specific threats or assets. You don’t need 1,000 correlation rules anymore – you simply can elevate risk scores on whatever object you want, based on the behavior you’re seeing in the environment. So the idea here is, a correlation rule fires, and then a risk modifier takes effect and changes the risk score based on cumulative scoring of whatever else has happened to that user, or system, or other object.
On the dashboard, we can define filters to find a particular system or user or timeframe.
Note the natural language descriptions (in the screenshot they are medium and low). We track how your overall risk scoring is doing over time, and constantly re-calculate the baseline. Got a lot of activity going on that isn’t “normal” for that timeframe and you might see things going from “increasing minimally” to “extremely increasing” – all based on what the historical norm is.
We can of course see which objects have the highest risk and which correlation rules are contributing the most to the highest risk.
In version 3.1 of Enterprise Security we introduced a full Risk Analysis framework. This is unique because we allow you to assign an arbitrary risk number, that means something to you, based on a notable event. You can assign risk to a user, or to a system, or to some other object that you see in the environment – perhaps a particular piece of malware is considered risky to you so you elevate the risk on the malware “object” itself.
Let’s bring up the Risk Analysis page associated with Advanced Threat:
On the dashboard we can see that we’re using the power of Splunk search to match artifacts in our incoming data against IoC’s we find in our threat feeds. Splunk de-duplicates the threat feeds so that if an artifact shows up in multiple feeds you don’t get duplicate notifications.
We can filter the display by threat_group, which is essentially the source of the IoCs. This could be something commercial like ThreatStream or ThreatConnect or Norse, something open-source like Sans or iblocklist, or something from your ISAC that is delivered over a TAXII feed in STIX format.
The threat collection shows that we can use various IoCs to match up against artifacts in our data – IP addresses, domain names, URLs, filenames, certificate common names and organizations, email addresses, registry keys – as long as it can be defined in your incoming feed or locally, you can use it as an IoC.
You can see the most active threat sources, and if you scroll down, you can see the most recent matches against your threat feeds.
How are these configured? Let’s go to the configuration, and see.
In version 3.1 of Enterprise Security we introduced a full Risk Analysis framework. This is unique because we allow you to assign an arbitrary risk number, that means something to you, based on a notable event. You can assign risk to a user, or to a system, or to some other object that you see in the environment – perhaps a particular piece of malware is considered risky to you so you elevate the risk on the malware “object” itself.
Let’s bring up the Risk Analysis page associated with Advanced Threat:
Rounding out the Threat Intelligence capabilities are the Threat Artifacts browser, which allows us to search through all of the artifacts stored in ES:
We don’t have time to go through each and every one of the advanced threat capabilities in the ES app. However, let’s just see that up here under Advanced Threat we have some very interesting capabilities:
Some of the most useful ones are the Protocol Intelligence that leverages wire data from things like Splunk Stream, Netflow, and Bro. Also the Access Anomalies and User Activity, which are very useful to detect possible insider threat. And the New Domain Analysis, which analyzes traffic patterns and DNS queries to domains, and then tells you if you have devices communicating with recently registered garbage domains (that are often associated with DGA). Again – this is something you can go through on your own time.
We will see all of the details of the event, including our most recent comments and ownership activity.
So we know from the title of the event that we have a device on our network communicating out to a known bad IP address that’s a Zeus C2 address. But Splunk has enriched this event with some very useful info. We can see here that this particular machine is a laptop, and that it is owned by someone in Sales named Chris Gilbert. We see the IP addresses associated with the communication. We see the locations that this person Chris Gilbert works from. This correlation happens automatically against our ES Asset and Identity frameworks – we get the information an incident responder needs right up front.
Everything we see here is pivotable. We can go to places within ES, within Core Splunk and outside of Splunk too, and use that field as an argument. As an example, let’s drill into the arrow next to “Destination” and see what Domain Dossier has to say about this external IP address:
We can see that this netblock is assigned to an organization in China.
While there are a lot of these “workflow actions” associated with Notable Events configured already in the product, you can feel free to create custom ones.
Next, let’s understand what else has been going on with this laptop.
One thing that we assume is that traffic from laptops outbound to C2 servers occurs via web proxy, at least when the laptops are on our corporate network. So we can look in our proxy logs to verify.
Note that we have only one source machine (Chris Gilbert’s laptop at 192.168.56.102) communicating with this known bad IP. That’s good at least – this doesn’t appear to be a widespread infection.
Some other interesting things about this data – notice a fairly large transaction in terms of bytes. Notice also that the connection is “tcp” over port 443 not “https” which would be considered normal.
Go back to the notable event and let’s look at Asset Investigator to get a more detailed view of this possibly-infected asset:
Asset Investigator shows us, at the top, all of the things we know about this asset from sources such as CMDBs or Active Directory. It also has multiple “swimlanes” that visually show you what’s been going on with the asset:
We can see Threat List, Exec File, IDS, and Notable Events associated with this asset, most of those happening right around the same time (this was likely the time of infection).
Asset Investigator shows us, at the top, all of the things we know about this asset from sources such as CMDBs or Active Directory. It also has multiple “swimlanes” that visually show you what’s been going on with the asset:
We can see Threat List, Exec File, IDS, and Notable Events associated with this asset, most of those happening right around the same time (this was likely the time of infection).
These are all Microsoft Sysmon events. Sysmon is a great, free utility from Microsoft that is lightweight and runs on all modern Windows variants. We’re simply collecting this data from Sysmon into Splunk, in real time, from our workstations. It gives us granular process data that includes parent/child relationships, hash data, and network connections, among other things.
Note that the second event is that strange calc.exe event. Let’s click the small arrow to the far left of the event:
Note that the second event is that strange calc.exe event. Let’s click the small arrow to the far left of the event:
Finally, let’s see some of the auditing that ES does of the activity carried out against Notable Events.
The recent activity that you have carried out should appear in the panels. Clicking on a reviewer’s name will bring you detail about that reviewer’s activity.
This is a search that’s been created that returns any IP address where we see, over the timeframe selected, a lot of login attempts (greater than 10) and then loading of the admin pages of the portal from that same IP. If any IP address returns from the search, we can consider this an alertable event.
Note that you could turn this into a simple Splunk alert by just doing a “save as” alert and running it regularly. But we want to see how to turn this into a Notable Event in ES.
Using your mouse, select the entire text of the search and copy it to the clipboard.
So these are overall malware attacks over the last 60 minutes in our environment. Let’s split out by the signatures….
The main reason why this risk framework is important is that it gets you away from writing specific rules for specific threats or assets. You don’t need 1,000 correlation rules anymore – you simply can elevate risk scores on whatever object you want, based on the behavior you’re seeing in the environment. So the idea here is, a correlation rule fires, and then a risk modifier takes effect and changes the risk score based on cumulative scoring of whatever else has happened to that user, or system, or other object.
On the dashboard, we can define filters to find a particular system or user or timeframe.
Note the natural language descriptions (in the screenshot they are medium and low). We track how your overall risk scoring is doing over time, and constantly re-calculate the baseline. Got a lot of activity going on that isn’t “normal” for that timeframe and you might see things going from “increasing minimally” to “extremely increasing” – all based on what the historical norm is.
We can of course see which objects have the highest risk and which correlation rules are contributing the most to the highest risk.
On the dashboard we can see that we’re using the power of Splunk search to match artifacts in our incoming data against IoC’s we find in our threat feeds. Splunk de-duplicates the threat feeds so that if an artifact shows up in multiple feeds you don’t get duplicate notifications.
We can filter the display by threat_group, which is essentially the source of the IoCs. This could be something commercial like ThreatStream or ThreatConnect or Norse, something open-source like Sans or iblocklist, or something from your ISAC that is delivered over a TAXII feed in STIX format.
The threat collection shows that we can use various IoCs to match up against artifacts in our data – IP addresses, domain names, URLs, filenames, certificate common names and organizations, email addresses, registry keys – as long as it can be defined in your incoming feed or locally, you can use it as an IoC.
You can see the most active threat sources, and if you scroll down, you can see the most recent matches against your threat feeds.
How are these configured? Let’s go to the configuration, and see.
Asset Investigator shows us, at the top, all of the things we know about this asset from sources such as CMDBs or Active Directory. It also has multiple “swimlanes” that visually show you what’s been going on with the asset:
We can see Threat List, Exec File, IDS, and Notable Events associated with this asset, most of those happening right around the same time (this was likely the time of infection).
The recent activity that you have carried out should appear in the panels. Clicking on a reviewer’s name will bring you detail about that reviewer’s activity.
We’re headed to the East Coast!
2 inspired Keynotes – General Session and Security Keynote + Super Sessions with Splunk Leadership in Cloud, IT Ops, Security and Business Analytics!
165+ Breakout sessions addressing all areas and levels of Operational Intelligence – IT, Business Analytics, Mobile, Cloud, IoT, Security…and MORE!
30+ hours of invaluable networking time with industry thought leaders, technologists, and other Splunk Ninjas and Champions waiting to share their business wins with you!
Join the 50%+ of Fortune 100 companies who attended .conf2015 to get hands on with Splunk. You’ll be surrounded by thousands of other like-minded individuals who are ready to share exciting and cutting edge use cases and best practices. You can also deep dive on all things Splunk products together with your favorite Splunkers.
Head back to your company with both practical and inspired new uses for Splunk, ready to unlock the unimaginable power of your data! Arrive in Orlando a Splunk user, leave Orlando a Splunk Ninja!
REGISTRATION OPENS IN MARCH 2016 – STAY TUNED FOR NEWS ON OUR BEST REGISTRATION RATES – COMING SOON!