Splunk Fundamentals: Investigations with Core Splunk - Splunk Tech Day

© 2015 Global Technology Resources, Inc. All Rights Reserved.
Contents herein contain confidential information not to be copied.
1
Splunk Fundamentals
Investigations with Core Splunk
Hosted by Global Technology Resources, Inc.
Taylor Williams
twilliams@gtri.com

2
GTRI Quick Facts
 Unique federal qualifications
 Cleared to support mission-critical projects
 Highly successful SBA 8 (a) program graduate (2010)
 Proven graduate of DoD Mentor Protégé of
NGA/Raytheon IIS (2010)
 Solutions-oriented consultants
 Averaging over 10 years of hands-on experience
 Culture of customer focus
 Relentless Commitment
 Operational excellence
 ISO 9001:2008 quality management certified
 Proven processes designed to mitigate risk

GTRI Splunk-Practice Overview
Highlights:
• Splunk’s 1st Elite Partner and one of only two Splunk Certified Training
Centers in the U.S.
• GTRI provides end-to-end support for Splunk from pre-sales engineering to
post-sales professional services, implementation, training and optimization
• Splunk’s most credentialed partner in N. America:
– GTRI holds over 60 Splunk Certifications:
• 8 Certified Architects
• 14 Certified Solutions Engineers (SE-I & SE-2)
• Certified Training Center
3

4
GTRI Overview and Capabilities

Agenda
• Provide a fundamental understanding of the components
in a Splunk implementation and how they scale
• Provide hands-on examples of Splunk tasks to provide
insight on how Splunk expedites system diagnostics and
investigation
• Labs are incorporated to allow the attendees to learn by
exploring. They have practical instruction not directly
covered by the lecture.
5http://www.splunk.com/view/SP-CAAAH9Q

Topics
• Splunk Overview
• Splunk Architecture
• Data Collection
• Splunk for Discovery
• Automation: Let Splunk Do the Work
• Splunk Apps
6

Splunk Overview: What is Splunk?
Splunk is a big data platform designed to
make machine data accessible and
meaningful
7
Data Collection
Ad-hoc searches
Dashboards

Splunk Overview: How is Splunk Used?
Traditional
• Applications
• Security
• SOC
• NOC
8
Custom
• SCADA
• Election Data
• Energy Consumption
• …
Use Cases

Architecture: Main Splunk Server Functions
9
Searching and Reporting
A Splunk install can be one or all roles…
Indexing and Search Services
Data Collection and Forwarding
Search Head
Indexer
Forwarder

Architecture: Multi-tiered Environment
10
Single Server: Demos, POC, … Enterprise Scale

Architecture: Operating Systems
Splunk runs on most Windows and
Unix Variants (32 bit/64 bit)
All binaries
• have identical disk structure
• have identical command line
interface
• communicate via network bridging
operating systems
Windows binaries include extra inputs
• Registry
• Event Logs
11http://docs.splunk.com/Documentation/Splunk/6.1/Installation/Systemrequirements

Lab Access
This is a hands on class where everyone has their own sandbox to
work in. Logins are a part of the
provided class materials.
Server: http://bootcamp.gtri-training.com
Credentials are provided in logins.pdf
12http://docs.splunk.com/Documentation/Splunk/latest/Installation/Systemrequirements#Supported_browsers

13
LAB 1: Getting Connected (5 minutes)
• Log in to Splunk
• Customize display information

Splunk is licensed by volume of uncompressed data/day –
collect what is needed.
Questions to ask:
• What are you trying so solve?
• Where is that information?
• How is your data accessed?
• How long do you want that data searchable?
• Is there information that needs to have limited visibility?
• What kind of archival strategy(ies) is needed?
Data Collection: Know Your Data
14

15
Data Collection: Where is Your Data?
• Any log files
• Custom applications
• Web servers
• User clickstreams
• Social platforms
• Configuration files
• Telecoms devices
• Storage devices
• Network devices
• Databases
• Web Services
• System metrics
Splunk can digest any type of text data –
What data do you want to Splunk?
• GPS
• Security devices
• Servers/Hypervisors/VMs
• DNS, DHCP
• AAA Logs
• Proxy servers
• Errors
• Scripts
• Sensors
• …

Data Collection: Inputs
16
Types of inputs
• Files and directories – monitor physical files on disk
• Network inputs – monitor network data feeds on specific ports
• Scripted inputs – import from non-traditional sources, APIs,
databases, etc.
• Windows inputs – are Windows specific; Windows event logs,
performance monitoring, AD monitoring, and local registry
monitoring
• File system change monitoring – monitor the state: permissions,
read only, last changed, etc. of key config or security files

17
LAB 2: Exploring the Splunk
Interface (10 minutes)
• Find the data
• Run basic key word searches
• Challenge: Using a key word search, find
the number of times in the last 8 hours
that the access control list blocked an
action.

18
Lab Review
• Search Application
• Keyword Searches
• Booleans: AND, OR, NOT
• Not Case Sensitive

Splunk for Discovery: Definitions
Event
A single piece of data in Splunk,
similar to a record in a log file.
When you run a search, events are
what you get back.
Can be single or multiple lines
Each event has the following fields
• timestamp
• host
• source
• source type
• index
19

Field
Searchable name/value pair associated with
Splunk event data.
Fields give you more precision in searches.
20

Source type
the data format from which the event originates, such as
ps or cisco:asa.
• Splunk has many source types pre-trained
• Additional source types can be created as needed
• Field definitions are defined per source type.
• Splunk Common Information Model (CIM) defines
what fields should be extracted from source types
and what their names should be. This facilitates field
reuse in different applications.
21
http://docs.splunk.com/Documentation/Splunk/6.1/Data/Listofpretrainedsourcetypes

Splunk for Discovery: Demo
Purchasing Problem Scenario:
A call comes in about users not being able to make
purchases.
The lab has a multi-tier implementation so let’s step through
the investigation and where it leads us.
22

23
LAB 3: Core Splunk Investigation
• Calls have come in reporting users having difficulty
connecting to a web application..

24
Lab Review
1. Search for concur*
2. Chart the status

25
Lab Review: (cont.)
3. Check the Remedy
Change Ticket data
4. Chart the Firewall
connections denied over
time

Automation: Let Splunk Do the Work
Rapid
Investigation
26
Proactive
Monitoring

Automation: Saving Searches
After you develop a search you may want to persist it
as a Knowledge Object to be reused or referenced
• Report
• Alert
• Dashboard Panel
27

Automation: Report Knowledge Object
28
Save a Report from the search screen Access the report using the Reports menu item
NOTE: You can also access Reports from the Settings link

Automation: Alerting
Anything that can be searched for can be alerted upon
Alert Actions
• list in triggered alerts
• send an email
• custom action via a script
– automatic actions
– entries into a ticketing system
29

Automation: Creating an Alert
30
Step one Step two
Save As Alert Flow

Automation: Reports, and Alerts
Reports and alerts are really the same
type of Knowledge Object. Reports
just have more fields filled out.
31
Search
String
Schedule Alert
Defined
Report x
Scheduled
Report
x x
Alert x x x

Automation: Simple Dashboards
32
Add the Report to a new or existing
dashboard
Start with a developed search and
select the Save As > Dashboard Panel

Automation: Editing Dashboards
33
Use the Edit->Edit Panels feature to
rearrange charts or modify searches
• Drag the header to change rows or
combine panels on one row
• Use the paint brush to modify the
labeling
• Use the chart line to modify the
visualization
• Use the magnify icon to modify the
search

34
LAB 4: Splunk Investigation and Simple Dashboard (15-20 min)
• Users have reported that internet applications have suddenly gotten
sluggish. The likely cause is network bandwidth, so we will look at the proxy
logs to determine how bandwidth is being used
• Add the result to a new dashboard
• Add the searches from the previous 2 labs to the
dashboard

Splunk Apps: Definition
From Splunk website: “Apps are a self-service, out-of-the box extension for Splunk.
Apps serve as workspaces for tailored configuration for configuration or display.”
Most apps are just fancy versions of the dashboard you created in the lab.
Apps may contain
• A UI context selected from the App list dropdown
• Knowledge objects including saved reports, alerts, and custom-designed views
and dashboards
• Configuration
Apps are available at apps.splunk.com for free with the exception of Enterprise
Security, VMWare, PCI, Exchange
35

Splunk Apps: Repository
apps.splunk.com
• Many providers, from Splunk
to individuals
• Collected data is immutable,
apps will not change existing
data
• Apps can change how data
is collected and how it is
displayed
36

37
Cisco Security Suite App
Provided by Cisco
Dashboards and searches for Cisco appliance data
• Cisco Client Security Agent (CSA)
• Cisco IronPort Email Security Appliance
(ESA)
• Cisco IronPort Web Security Appliance (WSA)
• Cisco Firewalls (PIX, FWSM, ASA)
• Cisco IPS
• Cisco MARS

38
Application Management App
Custom App Developed by Splunk for Demo
purposes
Provides an example of how Splunk can
monitor a multi-tiered system.
Demo:
Investigating Environment Issues within an
application

39
Application Management: Recap
From the Environment State dashboard, it is
clear that the DB layer is taking too much
time
Further, the Queue size is running higher
than the 7 day average
Drilling into the DB State
dashboard, there is a recent
unauthorized change to the
queue size.

40
Return to normal…
Observe the changes to your dashboard
when conditions are corrected and activity
returns to normal.
page Not Found errors drop off
DirecTV load disappears

Summary
Topics
• Splunk Overview
• Splunk Architecture
• Data Collection
• Splunk for Discovery
• Automation: Let Splunk Do the Work
• Splunk Apps
41

43
Thank You!
Global Technology Resources, Inc.

Splunk Fundamentals: Investigations with Core Splunk - Splunk Tech Day

More Related Content

What's hot

Viewers also liked

Similar to Splunk Fundamentals: Investigations with Core Splunk - Splunk Tech Day

More from Zivaro Inc

Recently uploaded

Splunk Fundamentals: Investigations with Core Splunk - Splunk Tech Day

Editor's Notes