Splunk Enterprise for IT Troubleshooting Hands-On

1. Copyright © 2016 Splunk Inc.
Splunk Enterprise for
IT Troubleshooting
Tyler Germer
Sales Engineer
tgermer@splunk.com

2. 2
Safe Harbor Statement
During the course of this presentation, we may make forward-looking statements regarding
future events or the expected performance of the company. We caution you that such
statements reflect our current expectations and estimates based on factors currently known
to us and that actual events or results could differ materially. For important factors that may
cause actual results to differ from those contained in our forward-looking statements, please
review our filings with the SEC. The forward-looking statements made in this presentation
are being made as of the time and date of its live presentation. If reviewed after its live
presentation, this presentation may not contain current or accurate information. We do not
assume any obligation to update any forward-looking statements we may make. In addition,
any information about our roadmap outlines our general product direction and is subject to
change at any time without notice. It is for informational purposes only and shall not be
incorporated into any contract or other commitment. Splunk undertakes no obligation either
to develop the features or functionality described or to include any such feature or
functionality in a future release.

3. 3
Agenda
1. Splunk Enterprise Overview
2. Troubleshooting with Splunk (Live Demo /
Walkthrough)
• Installing & Setting up an App
• Logging in and Searching
• Extracting Fields
• Troubleshooting Infrastructure
• Troubleshooting Applications
• Creating an Alert
• Creating Reports and Dashboards
3. Wrap Up
4. Q&A

4. 4
Escalating IT Complexity…
SaaS/PaaS
IaaS
VIRTUALIZATION
STORAGE
PACKAGED
APPLICATIONS
CUSTOM
APPLICATIONS
HR
Email
Finance
App Svr
DB
Web Svr
INFRASTRUCTURE
APPLICATIONS
VPN
IP Phone
Identify
SERVERS NETWORKING
4

5. 5
… Plaguing IT Operations
SaaS/PaaS
IaaS
VIRTUALIZATION
STORAGE
PACKAGED
APPLICATIONS
CUSTOM
APPLICATIONS
HR
Email
Finance
App Svr
DB
Web Svr
INFRASTRUCTURE
APPLICATIONS
VPN
IP Phone
Identify
SERVERS NETWORKING
Complex, silo-based technologies
Disconnected and outdated point solutions
Reactive brute-force problem resolution
Over 80% of time on maintaining, not innovating
5

6. 6
Enterprise
Scalability
Industry-Leading Platform for Machine Data
Any Machine Data Operational Intelligence
Online
Services Web
Services
Servers
Security GPS
Location
Storage
Desktops
Networks
Packaged
Applications
Custom
ApplicationsMessaging
Telecoms
Online
Shopping
Cart
Web
Clickstreams
Databases
Energy
Meters
Call Detail
Records
Smartphones
and Devices
RFID
Private
Cloud
Public
Cloud
Search and
Investigation
Proactive
Monitoring
Operational
Visibility
Real-Time
Business
Insights
6
Datacenter
Containers

7. 7
Enterprise
Scalability
Any Machine Data Operational Intelligence
Online
Services Web
Services
Servers
Security GPS
Location
Storage
Desktops
Networks
Packaged
Applications
Custom
ApplicationsMessaging
Telecoms
Online
Shopping
Cart
Web
Clickstreams
Databases
Energy
Meters
Call Detail
Records
Smartphones
and Devices
RFID
Datacenter
Private
Cloud
Public
Cloud
Containers
Search and
Investigation
Proactive
Monitoring
Operational
Visibility
Real-Time
Business
Insights
Industry-Leading Platform for Machine Data
Any amount, any location, any source
Schema-
on-the-fly
Universal
indexing
No
backend
RDBMS
No need
to filter
data
7

8. 8
The Focus
Application
Delivery
IT
Operations
Security,
Compliance
and Fraud
Business
Analytics
Internet of Things
and Industrial Data
Developer Platform (REST API, SDKs)
8

9. 9
Turning Machine Data Into Operational Intelligence
Reactive
Proactive
9
Proactive
Monitoring
and Alerting
Real-Time
Business
InsightOperational
Visibility
Search
and
Investigate

10. 10
Index and Analyze Data Across Your Technology Stack
Splunk Add-Ons, Templates and Apps Accelerate Value From Machine Data
No rigid schemas– add in data from any other source.
API
SDKs UI
Server, Storage,
Network
Virtualization,
Containers
Operating Systems
+ Databases
Custom
Applications
Business
Applications
Cloud Services
Web Intelligence
Mobile
Applications
Stream
Operations and
Service Desks
App Performance
Monitoring
DB Connect
1

11. 11
Install Splunk and App
Ø Install Splunk (installers on USB keys)
Ø Start Splunk
Ø splunk start --accept-license --answer-yes --no-prompt
Ø Log in
Ø http://localhost:8000
Ø Default credentials: admin/changeme
Ø Install app
Ø Click the widget next to “Apps”
Ø Install app from file
Ø Choose the app from the USB key
Ø Restart Splunk

12. 12
Splunk Demo Environment - Cloud
Ø Go to one of the following URLs:
Ø https://od-sl-longbeach-itops-01.splunkoxygen.com
Ø https://od-sl-longbeach-itops-02.splunkoxygen.com
Ø https://od-sl-longbeach-itops-03.splunkoxygen.com
Ø Log in to Splunk using the following credentials:
– User: user[1-10]@buttercupgms.com
– Password: Changeme[1-10]

13. 13
Login to Splunk
Click Search & Reporting
to get started using
Splunk!

14. 14
Searching with Splunk
Start by typing *
in the search bar!

15. 15
Search Results
Explore the results!
Click on host
Click on sourcetype
Look at the other
fields
Next, we’ll extract
new fields!

16. 16
Extracting Fields
1
Start with this search:
sourcetype=customlog
Then scroll down and click on Extract New
Fields at the bottom of the field list.

17. 17
Extracting Fields
Click on any event in the list
Fields that are already being extracted
are highlighted.
Click Next to continue

18. 18
Extracting Fields
Choose Regular
Expression
Don’t worry – we won’t
be writing any regexes!
Click Next

19. 19
Extracting Fields
Click and drag to
highlight the last field
(it will be OK or NSF)
Type status_description
in the Field Name box
Click Add Extraction

20. 20
Extracting Fields
2
Check out the
status_description
column!
Click Next

21. 21
Extracting Fields
2
Notice the Permissions
(You don’t need to change
anything now, but you may
need to look this up on
docs.splunk.com later if you
have trouble!)
Click Finish
Your new field is now
EXTRACTED!

22. 22
Troubleshooting Infrastructure
We have reports of problems
on the website, so let’s search
sourcetype=access*
Look at the available fields –
feel free to explore!
Click on the status field to see
the top values
Click on Top values by time

23. 23
Extracting Fields
2
Click on Explore
the fields I just
created in Search
When your search
results show up,
expand an event
by clicking on the >
Check out your
new field!

24. 24
Troubleshooting Infrastructure
2
Change the graph from a
Line to a Column
Change Format to Stacked
Now we can see the
distribution of status codes
over the last hour!
Click on 503 in the legend
on the far right

25. 25
Troubleshooting Infrastructure
2
Now we can see all the events with a
503 status!
Add | stats count by host to your
search to see how many 503 status
codes each server has
sourcetype=access* status=503 | stats
count by host
Now we can see that webserver-01 has
more errors!
Click webserver-01 then click New
Search

26. 26
Troubleshooting Infrastructure
2
Notice that your search is
now * host=“webserver-01”
Click on sourcetype to see
what kinds of data we have
Let’s start by checking for
disk space problems – click
on df

27. 27
Troubleshooting Infrastructure
2
Notice that your search is now
* host=“webserver-01” sourcetype=df
Scroll down and click on the field
PercentUsedSpace
Click on Maximum value over time
This will help us see if we have a disk
full problem!

28. 28
Troubleshooting Infrastructure
2
No disk space issues here – the maximum
doesn’t go much over 70%!
Let’s change our search to look for a
different sourcetype – modify the search
to look for CPU data
Your search should look like this:
* host=“webserver-01” sourcetype=cpu

29. 29
Troubleshooting Infrastructure
2
Scroll down and click on the field PercentUserTime, then
click on Maximum value over time
It looks like we found the problem! The CPU is maxed out.

30. 30
Troubleshooting Applications
3
Now search for error
Many results are coming
from a test machine – on
any event where host = test-
01, click test-01, then click
Exclude from search
Your search will now be
error host!=“test-01”
The events that remain
indicate a problem with one
of the MySQL servers –
which one?

31. 31
Troubleshooting Applications
3
It looks like mysql-02 is the server
having issues
The errors show a problem
writing log files, so let’s check for
disk space issues
Search for host=mysql-02
sourcetype=df
Click on PercentUsedSpace and
choose Maximum value over time

32. 32
Creating an Alert
3
We found the problem – a full disk!
But wouldn’t an alert be better?
Timechart is great for data over time, but let’s
change the search to use stats, which will give
us a single number on the Statistics tab
To make it easier to read, we’ll rename the field
max(PercentUsedSpace) to maxused by adding
as maxused to the end your search
Now your search should be host=mysql-02
sourcetype=df | stats max(PercentUsedSpace)
as maxused

33. 33
Creating an Alert
3
Click Save As and choose Alert to
bring up the settings
Add a Title
Set a schedule or choose Real-time
Set Trigger Conditions – use the
maxused field that we created
Throttle alerts to reduce noise
Set an Action for the alert
That’s it!

34. 34
Creating Reports
3
Start with the same search as for the alert:
host=mysql-02 sourcetype=df | stats
max(PercentUsedSpace) as maxUsed
On the Visualization tab, choose Radial
Gauge as the chart type
You can change the color thresholds by
choosing Color Ranges under Format
Click Save As and choose Report

35. 35
Creating a Dashboard
3
From your report, click Add to
Dashboard to create a new
dashboard
Click Edit so we can add more
information to this dashboard
Click Add Panel, then
choose A Sample Panel
from Add Prebuilt Panel
This panel shows the
disk space for another
server – mysql-03
Rearrange panels by
dragging and dropping

36. 36
Using Dashboards
3
Click on Dashboards and then
choose Website Health
This dashboard lets us see that
there are some spikes in 503
errors and that webserver-01
has the most errors – easier
than all the searches we started
with!
Click on the pie chart in the
panel labeled Errors by Server

37. 37
Using Dashboards
3
This dashboard shows us
some OS statistics, and
we can see the CPU
issue with webserver-01
in the Maximum CPU by
Server over Time panel
Dashboards let us
troubleshoot common
problems faster!

38. 38
Dramatic Results, Rapid ROI
3
99.7% Uptime
Accelerated from monthly
releases to 900 deploys per day
Incident reduction by
90%
95% reduction in MTTR
30% acceleration in SDLC

39. 39
Now what?
3
Full-featured platform for real-time Operational Intelligence
Download Splunk Enterprise for free!
Splunk Enterprise as a cloud service
Try out Splunk Cloud with a free trial!
Feel free to keep working with the data from your USB key!
Learn more with Splunk Education!

40. Copyright © 2015 Splunk Inc.
• 5,000+ IT and Business Professionals
• 175+ Sessions
• 80+ Customer Speakers
PLUS Splunk University
• Three days: Sept 23-25, 2017
• Get Splunk Certified for FREE!
• Get CPE credits for CISSP, CAP, SSCP
SEPT 25-28, 2017
Walter E. Washington Convention Center
Washington, D.C.
CONF.SPLUNK.COM
The 8th Annual Splunk Worldwide Users’ Conference

41. 4
Thank You