Are you currently using Splunk to troubleshoot and monitor your IT environment? Do you want more out of Splunk but don’t know how? Here’s your chance to learn more about Splunk IT Service Intelligence (Splunk ITSI) and get hands-on with it for the very first time. We’ll kick off this session with a discussion on the concept of services, KPIs and entities and demonstrate how to use them in Splunk IT Service Intelligence. We’ll help you build custom visualizations and dashboards for personalized service-centric views. We’ll teach you how to navigate across multiple KPIs, entities and events with built-in visualizations and intelligently troubleshoot and resolve problems faster using Splunk ITSI. We’ll also show you how to create correlations across KPIs easily and be alerted of “notable events” to catch these emerging problems quickly. At the end of this session, you will leave with an understanding of the unique monitoring approach Splunk ITSI delivers to maximize the value of your data in Splunk and how to accelerate visibility into your critical IT services.

1. Copyright © 2015 Splunk Inc.
Getting Started with
IT Service Intelligence
Naman Joshi
Snr Sales Engineer

2. Setup Before You Can Play
Download this presentation slide deck: https://splunk.box.com/splunkliveitsi16
Follow the instructions on your paper hand-out to log in to your VM.
Please log in as either
• user1@buttercupgms.com OR
• user2@buttercupgms.com
• Password is “Changeme1” or
“Changeme2”
After logging in, select
IT Service Intelligence from the
list of apps at the left
2

3. Agenda
3

4. ITSI Core Concepts
4

5. What is a Service?
Service
Requests
Responses
In ITSI, a Service is a logical group of technology components that a user
deems need to be monitored together.
It can often be generalized as a “black box” which we send requests, and
expect responses
5

6. What is a Service?
DNS
Requests
Responses
Technical Services
Auth
Requests
Responses
Web
Requests
Responses
Services can be lower level (technical) …
6

7. What is a Service?
DNS
Requests
Responses
Technical Services
Customer
Transactions
Requests
Responses
Business Services
Auth
Requests
Responses
Web
Requests
Responses
Support Desk
Requests
Responses
Services can also be higher level (business) …
7

8. What is a Service?
Packet Network
Hypervisor and Hosts
RBMDBs
Storage Tier
API Services
Web Services
Customer Transactions
Mobile
API/Middleware
Partner Portal
DNS
Services can encompass multiple tiers of the IT domain.
Services may also depend upon other services
8

9. What is a KPI?
DNS
Requests
Responses
KPI: Number of requests
KPI: Error rate
KPI: Average response time
KPI: Server CPU load
KPI: Server network I/F errors
Customer
Transactions
Requests
Responses
KPI: Number of transactions
KPI: Error rate
KPI: Average response time
KPI: Count of Incident Tickets
KPI: Synthetic Transx Health
KPIs and Health scores constitute the means by which
Services are monitored.
9

10. Key Performance Indicators (KPIs)
10
A Key Performance Indicator (KPI) is a Splunk saved search created within the
ITSI UI that helps monitor a specific field like CPU, Memory, Number of Errors
and so on. KPIs are contained within Services.

11. Service Health Scores
11
A Health score is a score form 0-100 (0 being critical and 100 being normal)
that helps determine the health of a Service. It is calculated based on all KPIs
importance and its status (e.g. green, orange, red), once every minute.

12. Let’s Talk Entities
12
● Entities are the relevant
components that support a
service (often but not
always hosts)
● Select the correct entities
with filters, ANDs, ORs
● Entity list can come from a
CMDB, a spreadsheet, a
Splunk search…

13. Service Decomposition
13

14. Service Decomposition in ITSI
14
CLICK
“Glass Tables”

15. Service Decomposition in ITSI
15
CLICK (open in new tab)
“Buttercup Games
Business Process (IN
PROGRESS)”

16. Service Decomposition in ITSI
16
CLICK (open in new tab)
“Buttercup Games
Online Store”

17. Service Decomposition in ITSI
17
Identify a high-value business service

18. Service Decomposition in ITSI
18
Identify the process flow and underlying sub-services
(Web -> Middleware -> DB -> Middleware -> Web)

19. Service Decomposition in ITSI
19
For each sub-service, identify
KPIs that will show health and
status
(Requests, response time,
errors, OS health…)

20. Service Decomposition in ITSI
20
For each KPI,
define a
Splunk search

21. Service Decomposition in ITSI
21

22. ITSI Demo
22

23. Service Decomp: The Business Processes
23

24. Service Decomp: End-To-End Process Flow
24

25. New Requirements!
25
● Create a new KPI for the DB Service:
● Network Utilization
● Modify the Executive Glass Table
in order to show off the services
you slave over
“WE only have about 15min
TO DO WHAT ???!!???”
Think about how long this
would take you today?

26. 26
Configuration of DB Service
Click Configure >
Click Services

27. A KPI in 5 minutes? Absolutely!
27
Click New – Generic KPI
Select Data Model
● Host Operating System
● Network
● # bytes
● Next

28. KPIs Continued….
28
Splunk Builds Searches for you –
Oh Yeah, that’s happening J
● Select Yes for Split by & Filter options
● Select host for Entity Lookup & Alias options
● Click Next

29. Almost There…
29
Select
● KPI Search Schedule: Every Minute
● Entity Calculation: Average
● Service/Agg Calculation: Average
● Calculation Window: Last Minute
● Click Next
● Unit: Bps
● Click Next

30. Final Steps …
30
Set your thresholds:
● Aggregate (All)
● Per Entity
● Click “Add Threshold” TWICE
● Make the Neapolitan ice cream colors
Yellow, Green, Yellow
● Drag the sliders around in order to get
the current data graph entirely inside the
Green (normal) band
● Click Finish
● Other options are also available,
including adaptive thresholds and
anomaly detection

31. Adaptive Thresholds
31
What if your KPI data looks like this?

32. 32
Adaptive Thresholds
Static thresholds will not work…

33. 33
Adaptive Thresholds
Adaptive Thresholding works beautifully with cyclical (and other dynamic) data

34. Anomaly Detection
34
● Machine Learning
● Works well for data with patterns
● Requires some “training” (trial & error)
to zero in on best sensitivity

35. ITSI Demo –
Troubleshooting
35

36. Name that KPI!
36
From the list of KPIs, select your new one (at the bottom)
● Click on the little pencil next to the name
● Call it “Network Utilization”,
with your username up front
● Click on Save at bottom right when finished!

37. Let’s Fix that Glass Table
37

38. Clone the Glass Table
38
Return to Saved Glass Tables page
(click on Glass Tables in the upper menu bar)
CLICK Edit for “Buttercup Games Business Process (IN
PROGRESS)”
• Select Clone
• Title: Add your username
to the front
• Permissions: Shared in App
• Click Clone Page
• Click on your new Glass Table
from the list, to view it

39. Edit & Have Fun!
39
Click on Edit in the upper right corner of your Glass Table
Use the “Services” panel on the left to select Individual KPIs,
or Aggregate Service Health Scores
• Choose 2 KPIs from Online Store that would be useful in
the “Order Process” section
• Drag the selected widgets onto the canvas, positioning in
the gray oval
• What’s the difference between the
and tools at the top left?

40. More Fun with the Glass Table Editor…
40
Use the Configurations panel on the right to edit a
selected widget
• Can change the visualization type, drilldown
behavior, and other settings
• You should hit Save frequently
• I wonder what Auto Layout does?
• (YIKES!) Revert All Changes might be helpful

41. Finishing up …
41
• Add a ServiceHealthScore widget for Online
Store under Buttercup
• Choose a Viz Type with a sparkline graph, then
resize to make it look pretty
• Modify the Custom Drilldown action to go to
the saved glass table,
Buttercup Games Online Store
• Bonus Points: Make the label bigger, more
readable
• Click Save
• View when done

42. A Troubleshooting Exercise
42
Let’s use ITSI to troubleshoot an outage
● Start at your Glass Table, “<UserName> Buttercup Business Process”
● Customer Care reports that unhappy customers are complaining of failures
and long delays when trying to purchase
● The calls began coming in at around ten minutes after the hour.
● In the upper right corner of the Glass Table, change the time picker from Now
to XX:10:00.0, where XX is the appropriate hour. For example, if it is currently
14:05, set the time picker to 13:10:00.0, then Apply
● This is how we can “time travel” back to see conditions at a particular
outage– oh yeah!

43. A Troubleshooting Exercise, cont’d
43
● The Online Store seems to be degraded, just as Customer Care reported.
Click on the widget under Buttercup to drill down further

44. A Troubleshooting Exercise, cont’d.
44
● The Online Store Glass Table shows a much more detailed view, including the impacted customer-facing KPIs
at the far left (Revenue, etc)
● Based on this view of all the relevant
services, where do you think the root cause
lies?
● Which service should we troubleshoot first?
● Click on Health widget for that service, to
drill down to a Deep Dive

45. Deep Dive
45
● Deep Dive shows multiple KPIs and Health
Scores in parallel “swim lanes”.
● The Health Score for this Service is the top
swim lane. Can you see when it begins to
degrade from 100%?
● Mousing over this point in time, can you
spot the KPI with the leading fault
indication, i.e., what failed first?

46. Multi-KPI Alerts and Notable Events
46
● Click on Notable Events Review
● Multiple KPIs and Healthscores can
be combined in sophisticated ways
to create Multi-KPI alerts
● When a Multi-KPI alert fires, one
of the outcomes is the creation of
a Notable Event
● Notable Events allow NOC
personnel and others to triage and
coordinate event management
efforts

47. Service Analyzer
47
● Click on Service Analyzer > Default Service Analyzer
● Back where we started!
● This view shows a “no-frills” list of
services (top) and hottest KPIs
(bottom)
● Provides a quick jumping off point
into Deep Dives and the Notable
Events Review
● It is useful for NOCs and others
who need a high-level situational
view

48. Wrap Up - Review
48
● High-value services can be decomposed and modeled in ITSI, using machine data
from the relevant systems
● Services and KPIs can be created in minutes, with sophisticated thresholding
techniques to distinguish “normal” from “not normal”
● Glass Tables allow service health and KPI metrics to be displayed in a way that
makes sense to specific groups, such as Executive Leadership, Business Service
Owners, the NOC, DevOps & Others
● Deep Dives allow KPIs to be compared side-by-side across any time range,
accelerating root cause analysis and significantly reducing MTTR
● Multi-KPI Alerts and Notable Events reduce alert noise, producing actionable
events and a means to manage them
● … and it’s fun to build!

49. Want to explore on your own?
49
Sign up for your very own seven-day free sandbox!
http://splunk.com/ITSI
Then click:
You’ll find a Sandbox Guide in the Dashboards!
In the ITSI app of your sandbox, go to Search > Dashboards > ITSI Sandbox Guide

50. Northern Cal Tech Talks!
Monthly WebEx Sessions
• Ted Talk style presentation
• Q&A Chat forum
So what’s next on the agenda?
• April 20th @ 10AM PST - Top 5 most useful
search commands.
• May 18th @ 10AM PST – Splunk for IT
Service Intelligence
See more at:
http://live.splunk.com/NorCalTechTalks

51. 51
SEPT 26-29, 2016
WALT DISNEY WORLD, ORLANDO
SWAN AND DOLPHIN RESORTS
• 5000+ IT & Business Professionals
• 3 days of technical content
• 165+ sessions
• 80+ Customer Speakers
• 35+ Apps in Splunk Apps Showcase
• 75+ Technology Partners
• 1:1 networking: Ask The Experts and Security
Experts, Birds of a Feather and Chalk Talks
• NEW hands-on labs!
• Expanded show floor, Dashboards Control
Room & Clinic, and MORE!
The 7th Annual Splunk Worldwide Users’ Conference
PLUS Splunk University
• Three days: Sept 24-26, 2016
• Get Splunk Certified for FREE!
• Get CPE credits for CISSP, CAP, SSCP
• Save thousands on Splunk education!

52. A flying start to Service Intelligence
Start With A problem worth solving
Collaborate with Subject Matter Experts
Design Before Configuring

53. Sign Up Here - We’re Here To Help!
Harness the creativity and domain knowledge of your organization to unlock
the value of data and solve an important service problem through a joint
service intelligence workshop with key stakeholders
Define methods for:
• Proactive service monitoring
• Reduced risk and failures
• Faster issue resolution
• Increased business
performance
What is it?
• 1 Day Onsite Workshop
• Tightly linked with value
• Collaborative approach
• Build your own Splunk
ITSI Glass Table……

54. Thank You