This document provides an overview of IT service intelligence concepts including services, key performance indicators (KPIs), and service decomposition using Splunk IT service intelligence (ITSI). It discusses how services can represent different levels of an IT environment from technical to business services. KPIs and health scores are used to monitor services. The document demonstrates how to create a new KPI, edit a glass table dashboard, and use ITSI for troubleshooting an outage by analyzing service and KPI data over time.

1. Copyright © 2015 Splunk Inc.
Getting Started with
IT Service Intelligence
Naman Joshi
Snr Sales Engineer

2. Agenda
2

3. ITSI Core Concepts
3

4. What is a Service?
Service
Requests
Responses
In ITSI, a Service is a logical group of technology components that a user
deems need to be monitored together.
It can often be generalized as a “black box” which we send requests, and
expect responses
4

5. What is a Service?
DNS
Requests
Responses
Technical Services
Auth
Requests
Responses
Web
Requests
Responses
Services can be lower level (technical) …
5

6. What is a Service?
DNS
Requests
Responses
Technical Services
Customer
Transactions
Requests
Responses
Business Services
Auth
Requests
Responses
Web
Requests
Responses
Support Desk
Requests
Responses
Services can also be higher level (business) …
6

7. What is a Service?
Packet Network
Hypervisor and Hosts
RBMDBs
Storage Tier
API Services
Web Services
Customer Transactions
Mobile
API/Middleware
Partner Portal
DNS
Services can encompass multiple tiers of the IT domain.
Services may also depend upon other services
7

8. What is a KPI?
DNS
Requests
Responses
KPI: Number of requests
KPI: Error rate
KPI: Average response time
KPI: Server CPU load
KPI: Server network I/F errors
Customer
Transactions
Requests
Responses
KPI: Number of transactions
KPI: Error rate
KPI: Average response time
KPI: Count of Incident Tickets
KPI: Synthetic Transx Health
KPIs and Health scores constitute the means by which
Services are monitored.
8

9. Key Performance Indicators (KPIs)
9
A Key Performance Indicator (KPI) is a Splunk saved search created within the
ITSI UI that helps monitor a specific field like CPU, Memory, Number of Errors
and so on. KPIs are contained within Services.

10. Service Health Scores
10
A Health score is a score form 0-100 (0 being critical and 100 being normal)
that helps determine the health of a Service. It is calculated based on all KPIs
importance and its status (e.g. green, orange, red), once every minute.

11. Let’s Talk Entities
11
● Entities are the relevant
components that support a
service (often but not
always hosts)
● Select the correct entities
with filters, ANDs, ORs
● Entity list can come from a
CMDB, a spreadsheet, a
Splunk search…

12. Service Decomposition
12

13. Service Decomposition in ITSI
13
CLICK
“Glass Tables”

14. Service Decomposition in ITSI
14
CLICK (open in new tab)
“Buttercup Games
Business Process (IN
PROGRESS)”

15. Service Decomposition in ITSI
15
CLICK (open in new tab)
“Buttercup Games
Online Store”

16. Service Decomposition in ITSI
16
Identify a high-value business service

17. Service Decomposition in ITSI
17
Identify the process flow and underlying sub-services
(Web -> Middleware -> DB -> Middleware -> Web)

18. Service Decomposition in ITSI
18
For each sub-service, identify
KPIs that will show health and
status
(Requests, response time,
errors, OS health…)

19. Service Decomposition in ITSI
19
For each KPI,
define a
Splunk search

20. Service Decomposition in ITSI
20

21. ITSI Demo
21

22. Service Decomp: The Business Processes
22

23. Service Decomp: End-To-End Process Flow
23

24. New Requirements!
24
● Create a new KPI for the DB Service:
● Network Utilization
● Modify the Executive Glass Table
in order to show off the services
you slave over
“WE only have about 15min
TO DO WHAT ???!!???”
Think about how long this
would take you today?

25. 25
Configuration of DB Service
Click Configure >
Click Services

26. A KPI in 5 minutes? Absolutely!
26
Click New – Generic KPI
Select Data Model
● Host Operating System
● Network
● # bytes
● Next

27. KPIs Continued….
27
Splunk Builds Searches for you –
Oh Yeah, that’s happening J
● Select Yes for Split by & Filter options
● Select host for Entity Lookup & Alias options
● Click Next

28. Almost There…
28
Select
● KPI Search Schedule: Every Minute
● Entity Calculation: Average
● Service/Agg Calculation: Average
● Calculation Window: Last Minute
● Click Next
● Unit: Bps
● Click Next

29. Final Steps …
29
Set your thresholds:
● Aggregate (All)
● Per Entity
● Click “Add Threshold” TWICE
● Make the Neapolitan ice cream colors
Yellow, Green, Yellow
● Drag the sliders around in order to get
the current data graph entirely inside the
Green (normal) band
● Click Finish
● Other options are also available,
including adaptive thresholds and
anomaly detection

30. Adaptive Thresholds
30
What if your KPI data looks like this?

31. 31
Adaptive Thresholds
Static thresholds will not work…

32. 32
Adaptive Thresholds
Adaptive Thresholding works beautifully with cyclical (and other dynamic) data

33. Anomaly Detection
33
● Machine Learning
● Works well for data with patterns
● Requires some “training” (trial & error)
to zero in on best sensitivity

34. ITSI Demo –
Troubleshooting
34

35. Name that KPI!
35
From the list of KPIs, select your new one (at the bottom)
● Click on the little pencil next to the name
● Call it “Network Utilization”,
with your username up front
● Click on Save at bottom right when finished!

36. Let’s Fix that Glass Table
36

37. Clone the Glass Table
37
Return to Saved Glass Tables page
(click on Glass Tables in the upper menu bar)
CLICK Edit for “Buttercup Games Business Process (IN
PROGRESS)”
• Select Clone
• Title: Add your username
to the front
• Permissions: Shared in App
• Click Clone Page
• Click on your new Glass Table
from the list, to view it

38. Edit & Have Fun!
38
Click on Edit in the upper right corner of your Glass Table
Use the “Services” panel on the left to select Individual KPIs,
or Aggregate Service Health Scores
• Choose 2 KPIs from Online Store that would be useful in
the “Order Process” section
• Drag the selected widgets onto the canvas, positioning in
the gray oval
• What’s the difference between the
and tools at the top left?

39. More Fun with the Glass Table Editor…
39
Use the Configurations panel on the right to edit a
selected widget
• Can change the visualization type, drilldown
behavior, and other settings
• You should hit Save frequently
• I wonder what Auto Layout does?
• (YIKES!) Revert All Changes might be helpful

40. Finishing up …
40
• Add a ServiceHealthScore widget for Online
Store under Buttercup
• Choose a Viz Type with a sparkline graph, then
resize to make it look pretty
• Modify the Custom Drilldown action to go to
the saved glass table,
Buttercup Games Online Store
• Bonus Points: Make the label bigger, more
readable
• Click Save
• View when done

41. A Troubleshooting Exercise
41
Let’s use ITSI to troubleshoot an outage
● Start at your Glass Table, “<UserName> Buttercup Business Process”
● Customer Care reports that unhappy customers are complaining of failures
and long delays when trying to purchase
● The calls began coming in at around ten minutes after the hour.
● In the upper right corner of the Glass Table, change the time picker from Now
to XX:10:00.0, where XX is the appropriate hour. For example, if it is currently
14:05, set the time picker to 13:10:00.0, then Apply
● This is how we can “time travel” back to see conditions at a particular
outage– oh yeah!

42. A Troubleshooting Exercise, cont’d
42
● The Online Store seems to be degraded, just as Customer Care reported.
Click on the widget under Buttercup to drill down further

43. A Troubleshooting Exercise, cont’d.
43
● The Online Store Glass Table shows a much more detailed view, including the impacted customer-facing KPIs
at the far left (Revenue, etc)
● Based on this view of all the relevant
services, where do you think the root cause
lies?
● Which service should we troubleshoot first?
● Click on Health widget for that service, to
drill down to a Deep Dive

44. Deep Dive
44
● Deep Dive shows multiple KPIs and Health
Scores in parallel “swim lanes”.
● The Health Score for this Service is the top
swim lane. Can you see when it begins to
degrade from 100%?
● Mousing over this point in time, can you
spot the KPI with the leading fault
indication, i.e., what failed first?

45. Multi-KPI Alerts and Notable Events
45
● Click on Notable Events Review
● Multiple KPIs and Healthscores can
be combined in sophisticated ways
to create Multi-KPI alerts
● When a Multi-KPI alert fires, one
of the outcomes is the creation of
a Notable Event
● Notable Events allow NOC
personnel and others to triage and
coordinate event management
efforts

46. Service Analyzer
46
● Click on Service Analyzer > Default Service Analyzer
● Back where we started!
● This view shows a “no-frills” list of
services (top) and hottest KPIs
(bottom)
● Provides a quick jumping off point
into Deep Dives and the Notable
Events Review
● It is useful for NOCs and others
who need a high-level situational
view

47. Wrap Up - Review
47
● High-value services can be decomposed and modeled in ITSI, using machine data
from the relevant systems
● Services and KPIs can be created in minutes, with sophisticated thresholding
techniques to distinguish “normal” from “not normal”
● Glass Tables allow service health and KPI metrics to be displayed in a way that
makes sense to specific groups, such as Executive Leadership, Business Service
Owners, the NOC, DevOps & Others
● Deep Dives allow KPIs to be compared side-by-side across any time range,
accelerating root cause analysis and significantly reducing MTTR
● Multi-KPI Alerts and Notable Events reduce alert noise, producing actionable
events and a means to manage them
● … and it’s fun to build!

48. Want to explore on your own?
48
Sign up for your very own seven-day free sandbox!
http://splunk.com/ITSI
Then click:
You’ll find a Sandbox Guide in the Dashboards!
In the ITSI app of your sandbox, go to Search > Dashboards > ITSI Sandbox Guide

49. Northern Cal Tech Talks!
Monthly WebEx Sessions
• Ted Talk style presentation
• Q&A Chat forum
So what’s next on the agenda?
• April 20th @ 10AM PST - Top 5 most useful
search commands.
• May 18th @ 10AM PST – Splunk for IT
Service Intelligence
See more at:
http://live.splunk.com/NorCalTechTalks

50. 50
SEPT 26-29, 2016
WALT DISNEY WORLD, ORLANDO
SWAN AND DOLPHIN RESORTS
• 5000+ IT & Business Professionals
• 3 days of technical content
• 165+ sessions
• 80+ Customer Speakers
• 35+ Apps in Splunk Apps Showcase
• 75+ Technology Partners
• 1:1 networking: Ask The Experts and Security
Experts, Birds of a Feather and Chalk Talks
• NEW hands-on labs!
• Expanded show floor, Dashboards Control
Room & Clinic, and MORE!
The 7th Annual Splunk Worldwide Users’ Conference
PLUS Splunk University
• Three days: Sept 24-26, 2016
• Get Splunk Certified for FREE!
• Get CPE credits for CISSP, CAP, SSCP
• Save thousands on Splunk education!

51. A flying start to Service Intelligence
Start With A problem worth solving
Collaborate with Subject Matter Experts
Design Before Configuring

52. Sign Up Here - We’re Here To Help!
Harness the creativity and domain knowledge of your organization to unlock
the value of data and solve an important service problem through a joint
service intelligence workshop with key stakeholders
Define methods for:
• Proactive service monitoring
• Reduced risk and failures
• Faster issue resolution
• Increased business
performance
What is it?
• 1 Day Onsite Workshop
• Tightly linked with value
• Collaborative approach
• Build your own Splunk
ITSI Glass Table……

53. Thank You