Splunk IT Service Intelligence Sandbox Guidebook
•
0 likes•1,547 views
Splunk IT Service Intelligence Sandbox Guidebook
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Splunk IT Service Intelligence Sandbox Guidebook
Similar to Splunk IT Service Intelligence Sandbox Guidebook (20)
More from Splunk
More from Splunk (20)
Recently uploaded
Recently uploaded (20)
Splunk IT Service Intelligence Sandbox Guidebook
- 3. 3 1 - Fly Over the Product For the traveller who is in a hurry, who wants the 30,000-foot view, this is the section for you! It is also the best place to begin, for the student who is largely unfamiliar with IT Service Intelligence. Instructions 1. After logging into Splunk, click on "Product Tour" 2. Click through the 11 slides to preview services, entities, KPIs, thresholding, Deep Dives, Multi-KPI Alerts, Notable Events and the Service Analyzer 3. These topics, and more, are covered in more detail in the following chapters
- 5. 5 Packet Network Hypervisor and Hosts RBMDBs Storage Tier API Services Web Services In ITSI, a Service is a logical group of KPIs which describe its health and status; they typically span mulGple IT domains IT Service Intelligence – Core Concepts Service Requests Responses Web Technical Services Services Customer Transac9ons Web Customer Transac9ons Requests Responses Business Services Mobile API/ Middleware Support Desk DNS
- 12. 12 ITSI represents a new way of dealing with IT Service challenges: • Data-driven approach uses ALL IT Data - events, metrics, logs, structured, unstructured, from-the-device, from-the-wire, etc. • Service-awareness provides actionable insights into high-visibility services • Customized contextual visualizations can be tailored for any person or group: highly technical to business-oriented • Mitigate problems before they impact customers
- 14. 14 3. Select "Business Status (Medium)" This Glass Table shows the "bottom line" status of an online store, including overall health, revenue and checkouts metrics. It could be used by service owners, executive management or others who need to quickly understand the "big picture". 4. Select "On Line Transaction Service" This Glass Table shows a detailed view of a customer-facing service, including transaction flow, component relationships and dependencies, and critical health scores and metrics of key service points along the way. It makes excellent use of a pre- existing drawing, with live ITSI "widgets" placed strategically on top. This Glass Table would helpful for NOC, Tier 1&2 and similar support personnel who need to understand the complex relationships of all the service components supporting an important business service. 5. Select "End to End Health (Medium)" This Glass Table shows a streamlined view of a customer-facing service-- the "online store" summarized in the "Business Status" Glass Table. This view provides more detail of the underlying technical services, their dependencies, and the overall transaction flow. It uses native Glass Table drawing tools, as well as service & KPI widgets which display health & metric values live (updating over time). These widgets have configurable drill-down capabilities, including the ability to navigate to other, even-more-detailed Glass Tables. For example, if you click on the widget next to "Web Tier", you will navigate to ... 6. "Web Tier (Medium)" This Glass Table represents a more detailed visualization of the KPIs, overall Web Tier healthscore, and the healthscore of its dependent service, "Middleware". Such Glass Tables allow technical personnel to quickly troubleshoot problems by being able to drill down to the detailed technical metrics which matter. 7. Select "End to End Health (Medium)" (again) Several drill-down options are available when a widget is clicked. Click on the widget next to "Database"; this will navigate to a Deep Dive.
- 16. 16 4 - Troubleshoot with Glass Tables and Deep Dives This section describes a possible problem scenario, and how ITSI could be used to efficiently troubleshoot to find root cause. This would typically be driven by a NOC or Tier 1/2 support person. We're going to "set up" the failure scenario and first see how Glass Tables can accelerate the troubleshooting process, then continue isolating root cause with Deep Dives. Pre-Requisites You should already be familiar with: • Core Concepts (Ch. 2) • Glass Tables (Ch. 3) About the event generator... In order to make the ITSI Sandbox more interesting to play in, an event generator is included which continuously generates a simulated stream of realistic machine events, including web access, database, linux metrics (from the *nix Technology Add-on App) and others. Included in this stream of events are a failure scenario, showing a sequence of failures and resulting service degradations, repeating approximately once per hour. Typically, the initial failures occur at about 30 minutes past the hour (XX:30), and reset back to "OK" around the top of the hour (XX:00). However, the event generator (eventgen) timing is not precise. The failure scenarios may occur at slightly different times from hour to hour, and may vary from sandbox to sandbox. Thus, within the Sandbox, it is impossible to predict exactly how the healthscores and KPIs will appear, during any specific hour. This makes it difficult to setup a "clean" failure simulation. Please pardon any eventgen inconsistencies. We decided to put most of our effort into developing ITSI-- not an event generator.
- 17. 17 Instructions 1. Navigate to the Glass Table called, "End to End Health (Medium)": a. Click on "Glass Tables" in the upper menu bar to navigate to the page, "Saved Glass Tables" b. Click on "End to End Health (Medium)" to navigate to this Glass Table As noted above, it is not possible to predict exactly when a failure scenario occurs within a particular past hour. However, interesting things usually occur around XX:40 to XX:50. 2. Modify the view time by clicking on the time picker in the upper right corner. In the pop-up window, type in an explicit time from the past, such as XX:50.0 from the previous hour (or the hour before that, etc). Be sure to use the correct HH:MM:SS.sss format (example: "22:50:00.0")
- 19. 19 5. The scenario: Customer Care has informed us that customers are calling to complain when they try to purchase through the Online Store; they are seeing slow response and occasional errors. The problems seem to be affecting both web-based and mobile-based customers. 6. Based on just the reports that the customer-facing web-based service is having problems, most support persons would begin troubleshooting "from the top"-- the web & mobile tiers in this case. If no obvious problems are found, they would proceed down the service dependency tree-- to the middleware tier, etc. 7. But using a Glass Table such as "End to End Health" provides instant and context-relevant visibility into service healthscores and important KPIs, all in one place. In the above example, which supporting tier seems to be in distress? (Database) By being able to visualize the relevant services and their health scores, we have the ability to immediately focus our troubleshooting on the areas which are degraded. This can save huge amounts of time and greatly reduce the time required to find root cause. 8. On your Sandbox Glass Table, click on the widget beneath "Database" to drill down into the Database tier to continue the troubleshooting exercise. (Select "Leave This Page" if prompted)
- 20. 20 (Now in "DB Deep Dive") 9. Click on the ">" next to "Focus" to collapse the service tree navigator panel on the right side; we will explore this feature later. 10. Change the Primary Time Range to "Last 90 minutes" by clicking on the time picker in the lower left corner: a. In the "Relative" section, type in "90" and select "Minutes Ago" b. Click on "Apply" (The Deep Dive now displays a couple of periods of "mostly green" and "mostly red")
- 21. 21 11. Position the pointer near the left side of a "mostly green" section, then click-hold and drag right to include most of a "mostly red" section as well. Release the mouse to zoom in to this time period. At this point, your Deep Dive should look similar to this: 12. We are looking at the Aggregated Health Score (top swimlane) and KPIs for the DB Service, across a range of time which shows the service moving from "healthy" to "not healthy". 13. Slowly mouse over the swim lanes to compare values at various points in time. 14. Toggle on/off the "Enable Threshold" in the upper left corner, to compare the swim lanes with and without the threshold colors/states overlaid. 15. Note that the ServiceHealthScore in the top swimlane is an aggregation of the service's KPIs and dependent services, ranging from 100-0. When did the healthscore begin to deteriorate, and which KPI(s) may have been part of the root cause?
- 22. 22 16. Click on the name-box for "DB Hosts Used Disk Space Percent", then drag it upwards to reposition this swimlane. 17. Mouseover the name-box for "DB Host CPU Load Percent" to reveal the "options wheel", then select it to view available options. Select "Delete Lane" to (temporarily) remove this swimlane from our Deep Dive. 18. Click on the darker blue tile within the "DB Errors" swimlane to reveal "raw errors" from the underlying Splunk search. Click on "Hide Events" to dismiss. 19. Mouseover the "DB Hosts Used Disk Space Percent" swimlane, in the place where it goes from green to red. Note the high & low metric values shown for the swimlane, and that this metric has gone to 100%, indicating that a filesystem is full. 20. Mouseover the name-box for "DB Hosts Used Disk Space Percent" to reveal the "options wheel", then select it to view available options. Select "Lane Overlay Options". When the popup appears, select "Enable Overlays - Yes", then "Done". (Several colored lines have now replaced the single black line in this swimlane)
- 24. 24 5 - Tour the Deep Dive Deep Dives allow KPI metrics and healthscores to be compared in side-by-side swimlanes, which allows trends and correlations to be more easily and quickly discovered. This chapter explores Deep Dives and how they can be used. Pre-Requisites You should already be familiar with: • Core Concepts (Ch. 2) • Troubleshooting (Ch. 4) also goes into Deep Dives Instructions 1. Navigate to the Deep Dive called, "DB Deep Dive": a. Click on "Deep Dives" in the upper menu bar to navigate to the page, "Saved Deep Dives" b. Click on "DB Deep Dive" to navigate to this Deep Dive 2. Click on the ">" next to "Focus" to collapse the service tree navigator panel on the right side; we will explore this feature later. 3. Select an arbitrary time range by clicking on the Primary Time Range menu option at the bottom right; it functions like a standard Splunk search bar time picker 4. Zoom in to a tighter time range in the current view by click-holding anywhere in the swimlanes, then dragging horizontally to select the range. 5. Toggle the threshold healthscore colors by clicking on "Enable Threshold" in the upper left corner. 6. Click on the "<" above "Focus" to open the service tree navigator panel on the right side. a. Click on a service node to navigate up and down the dependency tree of services b. After clicking on a service node, note that those service's KPIs are listed below. c. Click on the "+" on a listed KPI to add it to the current swimlanes d. Click on the ">" next to "Focus" to collapse the service tree navigator panel on the right side
- 25. 25 7. Mouseover the name-box for any swimlane to reveal the "options wheel", then select it to view available options: 8. The student is encouraged to explore these options, which are covered in more detail at http://docs.splunk.com/Documentation/ITSI/latest/User/DeepDives 9. Click-hold on the name-box for any swimlane, then drag it vertically to reposition this swimlane. 10. Click on the darker blue tile within the "DB Errors" (or any "event"-style) swimlane to reveal "raw errors" from the underlying Splunk search. Click on "Hide Events" to dismiss. 11. To save a Deep Dive after modifying the layout and/or visualization options, click on the "Edit" menu option in the upper right corner, then select "Save" 12. To compare the current time range against a different time range, click on "Compare to ..." in the lower left corner, then select a comparison time range. This causes each KPI to display twin swimlanes: primary time range above comparison time range. Note that when mousing over the swimlanes, the time display at the top now shows both times. 13. To dismiss the "twin" lanes display, deselect the checkbox next to "Compare to ..." in the lower left corner
- 26. 26 6 - Tour the Notable Event Viewer Multi-KPI Alerts are correlation searches which can combine any KPIs to create meaningful, actionable alerts, using multiple correlation factors such KPI threshold indications, length of time in this state, time-of-day, and others. Multi-KPI alerts can find not just "failures", but early "canary in the coal mine" indications that the service is becoming unstable; it is possible to find problems BEFORE they impact customer-facing services. When a Multi-KPI alert fires, it creates a Notable Event; it could also execute a script and/or send email. A Notable Event is like a "Poor man's trouble-ticket", useful for tactical triage of problems or potential problems. The Notable Event Viewer has the ability filter Notable Events based on various criteria, such as Severity, Status, Service and others. It also allows Notable Events to be modified, to change Owner, Severity, Status, and/or add comments. Notable events can also have workflow actions associated with them, to allow an operator the ability to quickly hit troubleshooting options, fire mitigation scripts, or open a "real" Incident Management trouble-ticket. Pre-Requisites You should already be familiar with: • Core Concepts (Ch. 2) Instructions 1. Navigate to the Notable Events Viewer by clicking on "Notable Events Review" in the upper menu bar 2. Filter the Notable Events by Severity: Click on a colored Severity bar to select/deselect that particular severity. Severities which have been deselected appear as gray. 3. Filter by Status, Owner, Service, Time Range, Name ("Title") or freeform search criteria by using the input options in the upper left quadrant. 4. Click "Submit" to (re)apply search filter criteria 5. Edit one or more Notable Events by selecting their checkboxes on the left side, then select "Edit all selected" a. Change Status, Severity or Owner b. Add freeform comments c. Click "Done" 6. View available workflow actions for a Notable Event by clicking the "V" under "Actions"
- 27. 27 7 - Tour the Service Analyzer The Service Analyzer is a "Big Picture" view of all services, and the "most interesting" KPIs (i.e., KPIs with degraded healthscores). It is "no frills", designed for NOCs, Tier 1/2 support, and others who need a high level view of all services/KPIs, or a subset. Pre-Requisites You should already be familiar with: • Core Concepts (Ch. 2) Instructions 1. Navigate to the Service Analyzer by clicking on " Service Analyzer " in the upper menu bar 2. Click on any service to drill down into a deep dive containing the KPIs for that service a. Notice the deep dive has been built for you on the fly, containing all the KPI’s associated with that service 3. Navigate back to Service Analyzer What if I only want to see a few services or KPIs? 4. Click in the "Select service(s) to monitor" box to select & show only certain services 5. Click on the "Option Wheel" next to "Top ... Services" to control how many services are shown 6. Click on the "Option Wheel" next to "Top ... KPIs" to control how many KPIs are shown, and to select which KPIs are shown I want to combine & drill down into "interesting" KPIs and Service Health Scores ... 7. To create an ad-hoc Deep Dive: a. Mouseover one or more Service or KPI tiles, then select the checkbox in the upper right corner of the tile (select at least three tiles) b. Click "Drilldown on Selection"