SplunkLive! London 2016 Get your service intelligence off to a flying start
•
0 likes•314 views
IT Ops Track Session, hands on
Recommended
More Related Content
What's hot
What's hot (19)
Viewers also liked
Viewers also liked (13)
Similar to SplunkLive! London 2016 Get your service intelligence off to a flying start
Similar to SplunkLive! London 2016 Get your service intelligence off to a flying start (20)
More from Splunk
More from Splunk (20)
Recently uploaded
Recently uploaded (20)
SplunkLive! London 2016 Get your service intelligence off to a flying start
- 2. Setup Before You Can Play Download this presentation slide deck: https://splunk.box.com/v/ITSI-HandsOn Follow the instructions on your paper hand-out to log in to your VM. Please log in as either • user1@buttercupgms.com OR • user2@buttercupgms.com • Password is “Changeme1” or “Changeme2” After logging in, select IT Service Intelligence from the list of apps at the left 2
- 4. What is a Service? Service Requests Responses In ITSI, a Service is a logical group of technology components that a user deems need to be monitored together. It can often be generalized as a “black box” which we send requests, and expect responses 4
- 11. ITSI Tour 11
- 17. Service Decomposition (Refresher) 17 1 - What is a high-value business service? (Online Store) 2- Process flow, and underlying sub-services? (Web -> Middleware …) 3- For each (sub)service: KPIs to show health & status? (Database: errors, SQL hits, response time, …)
- 18. Service Decomposition (Refresher) 18 1 - What is a high-value business service? (Online Store) 2- Process flow & underlying sub-services? (Web -> Middleware …) 3- For each (sub)service: KPIs? (Database: errors, SQL hits, …) 4- For each KPI: Need a Splunk search (index=DB (warn* OR error*) | stats count)
- 19. Service Decomposition (Refresher) 19 1 - What is a high-value business service? (Online Store) 2- Process flow & underlying sub-services? (Web -> Middleware …) 3- For each (sub)service: KPIs? (Database: errors, SQL hits, …) 4- For each KPI: Need a Splunk search (index=DB (warn* OR error*) | stats count)
- 22. New Requirements! 22 ● Create a new KPI for the DB Service: ● Network Utilization ● Modify the Executive Glass Table in order to show off the services you slave over “WE only have about 15min TO DO WHAT ???!!???” Think about how long this would take you today?
- 24. Let’s Talk Entities 24 ● Select DB Service ● Entities are the relevant things which support this service (usually hosts) ● Select the right entries with filters, ANDs, ORs ● Original Entity list can come from CMDB, spreadsheet, Splunk search, others
- 27. Now start 27 Select Data Model ● Host Operating System ● Network ● # bytes ● Next
- 28. KPI Continued…. 28 Splunk Builds Searches for you – Oh Yeah, that’s happening J ● Select Yes for Split by & Filter options ● Select host for Entity Lookup & Alias options ● Click Next
- 29. Almost There… 29 Select ● KPI Search Schedule: Every Minute ● Entity Calculation: Average ● Service/Agg Calculation: Average ● Calculation Window: Last Minute ● Next ● Unit: Bps ● Next
- 30. Final Steps … 30 Set your thresholds ● Aggregate (All) ● Per Entity ● Click “Add Threshold” TWICE ● Make the Neapolitan ice cream colors Yellow, Green, Yellow ● Drag the sliders around in order to get the current data graph entirely inside the Green (normal) band ● Finish ● Other options are also available, including adaptive thresholds and anomaly detection
- 34. Anomaly Detection 34 ● Machine Learning ● Works well for data with patterns ● Requires some “training” (trial & error) to zero in on best sensitivity ● More sophisticated capabilities coming! (multivariate, more algorithms, etc)
- 36. Clone the Glass Table 36 Return to Saved Glass Tables page (click on Glass Tables in the upper menu bar) CLICK Edit for “Buttercup Games Business Process” • Select Clone • Title: Add your username to the front • Permissions: Shared in App • Clone Page • Click on your new Glass Table from the list, to view it
- 37. Edit & Have Fun! 37 Click on Edit in the upper right corner of your Glass Table Use the “Services” panel on the left to select Individual KPIs, or Aggregate Service Health Scores • Choose 2 KPIs from Online Store that would be useful in the “Order Process” section • Drag the selected widgets onto the canvas, positioning in the gray oval • What’s the difference between the and tools at the top left?
- 38. More Fun with the Glass Table Editor… 38 Use the Configurations panel on the right to edit a selected widget • Can change the visualization type, drilldown behavior, and other settings • You should hit Save frequently • I wonder what Auto Layout does? • (YIKES!) Revert All Changes might be helpful
- 39. Finishing up … 39 • Add a ServiceHealthScore widget for Online Store under Buttercup • Choose a Viz Type with a sparkline graph, then resize to make it look pretty • Modify the Custom Drilldown action to go to the saved glass table, Buttercup Games Online Store • Bonus Points: Make the label bigger, more readable • Save • View when done
- 40. A Troubleshooting Exercise 40 Let’s use ITSI to troubleshoot an outage ● Start at your Glass Table, “<UserName> Buttercup Business Process” ● Customer Care reports that unhappy customers are complaining of failures and long delays when trying to purchase ● The calls began coming in at around 40 minutes after the (previous) hour. ● In the upper right corner of the Glass Table, change the time picker from Now to XX:40:00.0, where XX is the previous hour. For example, if it is currently 14:05, set the time picker to 13:40:00.0, then Apply ● This is how we can “time travel” back to see conditions at a particular outage– oh yeah!
- 42. A Troubleshooting Exercise, cont’d 42 ● The Online Store Glass Table shows a much more detailed view, including the impacted customer-facing KPIs at the far left (Revenue, etc) ● Based on this view of all the relevant services, where do you think the root cause lies? ● Which service should we troubleshoot first? ● Click on Health widget for that service, to drill down to a Deep Dive
- 43. Deep Dive 43 ● Deep Dive shows multiple KPIs and Health Scores in parallel “swim lanes”. The initial time span shown is 15 minutes. ● The Health Score for this DB Service is the top swim lane. Can you see when it begins to degrade from 100%? ● Mousing over this point in time, can you spot the KPI with the leading fault indication? I.e., what busted first? ● To improve readability, change the Primary Time Range (lower left corner) to Presets > Last 60 minutes
- 44. Multi-KPI Alerts and Notable Events 44 ● Click on Notable Events Review ● Multiple KPIs and Healthscores can be combined in sophisticated ways to create Multi-KPI alerts ● When a Multi-KPI alert fires, one of the outcomes is the creation of a Notable Event ● Notable Events allow NOC personnel and others to triage and coordinate event management efforts
- 45. Service Analyzer 45 ● Click on Service Analyzer > Default Service Analyzer ● Back where we started! ● This view shows a “no-frills” list of services (top) and hottest KPIs (bottom) ● Provides a quick jumping off point into Deep Dives and the Notable Events Review ● It is useful for NOCs and others who need a high-level situational view
- 46. Review 46 ● High-value services can be decomposed and modeled in ITSI, using machine data from the relevant systems ● Services and KPIs can be created in minutes, with sophisticated thresholding techniques to distinguish “normal” from “not normal” ● Glass Tables allow service health and KPI metrics to be displayed in a way that makes sense to specific groups, such as Executive Leadership, Business Service Owners, the NOC, DevOps & Others ● Deep Dives allow KPIs to be compared side-by-side across any time range, accelerating root cause analysis and significantly reducing MTTR ● Multi-KPI Alerts and Notable Events reduce alert noise, producing actionable events and a means to manage them ● … and it’s fun to build!
- 47. PLAY TIME IS OVER! Everyone out of the sandbox! 47 NOT! You can have your very own 15-day free eval sandbox, to continue playing: ● http://splunk.com/ITSI Then select: And a Guidebook to help you explore ITSI’s capabilities: ● https://splunk.box.com/ITSI-Sandbox-Guidebook
- 48. 48 SEPT 26-29, 2016 WALT DISNEY WORLD, ORLANDO SWAN AND DOLPHIN RESORTS • 5000+ IT & Business Professionals • 3 days of technical content • 165+ sessions • 80+ Customer Speakers • 35+ Apps in Splunk Apps Showcase • 75+ Technology Partners • 1:1 networking: Ask The Experts and Security Experts, Birds of a Feather and Chalk Talks • NEW hands-on labs! • Expanded show floor, Dashboards Control Room & Clinic, and MORE! The 7th Annual Splunk Worldwide Users’ Conference PLUS Splunk University • Three days: Sept 24-26, 2016 • Get Splunk Certified for FREE! • Get CPE credits for CISSP, CAP, SSCP • Save thousands on Splunk education!