IT Service Intelligence Hands On Breakout Session
•
4 likes•935 views
This document provides an overview of using IT Service Intelligence (ITSI) to model, monitor, and troubleshoot IT services. It discusses how to decompose high-level business services into underlying technical services and KPIs. It demonstrates how to quickly create a new KPI and customize a glass table dashboard. It also shows how to use ITSI for root cause analysis of an outage, including deep diving on services and reviewing notable events. The document encourages participants to continue exploring ITSI capabilities on their own sandbox.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (8)
Similar to IT Service Intelligence Hands On Breakout Session
Similar to IT Service Intelligence Hands On Breakout Session (20)
More from Splunk
More from Splunk (20)
Recently uploaded
Recently uploaded (20)
IT Service Intelligence Hands On Breakout Session
- 1. Copyright © 2015 Splunk Inc. IT Service Intelligence The Hands-On Version David Millis Staff Architect, IT Operations Analytics
- 2. Setup Before You Can Play Download this presentation slide deck: https://splunk.box.com/ITSI-HandsOn Follow the instructions on your paper hand-out to log in to your VM. Please log in as either • user1@buttercupgms.com OR • user2@buttercupgms.com • Password is “Changeme1” or “Changeme2” After logging in, select IT Service Intelligence from the list of apps at the left 2
- 4. What is a Service? Service Requests Responses In ITSI, a Service is a logical group of technology components that a user deems need to be monitored together. It can often be generalized as a “black box” which we send requests, and expect responses 4
- 5. What is a Service? DNS Requests Responses Technical Services Auth Requests Responses Web Requests Responses Services can be lower level (technical) … 5
- 6. What is a Service? DNS Requests Responses Technical Services Customer Transactions Requests Responses Business Services Auth Requests Responses Web Requests Responses Support Desk Requests Responses Services can also be higher level (business) … 6
- 7. What is a Service? Packet Network Hypervisor and Hosts RBMDBs Storage Tier API Services Web Services CustomerTransactions Mobile API/Middleware PartnerPortal DNS Services can encompass multiple tiers of the IT domain. Services may also depend upon other services 7
- 8. What is a KPI? DNS Requests Responses KPI: Number of requests KPI: Error rate KPI: Average response time KPI: Servicer CPU load KPI: Server network I/F errors Customer Transactions Requests Responses KPI: Number of transactions KPI: Error rate KPI: Average response time KPI: Count of Incident Tickets KPI: Synthetic Transx Health KPIs and Health scores constitute the means by which Services are monitored. 8
- 9. Key Performance Indicators (KPIs) 9 A Key Performance Indicator (KPI) is a Splunk saved search created within the ITSI UI that helps monitor a specific field like CPU, Memory, Number of Errors and so on. KPIs are contained within Services.
- 10. Service Health Scores 10 A Health score is a score form 0-100 (0 being critical and 100 being normal) that helps determine the health of a Service. It is calculated based on all KPIs importance and its status (e.g. green, orange, red), once every minute.
- 11. ITSI Tour 11
- 12. Service Decomposition in ITSI 12 CLICK “Glass Tables”
- 13. Service Decomposition in ITSI 13 CLICK (open in new tab) “Buttercup Games Business Process”
- 14. Service Decomposition in ITSI 14 CLICK (open in new tab) “Buttercup Games Online Store”
- 15. Service Decomposition (Refresher) 15 1 - What is a high-value business service? (“Online Store” in Buttercup Games)
- 16. Service Decomposition (Refresher) 16 1 - What is a high-value business service? (Online Store) 2- Process flow, and underlying sub-services? (Web -> Middleware -> DB -> Middleware -> Web)
- 17. Service Decomposition (Refresher) 17 1 - What is a high-value business service? (Online Store) 2- Process flow, and underlying sub-services? (Web -> Middleware …) 3- For each (sub)service: KPIs to show health & status? (Database: errors, SQL hits, response time, …)
- 18. Service Decomposition (Refresher) 18 1 - What is a high-value business service? (Online Store) 2- Process flow & underlying sub-services? (Web -> Middleware …) 3- For each (sub)service: KPIs? (Database: errors, SQL hits, …) 4- For each KPI: Need a Splunk search (index=DB (warn* OR error*) | stats count)
- 19. Service Decomposition (Refresher) 19 1 - What is a high-value business service? (Online Store) 2- Process flow & underlying sub-services? (Web -> Middleware …) 3- For each (sub)service: KPIs? (Database: errors, SQL hits, …) 4- For each KPI: Need a Splunk search (index=DB (warn* OR error*) | stats count)
- 20. Service Decomp: The Business Processes 20
- 21. Service Decomp: End-To-End Process Flow 21
- 22. New Requirements! 22 ● Create a new KPI for the DB Service: ● Network Utilization ● Modify the Executive Glass Table in order to show off the services you slave over “WE only have about 15min TO DO WHAT ???!!???” Think about how long this would take you today?
- 23. 23 Configuration of DB Service Click Configure > Click Services
- 24. Let’s Talk Entities 24 ● Select DB Service ● Entities are the relevant things which support this service (usually hosts) ● Select the right entries with filters, ANDs, ORs ● Original Entity list can come from CMDB, spreadsheet, Splunk search, others
- 25. A KPI in 5 minutes? Absolutely! 25 Click New – Generic KPI Select Data Model ● Host Operating System ● Network ● # bytes ● Next
- 26. KPI Continued…. 26 Splunk Builds Searches for you – Oh Yeah, that’s happening ● Select Yes for Split by & Filter options ● Select host for Entity Lookup & Alias options ● Click Next
- 27. Almost There… 27 Select ● KPI Search Schedule: Every Minute ● Entity Calculation: Average ● Service/Agg Calculation: Average ● Calculation Window: Last Minute ● Next ● Unit: Bps ● Next
- 28. Final Steps … 28 Set your thresholds ● Aggregate (All) ● Per Entity ● Click “Add Threshold” TWICE ● Make the Neapolitan ice cream colors Yellow, Green, Yellow ● Drag the sliders around in order to get the current data graph entirely inside the Green (normal) band ● Finish ● Other options are also available, including adaptive thresholds and anomaly detection
- 29. Name that KPI! 29 From the list of KPIs, select your new one (at the bottom) ● Click on the little pencil next to the name ● Call it “Network Utilization”, with your username up front ● Click on Save at bottom right when finished!
- 30. Adaptive Thresholds 30 What if your KPI data looks like this?
- 31. 31 Adaptive Thresholds Static thresholds will not work…
- 32. 32 Adaptive Thresholds Adaptive Thresholding works beautifully with cyclical data
- 33. Anomaly Detection 33 ● Machine Learning ● Works well for data with patterns ● Requires some “training” (trial & error) to zero in on best sensitivity ● More sophisticated capabilities coming! (multivariate, more algorithms, etc)
- 34. Let’s Fix that Glass Table 34
- 35. Clone the Glass Table 35 Return to Saved Glass Tables page (click on Glass Tables in the upper menu bar) CLICK Edit for “Buttercup Games Business Process” • Select Clone • Title: Add your username to the front • Permissions: Shared in App • Clone Page • Click on your new Glass Table from the list, to view it
- 36. Edit & Have Fun! 36 Click on Edit in the upper right corner of your Glass Table Use the “Services” panel on the left to select Individual KPIs, or Aggregate Service Health Scores • Choose 2 KPIs from Online Store that would be useful in the “Order Process” section • Drag the selected widgets onto the canvas, positioning in the gray oval • What’s the difference between the and tools at the top left?
- 37. More Fun with the Glass Table Editor… 37 Use the Configurations panel on the right to edit a selected widget • Can change the visualization type, drilldown behavior, and other settings • You should hit Save frequently • I wonder what Auto Layout does? • (YIKES!) Revert All Changes might be helpful
- 38. Finishing up … 38 • Add a ServiceHealthScore widget for Online Store under Buttercup • Choose a Viz Type with a sparkline graph, then resize to make it look pretty • Modify the Custom Drilldown action to go to the saved glass table, Buttercup Games Online Store • Bonus Points: Make the label bigger, more readable • Save • View when done
- 39. A Troubleshooting Exercise 39 Let’s use ITSI to troubleshoot an outage ● Start at your Glass Table, “<UserName> Buttercup Business Process” ● Customer Care reports that unhappy customers are complaining of failures and long delays when trying to purchase ● The calls began coming in at around 40 minutes after the (previous) hour. ● In the upper right corner of the Glass Table, change the time picker from Now to XX:40:00.0, where XX is the previous hour. For example, if it is currently 14:05, set the time picker to 13:40:00.0, then Apply ● This is how we can “time travel” back to see conditions at a particular outage– oh yeah!
- 40. A Troubleshooting Exercise, cont’d 40 ● The Online Store seems to be degraded, just as Customer Care reported. Click on the widget under Buttercup to drill down further
- 41. A Troubleshooting Exercise, cont’d 41 ● The Online Store Glass Table shows a much more detailed view, including the impacted customer-facing KPIs at the far left (Revenue, etc) ● Based on this view of all the relevant services, where do you think the root cause lies? ● Which service should we troubleshoot first? ● Click on Health widget for that service, to drill down to a Deep Dive
- 42. Deep Dive 42 ● Deep Dive shows multiple KPIs and Health Scores in parallel “swim lanes”. The initial time span shown is 15 minutes. ● The Health Score for this DB Service is the top swim lane. Can you see when it begins to degrade from 100%? ● Mousing over this point in time, can you spot the KPI with the leading fault indication? I.e., what busted first? ● To improve readability, change the Primary Time Range (lower left corner) to Presets > Last 60 minutes
- 43. Multi-KPI Alerts and Notable Events 43 ● Click on Notable Events Review ● Multiple KPIs and Healthscores can be combined in sophisticated ways to create Multi-KPI alerts ● When a Multi-KPI alert fires, one of the outcomes is the creation of a Notable Event ● Notable Events allow NOC personnel and others to triage and coordinate event management efforts
- 44. Service Analyzer 44 ● Click on Service Analyzer > Default Service Analyzer ● Back where we started! ● This view shows a “no-frills” list of services (top) and hottest KPIs (bottom) ● Provides a quick jumping off point into Deep Dives and the Notable Events Review ● It is useful for NOCs and others who need a high-level situational view
- 45. Review 45 ● High-value services can be decomposed and modeled in ITSI, using machine data from the relevant systems ● Services and KPIs can be created in minutes, with sophisticated thresholding techniques to distinguish “normal” from “not normal” ● Glass Tables allow service health and KPI metrics to be displayed in a way that makes sense to specific groups, such as Executive Leadership, Business Service Owners, the NOC, DevOps & Others ● Deep Dives allow KPIs to be compared side-by-side across any time range, accelerating root cause analysis and significantly reducing MTTR ● Multi-KPI Alerts and Notable Events reduce alert noise, producing actionable events and a means to manage them ● … and it’s fun to build!
- 46. PLAY TIME IS OVER! Everyone out of the sandbox! 46 NOT! You can have your very own 15-day free eval sandbox, to continue playing: ● http://splunk.com/ITSI Then select: And a Guidebook to help you explore ITSI’s capabilities: ● https://splunk.box.com/ITSI-Sandbox-Guidebook
- 47. Northern Cal Tech Talks! Monthly WebEx Sessions • Ted Talk style presentation • Q&A Chat forum So what’s next on the agenda? • March 23rd @ 10AM PST - Building & Deploying Apps. • April 20th @ 10AM PST - Top 5 most useful search commands. See more at: http://live.splunk.com/NorCalTechTalks
- 48. 48 SEPT 26-29, 2016 WALT DISNEY WORLD, ORLANDO SWAN AND DOLPHIN RESORTS • 5000+ IT & Business Professionals • 3 days of technical content • 165+ sessions • 80+ Customer Speakers • 35+ Apps in Splunk Apps Showcase • 75+ Technology Partners • 1:1 networking: Ask The Experts and Security Experts, Birds of a Feather and Chalk Talks • NEW hands-on labs! • Expanded show floor, Dashboards Control Room & Clinic, and MORE! The 7th Annual Splunk Worldwide Users’ Conference PLUS Splunk University • Three days: Sept 24-26, 2016 • Get Splunk Certified for FREE! • Get CPE credits for CISSP, CAP, SSCP • Save thousands on Splunk education!
- 49. Thank You David Millis Staff Architect, IT Operations Analytics
Editor's Notes
- A services can literally be sources of data a customer wants to group together to monitor in a single healthscore or just wants to logically group together because they need to be managed by a specific team or needs to be reported in such a way. Services derive their value when KPIs are defined within them or dependencies are defined to other services. Therefore you could have a more abstractly defined service which only depends on other services to derive its own health. E.g. Partner portal is a conceptual service which depends on the API service which in turn has its own KPIs but depends on Web Services. Alternatively you could have Partner portal depend on each and every one in blue, or not even have all the ones in blue and have all the Kpis be inside Partner portal. Everything you see in the diagram above could be a service in ITSI.
- A services can literally be sources of data a customer wants to group together to monitor in a single healthscore or just wants to logically group together because they need to be managed by a specific team or needs to be reported in such a way. Services derive their value when KPIs are defined within them or dependencies are defined to other services. Therefore you could have a more abstractly defined service which only depends on other services to derive its own health. E.g. Partner portal is a conceptual service which depends on the API service which in turn has its own KPIs but depends on Web Services. Alternatively you could have Partner portal depend on each and every one in blue, or not even have all the ones in blue and have all the Kpis be inside Partner portal. Everything you see in the diagram above could be a service in ITSI.
- A services can literally be sources of data a customer wants to group together to monitor in a single healthscore or just wants to logically group together because they need to be managed by a specific team or needs to be reported in such a way. Services derive their value when KPIs are defined within them or dependencies are defined to other services. Therefore you could have a more abstractly defined service which only depends on other services to derive its own health. E.g. Partner portal is a conceptual service which depends on the API service which in turn has its own KPIs but depends on Web Services. Alternatively you could have Partner portal depend on each and every one in blue, or not even have all the ones in blue and have all the Kpis be inside Partner portal. Everything you see in the diagram above could be a service in ITSI.
- A services can literally be sources of data a customer wants to group together to monitor in a single healthscore or just wants to logically group together because they need to be managed by a specific team or needs to be reported in such a way. Services derive their value when KPIs are defined within them or dependencies are defined to other services. Therefore you could have a more abstractly defined service which only depends on other services to derive its own health. E.g. Partner portal is a conceptual service which depends on the API service which in turn has its own KPIs but depends on Web Services. Alternatively you could have Partner portal depend on each and every one in blue, or not even have all the ones in blue and have all the Kpis be inside Partner portal. Everything you see in the diagram above could be a service in ITSI.
- A services can literally be sources of data a customer wants to group together to monitor in a single healthscore or just wants to logically group together because they need to be managed by a specific team or needs to be reported in such a way. Services derive their value when KPIs are defined within them or dependencies are defined to other services. Therefore you could have a more abstractly defined service which only depends on other services to derive its own health. E.g. Partner portal is a conceptual service which depends on the API service which in turn has its own KPIs but depends on Web Services. Alternatively you could have Partner portal depend on each and every one in blue, or not even have all the ones in blue and have all the Kpis be inside Partner portal. Everything you see in the diagram above could be a service in ITSI.
- KPIs are created by the user and the user has to define which Splunk field to monitor, what stat operations to use (e.g. avg cpu, max cpu etc.), what the thresholds for good bad ugly should be, what the frequency of monitoring that field should be and how important it is towards the health of the service. Images: KPIs on the left, health scores on the right.
- KPIs are created by the user and the user has to define which Splunk field to monitor, what stat operations to use (e.g. avg cpu, max cpu etc.), what the thresholds for good bad ugly should be, what the frequency of monitoring that field should be and how important it is towards the health of the service. Images: KPIs on the left, health scores on the right.
- Describe how GTs can show KPIs & health scores to any audience: Show GTs: Buttercup Games Business Process (executives, business service owners) On Line Transaction Service (NOC, Tier2); can use visio diagrams… Buttercup Games Online Store (service flow, sub-services) Show saved Deep Dive “DB Deep Dive”; briefly describe DD functionality Show Notable Event Review, briefly describe Show Service Analyzer, briefly describe
- These actions set the student up for decomp discussion, later
- These actions set the student up for decomp discussion, later
- These actions set the student up for decomp discussion, later
- In the “real world”, it will probably be necessary to iterate up & down these steps a few times. For example, what if a KPI requires data which is not being collected by Splunk?
- This Glass Table shows the high-level business process for Buttercup Games. We need better visibility into our Online Store, which is part of the Order Entry process.
- This Glass Table shows a more detailed process flow for the Online Store service.
- Based on a recent DB outage which was caused by a saturated network interface, we’ve decided that network utilization would be a handy KPI for our Database Service. We’re also going to tweak the high-level Business Process Glass Table to provide more visibility into the Online Store service. And we’ve only got 15 minutes to do it!
- SHORT discussion of entities
- So is it fair to say that YOU, the business teams and YOU, the technical teams can now use these Glass Tables to help each other start defining KPI’s that interrelate, correlate and integrate the technology and Business Micro Services that make up the Many Services in an Environment? Show of hands Business Team who believes this can happen? Show of hands Technical Teams who believes this can happen? YES they CAN and YES it will actually happen both of the teams can and WILL start helping one another to identify Entities, KPI’s and Services? So lets build on that excitement for a second. What is the typical answer the Business gets from the Technical Teams as to how long something will take to build?
- "Many Quarters and realistically a Year on the conservative side right?" To quantify that, Show of hands, anyone here been involved in a IT Service Management / Business Management team trying to map every Server to a Service or Business Function? Net of that LONG Conversation it is not fun nor pretty and Job Longevity - Yeah as soon as you finish you have to start over. So why ITSI - Because this is “EASY" with ITSI and we NEED the tech teams to let the Business Teams know that it can happen in weeks but, even more important we NEED the Business Teams to let the Technical Teams know it can happen in weeks… But just like the infomercials, but wait there's more …
- This might take a while for “waiting for data” to produce an actual graph for the students. Be prepared to kill some time, and if necessary, punt out of this step if the AWS instances are too slow.
- How long did it take to make this KPI?
- Talk through for SplunkLive-- NOT HANDS ON
- Talk through for SplunkLive-- NOT HANDS ON
- Talk through for SplunkLive-- NOT HANDS ON
- Talk through NOT WORK
- We’ve already discussed the high-level business process for Buttercup Games. We need better visibility into our Online Store, which is part of the Order Entry process.
- We’ve already discussed the high-level business process for Buttercup Games. We need better visibility into our Online Store, which is part of the Order Entry process.
- We’ve already discussed the high-level business process for Buttercup Games. We need better visibility into our Online Store, which is part of the Order Entry process.
- We’ve already discussed the high-level business process for Buttercup Games. We need better visibility into our Online Store, which is part of the Order Entry process.
- If pressed for time, talk through and show this process– but don’t have the students attempt to “click along” in real time
- If pressed for time, talk through and show this process– but don’t have the students attempt to “click along” in real time
- If pressed for time, talk through and show this process– but don’t have the students attempt to “click along” in real time
- If pressed for time, talk through and show this process– but don’t have the students attempt to “click along” in real time
- If pressed for time, talk through and show this process– but don’t have the students attempt to “click along” in real time
- If pressed for time, talk through and show this process– but don’t have the students attempt to “click along” in real time
- Talk through NOT WORK
- We’re headed to the East Coast! 2 inspired Keynotes – General Session and Security Keynote + Super Sessions with Splunk Leadership in Cloud, IT Ops, Security and Business Analytics! 165+ Breakout sessions addressing all areas and levels of Operational Intelligence – IT, Business Analytics, Mobile, Cloud, IoT, Security…and MORE! 30+ hours of invaluable networking time with industry thought leaders, technologists, and other Splunk Ninjas and Champions waiting to share their business wins with you! Join the 50%+ of Fortune 100 companies who attended .conf2015 to get hands on with Splunk. You’ll be surrounded by thousands of other like-minded individuals who are ready to share exciting and cutting edge use cases and best practices. You can also deep dive on all things Splunk products together with your favorite Splunkers. Head back to your company with both practical and inspired new uses for Splunk, ready to unlock the unimaginable power of your data! Arrive in Orlando a Splunk user, leave Orlando a Splunk Ninja! REGISTRATION OPENS IN MARCH 2016 – STAY TUNED FOR NEWS ON OUR BEST REGISTRATION RATES – COMING SOON!