Service Intelligence hands on workshop
•
2 likes•499 views
The document discusses building IT service intelligence with Splunk. It introduces key concepts like services, KPIs, health scores, and the benefits of Splunk's approach to machine data. The presentation provides an example of designing service intelligence for an online store and supply chain at a toy company called Buttercup Games. It demonstrates how to configure a new KPI for database network utilization and modify a dashboard in Splunk IT Service Intelligence.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Service Intelligence hands on workshop
Similar to Service Intelligence hands on workshop (20)
More from Splunk
More from Splunk (20)
Recently uploaded
Recently uploaded (20)
Service Intelligence hands on workshop
- 2. Agenda 2 u Introductions and Set Up u Splundamentals – IT Troubleshooting with Splunk u What is IT Service Intelligence? u Service Intelligence Design Practices u Let's Play! u What's Next? u Happy Hour!
- 3. Safe Harbor Statement During the course of this presentation, we may make forward looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release. 3
- 5. Key Takeaways 1 Build on what you are already doing with Splunk Service Intelligence design and configuration practices 3 What is possible with Splunk IT Service Intelligence
- 7. Challenging Traditional Methods Network Infrastructure Layer Storage Server 74% -36% Application Layer Synthetic APM Byte Code Instrumentation Adaptive Thresholding HP Run-Time Service Model CA Service Operations Insight IBM NetCool/Omnibus Service Model definition & Correlation Engine Business Layer Aggregation/Correlation/Visualization Service Layer Challenges • Too many disparate components • Difficult to define Service Model • Labor intensive • Most implementations fail • Very important source is missing! (machine data)
- 8. Data-Defined & Driven Service Insights Infrastructure LayerApplication Layer Splunk> is the missing link • Data Fidelity • Single Repository for ALL data • Easier to Manage Services • Reduced Integrations • Reduced Point Solutions • Collaborative Approach • Quick time to value Data Fabric Platform Service Intelligence Network Packet, Payload, Traffic, Utilization, Perf Synthetic APM Availability, Capacity, User Experience Byte Code Instrumentation Usage, Experience, Performance, Quality Adaptive Thresholding Apps, Services, Systems74% -36% Server Performance, Usage, Dependency Storage Utilization, Capacity, Performance MACHINE DATA
- 9. Splunk Approach to Machine Data 9 Structured RDBMS SQL Schema on Write Traditional ETL Search Schema on Read Splunk Universal Indexing Volume Velocity Variety Unstructured • Define Static schema • ETL into Schema • Enrich at write • New data = new columns • New questions = new columns • “Data at rest” (delayed info) • Labor Intensive & time consuming Ideal for Reporting • “Schema-on-the-Fly” • Data in native format • Enrich at read • New data = no changes needed • New questions = no changes needed • “Data in motion” (Real time) • Fast time to value Ideal for Investigation
- 12. IT Service Intelligence Value Stack § Adaptive Threshold § Behavior Anomaly § Correlates Data into Knowledge § Visualizes entire stack § View the entire Ecosystem § 3 clicks to get the answer versus 10 § Time Series Index § Schema on Read § Data Model Service Model ML § Accelerators § Trend aggregation § Multi KPI Alerts ITSI
- 30. 30 Your Service Intelligence Collaborators Service Owners • Business functions • Performance indicators • Common business issues • Frequency of issues • Business impact of issues Operations and Support • Common issues • Performance indicators • Resolution processes • Tools used for resolving issues • Frequency of issues • IT impact of issues Enterprise Architecture • Business processes • Key inputs and outputs • Technology architecture • Data architecture • Common issues Administrators • Current tools and usage, and adoption levels • Splunk expertise • Environment expertise • Personal pain
- 33. Service Intelligence Design – Buttercup Games Infrastructure Layer Application Layer Business Layer Service Layer Order Entry Manufacturing Shipping Fulfillment Supply Chain Online Store EDI Web Tier Middleware • Total Orders • Total Revenue • Unit Count • Unit Failures • Service Level • Delivery Time • Online Orders • Online Revenue • Response Time • ServiceHealth • Incidents/Changes • Customer Satisfaction • HTTP Hits • Error Rate • CPU Load • Memory Used • Disk Used • IO Latency • CPU Load • Memory Used • Disk Used • IO Latency • Response Time • Error Rate • Response Time • Storage Free
- 35. Service Intelligence Design in ITSI 1. High-value business services • Buttercup Games Online Store and Supply Chain 2. Major business functions • Order Entry, Manufacturing, Shipping Fulfillment 3. Supporting services • Web, Middleware, Database 4. Relevant KPIs for each service • Database:, errors, SQL hits, …) 5. Splunk search for each KPI • (index=DB (warn* OR error*) | stats count) 35
- 36. Service Decomposition – Buttercup Games Infrastructure Layer Application Layer Business Layer Service Layer Order Entry Manufacturing Shipping Fulfillment Supply Chain Online Store EDI Web Tier Middleware
- 37. Putting It All Together Infrastructure Layer Application Layer Business Layer Service Layer Order Entry Manufacturing Shipping Fulfillment Supply Chain Online Store EDI Web Tier Middleware • Total Orders • Total Revenue • Unit Count • Unit Failures • Service Level • Delivery Time • Online Orders • Online Revenue • Response Time • ServiceHealth • Incidents/Changes • Customer Satisfaction • HTTP Hits • Error Rate • CPU Load • Memory Used • Disk Used • IO Latency • CPU Load • Memory Used • Disk Used • IO Latency • Response Time • Error Rate • Response Time • Storage Free
- 38. Typical Data Sources Infrastructure Layer Application Layer Business Layer Service Layer Order Entry Manufacturing Shipping Fulfillment Supply Chain Online Store EDI Web Tier Middleware • Application Logs • Corporate Databases • Service Management • Application Logs • Webserver Logs • DB Perf Counters • Wire data • Perf Counters • Access Logs • Network Logs
- 45. New Requirements! 45 ● Create a new KPI for the DB Service: ● Network Utilization ● Modify the Executive Glass Table in order to show off the services you slave over “WE only have about 15min TO DO WHAT ???!!???” Think about how long this would take you today?
- 47. Let’s Talk Entities 47 ● Select DB Service ● Entities are the relevant things which support this service (usually hosts) ● Select the right entries with filters, ANDs, ORs ● Original Entity list can come from CMDB, spreadsheet, Splunk search, others
- 48. A KPI in 5 minutes? Absolutely! 48 Click New – Generic KPI Select Data Model ● Host Operating System ● Network ● # bytes ● Next Call it “Network Utilization”, with your username up front
- 49. KPIs Continued…. 49 Splunk Builds Searches for you – Oh Yeah, that’s happening J ● Select Yes for Split by & Filter options ● Select host for Entity Lookup & Alias options ● Click Next
- 50. Almost There… 50 Select ● KPI Search Schedule: Every Minute ● Entity Calculation: Average ● Service/Agg Calculation: Average ● Calculation Window: Last Minute ● Click Next ● Unit: Bps ● Click Next
- 51. Final Steps … 51 Set your thresholds: ● Aggregate (All) ● Per Entity ● Click “Add Threshold” TWICE ● Make the Neapolitan ice cream colors Yellow, Green, Yellow ● Drag the sliders around in order to get the current data graph entirely inside the Green (normal) band ● Click Finish ● Other options are also available, including adaptive thresholds and anomaly detection
- 55. Anomaly Detection 55 ● Machine Learning ● Works well for data with patterns ● Requires some “training” (trial & error) to zero in on best sensitivity ● More sophisticated capabilities coming! (multivariate, more algorithms, etc)
- 57. Clone the Glass Table 57 Return to Saved Glass Tables page (click on Glass Tables in the upper menu bar) CLICK Edit for “Buttercup Games Business Process (IN PROGRESS)” • Select Clone • Title: Add your username to the front • Permissions: Shared in App • Click Clone Page • Click on your new Glass Table from the list, to view it
- 58. Edit & Have Fun! 58 Click on Edit in the upper right corner of your Glass Table Use the “Services” panel on the left to select Individual KPIs, or Aggregate Service Health Scores • Choose 2 KPIs from Online Store that would be useful in the “Order Process” section • Drag the selected widgets onto the canvas, positioning in the gray oval • What’s the difference between the and tools at the top left?
- 59. More Fun with the Glass Table Editor… 59 Use the Configurations panel on the right to edit a selected widget • Can change the visualization type, drilldown behavior, and other settings • You should hit Save frequently • Revert All Changes can be helpful, occasionally
- 60. Finishing up … 60 • Add a ServiceHealthScore widget for Online Store under Buttercup • Choose a Viz Type with a sparkline graph, then resize to make it look pretty • Modify the Custom Drilldown action to go to the saved glass table, Buttercup Games Online Store • Bonus Points: Make the label bigger, more readable • Click Save • View when done
- 62. A Troubleshooting Exercise 62 Let’s use ITSI to troubleshoot an outage ● Start at your Glass Table, “<UserName> Buttercup Business Process” ● Customer Care reports that unhappy customers are complaining of failures and long delays when trying to purchase ● The calls began coming in at around the top of the last hour. ● In the upper right corner of the Glass Table, change the time picker from Now to XX:00:00.0, where XX is the previous hour. For example, if it is currently 14:05, set the time picker to 13:00:00.0, then Apply ● This is how we can “time travel” back to see conditions at a particular outage– oh yeah!
- 64. A Troubleshooting Exercise, cont’d. 64 ● The Online Store Glass Table shows a much more detailed view, including the impacted customer-facing KPIs at the far left (Revenue, etc) ● Based on this view of all the relevant services, where do you think the root cause lies? ● Which service should we troubleshoot first? ● Click on Health widget for that service, to drill down to a Deep Dive
- 65. Deep Dive 65 ● Deep Dive shows multiple KPIs and Health Scores in parallel “swim lanes”. ● The Health Score for this Service is the top swim lane. Can you see when it begins to degrade from 100%? ● Mousing over this point in time, can you spot the KPI with the leading fault indication, i.e., what failed first? ● To improve readability, make sure the Primary Time Range (lower left corner) is set to Presets > Last 60 minutes
- 66. Multi-KPI Alerts and Notable Events 66 ● Click on Notable Events Review ● Multiple KPIs and Healthscores can be combined in sophisticated ways to create Multi-KPI alerts ● When a Multi-KPI alert fires, one of the outcomes is the creation of a Notable Event ● Notable Events allow NOC personnel and others to triage and coordinate event management efforts
- 67. Service Analyzer 67 ● Click on Service Analyzer > Default Service Analyzer ● Back where we started! ● This view shows a “no-frills” list of services (top) and hottest KPIs (bottom) ● Provides access into Service Details ● It is useful for NOCs and others who need a high-level situational view
- 69. Summary 69 ● High-value services can be decomposed and modeled in ITSI, using machine data from the relevant systems ● Services and KPIs can be created in minutes, with sophisticated thresholding techniques to distinguish “normal” from “not normal” ● Glass Tables allow service health and KPI metrics to be displayed in a way that makes sense to specific groups, such as Executive Leadership, Business Service Owners, the NOC, DevOps & Others ● Deep Dives allow KPIs to be compared side-by-side across any time range, accelerating root cause analysis and significantly reducing MTTR ● Multi-KPI Alerts and Notable Events reduce alert noise, producing actionable events and a means to manage them ● … and it’s fast+fun to build!
- 71. Splunk IT Service Intelligence Machine Learning-Powered, Analytics-Driven IT Operations Simplify service operations Prioritize incidents with context Redefine the role of IT Combine events & metrics across silos with ease, flexibility & scale in days Unify siloed monitoring Leverage machine learning to detect anomalies & highlight events that matter Deliver business & service context to prioritize incident investigation & action Support decisions & communicate results with powerful service-level insights
- 73. Call Center Service Service Health Transactions ACD Analysis – Core Splunk Call Wait History Inbound Analysis Social Media Online Msg Social Media Mail SupportVOIP Service Inbound Calls
- 74. Online Transactions Internal Transfer Service External Wire Service Money Exchange Service Money Transfer Services Service Health Corporate Reconciliation Service Fed Exchange Service Core Splunk Searches Transaction History System Investigation Heat Map Analysis
- 75. CIO Scorecard Enterprise Service Status Major Incidents Service Health Continuous Operational Visibility Volume Revenue Incidents Changes Major Changes Service Health Volume Revenue Incidents Changes Service Health Volume Ontime DeliveryIncidents Changes Service Health VolumeRevenue Incidents Changes Service Health Volume Revenue Incidents Changes Container UtilService Health Throughput Incidents Changes
- 76. The Vision - Business Operations Center • Splunk ITSI has the fundamentals to deliver on the promise of real time business visualizations • Modeled after your Security, Network, and IT Operations Centers • Monitoring and diagnosis of important ecommerce and brick and mortar operations • Enhanced with process insight from end-to-end, alerts, machine learning and real-time response NOC SOC BOC
- 77. Sign Up Now – We’re here to help! Harness the creativity and domain knowledge of your organization to unlock the value of data and solve an important Business Service problem through a joint service intelligence workshop with key stakeholders Define methods for: › Proactive service monitoring › Reduced risk and failures › Faster issue resolution › Increased business performance What is it? › 1 Day Onsite Workshop › Tightly linked with value › Collaborative approach › Build your own Glass Table
- 78. Our Workshop In Action