![From Days to Minutes With Splunk
“First
Responder”
2012-12-05 07:04:44 Id=Rd910EAJ City=New York Email.jdoe@gmail.com
product_id=product_i BD-
66.57.19.112 ..[05/Dec/2012 07:05:22:152]”GET /card.do?action=addtocart
&itemid=K9
[1208/12 02:39:03:209 UTC] 000000c6 ConnectionEve A J2CA00561:
ConnectionExeception:[IBM][CLI Driver] SQL1224N
Report and
analyze
Custom
dashboards
Monitor
and alert
Ad hoc
search
2012-12-05 07:04:44 Id=Rd910EAJ
City=New York Email.jdoe@gmail.com
product_id=product_i BD-
66.57.19.112 ..[05/Dec/2012
07:05:22:152]”GET
/card.do?action=addtocart
&itemid=K9
[1208/12 02:39:03:209 UTC]
000000c6 ConnectionEve A
J2CA00561:
ConnectionExeception:[IBM][CLI
Driver] SQL1224N
Outage
Occurs](https://image.slidesharecdn.com/2016-03-09itoperationsbreakoutmuc-160310115943/75/SplunkLive-Munchen-2016-Splunk-fur-IT-Operations-5-2048.jpg)
Uploaded bySplunk
SplunkLive! München 2016 - Splunk für IT Operations
The document discusses Splunk for IT operations (ITOps). It provides an overview of how Splunk can help organizations gain operational intelligence and visibility across their IT infrastructure and applications. Some key points: - Splunk consolidates machine data from different sources like servers, storage, networking devices, applications etc. into a single platform for monitoring, searching and analyzing data. - It helps overcome issues of disconnected point solutions, siloed teams and outdated tools that take up majority of IT time for maintenance instead of innovation. - Splunk provides real-time search capabilities to help IT teams act as "first responders" and reduce problem resolution time from days to minutes by quickly searching across all log data.
More Related Content
Similar to SplunkLive! München 2016 - Splunk für IT Operations
SplunkLive! München 2016 - Splunk für IT Operations
- 1. Copyright © 2016Splunk Inc. SPLUNK FOR IT-OPERATIONS Udo Götzen CISSP, Senior Sales Engineer In diesem Raum: 13:00 Splunk Enterprise 6.3 14:45 Splunk for ITOps
- 2. CIO Obstacle: EscalatingIT Complexity SERVERS STORAGE NETWORKING VIRTUALIZATION INFRASTRUCTURE APPLICATIONS PACKAGED APPLICATIONS CUSTOM APPLICATIONS Identity VPN IP Phone HR Email Finance App Svr DB Web Svr SaaS/PaaS IaaS
- 3. CIO Obstacle: EscalatingIT Complexity SERVERS STORAGE NETWORKING VITUALIZATION INFRASTRUCTURE APPLICATIONS PACKAGED APPLICATIONS CUSTOM APPLICATIONS Identity VPN IP Phone HR Email Finance App Svr DB Web Svr SaaS/PaaS IaaS Complex, silo-based technologies Disconnected and outdated point solutions Over 70% of time spent on maintaining, not innovating
- 4. Before Splunk Data Gathering DB App NW Storage Now What? …. War Room Outage Occurs Humanlatency measured in hours or days
- 5. From Days toMinutes With Splunk “First Responder” 2012-12-05 07:04:44 Id=Rd910EAJ City=New York Email.jdoe@gmail.com product_id=product_i BD- 66.57.19.112 ..[05/Dec/2012 07:05:22:152]”GET /card.do?action=addtocart &itemid=K9 [1208/12 02:39:03:209 UTC] 000000c6 ConnectionEve A J2CA00561: ConnectionExeception:[IBM][CLI Driver] SQL1224N Report and analyze Custom dashboards Monitor and alert Ad hoc search 2012-12-05 07:04:44 Id=Rd910EAJ City=New York Email.jdoe@gmail.com product_id=product_i BD- 66.57.19.112 ..[05/Dec/2012 07:05:22:152]”GET /card.do?action=addtocart &itemid=K9 [1208/12 02:39:03:209 UTC] 000000c6 ConnectionEve A J2CA00561: ConnectionExeception:[IBM][CLI Driver] SQL1224N Outage Occurs
- 6. “Splunk reduced our escalationsby 90% and our problem resolution time by 67%. “Escalations reduced by 90% and MTTR dropped by 67%” Splunk at Service Desk: Vodafone Paulo Carvalho Director Operations Theoldway:DisparateITsilos impactCustomerService • Manuallyintensive,error-proneprocessesresultinconstantescalationsandlongdelays • Expensive,home-growntoolsforlogcollectionandanalysisdon’tprovidethecompletepicture • Disconnectedsystemscreatetroubleinmeetingsecurityandcompliancemandates Thenewway:Providecomprehensivevisibility andcontrol ✓ Asingle Tier 1support person can now perform iterative searches across alltheir IT data to investigate, identify, and fixthe problem – escalations reduced by90percent ✓ Splunk consolidates logs from disparate systems into asingle view, providing visibility across end- to-end service delivery from one place -time to problem resolution dropped by67% ✓ Role-based secure access to logs viaSplunk ensures SOX compliance ✓ Monitor IT data and find issues before they become visible to customers
- 7. Splunk : TheBetter Approach For IT 7 Customer Facing Data Outside the Datacenter Applications Web logs Log4J, JMS, JMX .NET events Code and scripts Networking Configurations syslog SNMP netflow Databases Configurations Audit/query logs Tables Schemas Virtualization & Cloud Hypervisor Guest OS, Apps Cloud Linux/Unix Configurations syslog File system ps, iostat, top Windows Registry Event logs File system sysinternals Logfiles Configs Messages Traps Alerts Metrics Scripts TicketsChanges Click-stream data Shopping cart data Online transaction data Manufacturing, logistics… CDRs & IPDRs Power consumption RFID data GPS data Powerful, end-to-end, real-time platform for Machine Data
- 8. Splunk : TheBetter Approach For IT 8 Customer Facing Data Outside the Datacenter Applications Web logs Log4J, JMS, JMX .NET events Code and scripts Networking Configurations syslog SNMP netflow Databases Configurations Audit/query logs Tables Schemas Virtualization & Cloud Hypervisor Guest OS, Apps Cloud Linux/Unix Configuration s syslog File system ps, iostat, top Windows Registry Event logs File system sysinternals Logfiles Configs Messages Traps Alerts Metrics Scripts TicketsChanges Click-stream data Shopping cart data Online transaction data Manufacturing, logistics… CDRs & IPDRs Power consumption RFID data GPS data Powerful, end-to-end, real-time platform for Machine Data Noupfrontschema Nocustomconnectors NoRDBMS •Any amount, any location, any source.
- 9. Copyright © 2016Splunk Inc. SPLUNK APPS ACCELEERATE INSIGHT
- 10. Copyright © 2015Splunk Inc. Apps Provide Deep Insights By Role Find and resolve problems fast in individual technology areas Exchange Admin Service Health Performance Message tracking VMware/Win/ Linux Admin Infrastructure Health Performance Anomalies/Outliers Storage Admin Infrastructure Health Performance Anomalies/Outliers 10
- 11. Reduce Costs: Consolidatetools, eliminate silos, find root cause faster! Exchange Admin Linux/Win Admin Network Admin Applications Admin Line of Business User Application Support VMware/Linux/ Win Admin Security Admin Storage Admin IT Management
- 12. Copyright © 2015Splunk Inc. Splunk : Platform For IT Operational Intelligence 12 Plug-Ins, Templates and Apps Accelerate Value From Machine Data No rigid schemas– Add in data from any other source. API SDKs UI Server, Storage, Network Server Virtualization Operating Systems Custom Applications Business Applications Cloud Services App Performance Monitoring Ticketing/ and Other Web Intelligence Mobile Applications Stream
- 13. Copyright © 2015Splunk Inc. Splunk For Operating Systems Proactive Monitoring Operational Analytics End-to-End Visibility Get instant insight into infrastructure health OS Metrics for Performance, Capacity & Resource Allocation Analyses Scale And Correlate Across All Tiers Of Your Technology Stack 13
- 14. Copyright © 2015Splunk Inc. Splunk For Virtualization & Storage Proactive Monitoring Operational Analytics End-to-End Visibility Real-time actionable insights into problem spots and health issues Real-time & historical insights into performance, security, capacity, forecasting and change tracking Scalable Big Data solution for holistic visibility across all technology tiers 14
- 15. Copyright © 2015Splunk Inc. Add-on for AWS Splunk App for AWS Explore Analyze Dashboard Alert CloudTrail EC2 EMR VPC ELBCloudfront Lambda ConfigCloudWatch Other Services S3 Kinesis IoT Splunk Insights for AWS Machine Data (api audit trail) (logs and metrics) (change mgmt) Why Splunk for AWS? Security Intelligence (Cloudtrail, Config Cloudwatch, VPC) Operational Intelligence (Cloudwatch, Config, ELB, Cloudfront) DevOps Intelligence (Cloudwatch, Lambda) Big Data Insights (Kinesis, EMR, IoT, S3)
- 16. Copyright © 2015Splunk Inc. Live Demo Splunk for AWS 16
- 17. Copyright © 2016Splunk Inc. IT-SI IT-SERVICE INTELLIGENCE
- 18. Data-driven service insights forroot-cause isolation and improved service operations INTRODUCING
- 19. Copyright © 2015Splunk Inc. Splunk IT Service Intelligence Deploy flexible and scalable solution in days, not months Any IT data (metrics and events) from anywhere Quick to install, immediate value and on-the-fly customization Flexible deployment options (on-premises, Cloud and hybrid) Scale and robustness of the Splunk platform
- 20. Copyright © 2015Splunk Inc. Splunk IT Service Intelligence Transform IT monitoring with data-driven analytics Dynamically adapting KPIs for dynamic thresholds Machine learning to baseline normal operations Detection of anomalous behavior to drive meaningful actions Correlation searches to create meaningful “alerts” off KPIs
- 21. Copyright © 2015Splunk Inc. Splunk IT Service Intelligence Redefine the role of IT (as a strategic business partner) with service awareness Data-driven insights – ask any question, any time Flexible and powerful framework to map data into services Easy definition of KPIs to easily measure what matters most Drill downs for in-depth investigation and resolution
- 22. What is aService? DNS Requests Responses Technical Services Customer Transactions Requests Responses Business Services Auth Requests Responses Web Requests Responses Support Desk Requests Responses
- 23. What is aService? Packet Network Hypervisor and Hosts RBMDBs Storage Tier API Services Web ServicesCustomerTransactions Mobile API/Middleware PartnerPortal DNS
- 24. What is aKPI? DNS Requests Responses KPI: Number of requests KPI: Error rate KPI: Average response time KPI: Servicer CPU load KPI: Server network I/F errors Customer Transactions Requests Responses KPI: Number of transactions KPI: Error rate KPI: Average response time KPI: Count of Incident Tickets KPI: Synthetic Transx Health
- 25. Copyright © 2015Splunk Inc. 25 Achieve Service Visibility Faster Service Analyzer High-level view of services and composite health scores Glass Tables Personalized visualizations of your services Deep Dives Organized view of performance indicators across silos Multi KPI Alerts Correlation rules to generate notable events Notable Events Easy-to-understand report on results of correlation searches Anomaly Detection and Adaptive Thresholds Machine learning to baseline normal operations and identify anomalous behavior
- 26. Copyright © 2015Splunk Inc. Live Demo IT-SI 26
- 27.
- 28.
- 29.
Editor's Notes
- #2 Welcome to SplunkLive [City]. Thank you for taking the time to attend today’s event.
- #7 Company Background: Vodafone Group Plc is the world's leading mobile telecommunications company, providing a wide range of services including voice and data communications. Paulo Carvalho works in Vodafone's DSSL group supports Vodafone live! Which includes popular mobile video, news, music and other services. Paulo is the Services Network Manager at Vodafone Portugal and is responsible for all services on top of GSM Network, MMS, SMS, Voice Mail, Unified Messaging, streaming, Mobile Portal, VAS Services, Prepaid Services. Other Notes: Vodafone uses Splunk for application troubleshooting and management of services they offer over their 3G network. The environment is complex, with many services being offered, running on many platforms and servers - Solaris, Redhat Linux and introducing virtualized environments. They also have a huge Java and J2EE infrastructure and often need to search quickly for errors or exceptions occurring within the last sixty minutes. Vodafone has been a successful user of Splunk realizing significant material benefits. They have also moved to a proactive phase with Splunk, using it to monitor IT data such as threshold levels for specific systems, and fixing issues before they become visible to their customers.
- #8 So how does Splunk help? We offer a powerful, end-to-end, real-time platform for Machine Data. Splunk can collect data from any source, giving our customers real-time visibility and intelligence into what’s happening across the IT infrastructure – whether it’s physical, virtual or in the cloud.
- #9 Splunk’s highly capable platform for machine data can handle any machine generated data from any location and any source – without the need to transform the data to fit a schema, without the need for custom connectors-because unlike most other tools on the market, Splunk does not have a database backend. Splunk’s proprietary map-reduce based high speed index and retrieval system allows management of very large quantities of data at scale with just commodity x86 servers.
- #10 Welcome to SplunkLive [City]. Thank you for taking the time to attend today’s event.
- #13 Remember we said before, that Splunk is a “platform” for machine data? Splunk has evolved over the years from an engine for any kind of machine data to a robust platform, complete with a REST API, 6 different SDKs and numerous “apps” that sit on top of Splunk and provide out of the box value from your data. These “apps” are available on Splunkbase and they accelerate getting g data into Splunk and getting pre-built visualizations for that data. Note that these apps are not like connectors because they don’t lock away the data in a silo or restrict its usage to particular sets of views – the data is in Splunk and can be used side by side with any other data in Splun k. You can move dashboards and key indicators across apps or customize them in any way you want. Apps make it faster to get value out of your data and several key apps provide new visibility into areas that were formerly “black box” in the infrastructure – such as the virtualization apps. We also recently introduced the 2 new offerings – one to collect wire data, with the Splunk App for Stream (stemming from the acquisition of Cloudmeter) and MINT (Mobile Intelligence) that stems from our acquisition of Bugsense. The Splunk App for Stream enables the capture of real-time streaming wire data, which is the data transmitted between applications over the network. It enables visibility into application, business and user activity without the need for instrumentation, enhancing various operational use cases across IT, security and the business. And Splunk MINT helps you gain visibility into mobile app performance and quality, so you can deliver better mobile apps Splunk MINT helps you combine and correlate mobile app data with other data in Splunk so you can pinpoint problems faster and analyze user experience/behavior across mobile, desktop and web channels. The main value from the apps is providing context for data from silos and making it available inside Splunk for correlation with other data from other silos. In addition to prebuilt apps, customers can also build their own. What have developers been building using Splunk Enterprise? Examples include the following: Run searches and retrieve Splunk data from existing Customer Service/Call Center applications (Comcast use case) Integrate Splunk data into existing BI tools and dashboard (Tableau, MS Excel) Build mobile applications with KPI dashboards and alerts powered by Splunk (Otto Group use case) Log directly to Splunk from remote devices (Bosch use cases) Build customer-facing dashboards powered by user-specific data in Splunk (Socialize, Hurricane Labs use cases) Programmatically extract data from Splunk for long-term data warehousing We hope this is just the beginning. We hope to open up a whole new world of enterprise apps.
- #18 Welcome to SplunkLive [City]. Thank you for taking the time to attend today’s event.
- #19 That brings us to Splunk IT Service Intelligence – a packaged solution that enables real-time visibility into services driven by machine data. Splunk ITSI speeds and simplifies service monitoring and analytics and enables IT to make better, smarter and informed business decisions. This solution allows you to gain a deep understanding of your services. With Splunk ITSI, you have real-time views into the health of your services, and can use advanced analytics to find patterns, detect anomalies and trends to proactively monitor and address issues. As a result you have improved service visibility, reduced resolution times, and a transformative approach to monitoring and analytics driven by machine-data.
- #23 A services can literally be sources of data a customer wants to group together to monitor in a single healthscore or just wants to logically group together because they need to be managed by a specific team or needs to be reported in such a way. Services derive their value when KPIs are defined within them or dependencies are defined to other services. Therefore you could have a more abstractly defined service which only depends on other services to derive its own health. E.g. Partner portal is a conceptual service which depends on the API service which in turn has its own KPIs but depends on Web Services. Alternatively you could have Partner portal depend on each and every one in blue, or not even have all the ones in blue and have all the Kpis be inside Partner portal. Everything you see in the diagram above could be a service in ITSI.
- #24 A services can literally be sources of data a customer wants to group together to monitor in a single healthscore or just wants to logically group together because they need to be managed by a specific team or needs to be reported in such a way. Services derive their value when KPIs are defined within them or dependencies are defined to other services. Therefore you could have a more abstractly defined service which only depends on other services to derive its own health. E.g. Partner portal is a conceptual service which depends on the API service which in turn has its own KPIs but depends on Web Services. Alternatively you could have Partner portal depend on each and every one in blue, or not even have all the ones in blue and have all the Kpis be inside Partner portal. Everything you see in the diagram above could be a service in ITSI.
- #25 A services can literally be sources of data a customer wants to group together to monitor in a single healthscore or just wants to logically group together because they need to be managed by a specific team or needs to be reported in such a way. Services derive their value when KPIs are defined within them or dependencies are defined to other services. Therefore you could have a more abstractly defined service which only depends on other services to derive its own health. E.g. Partner portal is a conceptual service which depends on the API service which in turn has its own KPIs but depends on Web Services. Alternatively you could have Partner portal depend on each and every one in blue, or not even have all the ones in blue and have all the Kpis be inside Partner portal. Everything you see in the diagram above could be a service in ITSI.
- #29 How can you leverage Splunk?
- #31 (Optional) Showing Adaptive Thresholding (Time: 3 min) Opening Tell Major points/highlights: Most KPIs are depending on business usage, weekly or daily patterns Static thresholds must compromise between peak utilization & low-use times Situation – IT systems support a particular business need. Based on that business usage, typical utilization patterns emerge. Examples: Email systems have higher utilization in early office hours Service Management Systems have higher utilization at the beginning of the week as well as in early office hours Retail system utilization often reflects seasonality or active promotions Trading Systems consume most resources during stock market hours Human Resource Systems may sit almost idle for the majority of the month unless Payroll data is being processed Static thresholds like those found in most monitoring systems are typically tuned to deliver accurate results during peak system usage, ignoring issues and leading indication during idle times. Adaptive thresholds on the other hand learn “typical” application behavior. They can adjust to various business patterns and still detect irregular behavior or exceptions during low-use times because they have a more granular threshold definition. This threshold definition takes past behavior into account and overlays it with a comparable timeframe. Highlight – 2 Components System Learning of Past behavior Time Windows to apply these thresholds Show (Demo) We’ll be looking at the service, "Web service”, "Corporate Web Site Successful Transactions” KPI. Applying simple thresholds against KPI metrics is sometimes helpful, but most metrics in the "real world" vary cyclically, according to human activities. ITSI allows more sophisticated thresholding, including time-based and adaptive, based on range, quantile and standard deviation methods. Note that the template "2-hour blocks every day (adaptive/stdev) is selected" A number of templates are available, or custom templates can be created.
- #32 After you have chosen your thresholding template, you can either set your thresholds manually or use Adaptive Thresholding <Click Apply Adaptive Thresholding>
- #33 Thus, thresholds can adapt to the real-world behavior of the data. We've only just seen the tip of the iceberg here; the point is that ITSI allows a variety of sophisticated thresholding approaches, to track the health of the metric in realistic and actionable ways. Additional points (optional): Explain the differences between Static, Standard Deviation, Quantile, and Range based approaches Explain that you can still take advantage of adaptive thresholds even if you don’t feel comfortable with resetting the thresholds automatically on a daily basis. This can be done by using the Std Deviations, Range or Quantile based approach to initially calculate the thresholds but then toggling them back to static. This way the algorithm uses the past data to come up with the “best/adaptive” thresholds but your keep control on when they need to be updated. Closing Tell In a couple minutes we’ve build a threshold based system that allows to find issues in low-use times (something they most likely don’t do today). We've shown adaptive methods to determine standard system behavior and calculate threshold suggestions without being familiar with the usage patterns of this KPI We’ve shown how these thresholds can be adjusted based on new usage behavior without admins or experts having to get involved