Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Machine Learning + Analytics in Splunk

3,318 views

Published on

Machine Learning + Analytics in Splunk

Published in: Technology
  • Don't even THINK about buying any uterine fibroids product, drugs or going on a gimmick fibroids program until you read my revealing, no-holds barred holistic uterine fibroids cure book. ➤➤ http://t.cn/Aig7bySW
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Machine Learning + Analytics in Splunk

  1. 1. Copyright © 2015 Splunk Inc. Operationalizing Machine Learning
  2. 2. 2 Disclaimer During the course of this presentation, we may make forward looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in the this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release.
  3. 3. Copyright © 2015 Splunk Inc. Why Machine Learning?
  4. 4. 4 Humans are good at learning, but we get lost in volume and details…
  5. 5. 5 Why do we need Machine Learning? - Improve Decision Making - Forecast or Predict KPIs - Alert on Deviation - Uncover hidden trends or relationships All of this requires Diverse Data from across Many Silos. Lots of Unstructured, Real Time Data.
  6. 6. 6 Run the Business in Real-time Data From the Past Real-time Data Statistical Forecast T – a few days T + a few days Security Operations Center IT Operations Center Business Operations Center Predictive (Models) Descriptive (BI Tools, Data Lakes) Grey space
  7. 7. Copyright © 2015 Splunk Inc. What is Machine Learning?
  8. 8. 8 ML 101: What is Machine Learning? What: “Field of study that gives computers the ability to learn without being explicitly programmed” – A. Samuel, 1959 How: Generalizing (learning) from examples (data) Simple ML workflow: – EXPLORE data – FIT models based on data – APPLY models in production – VALIDATE models – REPEAT
  9. 9. 9 How Machines Learn [Prediction] • When we see thick clouds and an overcast sky, we predict that it’s likely going to rain [Estimation/ Regression] • Estimate how much an apartment costs based on its location, condition and prices of properties in that neighborhood [Classification/ Clustering] • Determine the gender of a person based on her/his features, hair style and the way s/he dresses [Anomaly Detection] • Identify the odd one out [Reinforcement Learning] • If I made a mistake this time, can I do better next time? All of us have had some experience in learning. But… what’s behind our experience? How do we translate that knowledge to code?
  10. 10. 10 Major Types of Machine Learning 1. Supervised Learning: generalizing from labeled data
  11. 11. 11 Major Types of Machine Learning 2. Unsupervised Learning: generalizing from unlabeled data ?
  12. 12. 12 3. Reinforcement Learning: • System is rewarded (or punished) based on the outcomes it generates • Action leads to a change in the state of the world and generates an error score Major Types of Machine Learning
  13. 13. Copyright © 2015 Splunk Inc. Splunk’s Machine Learning Tour
  14. 14. 14 Overview of ML at Splunk Core Platform Search Packaged Premium Solutions Custom ML Platform for Operational Intelligence
  15. 15. 15 Search Includes Machine Learning Core Platform Search is a powerful and highly flexible interface built with ML anomalydetection
  16. 16. 16 Splunk IT Service Intelligence Get Data Define services, entities and KPIs Monitor and troubleshoot Analyze and detect Data-Defined, Data-Driven Service Insights Packaged ML : Adaptive Thresholds and Anomaly Detection One of several Premium Solutions
  17. 17. 17 Splunk Machine Learning Toolkit Assistants: Guide model building, testing, & deploying for common objectives Showcases: Interactive examples for typical IT, security, business, IoT use cases Algorithms: 25+ standard algorithms available prepackaged with the toolkit SPL ML Commands: New commands to fit, test and operationalize models Python for Scientific Computing Library: 300+ open source algorithms available for use Build custom analytics for any use case Extends Splunk platform functions and provides a guided modeling environment
  18. 18. 18 Algorithms supported (v2.0, .conf2016)
  19. 19. ITSI, UBA Domain Expertise (IT, Security, …) Data Science Expertise Splunk Expertise Custom Machine Learning – Success Formula Identify use cases Drive decisions Set business/ops priorities SPL Data prep Statistics / math background Algorithm selection Model building Splunk ML Toolkit facilitates and simplifies via examples & guidance Operational success
  20. 20. 20 Summary: The ML Process Problem: <Stuff in the world> causes big time & money expense. Value Hypothesis Solution: Build ML model to forecast <possible incidents>, act pre-emptively & learn Operationalize 1. Get all the relevant data to the problem; Explore the data 2. Select and Fit an algorithm on the data, generating a model 3. Apply & Validate models until predictions solve the problem 4. Surface the model to X Ops, who consume the model to solve the problem
  21. 21. 21 Machine Learning Process with Splunk 2 Collect Data Explore/ Visualize Model Evaluate Clean/ Transform Publish/ Deploy props.conf, transforms.conf, Datamodels Add-ons from Splunkbase, etc. Pivot, Table UI, SPL ML Toolkit Alerts, Dashboards, Reports
  22. 22. Copyright © 2015 Splunk Inc. Splunk Architecture & ML
  23. 23. 23 Continuous Data Ingest at Scale DevelopVisualize PredictAlertSearch Engineers Data Analysts Security Analysts Business Users Native Inputs TCP, UDP, Logs, Scripts, Wire, Mobile Industrial Data SCADA, AMI, Meter Reads Modular Inputs MQTT, AMQP, COAP, REST, JMS HTTP Event Collector Token Authenticated Events Real Time Technology Partnerships Kepware, AWS IoT, Cisco, Palo Alto Maintenance Info Asset Info Data Stores External Lookups/Enrichment 2 OT Industrial Assets IT Consumer and Mobile Devices
  24. 24. 24 Sense and Respond Real Time Search Alert Third-Party Applications Smartphones and Devices Tickets Email Send an email File a ticket Send a text Flash lights Trigger process flow 2 OT Industrial Assets IT Consumer and Mobile Devices Every Search Can Use Machine Learning
  25. 25. 25 Splunk: Data Fabric 2 OT Industrial Assets IT Consumer and Mobile Devices Real Time IT users Analysts Business Users Ad Hoc Search Custom Dashboards Monitor and Alert Reports/ Analyze Clickstreams HadoopDevices Networks GPS/ Cellular Online Shopping Carts Servers Applications Analysts Business Users Data Warehouses Structured Data Sources CRM ERP HR Billing Product Finance DB Connect Look-ups ODBC SDK API
  26. 26. Different lenses into the same data SCADA Ops Center Biz Ops Center IT Ops Center Compliance Security Ops Center Data Reuse = Greater Data Leverage Fraud Ops Center, etc…
  27. 27. Copyright © 2015 Splunk Inc. ML Use Cases And Customer Stories
  28. 28. 28 ML Is All Around You! Recall: EXPLORE > FIT > APPLY > VALIDATE > REPEAT • Face detection: find faces in images • Spam filtering: identify SPAM messages • Shopping Recommendations: predict what customers would like to buy • Fraud detection: identify credit card transactions which may be fraudulent in nature • Weather forecast: predict whether or not it will rain tomorrow; estimate daily max/min
  29. 29. 29 Machine Learning Customer Success Network Incident Detection Service Degradation Detection Security / Fraud Prevention Prioritize Website Issues and Predict Root Cause Predict Gaming Outages Fraud Prevention Machine Learning Consulting Services Analytics App built on ML Toolkit Optimizing operations and business results Cell Tower Incident Detection Optimize Repair Operations Entertainment Company 1
  30. 30. 30 ML Toolkit Customer Use Cases 30 Speeding website problem resolution by automatically ranking actions for support engineers Reducing customer service disruption with early identification of difficult-to-detect network incidents Minimizing cell tower degradation and downtime with improved issue detection sensitivity Improving cell tower uptime and reducing repair truck roles with anomaly detection and root cause analysis Predicting and averting potential gaming outage conditions with finer-grained detection Ensuring mobile device security by detecting anomalies in ID authentication Preventing fraud by Identifying malicious accounts and suspicious activities Entertainment Company
  31. 31. 31 Detect Network Outliers Reduced downtime + increased service availability = better customer satisfaction 3 ML Use Case Monitor noise rise for 20,000+ cell towers to increase service and device availability, reduce MTTR Technical overview • A customized solution deployed in production based on outlier detection. • Leverage previous month data and voting algorithms “The ability to model complex systems and alert on deviations is where IT and security operations are headed … Splunk Machine Learning has given us a head start...”
  32. 32. 32 Reliable website updates Proactive website monitoring leads to reduced downtime 3 “Splunk ML helps us rapidly improve end-user experience by ranking issue severity which helps us determine root causes faster thus reducing MTTR and improving SLA” • Very frequent code and config updates (1000+ daily) can cause site issues • Find errors in server pools, then prioritize actions and predict root cause • Custom outlier detection built using ML Toolkit Outlier assistant • Built by Splunk Architect with no Data Science background ML Use Case Technical overview
  33. 33. Copyright © 2015 Splunk Inc. Show me the ML!
  34. 34. 34 Next Steps with Splunk ML • Reach out to your Tech Team! We can help architect ML workflows. • Lots of ML commands in Core Splunk (predict, anomalydetection, stats) • ML Toolkit & Showcase – available and free, ready to use • Splunk ITSI: Applied ML for ITOA use cases – Manage 1000s of KPIs & alerts – Adaptive Thresholding & Anomaly Detection • Splunk UBA: Applied ML for Security – Unsupervised learning of Users & Entities – Surfaces Anomalies & Threats • ML Customer Advisory Program: – Connect with Product & Engineering teams - mlprogram@splunk.com
  35. 35. 35 What Else? • Get the Machine Learning Toolkit from Splunkbase • Go watch Machine Learning Videos on Splunk Youtube Channel http://tiny.cc/splunkmlvideos • Go watch the Machine Learnings talks from Conf 2016: – Advanced Machine Learning in SPL with the Machine Learning Toolkit by Jacob Leverich – Extending SPL with Custom Search Commands and the Splunk SDK for Python by Jacob Leverich • Early Adopter And Customer Advisory Program : mlprogram@splunk.com • Field ML Architects : Andrew Stein (astein@), Brian Nash (bnash@)
  36. 36. 36 Mark Your Calendars! • .conf2017 is going to DC! • Sept 26-28, 2017 • Walter E Washington Convention Center
  37. 37. Copyright © 2015 Splunk Inc. Thank you!

×