Downloaded 19 times
Uploaded bySplunk
Power of SPL
The document discusses Splunk's Search Processing Language (SPL) and provides examples of how to use SPL commands. It covers SPL's over 140 search commands for data searching, filtering, manipulation and more. The presentation agenda includes an overview of SPL, examples of SPL commands for searching, charting, converging data sources, identifying anomalies, and custom visualizations. It also discusses using SPL with Splunk's Machine Learning Toolkit.
More Related Content
Similar to Power of SPL
Power of SPL
- 1. © 2017 SPLUNKINC.© 2017 SPLUNK INC. Power of Splunk Search Processing Language (SPL™) August 16, 2017 | Cincinnati, OH Tony Peveler | Sr. Sales Engineer
- 2. © 2017 SPLUNKINC. During the course of this presentation, we may make forward-looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward-looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release. Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2017 Splunk Inc. All rights reserved. Forward-Looking Statements
- 3. © 2017 SPLUNKINC. Set Up Before You Can Play Download the following at splunk.com and splunkbase.com ▶ Splunk Enterprise: • https://www.splunk.com/download ▶ Power of SPL App: • https://splunkbase.splunk.com/app/3353/
- 4. © 2017 SPLUNKINC. Set Up Before You Can Play ▶ Install Power of SPL App ▶ Now we’re ready to go!
- 5. © 2017 SPLUNKINC. ▶ License expired (already had older version installed) • Close browser, empty cache, open browser. If that doesn’t work: • Stop Splunk. • Uninstall all Splunk versions • Windows Control Panel->Uninstall programs->Splunk • OS X. Finder->Applications->Right click Splunk, Move to trash • Reinstall • Start Splunk ▶ Can’t start Splunk • Windows, Search Control panel ->Services->Splunk start • Linux; cd <SPLUNK dir>/splunk/bin;./splunk start Common problems at this point
- 6. © 2017 SPLUNKINC. 1. Installation and Setup (~15min) 2. Power of SPL Walkthrough (~1h 30min) • Overview & Anatomy of a Search • SPL Commands & Examples for Searching, charting, converging, mapping, transactions, anomalies, exploring data, custom 3. Custom Visualizations (~30min) 4. SPL and the Machine Learning Toolkit (~45min) Agenda
- 7. © 2017 SPLUNKINC. SPL Overview
- 8. © 2017 SPLUNKINC. ▶ Over 140 search commands ▶ Syntax was originally based upon the Unix pipeline and SQL and is optimized for time-series data ▶ The scope of SPL includes data searching, filtering, modification, manipulation, enrichment, insertion and deletion ▶ Includes machine learning such as anomaly detection SPL Overview Disk Intermediate results table Intermediate results table Final results table
- 9. © 2017 SPLUNKINC. ▶ Flexibility and effectiveness on small and big data ▶ Late-binding schema ▶ More/better methods of correlation ▶ Not just analyze, but visualize Why Create a New Query Language? Data BIG Data
- 10. © 2017 SPLUNKINC. search and filter | munge | report | cleanup | rename sum(KB) AS "Total KB" dc(clientip) AS "Unique Customers" | eval KB=bytes/1024 sourcetype=access* | stats sum(KB) dc(clientip) SPL Basic Structure
- 11. © 2017 SPLUNKINC. SPL Examples
- 12. © 2017 SPLUNKINC. ▶ Find the needle in the haystack ▶ Charting statistics and predicting values ▶ Enriching and converging data sources ▶ Map geographic data in real time (We’ll add some custom viz here!) ▶ Identifying anomalies ▶ Transactions ▶ Data exploration & finding relationships between fields ▶ Custom commands SPL Examples and Recipes
- 13. © 2017 SPLUNKINC. ▶ Find the needle in the haystack ▶ Charting statistics and predicting values ▶ Enriching and converging data sources ▶ Map geographic data in real time ▶ Identifying anomalies ▶ Transactions ▶ Data exploration & finding relationships between fields ▶ Custom commands SPL Examples and Recipes
- 14. © 2017 SPLUNKINC. Eval – Just Getting Started! Splunk Search Quick Reference Guide
- 15. © 2017 SPLUNKINC. ▶ Find the needle in the haystack ▶ Charting statistics and predicting values ▶ Enriching and converging data sources ▶ Map geographic data in real time ▶ Identifying anomalies ▶ Transactions ▶ Data exploration & finding relationships between fields ▶ Custom commands SPL Examples and Recipes
- 16. © 2017 SPLUNKINC. Stats/Timechart – But Wait, There’s More! Splunk Search Quick Reference Guide
- 17. © 2017 SPLUNKINC. ▶ Find the needle in the haystack ▶ Charting statistics and predicting values ▶ Enriching and converging data sources ▶ Map geographic data in real time ▶ Identifying anomalies ▶ Transactions ▶ Data exploration & finding relationships between fields ▶ Custom commands SPL Examples and Recipes
- 18. © 2017 SPLUNKINC. Converging Data Sources Index Untapped Data: Any Source, Type, Volume Ask Any Question Application Delivery Security, Compliance and Fraud IT Operations Business Analytics Industrial Data and the Internet of Things On-Premises Private Cloud Public Cloud Storage Online Shopping Cart Telecoms Desktops Security Web Services Networks Containers Web Clickstreams RFID Smartphones and Devices Servers Messaging GPS Location Packaged Applications Custom Applications Online Services DatabasesCall Detail Records Energy Meters Firewall Intrusion Prevention
- 19. © 2017 SPLUNKINC. ▶ Find the needle in the haystack ▶ Charting statistics and predicting values ▶ Enriching and converging data sources ▶ Map geographic data in real time (Let’s add some viz!) ▶ Identifying anomalies ▶ Transactions ▶ Data exploration & finding relationships between fields ▶ Custom commands SPL Examples and Recipes
- 20. © 2017 SPLUNKINC. Custom Visualizations
- 21. © 2017 SPLUNKINC. ▶ Native charts and maps • Bar / Line / Area charts • Bubble / Scatter plots • Gauges • Maps • Single Value Displays • Tables ▶ Generalized to fit use cases across many different areas ▶ Can be customized to some extent to cover specific use cases Native Visualizations In Splunk 42
- 22. © 2017 SPLUNKINC. ▶ Many use cases require a more specific visualization ▶ Specific custom appearance ▶ Represent data where native visualizations are not suitable • You can Splunk everything! • We won’t be able to predict every possible use case • Still uses SPL to drive visualizations Custom Visualizations FTW!
- 23. © 2017 SPLUNKINC. ▶ Platform extensibility framework and API ▶ Targeted at internal and external developers with web development / JS skills and basic knowledge of the Splunk platform ▶ Developers can make use of any third party libraries (d3.js, three.js, highcharts.js, etc…) that run in the browser* * with minor adjustments, and if third party license permits such use Custom Visualizations
- 24. © 2017 SPLUNKINC. Custom Visualizations For Admins In-product • Packaged as an app! • Installed like any other app • Users can search for visualizations on Splunkbase and directly in the product Installation
- 25. © 2017 SPLUNKINC. ▶ Choose from potentially dozens of installed visualizations! ▶ Appears as a first-class citizen alongside native visualizations • Looks and works just like packaged native visualizations ▶ Customize functionality and appearance of the visualization without touching any code, straight from the UI ▶ SPL Example provided as you hover over each visualization option. Custom Visualizations How-to
- 26. © 2017 SPLUNKINC. New Splunk Visualizations Treemap Sankey Diagram Punchcard Calendar Heat Map Parallel Coordinates Bullet GraphLocation Tracker Horseshoe Meter Machine Learning Charts Timeline Horizon Chart Multiple use cases across IT, security, IoT, and business analytics
- 27. © 2017 SPLUNKINC. Box Plot 3D scatter plot New Partner/Community Visualizations Wordcloud Donut Chart Heat Map
- 28. © 2017 SPLUNKINC. New Partner/Community Visualizations Geo Heatmap Custom Cluster Map Clustered Single Value Map Missile Map
- 29. © 2017 SPLUNKINC. Custom Visualizations – Demo!
- 30. © 2017 SPLUNKINC. ▶ Find the needle in the haystack ▶ Charting statistics and predicting values ▶ Enriching and converging data sources ▶ Map geographic data in real time ▶ Identifying anomalies ▶ Transactions ▶ Data exploration & finding relationships between fields ▶ Custom commands SPL Examples and Recipes
- 31. © 2017 SPLUNKINC. ▶ Find the needle in the haystack ▶ Charting statistics and predicting values ▶ Enriching and converging data sources ▶ Map geographic data in real time ▶ Identifying anomalies ▶ Transactions ▶ Data exploration & finding relationships between fields ▶ Custom commands SPL Examples and Recipes
- 32. © 2017 SPLUNKINC. ▶ Find the needle in the haystack ▶ Charting statistics and predicting values ▶ Enriching and converging data sources ▶ Map geographic data in real time ▶ Identifying anomalies ▶ Transactions ▶ Data exploration & finding relationships between fields ▶ Custom commands SPL Examples and Recipes
- 33. © 2017 SPLUNKINC. Data Exploration | analyzefields | anomalies | arules | associate | cluster | contingency | correlate | fieldsummary
- 34. © 2017 SPLUNKINC. ▶ Predict Numeric Fields ▶ Predict Categorical Fields ▶ Detect Numerical Outliers ▶ Detect Categorical Outliers ▶ Forecast Time Series ▶ Cluster Events Machine Learning Toolkit and Showcase Examples
- 35. © 2017 SPLUNKINC. ▶ Find the needle in the haystack ▶ Charting statistics and predicting values ▶ Enriching and converging data sources ▶ Map geographic data in real time ▶ Identifying anomalies ▶ Transactions ▶ Data exploration & finding relationships between fields ▶ Custom commands SPL Examples and Recipes
- 36. © 2017 SPLUNKINC. ▶ What is a Custom Command? • “| haversine origin="47.62,-122.34" outputField=dist lat lon” ▶ Why do we use Custom Commands? • Run other/external algorithms on your Splunk data • Save time munging data (see Timewrap!) • Because you can! ▶ Create your own or download as Apps • Haversine (Distance between two GPS coords) • Timewrap (Enhanced Time overlay) • Levenshtein (Fuzzy string compare) • Base64 (Encode/Decode) Custom Commands
- 37. © 2017 SPLUNKINC. SPL & The Machine Learning Toolkit
- 38. © 2017 SPLUNKINC. ▶ Predict Numeric Fields ▶ Predict Categorical Fields ▶ Detect Numerical Outliers ▶ Detect Categorical Outliers ▶ Forecast Time Series ▶ Cluster Events Machine Learning Toolkit and Showcase Examples
- 39. © 2017 SPLUNKINC. Machine Learning with the Splunk Platform Visualize Share Clean Transform Operationalize Monitor Alert Build Model Search Explore Collect Data Test, Improve Models Ecosystem MLTK Choose Algorithm Ecosystem Splunk Splunk Splunk Splunk MLTK Splunk MLTK Splunk MLTK Splunk Ecosystem Splunk Real-time Data Science Pipeline Ecosystem MLTK Splunk Splunk’s App Ecosystem contains 1000’s of free add-ons for getting data in, applying structure and visualizing your data giving you faster time to value. The Machine Learning Toolkit delivers new SPL commands, custom visualizations, assistants, and examples to explore a variety of ml concepts. Splunk Enterprise is the mission-critical platform for indexing, searching, analyzing, alerting and visualizing machine data. Packaged: UBA, ITSI
- 40. © 2017 SPLUNKINC. ML SPL Visualize Share Correlate Clean Munge Operationalize Monitor Alert Build Model Search Explore Universal Indexing Test, Improve Models Ecosystem MLTK Choose Algorithm Ecosystem Splunk Splunk Splunk Splunk MLTK Splunk MLTK Splunk MLTK Splunk Ecosystem Splunk fit sample apply listmodels deletemodel summary eval rex stats eventstats streamstats table … timechart chart stats geostats geom sendalert sendemail table … MLTK Library predict (cmd) anomalydetection (cmd) analyzefields anomalies arules associate cluster contingency correlate fieldsummary …
- 41. © 2017 SPLUNKINC. MLTK Commands The Machine Learning Toolkit contains several custom search commands that implement classic machine learning and statistical learning tasks: • fit: Fit and apply a machine learning model to search results. • apply: Apply a machine learning model that was learned using the fit command. • summary: Return a summary of a machine learning model that was learned using the fit command. • listmodels: Return a list of machine learning models that were learned using the fit command. • deletemodel: Delete a machine learning model that was learned using the fit command. • sample: Randomly sample or partition events.
- 42. © 2017 SPLUNKINC. ML-SPL Demo
- 43. © 2017 SPLUNKINC. Set Up Before You Can Play Download the following at splunkbase.com ▶ Machine Learning Toolkit: • https://splunkbase.splunk.com/app/2890/ ▶ Python for Scientific Computing: • https://splunkbase.splunk.com/app/2881/ *Note – For the Python for Scientific Computing App you need to download the platform specific version – Mac, Linux, Windows
- 44. © 2017 SPLUNKINC. ▶ Additional information can be found in: • Power of SPL App! • Docs - Search Manual • Docs - MLTK Search Commands • MLTK Quick Reference Guide • Blogs • Answers • Exploring Splunk For More Information
- 45. © 2017 SPLUNKINC. • SPL Examples App Other Useful Apps to download! • Splunk 6.x Dashboard Examples • Splunk 6.5 Overview App
- 46. © 2017 SPLUNKINC. ▶ Benefit of using your own laptop is a ‘Take it with you after’ approach – and bang on it without messing up production Splunk • Promote Personalized 50GB 6 month Dev/Test license to use for Workshop (vs. 30 day download trial) • Encourages long term playing with Splunk and continuously testing out new data sources How about an extra 50GB?
- 47. © 2017 SPLUNKINC. Q & A
- 48. © 2017 SPLUNKINC.© 2017 SPLUNK INC. Thank You
Editor's Notes
- #2 This presentation has some animations and content to help tell stories as you go. Feel free to change ANY of this to your own liking! I would definitely practice your flow once or twice before a presentation. There is A LOT of content to get through in 1 hour. The slides with search examples can be unhidden if needed. Here is what you need for this presentation: You should have the following installed: PowerOfSPL App - https://splunkbase.splunk.com/app/3353/ Custom Cluster Map Visualization - https://splunkbase.splunk.com/app/3122/ Clustered Single Value Map Visualization - https://splunkbase.splunk.com/app/3124/ Geo Heatmap Custom Visualization - https://splunkbase.splunk.com/app/3217/ Timewrap Custom Command (NOTE this command is now included in CORE) - https://splunkbase.splunk.com/app/1645/ Haversine Custom Command - https://splunkbase.splunk.com/app/936/ Levenshtein Custom Command - https://splunkbase.splunk.com/app/1898/ Optional: Splunk Search Reference Guide handouts Mini buttercups or other prizes to give out for answering questions during the presentation Shake! Demo can be used for interactivity on some of these search examples if you want… definitely adds some flare to the presentation
- #4 Intro Mention to people to start downloading Splunk as it’s a large download if they haven’t already. If they already have Splunk installed than they just need to download the Power Of SPL App. How to get help (recommended to have 1-2 staffers in room to help with install problems etc)
- #5 Intro Mention to people to start downloading Splunk as it’s a large download if they haven’t already. If they already have Splunk installed than they just need to download the Power Of SPL App. How to get help (recommended to have 1-2 staffers in room to help with install problems etc)
- #7 Intro: 1. Get everyone set up 2. This will essentially be a follow along – with me demoing and you following along. After today’s session you’ll have a useful, live instance with everything we do today on it. 3. We’ll walk through a bunch of different SPL commands, how SPL works with Custom Visualizations and finally SPL with a light Machine Learning intro.
- #9 - We call them search commands, but they really do so much more and that’s what I hope to get across with you today. “The Splunk search language has over 140+ commands, is very expressive and can perform a wide variety of tasks ranging from filtering to data, to munging or modifying, and reporting.” “The Syntax was …” “Why? Because SQL is good for certain tasks and the Unix pipeline is amazing!” This is great BUT… WHY WOULD WE WANT TO CREATE A NEW LANGUAGE AND WHY DO YOU CARE?
- #10 <Engage audience here.. Before showing bullet points ask “Why do you think we would want to create a new language?”> <Also Feel free to change pictures or flow of this slide..> -- have buttercups to throw out if anyone answers correctly? - Today we require the ability to quickly search and correlate through large amounts of data, sometimes in an unstructured or semi-unstructured way. Conventional query languages (such as SQL or MDX) simply do not provide the flexibility required for the effective searching of big data. Not only this but STREAMING data. (SQL can be great at joining a bunch of small tables together, but really large joins on datasets can be a problem whereas hadoop can be great with larger data sets, but sometimes inefficient when it comes to many small files or datasets. ) - Machine Data is different: - It is voluminous unstructured time series data with no predefined schema - It is generated by all IT systems– from applications and servers, to networks and RFIDs. - It is non-standard data and characterized by unpredictable and changing formats Traditional approaches are just not engineered for managing this high volume, high velocity, and highly diverse form of data. Splunk’s NoSQL query approach does not involve or impose any predefined schema. This enables the increased flexibility mentioned above, as there are No limits on the formats of data – No limits on where you can collect it from No limits on the questions that you can ask of it And no limits on scale Methods of Correlation enabled by SPL Time & GeoLocation: Identify relationships based on time and geographic location Transactions: Track a series of events as a single transaction Subsearches: Results of one search as input into other searches Lookups: Enhance, enrich, validate or add context to event data SQL-like joins between different data sets In addition to flexible searching and correlation, the same language is used to rapidly construct reports, dashboards, trendlines and other visualizations. This is useful because you can understand and leverage your data without the cost associated with the formal structuring or modeling of the data first. (With hadoop or SQL you run a job or query to generate results, but then you have need to integrate more software to actually visualize it!) “OK.. Let’s move on..”
- #11 “Let’s take a closer look at the syntax, notice the unix pipeline” “The structure of SPL creates an easy way to stitch a variety of commands together to solve almost any question you may ask of your data.” “Search and Filter” - The search and filter piece allows you to use fields or keywords to reduce the data set. It’s an important but often overlooked part of the search due to the performance implications. “Munge” - The munge step is a powerful piece because you can “re-shape” data on the fly. In this example we show creating a new field called KB from an existing field “bytes”. “Report” - Once we’ve shaped and massaged the data we now have an abundant set of reporting commands that are used to visualize results through charts and tables, or even send to a third party application in whatever format they require. “Cleanup” - Lastly there are some cleanup options to help you create better labeling and add or remove fields. Again, sticthing together makes it easier to utilize and understand advanced commands, better flow etc. Additionally the implicit join on time and automatic granularity helps reduces complexity compared to what you would have to do in SQL and excel or other tools. “Let’s look at some more in depth examples”
- #13 “In this next section we’ll take a more in depth look at some search examples and recipes. It would be impossible for us to go over every command and use case so the goal of this is to show a few different commands that can help solve most problems and generate quick time to value in the following area."
- #14 “In this next section we’ll take a more in depth look at some search examples and recipes. It would be impossible for us to go over every command and use case so the goal of this is to show a few different commands that can help solve most problems and generate quick time to value in the following area."
- #15 Note how the search assistant shows the number of both exact and similar matched terms before you even click search. This can be very useful when exploring and previewing your data sets without having to run searches over and over again to find a result.
- #16 Additionally we can further filter our data set down to a specific host.
- #17 Lastly we can combine filters and keyword searches very easily. “This is pretty basic, but the key here is that SPL makes it incredibly easy and flexible to filter your searches down and reduce your data set to exactly what you’re looking for.
- #18 Remember Munging or Re-shaping our data on the fly? Talk about Eval and it’s importance sourcetype=access* |eval KB=bytes/1024
- #19 sourcetype=access* | eval http_response = if(status == 200, "OK", "Error”)
- #20 sourcetype=access* | eval connection = clientip.":".port
- #21 “There are tons of EVAL commands to help you shape or manipulate your data the way you want it.” Optional <Click on image to go to show and scroll through online quick reference quide>
- #26 Show difference between stats and timechart (adds _time buckets, visualize, etc.) Why is this awesome? We can do all of the same statistical calculations over time with almost any level of granularity. For example… <change timepicker from 60min to 15min, add span=1s to search and zoom in> Add below? Due to the implicit time dimension, it’s very easy to use timechart to visualize disparate data sets with varying time frequencies. SQL vs Timechart actual comparison?
- #27 Walk through trendline basic options
- #28 Walk through predict basic options “The timechart command plus other SPL commands make it very easy to visualize your data any way you want.”
- #31 “Again, don’t forget about the quick reference guide. There are many more statistical functions you can use with these commands on your data.”
- #33 Context is everything when it comes to building successful operational intelligence. When you are stuck analyzing events from a single data source at a time, you might be missing out on rich contextual information or new insights that other data sources can provide. Let’s take a quick look at a few powerful SPL commands that can help make this happen.
- #51 Make sure everyone is on the “4.1 Geographic” Dashboard. The bottom 3 visualizations will show errors. Each example headline can be clicked to go to the correct Splunkbase page for download. Have each user (including yourself) download each app (3 of them), install them from file, and restart Splunk to ensure correct functionality. Walkthrough the SPL required to generate each custom visualization. Point out the different commands used and why. Now let’s install one more … Search for Missile Map on Splunkbase Download and install Complete the following search to get the visualization to work (starting coordinates can be whatever you like) index=power_of_spl | stats count by clientip | iplocation clientip | dedup Country | eval start_lat="-104.99", start_lon="39.76" | Let’s add some of the custom options specific to this visualization: Make it animated Make it colored Make it pulse index=power_of_spl | stats count by clientip | iplocation clientip | dedup Country | tail 10 | eval start_lat="-104.99", start_lon="39.76", end_lat=lon, end_lon=lat | eval color=if(Country="Iran","#FF0000","") | eval pulse_at_start="false" | eval animate=if(Country="Iran","true","false") | table color, animate, pulse_at_start, start_lat start_lon end_lat end_lon
- #55 Walk through the dashboard which points out several uses of anomalydetection
- #57 sourcetype=access* | transaction JSESSIONID
- #58 sourcetype=access* | transaction JSESSIONID | stats min(duration) max(duration) avg(duration)
- #59 NOTE: Many transactions can be re-created using stats. Transaction is easy but stats is way more efficient and it’s a mapable command (more work will be distributed to the indexers). sourcetype=access* | stats min(_time) AS earliest max(_time) AS latest by JSESSIONID | eval duration=latest-earliest | stats min(duration) max(duration) avg(duration)
- #61 Feel free to change this and use your own story! “My interpretation of Data Exploration when it comes to Splunk is the process of characterizing and researching behavior of both existing and new data sources.” “ For example while you may have an existing data source you are already used to, but there still could be some unknown value in in terms of patterns, relationships between fields and rare events that could point you to new insights or help with predictive analytics. This capability gives you confidence to explore new data sources as well because you can quickly look for replacements and nuggets that stick out or help classify data. A friend once asked me to look at some biomedical data with DNA information. The vocabulary and field definitions were way above me, but I was able to quickly understand patterns and relationships with Splunk and provide them value instaneously. With Splunk you literally become afraid of no data!” Let’s look at a few quick examples.
- #62 “The cluster command is used to find common and/or rare events within your data” <Show simple table search first and point out # of events, then run cluster and sort on cluster count to show common vs rare events> * | table _raw _time * | cluster showcount=t t=.1 | table _raw cluster_count | sort - cluster_count
- #63 Fieldsummary gives you a quick breakdown of your numerical fields such as count, min, max, stdev, etc. It also shows you examples values in the event. I used maxvals to limit the number of samples it shows per field. sourcetype=access_combined | fields – date* source* time* | fieldsummary maxvals=5
- #64 “The correlate command is used to find co-occurrence between fields. Basically a matrix showing the ‘Field1 exists 80% of the time when Field2 exists’” sourcetype=access_combined | fields – date* source* time* | correlate “This can be useful for both making sure your field extractions are correct (if you expect a field to exist %100 of the time when another field exists) and also helping you identify potential patterns and trends between different fields.”
- #65 “The contingency command is used to look for relationships of between two fields. Basically for these two fields, how many different value combinations are there and what are they / most common” sourcetype=access_combined | contingency uri status
- #66 This command is extremely useful for not only looking for meaningful fields in your data, but also for determining which fields to use in linear or logistical regression algorithms in the machine learning app. sourcetype=access_combined | analyzefields classfield=status
- #67 These commands are not only useful for learning about the patterns and characteristics of your data, but they are a stepping stone into machine learning as well. If you want to learn more about Data Science, Exploration and Machine Learning, download the Machine Learning App! You’ll use new SPL commands like “fit” and “apply” to train models on data in Splunk. We will take a look at this in more detail after the last SPL section, custom commands. New SPL commands: fit, apply, summary, listmodels, and deletemodel * Predict Numeric Fields (Linear Regression): e.g. predict median house values. * Predict Categorical Fields (Logistic Regression): e.g. predict customer churn. * Detect Numeric Outliers (distribution statistics): e.g. detect outliers in IT Ops data. * Detect Categorical Outliers (probabilistic measures): e.g. detect outliers in diabetes patient records. * Forecast Time Series: e.g. forecast data center growth and capacity planning. * Cluster Events (K-means, DBSCAN, Spectral Clustering, BIRCH).
- #68 Let’s talk about custom commands before we get into Machine Learning.
- #69 Only need to go through one example here, I usually just do haversine. “We’ve gone over a variety of Splunk search commands.. but what happens when we can’t find a command that fits our needs OR want to use a complex algorithm someone already OR even create your own?? Enter Custom Commands.” Additional Text: Splunk's search language includes a wide variety of commands that you can use to get what you want out of your data and even to display the results in different ways. You have commands to correlate events and calculate statistics on your results, evaluate fields and reorder results, reformat and enrich your data, build charts, and more. Still, Splunk enables you to expand the search language to customize these commands to better meet your needs or to write your own search commands for custom processing or calculations.
- #70 Let’s see Haversine in action. <Pull up search>
- #71 *Note – Coordinates of origin in this Haversine example is currently “Seattle”
- #73 In this final section today we’re going to do a brief overview of the ML Toolkit. More specifically since this whole workshop is based on SPL, we’ll focus in on the SPL aspects of Machine Learning. First things first, Let’s install it!
- #74 Collect Data: Correlate, Clean Munge: Search & Explore: Choose Algorithm: Build Models: Test, Improve Models: Operationalize, Monitor, Alert: Visualize & Share:
- #75 The purpose of this slide is to point out some of the SPL commands and how they contribute or can be applied to a Machine Learning use case or process every step of the way. Key Points: - We’ve gone over some of these commands already, but here’s where they fit in the ML process, from cleaning, to stats, etc. - Some of these commands such as predict and anomalydetection are part of core Splunk - As of today, there are 6 commands specific too and added by the MLTK – all highlighted in blue - Let’s go over them briefly
- #77 The Demo will show each of the 6 commands in use. I decided to use the ”Predict Categorical Fields” and the ”Predict Car Make and Model” example as it’s very easy to understand. Feel free to use any other example you prefer. The backup slides below are of this example. The next slide contains the directions to help everyone install the MLTK so they can follow along. Once everyone is ready to go, explain the Showcase at a high level before drilling into the example. You’ll go in the following order and can access most of the SPL just by using the assistant. 1. fit Start on the Showcase tab. Click “Predict Vehicle Make and Model” under “Predict Categorical Fields”. Wait for page to load. Explain the basic operations of the page. Then click on “Show SPL” next to fit model to show the SPL portion. Then show them how to open this in search. 2. sample Once You’ve opened this in search, remove all of the SPL except for the inputlookup statement. Notice the 50,000 results. Now add | sample ratio=.12 to the query and show how the results have been cut down and is a randomized sample. 3. apply Go back to the assistant. Scroll down to look at the prediction results and other statistical information. Hover over the helpers and explain some of the features. Now click “open in search” button in the “Prediction Results Table”. This will show the Apply command in action. Point out how there is a new field called “predicted(vehicleType)” 4. listmodels Remove all of the SPL and type “| listmodels”. Explain how this lists all of the models you’ve created using the fit command. 5. summary Remove all of the SPL and type “| summary example_vehicle_type”. Explain how this shows additional detail about the model. The results and what is shown depends on what algorithm has been used. This is a good time to play with other examples in the MLTK to see the differences. 6. deletemodel Remove all of the SPL and type “| deletemodel example_vehicle_type”. This will simply delete models.
- #78 Again you can either install through the App Management GUI using “Browse for Apps” or you can download them from Splunkbase and choose ”Install from File” You’ll most likely have to restart after installing the apps. (You can wait to restart until you’ve installed both of them)
- #85 References: Make sure to reference this App is now available for download!!
- #86 References: Make sure to reference this App is now available for download!!