This session will unveil the power of the Splunk Search Processing Language (SPL). See how to use Splunk's simple search language for searching and filtering through data, charting statistics and predicting values, converging data sources and grouping transactions, and finally data science and exploration. We'll begin with basic search commands and build up to more powerful advanced tactics to help you harness your Splunk Fu!

1. Copyright © 2015 Splunk Inc.
Power of Splunk
Search Processing Language
(SPL™)
Brian Heffernan, CISSP
heff@splunk.com

2. About Me
• My name is Brian Heffernan
and I have Splunked:
• BBQ Smokers - predicted
cook times
• Nest Thermostats
• Alerts when my wife
turns on the heat
• Children’s Browsing
History
• Splunking for 4+ years
• Northeastern University2

3. Rules & Agendah
Goal: Get Gooder!
 Ask Questions- there will be prizes
 Don’t take notes – I will provide slides
 No texting and searching
 No Sleeping – I will make fun of you.
• Overview & Anatomy of a Search
– Quick refresher on search language and
structure
• SPL Commands and Examples
– Doing More with Less
– Searching, charting, converging,
transactions, anomalies, exploring
• Custom Commands
– Doing Less with More
– Extend the capabilities of SPL
• Q&A

4. SPL Overview

5. SPL Overview
● Over 140+ search commands
● Syntax was originally based upon the Unix pipeline and SQL
and is optimized for time series data
● The scope of SPL includes data searching, filtering, modification, manipulation,
enrichment, insertion and deletion
● Includes anomaly detection and machine learning
5

6. Why Create a New Query Language?
● Flexibility and
effectiveness on
small and big data
● Late-binding schema
● More/better methods
of correlation
● Not just analyze, but
visualize
6
Data
BIG Data

7. new pipe = new line + space + pipe
search and filter | munge | report | cleanup
| rename sum(KB) AS "Total KB" dc(clientip) AS "Unique Customers"
| eval KB=bytes/1024
sourcetype=access*
| stats sum(KB) dc(clientip)
SPL Basic Structure
Bonus
Points

8. SPL Examples

9. SPL Examples and Recipes
● Search and filter + creating/modifying fields
● Charting statistics and predicting values
● Converging data sources
● Identifying transactions and anomalies
● Data exploration & finding relationships between fields
9

10. SPL Examples and Recipes
● Search and filter + creating/modifying fields
● Charting statistics and predicting values
● Converging data sources
● Identifying transactions and anomalies
● Data exploration & finding relationships between fields
10

11. Search and Filter
Examples
● Keyword search:
sourcetype=access* http
● Filter:
sourcetype=access* http
host=webserver-02
● Combined:
sourcetype=access* http
host=webserver-02 (503 OR 504)
11

12. Search and Filter
Examples
● Keyword search:
sourcetype=access* http
● Filter:
sourcetype=access* http
host=webserver-02
● Combined:
sourcetype=access* http
host=webserver-02 (503 OR 504)
12

13. Search and Filter
Examples
● Keyword search:
sourcetype=access* http
● Filter:
sourcetype=access* http
host=webserver-02
● Combined:
sourcetype=access* http
host=webserver-02 (status=503 OR
status=504)
13

14. Eval – Modify or Create New Fields and Values
Examples
● Calculation:
sourcetype=access*
|eval KB=bytes/1024
● Evaluation:
sourcetype=access*
| eval http_response =
if(status != 200, ”Error", ”OK”)
● Concatenation:
sourcetype=access*
| eval connection = device.“-“.clientip
14

15. Eval – Modify or Create New Fields and Values
Examples
● Calculation:
sourcetype=access*
|eval KB=bytes/1024
● Evaluation:
sourcetype=access*
| eval http_response =
if(status != 200, ”Error", ”OK”)
● Concatenation:
sourcetype=access*
| eval connection = device.“-“.clientip
15

16. Eval – Modify or Create New Fields and Values
Examples
● Calculation:
sourcetype=access*
|eval KB=bytes/1024
● Evaluation:
sourcetype=access*
| eval http_response =
if(status != 200, ”Error", ”OK”)
● Concatenation:
sourcetype=access*
| eval connection = clientip.“-”.device
16

17. Eval – Just Getting Started!
Splunk Search Quick Reference Guide
17

18. SPL Examples and Recipes
● Search and filter + creating/modifying fields
● Charting statistics and predicting values
● Converging data sources
● Identifying transactions and anomalies
● Data exploration & finding relationships between fields
18

19. Stats, Chart, Timechart
19

20. Stats – Calculate Statistics Based on Field Values
Examples
● Calculate stats and rename
sourcetype=netapp:perf
| stats avg(read_ops) AS “Read OPs”
● Multiple statistics
sourcetype=netapp:perf
| stats avg(read_ops) AS Read_OPs
sparkline(avg(read_ops)) AS Read_Trend
● By another field
Sourcetype=netapp:perf
| stats avg(read_ops) AS Read_OPs
sparkline(avg(read_ops)) AS Read_Trend
by instance
20

21. Stats – Calculate Statistics Based on Field Values
Examples
21
● Calculate stats and rename
sourcetype=netapp:perf
| stats avg(read_ops) AS “Read OPs”
● Multiple statistics
sourcetype=netapp:perf
| stats avg(read_ops) AS Read_OPs
sparkline(avg(read_ops)) AS Read_Trend
● By another field
Sourcetype=netapp:perf
| stats avg(read_ops) AS Read_OPs
sparkline(avg(read_ops)) AS Read_Trend
BY instance

22. Stats – Calculate Statistics Based on Field Values
Examples
22
● Calculate stats and rename
sourcetype=netapp:perf
| stats avg(read_ops) AS “Read OPs”
● Multiple statistics
sourcetype=netapp:perf
| stats avg(read_ops) AS Read_OPs
sparkline(avg(read_ops)) AS Read_Trend
● By another field
Sourcetype=netapp:perf
| stats avg(read_ops) AS Read_OPs
sparkline(avg(read_ops)) AS Read_Trend
BY instance

23. Timechart – Visualize Statistics Over Time
Examples
● Visualize stats over time
sourcetype=netapp:perf
| timechart avg(read_ops)
● Add a trendline
sourcetype=netapp:perf
| timechart avg(read_ops) AS
read_ops | trendline sma5(read_ops)
● Add a prediction overlay
sourcetype=netapp:perf
| timechart avg(read_ops) AS
read_ops | predict read_ops
23

24. Timechart – Visualize Statistics Over Time
Examples
24
● Visualize stats over time
sourcetype=netapp:perf
| timechart avg(read_ops)
● Add a trendline
sourcetype=netapp:perf
| timechart avg(read_ops) AS
read_ops | trendline sma5(read_ops)
● Add a prediction overlay
sourcetype=netapp:perf
| timechart avg(read_ops) AS
read_ops | predict read_ops

25. Timechart – Visualize Statistics Over Time
Examples
25
● Visualize stats over time
sourcetype=netapp:perf
| timechart avg(read_ops)
● Add a trendline
sourcetype=netapp:perf
| timechart avg(read_ops) AS
read_ops | trendline sma5(read_ops)
● Add a prediction overlay
sourcetype=netapp:perf
| timechart avg(read_ops) AS
read_ops | predict read_ops

26. Stats/Timechart – But Wait, There’s More!
Splunk Search Quick Reference Guide
26

27. SPL Examples and Recipes
● Search and filter + creating/modifying fields
● Charting statistics and predicting values
● Converging data sources
● Identifying transactions and anomalies
● Data exploration & finding relationships between fields
27

28. 28
Converging Data Sources
Index Untapped Data: Any Source, Type, Volume
Online
Services Web
Services
Servers
Security GPS
Location
Storage
Desktops
Networks
Packaged
Applications
Custom
ApplicationsMessaging
Telecoms
Online
Shopping
Cart
Web
Clickstreams
Databases
Energy
Meters
Call Detail
Records
Smartphones
and Devices
RFID
On-
Premises
Private
Cloud
Public
Cloud
Ask Any Question
Application Delivery
Security, Compliance,
and Fraud
IT Operations
Business Analytics
Industrial Data and
the Internet of Things

29. Converging Data Sources
Examples
● Implicit join on time
index=* http | timechart count BY
sourcetype
● Enrich data with lookup
sourcetype=access_combined status=503
| lookup customer_info uid |
stats count BY customer_value
● Append results from another
search
… | appendcols [search earliest=-1h
sourcetype=Kepware units=W row=A
| stats stdev(Value) AS hr_stdev] …
29

30. Lookup – Converging Data Sources
Examples
30
● Implicit join on time
index=* http | timechart count by
sourcetype
● Enrich data with lookup
sourcetype=access_combined status=503
| lookup customer_info uid |
stats count by customer_value
● Append results from another
search
… | appendcols [search earliest=-1h
sourcetype=Kepware units=W row=A
| stats stdev(Value) AS hr_stdev] …

31. Appendcols – Converging Data Sources
Examples
31
● Implicit join on time
index=* http | timechart count BY
sourcetype
● Enrich data with lookup
sourcetype=access_combined status=503
| lookup customer_info uid
| stats count BY customer_value
● Append results from another
search
… | appendcols [search earliest=-1h
sourcetype=Kepware units=W row=A
| stats stdev(Value) AS hr_stdev] …

32. SPL Examples and Recipes
● Search and filter + creating/modifying fields
● Charting statistics and predicting values
● Converging data sources
● Identifying transactions and anomalies
● Data exploration & finding relationships between fields
32

33. Transaction – Group Related Events Spanning Time
Examples
● Group by session ID
sourcetype=access*
| transaction JSESSIONID
● Calculate session durations
sourcetype=access*
| transaction JSESSIONID
| stats min(duration) max(duration)
avg(duration)
● Stats is better
sourcetype=access*
| stats min(_time) AS earliest max(_time)
AS latest by JSESSIONID
| eval duration=latest-earliest
| stats min(duration) max(duration)
avg(duration)
33

34. Transaction – Group Related Events Spanning Time
Examples
34
● Group by session ID
sourcetype=access*
| transaction JSESSIONID
● Calculate session durations
sourcetype=access*
| transaction JSESSIONID
| stats min(duration) max(duration)
avg(duration)
● Stats is better
sourcetype=access*
| stats min(_time) AS earliest max(_time)
AS latest by JSESSIONID
| eval duration=latest-earliest
| stats min(duration) max(duration)
avg(duration)

35. Transaction – Group Related Events Spanning Time
Examples
35
● Group by session ID
sourcetype=access*
| transaction JSESSIONID
● Calculate session durations
sourcetype=access*
| transaction JSESSIONID
| stats min(duration) max(duration)
avg(duration)
● Stats is better
sourcetype=access*
| stats min(_time) AS earliest max(_time)
AS latest by JSESSIONID
| eval duration=latest-earliest
| stats min(duration) max(duration)
avg(duration)

36. Anomaly Detection – Find anomalies in your data
Examples
36
● Find anomalies
| inputlookup car_data.csv |
anomalydetection
● Summarize anomalies
| inputlookup car_data.csv |
anomalydetection action=summary
● Use IQR and remove outliers
| inputlookup car_data.csv |
anomalydetection method=iqr
action=remove

37. SPL Examples and Recipes
● Search and filter + creating/modifying fields
● Charting statistics and predicting values
● Converging data sources
● Identifying transactions and anomalies
● Data exploration & finding relationships between fields
37

38. Data Exploration
| analyzefields
| anomalies
| arules
| associate
| cluster
| contingency
| correlate
| fieldsummary
38

39. Cluster – Exploring Your Data
Examples
● Find most/least common events
* | cluster showcount=t t=.1
| table _raw cluster_count
● Display Summary of Fields.
sourcetype=access_combined
| fields – date* source* time*
| fieldsummary maxvals=5
● Show patterns of co-occurring fields.
sourcetype=access_combined
| fields – date* source* time* | correlate
● View field relationships
sourcetype=access_combined
| contingency uri status
● Find predictors of fields
sourcetype=access_combined
| analyzefields classfield=status
39

40. Cluster – Exploring Your Data
Examples
● Find most/least common events
* | cluster showcount=t t=.1
| table _raw cluster_count
● Display Summary of Fields.
sourcetype=access_combined
| fields – date* source* time*
| fieldsummary maxvals=5
● Show patterns of co-occurring fields.
sourcetype=access_combined
| fields – date* source* time* | correlate
● View field relationships
sourcetype=access_combined
| contingency uri status
● Find predictors of fields
sourcetype=access_combined
| analyzefields classfield=status
40

41. Correlate – Exploring Your Data
Examples
● Find most/least common events
* | cluster showcount=t t=.1
| table _raw cluster_count
● Display Summary of Fields.
sourcetype=access_combined
| fields – date* source* time*
| fieldsummary maxvals=5
● Show patterns of co-occurring fields.
sourcetype=access_combined
| fields – date* source* time* | correlate
● View field relationships
sourcetype=access_combined
| contingency uri status
● Find predictors of fields
sourcetype=access_combined
| analyzefields classfield=status
41

42. Contingency – Exploring Your Data
Examples
● Find most/least common events
* | cluster showcount=t t=.1
| table _raw cluster_count
● Display Summary of Fields.
sourcetype=access_combined
| fields – date* source* time*
| fieldsummary maxvals=5
● Show patterns of co-occurring fields.
sourcetype=access_combined
| fields – date* source* time* | correlate
● View field relationships
sourcetype=access_combined
| contingency uri status
● Find predictors of fields
sourcetype=access_combined
| analyzefields classfield=status
42

43. Analyzefields – Exploring Your Data
Examples
● Find most/least common events
* | cluster showcount=t t=.1
| table _raw cluster_count
● Display Summary of Fields.
sourcetype=access_combined
| fields – date* source* time*
| fieldsummary maxvals=5
● Show patterns of co-occurring fields.
sourcetype=access_combined
| fields – date* source* time* | correlate
● View field relationships
sourcetype=access_combined
| contingency uri status
● Find predictors of fields
sourcetype=access_combined
| analyzefields classfield=status
43

44. Custom Commands

45. Custom Commands
● What is a Custom Command?
– “| haversine origin="47.62,-122.34" outputField=dist lat lon”
● Why do we use Custom Commands?
– Run other/external algorithms on your Splunk data
– Save time munging data (see Timewrap!)
– Because you can!
● Create your own or download as Apps
– Haversine (Distance between two GPS coordinates)
– Timewrap (Enhanced Time overlay)
– Levenshtein (Fuzzy string compare)
– R Project (Utilize R!)
45

46. Custom Commands to the rescue!
46

47. Custom Commands – Haversine
Examples
● Download and install App
Haversine
● Read documentation then
use in SPL!
sourcetype=access*
| iplocation clientip
| search City=A*
| haversine origin="47.62,-122.34"
units=mi outputField=dist lat lon
| table clientip, City, dist, lat, lon
47

48. Custom Commands – Haversine
Examples
● Download and install App
Haversine
● Read documentation then
use in SPL!
sourcetype=access*
| iplocation clientip
| search City=A*
| haversine origin="47.62,-122.34"
units=mi outputField=dist lat lon
| table clientip, City, dist, lat, lon
48

49. Tools
 Add Splunk Search to Chrome:
It is a cool way to search for a .conf or a Splunk command directly from the Chrome search bar. Install the Chrome Extension, and then add the
following commands under the Chrome/Settings/Manage Search Engines
1- Add "Bookmark Search": https://chrome.google.com/webstore/detail/bookmark-search/hhmokalkpaiacdofbcddkogifepbaijk?utm_source=gmail
2-Add the following Search Engines
Splunk Apps SA: https://splunkbase.splunk.com/apps/#/order/relevance/search/%s
Splunk Conf SC: http://docs.splunk.com/Documentation/Splunk/latest/admin/%sconf
Splunk Search Commands SS: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/%s
Splunk Docs SD: http://docs.splunk.com/Special:SplunkSearch/docs?q=%s

51. Machine Learning Toolkit and Showcase
Examples
● Predict Numeric Fields
● Predict Categorical Fields
● Detect Numerical Outliers
● Detect Categorical Outliers
● Forecast Time Series
● Cluster Events
51

52. Splunk 6.x Dashboard Examples

53. For More Information
● Additional information can be
found in:
– Search Manual
– Blogs
– Answers
– Operational Intelligence
Cookbook
– Exploring Splunk
53

54. Thank you!