The document provides workshop notes for presenters on conducting a Splunk workshop, focusing on hands-on use of Apache access logs and data preparation instructions. It emphasizes best practices for presenting, including indexing data correctly, using appropriate technology tools, and providing comprehensive resources for attendees. The notes also cover the functionalities of Splunk for machine data handling, data sources, and integration with various platforms and applications.

1. © 2017 SPLUNK INC.
Workshop Notes for Presenter
Tip #1: The hands-on portion of the workshop uses Apache access logs
exported from the Splunk Oxygen Application Management demo.
A 4-hour sample is plenty of data (~12 MB).
Tip #2: Be sure to supply this data sample along with instructions on
downloading Splunk to the attendees several days before the workshop.

2. © 2017 SPLUNK INC.
How to for the Presenter

3. © 2017 SPLUNK INC.
Workshop Notes for Presenter
Tip #3: Index the sample data set as sourcetype=access_combined.
All the hands-on examples use fields and constructs based on this sourcetype.
Tip #4: Use an iPad with the presenter version to keep track of where you are
in the workshop. Remember to set the screen timeout to something high. :)

4. © 2017 SPLUNK INC.
Workshop Notes for Presenter
Tip #5: Show
Any slides with the “Show” comment (like this one) are things you should
do in Splunk. In other words, show them the step within Splunk.
SHOW

5. © 2017 SPLUNK INC.
Welcome

6. © 2017 SPLUNK INC.
Download Splunk
www.splunk.com > Free
Splunk > Splunk Enterprise
Download Data Sample and Lookup
https://splunk.box.com/v/MD101Workshop
1
2
1

7. © 2017 SPLUNK INC.© 2017 SPLUNK INC.
Machine Data Workshop 101
Beyond the Basics
Jon Yost | Sales Engineer | Florida
7/13/17

8. © 2017 SPLUNK INC.
Splunk Approach to Machine Data
SQL Search
Schema at Write Schema at Read
Traditional Splunk
ETL Universal Indexing
Volume Velocity Variety
UnstructuredStructured
RDBMS

9. © 2017 SPLUNK INC.
Industry Leading Platform For Machine Data
Custom
dashboards
Report and
analyze
Monitor
and alert
Developer
Platform
Ad hoc
search
On-Premises
Private Cloud
Public
Cloud
Storage
Online
Shopping Cart
Telecoms
Desktops
Security
Web
Services
Networks
Containers
Web
Clickstreams
RFID
Smartphones
and Devices
Servers
Messaging
GPS
Location
Packaged
Applications
Custom
Applications
Online
Services
DatabasesCall Detail
Records
Energy MetersFirewall
Intrusion
Prevention
Platform Support (Apps / API / SDKs)
Enterprise Scalability
Universal Indexing
Machine Data: Any Location, Type, Volume Answer Any Question
Any Amount, Any Location, Any Source
Schema
on-the-fly
Universal
indexing
No
back-end
RDBMS
No need
to filter
data

10. © 2017 SPLUNK INC.
The Splunk Portfolio
Rich Ecosystem of
Apps & Add-Ons
Splunk Premium
Solutions
Mainframe
Data
Relational
Databases
MobileForwarders
Syslog/
TCP
IoT
Devices
Network
Wire Data
Hadoop
Platform for Operational Intelligence

11. © 2017 SPLUNK INC.
▶ Data Sources
▶ Data Enrichment
▶ Level Up on Search and Reporting Commands
▶ Data Models and Pivot
▶ Custom Visualizations and the Web Framework
Agenda

12. © 2017 SPLUNK INC.
Workshop Setup

13. © 2017 SPLUNK INC.
Download Splunk or Sign Up For Splunk Cloud
www.splunk.com > Free Splunk > Splunk Enterprise or Splunk Cloud
1
2
3

14. © 2017 SPLUNK INC.
▶ Box > access_datasample_last4h.log
▶ Box > http_status.csv
Download Data Sample and Lookup
https://splunk.box.com/v/MD101Workshop

15. © 2017 SPLUNK INC.
▶ Browser: http://localhost:8000
▶ Default username/password is admin/changeme
Index Data SampleSHOW
1
2

16. © 2017 SPLUNK INC.
Index Data SampleSHOW
3
2
1
4
5

17. © 2017 SPLUNK INC.
Index Data SampleSHOW
1
2

18. © 2017 SPLUNK INC.
Index Data SampleSHOW
1
2
You will need to refresh
the search after a few
moments for all events
to show up

19. © 2017 SPLUNK INC.
12.130.60.4 - - [18/Sep/2014 05:26:50:193] "GET
/product.screen?product_id=AV-CB-01&JSESSIONID=SD8SL4FF8ADFF5
HTTP 1.1" 200 3221
"http://www.myflowershop.com/category.screen?category_id=BOUQUET
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"634
▶ Keyword searching
▶ Interesting fields sourcetype=access_combined
▶ Field extractions + why are they important
• IFX | rex | auto kv through app logging best practices
▶ Filters
Quick UI/Search OrientationSHOW
clientip method url
bytes xfered
status return code
user agent

20. © 2017 SPLUNK INC.
▶ Data discovery
▶ Group like events
▶ Save as event type
▶ Create alert
Pattern DetectionSHOW
Back to
Slides

21. © 2017 SPLUNK INC.
Data Sources

22. © 2017 SPLUNK INC.
▶ Captures events from log files in real time
▶ Runs scripts to gather system metrics,
connect to APIs and databases
▶ Listens to syslog and gathers Windows events
▶ Universally indexes any data format so it
doesn’t need adapters
Traditional Data Sources
Windows
• Registry
• Event logs
• File system
• sysinternals
Linux/Unix
• Configurations
• Syslog
• File system
• Ps, iostat, top
Virtualization
• Hypervisor
• Guest OS
• Guest Apps
Applications
• Web logs
• Log4J, JMS, JMX
• .NET events
• Code and scripts
Databases
• Configurations
• Audit/query logs
• Tables
• Schemas
Network
• Configurations
• syslog
• SNMP
• netflow

23. © 2017 SPLUNK INC.
▶ Network Inputs
▶ HTTP Event Collector
▶ Log Event Alert Action
▶ Splunk Stream
▶ Scripted Inputs
▶ Database Inputs
▶ Splunk ODBC Driver
▶ Modular Inputs
▶ zLinux Forwarder
▶ MINT
▶ Non-Splunk Datastores
Non-Traditional Data Sources

24. © 2017 SPLUNK INC.
▶ Collect data over any UDP or TCP port
• Some devices only send data over a network port
▶ Best Practice: use syslog-ng or rsyslog
• Offers persistence
• Categorizes data by host
Network Inputs

25. © 2017 SPLUNK INC.
▶ Collect data over HTTP or HTTPS directly to Splunk
• Application Developer focus – few lines of code in app to send data
▶ HEC Features Include:
• Token-based, not credential based
• Indexer Acknowledgements – guarantees data indexing
• Raw and JSON formatted event payloads
• SSL, CORS (Cross-Origin Resource Sharing), and Network Restrictions
HTTP Event Collector (HEC)

26. © 2017 SPLUNK INC.
▶ Use Splunk alerting to index a custom log event
• Splunk searchable index of custom alert events
▶ Configurable Features Include:
• Host
• Source
• Sourcetype
• Index
• Event text – construct the exact syntax of the log event, including
any text, tokens, or other information
Log Event Alert Action

27. © 2017 SPLUNK INC.
Wire Data Enhances the Platform
for Operational Intelligence
Efficient, Cloud-Ready Wire
Data Collection
Simple Deployment Supports
Fast Time to Value
The Splunk Stream
Log Files Configurations Wire Data Alerts Metrics Scripts Changes Tickets
Sensors Security Custom
Applications
Networks Databases Servers Smartphones
and Devices
Web
Services
Virtual
Machines

28. © 2017 SPLUNK INC.
Solution Area Contextual Data Wire Data Enriched View
Application Management application logs,
monitoring data,
metrics, events
protocol conversations on
database performance, DNS
lookups, client data,
business transaction
paths…
Measure application
response times, deeper
insights for root-cause
diagnostics, trace tx paths,
establish baselines…
IT Operations application logs,
monitoring data,
metrics, events
payload data including
process times, errors,
transaction traces, ICA
latency, SQL statements,
DNS records…
Analyze traffic volume,
speed and packets to
identify infrastructure
performance issues,
capacity constraints,
changes; establish
baselines…
Stream = Better Insights for *

29. © 2017 SPLUNK INC.
Solution Area Contextual Data Wire Data Enriched View
Security app + infra logs, monitoring
data, events
protocol identification,
protocol headers, content
and payload information,
flow records
Build analytics and context
for incident response, threat
detection, monitoring and
compliance
Digital Intelligence website activity,
clickstream data, metrics
browser-level customer
interactions
Customer Experience –
analyze website and application
bottlenecks to improve customer
experience and online revenues
Customer Support (online,
call center) – faster root cause
analysis and resolution of customer
issues with website or apps
Stream = Better Insights for *

30. © 2017 SPLUNK INC.
▶ Send data to Splunk via a custom script
• Splunk indexes anything written to stdout
• Splunk handles scheduling
• Supports shell, Python scripts, WIN batch, PowerShell
• Any other utility that can format and stream data
Scripted Inputs
Streaming Mode
• Splunk executes script and indexes stdout
• Checks for any running instances
Write to File Mode
• Splunk launches script which produces
output file, no need for external scheduler
• Splunk monitors output file

31. © 2017 SPLUNK INC.
▶ Alternative to file-base or network-based inputs
▶ Stream data from command-line tools, such as vmstat and iostat
▶ Poll a web service, API or database and process the results
▶ Reformat complex or binary data for easier parsing into events and fields
▶ Maintain data sources with slow or resource-intensive startup procedures
▶ Provide special or complex handling for transient or unstable inputs
▶ Scripts that manage passwords and credentials
▶ Wrapper scripts for command line inputs that contain special characters
Use Cases for Scripted Inputs

32. © 2017 SPLUNK INC.
▶ DB Connect provides reliable, scalable,
real-time integration between Splunk and
traditional relational databases
• Create value with structured data
• Enrich search results with additional business context
• Easily import data for deeper analysis
• Integrate multiple DBs concurrently
• Simple set-up, non-invasive and secure
Database Inputs
DB CONNECT
JRE
JDBC
DATABASE DRIVER
DATABASE

33. © 2017 SPLUNK INC.
▶ DB Connect App
• Real-time, scalable integration with relational DBs
• Browse and navigate schemas and tables before data import
• Reliable scheduled import
• Seamless installation and UI configuration
• Supports connection pooling and caching
▶ “Tail” tables or import entire tables
• Detect and import new/updated rows using timestamps or unique IDs
▶ Supports many RDBMS flavors
• AWS RDS Aurora, AWS RedShift, IBM DB2 for Linux, Informix, MemSQL, MS SQL, MySQL,
Oracle, PostgreSQL, SAP SQL Anywhere (aka Sybase SA), Sybase ASE and IQ, Teradata
Configure Database Inputs

34. © 2017 SPLUNK INC.
▶ Interact with, manipulate and visualize machine data in Splunk Enterprise using
business software tools
▶ Leverage analytics from Splunk alongside Microsoft Excel, Tableau Desktop or
Microstrategy Analytics Desktop
▶ Industry-standard connectivity to Splunk Enterprise
▶ Empowers business users with direct and secure access to machine data
▶ Combine machine data with structured data for better operational context
Splunk ODBC Driver

35. © 2017 SPLUNK INC.
ODBC: How it Works
Splunk AdminAnalyst
Step 3:
Business Analyst uses Microsoft Excel, Tableau or Mocrostrategy to access
Data Models and saved searches and retrieve machine data from Splunk Enterprise
Step 2:
Splunk Admin authors Data Models or saved
searches in Splunk Enterprise
Step 1:
Business Analyst communicates data
requirements to Splunk Admin
Analyst
REQUIREMENTS
Saved Searches
or Data Models
Tableau or MS Excel
or Microstrategy ODBC Driver

36. © 2017 SPLUNK INC.
▶ Create your own custom inputs
• Scripted input with structure and intelligence
• First class citizen in the Splunk management interface
• Appears under Settings > Data Inputs
▶ Benefits over simple scripted input
• Instance control: launch a single instance or multiple instances
• Input validation
• Support multiple platforms
• Stream data as text or XML
• Secure access to mod input scripts via REST endpoints
Modular Inputs

37. © 2017 SPLUNK INC.
▶ Twitter
• Stream JSON data from a Twitter source to Splunk using Tweepy
▶ Amazon S3 Online Storage
• Index data from the Amazon S3 online storage web service
▶ Java Messaging Service (JMS)
• Poll message queues and topics through JMS Messaging API
• Talks to multiple providers: MQSeries (Websphere MQ), ActiveMQ,
TibcoEMS, HornetQ, RabbitMQ, Native JMS, WebLogic JMS, Sonic MQ
▶ Splunk Windows Inputs
• Retrieve WIN event logs, registry keys, perfmon counters
Example Modular Inputs

38. © 2017 SPLUNK INC.
More Modular Inputs

39. © 2017 SPLUNK INC.
▶ Easily collect and index data on IBM mainframes
▶ Collect application and platform data
▶ Download as new Forwarder distribution for s390x Linux
zLinux Forwarder

40. © 2017 SPLUNK INC.
​Deliver Better
Performing, More
Reliable Apps
​End-to-End
Performance and
Capacity Insights
​Deliver Real-Time
Omni-Channel
Analytics
Extend Operational Intelligence
to Mobile Apps

41. © 2017 SPLUNK INC.
▶ Improve user retention by quickly
identifying crashes and performance
issues
▶ Establish whether issues are caused
by an app or the network(s)
▶ Correlate app, OS and device type
to diagnose crash and network
performance issues
Monitor App Usage and Performance

42. © 2017 SPLUNK INC.
▶ Hunk Archive functionality
moves under Splunk Enterprise
as Data Roll
▶ Hunk searching of third party
data is rebranded as Splunk
Analytics for Hadoop
▶ Pricing model stays the same
as Hunk - no new SKU
HUNK > Splunk Analytics for Hadoop
Hadoop
Clusters
Splunk Analytics for
Hadoop Add-on

43. © 2017 SPLUNK INC.
▶ Build custom streaming resource
libraries
▶ Search and analyze data from other
data stores in Splunk
▶ In partnership with leading NoSQL
vendors
▶ Use in conjunction with DB Connect
for relational database lookups
Connect to NoSQL and Other Data Stores
Splunk Analytics for Hadoop Add-on
STREAMING ERP

44. © 2017 SPLUNK INC.
▶ Rolls historical data into
existing Hadoop distribution
▶ Reduces storage up to 80%*
▶ Retains Splunk search
capability
with performance tradeoffs
▶ Integrated, zero-cost option
of Splunk Enterprise
* Achieved by reducing Splunk performance optimization data
Hadoop Data Roll
Amazon EMR
on S3
Hadoop
Clusters
Leverage existing Hadoop
Datastore to reduce TCO

45. © 2017 SPLUNK INC.
▶ Enables seamless use of almost the
entire Splunk stack on data
▶ Automatically handles MapReduce
▶ Technology is patent pending
Virtual Indexes

46. © 2017 SPLUNK INC.
Data Enrichment

47. © 2017 SPLUNK INC.
▶ Tags – categorize and add meaning to data
▶ Field Aliases – simplify search and correlation
▶ Calculated Fields – shortcut complex/repetitive computations
▶ Event Types – group common events and share knowledge
▶ Lookups – augment data with additional external fields
Agenda

48. © 2017 SPLUNK INC.
▶ Adds inline meaning/context/specificity to raw data
▶ Used to normalize metadata or raw data
▶ Simplifies correlation of multiple data sources
▶ Created in Splunk
▶ Transferred from external sources
What is Data Enrichment?

49. © 2017 SPLUNK INC.
▶ Add meaning/context/specificity to raw data
▶ Labels describing team, category, platform, geography
▶ Applied to field-value combination
▶ Multiple tags can be applied for each field-value
▶ Case sensitive
Tags

50. © 2017 SPLUNK INC.
Create TagsSHOW

51. © 2017 SPLUNK INC.
Search events with tag in any field
Search events with tag in a specific field
Search events with tag using wildcards
Find the Web Servers
Tags in Action
tag=webserver
tag::host=webserver
tag=web*
Tag the host
as webserver
Tag the sourcetype
as web
1
2
3
4
5
SHOW
Back to
Slides

52. © 2017 SPLUNK INC.
▶ Normalize field labels to simplify search and correlation
▶ Apply multiple aliases to a single field
• Example: Username | cs_username | User à user
• Example: c_ip | client | client_ip à clientip
▶ Processed after field extractions + before lookups
▶ Can apply to lookups
▶ Aliases appear alongside original fields
Field Aliases

53. © 2017 SPLUNK INC.
Re-Label Field to Intuitive Name
Create Field Alias
SHOW
1
2
3

54. © 2017 SPLUNK INC.
Create field alias of clientip = customer
Search events in last 15 minutes, find
customer field
Field alias (customer) and original field
(clientip) are both displayed
Search using an Intuitive Field Name
Field Alias in Action
sourcetype=access_combined
SHOW
1
2
3

55. © 2017 SPLUNK INC.
▶ Shortcut for performing
repetitive/long/complex
transformations using eval
command
▶ Based on extracted or discovered
fields only
▶ Do not apply to lookup or
generated fields
Calculated Fields
1
2
3
3

56. © 2017 SPLUNK INC.
Compute Kilobytes from Bytes
Create Calculated Field
SHOW
1
2
3

57. © 2017 SPLUNK INC.
Create kilobytes = bytes/1024
Search events in last 15 minutes for kilobytes
and bytes
Search Using Kilobytes instead of Bytes
Calculated Fields in Action
SHOW
Back to
Slides
1
2
sourcetype=access_combined

58. © 2017 SPLUNK INC.
▶ Classify and group common events
▶ Capture and share knowledge
▶ Based on search
▶ Use in combination with fields and tags to define event topography
Event Types

59. © 2017 SPLUNK INC.
▶ Best Practice: Use punct field
• Default metadata field describing event structure
• Built on interesting characters: ",;-#$%&+./:=?@'|*nr"(){}<>[]^! »
• Can use wildcards
Create Event Types
event punct
####<Jun 3, 2014 5:38:22 PM MDT> <Notice>
<WebLogicServer> <bea03> <asiAdminServer>
<WrapperStartStopAppMain> <>WLS Kernel<>
<> <BEA-000360> <Server started in
RUNNING mode>
####<_,__::__>_<>_<>_<>_<>_
<>_
172.26.34.223 - - [01/Jul/2005:12:05:27 -0700]
"GET /trade/app?action=logout HTTP/1.1" 200
2953
..._-_-_[:::_-]_"_?=_/."__

60. © 2017 SPLUNK INC.
Show punct for sourcetype=access_combined
Pick a punct, then wildcard it after the timestamp
Add NOT status=200
Save as “bad” event type + Color:red + Priority:1
(shift reload in browser to show coloring)
Classify Events as Known Bad
Create Event Type
sourcetype="access_combined" punct="..._-_-_[//_:::]*" NOT status=200
SHOW
Back to
Slides
1
2
3
4
eventtype=bad

61. © 2017 SPLUNK INC.
Lookups to Enrich Raw Data
CRM/
ERP
External Data Sources
Data goes in
Create additional fields
from the raw data with
a lookup to an external
data source
Insight comes out
Watch
Lists
LDAP
AD
CMDB

62. © 2017 SPLUNK INC.
▶ Augment raw events with additional fields
• Provide context or supporting details
▶ Translate field values to more descriptive data
• Example: add text descriptions for error codes, IDs
• Example: add contact details to user names or IDs
• Example: add descriptions to HTTP status codes
▶ File-based or scripted lookups
Lookups

63. © 2017 SPLUNK INC.
Convert a Code into a Description
Configure a Static Lookup
1. Upload/create table
2. Assign table to lookup object
3. Map lookup to data set
SHOW

64. © 2017 SPLUNK INC.
Get the lookup from the Splunk Wiki (save to .csv file)
http://wiki.splunk.com/Http_status.csv
Lookup table files > Add new
• Name: http_status.csv (must have .csv file extension)
• Upload: <path to .csv>
Verify lookup was created successfully
1. Create HTTP Status TableSHOW
1
2
3
| inputlookup http_status.csv

65. © 2017 SPLUNK INC.
Lookup definitions > Add new
• Name: http_status
• Type: File-based
• Lookup file: http_status.csv
Invoke the lookup manually
2. Add Lookup DefinitionSHOW
sourcetype=access_combined | lookup http_status
status OUTPUT status_description
1
2

66. © 2017 SPLUNK INC.
Automatic lookups > Add new
• Name: http_status (cannot have spaces)
• Lookup table: http_status
• Apply to: sourcetype = access_combined
• Lookup input field: status
• Lookup output field: status_description
Verify lookup is invoked automatically
3. Configure Automatic LookupSHOW
1
2

67. © 2017 SPLUNK INC.
▶ Temporal lookups for time-based lookups
• Example: Identify users on your network based on their IP address and the
timestamp in DHCP logs
▶ Use search results to populate a lookup table
• … | outputlookup <tablename|filename>
▶ Call an external command or script
• Python scripts only
• Example: DNS lookup for IP ßà Host
▶ Create a lookup table using a relational database
• Review matches against a database column or SQL query
Fancy Lookups

68. © 2017 SPLUNK INC.
▶ Creating and Managing Alerts (Job Inspector)
▶ Macros
▶ Workflow Actions
More Data Enrichment

69. © 2017 SPLUNK INC.
BREAK
15 MINUTES

70. © 2017 SPLUNK INC.
15 minute break

71. © 2017 SPLUNK INC.
Level Up on Search &
Reporting Commands

72. © 2017 SPLUNK INC.
▶ Doing more with basic search commands
▶ Advanced search commands
▶ Doing more with basic reporting commands
Agenda

73. © 2017 SPLUNK INC.
Search Syntax Components

74. © 2017 SPLUNK INC.
Anatomy of a Search
Disk

75. © 2017 SPLUNK INC.
▶ top – limit
▶ rare – same options as top
▶ timechart – parameters
▶ stats – functions (sum, avg, list, values, sparkline)
▶ sort – inline ascending or descending
▶ addcoltotals
▶ addtotals
Doing More with Basic Search Commands

76. © 2017 SPLUNK INC.
Workshop Notes for Presenter
Tip #6:
In the next section, after each search, have the participants
save the search as a dashboard panel. At the end of the workshop,
they will have a living document of the workshop exercises to reference
later. A complete version of this dashboard is packaged as an app.
It is uploaded to the Box folder as a leave behind.

77. © 2017 SPLUNK INC.
... | rare limit=20 clientip
... | top limit=20 clientip
▶ Commands have parameters or qualifiers
▶ Top and rare have similar syntax
▶ Each search command has its own syntax – show inline help
Find Most and Least Active Customers
Using the top + rare Commands
SHOW
IPs with the
most visits
IPs with the
least visits

78. © 2017 SPLUNK INC.
... | stats count by clientip | sort + count
... | stats count by clientip | sort - count
▶ Sort inline descending or ascending
The Number of Customer Requests
Using the sort Command
SHOW
Number of requests by
customer - descending
Number of requests by
customer - ascending

79. © 2017 SPLUNK INC.
▶ Show Search Command Reference Docs
• Functions for eval + where
• Functions for stats + chart and timechart
▶ Invoke a function
▶ Rename inline
... | stats sum(bytes) as totalbytes by clientip | sort - totalbytes
... | stats sum(bytes) by clientip | sort - sum(bytes)
Determine Total Customer Payload
Using functions + rename command
SHOW
Total payload by
customer - descending
Total payload by
customer - ascending

80. © 2017 SPLUNK INC.
▶ List all values of a field
▶ List only distinct values of a field
Observe Customer Activity
Using the list + values Functions
... | stats values(action) by clientip
... | stats list(action) by clientip
SHOW
Activity by customer
Distinct actions by
customer

81. © 2017 SPLUNK INC.
▶ Show distinct actions and cardinality of each action
Analyze Customer Activity
Combine list + values Functions
sourcetype=access_combined
| stats count(action) as value by clientip, action
| eval pair=action + " (" + value + ")"
| stats list(pair) as values by clientip
SHOW

82. © 2017 SPLUNK INC.
... | stats sum(bytes) as totalbytes, avg(bytes) as avgbytes, count as
totalevents by clientip | addcoltotals totalbytes, totalevents
▶ Add columns
▶ Sum specific columns
Building a Table of Customer Activity
Add Columns and Sum Columns
... | stats count by clientip, action
SHOW
2 cols: clientip + action
Sum totalbytes and
totalevents columns

83. © 2017 SPLUNK INC.
... | stats sum(bytes) as totalbytes, sum(other) as totalother
by clientip | addtotals fieldname=totalstuff
Building a Table of Customer Activity
Sum Across Rows
SHOW
Sum totalbytes and
totalevents columns
A better example:
physical memory + virtual
memory = total memory

84. © 2017 SPLUNK INC.
... | stats sparkline(count) as trendline sum(bytes) by clientip
Trend Individual Customer Activity
Sparklines in Action
... | stats sparkline(count) as trendline by clientip
SHOW
In context of larger
event set
Inline in tables
Back to
Slides

85. © 2017 SPLUNK INC.
Advanced Search Commands
Command Short Description Hints
transaction Group events by a common field value. Convenient, but resource intensive.
cluster Cluster similar events together. Can be used on _raw or field.
associate Identifies correlations between fields. Calculates entropy btn field values.
correlate Calculates the correlation between different fields.
Evaluates relationship of all fields
in a result set.
contingency Builds a contingency table for two fields.
Computes co-occurrence, or % two fields
exist in same events.
anomalies Computes an unexpectedness score for an event.
Computes similarity of event (X) to a
set of previous events (P).
anomalousvalue
Finds and summarizes irregular, or uncommon,
search results.
Considers frequency of occurrence or
number of stdev from the mean.

86. © 2017 SPLUNK INC.
▶ Sew events together + creates duration + eventcount
View Customer Activity by Session
Using the transaction Command
... | transaction JSESSIONID | table JSESSIONID, action, product_id
SHOW
Group by JSESSIONID

87. © 2017 SPLUNK INC.
▶ Intelligent group (creates cluster_count and cluster_label)
ClusterSHOW
Back to
Slides
... | cluster showcount=1 | table _raw, cluster_count, cluster_label

88. © 2017 SPLUNK INC.
▶ Predict over time
▶ Chart Overlay with and without streamstats
▶ Maps with iplocation + geostats
▶ Single value
▶ Metered visuals with gauge
Doing More with Basic Reporting Commands

89. © 2017 SPLUNK INC.
▶ Predict future values using lower/upper bounds – single and multiple series
Predict Website Traffic
Using the predict Command
... | timechart count as traffic | predict traffic
SHOW

90. © 2017 SPLUNK INC.
Compare Browsing vs. Buying Activity
Simple Chart Overlay
SHOW
sourcetype=access_combined (action=view OR action=purchase)
| timechart span=10m count(eval(action="view")) as Viewed,
count(eval(action="purchase")) as Purchased

91. © 2017 SPLUNK INC.
Map Customer Activity Geographically
Geolocation in Action
SHOW
... | iplocation clientip | geostats count by clientip Combine IP lookup with
geo mapping

92. © 2017 SPLUNK INC.
Display a Simple Count of Events
Single Value in Action
SHOW
... | stats count

93. © 2017 SPLUNK INC.
Display Counts Using Gauges
Single Value, Radial and Filler Gauges in Action
SHOW
... | stats count | gauge count 10000 20000 30000 40000 50000
Back to
Slides

94. © 2017 SPLUNK INC.
BREAK
15 MINUTES

95. © 2017 SPLUNK INC.
15 minute break,
or end of session

96. © 2017 SPLUNK INC.
Data Model and Pivot

97. © 2017 SPLUNK INC.
▶ What is a data model?
▶ Build a data model
▶ Pivot Interface
▶ Accelerate a data model
Agenda

98. © 2017 SPLUNK INC.
Pivot
Enables non-technical users to build complex
reports without the search language
Powerful Analytics Anyone Can Use
Data
Model
Provides more meaningful representation of
underlying raw machine data
Analytics
Store
Acceleration technology delivers up to 1000x
faster analytics over Splunk 5

99. © 2017 SPLUNK INC.
▶ Data Model
• Describes how underlying
machine data is represented and
accessed
• Defines meaningful relationships
in the data
• Enables single authoritative view
of underlying raw data
Define Relationships in Machine Data
Hierarchical object view of underlying data
Add constraints to filter out events

100. © 2017 SPLUNK INC.
▶ High Performance
Analytics Store
• Automatically collected
− Handles timing issues, backfill…
• Automatically maintained
− Uses acceleration window
• Stored on the indexers
− Peer to the buckets
• Fault tolerant collection
Transparent Acceleration
Check to enable
acceleration of data model
Time window of data
that is accelerated

101. © 2017 SPLUNK INC.
▶ Pivot
• Drag-and-drop interface enables
any user to analyze data
• Create complex queries and
reports without learning search
language
• Click to visualize any chart type;
reports dynamically update when
fields change
Easy-to-Use Analytics
All chart types available in
the chart toolbox
Select fields from
data model
Time window
Save report to share

102. © 2017 SPLUNK INC.
▶ Defines least common denominator for a data domain
▶ Standard method to parse, categorize, normalize data
▶ Set of field names and tags by domain
▶ Packaged as Data Models in a Splunk App
• Domains: security, web, inventory, JVM,
performance, network sessions, and more
• Minimal setup to use Pivot interface
Common Information Model (CIM) App

103. © 2017 SPLUNK INC.
Apps > Find More Apps >
Search: “Common Information Model”
Install free
Show fields for web + Web Data Model
Download CIM AppSHOW
Back to
Slides
1
2
3
4

104. © 2017 SPLUNK INC.
http://docs.splunk.com/Documentation/Splunk/latest/PivotTutorial/WelcometothePivotTutorial
Data Model & Pivot Tutorial

105. © 2017 SPLUNK INC.
Custom Visualizations and
the Web Framework Toolkit

106. © 2017 SPLUNK INC.
▶ Custom Visualizations
▶ Developer Platform
▶ Resources
Agenda

107. © 2017 SPLUNK INC.
▶ Native charts and maps
• Bar / Line / Area charts
• Bubble / Scatter plots
• Gauges
• Maps
• Single Value Displays
• Tables
▶ Generalized to fit use cases
across many different areas
▶ Can be customized to some
extent to cover specific use cases
Native Visualizations In Splunk

108. © 2017 SPLUNK INC.
▶ Many use cases require a more
specific visualization
▶ Specific custom appearance
▶ Represent data where native
visualizations are not suitable
• You can Splunk everything!
• We won’t be able to predict every possible
use case
• Still uses SPL to drive visualizations
Custom Visualizations FTW!

109. © 2017 SPLUNK INC.
▶ Platform extensibility framework and API
▶ Targeted at internal and external
developers with web development / JS
skills and basic knowledge of the
Splunk platform
▶ Developers can make use of any third party
libraries (d3.js, three.js, highcharts.js, etc…)
that run in the browser*
* with minor adjustments, and if third party license permits such use
Custom Visualizations

110. © 2017 SPLUNK INC.
▶ Packaged as an app!
▶ Installed like any other app
▶ Users can search for
visualizations on Splunkbase
and directly in the product
Custom Visualizations For Admins
In-productInstallation

111. © 2017 SPLUNK INC.
▶ Choose from potentially dozens of installed
visualizations!
▶ Appears as a first-class citizen alongside
native visualizations
• Looks and works just like packaged native
visualizations
▶ Customize functionality and appearance of
the visualization without touching any code,
straight from the UI
SPL Example provided as you hover
over each visualization option.
Custom Visualizations How-to

112. © 2017 SPLUNK INC.
New Splunk Visualizations
Multiple use cases across IT, security, IoT, and business analytics
Treemap
Sankey
Diagram
Punchcard Calendar
Heat Map
Parallel
Coordinates
Bullet GraphLocation
Tracker
Horseshoe
Meter
Machine Learning
Charts
Timeline
Horizon Chart

113. © 2017 SPLUNK INC.
Box Plot
3D scatter plot
New Partner/Community Visualizations
Wordcloud
Donut Chart
Heat Map

114. © 2017 SPLUNK INC.
New Partner/Community Visualizations
Geo
Heatmap
Custom Cluster Map
Clustered
Single
Value Map
Missile Map

115. © 2017 SPLUNK INC.
The Splunk Enterprise Platform
Collection
Indexing
Search Processing Language
Core Functions
Inputs, Apps, Other Content
Content
Core Engine
User and Developer Interfaces
Core Engine
User and Developer Interfaces
Content
Web Framework
SDK
Rest API

116. © 2017 SPLUNK INC.
Developer Platform
What’s Possible with the
Splunk Enterprise Platform?
Power
Mobile Apps
Log
Directly
Extract
Data
Customer
Dashboards
Integrate
BI Tools
Integrate Platform
Services

117. © 2017 SPLUNK INC.
Web Framework Toolkit

118. © 2017 SPLUNK INC.
SDKs
Powerful Platform for Enterprise Developers
Developers Can Customize and Extend
Rest API
Web Framework Java
JavaScript
Python
Simple XML
JavaScript
HTML5
Data Models
Search Extensibility
Modular Inputs
Ruby
C#
PHP
Extend and Integrate SplunkBuild Splunk Apps

119. © 2017 SPLUNK INC.
Splunk Software for Developers
GAIN APPLICATION
INTELLIGENCE
INTEGRATE AND
EXTEND SPLUNK
BUILD SPLUNK
APPS

120. © 2017 SPLUNK INC.
A Wealth of Splunk Apps
Over 1,300 apps available on the Splunk apps site
Server, Storage,
Network
Server Virtualization Operating Systems
Custom
Applications
Business
Applications
Cloud Services
App Performance
MonitoringTicketing/ and Other
Web Intelligence
Mobile
Applications
Stream
API
SDKs UI

121. © 2017 SPLUNK INC.
▶ Interactive, cut/paste examples from popular source repositories:
D3, GitHub, jQuery
▶ Splunk 6.x Dashboard Examples App
https://apps.splunk.com/app/1603
▶ Custom SimpleXML Extensions App
https://apps.splunk.com/app/1772
▶ Splunk Web Framework Toolkit App
https://apps.splunk.com/app/1613
Example Advanced Visualizations

122. © 2017 SPLUNK INC.
Resources

123. © 2017 SPLUNK INC.
▶ http://docs.splunk.com
▶ Official Product Docs
▶ Wiki and community topics
▶ Updated daily
▶ Can be printed to .PDF
Splunk Documentation

124. © 2017 SPLUNK INC.
▶ http://answers.splunk.com
▶ Community driven
▶ Splunk supported
▶ Knowledge exchange
▶ Q & A
Splunk Answers

125. © 2017 SPLUNK INC.
▶ Recommended for Users
• Using Splunk
• Searching & Reporting
▶ Recommended for UI/Dashboard Developers
• Developing Apps
▶ Instructor-Led Courses
• Web
• Onsite
Splunk Education

126. © 2017 SPLUNK INC.
Delivered Globally:
Online, Classroom,
Self-Paced
15 Free
Getting Started Videos
Get Splunk Certified
in 5 Days
20 Classes
For more information: splunk.com/education
Knowledge is Power
Splunk Education

127. © 2017 SPLUNK INC.
Become a Splunk Expert
Knowledge is Power
Using Splunk
Searching and Reporting with Splunk
Creating Splunk Knowledge Objects
Splunk Administration
Architecting and Deploying Splunk
Developing Apps with Splunk
Splunk Architect Certification Lab

128. © 2017 SPLUNK INC.
Splunk Education for Security
Knowledge is Power
Using Splunk
Searching and Reporting with Splunk
Creating Splunk Knowledge Objects
Using the Splunk App for
Enterprise Security
Splunk Administration
Architecting and Deploying Splunk
Administering the Splunk App
for Enterprise Security

129. © 2017 SPLUNK INC.
Splunk Education for IT Service Intelligence
Knowledge is Power
Using Splunk
Searching and Reporting with Splunk
Creating Splunk Knowledge Objects
Splunk Administration
Implementing IT Service Intelligence

130. © 2017 SPLUNK INC.
​Course Topics
• Overview of ITSI features
• ITSI architecture and deployment
• Installing ITSI
• Designing and implementing services
and entities
• Configuring correlation searches and
notable events
• Creating deep dive pages
• Creating glass tables
• ITSI troubleshooting
Splunk Education for IT Service Intelligence
Knowledge is Power

131. © 2017 SPLUNK INC.
Splunk Education for IT Service Intelligence
Knowledge is Power
​Course Topics
• Overview of ITSI features
• ITSI architecture and deployment
• Installing ITSI
• Designing and implementing services and entities
• Configuring correlation searches and notable
events
• Creating deep dive pages
• Creating glass tables
• ITSI troubleshooting
PREREQUISITES | 13.5 hour course
Using Splunk, Searching and Reporting with Splunk,
Creating Splunk Knowledge Objects, Splunk Administration

132. © 2017 SPLUNK INC.
Q&A

133. © 2017 SPLUNK INC.
Get Started Fast!
splunk.com/education

134. © 2017 SPLUNK INC.© 2017 SPLUNK INC.
Thank You