This document provides an overview and agenda for a Machine Data 101 presentation. The presentation covers Splunk fundamentals including the Splunk architecture and components, data sources both traditional and non-traditional, data enrichment techniques including tags, field aliases, calculated fields, event types, and lookups. Labs are included to help attendees get hands-on experience with indexing sample data, performing data discovery, and enriching data.

2. 2
Splunk>
Mo Hassan
Sales Engineer
presenter
April 13 2017
St. Louis, MO
Machine Data 101

3. Agenda (modules)
 Splunk Overview
 Non-Traditional Data Sources
 Data Enrichment (lab)
 Level Up on Search & Reporting Commands (lab)
 Visualizations commands (lab)
 Data Models and Pivot
 Advanced Visualizations and the Web Framework
3

4. Splunk Overview

5. A Platform for Universal Data
5

6. 6
Service Desk
Application
Support
Systems
Administrator
Application
Developer
Application
Developer
Database
Administrator
Java monitoring
tools don’t
show anything
either. Call
developer.
Log call, console
says everything
is green.
Stop working on
new code to
troubleshoot.
Need production
logs!
Stop what
they’re doing to
identify and
gather
production logs
for developer.
Manual
investigation
establishes not
application
problem
DBA analyzes
audit logs which
points to bad
query.
Splunk breaking the silos

7. 7
SEARCH HEAD (searching & reporting)
INDEXER (procesing & storage)
FORWARDER (data collection)
DEP/DS/CM/LIC/MC SERVER (management)
Architecture Components

8. 8
n+1
n+1
Horizontal Scaling (Petabytes)
Indexing Tier
Search Heads Tier
Forwarding Tier

9. Storage Optimization
Driving down data retention costs
Buckets
9
New Data Storage Controls
• 40-80% reduction in data footprint
• No functionality loss
• Limited performance tradeoff for typical use
cases
How does it work?
Certain Splunk performance optimization
data (TSIDX) is removed – yielding a
smaller footprint.

10. High Availability
• As Splunk software collects data, it keeps
multiple identical copies
• Data integrity and resilience without a SAN
• If indexer fails, incoming data continues
to get indexed
• Indexed data continues to be searchable
10

11. Visibility Across Datacenters (multi-site clustering)
New York Tokyo
London Cloud
• Distributed search unifies
the view across locations
• Role-based access controls how
far a given user's search will span
11

12. 1.
2.
3.
4.
Simple Steps to Deploy Splunk Enterprise
Download
Install
Forward Data
Search
Four steps:
DatabasesNetworks Servers Virtual
Machines
Smartphones
and Devices
Custom
Applications
Security Web
Server
Sensors
12

13. 1.
2.
3.
Simple Steps to Deploy Splunk Cloud
Sign Up
Forward Data
Search
Three steps:
DatabasesNetworks Servers Virtual
Machines
Smartphones
and Devices
Custom
Applications
Security Web
Server
Sensors
13

14. Data Sources

15. Traditional Data Sources
 Captures events from log files in real time
 Runs scripts to gather system metrics, connect
to APIs and databases
 Listens to syslog and gathers Windows events
 Universally indexes any data format so it
doesn’t need adapters
15
Windows
• Registry
• Event logs
• File system
• sysinternals
Linux/Unix
• Configurations
• Syslog
• File system
• Ps, iostat, top
Virtualization
• Hypervisor
• Guest OS
• Guest Apps
Applications
• Web logs
• Log4J, JMS, JMX
• .NET events
• Code and scripts
Databases
• Configurations
• Audit/query logs
• Tables
• Schemas
Network
• Configurations
• syslog
• SNMP
• netflow

16. Non-Traditional
Data Sources

17. Non-Traditional Data Sources
Network Inputs (TCP, UDP)
Splunk App for Stream (PCAP,NETFLOW)
Scripted Inputs (PS, BSH, PHP)
RDMBS (SQL)
Splunk ODBC Driver
Modular Inputs (python, C++, .Net)
zLinux Forwarder(mainframes)
MINT (mobile devices)
Hadoop, NoSQL
HTTP Event Collector (HTTP, HTTPS)
17
1
2
3
4
5
6
7
8
9
1
10

18. zLinux Forwarder
18
 Easily collect and index data on IBM mainframes
 Collect application and platform data
 Download as new Forwarder distribution for s390x Linux
Data inputs

19. The Splunk App for Stream
Wire Data Enhances the Platform for
Operational Intelligence
Efficient, Cloud-ready Wire Data Collection
Simple Deployment Supports Fast Time to Value
19
Data inputs

20. Stream PCACP For Better Insights
20
•Payload data including process
times, errors, transaction traces,
ICA latency, SQL statements, DNS
records…
•Analyze traffic volume, speed and
packets to identify infrastructure
performance issues, capacity
constraints, changes; establish
baselines
•Measure application response times,
deeper insights for root-cause
diagnostics, trace tx paths, establish
baselines
•Protocol conversations on database
performance, DNS lookups, client data,
business transaction paths…
•Customer Experience –
analyze website and
application bottlenecks to
improve customer experience
•Faster root cause analysis and
resolution of customer issues
with website or apps
•Protocol identification, protocol
headers, content and payload
information, flow record
•Build analytics and context for
incident response, threat detection,
monitoring and compliance
Security Digital
Intelligence
IT
Operations
Application
Management
Data inputs

21. Scripted Inputs
21
 Send data to Splunk via a custom script
 Splunk indexes anything written to STDOUT
 Splunk handles scheduling
 Supports shell, Python scripts, WIN batch, PowerShell
 Any other utility that can format and stream data
Streaming Mode
 Splunk executes script and indexes stdout
 Checks for any running instances
Write to File Mode
 Splunk launches script which produces
output file, no need for external scheduler
 Splunk monitors output file
Data inputs

22. Scripted Inputs Use Cases
22
Alternative to file-base or network-based inputs
Stream data from command-line tools, such as vmstat and iostat
Poll a web service, API or database and process the results
Reformat complex or binary data for easier parsing into events and fields
Maintain data sources with slow or resource-intensive startup procedures
Provide special or complex handling for transient or unstable inputs
Scripts that manage passwords and credentials
Wrapper scripts for command line inputs that contain special characters
Data inputs

23. Modular Inputs
23
More control…
• Instance control: launch a single or
multiple instances
• Input validation
• Stream data as text or XML
• Support multiple platforms
• Secure access to mod input scripts via
REST endpoints
Data inputs

24. Modular Inputs Use Cases
24
Twitter
 Stream JSON data from a Twitter source to Splunk using Tweepy
Amazon S3 Online Storage
 Index data from the Amazon S3 online storage web service
Java Messaging Service (JMS)
 Poll message queues and topics through JMS Messaging API
 Talks to multiple providers: MQSeries (Websphere MQ), ActiveMQ,
TibcoEMS, HornetQ, RabbitMQ, Native JMS, WebLogic JMS, Sonic MQ
Splunk Windows Inputs
 Retrieve WIN event logs, registry keys, perfmon counters
Data inputs

25. Database Inputs (DB Connect)
 Create value with structured data
 Enrich search results with additional
business context
 Easily import data for deeper analysis
 Integrate multiple DBs concurrently
 Simple set-up, non-invasive and secure
DB Connect provides reliable, scalable,
real-time integration between Splunk and
traditional relational databases
Microsoft SQL
Server
JDBC
Database
Lookup
Database
Query
Connection
Pooling
Other
Databases
Oracle
Database
Java Bridge Server
25
Data inputs

26. Configure Database Inputs
26
 DB Connect App
 Real-time, scalable integration with relational DBs
 Browse and navigate schemas and tables before data import
 Reliable scheduled import
 Seamless installation and UI configuration
 Supports connection pooling and caching
 “Tail” tables or import entire tables
 Detect and import new/updated rows using timestamps or unique IDs
 Supports many RDBMS flavors
 IBM DB2, Oracle, Microsoft SQL Server, MySQL, SAP Sybase, PostgreSQL, +
Data inputs

27. Splunk ODBC Driver
27
• Interact with, manipulate and visualize machine data in
Splunk Enterprise using business software tools
• Leverage analytics from Splunk alongside Microsoft
Excel, Tableau Desktop or Micro strategy Analytics
Desktop
• Industry-standard connectivity to Splunk Enterprise
• Empowers business users with direct and secure access
to machine data
• Combine machine data with structured data for better
operational context
Data inputs

28. HTTP Event Collector (HEC)
 Collect data over HTTP or HTTPS directly to Splunk
 Application Developer focus – few lines of code in app
to send data
 HEC Features Include:
 Token-based, not credential based
 Indexer Acknowledgements – guarantees data indexing
 Raw and JSON formatted event payloads
 SSL, CORS (Cross Origion access), and Network Restrictions
28
Data inputs

29. Configuring HEC
• Enable HTTP Event Collector
• Create/Get a token
• Send events to Splunk using the token
– Use HTTP Directly
 Create a POST request and
set the Auth header with the token
 POST JSON/RAW in our event format to the
collector
– Use logging libraries
 Support for .NET, Java and JavaScript loggers
29
LA
B

30. 30
Using HEC in bash script
JSON Objects
{“event:{
“message”:”…”
”severity”:”warn”
“category”:”web”
}
}
Batching
{“event”:”event 1”}
{“event”:”event 2”}
{“event”:”event 3”}
Metadata:
{
“source”:”foo”,
“sourcetype”:”bar”,
“host”:”192.168.1.1”,
“time”:1441139779
“event”:”Hello World”
}
Index Selection:
{
“index”:”collector”
“event”:”Hellow World”
}
curl -k http://localhost:8088/services/collector/event -H "Authorization: Splunk FC24BA71-46AF-
40C9-A52D-779F7062D514" -d '{"event": "hello world"}'
Steps:
Settings > Data Inputs > HTTP Event Collector > Global Settings
LA
B

31. Extend Operational Intelligence to Mobile Apps
31
Deliver Better
Performing, More
Reliable Apps
Deliver Real-Time
Omni-Channel
Analytics
End-to-End
Performance and
Capacity Insights

32. Monitor App Usage and Performance
• Improve user retention by quickly
identifying crashes and
performance issues
• Establish whether issues are
caused by an app or the network(s)
• Correlate app, OS and device type
to diagnose crash and network
performance issues
32

33. Splunk Analytics for Hadoop
Explore Visualize Dashboard
s
ShareAnalyze
Hadoop Clusters NoSQL and Other Data Stores
Hadoop Client Libraries Streaming Resource Libraries
Splunk Analytics for Hadoop
 Data stays in Hadoop
 Automatically handles
MapReduce
 Use SPL to search
Hadoop
 Bi-directional
Integration
33

34. Connect to NoSQL and Other Data Stores
• Build custom streaming resource
libraries
• Search and analyze data from
other data stores in Hunk
• In partnership with leading
NoSQL vendors
• Use in conjunction with DB
Connect for relational database
lookups
Splunk Analytics for Hadoop
34

35. Integrated Hadoop Features
Access, analysis and storage flexibility with data lake
Amazon
EMR on S3
Hadoop
Clusters
Roll historical Splunk data into
existing Hadoop distribution
Seamlessly search your Hadoop
data within Splunk *
Enrich data in Hadoop with Splunk
search results
Import Hadoop data into Splunk
35

36. Integrates with Third-Party Business Tools
Analyst Splunk admin
Requirements
STEP 1 Business user
communicates data
requirements to
Splunk admin
STEP 2 Splunk admin authors saved
searches in Splunk Enterprise
thereby making the searches
available to ODBC driver
STEP 3 Business user
uses tool to access
saved searches and
retrieve data from
Splunk Enterprise ODBC driver
(SQL to SPL
translation layer)
Analyst
Saved
Searches
36

37. 37
BREAK….
10 minutes

38. Workshop Setup
LAB #1
30 minutes

39. LAB #1
(Getting ready!, 30 min)
 Installing Splunk (10 mins)
 Indexing sample dataset (10 mins)
 Fields extractor (10 mins)
 Exporting data (5 mins)
39
LA
B

40. Download Splunk or Sign Up For Splunk Cloud
40
 www.splunk.com > Free Splunk > Splunk Enterprise or Splunk Cloud
1
2 3
LA
B

41. Download Data Sample
41
• ftp://34.206.145.1/Lunch_Learn/
• access_datasample_last4h.log (data file)
• http_status.csv (lookup table)
LA
B

42. Index Data Sample
42
 Browser: http://localhost:8000
 Default user: admin pass: changeme
1
2
LA
B

43. Index Data Sample
43
3
2
1
4
5
LA
B

44. Index Data Sample
44
1
2
LA
B

45. Index Data Sample
45
1
2
You will need to refresh the
search after a few moments
for all events to show up
LA
B

46. Pattern Detection
46
 Data discovery
 Group like events
 Save as event type
 Create alert
LA
B

47. 47
Exporting Splunk DataLA
B

48. 12.130.60.4 - - [18/Sep/2014 05:26:50:193] "GET
/product.screen?product_id=AV-CB-01&JSESSIONID=SD8SL4FF8ADFF5 HTTP
1.1" 200 3221
"http://www.myflowershop.com/category.screen?category_id=BOUQUETS"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" 634
Lab dataset
48
Client IP: 12.130.60.4
Method: GET
URL: /product.screen?product_id=AV-CB-01&JSESSIONID=SD8SL4FF8ADFF5 HTTP 1.1"
Return Code: 200
User Agent: "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
Bytes xfered: 634
LA
B
Apache sample event:

49. Fields Extractor
49
LA
B

50. Data Enrichment
LAB #2
50 minutes

51. LAB #2
(data enrichment, 50 min)
 Tags (10 mins)
 Field Aliases (10 mins)
 Calculated Fields (10 mins)
 Event Types (10 mins)
 Lookups (10 mins)
51
LA
B

52.  Adds inline meaning/context/specificity to raw data
 Used to normalize metadata or raw data
 Simplifies correlation of multiple data sources
 Created in Splunk
 Transferred from external sources
What is Data Enrichment?
52
LA
B

53.  Add meaning/context/specificity to raw data
 Labels describing team, category, platform, geography
 Applied to field-value combination
 Multiple tags can be applied for each field-value
 Case sensitive
Tags
53
LA
B

54. How to Create Tags
54
LA
B

55.  Search events with tag in any field
 Search events with tag in a specific field
 Search events with tag using wildcards
Searching with Tags
(find the web servers)
55
tag=webserver
tag::host=webserver
tag=web*
 Tag the host as
webserver
 Tag the sourcetype
as web
1
2
3
4
5
LA
B

56.  Normalize field labels to simplify search and correlation
 Apply multiple aliases to a single field
 Example: Username | cs_username | User  user
 Example: c_ip | client | client_ip  clientip
 Processed after field extractions + before lookups
 Can apply to lookups
 Aliases appear alongside original fields
Field Aliases
56
LA
B

57. How to Create Field Aliases
(re-label field to intuitive name)
57
1
2
3
LA
B

58.  Create field alias of clientip = customer
 Search events in last 15 minutes, find
customer field
 Field alias (customer) and original field
(clientip) are both displayed
How To Search With Field Alias
58
1
3
2
sourcetype=access_combined
LA
B

59.  Shortcut for performing
repetitive/long/complex transformations using
eval command
 Based on extracted or discovered fields only
 Do not apply to lookup or generated fields
Calculated Fields
59
LA
B

60. How to Create Calculated Field
(compute kilobytes from bytes)
60
1
21
2
3
LA
B

61.  Create kilobytes = bytes/1024
 Search events in last 15 minutes for
kilobytes and bytes
*use verbose mode if needed!
Search with Calculated Fields
(search using KB instead of bytes)
61
1
2
sourcetype=access_combined
LA
B

62.  Classify and group common events
 Capture and share knowledge
 Based on search
 Use in combination with fields and tags to define
event topography
Event Types
62

63.  Best Practice: Use punct field
 Default metadata field describing event structure
 Built on interesting characters: ",;-#$%&+./:=?@'|*nr"(){}<>[]^! »
 Can use wildcards
How to Create Event Types
63
event punct
####<Jun 3, 2014 5:38:22 PM MDT> <Notice>
<WebLogicServer> <bea03> <asiAdminServer>
<WrapperStartStopAppMain> <>WLS Kernel<> <>
<BEA-000360> <Server started in RUNNING mode>
####<_,__::__>_<>_<>_<>_<>_<>_
172.26.34.223 - - [01/Jul/2005:12:05:27 -0700]
"GET /trade/app?action=logout HTTP/1.1" 200 2953
..._-_-_[:::_-]_"_?=_/."__

64.  Show punct for sourcetype=access_combined
 Pick a punct, then wildcard it after the timestamp
 Add NOT status=200
 Save as “bad” event type + Color:red + Priority:1 (shift
reload in browser to show coloring)
Searching with Event Type
(classify events as known “bad”)
64
eventtype=bad
sourcetype="access_combined" punct="..._-_-_[//_:::]*" NOT status=200
1
2
3
4
LA
B

65.  Augment raw events with additional fields
 Provide context or supporting details
 Translate field values to more descriptive data
 Example: add text descriptions for error codes, IDs
 Example: add contact details to user names or IDs
 Example: add descriptions to HTTP status codes
 File-based or scripted lookups
Lookups
65
LA
B

66. Lookups to Enrich Raw Data
LDAP
AD
Watch
Lists
CRM/
ERP
CMDB
External Data SourcesCreate additional fields
from the raw data with
a lookup to an external
data source
66

67. Simplest way to create a lookup table
67
sourcetype=access_combined | dedup status | table status | outputlookup file.csv
1
2
LA
B

68. ConfigureStatic Lookup
(convertacodeintodescription)
68
1. Upload/create table
2. Assign table to lookup object
3. Map lookup to data set
1
2
3
LA
B

69.  Get the lookup from the Splunk Wiki (save to .csv file)
 ftp://34.206.145.1/WorkShop_4_13_2017/MachineData101/ http_status.csv
 Lookup table files -> Add new
 Name: http_status.csv (must have .csv file extension)
 Upload: <path to .csv>
 Verify lookup was created successfully
Import HTTP Status Table
(step 1)
69
| inputlookup http_status.csv
1
2
3
LA
B

70.  Lookup definitions > Add new
 Name: http_status
 Type: File-based
 Lookup file: http_status.csv
 Invoke the lookup manually
Add Lookup Definition
(step 2)
70
1
2
sourcetype=access_combined | lookup
http_status status OUTPUT status_description
LA
B
Filed extracted from source file Found in lookup table

71.  Automatic lookups > Add new
 Name: http_status (cannot have spaces)
 Lookup table: http_status
 Apply to: sourcetype = access_combined
 Lookup input field: status
 Lookup output field: status_description
 Verify lookup is invoked automatically
Configure Automatic Lookup
(step 3)
71
1
2
sourcetype=access_combined
LA
B

72. 72
Lookup output fields
Provide one or more pairs of
output fields. The first field is
the corresponding field that
you want to output to
events. The second field is
the name that the output
field should have in your
events.
Lookup input fields
Provide one or more pairs of
input fields. The first field is
the field in the lookup table
that you want to match. The
second field is a field from
your events that should
match the lookup table field.
.

73. Temporal lookups for time-based lookups
• Example: Identify users on your network based on their IP address and the
timestamp in DHCP logs
Call an external command or script
• Python scripts only
• Example: DNS lookup for IP  Host
Create a lookup table using a relational database
• Review matches against a database column or SQL query
Use search results to populate a lookup table
• sourcetype=access_combined | dedup status | table status | outputlookup file.csv
Other methods for creating Lookups
73

74.  Creating and Managing Alerts (Job Inspector)
 Macros
 Workflow Actions
More Data Enrichment…
74

75. 75
BREAK….
10 minutes

76. Level Up on Search &
Reporting Commands
LAB #3
50 mins

77. LAB #3
level up on searching & reporting (50 mins)
77
 top – limit – rare (5 mins)
 sort – inline ascending or descending (5 mins)
 timechart – parameters (5 mins)
 stats – functions (sum, avg, list, values, sparkline)
 rename - eval – list – values (5 mins)
 addcoltotals - addtotals (5 mins)
 cluster – transactions (5 mins)

78. Search Syntax Components
78

79. Anatomy of a Search
79
Disk

80. How Search Works
80

81. Tips for tuning searches
81
http://docs.splunk.com/Documentation/Splunk/latest/Search/Writebettersearches
 Be more specific (as early as possible)
 Avoid using NOT expressions when possible (inclusive search)
 Restrict searches to the specific index
 Use indexed and default fields (host, sourcetype, source)
 Disable field discovery to improve search performance
 Summarize your data (create summary indexers)
 Use the Search Job Inspector
 Book mark SPL link in browser
http://docs.splunk.com/Documentation/Splunk/6.5.3/SearchReference/Abstract

82. Workshop Notes for Presenter
Tip #5: In the next section, after each search, have the
participants save the search as a dashboard panel. At the end
of the workshop, they will have a living document of the
workshop exercises to reference later.
A complete version of this dashboard is packaged as an app.
It is uploaded to the Box folder as a leave behind.
82

83.  Commands have parameters or qualifiers
 top and rare have similar syntax
 Each search command has its own syntax – show inline help
top, rare commands
(find most & least active customers)
83
... | top clientip limit=20
... | rare clientip limit=20
IPs with the
most visits
IPs with the
least visits
LA
BSave as dashboard
UI TIP:
Linux/Windows: Ctrl
Mac: Command

84.  Sort inline descending or ascending
sort
(sort the Number of Customer Requests)
84
... | stats count by clientip | sort - count
... | stats count by clientip | sort + count
Number of requests by
customer - descending
Number of requests by
customer - ascending
LA
BSave as dashboard

85.  http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Commandsbycategory
 Functions for eval + where Functions for stats + chart and timechart
functions + rename
(determine total customer payload)
85
... | stats sum(bytes) by clientip | sort - sum(bytes)
... | stats sum(bytes) as totalbytes by clientip | sort - totalbytes
Total payload by
customer - descending
Total payload by
customer - ascending
LA
B
 Rename inline
 Invoke a function
Save as dashboard

86.  List all values of a field
list + values functions
86
... | stats values(action) by clientip
... | stats list(action) by clientip
Activity by customer
Distinct actions by
customer
LA
B
 List only distinct values of a field
Save as dashboard

87.  Show distinct actions and cardinality of each action
Combine list + values functions
87
sourcetype=access_combined
| stats count(action) as value by clientip, action
| eval pair=action + " [" + value + "]"
| stats list(pair) as action by clientip
LA
B

88.  Add columns
addcoltotals command
88
sourcetype=access_combined | stats count by clientip, action
2 cols: clientip + action
sourcetype=access_combined
| stats sum(bytes) as total_bytes, avg(bytes) as avg_gbytes, count as total_events by clientip
| addcoltotals total_bytes, total_events
Sum total bytes and
total events columns
LA
B
 Sum specific columns
Save as dashboard

89.  Sum across rows
89
sourcetype=access_combined
| stats sum(bytes) as totalbytes, sum(other) as totalother by clientip
| addtotals fieldname=totalstuff
For each row, add
totalbytes + totalother
A better example:
physical memory + virtual memory =
total memory
addtotals commandLA
BSave as dashboard

90.  Sew events together + creates duration + eventcount
transaction command
90
... | transaction JSESSIONID | table JSESSIONID, action, product_id
Group by
JSESSIONID
LA
BSave as dashboard

91.  Intelligent group (creates cluster_count and cluster_label)
Cluster command
91
... | cluster showcount=1 | table _raw, cluster_count, cluster_label
LA
B
 Sparklines inline in tables

92. Advanced Search Commands
Command Short Description Hints
transaction Group events by a common field value. Convenient, but resource intensive.
cluster Cluster similar events together. Can be used on _raw or field.
associate Identifies correlations between fields. Calculates entropy btn field values.
correlate Calculates the correlation between
different fields.
Evaluates relationship of all fields in
a result set.
contingency Builds a contingency table for two fields. Computes co-occurrence, or % two
fields exist in same events.
anomalies Computes an unexpectedness score for
an event.
Computes similarity of event (X) to a
set of previous events (P).
anomalousvalue Finds and summarizes irregular, or
uncommon, search results.
Considers frequency of occurrence
or number of stdev from the mean
92

93. 93
BREAK….
10 minutes

94. Visualization
LAB #4
30 mins

95.  Predict over time (5 mins)
 Chart Overlay with and without streamstats (5 mins)
 Maps with iplocation + geostats (5 mins)
 Predict – sparklines (5 mins)
 Single value (5 mins)
 Metered visuals with gauge (5 mins)
LAB #4
Visualizations Commands(30 mins)
95

96.  Predict future values using lower and upper bounds
predict command
96
... | timechart count as traffic | predict traffic
Predict website
traffic
LA
BSave as dashboard

97.  Sparklines for trending of individual fields
Sparklines
97
... | stats sparkline(count) as trendline by clientip
In context of
larger event set
... | stats sparkline(count) as trendline sum(bytes) by clientip
Inline in tables
LA
BSave as dashboard

98.  Simple overlay
Chart Overlays
98
sourcetype=access_combined (action=view OR action=purchase)
| timechart span=10m count(eval(action="view")) as Viewed,
count(eval(action="purchase")) as Purchased
Show browsing
vs. buying
LA
BSave as dashboard

99. Geolocation
99
... | iplocation clientip | geostats count by clientip
Combine IP lookup
with geo mapping
LA
B Save as dashboard

100. Single Values
100
sourcetype=access_combined | stats count
Count of events
LA
B Save as dashboard

101. Gauges
101
... | stats count | gauge count 10000 20000 30000 40000 50000
Count of eventsLA
B Save as dashboard

102. Data Model and Pivot
LAB #5

103. LAB #5
(instructor only)
 What is a data model?
 Build a data model
 Pivot Interface
 Accelerate a data model
103

104.  Apps > Find More Apps >
 Search: “Common Information Model”
 Install free
 Show fields for web + Web Data Model
Download CIM App
104
1
2
3
4
LA
B Show Splunkbase
Must have tag=web created

105. 105
Enables non-technical users to build complex
reports without the search language
Provides more meaningful representation of
underlying raw machine data
Acceleration technology delivers up to 1000x
faster analytics over Splunk 5
Pivot
Data
Model
Analytics
Store
Powerful Analytics Anyone Can Use

106. Define Relationships in Machine Data
Data Model
• Describes how underlying
machine data is represented
and accessed
• Defines meaningful
relationships in the data
• Enables single authoritative
view of underlying raw data
Hierarchical object view of underlying data
Add constraints to
filter out events
106

107. Transparent Acceleration
• Automatically collected
– Handles timing issues,
backfill…
• Automatically maintained
– Uses acceleration window
• Stored on the indexers
– Peer to the buckets
• Fault tolerant collection
Time window of data
that is accelerated
Check to enable
acceleration of
data model
High Performance
Analytics Store
107

108. Easy-to-Use Analytics
• Drag-and-drop interface
enables any user to analyze
data
• Create complex queries and
reports without learning
search language
• Click to visualize any chart
type; reports dynamically
update when fields change
Select fields from
data model
Time window
All chart types available in the chart toolbox
Save report
to share
Pivot
108

109.  Defines least common denominator for a
data domain
 Standard method to parse, categorize,
normalize data
 Set of field names and tags by domain
 Packaged as a Data Models in a Splunk App
 Domains: security, web, inventory, JVM,
performance, network sessions, and more
 Minimal setup to use Pivot interface
Common Information Model (CIM) App
109

110. Data Model & Pivot Tutorial
http://docs.splunk.com/Documentation/Splunk/latest/PivotTutorial/WelcometothePivotTutorial
110

111. Custom Visualizations
and the Web
Framework Toolkit

112.  Developer Platform
 Web Framework Toolkit (WFT)
 REST API and SDKs
 Get a Flying Start
112
LAB #6
(instructor only)

113. The Splunk Enterprise Platform
Collection
Indexing
Search Processing Language
Core Functions
Inputs, Apps, Other Content
SDKContent
Core Engine
User and Developer Interfaces
Web Framework
REST API
113

114. Powerful Platform for Enterprise Developers
REST API
Web Framework
Web
Framework
Ruby
C#
PHP
Data Models
Search Extensibility
Modular Inputs
SDKsSimple XML
JavaScript
Django
Developers Can Customize and Extend
114

115. Splunk Software for Developers
Gain
Application
Intelligence
Build Splunk
Apps
Integrate and
Extend Splunk
115

116. Splunk ValueTransaction Profiling
Value
Transaction Profiling and Splunk
End users
Messaging
Legacy
Systems Databases
Web
Servers
App
Servers
Security
End usersEnd users
Java, .NET, PHP, etc.
116
Transaction path discovery
through entire stack
Pinpoint
bottlenecks/source of
transactions
End user experience
Detect slow code execution
for Java, .NET, PHP, node.js,
etc.
Single source for analyzing
and correlating logs and app
metrics
Pinpoint problems not
related to application
execution code
Application logs provide
developer insight to apps
Network-based insight to
transactions via Splunk
Stream
Customer/business context
for transactions
Virtualization
Servers
Storage
Networking/
Loadbalancing

117.  Interactive, cut/paste examples from popular
source repositories: D3, GitHub, jQuery
 Splunk 6.x Dashboard Examples App
https://apps.splunk.com/app/1603
 Custom SimpleXML Extensions App
https://apps.splunk.com/app/1772
 Splunk Web Framework Toolkit App
https://apps.splunk.com/app/1613
Example Advanced Visualizations
117
Download & install from Splunkbase

118. 118
http://www.d3js.org

119. Add a D3 Bubble Chart
119
1. Go to Find More Apps and Install the
Splunk 6.x Dashboard Examples App
2. Enter the App
3. Go to Examples > Custom Visualizations >
D3 Bubble Chart
4. Copy autodiscover.js (file) + components/bubblechart (dir)
from: $SH/etc/apps/simple_xml_examples/appserver/static
to: $SH/apps/search/appserver/static
5. Copy and paste simple XML to new dashboard
LA
B

120. Resources

121. Splunk Documentation
121
 http://docs.splunk.com
 Official Product Docs
 Wiki and community topics
 Updated daily
 Can be printed to .PDF

122. © 2017 SPLUNK INC.
Thriving Community
dev.splunk.com
40,000+ questions
and answers
1,300+ apps
Local User Groups &
SplunkLive! events

123. Splunk Education
123
• Recommended for Users
– Using Splunk
– Searching & Reporting
• Recommended for UI/Dashboard Developers
– Developing Apps
• Instructor-Led Courses
– Web
– Onsite

124. Links
 http://docs.splunk.com/Documentation/Splunk/latest/SearchTutorial/Welcom
etotheSearchTutorial?r=searchtip
 http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Splun
kEnterpriseQuickReferenceGuide?r=searchtip
 http://www.splunk.com/content/dam/splunk2/pdfs/solution-guides/splunk-
quick-reference-guide.pdf
 http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Comm
andsbycategory
 https://tekhnologic.wordpress.com/2015/09/07/need-a-timer-for-your-lesson-try-a-video-timer/#Add%20a%20timer
124

125. Thank you!
125
SEPT 25-28, 2017
Walter E. Washington Convention Center
Washington, D.C.