Splunk conf2014 - Lesser Known Commands in Splunk Search Processing Language (SPL)

Lesser
Known

Search
Commands

Kyle
Smith

Infrastructure
Analyst,

The
Hershey
Company

Disclaimer

During
the
course
of
this
presenta?on,
we
may
make
forward-‐looking
statements

regarding
future
events
or
the
expected
performance
of
the
company.
We

cau?on
you
that
such
statements
reflect
our
current
expecta?ons
and
es?mates

based
on
factors
currently
known
to
us
and
that
actual
events
or
results
could

differ
materially.
For
important
factors
that
may
cause
actual
results
to
differ

from
those
contained
in
our
forward-‐looking
statements,
please
review
our

filings
with
the
SEC.
The
forward-‐looking
statements
made
in
the
this

presenta?on
are
being
made
as
of
the
?me
and
date
of
its
live
presenta?on.
If

reviewed
aLer
its
live
presenta?on,
this
presenta?on
may
not
contain
current
or

accurate
informa?on.
We
do
not
assume
any
obliga?on
to
update
any
forward-‐
looking
statements
we
may
make.
In
addi?on,
any
informa?on
about
our

roadmap
outlines
our
general
product
direc?on
and
is
subject
to
change
at
any

?me
without
no?ce.
It
is
for
informa?onal
purposes
only,
and
shall
not
be

incorporated
into
any
contract
or
other
commitment.
Splunk
undertakes
no

obliga?on
either
to
develop
the
features
or
func?onality
described
or
to
include

any
such
feature
or
func?onality
in
a
future
release.

2

Me

!   Infrastructure
Analyst
at
The
Hershey
Company

!   Working
with
Splunk
for
~5
years

!   WriRen
4
Public
Splunk
Apps
(on
apps.splunk.com)

!   Ac?ve
on
both
#splunk
on
IRC
and
answers.splunk.com

–  My
Handle
is
“alacercogitatus”
or
just
“alacer”

!   Totally
not
above
asking
for
Karma

3

Agenda

!   ?mechart
eval

!   dynamic
eval

!   con?ngency

!   map

!   gen?mes

!   xyseries

!   foreach

!   cluster

!   correlate

!   autoregress

!   CLI
Commands

4

Timeline
Eval

!   You
can
use
an
eval
statement
in
a
?mechart
command

`nest_data`| autoregress temp_f | eval diff = temp_f - temp_f_p1 | timechart span=15m
avg(diff) as "Average Difference" eval(avg(temp_f) - avg(temp_f_p1)) as "Eval Diff”
5

Dynamic
Eval
(aka
Indirect
Reference)

!   Not
a
search
command

!   NOTE:
It’s
a
python
hack,
so
it
might
not
work
in
the
future.

!   Works
great
for
perfmon
sourcetypes,
but
can
be
applied
to

any
search

!   ?mechart
will
not
use
wildcard
to
expand
ﬁelds

<your_search> | eval sample_{field} = value| stats avg(sample_*) as *
6

Dynamic
Eval
–
How
to
Use
It

<your_search> | eval m_{metric} = Value| stats avg(m_*) as “*”
The Raw Event
07/17/2014 09:03:59.865
-0700
collection=CPUTime
object=Processor
metric=kbps
Value=0.97397499837021373
The New Event
07/17/2014 09:03:59.865 -0700
collection=CPUTime
object=Processor
metric=kbps
Value=0.97397499837021373
kbps = 0.97397499837021373
Why?

Allows
the
dynamic
crea?on
of
fields
based

on
the
values
of
other
fields
within
the
same
event

Ref:hRp://answers.splunk.com/answers/103700/how-‐do-‐i-‐create-‐a-‐field-‐whose-‐name-‐is-‐the-‐value-‐of-‐another-‐field-‐like-‐back?cks-‐or-‐eval-‐in-‐other-‐languages

7

Dynamic
Eval
–
Subsearch

!   Not
a
search
command

!   NOTE:
It’s
a
Splunk
hack,
so
it
might
not
work
in
the
future

<your_search> | eval subsearch = if(host==[ <subsearch> | head 1 | rename host as query |
fields query | eval query=""".query.""" ],”setting_1”,”setting_2”)
8

con?ngency

9

!   Web
analy?cs
-‐
browsers
with
versions

!   Demographics
-‐
ages
with
loca?ons
or
genders

!   Security
-‐
usernames
with
proxy
categories

!   Compare
categorical
ﬁelds

contingency [<contingency-option>]* <field> <field>
Builds a contingency table for two fields
A contingency table is a table showing the distribution
(count) of one variable in rows and another in columns, and
is used to study the association between the two variables

con?ngency
–
How
to
Use
It

`get_iis_data` | contingency ua_browser ua_browser_version usetotal=f
10

con?ngency
–
How
to
Use
It

`get_iis_data` | contingency ua_browser ua_browser_version usetotal=f
11

con?ngency
–
How
to
Use
It

`get_weather_data` | contingency weather range usetotal=f
range in this case is the absolute value of (Actual Temperature - Relative Humidity)
12

map

13

!   Uses
“tokens”
($field$)
to
pass
values
into
the
search

!   Best
with
either:
Very
small
input
set
And/Or
very
specific
search.
Can
take
a
long

amount
of
?me

!   Map
is
a
type
of
subsearch

!   Is
“?me
agnos?c”
–
?me
is
not
necessarily
linear,
and
can
be
based
off
of
the

master
search

map (<searchoption>|<savedsplunkoption>) [maxsearches=int]
Looping operator, performs a search over each search result

map
–
How
to
Use
It

First Search:
sourcetype=syslog sudo | stats min(_time) as et max(_time) as lt by user host
14

map
-‐
How
to
Use
It

_?me
computername
computer?me
username
user?me

10/12/12
8:31:35.00
AM
Worksta?on$
10/12/2012
08:25:42
userA
10/12/2012
08:31:35
AM

Map Search
`first_search` | map search="search index=windows latest=$et$ username=$user$
eventtype=ad_last_logon "
It takes each of the results from the previous search and searches in the windows index
for the user's logon event. The results are returned as a table, such as:
So
the
informa?on
you
have
is:
From
what
computer
on
the
domain,
did
a
user
most
likely

execute
a
sudo
command
on
a
server?
You
can
then
use
this
to
determine
if
access
was

authorized
from
that
loca?on,
poten?ally
spopng
a
breach.

15

gen?mes

16

!   Useful
for
genera?ng
?me
buckets
not
present
due
to
lack
of
events
within
those

?me
buckets

!   Must
be
the
first
command
of
a
search
(useful
with
map
)

!   “Suppor?ng
Search”
-‐
no
real
use
case
for
basic
searching

!   Can
be
used
to
show
different
“top”
fields
over
a
?mechart!

| gentimes start=<timestamp> [end=<timestamp>] [<increment>]
Generates time range results. This command is useful in
conjunction with the map command

gen?mes
–
How
to
Use
It

|gentimes start=10/1/2013 end=10/2/2013
increment=1h
<timestamp>
Syntax: (MM/DD/YY)?:(HH:MM:SS)?|<int>
Description: Indicate the time, for
example: 10/1/07:12:34:56 (for October 1,
2007 12:34:56) or -5 (five days ago)
17

gen?mes
–
How
to
Use
It

|gentimes start=-1 end=0 increment=1h | map maxsearches=24 search="search earliest=
$starttime$ latest=$endtime$ <your_base_search> | bucket _time span=1h | top
useother=t limit=5 <some_field> by _time | fields - percent" | timechart sum(count) by
<some_field>
hRp://answers.splunk.com/answers/154723/?mechart-‐other-‐category-‐per-‐span

18

BONUS
SEARCH!

`find_asa_vpn_events` | `combine_user_names`| eventstats
min(_time) as earliest max(_time) as latest by asaUser |
convert ctime(earliest) as et timeformat="%m/%d/%Y:%H:%M:
%S" | convert ctime(latest) as lt timeformat="%m/%d/%Y:%H:
%M:%S" | map maxsearches=5000 search="|gentimes start=$et$
end=$lt$ increment=1h | eval asaUser="$asaUser$" | eval
_time = starttime "| fields _time asaUser | timechart
span=1h dc(asaUser) as "ASA Users"
RUNNING
THIS
OVER
LARGE
TIME
RANGES
WILL
TAKE
A
LONG
TIME
TO
COMPLETE

(
This
one
took
9.7
minutes
to
run
)

19

BONUS
SEARCH!

Connect
event:

Sep

5
08:22:09
asadevice.example.com
:Sep
05
12:22:09
UTC:
%ASA-‐auth-‐6-‐113004:
AAA
user
authentication
Successful
:
server
=

10.140.41.8
:
user
=
myuser100

Disconnect
event:

Sep

5
09:03:01
asadevice.example.com
:Sep
05
13:03:01
UTC:
%ASA-‐auth-‐4-‐113019:
Group
=
DefaultWEBVPNGroup,
Username
=
myuser100,

IP
=
192.168.100.100,
Session
disconnected.
Session
Type:
SSL,
Duration:
16h:00m:05s,
Bytes
xmt:
58502687,
Bytes
rcv:
15322893,

Reason:
Max
time
exceeded

Fields
being
passed
to
map
command:

20

BONUS
Karma!

Free
Karma?
How?

1.  Login
to
answers.splunk.com

(create
an
account
if
needed)

2.  Find
the
Answer
where
I
used

this
image

3.  Up
vote
the
answer

4.  I’ll
give
the
ﬁrst
person
today

to
up
vote
100
Karma
points!

22

xyseries

23

!   Email
flow
[
xyseries
email_domain
email_direc?on
count
]

!   One
to
many
rela?onships
[
example
Weather
Icons
]

!   Any
data
that
has
values
INDEPENDENT
of
the
field
name

–  host=myhost
domain=splunk.com
metric=kbps
metric_value=100

–  xyseries
domain
metric
metric_value

!   Works
great
for
categorical
field
comparison

xyseries [grouped=<bool>] <x-field> <y-name-field> <y-data-field>...
[sep=<string>] [format=<string>]
Converts results into a format suitable for graphing

xyseries
–
How
to
Use
It

`weather_data` | xyseries icon weather weather
24

foreach

25

!   Rapidly
perform
evalua?ons
and
other
commands
on
a
series
of
ﬁelds

!   Can
help
calculate
Z
scores
(sta?s?cal
inference
comparison)

!   Reduces
the
number
of
evals
required

foreach <wc-field> [fieldstr=<string>] [matchstr=<string>]
[matchseg1=<string>] [matchseg2=<string>] [matchseg3=<string>]
<subsearch>
Runs a templated streaming subsearch for each field in a
wildcarded field list.
Example. Equivalent to ... | eval foo="foo" | eval bar="bar" | eval
baz="baz"
... | foreach foo bar baz [eval <<FIELD>> = "<<FIELD>>"]

foreach
–
How
to
Use
It

`per60m_firewall_actions`
| timechart span=60m sum(countaction) by action
| streamstats window=720 mean(*) as MEAN* stdev(*) as STDEV*
| foreach * [eval Z_<<FIELD>> = ((<<FIELD>>-MEAN<<MATCHSTR>>) / STDEV<<MATCHSTR>>)] | fields
_time Z*
26

foreach
–
How
to
Use
It

`per60m_firewall_actions`
| timechart span=60m sum(countaction) by action
| streamstats window=720 mean(*) as MEAN* stdev(*) as STDEV*
| foreach * [eval Z_<<FIELD>> = ((<<FIELD>>-MEAN<<MATCHSTR>>) / STDEV<<MATCHSTR>>)] | eval
Z_PROB=3.2 | fields _time Z*
27

cluster

28

!   Find
common
and/or
rare
events

!   Great
for
“WAG”
searching

!   Finds
anomalies
(outliers)
in
your
web
logs,
security
logs,
etc
by

checking
for
cluster_counts

!   Find
common
errors
in
event
logs

cluster [slc-option]*
Cluster similar events together

cluster
–
How
to
Use
It

index=_internal source=*splunkd.log* log_level!=info | cluster showcount=t | table
cluster_count _raw | sort -cluster_count
29

autoregress

30

!   Allows
advanced
sta?s?cal
calcula?ons
based
on
previous
values

!   Moving
averages
of
numerical
ﬁelds

–  Network
bandwidth
trending
-‐
kbps,
latency,
dura?on
of
connec?ons

–  Web
analy?cs
trending
-‐
number
of
visits,
dura?on
of
visits,
average
download

size

–  Malicious
traﬃc
trending
-‐
excessive
connec?on
failures

autoregress field [AS <newfield>] [p=<p_start>[-<p_end>]]
Sets up data for calculating the moving average.
A Moving Average is a succession of averages calculated from
successive events (typically of constant size and
overlapping) of a series of values.

autoregress
–
How
to
Use
It

`temperature_data` | autoregress temp_f | table _time temp_f temp_f_p1
31

autoregress
–
How
to
Use
It

`temperature_data` | autoregress temp_f | eval diff = abs(temp_f - temp_f_p1) | timechart span=15m eval(avg(diff)) as
"Average Difference"
32

CLI
Commands

33

!   $SPLUNK_HOME/bin/splunk
reload
index

–  Reloads
index
conﬁgura?on,
making
immediately
eﬀec?ve
all

–  "add/edit/enable/disable
index"
commands
since
last
reload
or
Splunk
restart

!   Why?

–  Adding
a
new
app

–  Changing
a
frozen
?me
period

–  New
loca?on
for
data

CLI
Commands

34

cmd
pcregextest

–  Useful
for
tes?ng
modular
regular
expressions
for
extrac?ons

splunk cmd pcregextest mregex="[[ip:src_]] [[ip:dst_]]" ip="(?<ip>d+[[dotnum]]{3})"
dotnum=".d+" test_str="1.1.1.1 2.2.2.2"
Original Pattern: '[[ip:src_]] [[ip:dst_]]'
Expanded Pattern: '(?<src_ip>d+(?:.d+){3}) (?<dst_ip>d+(?:.d+){3})'
Regex compiled successfully. Capture group count = 2. Named capturing groups = 2.
SUCCESS - match against: '1.1.1.1 2.2.2.2'
#### Capturing group data #####
Group | Name | Value
--------------------------------------
1 | src_ip | 1.1.1.1
2 | dst_ip | 2.2.2.2

CLI
Commands

35

cmd
btool

ê  Btool
allows
you
to
inspect
conﬁgura?ons
and
what
is
actually
being
applied
to
your

sourcetypes

!   splunk
cmd
btool
-‐-‐debug
props
list
wunderground
|
grep
-‐v
"system/default"

/opt/splunk/etc/apps/TA-wunderground/default/props.conf [wunderground]
/opt/splunk/etc/apps/TA-wunderground/default/props.conf KV_MODE = json
/opt/splunk/etc/apps/TA-wunderground/default/props.conf MAX_EVENTS = 100000
/opt/splunk/etc/apps/TA-wunderground/default/props.conf MAX_TIMESTAMP_LOOKAHEAD = 30
/opt/splunk/etc/apps/TA-wunderground/default/props.conf REPORT-extjson = wunder_ext_json
/opt/splunk/etc/apps/TA-wunderground/default/props.conf SHOULD_LINEMERGE = true
/opt/splunk/etc/apps/TA-wunderground/default/props.conf TIME_PREFIX = observation_epoch
/opt/splunk/etc/apps/TA-wunderground/default/props.conf TRUNCATE = 1000000

Resources
and
Ques?ons

36

!   IRC
#splunk
on
efnet.org
(look
for
alacer)

!   docs.splunk.com

!   answers.splunk.com
(I’m
alacercogitatus
-‐
can
I
haz
karma?)

!   wiki.splunk.com

THANK
YOU

alacercogitatus
–

Did
I
men?on
Karma?

Splunk conf2014 - Lesser Known Commands in Splunk Search Processing Language (SPL)

More Related Content

What's hot

Similar to Splunk conf2014 - Lesser Known Commands in Splunk Search Processing Language (SPL)

More from Splunk

Recently uploaded

Splunk conf2014 - Lesser Known Commands in Splunk Search Processing Language (SPL)