Welcome to SplunkLive [City]. Thank you for taking the time to attend today’s event.
Event types can help you automatically identify events based on a search. An event type is a field based on a search, it’s a way of classifying data for searching and reporting and it’s useful for user knowledge capture and sharing. Tags are different, in that they allow you to search for events with related field values. You can assign any field/value combination. So as an example, server names aren’t always helpful. Sometimes they contain ambiguous information. Using tags you can use a more meaningful term. The Splunk Manager allows you to enable/disable, copy, delete and edit tags that you’ve created.
Extracting fields that aren’t already pulled out at search time is a necessary step to doing more with your data like reporting.
Show example of field extraction with IFX and an example using rex. Show other field extractor.
Real-time alerts always trigger immediately for every returned result Real-time monitored alerts monitor a real-time window and can trigger immediately, or you can define conditions Scheduled alerts run a search on a regular interval that you define and triggers based on conditions that you define
Run alert in Splunk.
Splunk alerts are based on searches and can run either on a regular scheduled interval or in real-time. Alerts are triggered when the results of the search meet a specific condition that you define. Based on your needs, alerts can send emails, trigger scripts and write to RSS feeds.
Consider how you might use a scripted alert.
How can you leverage Splunk?
Show dashboard examples:
Why with the same settings is the shadow so dark?
Splunk deployments can grow to encompass thousands of Splunk instances, including forwarders, indexers, and search heads. Splunk offers a deployment monitor app that helps you to effectively manage medium- to large-scale deployments, keeping track of all your Splunk instances and providing early warning of unexpected or abnormal behavior.
The deployment monitor provides chart-rich dashboards and drilldown pages that offer a wealth of information to help you monitor the health of your system. These are some of the things you can monitor:
Index throughput over time Number of forwarders connecting to the indexer over time Indexer and forwarder abnormalities Details for individual forwarders and indexers, such as status and forwarding volume over time Source types being indexed by the system License usage
What Does Splunk Really Do?
Into thisIt turns this
[Thu Sep 24 14:57:33 2009] [error] [client 10.2.1.44] ap_proxy: trying GET /petstore/
enter_order_information.screen at backend host '127.0.0.1/7001; got exception
'CONNECTION_REFUSED [os error=0, line 1739 of ../nsapi/URL.cpp]: Error connecting to host
127.0.0.1:7001', referer: http://10.2.1.223/petstore/cart.do?action= purchase&itemId=EST-14
Industry Leading Platform For Machine Data
Machine Data: Any Location, Type, Volume
Platform Support (Apps / API / SDKs)
Answer Any Question
Developer Platform (REST API, SDKs)
Small Data. Big Data. Huge Data.
In this room:
13:15 – 14:15 Getting Started with Splunk
14:15 – 15:15 Splunk for Security
• WIN: Program FilesSplunk
• Other: /opt/splunk (Applications/splunk)
• WIN: Program FilesSplunkbinsplunk.exe start (services start)
• *NIX: /opt/splunk/bin/splunk start
• 32 or 64 Bit?
• Indexer or Universal Forwarder?
Free Download Limits Indexing to 500MB/day
• Enterprise Trial License expires after 60 days
• Reverts to Free License
Features Disabled in Free License
• Multiple user accounts and role-based access controls
• Distributed search
• Forwarding to non-Splunk Instances
• Deployment management
• Scheduled saved searches and alerting
• Summary indexing
Other License Types
• Enterprise, Forwarder, Trial
Default installation on: http://localhost:8000
Splunk Web Basics
• Firefox 10.x and latest
• Internet Explorer 7, 8, 9 and 10
• Safari (latest)
• Chrome (latest)
• Add data
• Getting Started App
• Install an App (Splunk for Windows, *NIX)
Add some data
Search app – Summary view
app navigation time range
Search > *
Select Time Range
• Historical, custom, or real-time
• Smart, Fast, Verbose
Using the timeline
• Click events and zoom in and out
• Click and drag over events for a specific range
Everything is searchable
Everything is searchable
• * wildcards supported
• Search terms are case insensitive
• Booleans AND, OR, NOT
– Booleans must be uppercase
– Implied AND between terms
– Use () for complex searches
• Quote phrases
error OR 404
error OR failed OR (sourcetype=access_*(500 OR 503))
- advanced type-ahead
- short/long description
suggests search terms
updates as you type
shows examples and help
toggle off / on
Searches can be managed as
Jobs can be
• Moved to background tasks
• Paused, stopped, resumed, finalized
Modify Job Settings
Search > error | head 1
Search results are “piped” to the command
• Manipulating fields
• Handling results
• host, source, sourcetype, linecount, etc.
• View on left panel in search results or all in field picker
Where do fields come from?
• Pre-defined by sourcetypes
• Automatically extracted key-value pairs
• User defined
- hostname, IP address,
or name of the network
host from which the
- the name of the file,
stream, or other input
- a specific data type or
Tagging and Event Typing
Eventtypes for more human-readable reports
• to categorize and make sense of mountains of data
• punctuation helps find events with similar patterns
Search > eventtype=failed_login instead of
Search > “failed login” OR “FAILED LOGIN” OR “Authentication failure” OR “Failed to
Tags are labels
• apply ad-hoc knowledge
• create logical divisions or groups
• tag hosts, sources, fields, even eventtypes
Search > tag=web_servers instead of
Search > host=“apache1.splunk.com” OR host=“apache2.splunk.com” OR
Leverage Searches for future Insights!
Add a Time Range Picker
• Date & Time Range
Scheduled or Real-Time
• Define Time Ranges
Searches run on a schedule and fire an alert
• Example: Run a search for “Failed password” every 15 min
over the last 15 min and alert if the number of events is
greater than 10
Searches are running in real-time and fire an alert
• Example: Run a search for “Failed password user=john.doe” in
a 1 minute window and alert if an event is found
results of any search
Define your Search and set your time range,
accelerate you search and more
Choose the type of chart (line, area, column, etc) and
other formatting options
Build reports from
• Use wizard or reporting commands (timechart, top, etc)
• Build real-time reports with real-time searches
• Save reports for use on dashboards
Create dashboards from search results
Splunk Has Four PrimaryFunctions
A Splunk install can be one or all roles…
Mounted File Systems
and network devices
Unix, Linux and Windows hosts
Local File Monitoring
Scripted or Modular Inputs
Splunk App for Stream
Forward data without negatively impacting production performance.
Universal Forwarder Deployment
Logs ConfigurationsMessages Metrics
Central Deployment Management
Monitor files, changes and the system registry; capture metrics and status.
Universal Forwarder Regular (Heavy) Forwarder
Splunk Web ✔
Load balanced search and indexing for massive, linear scale out.
London Hong Kong Tokyo New York
Index and store locally. Distribute searches to datacenters, networks & geographies.
As Splunk collects data, it keeps
multiple identical copies
If indexer fails, incoming data
continues to get indexed
Indexed data continues to be
Easy setup and administration
Data integrity and resilience
without a SAN
Send Data to Other Systems
Route raw data in real time or send alerts based on searches.
Integrate External Data
LDAP, AD Watch
Correlate IP addresses with locations, accounts with regions
Extend search with lookups to external data sources.
Integrate Usersand Roles
Problem Investigation Problem Investigation Problem Investigation
Users and Groups
Splunk Flexible Roles
Map LDAP & AD groups to flexible Splunk roles. Define any search as a filter.
Integrate authentication with LDAP and Active Directory.
Build Splunk Apps Extend and Integrate Splunk
SupportThroughthe Splunk Community
Browse and share Apps
from Splunk, Partners and
2015 -> more than 140
Where to Go for Help