Splunk 6.4 Administration.pdf

Listen to your data.
Splunk 6.4 Administration
Listen to your data. 1 Copyright © 2016 Splunk, Inc. All rights reserved | 7 July 2016
Generated for chandrika seela (chandrika.seela@accenture.com) (C) Splunk Inc, not for distribution

Document Usage Guidelines
Y Should be used only for enrolled students
Y Not meant to be a self-paced document, an instructor is needed
Y Do not distribute
7 July 2016

Course Prerequisites
Y Required:
– Using Splunk
– SplunkArchitecture Overview (eLearning)
Y Strongly Recommended:
– Searching and Reporting with Splunk
– Creating Splunk Knowledge Objects
Note
In order to receive credit for this
course, you must complete all lab
exercises.

Course Goals
Y Build and manage a production Splunk environment
Y Create and maintain Splunk indexes
Y Manage users, roles, and authentication options
Y Deploy forwarders with Forwarder Management
Y Configure common Splunk data inputs
Y Customize the input parsing process
Y Configure a distributed search environment
Y Monitor a Splunk instance with Management Console
Y Configure system alerts

Course Outline
SettingupaSplunk Enterprise Environment
–  Mod1:SettingupSplunk
–  Mod2:LicenseManagement
–  Mod3:SplunkApps
–  Mod4:SplunkConfigurationFiles
–  Mod5:SplunkIndexManagement
–  Mod6:Users,Roles,andAuthentication
BuildingaBasicProduction Environment
–  Mod7:UniversalForwarders
–  Mod8:ForwarderManagement
SplunkInputs
–  Mod9:GettingDataIn
–  Mod10:MonitorInputs
–  Mod11:NetworkandScriptedInputs
–  Mod12:WindowsandAgentlessInputs
–  Mod13:Fine-tuningInputs
ParsingandSearching
–  Mod14:ParsingPhaseandDataPreview
–  Mod15:ManipulatingRawData
–  Mod16:SupportingKnowledgeObjects
–  Mod17:DistributedSearch
SplunkResourceManagement
–  Mod18:BasicPerformanceTuning
–  Mod19:ProblemIsolationOverview
–  Mod20:IntroductiontoLarge-scaleDeployment
–  CourseWrap-up

Module 1:
Setting up Splunk

Module Objectives
Y Describe the Splunk installation options
Y Identify Splunk instance types
Y Identify Splunk hardware requirements
Y List steps to install Splunk
Y Perform post-installation configuration tasks
Y Start, stop, and restart Splunk
Y Enable Distributed Management Console (DMC)

Y Splunk can be deployed in a variety of configurations
Y Scales from a single server to a distributed infrastructure
–  Acceptsanytext data as input
–  Parsestheinputs into events
–  Storesevents in indexes
–  Searchesand reports
–  Authenticates users
Splunk Overview
DB
Servers
Networks Servers Web
Services
Mobile
Devices
Custom
Apps
Security
Any Text Data
Searching
Users
Input
Parsing
Indexing

Splunk Deployment – Standalone
Y Single Server
– All functions in a single instance of
Splunk
– For testing, proof of concept,
personal use, and learning
– This is what you get when you
download Splunk and install with
default settings
Y Recommendation
– Have at least one test/development
setup at your site
Parsing
Indexing
Input
Searching
Note
This is the initial configuration in
class.

Splunk Deployment – Basic
Y Splunk server
– Similar to server in standalone
configuration
– Manage deployment of forwarder
configurations
Y Forwarders
– Forwarders collect data and send it
to Splunk servers
– Install forwarders at data source
(usually production servers)
Parsing
Indexing
Input
Searching
Forwarder
Management
Note
Your lab environment will evolve to
include a separate forwarder.

Splunk Deployment – Distributed
Y Splunk can be distributed and scaled
in a variety of ways
–  Moreindexerstohandlemoreinput
–  MoreindexersANDsearchheadsto
handlemoresearching
Y Manage forwarder configurations
from a dedicated Deployment Server
Parsing
Indexing
Input
Searching
Search Head
Indexers
(Search peers)
Forwarders
Deployment
Server
Forwarder
Management
Note
You will add a single search peer
to your environment in a later lab
exercise.

Y Included in the Splunk Enterprise software package
Y Included in the Universal Forwarder software package
What Software Do You Install?
Indexer
(Search peer)
Search Head Heavy
Forwarder
Universal Forwarder
Deployment
Server
License
Master
Cluster
Master
Search Head
Cluster
Deployment Client
Splunk Enterprise

Reference Servers
Y Hardware requirements and sizing are discussed in detail in
– Architecting and Deploying Splunk class
– docs.splunk.com/Documentation/Splunk/latest/Capacity/Referencehardware
Indexer Search Head
OS Linux or Windows 64-bit distribution
Network 1Gb Ethernet NIC
Optional second NIC for a management network
Memory 12 GB RAM
CPU Intel 64-bit chip architecture
12 CPU cores
Running at 2+ GHz
Intel 64-bit chip architecture
4 CPUs, quad-core per CPU
Running at 2+ GHz
Disk Disk subsystem capable of 800 IOPS 2 x 10K RPM 300GB SAS drives - RAID 1

Further Reading: Hardware and Virtualization
Y System requirements
docs.splunk.com/Documentation/Splunk/latest/Installation/Systemrequirements#Recommended_hardware
Y Hardware capacity planning (Dimensions of a Splunk deployment)
docs.splunk.com/Documentation/Splunk/latest/Capacity/DimensionsofaSplunkEnterprisedeployment
Y Summary of performance recommendations
docs.splunk.com/Documentation/Splunk/latest/Capacity/Summaryofperformancerecommendations
Y Virtualization tech brief
www.splunk.com/web_assets/pdfs/secure/Splunk_and_VMware_VMs_Tech_Brief.pdf

Splunk Installation Overview
Y Pre-installation Checklist
– Start-up account
– Time synchronization
– Splunk ports
– Linux setting recommendations
Y Installation
– Splunk directory structure
Y Post-installation configuration
– Run Splunk at boot
– Configure system settings
– Optionally, enable Distributed Management Console

Start-up Account
Y Best practice: Do not run Splunk as super-user
– For example, root on *NIX, administrator on Windows
Y  Create a user account that is used to run Splunk
– For input, Splunk must be able to access data sources
ê  On *NIX, /var/log is not typically open to non-root accounts
– On *NIX, non-root accounts cannot access ports < 1024
– On Windows
ê  Use a domain account if Splunk has to connect to other servers
ê  Otherwise, use a local machine account
– Make sure the Splunk account can access scripts used for inputs and alerts

Time Synchronization
Y Best practice: Use a time synchronization service such as NTP
Y Splunk searches depend on accurate time
– Correct event timestamping is essential
Y It is imperative that your Splunk indexer and production servers have
standardized time configuration
– Clock skew between hosts can affect search results

Splunk Ports
Usage Splunk Enterprise Universal Forwarder
splunkd 8089 8089
Splunk Web 8000 -
Web app-server proxy 8065 -
KV Store 8191 -
S2S receiving port(s) No default -
Any network/http input(s) No default No default
Index replication port(s) Optional (no default) -
Search replication port(s) Optional (no default) -

Linux Setting Recommendations
Y Increase ulimit settings
– The following OS parameters need to be increased to allow for a large number of
buckets/forwarders/users
docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/ulimitErrors
Y Turn Transparent Huge Pages (THP) off on Splunk Enterprise servers
docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/SplunkandTHP
ulimit -a
ulimit -c 1073741824 (1 GB) (unlimited)
ulimit -n 48 x default (48 x 1024 = 49,152) (65536)
ulimit -u 12 x default (12 x 1024 = 12,288) (258048)
core file size
open files
max user processes

Installation
Y Download Splunk Enterprise from www.splunk.com/download
Y Installation: (as account running Splunk)
– *NIX – un-compress the .tar.gz file in the path you want Splunk to run from
– Windows – execute the .msi installer and follow the wizard steps
Y Complete installation instructions at:
docs.splunk.com/Documentation/Splunk/latest/Installation/Chooseyourplatform
Y After installation:
– Splunk starts automatically on Windows
– Splunk must be manually started on *NIX until boot-start is enabled

Run Splunk at Boot
Y *NIX
– Splunk on *NIX does not auto-start at boot time (default)
– To enable boot-start, run as root:
– This modifies the *NIX boot-up configuration
ê  Modifies /etc/init.d depending on your *NIX flavor
– Pass the -user parameter to start Splunk as the correct user
Y Windows
ê  Runs as splunkd service and starts child processes
ê  The service starts and stops like any Windows services
# ./splunk enable boot-start -user splunker

Splunk Directory Structure
Note
$SPLUNK_HOME depicted in the
documentation is not an exported
environment variable. It is used as
a placeholder for "the top directory
where Splunk is installed.”
SPLUNK_HOME is used in this
training.
SPLUNK_HOME
bin etc var
system apps users
search launcher <custom app>
lib
splunk
executables licenses, configs
indexes

Splunk Processes
splunkd
Y Runs on port 8089 (default) using SSL
Y Spawns and controls Splunk child
processes (helpers):
– Splunk Web proxy, KV store, and
Introspection services
– Each search, scripted input, or
scripted alert
Y Accesses, processes, and indexes
incoming data
Y Handles all search requests and
returns results
Splunk Web
Y Splunk browser-based user interface
– Provides both a search and
management front end for splunkd
process
Y Runs on port 8000 by default
– http://<server_name>:<port>
Y Initial default login:
– Username: admin
– Password: changeme

Splunk Web - Server Settings
Y Administrators can select Settings > Server settings > General settings
1
2
3
-  Overall server configuration
-  Used to set server options

Describing General Settings
docs.splunk.com/Documentation/Splunk/latest/Security/
AboutsecuringyourSplunkconfigurationwithSSL
Identifies this server to
other Splunk servers
SPLUNK_HOME
Splunkd port
Default host name
assigned to events
from this server
Change if indexes are
on a different volume
Set minimum free space
- Prevents the file system from being filled by Splunk
- Splunk data loss can occur if this value is reached
Changes require a restart of Splunk
6
5
4
3
2
1
7
6
5
4
3
2
1
7
8191

Restarting the Server from Splunk Web
1
2
3
Note
Any changes to General settings
generates a message. Clicking the
indicator opens a message
prompting you to restart.
You can also restart by selecting
Settings > Server controls or
from the CLI. (splunk restart)

The Splunk Command Line Interface (CLI)
Y splunk is the program in the bin directory to run the CLI
– Same syntax on all supported platforms
Command Operation
splunk help Display a usage summary
splunk help <object> Display the details of a specific object
splunk [start | stop | restart] Manages the Splunk processes
splunk start –-accept-license Automatically accept the license without prompt
splunk status Display the Splunk process status
splunk show splunkd-port Show the port that the splunkd listens on
splunk show web-port Show the port that Splunk Web listens on
splunk show servername Show the servername of this instance
splunk show default-hostname Show the default host name used for all data inputs

Distributed Management Console (DMC)
Y Splunk collects a lot of data about itself
Y DMC is a Splunk admin-only app that lets you monitor and investigate
Splunk performance, resource usage, and more
Note
You will use DMC to monitor your
activities as you learn more about
Splunk components.

Enabling DMC in Standalone Mode
Y DMC runs un-configured in standalone mode by default
Y To enable, click Settings > General Setup >Apply Changes
3
2
1
The default server roles

More Resources
Y Splunk Documentation: http://docs.splunk.com/Documentation
Y SplunkApp Repository: https://splunkbase.splunk.com/
Y SplunkAnswers: http://answers.splunk.com/
Y Splunk Blogs: http://blogs.splunk.com/
Y Splunk Wiki: http://wiki.splunk.com/

Lab Exercise 1 – Configure Splunk
Time: 15 - 20 minutes
Tasks:
– Log into Splunk Web
– Change your Splunk server name
– Restart Splunk
– Enable DMC and check the system overview
– Access your environment with SSH (Linux) or Remote Desktop Connection (Windows)
– Use Splunk CLI to confirm the status and changes

Lab Exercise 1 – Configure Splunk (cont.)
ssh {EIP}
RDC {EIP}
Provision:
Go to Splunk Web with your browser
using the external IP address
(xxx.xxx.xxx.xxx:8000)
Verification:
Linux:
1.  Connect to your indexer via SSH
2.  Execute Splunk CLI
Windows:
1.  Connect to your indexer via RDC
2.  Run cmd to execute Splunk CLI
http://{EIP}:8000
OR
Your
Computer
Your
Computer

Module 2:
License Management

Module Objectives
Y Identify license types
Y Describe license violations
Y Add and remove licenses

Managing Licenses
Select Settings > Licensing
Change license group
Add a license
Check license alerts and violations
View stacks
Edit and add pools
Designate the license server type
– Master or slave
6
1
2
3
4
5
6
5
4
3
2
1

Splunk License Types
Y Enterprise trial license
– Downloads with product
– Features same as Enterprise except for 500mb per day limit
– Only valid for 60 days, after which one of the other 3 license types must be
activated
– Sales trial license is a trial Enterprise license of varying size and duration
Y Enterprise license
– Purchased from Splunk
– Full functionality for indexing, search head, deployment server, etc.
– Sets the daily indexing volume

Splunk License Types (cont.)
Y Free license
– Disables alerts, authentication, clustering, distributed search, summarization, and
forwarding to non-Splunk servers
– Allows 500mb/day of indexing and forwarding to other Splunk instances
– After 60 days on enterprise trial license, you'll be automatically prompted to convert to
this license type
ê  Canbeactivatedbefore60daysbychanginglicensetype
Y Forwarder license
– Sets the server up as a heavy forwarder
– Applies to non-indexing forwarders
– Allows authentication, but no indexing
Y Splunk license comparison: www.splunk.com/view/SP-CAAAE8W

Adding a License
Y CanuseCLI,upload,orcopy&paste
–  Licensegroupchangerequiresarestart
Y Licensesarestoredunder
SPLUNK_HOME/etc/licenses
Y Canaddmultiplelicenses(stacked)
splunk add licenses <path_to_file>

License Warnings and Violations
Y If the indexing exceeds the allocated daily quota in a pool, an alert is raised
– The daily license quota resets at midnight and you have until then to fix
Y If you don't correct the situation, the alert becomes a warning
– The warning message persists for 14 days
Y The fifth warning in a rolling 30-day period causes violation
– Search is disabled for all non-internal indexes
– All other features remain functional, such as indexing and forwarding
– Violation remains in effect for 30 days
– Free license gets only three warnings
Y To unlock the license and enable searching, contact Splunk Support or
Sales

What Counts As Daily License Quota?
Y All data that is indexed, regardless of source, sourcetype, or host
– About how much data you index each day
– Not about how much data you store in Splunk
Y What does not count as daily quota?
– Data that is replicated in a cluster
ê  Data is metered once; the copies do not count
– Summary indexes, using a summarization technique
– Splunk internal logs that are stored in _internal, _audit, etc.
– Data that is eliminated during the parsing process
– Metadata fields, search terms, etc.

Viewing Alerts

Change to Slave
Change an instance to slave by
entering the master license server URI
License Master
with a license stack
All instances collectively share the stack entitlement

Y Pools allow licenses to be subdivided and assigned to a group of indexers
– Can be created for a given stack
– Warnings and violations occur per pool
Y Example: Master has a stack for a total of 500GB
License Pooling
Default Pool
500 GB Shared Entitlement
Enterprise Stack with Single Pool
(Most common)
Pool 2
(200GB Entitlement)
Pool 3
(200GB Entitlement)
Default Pool
(100GB Entitlement)
Enterprise Stack with Multiple Pools
(Multi-tenant environment)

Managing Soft License Warnings
Y DO NOT ignore license warnings
Y Proactively monitor the consumption of your Splunk license
– DMC provides a couple of alerts
– If possible, give yourself wiggle room by rearrange license pools

Lab Exercise 2 –Add and Configure Splunk Licenses
Y Time: 15 – 20 minutes
Y Tasks:
– Add a license with Splunk Web UI (optionally with CLI)
– Enable DMCAlert -Total License Usage Near Daily Quota

Module 3:
Splunk Apps

Module Objectives
Y Describe Splunk apps and add-ons
Y Install an app on a Splunk instance
Y Manage app accessibility and permissions

.
.
.
What is an App?
Y An app is an independent collection of:
– Configuration files
ê  Defininginputs,indexes,sourcetypes,field
extractions,transformations
ê  Providingeventtypes,tags,reports,dashboards
andotherknowledgeobjects
– Scripts, web assets, etc.
Y  Most apps are focused on:
–  Aspecific type of data from a vendor, operating
system, or industry
–  Aspecific business need
Y  Apps may be installed on any Splunk instance
Y  Splunk includes a number default apps
app.conf
tags.conf
mylookup.csv
indexes.conf
myviews.xml
default.meta
transforms.conf
props.conf
inputs.conf
App B
App A

View All Installed Apps
Y From the Search app, select Apps > ManageApps
Y Or, from the Home view, click
Y Apps can be Visible or hidden
– Several apps are installed by default that are hidden or disabled
ê  Internal apps used by Splunk should not be modified
ê  Legacy apps
ê  Sample apps
Y Apps are installed under SPLUNK_HOME/etc/apps

Managing Apps
Controls who can
use/modify an app
Enable or disable
an app
Add more apps

Installing an App from splunkbase.splunk.com
Y From the Apps page, click Browse more apps
Y Or, click Apps > Find MoreApps
– Splunk Web will try to access splunkbase.splunk.com
– Search and browse to find the app you want
– Select Install (most apps are free)
ê  You must provide your Splunk.com user ID and password
ê  The app is installed into a sub-directory below SPLUNK_HOME/etc/apps
ê  Some apps may require a restart
Y Or, go directly to splunkbase.splunk.com and download the app as a file
Note
Anyone can sign up for a user
account in Splunk.com.
A support contract is not required.

Installing an App From a File
Y Download the file for the app from splunkbase.splunk.com
– The file may be a .tar.gz, .tgz, .zip, or .spl file
Y Install the app:
– From Splunk Web, click Install app from file
– Using the CLI
splunk install app path-to-appfile
– Or extract the app in the proper location
cd SPLUNK_HOME/etc/apps
tar –xf path-to-appfile
Y Some apps may require a restart
Y Configure the app after the install according to its documentation

Apps on Forwarders?
Y Universal forwarders don't have a web interface, but they can still benefit
from an app
Y An add-on is a subset of an app
– Usually contains data collection but no GUI (reports or dashboards)
Y To install an add-on or app on a forwarder
– Run the CLI command described earlier, or
– Deploy it using a deployment server
ê  deployment server is discussed in a later module

Deleting an App
Y When you delete an app, all of its related configurations and scripts are
removed from a Splunk server
– User’s private app artifacts remain untouched
Y To remove an app:
– splunk remove app <app_folder>
– Or, navigate to SPLUNK_HOME/etc/apps and delete the app's folder and all its
contents
– Restart the Splunk server
Y It can be reinstalled later
Y Alternatively, either disable it or move it to another (backup) location

App Permissions
Y Users with read permission can see the
app and use it
Y Users with write permission can add/
delete/modify knowledge objects used in
the app
– By default, the user role does not have write
permissions within the search app

Lab Exercise 3 -- Install an App
Y Time: 5 - 10 minutes
Y Tasks:
– Download an app
– Install the app
– Change the app permission
– Verify if the app's dashboard displays reports

Module 4:
Splunk Configuration Files

Module Objectives
Y Describe Splunk configuration directory structure
Y Understand configuration layering process
– Index-time process
– Search-time process
Y Use btool to examine configuration settings

Y Each configuration file governs a particular aspect of Splunk functionality
Y All configuration changes are saved in .conf files under SPLUNK_HOME/etc/...
– .conf files are text files using a simple stanza
and name/value (attribute) format
– The syntax is case-sensitive
Y You can change settings using Splunk Web, CLI, SDK,
app install, and/or direct edit
Y All .conf files have documentation and examples:
– SPLUNK_HOME/etc/system/README
ê  *.conf.spec
ê  *.conf.example
ê  Splunkdocumentation: docs.splunk.com
Splunk Configuration Files
[default]
host=www
[monitor:///var/log/httpd]
sourcetype = access_common
ignoreOlderThan = 7d
index = web
Splunk Web
CLI SDK
inputs.conf

Configuration Directories
SPLUNK_HOME
etc
system apps users
joe mary admin
search
unix
local
local
default local unix
default local
search
default local
Out-of-the-box
Custom

Default vs. Local Configuration
Y Splunk ships with default .conf files
– Stored in the default directories
Y Add all configurations and edits to the
local directory
– Most configurations apply to only one app
Y Avoid storing configurations in
SPLUNK_HOME/etc/system
– Use the Searching and Reporting app
as the default location for storing your
configurations that are not app-specific
default local
Shipped with
Splunk or the app
Y Overwritten on
update
Y Do not modify
Your specific
configuration
changes
Y Preserved on
update

Index time vs. Search time
Y The priority of layered configurations are based on the context
– Global context: a network input to collect syslog data
– App/User context: Mary's private report in the Search app
Y For a list of configuration files and their context, go to:
docs.splunk.com/Documentation/Splunk/latest/Admin/Wheretofindtheconfigurationfiles
Index time Global context User-independent and background tasks
such as inputs, parsing, indexing, etc.
Search time App/User context User-related activity, such as searching

default
SPLUNK_HOME
etc
system apps
default local unix
default local
search
local
Index-Time Precedence
1.  etc/system/local
2.  etc/apps/search/local
3.  etc/apps/unix/local
4.  etc/apps/search/default
5.  etc/apps/unix/default
6.  etc/system/default
6 1
5 3 4 2
Server is performing
non-user background
processing
Note
If two or more apps at the same
level of precedence have conflicts
between them, the conflicts are
resolved in ASCII order by app
directory name.

SPLUNK_HOME
etc
system apps users
joe mary admin
search
unix
local
local
default local unix
default local
search
default local
Search Time Precedence Order
7
1
3 2 5 4
Example:
user mary working in
the unix app context
6
Note
Splunk evaluates and if
objects from the app are exported
globally with .meta file setting.
5
4

Runtime Merging of Configurations
Y When Splunk starts, configuration files are merged together into a single
run-time model for each file type
– Regardless of the number of inputs.conf files in various apps or the system
path, only one master inputs configuration model exists in memory at runtime
Y If there are no duplicate stanzas or common settings between the files, the
result is the union of all files
Y If there are conflicts, the setting with the highest precedence is used
– Remember that local always takes precedence over default

Example of Runtime Merging (No Conflict)
[a]
x=1
y=2

[b]
j=1
k=2
[b]
p=1
q=1

[c]
s=1
t=2
[a]
x=1
y=2

[b]
j=1
k=2
p=1
q=1

[c]
s=1
t=2
SPLUNK_HOME/etc/
system/local/
example.conf
SPLUNK_HOME/etc/
apps/search/local/
example.conf
Runtime
example.conf

Example of Runtime Merging (Conflict)
[a]
x=1
y=2

[b]
x=0
z=2
[b]
x=1
y=1

[c]
x=1
y=2
[a]
x=1
y=2

[b]
x=0
y=1
z=2

[c]
x=1
y=2
SPLUNK_HOME/etc/
system/local/
example.conf
SPLUNK_HOME/etc/
apps/search/local/
example.conf
Runtime
example.conf

Configuration Validation Command
Y splunk btool conf-name list [options]
– Shows on-disk configuration for requested file
– Useful for checking the configuration scope and permission rules
ê  Use--debugtodisplaytheexact .conffilelocation
ê  Add--user=<user>--app=<app>toseetheuser/appcontextlayering
Y Examples:
–  splunk help btool
–  splunk btool check
–  splunk btool inputs list
–  splunk btool inputs list monitor:///var/log
–  splunk btool inputs list monitor:///var/log --debug
docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Usebtooltotroubleshootconfigurations

Scenario: Where are the /var/log/secure.log input configurations specified?
btool Example
> splunk btool inputs list monitor:///var/log/secure.log --debug

etc/apps/search/local/inputs.conf [monitor:///var/log/secure.log]
etc/system/local/inputs.conf host = myIndexer
etc/system/default/inputs.conf index = default
etc/apps/search/local/inputs.conf sourcetype = linux_secure

[monitor:///var/log/secure.log]
host=myIndexer

[monitor:///var/log/secure.log]
sourcetype=linux_secure
host=webserver
etc/system/local/inputs.conf etc/apps/search/local/inputs.conf

Overriding Defaults
Y There are default settings in SPLUNK_HOME/etc/system/default and
SPLUNK_HOME/etc/apps/search/default
Y The correct method to override these settings, if needed, is to do so in the
local directory at the same scope
– Only add the items you are overriding—not a whole copy of the default conf file
Y Example:
– To disable a default attribute TRANSFORMS for [syslog]:
# etc/system/default/props.conf
[syslog]
TRANSFORMS = syslog-host
REPORT-syslog = syslog-extractions
...
# etc/system/local/props.conf
[syslog]
TRANSFORMS =


Reloading Configuration Files After Edit
Y Changes made using Splunk Web or the CLI do not require restart
– Amessage appears if restart is required (i.e. changing server settings)
Y Changes made manually are not automatically detected
Y To force reload, go to http://servername:webport/debug/refresh
– Reloads many of the configurations, including inputs.conf, but not all
Y To reload all configurations, restart Splunk
– Splunk Web: Settings > Server controls > Restart Splunk
– CLI: splunk restart

Lab Exercise 4 – Configuration Files
Y Time: 15 – 20 minutes
Y Tasks:
– Run the same search as different users
– Check the search results and compare
– Use the btool command to investigate the configurations

Module 5:
Splunk Index
Management

Module Objectives
Y Understand index structure and buckets
Y Create new indexes
Y Apply a data retention policy
Y Monitor indexes with DMC
Y Enable index integrity check
Y Reset monitor input checkpoints
Y Delete data from an index
Y Back up and restore frozen data

Y Splunk stores events in indexes under
SPLUNK_HOME/var/lib/splunk
–  SetinSettings > Server Settings
–  Canoverride on a per-index basis
Y The main index is
–  Usedwhenan input does not specifyanindex
–  Agoodexampleofahighvolumeindex
–  Locatedindefaultdbdirectory
Y Splunk users can explicitly specify which
index(es) to search
What are Indexes?
index=web action=purchase
main web
[Input X]
[Input Y]
index = web

Preconfigured Indexes
Splunk ships with several indexes already set up besides main:
summary – default index for summary indexing system
_internal – Splunk indexes its own logs and metrics from its processing here
_audit – Splunk stores its audit trails and other optional auditing information
_introspection – tracks system performance and Splunk resource usage data
_thefishbucket – contains checkpoint information for file monitoring inputs

Why Create Your Own Indexes?
Y Access control – segregate data into separate indexes to limit access by
Splunk role
Y More use cases are discussed inArchitecting and Deploying Splunk class
Web Team
Security Team
Web Index
web data
Security Index
security data

Y Retention
– Retention is set on a per-index basis
– Separate data into different indexes based on retention time
– Splunk data retention can be managed by data age and/or by size
Proxy Inputs
Why Create Your Own Indexes? (cont.)
Security Inputs
Web Inputs
Keep for 6 weeks
Keep for 12 months
Keep for 6 months Purge
Archive
Purge
web
security
proxy

Buckets
Y Internally, an index stores events in
buckets
Y Abucket is a directory containing a
set of rawdata and indexing data
Y Buckets have a maximum data size
and a time span
– Both can be configured
wiki.splunk.com/Deploy:UnderstandingBuckets
$SPLUNK_DB
Buckets restored
from archive
Hot & warm buckets
Cold buckets
Indexes

Index
Data Flow Through an Index
Y Hot:These are the newest buckets– still open for write
Y Warm: Recent data, buckets are closed (read only)
Y Cold: Oldest data still in the index (read only)
Y Frozen: No longer searchable; buckets either get archived or deleted
Archive
Delete
or
Inputs

Hot – Building Buckets
Y After data is read and parsed, it goes through the license meter and the
event is written into a hot bucket
Y All buckets are implemented as directories
– Hot buckets have a name that begins with hot_
– All buckets have unique identifiers within an index
Y When hot buckets reach their max size or time span, they are closed and
converted to warm status
– Hot buckets also roll to warm automatically when the indexer is restarted
– Hot and warm buckets are stored in the db directory for the index

Warm and Cold Bucket Names
Y Hot buckets are renamed when they roll to warm
– Bucket names identify the time range for the data they contain
– When a warm bucket rolls to cold, the entire bucket directory is moved
Y At search time, Splunk scans the time range on a bucket directory name to
determine whether or not to open the bucket and search inside
db_1389230491_1389230488_5
db_1390579247_1390579086_18
Youngest event in the
bucket
Oldest event in the bucket
Unique ID
Note
When clustering is used, index
replication adds further identifiers
to the directory name.

Freezing: Data Expiration
Y The oldest bucket is deleted from the index when:
– The index's maximum overall size is reached
– The bucket's age exceeds the retention time limit
ê  All the events in the bucket have expired
Y Splunk will never exceed the maximum overall size of an index
– Therefore, buckets can be deleted even if they have not reached the time limit
Y You can optionally configure the frozen path
–  Splunk copies the bucket's rawdata to this location before deletion
– Once frozen, buckets are not searchable
Y Frozen data can be brought back (thawed) into Splunk if needed

Index
Storing Buckets in a Separate Volume
Y Best Practice: Use a single high performance file system to store indexes
– The time span of the buckets and their storage type can affect search performance
Y However, you can use multiple volume partitions for index data
– Specify a separate volume for hot/warm and cold buckets during index creation
– Hot and warm buckets should be on the fastest partition and are searched first
– Cold can be located on a slower, cheaper storage (or SAN/NAS)
Y wiki.splunk.com/Deploy:BucketRotationAndRetention
On Fast SSD:
From -90d to Now
On Slower SAN/NAS
Older than -90d

Estimating Index Growth Rate
Y Splunk compresses raw data as it is indexed
– Index data is then added to each bucket
ê  If your data has many searchable terms, the index data is larger
ê  If the data contains fewer searchable terms and less variety, the index is smaller
Y Best practice: get a good growth estimate
– Input your data in a test/dev environment over a sample period
ê  You should index more than one bucket of data
– Examine the size of the index's db directory compared to the input
ê  DMC: Indexing > Indexes and Volumes > Index Detail: Instance
http://docs.splunk.com/Documentation/Splunk/latest/Capacity/Estimateyourstoragerequirements

Calculating Index Storage
Y Limiting size on disk is the most common method of controlling index growth
Y Allocate disk space to meet data retention needs
– Daily Rate * Compression Factor * Retention Period (in days) + Padding
Y Example: 5 GB/day of security data searchable for 6 months (with compression factor of .5)
– 900 GB (5 GB x 180 days) * .5 (CF) + 50 GB (padding) = 500 GB
– On average, data moves to frozen in this index after ~6 months
500

Managing Indexes with Splunk Web
Select Settings > Indexes
Click New to create a new index
Custom indexes can be enabled/disabled or deleted
Click an index name to edit it
Displays the app the index is
configured in and its home path

Adding an Index With Splunk Web
Specifynameforthenewindex
–  MustbeASCIIcharactersandcannotstartwithan“_”or“-”
Leaveblanktousethedefaultlocations
–  Default=$SPLUNK_DB/soc/[db|colddb|thaweddb]
–  Tousecustomlocations,specifyfullindexpaths
Enabledataintegritycheck(optional)
Setmaximumindexsize(default=500GB)
–  Thissettingoverridesallothersizeandretentionsettings
Setmaxsizeofabucket(default=auto(750MB))
–  Useauto_high_volume whendailyvolumeis >10GB
–  Or,provideaspecificsize
Specifythepathtoarchivetherawdatabuckets
Selectwheretheindexes.conffileshouldbesaved
300
soc

/mnt/ssd/soc/db

volume:raid/soc/colddb
6
5
4
3
2
1
6
5
4
3
2
1
auto_high_volume
7
7

Index Data Integrity Check
Y Provides an ability to validate that data has not been tampered with after indexing
docs.splunk.com/Documentation/Splunk/latest/Security/Dataintegritycontrol
Y When enabled, produces calculated hash files for auditing and legal purposes
– Works on index level (including clustering)
– Not for inflight data from forwarders
– To prevent data loss, use the indexer acknowledgment capability (useACK)
Y To verify the integrity of an index/bucket:
– splunk check-integrity -bucketPath [bucket_path] [verbose]
– splunk check-integrity -index [index] [verbose]
Y To re-generate hash files:
– splunk generate-hash-files [-bucketPath|-index] [bucket_path|index]

indexes.conf
Y Many advanced/optional attributes are not available in Splunk Web
– The index stanza is created in local/indexes.conf of the selected app
[volume:raid]
path = /datastore/splunk
[soc]
homePath = /mnt/ssd/soc/db
coldPath = volume:raid/soc/colddb
thawedPath = $SPLUNK_DB/soc/thaweddb
maxDataSize = auto_high_volume
maxTotalDataSizeMB = 307200
enableDataIntegrityControl = 0
Index name in
square brackets
You must specify
home, cold, and
thawed paths,
even when using
the defaults
Size of bucket set
to 10 GB
Setting the total
size of the index at
300 GB

Lab Exercise 5a – Add Indexes
Y Tasks:
– Create two new indexes: securityops and itops
– Add a file monitor input to send events to the securityops index.

Customizing Index Retention Policies
Y To set more advanced/specific options, edit the stanza in indexes.conf
Y New indexes default to 3 hot buckets at a time
– If it is likely that an index will receive events that are not in
time-sequence order, increase the number of available
hot buckets
Y High-volume indexes should have up to 10 hot buckets
– Set with the maxHotBuckets key
Y Best practice for high-volume indexes:
– Examine and copy settings of main index stanza and adjust for your case
Warning
Inappropriate retention settings
can cause premature bucket
rotation and even stop Splunk.

indexes.conf Options
Number of Hot buckets
(maxHotBuckets=3)
Age
(maxHotSpanSecs=7776000)
Bucket size
(maxDataSize = auto)
db directory size
(homepath.maxDataSizeMB=0)
Number of Warm buckets
(maxWarmDBCount=300)
Maximum Index Size (maxTotalDataSizeMB=500000)
Age (frozenTimePeriodInSecs=188697600)
Create Rename Move Delete or Archive
colddb directory size
(coldPath.maxDataSizeMB=0)
[itops]
frozenTimePeriodInSecs = 31536000
maxHotSpanSecs = 86400
homePath.maxDataSizeMB = 100000
coldPath.maxDataSizeMB = 400000
...
Note
Values in parenthesis
are the defaults.

Strict Time-based Retention Policies
Y Example: Purge HR data when it is more than 90 days old, but no sooner
Y Issues to consider:
– Splunk freezes entire buckets, not individual events
– If a bucket spans more than one day, you can't meet the 90 day requirement
Y Configuration option:
frozenTimePeriodInSecs=7776000 (90 days)
maxHotSpanSecs = 86400
ê  Automatically "snaps" the span to the beginning of the time period
ê  For 86400 (24 hours), hot buckets roll at midnight
ê  Consider increasing max. number of hot buckets: maxHotBuckets = 10

Volume-based Retention Policies
Y Example: Prevent data bursts in one index from triggering indexing issues
elsewhere in the same volume
– Splunk in itself cannot determine the maximum size for non-local volumes
– Hot/warm and cold buckets can be in different volumes
– If the volume runs out space, buckets roll to frozen before frozenTimePeriodInSecs
Y Configuration Options: Use volume reference if a retention based on size is desired
[volume:fast]
path = /mnt/ssd/
maxVolumeDataSizeMB = 500000
[volume:slow]
path = /mnt/raid/
maxVolumeDataSizeMB = 4000000
[soc]
homePath = volume:fast/soc/db
homePath.maxDataSizeMB = 50000
coldPath = volume:slow/soc/colddb
coldPath.maxDataSizeMB = 200000


Viewing Indexing Activity and Health
Y Distributed Management Console
– Provides comprehensive indexing activity details
– Snapshot shows averages over the previous 15 minutes
– Historical exposes trending and possible decaying health
Y  Queue fill-ratio
Y  Indexing rate
Y  CPU activity
Y  Volume usage per index
Y  Index size over time
Y  Detailed Indexing status
Y  Retention policies
Y  Bucket configuration details
Displays the usage vs. capacity,
if the volume-based retention is
used (maxVolumeDataSizeMB)

Inspecting Bucket Details
Y Search command: | dbinspect [index=name] [span|timeformat]
– Returns information like status, number of events, timestamps of oldest and
newest events, total bucket size, filepath, etc.
– docs.splunk.com/Documentation/Splunk/latest/SearchReference/Dbinspect

What to Back Up
Y Indexed event data – the Splunk index
– SPLUNK_HOME/var/lib/splunk/
– Or the directories where you placed your indexes (see indexes.conf for details)
Y (Optional) the source log data for additional redundancy
Y SPLUNK_HOME/etc for config and other important files
apps
users
system/local
licenses
init.d
passwd
andmore

Backup Recommendation
Y Use the incremental backup of your choice
– Warm and cold buckets of your indexes
– Configuration files
– User files
Y Hot buckets cannot be backed up without stopping Splunk
– Use the snapshot capability of underlying file system to take a snapshot of hot,
then back up the snapshot
– Schedule multiple daily incremental backups of warm
ê  Works best for high data volumes

Moving an Entire Index
1. Stop Splunk
2. Copy the entire index directory to new location while preserving
permissions and all subdirectories
– *NIX: cp -rp <source> <target>
– Windows: xcopy <source> <target> /s /e /v /o /k (or, robocopy)
3. If this is a global change, unset the SPLUNK_DB environment variable and
update SPLUNK_HOME/etc/splunk-launch.conf
4. Edit indexes.conf to indicate the new location
5. Start Splunk
6. After testing and verifying new index, the old one can be deleted

Removing Indexed Data
Y Sometimes you have unwanted data in an index
– There are no index editors
– First, you should change your configuration to omit the data in the future
Y What are your options for the data already in the index?
– Let the data age-out normally
– Use the delete command to make the unwanted data not show up in searches
– Run splunk clean command to delete all data from the index
– Delete the index

Deleting Events
Y Assign users to the can_delete role
– Specifically set up for deletions
– NOTrecommended to give this capability to other roles
ê  By default, even the admin role does not have the ability
Y Delete CANNOT be undone:
– Log into Splunk Web as a user of the can_delete role
– Create a specific search that identifies the data you want to delete
ê  Double check that the search ONLYincludes the data to delete
ê  Pay special attention to which index you are using and the time range
– After you are certain you’ve targeted only the data you want to delete,
pipe the delete command
Note
This is a "virtual" delete. Splunk
marks the events as deleted and
they never show in searches
again. However, they continue to
take up space on disk.

Cleaning Out an Index
Y To flush indexed data and reset an index, use the CLI clean command
– DATAWILLBE PERMANENTLYDESTROYED
– Typically used on test/dev systems, not for production systems
Y Command syntax:
splunk clean [eventdata|userdata|all] [-index name]
ê  eventdata – delete indexed events and metadata on each event
ê  userdata – delete user accounts
ê  all – everything - including users, saved searches, and alerts
– ALWAYS SPECIFYAN INDEXTOAVOIDTEARS
ê  If no index is specified, the default is to clean all indexes

The Fishbucket
Y The fishbucket index stores the checkpoint data for monitor inputs
Y To reset the individual input checkpoint, use the btprobe command:
Y Requires stopping the forwarder or indexer
Y Other options:
– splunk clean eventdata _thefishbucket
ê  Force re-indexing of all file monitors in the indexer
– rm -r ~/splunkforwarder/var/lib/splunk/fishbucket
ê  Manually delete the fishbucket on forwarders
splunk cmd btprobe –d SPLUNK_HOME/var/lib/splunk/
fishbucket/splunk_private_db --file <source> --reset
Note
Resetting the monitor checkpoint
re-indexes the data, resulting in
more license usage.

Restoring a Frozen Bucket
Y To thaw an archived bucket:
– Copy the bucket directory from the archive to the index's thaweddb directory
– Stop Splunk
– Run splunk rebuild <path to bucket directory>
ê  Also works to recover a corrupted directory
ê  Does not count against license
– Start Splunk
Y Data in thaweddb is searchable along with other data, is not frozen, and does not
count against index total size
– Delete the bucket directory when no longer needed and restart Splunk

Index Replication
Y Splunk indexers can function as a cluster
– Indexers in a cluster can be configured to replicate buckets amongst themselves
Y Index replication allows for rapid failure recovery
Y Fully configurable replication allows you to balance speed of recovery and
overall disk usage
Y Index replication requires additional disk space
Y Discussed in detail in Splunk ClusterAdministration class
Y Basic indexer cluster concepts:
– http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Basicconcepts

Further Reading
Y docs.splunk.com/Documentation/Splunk/latest/Indexer/
HowSplunkstoresindexes
Y docs.splunk.com/Documentation/Splunk/latest/Indexer/
Setlimitsondiskusage
Y docs.splunk.com/Documentation/Splunk/latest/Indexer/Automatearchiving
Y wiki.splunk.com/Deploy:BucketRotationAndRetention

Lab Exercise 5b –Configure Retention Policies
Y Tasks:
– Configure a more strict time-based retention policy for securityops
– Configure a volume-based retention policy for itops
– Use DMC to monitor indexing activities and retention settings

Module 6:
Users, Roles, and
Authentication

Module Objectives
Y Describe user roles in Splunk
Y Create a custom role
Y Integrate Splunk with LDAP or SAML

Managing Users and Roles
Y Users and roles define user privileges
Y To have access to a Splunk instance, a
user must have:
– ASplunk user account
– Assignment to one or more Splunk roles
Y User accounts can be assigned to:
– Native Splunk accounts
– LDAPorActive Directory
– SAML
– Scripted access to PAM, RADIUS, or other
user account systems
Select Settings > Access controls

Identifying Roles
Y Five built-in user roles:
– admin, power, and user
ê  Users can be assigned these roles
– can_delete
ê  Will be discussed separately
– splunk-system-role
ê  Special role that allows system services to
run without a defined user context
Y Administrators can add custom user
roles

Defining Custom User Roles
Give the role a name and
select a default app
Optional
Restrict searches on
certain fields, sources,
hosts, etc
Default is -1
(no restriction)

Defining Custom User Roles (cont.)
Optional user-level and role-level limits

Describing Role Inheritance
Y Anew role can be based on one or more existing roles
Y The new role inherits both capabilities and index access
Y You cannot turn off inherited capabilities or access

Implications of Inheritance
Y If you create a new role that inherits from another role—such as, user:
– The new role has all the capabilities of the inherited role
ê  For example, "run real-time searches"
– The new role inherits the index settings (both default and allowed)
– In the new role, you cannot turn off capabilities or index access that were
inherited from the original role
Y If you want a role that is "like" user but with some capabilities turned off:
– Make a new role that does not inherit from any other role
– Turn on all of the same capabilities as in User, except those you want turned off
– Assign the appropriate indexes to the new role

Defining Role Capabilities
Add or remove capabilities (authorize.conf.spec)
docs.splunk.com/Documentation/Splunk/latest/Admin/authorizeconf

edit_roles_grantable Capability
Y Example: I want to separate and delegate administration tasks between
sys-admins and data admins without granting full admin role
– With edit_roles and edit_user capabilities, users can promote self to full
admin role
– Want to restrict grantable capabilities only to the level sub-admins currently have
Y Configuration option:
– Add the edit_roles_grantable capability to the sub-admin role
ê  Can only create roles with subset of the capabilities that the current user role has
ê  Must use in conjunction with the edit_user capability

Example: edit_roles_grantable Capability
Add new role user_admin
Y Inheritance:
Y  power
Y  user
Y Capabilities:
Y  edit_roles_grantable
Y  edit_user
admin acurry
Assign acurry to the user_admin role acurry can only assign limited roles to users
New users

Defining Role Index Search Options
Y You can specify which indexes are searched if the user does
not specify "index=<index_name>"
– Usually, this should match the Indexes setting

Defining Role Index Access Options
Y Most important index setting
–  Controlswhichindexestheusersinthisrolecanaccess
–  Ifnotselected,userscannotsearchorevenseethisindex
Y Inherited indexes are still available even when they are not listed
Y The default is All non-internal indexes
–  Revisitaccesscontroleachtimeanewindexhasbeenadded

Splunk Authentication Options
Y You need the change_authentication Splunk capability to configure
Y Saves the settings in authentication.conf
– Splunk native
– LDAP
– SAML2.0
– Scripted SSO

Splunk Native Authentication
Y You can create user accounts directly in Splunk
– Example: the default admin user
Y Passwords are stored in SPLUNK_HOME/etc/passwd
Y Use a blank passwd file to completely disable native authentication BUT
– In all authentication scenarios, best practice is to keep a failsafe account here
with a VERYstrong password
Y You can have a mix of Splunk and LDAP or other users
– Splunk native authentication always takes precedence over others
Y Select Settings >Access Controls > Users to manage all users and to
create Splunk user accounts

Managing Users in Splunk
Y Splunk native users can be edited or deleted
Y Only time zone and default app can be changed on LDAPor other users
Add new Splunk user
Click to edit the user settings

Splunk Native Authentication: Add Users
Y Required:
– Username and password
Y Optional:
– Full name and email address (defaults to
none)
– Time zone (defaults to search head time zone)
– Default app (defaults to role default app, or
home if no role default app)
– Role(s)
ê  Defaults to user

Directory Server Integration
Y Best practice: integrate Splunk with a directory server
– Works with multiple LDAPservers, including OpenLDAPandActive Directory
– You can configure from Splunk Web settings
Y User accounts are stored in directory server
– Enforce the same user account and password policy
– Users use the same user name and password in Splunk that they use elsewhere
– Optionally, the groups in the directory server can be mapped to Splunk roles
ê  Or, this can be done manually in Splunk

LDAPAuthentication
LDAP maintains the user credentials - user ID and password, plus other
information - centrally and handles all authentication
Log user in Splunk
Web
Client
LDAP
Server
Splunk
Request Splunk login
Authentication granted
Check authentication
Create user session
Check group mapping

Creating an LDAP Strategy
1.  SelectLDAP
2.  ClickConfigure Splunk touse LDAP
–  Thelistofcurrent LDAPstrategies
displays
–  Astrategyisaconnectiontooneor
moreLDAPnodes on an LDAPserver
–  Candefinemultiple LDAPservers
3.  ClickNewtoaddanewLDAPstrategy
–  Namethestrategy and fillout the form
1
2
3

LDAP Strategy Settings
Y Normally the configuration is based on
the information given to you by the
LDAPadministrators
– LDAPconnection settings
– User settings
ê  DeterminewhichpartoftheLDAP
directorystoresSplunkusers
– Group settings & Dynamic group
settings
ê  Determinewhichnodeinthedirectory
containsyourgroupdefinitions
– Advanced settings
host = 10.0.0.150
port = 389
SSLEnabled = 0
bindDN = adsuser@buttercupgames.local
bindDNpassword = <some_hashed_pw>

userBaseDN = OU=splunk,DC=buttercupgames,DC=local
userNameAttribute = samaccountname
realNameAttribute = displayname

groupBaseDN = OU=splunk,DC=buttercupgames,DC=local
groupNameAttribute = cn
groupMemberAttribute = member
nestedGroups = 0
groupMappingAttribute = dn

network_timeout = 20
sizelimit = 1000
timelimit = 15

Mapping LDAP Groups to Roles
Select Map groups to define
relationships between LDAP
groups and Splunk roles
Click a LDAP group name to
map it to one or more Splunk
roles

Mapping LDAP Groups to Roles (cont.)
Y Not all groups must be mapped
Y Mappings can be changed at any time
– The LDAPserver is rechecked each time a user logs into Splunk
Click one or more role names
to map them to to this group
After completing the mapping for
all LDAP groups, the mapped
roles are shown here

Further Information about LDAP
Y Splunk caches user data from LDAP
Y New user data is cached the first time that user logs in
Y See docs for more details on setting up LDAPmapping:
docs.splunk.com/Documentation/Splunk/latest/Security/SetupuserauthenticationwithLDAP
Y Can also be done in config files: docs.splunk.com/Documentation/Splunk/latest/Admin/authenticationconf
Click this to force an
immediate reload

SAML 2.0 Single Sign On
Identity provider (IDP) maintains the user credentials and handles
authentication
Direct to default app
Web
Client
IDP
Splunk
Request Splunk login
Post Assertion (grant/deny)
Redirect authentication
Create session cookie
Validate assertion
Configured with trusted binding
Challenge for credentials
Check group mapping

Configuring SAML
Y Configure this on a search head:
1.  Select SAML
2.  Click Configure Splunk to use
SAML
ê  The list of current SAMLgroups
displays
3.  Click SAMLConfiguration to
configure the trusted binding and
other connection details
4.  Click New Group to map the
roles
ê  Group-to-Role mapping is same
as LDAP
1
2
3 4

Configuring SAML (cont.)
1.  Enter the following info and save: (provided by the IDPadministrator)
– Single Sign On (SSO) URL
– IDP's certificate file
– Entity ID
– Attribute query URL
– User / Password
2.  Export Splunk (Service Provider) metadata:
http://<splunk_web>/saml/spmetadata
– IDPadministrator imports it into its system and
configures its settings
– IDPadministrator exports its configured metadata
3.  Import IDPmetadata and update the SAML
configuration settings

Splunk
Single Sign On with Reverse Proxy
Y Splunk SSO allows you to use a web proxy to handle Splunk authentication
–  Authentication is moved to a web proxy,whichpassesalongauthenticationtoSplunkWeb
–  Webproxycanuseanymethodtoauthenticate(IDPinexample)
docs.splunk.com/Documentation/Splunk/latest/Security/HowSplunkSSOworks
Web
Client
Splunk request
Proxy passes request with user name
Proxy authorizes client
Proxy
Server
IDP
Splunk Web returns page to proxy
Proxy returns page

Scripted Authentication
Y There are other types of authentication systems that Splunk can integrate
with using scripts
Y For the most up-to-date information on scripted authentication, see the
README file in:
SPLUNK_HOME/share/splunk/authScriptSamples/
– The directory includes sample authentication scripts

Lab Exercise 6 – Add Roles and Users
Y Tasks:
– Create a custom role
– Configure Splunk to use LDAPauthentication
– Map LDAPgroups to Splunk roles
– Verify the configurations
Y Lab notes:
– open.sesam3 is the password for all LDAPaccounts

Module 7:
Universal Forwarders

Module Objectives
Y Install a universal forwarder
Y Configure the forwarder to connect to an indexer
Y Test the forwarder connection
Y Describe optional forwarder settings

Forwarders and Indexers
Y In a production environment
– Splunk indexer(s) runs on dedicated
servers
– The data you want is on remote machines
Y Install Splunk forwarders on the remote
machines to
– Gather the data
– Send it across the network to the Splunk
indexer(s)
Y Indexers listen on a receiving port for
the forwarded data

Splunk Universal Forwarder
Y Universal forwarder gathers data from a host and sends it to an indexer
Y Specifically designed to run on production servers
– Alightweight Splunk instance designed to run on a mission-critical system
– Minimal CPU and memory usage
– Output bandwidth constrained to 256 KBps by default
– No web interface
Y Aseparate installation binary
– Free built-in license, no limits
Y Best Practice: use the Universal Forwarder if possible

Configuration Steps
1. Set up a receiving port on each indexer
– It is only necessary to do this once
2. Download and install Universal Forwarder
– Change password from changeme
3. Set up forwarding on each forwarder
4. Add inputs on forwarders, using one of the following:
– Forwarder management (discussed in the next module)
– CLI
– Manually

Configure the Receiving Port on Each Indexer
Y In Splunk Web:
1.  Select Settings > Forwarding and receiving
2.  Next to Configure receiving, select Add new
3.  Enter a port number and click Save
Y  Or, with CLI:
splunk enable listen <port>
Y  The configuration is saved in
inputs.conf as:
[splunktcp://portNumber]

Installing Universal Forwarder Manually
Y *NIX: unpack the .tgz or .tgz.Z in the desired location
Y Windows: execute the .msi or use the command line
– Installed as a service
Y SPLUNK_HOME is the installation directory:
– /opt/splunkforwarder or c:Program FilesSplunkUniversalforwarder
Y Same splunk command-line interface in SPLUNK_HOME/bin
– Same commands for start/stop, restart, etc.
– The initial admin account password is changeme
ê  Use splunk edit user admin –password newpassword
Y When installing large numbers of forwarders, use an automated method

Using the Interactive Windows Installer
Y Most forwarder settings can be configured using the installer wizard
– Can run as a domain user without the domain user local administrator privileges
Y CLI installation is available for scripted installations
–  docs.splunk.com/Documentation/Splunk/latest/Forwarding/DeployaWindowsdfviathecommandline

Forwarder Configuration Files
Y Forwarders require outputs.conf
– outputs.conf points the forwarder to the receiver(s)
– Can specify additional options for load balancing, SSL, compression, alternate
indexers, and indexer acknowledgement
[tcpout:splunk_indexer]
server = 10.1.2.3:9997
Production Server with Forwarder
[splunktcp://9997]

outputs.conf inputs.conf
Data feeds
from
inputs.conf
Receiver 10.1.2.3 (indexer)
TCP
stream to
port 9997
This stanza instructs the indexer to
listen on port 9997 for feeds from
Splunk forwarders
server includes one or more target receivers,
separated by commas
server can be IP or DNS name plus receiver port

Defining Target Indexer on the Forwarder
Y Run: splunk add forward-server indexer:receiving-port
– For example, splunk add forward-server 10.1.2.3:9997
configures the outputs.conf as:
docs.splunk.com/Documentation/Splunk/latest/Forwarding/Configureforwarderswithoutputs.confd
[tcpout]
defaultGroup = default-group
[tcpout:default-group]
server = 10.1.2.3:9997
[tcpout-server://10.1.2.3:9997]

Testing the Connection
Y After running splunk add forward-server, the forwarder should be
communicating with the indexer
– Splunk forwarder logs are automatically sent to the indexer's _internal index
Y To check for successful connection:
– On the indexer, search index=_internal host=forwarder_hostname
– On the indexer, run splunk display listen
– On the forwarder, run splunk list forward-server
Y To remove the target indexer setting:
– On the forwarder, run splunk remove forward-server indexer:port

Troubleshooting Forwarder Connection
Y Is the forwarder sending data to the indexer?
– Check SPLUNK_HOME/var/log/splunk/splunkd.log on the forwarder
Y Does the indexer receive any data on the listening port?
– Search on indexer:
– To get the <uf>, run on the forwarder:

tail –f var/log/splunk/splunkd.log | egrep 'TcpOutputProc|TcpOutputFd'
index=_internal sourcetype=splunkd component=TcpInputConfig OR
(host=<uf> component=StatusMgr)
splunk show default-hostname

Additional Forwarding Options
Y Compressing the feed
Y Securing the feed
Y Automatic load balancing to multiple indexers
Y Forwarder queue size
Y Indexer acknowledgement to forwarder
Y Selectively forwarding data to indexers

Compressing the Feed
Receiving indexer 10.1.2.3
server = 10.1.2.3:9997
compressed = true

Forwarders
[splunktcp:9997]
compressed = true

Set compression on both sides
(slightly increases CPU usage)
outputs.conf inputs.conf
Compressed
stream
Non-compressed feeds for this
port are ignored

Securing the Feed – SSL
Receiver 10.1.2.3
server = 10.1.2.3:9997

sslPassword = ssl_for_m3
sslCertPath = SPLUNK_HOME/etc/auth/cert1/server.pem
sslRootCAPath = SPLUNK_HOME/etc/auth/cert1/cacert.pem
Forwarders
[splunktcp-ssl:9997]

[ssl]
password = ssl_for_m3
serverCert = SPLUNK_HOME/etc/auth/cert1/server.pem
rootCA = SPLUNK_HOME/etc/auth/cert1/cacert.pem

outputs.conf
inputs.conf
Secure
Feed
SSL settings for all inputs in
dedicated SSL stanza
Turning on SSL:
Y  Can increase the CPU usage
Y  Automatically compresses the feed

Splunk 6.4 Administration.pdf

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Splunk 6.4 Administration.pdf

Similar to Splunk 6.4 Administration.pdf (20)

More from nitinscribd

More from nitinscribd (8)

Recently uploaded

Recently uploaded (20)

Splunk 6.4 Administration.pdf