4. What is Machine Data?
Log files
Custom applications
Web servers
User clickstreams
Social platforms
Servers/hypervisors/virtual machines
Configurations
Telecom devices
Storage devices
Network devices
Security devices, firewalls, IDS
Databases
Web services
System metrics
GPS
DNS, DHCP
AAA logs
Proxy servers
Errors
Scripts
Sensors
5. What Does Machine Data Look Like?
Sources
Order Processing
Twitter
Care IVR
Middleware
Error
6. Machine Data Contains Critical Insights
Customer ID Order ID
Customer’s Tweet
Time Waiting On Hold
Twitter ID
Product ID
Company’s Twitter ID
Customer IDOrder ID
Customer ID
Sources
Order Processing
Twitter
Care IVR
Middleware
Error
7. Machine Data Contains Critical Insights
Order ID
Customer’s Tweet
Time Waiting On Hold
Product ID
Company’s Twitter ID
Order ID
Customer ID
Twitter ID
Customer ID
Customer ID
Sources
Order Processing
Twitter
Care IVR
Middleware
Error
8. Splunk Delivers Value Across IT and the Business
IT
Operations
Application
Delivery
Developer Platform (REST API, SDKs)
Business
Analytics
Industrial Data
and Internet of
Things
Business
Analytics
Industrial
Data and
Internet of
Things
Security,
Compliance
and Fraud
12. Splunk Licenses
Free Download Limits Indexing to 500MB/day
• Enterprise Trial License expires after 60 days
• Reverts to Free License
Features Disabled in Free License
• Multiple user accounts and role-based access controls
• Distributed search
• Forwarding to non-Splunk Instances
• Deployment management
• Alerting/monitoring
• Authentication and user management
Other License Types
• Enterprise, Forwarder, Trial
13. Default installation on: http://localhost:8000
Splunk Web Basics
Browser Support
• Firefox ESR (24.2) and latest
• Internet Explorer 9, 10, and 11
• Safari (latest)
• Chrome (latest)
Index data
• Add data
• Install an App (Splunk for Windows, *NIX,…)
14. Splunk Web Basics continued…
Splunk Home
• Provides Interactive portal to the Apps & data.
Splunk Apps
• Default Search & Reporting App
• Provide different contexts for your data out of
sets of views, dashboards, and configurations
• You can create your own!
15. Optional: add some test data
Download the sample file, follow this link and save the file to your
desktop, then unzip: http://www.splunkbook.com (Using Splunk Book)
To add the file to Splunk:
– From the Welcome screen, click Add Data.
– Click From files and directories on the bottom half of the screen.
– Select Skip preview.
– Click the radio button next to Upload and index a file.
– Click Save.
16. Best Practice Suggestion:
Create an individual Index based on
sourcetype.
• Easier to re-index data if you make a
mistake.
• Easier to remove data.
• Easier to define permissions and data
retention.
19. current view
global stats
app navigation time range
picker
Selecting Data
Summary:
• Host
• Source
• Sourcetype
start
search
search box
20. Searching
Search > *
Select Time Range
• Historical, custom, real-time or
indexed real-time
Select Mode
• Smart, Fast, Verbose
Using the timeline
• Click events and zoom in and out
• Click and drag over events for a specific range
21. Everything is searchable
Everything is searchable
• * wildcards supported
• Search terms are case insensitive
• Booleans AND, OR, NOT
– Booleans must be uppercase
– Implied AND between terms
– Use () for complex searches
• Quote phrases
fail*
fail* nfs
error OR 404
error OR failed OR (sourcetype=access_*(500 OR 503))
"login failure"
23. Search Assistant
Contextual Help
- advanced type-ahead
History
- search
- commands
Search Reference
- short/long description
- examples
suggests search terms
updates as you type
shows examples and help
toggle off / on
24. Searches can be managed as
asynchronous processes
Jobs can be
• Scheduled
• Moved to background tasks
• Paused, stopped, resumed, finalized
• Managed
• Archived
• Cancelled
Job Management
Modify Job Settings
pause
finalize
delete
25. Search Commands
Search > error | head 1
Search results are “piped” to the command
Commands for:
• Manipulating fields
• Formatting
• Handling results
• Reporting
29. Fields
Default fields
• host, source, sourcetype, linecount, etc.
• View on left panel in search results or all in field picker
Where do fields come from?
• Pre-defined by sourcetypes
• Automatically extracted key-value pairs
• User defined
30. Sources, Sourcetypes, Hosts
• Host
- hostname, IP address,
or name of the network
host from which the
events originated
• Source
- the name of the file,
stream, or other input
• Sourcetype
- a specific data type or
data format
32. Tagging and Event Typing
Eventtypes for more human-readable reports
• to categorize and make sense of mountains of data
• punctuation helps find events with similar patterns
Search > eventtype=failed_login instead of
Search > “failed login” OR “FAILED LOGIN” OR “Authentication failure” OR “Failed to
………………authenticate user”
Tags are labels
• apply ad-hoc knowledge
• create logical divisions or groups
• tag hosts, sources, fields, even eventtypes
Search > tag=web_servers instead of
Search > host=“apache1.splunk.com” OR host=“apache2.splunk.com” OR
…………….host=“apache3.splunk.com”
37. Alerting Continued…
Searches run on a schedule and fire an alert
• Example: Run a search for “Failed password” every 15 min
over the last 15 min and alert if the number of events is
greater than 10
Searches are running in real-time and fire an alert
• Example: Run a search for “Failed password user=john.doe” in
a 1 minute window and alert if an event is found
41. Reporting
results of any search
Define your Search and set your time range,
accelerate you search and more
Choose the type of chart (line, area, column, etc) and
other formatting options
Build reports from
42. Reporting Examples
• Use wizard or reporting commands (timechart, top, etc)
• Build real-time reports with real-time searches
• Save reports for use on dashboards
45. Manager Settings
For All of that Cool Stuff
You Just Created (and more!)
• Permissions
• Saved Searches/Reports
• Custom Views
• Distributed Splunk
• Deployment Server
• License Usage….
48. Splunk Has Four Primary Functions
Searching and Reporting (Search Head)
Indexing and Search Services (Indexer)
Data Collection and Forwarding (Forwarder)
Distributed Management (Deployment Server)
Data Governor (Cluster Master)
Databases
Networks
Servers
Virtual
Machines
Smart
phones
and
Devices
Custom
Applications
Security
WebServer
Sensors
A Splunk install can be one or all roles…
50. Scales to Hundreds of TBs/Day
Enterprise-Class Scale, Resilience and Interoperability
Send data from thousands of servers using any combination of Splunk Forwarders
Auto load-balanced forwarding to Splunk Indexers
Offload search load to Splunk Search Heads
51. Visibility Across Datacenters
Distributed search unifies the view
across locations
Role-based access controls how far a given
user's search will span
New York Tokyo
London Cloud
52. Delivers Mission-Critical Availability
• Data replication – maintain
searchability even if servers
go down
• Multi-site capable –
maintain searchability even
if a site goes down
• Search Affinity – optimized
searches by fetching from
the closest/fastest location
REPLICATION
Portland
Datacenter
New York
Datacenter
Clustering
53. Forwards Events to Third-Party Systems
Problem Investigation
Service Desk
Event Console
SIEM
RAW
Formatted
54. Enrich Raw Data to Make It More Meaningful
Create additional fields from
the raw data with a lookup to
an external data source
LDAP,
AD
Watch
Lists
CRM/ERP
CMDB
External Data Sources
Insight comes out
Data goes in
55. Integrate Users and Roles
Problem Investigation Problem Investigation Problem Investigation
Save
Searches
Share
Searches
LDAP, AD
Users and Groups
Splunk Flexible Roles
Manage
Users
Manage
Indexes
Capabilities& Filters
NOT
tag=PCI
App=ERP
…
Map LDAP & AD groups to flexible Splunk roles. Define any search as a filter.
Integrate authentication with LDAP and Active Directory.
56. FrozenWARM COLDHOT
Index
How the Data is Stored and Aged
Hot – Newest buckets of data that are still open for write
Warm – Recent data but closed for writing (read only)
Cold – Oldest data, commonly on cheaper, slower storage
Frozen – No longer searchable, commonly archived or deleted data
58. Support Through the Splunk Community
Browse and share Apps from
Splunk, Partners and the
Community
splunkbase.com
Community-driven
knowledge exchange and
Q&A
answers.splunk.com
Splunk Docs
docs.splunk.com
Welcome to SplunkLive [City].
Thank you for taking the time to attend today’s event.
The goal of today is to share ideas on how you can use machine-generated data to:
Stop the time-consuming cycles of data gathering, investigations and analysis based on the old model of doing things. And perhaps reclaim some personal time.
You’ll see our products and apps live, and hear amazing stories from our customers.
We have three excellent customer speakers who will take you through ‘why and how’ they are using Splunk, tips, best practices and the impact it’s having on them personally and their organization. [CUSTOMER NAMES]
For those of you who are more ‘hands on’ – we have afternoon sessions that help accelerate your practical understanding of Splunk. [DETAILS].
We have one break in the morning, lunch and then a drinks reception at [TIME] – all fantastic networking opportunities with your peers and with us.
You may ask yourself, what the heck is Splunk and why are we called that? Our name came from the idea that digging through machine data , which is what we do, can be a lot like Spelunking .
Machine data is data generated by devices such as servers, storage devices, web streams, mobile technology and more! This data has highly diverse formats This is the data that Splunk manages. We are as important to machine data, as Google is to text data. According to IDC, unstructured data , much of it generated by machines accounts for more than 90% of data in todays organizations.
Unlike traditional structured data or multi-dimensional data– for example data stored in a traditional relational database for batch reporting – machine data is non-standard, highly diverse, dynamic and high volume. You will notice that machine data events are also typically time-stamped – it is time-series data.
Take the example of purchasing a product on your tablet or smartphone: the purchase transaction fails, you call the call center and then tweet about your experience. All these events are captured - as they occur - in the machine data generated by the different systems supporting these different interactions.
Each of the underlying systems can generate millions of machine data events daily. Here we see small excerpts from just some of them.
When we look more closely at the data we see that it contains valuable information – customer id, order id, time waiting on hold, twitter id … what was tweeted.
What’s important is first of all the ability to actually see across all these disparate data sources, but then to correlate related events across disparate sources, to deliver meaningful insight.
If you can correlate and visualize related events across these disparate sources, you can build a picture of activity, behavior and experience. And what if you can do all of this in real-time? You can respond more quickly to events that matter.
For example, if an organizations captured the customers twitter ID in their customer profile this correlation would be possible. Where that didn’t exist, they could at least group by demographic with the tweets.
You can extrapolate this example to a wide range of use cases – security and fraud, transaction monitoring and analysis, web analytics, IT operations and so on.
So what does Splunk do? We turn chaos into order – and make it easy for customers to do their job. We have a Google like interface which makes it easy to use and enables them to get up and running in days and minutes
How can you leverage Splunk?
Follow along if you like!
See full list of supported platforms in Installation Manual.
Can choose different directory during installation.
This license allows forwarding (but not indexing) of unlimited data, and also enables security on the instance so that users must supply username and password to access it. (The free license can also be used to forward an unlimited amount of data, but has no security.)
Good analogy for Apps is iPhone/iPad. Same data, many uses. Apps change the presentation layer.
Illustrate add data, illustrate creating a new index, illustrate the *nix app to show performance metrics.
Also, new splunk overview app that ships with test data for DM and Pivot etc,
How can you leverage Splunk?
1. Wildcards are supported - *
2. Search terms are case insensitive.
3. Boolean searches are supported with AND, OR, NOT. Just remember that Booleans must be uppercase.
4. There is an implied AND between the search terms, and for complex searches, use parenthesis. (error OR failed)
5. You can also quote phrases such as “Login Failure”
6. Search Modes!
This is an example of a search by host excluding events with an error log level
The search assistant offers quick reference for the Splunk search language that updates as you type. That includes links to online documentation, and shows matching searches along with their count, matching terms and examples.
It also shows you your history of searches.
A search becomes a job for Splunk to process. While a search is processing, this job can be Canceled, Paused, sent to the background and Finalized.
The ability to cancel is handy if you made a mistake or chose the wrong time range.
Finalized = stop processing events but build the "number of events" count. Jobs can be accessed while running or after through the jobs menu. There, Paused Jobs can be resumed and those sent to the background can be accessed. Jobs results are kept for a configurable time of 10 minutes by default.
Splunk search language is very unix-like—use the pipe symbol to pass search results to search commands. Search commands can be chained. You can even create your own custom search commands.
These are common commands we find most useful to analyze and filter data. <review each command>
Search reference is available online in addition to the search assistance and covers all search commands.
Much like *nix* operating systems, chances are you’re not going to memorize all of the commands. You’ll memorize a handful, and rely on the “man pages” to get additional context to commands. We SEs here at Splunk use maybe twenty terms in our day to day.
How can you leverage Splunk?
Fields give you much more precision in searches. Fields are key value pairs associated with your data by Splunk. So, an example would be host=www1, status=503. Now there are two specific types of fields. There are default fields, (source, sourcetype and host) which are added to every event by Splunk during indexing.
And there are data-specific fields. These would be action=“purchase” or status=“503”.
What’s the difference between Sources, sourcetypes, and hosts?
A host would be the hostname, IP address or name of the network host from which events originate. An example might be a single windows server would be a host or specific firewall.
A Source is the name of a file, a stream or some other input, such as a config file, process, application or event log, on a server. So per our Windows server example, sources on that server, might include Windows event logs, exchange logs, DNS/DHCP logs, performance metrics as well as the windows event logs from the windows event viewer. Each of these is a different source.
A Sourcetype is a specific data format. Sourcetype would beALL exchange logs or ALL Cisco ASA. It’s a high level group. Running your searches against a sourcetype of Windows Event Log Security across multiple servers.
Extracting fields that aren’t already pulled out at search time is a necessary step to doing more with your data like reporting.
Show example of field extraction with IFX and an example using rex.
Show other field extractor.
Event types can help you automatically identify events based on a search. An event type is a field based on a search, it’s a way of classifying data for searching and reporting and it’s useful for user knowledge capture and sharing.
Tags are different, in that they allow you to search for events with related field values. You can assign any field/value combination. So as an example, server names aren’t always helpful. Sometimes they contain ambiguous information. Using tags you can use a more meaningful term.
The Splunk Manager allows you to enable/disable, copy, delete and edit tags that you’ve created.
How can you leverage Splunk?
Use the time range picker to set time boundaries on your searches. You can restrict the search to Preset time ranges, custom Relative time ranges, and custom Real-time time ranges. You can also specify a Date Range, a Date & Time Range, and use more advanced options for specifying the time ranges for a search.
Real-time alerts always trigger immediately for every returned result
Real-time monitored alerts monitor a real-time window and can trigger immediately, or you can define conditions
Scheduled alerts run a search on a regular interval that you define and triggers based on conditions that you define
Run alert in Splunk.
Splunk alerts are based on searches and can run either on a regular scheduled interval or in real-time.
Alerts are triggered when the results of the search meet a specific condition that you define.
Based on your needs, alerts can send emails, trigger scripts and write to RSS feeds.
Consider how you might use a scripted alert.
How can you leverage Splunk?
Demo building a traditional report. Reports can also be dashboards mailed out.
Demo building a report and dashboard.
Demo new dashboard workflow
Show dashboard examples:
Why with the same settings is the shadow so dark?
How can you leverage Splunk?
These are the five logical roles, a Splunk instance can be one or more of the roles.
The search head is what most users interact with. It is the webserver and app interpreting engine that provides the primary, web-based user interface. Since most of the data interpretation happens as-needed at search time, the role of the search head is to translate user and app requests into actionable searches for it’s indexer(s) and display the results. The Splunk web UI is highly customizable, either through our own view and app system, or by embedding Splunk searches in your own web apps or our API. Additional search heads can be deployed to scale with user or search load.
The core of the Splunk infrastructure is indexing. An indexer does two things – it accepts and processes new data, adding it to the index and compressing it on disk. The indexer also services search requests, looking through the data it has via it’s indices and returning the appropriate results to the searcher over a secure compressed communication channel. Indexers scale out almost limitlessly and with almost no degradation in overall performance, allowing Splunk to scale from single-instance small deployments to truly massive Big Data challenges.
The Splunk forwarder is an optional component that can be installed to forward data from servers, desktops, mainframes, and even ARM based devices. There are two types of forwarders; the full Splunk distribution or a dedicated “Universal Forwarder”. The full Splunk distribution can be configured to filter data before transmitting, execute scripts locally, or run SplunkWeb. This gives you several options depending on the footprint size your endpoints can tolerate. The universal forwarder is an ultra-lightweight agent designed to collect data in the smallest possible footprint. Both flavors of forwarder come with automatic load balancing, SSL encryption and data compression, and the ability to route data to multiple Splunk instances or third party systems.
The Cluster Master coordinates which indexers have copies of which buckets to ensure we have met the proper number of replication and searchable copies of each bucket. All clustered Indexers check in with the Master to alert them of their status, and the status of each of their replicated indexes and buckets. We will talk more about buckets later.
And at the bottom there is the there is the Deployment Server, which can be used to manage your distributed Splunk environment. Deployment server helps you synchronize the configuration of your search heads during distributed searching, as well as your forwarders to centrally manage your distributed data collection. Of course, Splunk has a simple flat-file configuration system, so feel free to use your own config management tools if your more comfortable with what you already have.
Getting data into Splunk is designed to be as flexible and easy as possible. Because the indexing engine is so flexible and doesn’t generally require configuration for most IT data, all that remains is how to collect and ship the data to your Splunk. There are many options.
First, you can collect data over the network, without an agent. The most common network input is syslog; Splunk is a fully compliant and customizable syslog listener over both TCP and UDP. Further, because Splunk is just software, any remote file share you can mount or symlink to via the operating system is available for indexing as well. To facilitate remote Windows data collection, Splunk has a its own WMI query tool that can remotely collect Windows Event logs and performance counters from your Windows systems. Finally, Splunk has a AD monitoring tool that can connect to AD and get your user meta data to enhance your searching context and monitor AD for replication, policy or user security changes.
When Splunk is running locally as an indexer or forwarder, you have additional options and greater control. Splunk can directly monitor hundreds or thousands of local files, index them and detect changes. Additionally, many customers use our out-of-the-box scripts and tools to generate data – common examples include performance polling scripts on *nix hosts, API calls to collect hypervisor statistics and for detailed monitoring of custom apps running in debug modes. Also, Splunk has Windows-specific collection tools, including native Event Log access, registry monitoring drivers, performance monitoring and AD monitoring that can run locally with a minimal footprint.
Historically, a Splunk forwarder was a stripped down version of the full Splunk distribution. Certain features, such as Splunk Web, were turned off to decrease footprint on a remote host. Our customers asked us for something even lighter and we delivered. The Universal Forwarder is a new, dedicated package specifically designed for collecting and sending data to Splunk. It’s super light on resources, easy to install, but still includes all the current Splunk inputs, without requiring python. Most deployments should only require the use of the Universal Forwarder but we have kept all features of forwarding in the Regular (or Heavy) Forwarder for cases when you need specific capabilities.
A single indexers it can index 50-100 gigabytes per day depending the data sources and load from searching. If you have terabytes a day you can linearly scale a single, logical Splunk deployment by adding index servers, using Splunk’s built in forwarder load balancing to distribute the data and using distributed search to provide a single view across all of these servers. Unlike some log management products you get full consolidated reporting and alerting not simply merged query results.
When in doubt, the first rule of scaling is ‘add another commodity indexer.’ Splunk indexers are designed to enable nearly limitless fan-out with linear scalability by leveraging techniques like MapReduce to fan-out work in a highly efficient manner.
Leverage distributed search to give each locale access to their own data, while providing a combined view to central teams back at headquarters. Whether to optimize your network traffic or meet data segmentation requirements, feel free to build your Splunk infrastructure as it makes sense for your organization.
Further, each distributed search head automatically creates the correct app and user context while searching across other datasets. No specific custom configuration management is required; Splunk handles it for you.
The insights from your data are mission-critical. With Splunk Enterprise 5 we wanted to deliver a highly available system, with enterprise-grade data resiliency, even as you scale on commodity storage. And we wanted to maintain Splunk’s robust, real-time and ease of use features.
Splunk indexers can now be grouped together to replicate each other’s data, maintaining multiple copies of all data – preventing data loss and delivering highly available data for Splunk search.
Using index replication, if one or more indexers fail, incoming data continues to get indexed and indexed data continues to be searchable.
By spreading data across multiple indexers, searches can read from many indexers in parallel, improving parallelism of operations and performance. All as you scale on commodity servers and storage. And without a SAN.
Splunk isn’t the only technology that can benefit from IT data collection, so let Splunk help send the data to those systems that need it. For those systems that want a direct tap into the raw data, Splunk can forward all or a subset of data in real time via TCP as raw text or RFC-compliant syslog. This can be done on the forwarder or centrally via the indexer without incrementing your daily indexing volume. Separately, Splunk can schedule sophisticated correlation searches and configure them to open tickets or insert events into SIEMs or operation event consoles. This allows you to summarize, mash-up and transform the data with the full power of the search language and import data into these other systems in a controlled fashion, even if they don’t natively support all the data types Splunk does.
MSSP, Cloud Services, etc.
Your logs and other IT data are important but often cryptic. You can extend Splunk’s search with lookups to external data sources as well as automate tagging of hosts, users, sources, IP addresses and other fields that appear in your IT data. This enables you to find and summarize IT data according to business impact, logical application, user role and other logical business mappings. In the example shown, Splunk is looking up the server’s IP address to determine which domain the servicing web host is located in, and the customer account number to show which local market the customer is coming from. Using these fields, a search user could create reports pivoted on this information easily.
Illustrate Lookups:
Splunk allows you to extend your existing AAA systems into the Splunk search system for both security and convenience. Splunk can connect to your LDAP based systems, like AD, and directly map your groups and users to Splunk users and roles. From there, define what users and groups can access Splunk, which apps and searches they have access to, and automatically (and transparently) filter their results by any search you can define. That allows you to not only exclude whole events that are inappropriate for a user to see, but also mask or hide specific fields in the data – such as customer names or credit card numbers – from those not authorized to see the entire event.
How can you leverage Splunk?
With thousands of enterprise customers and an order of magnitude more actual users, we have a thriving community.
We launched a dev portal a few months back and already have over 1,000 unique visitors per week.
We have over 300 apps contributed by ourselves, our partners and our community.
Our knowledge exchange Answers site has over 20,000+ questions answered.
And in August 2012 we ran our 3rd users’ conference with over 1,000 users in attendance, over 100 sessions of content, customers presenting.
Best of all, this community demands more from Splunk and gives us incredible feedback