What is Splunk? At the end of this session you’ll have a high-level understanding of the pieces that make up the Splunk Platform, how it works, and how it fits in the landscape of Big Data. You’ll see practical examples that differentiate Splunk while demonstrating how to gain quick time to value.
4. Splunk Inc.
4
Public company, founded in 2004
Headquartered in San Francisco
Universal Platform for Machine Data
Any Machine Data
Any Volume
Deployments from 10MB to 350TB/day
On Premise
In the Cloud
SAAS
9,000+ Customers in 100+ Countries
2/3 of the Fortune 100
5. What is Machine Data?
Sources
Order Processing
Twitter
Care IVR
Middleware
Error
6. Machine Data Contains Critical Insights
Order ID
Customer’s Tweet
Time Waiting On Hold
Product ID
Company’s Twitter ID
Order ID
Customer ID
Twitter ID
Customer ID
Customer ID
Sources
Order Processing
Twitter
Care IVR
Middleware
Error
7. Machine Data is Growing Exponentially
Volume | Velocity | Variety | Variability
GPS,
RFID,
Hypervisor,
Web Servers,
Email, Messaging,
Clickstreams, Mobile,
Telephony, IVR, Databases,
Sensors, Telematics, Storage,
Servers, Security Devices, Desktops
Machine data is the fastest growing, most
complex, most valuable area of big data
8. Universal Platform for Machine Data
8
Real-time indexing of ANY machine data
Online
Services Web
Services
Servers
Security GPS
Location
Storage
Desktops
Networks
Packaged
Applications
Custom
ApplicationsMessaging
Telecoms
Online
Shopping
Cart
Web
Clickstreams
Databases
Energy
Meters
Call Detail
Records
Smartphones
and Devices
RFID
On-
Premise
Private
Cloud
Public
Cloud
Local Storage SAN NoSQL
Explore Visualize ShareAnalyze Develop
9. Universal Platform for Machine Data
9
Real-time indexing of ANY machine data
Online
Services Web
Services
Servers
Security GPS
Location
Storage
Desktops
Networks
Packaged
Applications
Custom
ApplicationsMessaging
Telecoms
Online
Shopping
Cart
Web
Clickstreams
Databases
Energy
Meters
Call Detail
Records
Smartphones
and Devices
RFID
On-
Premises
Private
Cloud
Public
Cloud
Developer
Platform
Report
and
analyze
Custom
dashboards
Monitor
and alert
Ad hoc
search
Any amount, any location, any source
Schema-on-the-flyNo
Database
No need
to filter data
10. Splunk Delivers Value Across IT and the Business
IT
Operations
Security
&
Compliance
Web
Intelligence
Application
Management
Developer Platform (Java, Python, JavaScript, PHP, SDKs, REST API)
Business
Analytics
Industrial
Data
Small Data. Big Data. Huge Data.
12. Scales to Hundreds of TBs/Day
Enterprise-class Scale, Resilience and Interoperability
Collect machine data from thousands sources via Splunk forwarders
Compress and store data on Splunk Indexers
Initiate searches and visualize results via Search Heads
13. Delivers Mission-critical Availability
Data replication – maintain
searchability even if servers
go down
Multi-site capable – maintain
searchability even if a site
goes down
Search Affinity – optimized
searches by fetching from
the closest/fastest location
REPLICATION
Portland
Datacenter
New York
Datacenter
Clustering
14. Drastically Reduces Time-to-Value
Over 600 apps available on splunkbase
REST API
XenApp
XenDesktop
Server,
Storage,
Network
Server
Virtualization
Operating
Systems
Infrastructure
Applications
Mobile
Applications
Cloud Services
Other Monitoring
Ticketing/Help
Desk
Custom Biz
Applications
SDKs
Web Framework
16. Install Splunk
Splunk Home
• WIN: Program FilesSplunk
• Other: /opt/splunk (Applications/splunk)
Start Splunk
• WIN: Program FilesSplunkbinsplunk.exe start (services start)
• *NIX: /opt/splunk/bin/splunk start
www.splunk.com/download
17. Splunk Licenses
Free Download Limits Indexing to 500MB/day
• Enterprise Trial License expires after 60 days
• Reverts to Free License
Features Disabled in Free License
• Multiple user accounts and role-based access controls
• Distributed search
• Forwarding to non-Splunk Instances
• Deployment management
• Scheduled saved searches and alerting
• Summary indexing
18. Default installation on: http://localhost:8000
18
Splunk Console
Browser Support
• Firefox 10.x and latest
• Internet Explorer 7, 8, 9 and 10
• Safari (latest)
• Chrome (latest)
19. Indexing Demonstration
Download the sample file, follow this link and save the file to your
desktop, then unzip: http://bit.ly/UBPFWP (Exploring Splunk Book)
To add the file to Splunk:
– Click Add Data
– Click Upload files from my computer.
– Drag and drop you sample data zip file.
– Add a new Index
– Review and Finish.
19
22. Settings Demonstration
22
For All of that Cool Stuff
You Just Created (and more!)
• Permissions
• Saved Searches/Reports
• Custom Views
• Distributed Splunk
• Deployment Server
• License Usage….
27. Distributed Architecture
Indexer
27
Compresses, Index and Search up to 150GB/day
Compressed Raw Data (~15% raw data size)
Time Series Index (~35% raw data size)
Executes Searches
Scales Horizontally via Commodity Hardware
6X2 Core CPUs/12GB RAM/800+ IOPs
28. Distributed Architecture
Search Head
28
Initiates Distributed Searches
Publishes Reports/Dashboards/Apps
Scales Horizontally via Commodity Hardware
4X4 Core CPUs/12GB RAM/2 x 300GB, 10,000 RPM SAS Raid 1
29. Scalability & High Availability
29
Forwarders load balance across
Indexers
Indexed data can be replicated across
peers and different physical sites
Search Heads can be Clustered to
eliminate single point of failure and
handle large search loads
31. Integrate External Data
31
LDAP, AD Watch
Lists
CRM/ER
P
CMDB
Correlate IP addresses with locations, accounts with regions
Extend search with lookups to external data sources.
32. Integrate Users and Roles
32
Problem Investigation Problem Investigation Problem Investigation
Save
Searches
Share
Searches
LDAP, AD
Users and Groups
Splunk Flexible Roles
Manage
Users
Manage
Indexes
Capabilities& Filters
NOT
tag=PCI
App=ERP
…
Map LDAP & AD groups to flexible Splunk roles. Define any search as a filter.
Integrate authentication with LDAP and Active Directory.
35. Support Through the Splunk Community
35
Browse and share Apps
from Splunk, Partners and
the Community
splunkbase.splunk.com
Splunkbase
Community-driven
knowledge exchange
and Q&A
answers.splunk.com
5 tracks, more than 40
sessions, the smartest
Splunk users together
conf.splunk.com
.conf2014
36. Where to Go for Help
36
• Documentation
– http://www.splunk.com/base/Documentation
• Technical Support
– http://www.splunk.com/support
• Videos
– http://www.splunk.com/videos
• Education
– http://www.splunk.com/goto/education
• Community
– http://answers.splunk.com
• Splunk Book
– http://splunkbook.com
Splunk Inc. is a public company, founded in 2004 with the goal of providing a universal platform for machine data.
We have more than 9,000 global customers with deployments ranging from home use to massive enterprises indexing 250TB of data a day.
So what is machine data?
Every machine on the planet, from internal servers and applications to call center hardware and the networks they run on to social media all generate some kind of structured or unstructured machine data.
Within that machine data are critical insights about the performance availability and value of business services provided to the end user.
But there is a challenge with this Machine Data. It is growing exponentially.
But there is a big challenge with this Machine Data. It is growing exponentially both in volume as well as complexity. – And that’s where Splunk comes in.
No matter what type of machine data the systems powering your business produce, Splunk can collect and index that data in real-time; allowing you to immediately explore, Analyze and visualize that data, and achieving what we at Splunk refer to as “Operational Intelligence”.
As machine data flows into Splunk, it is compressed and stored on local storage or a SAN for real-time investigation and alerting of incidents or Security challenges.
NoSQL Datastores can be used to store longer-term machine data not required for real-time analysis or can be queried directly to correlate the petabytes of unstructured NoSQL Datastore data with the unlimited types of Machine data indexed by Splunk.
In addition to a wizard-driven console for easily visualizing all of your data, Splunk provides a full development platform for collecting new Machine Data types or sharing data with third party products, ticketing systems or custom interfaces.
All of this is done without the limitation of a back end relational database. Which means the integrity of your data is never compromised, any field within your machine data is fair game for investigation, and scalability is never an issue.
Machine data is useful across both IT and the Business. Splunk use cases range from standard IT Operations management to Security and Business Analytics.
Splunk Dashboards are easy to generate and customize to meet the needs of a wide variety of roles within your organization.
And no matter how much machine data is generated by your business processes and services, Splunk can scale to meet your needs.
Splunk Forwarders are lightweight components which collect Machine data throughout your environment. Forwarder deployment is highly customizable, you can have the forwarder remotely collect data or place the forwarder locally on hundreds of thousands of devices as some of our customers do.
Forwarders automatically load-balance their collected machine data across a pool of Indexers, which scale horizontally on commodity hardware to adjust to your growing pool of Machine Data.
Search Heads initiate map-reduced searches across the indexer tier, combine and return the results to the Splunk console or your interface of choice.
Like Indexers, Search Heads can scale horizontally to meet your needs on commodity hardware.
Even if an entire datacenter were to go down, Splunk’s ability to replicate data across sites ensures a true high availability environment.
Finally, Splunk’s robust community of users and partners have created more than 500 apps which plug into Splunk Enterprise to get you up and running quickly.
The vast majority of these apps are free. Some apps, which have dedicated SCRUM development and support teams are charged for.
For Example:
HUNK is an application which provides easy NoSQL Connectivity, Search, and Reporting capabilities across your NoSQL nodes
The Exchange App provides out of the box monitoring of your email environment
The VM Ware app monitors virtual environments.
And Splunk’s Enterprise Security app provides Security Analysts with the ability to search across all Machine Data within an environment, not just data limited by a particular SIEM; which has propelled Splunk to a “Leader” in the Gartner Magic Quadrant for Security.