Are you outgrowing your initial Splunk deployment? Is Splunk becoming mission critical and you need to make sure it's Enterprise ready? Attend this session led by Splunk experts to learn about taking your Splunk deployment to the next level. Learn about Splunk high availability architectures with Splunk Search Head Clustering and Index Replication. Additionally, learn how to manage your deployment with Splunk’s operational and management controls to manage Splunk capacity and end user experience.
2. 2
Splunk at the Next Level
Time to move beyond initial Splunk environment
• More use cases – how to tackle?
• More data – how do we scale?
• Splunk is mission critical == HA
• Global deployments
• Splunk user experience Screenshot here
4. 4
Growing your Splunk Deployment
Many customers start with a single use case…
• Ex: Monitor the web servers
• Help ensure up-time & response times
• Track usage, errors
• Provides business value
5. 5
Growing your Splunk Deployment
Value statement for each overall service
Your services exist in a larger context than just one app, or one tier.
What is the value of the service as a whole?
What are CIO commitments for the service?
• The company’s web store is one of the most critical parts of the business.
• Performance of the overall environment must be maintained at all times.
• Failures in any portion of the web store must be quickly identified, send
notification to the appropriate parties.
• Dependencies on external processes must be monitored as well.
6. 6
Growing your Splunk Deployment
The larger context
• Failure in one system cascades
• Map dependencies, estimate costs
• Use Splunk to track all dependencies.
• What happens when it is down?
Dependencies often include:
• Networking dependencies
• Shared storage
• Databases, middleware, custom apps
• Virtualization layer
Screenshot here
10. 10
Scaling - Storage
Simple storage to complex
Raw data rate net compression of ~ 50% on disk.
Simple: rate * compression * retention
200 GB / day * 50% * 100 days = 10TB
Consider cold storage on NAS
– Changes storage story.
– Retention on fast, retention on slow
Clustering
– Changes storage story
12. 12
Scaling - Storage
RAID + SSD deep dive
• For spinning disks, Splunk recommends RAID 1+0 with 1k IOPs
• SSDs provide extremely high IOPs (45,000 +)
• RAID 5 SSD arrays give great Splunk performance in most
scenarios.
Additional details: Splunk Docs, Capacity Planning Manual
13. 13
Forwarder Load Balancing
Have UF balance across multiple indexers
DNS round robin
Multiple hosts in outputs
LB not needed!
Geography-based routing
14. 14
Indexer Clustering
High-Availability, Out of the Box
Splunk indexer clustering
Active-Active= better performance
Specific terms:
– Master Node
– Peer Node
– Search Factor
– Replication Factor
Additional details: Splunk Docs, Distributed Deployment Manual
16. 16
Scaling the Search Heads
Splunk Search is critical, too!
Splunk Search high availability needs
Scale to handle # of concurrent queries
17. 17
SHP vs SHC
SHC
• SHP
• Available since v4.2
• Sharing configurations through NFS
• Single point of failure
• Performance issues
• No NFS
• Replication using local storage
• Commodity hardware
NFS
19. 19
Search Head Clustering
Use “Captain” for Master to avoid confusion with Index-Clustering
Minimum 3 nodes required. Odd is always preferred.
Cluster takes certain key decisions based on *majority* (consensus)
In multi-site setup have more nodes in main datacenter
21. 21
Deployment Server
Central management of Splunk Forwarders
Deployment Server manages Apps, Configs
Select one or more classes for each host
Class defines apps & configs
Works by phone-home
Notes:
DS does not push forwarder binaries
Use Cluster Master to manage indexers in cluster, not DS