Uploaded bySplunk
Taking Splunk to the Next Level - Architecture
The document outlines advanced concepts and strategies for utilizing Splunk in enterprise environments, including hardware scaling, indexer and search head clustering, and centralized configuration management. It emphasizes the importance of increasing ROI through effective use case implementations and details technical specifications for optimal performance. The document also promotes the 6th annual Splunk Worldwide Users' Conference, highlighting session topics and opportunities for professional development.
More Related Content
Similar to Taking Splunk to the Next Level - Architecture
Taking Splunk to the Next Level - Architecture
- 1. Copyright © 2015Splunk Inc. Splunk @ Level++
- 2. 2 What’s a Burch? Bostonarea Middleware Engineer for 8 years (+splunk) Splunk Admin for 1.5 years (splunk 4.3+) Knowledge Manager & Admin Certified Tenure @ Splunk Automatic App
- 3. 3 Splunk at theNext Level Time to move beyond initial Splunk environment • More use cases • More data • Splunk is mission critical == HA • Global deployments • Splunk user experience Where are you with Splunk Architecture? Screenshot here
- 4. 4 Agenda 1. Use Cases Business Cases 2. Hardware Scaling 3. Indexer Clustering 4. Search Head Clustering 5. Centralized Configuration Mgmt 6. License Manager 7. Distributed Management Console 8. Splunk Cloud & Hybrid Deployments 9. Q&A
- 5. 5 Use Cases Business Cases Many customers start with a single use case… • Monitor the web servers • Help ensure up-time & response times • Track usage, errors • Provides business value
- 6. 6 Use Cases Business Cases Dependencies: failure in one system cascades • Networking dependencies • Shared storage • Databases, middleware, custom apps • Virtualization layer Splunk increases ROI of investment • Estimate costs • Estimate failure costs (opportunity costs) • Splunk to track all dependencies Screenshot here
- 7.
- 8. 8 Hardware Scaling: Indexers Sizingfor index performance Indexers usually storage-bound Indexers: 150 - 250 GB per day each Ref HW: 12 cores (2 GHz+), 12 GB RAM, 800+ IOPs Optimal HW (normal disk): 16 CPU cores, 48 GB RAM Optimal HW (SSD): 24 CPU cores, 132 GB RAM
- 9. 9 Hardware Scaling: SSDAdvantage • Writes are not that much faster • Dense searches become CPU bound • Low cost random seeks (sparse) • http://blogs.splunk.com/2012/05/10/quantifying-the-benefits-of- splunk-with-ssds/ • SSD = more searches/min
- 10. 10 Hardware Scaling: Storage SizingCalculator: http://splunk-sizing.appspot.com/
- 11. 11 Hardware Scaling: Storage Rawdata compression ~ 50% Simple: rate * compression * retention – 200 GB / day * 50% * 100 days = 10TB Consider cold storage on NAS Clustering impact
- 12. 12 Hardware Scaling: Storage •For spinning disks, Splunk recommends RAID 1+0 with 1k IOPs • SSDs provide extremely high IOPs (45,000 +) • RAID 5 SSD arrays = Additional details: Splunk Docs -> Capacity Planning Manual
- 13. 13 Indexer Clustering High-Availability, Outof the Box Splunk indexer clustering Active-Active= better performance Specific terms: – Master Node – Search Peer Node – Search Factor – Replication Factor Additional details: Splunk Docs, Distributed Deployment Manual
- 14. 14 Indexer Clustering: Cross-site SearchAffinity by location “Search locally”, “Store Globally” DR scenarios
- 15. 15 Indexer Clustering: Forwarders HaveUF balance across multiple indexers DNS round robin Multiple hosts in outputs LB not needed! Geography-based routing
- 16. 16 Search Heads Why wouldyou scale search heads? high availability needs # of concurrent queries
- 17. 17 SHP vs SHC SHC •SHP • Available since v4.2 • Sharing configurations through NFS • Single point of failure • Performance issues • No NFS • Replication using local storage • Commodity hardware NFS
- 18. 18 Search Head Clustering Captainvs Master 3+ nodes required Odd preferred *majority* (consensus) In multi-site setup have more nodes in main datacenter
- 19. 19 Centralized Configuration Management DeploymentServer manages apps, configs phone-home not binaries
- 20. 20 Configuration Distribution Recap DeploymentServer Deployer Master Node Forwarders Search Head Cluster Index Cluster In a mature environment
- 21. 21 License Manager Master &Slave Docs -> Admin Manual -> Configure Splunk License
- 22. 22 Distributed Management Console ManageSplunk 6.2 environments Replaces Deployment Monitor App Incorporates SOS app prior to 6.2
- 23. 23 Cloud & Hybrid Scalewithout waiting for hardware
- 24. The 6th AnnualSplunk Worldwide Users’ Conference September 21-24, 2015 The MGM Grand Hotel, Las Vegas • 50+ Customer Speakers • 50+ Splunk Speakers • 35+ Apps in Splunk Apps Showcase • 65 Technology Partners • Register at conf.splunk.com • 4,000+ IT & Business Professionals • 2 Keynote Sessions • 3 days of technical content – 150+ sessions • 3 days of Splunk University – Get Splunk Certified – Get CPE credits for CISSP, CAP, SSCP, etc. – Save thousands on Splunk education! 24
- 25. 25 2 www.splunk.com/apptitude July 20th,2015 Submission deadline
- 26.
Editor's Notes
- #25 And finally, I would like to encourage all of you to attend our user conference in September. The energy level and passion that our customers bring to this event is simply electrifying. Combined with inspirational keynotes and 150+ breakout session across all areas of operational intelligence, It is simply the best forum to bring our Splunk community together, to learn about new and advanced Splunk offerings, and most of all to learn from one another.
- #26 ----- Meeting Notes (4/22/15 10:47) ----- Splunk Apptitude is live and open. You've got 90 days. To win more than $150,000 in cash and prizes. Last day to submit is July 20th, 2015. We'll announce the winners at Black Hat in August. Good luck!