Uploaded bySplunk
Taking Splunk to the Next Level – Architecture
The document presents a comprehensive guide for advancing Splunk deployments, focusing on technical architecture, scaling strategies, and the importance of dependencies and data management. It covers various roles within Splunk, such as forwarders, indexers, and search heads, while emphasizing the significance of high availability and performance in mission-critical environments. Best practices for deployment, storage, and system configurations are also outlined to enhance the user experience and operational efficiency.
More Related Content
![[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...](https://cdn.slidesharecdn.com/ss_thumbnails/draganberic-lakehousearchitecturewithdeltalakeanddatabricks-221130080712-6e817e95-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Taking Splunk to the Next Level – Architecture
Taking Splunk to the Next Level – Architecture
- 1. Copyright © 2015Splunk Inc. Take Splunk to the Next Level: Technical - Architecture
- 2. 2 Am I inthe right place? You should have familiarity with… • Splunk Roles • Administering a single Splunk instance • Great if you’ve worked with a distributed environment • General Server Hardware • Disk I/O • CPU Cores
- 3. 3 Who’s This Dude? JeffChampagne Client Architect Started with Splunk in Fall 2014 Former Splunk customer in the Financial Services Industry Lived previous lives as a Systems Administrator, Engineer, and Architect
- 4. 4 Splunk at theNext Level Time to move beyond initial Splunk environment • More use cases – how to tackle? • More data – how do we scale? • Splunk is mission critical == HA • Global deployments • Improving Splunk user experience Screenshot here
- 5. 5 Growing your SplunkDeployment Many customers start with a single use case… • Ex: Monitor the web servers • Help ensure up-time & response times • Track usage, errors • Provides business value
- 6. 6 Growing your SplunkDeployment Value statement for each overall service Your services exist in a larger context than just one app, or one tier. What is the value of the service as a whole? What are CIO commitments for the service? • The organization’s web site is one of the most critical parts of the business. • Performance of the overall environment must be maintained at all times. • Failures in any portion of the web site must be quickly identified, send notification to the appropriate parties. • Dependencies on external processes must be monitored as well.
- 7. 7 Growing your SplunkDeployment The larger context • Failure in one system cascades • Map dependencies, estimate costs • Use Splunk to track all dependencies. • What happens when it is down? Dependencies often include: • Networking dependencies • Shared storage • Databases, middleware, custom apps • Virtualization layer Screenshot here
- 8. 8 Scales to Hundredsof TBs/Day Enterprise-Class Scale, Resilience and Interoperability Send data from thousands of servers using any combination of Splunk Forwarders Auto load-balanced forwarding to Splunk Indexers Offload search load to Splunk Search Heads
- 9. 9 Product Roles Searching andReporting (Search Head) Indexing and Search Services (Indexer) Data Collection and Forwarding (Forwarder) Indexer Cluster Master, SHC Deployer Distributed Management / Deployment Server License Master, Distributed Mgmt Console Databases Networks Servers Virtual Machines Smart phones and Devices Custom Applications Security WebServer Sensors
- 10. Copyright © 2015Splunk Inc. Forwarders
- 11.
- 12. 12 Splunk Universal Forwarder Whyuse the UF over other methods? Collect syslog / event log / custom application logs Collect configuration files, registry settings Collect data NOT in log files: scripted inputs on current state Collect wire data – Splunk Stream Faster, Lower overhead than “agentless” polling Centrally administered … and
- 13. 13 Forwarder Load Balancing HaveUF balance across multiple indexers Load Balance – Multiple hosts in outputs – DNS round robin – LB not needed! Geography-based routing Optional SSL encryption Compressed 10 to 1
- 14. 14 Deployment Server Central managementof Splunk Forwarders Deployment Server manages Apps, Configs Select one or more classes for each host Class defines apps & configs Works by phone-home Notes: DS does not push forwarder binaries Use Cluster Master to manage indexers in cluster, not DS
- 15. 15 Forwarding Tier DesignBest Practices 15 • Use a Syslog Server for Syslog data • Deployment server (on a VM) for central management • Let AutoLB distribute data across available indexers • May need to increase UF throughput setting for high velocity sources – Enable forceTimebasedAutoLB (for more even distribution) – maxKBps (to adjust throttling) Questions?
- 16. Copyright © 2015Splunk Inc. Indexers
- 17. 17 Indexers Dedicated indexers servethree primary roles: Data Storage Processing and parsing at index-time Indexing Data Management Hot / warm / cold data rotation Aging and removal Data Retrieval Perform search upon request, return data to search heads
- 18. 18 Scaling - Indexers Sizingfor index performance Indexers are usually storage-bound Indexers: 150 to 250 GB per day, each. (With reference HW.) Ref HW: 12 cores (2 GHz+), 12 GB RAM, 800+ IOPs Optimal HW (normal disk): 16 CPU cores, 48 GB RAM Optimal HW (SSD): 24 CPU cores, 132 GB RAM Questions?
- 19. 19 Tiered Storage • Splunksupports tiered storage • Hot / Warm buckets – put on fastest disk • Size Hot/Warm for normal saved search durations. (7d, 30d) • Use slower / cheaper storage (NAS?) for long term access • Optional: Use Frozen to roll data to glacier, Hadoop, etc.
- 20. 20 SSD Advantage http://blogs.splunk.com/2012/05/10/quantifying-the-benefits-of- splunk-with-ssds/ • Lowcost random seeks • Writes are not that much faster – no great improvement with Indexing • Significant improvements with Sparse/needle-haystack searches • Dense searches become CPU bound • Searches run faster allowing for more completed searches/min • Use Enterprise-grade SSDs, not commercial-grade.
- 21. 21 Scaling - Storage Manualstorage calculation Raw data rate net compression of ~ 50% on disk. Simple: rate * compression * retention / #indexers Hot / warm requirements – 200 GB / day * 50% * 30 days = 3TB per indexer Cold storage requirements – 200 GB / day * 50% * 335 days = 33.5TB per indexer Clustering – Changes storage story completely
- 22. 22 Scaling - Storage Oneexample of good local storage A well configured indexer using local storage might look like: • SSDs in RAID 5, sized for 14 days of storage • SATA drives in RAID 5, sized for 6 months of storage SSDs: RAID 5 provides decent performance Spinning disks: • Hot/Warm, RAID 1+0, 800 IOPS or faster • Cold – RAID 5 with proper block / stripe sizing
- 23. 23 Scaling - Storage SizingCalculator: http://splunk-sizing.appspot.com/
- 24. Copyright © 2015Splunk Inc. Indexer Clustering
- 25. 25 Delivers Mission-Critical Availability •Data replication – maintain searchability even if servers go down • Multi-site capable – maintain searchability even if a site goes down • Search Affinity – optimized searches by fetching from the closest/fastest location REPLICATION Portland Datacenter New York Datacenter Clustering
- 26. 26 Indexer Clustering High-Availability, Outof the Box Splunk indexer clustering Active-Active= better performance Specific terms: – Master Node / Master Cluster Node – Peer Node – Search Factor – Replication Factor Additional details: Splunk Docs, Distributed Deployment Manual
- 27. 27 Cross-site Clustering Search Affinityby location “Search locally”, “Store Globally” DR scenarios
- 28. 28 How Clustering AffectsSizing • Increased storage: – 15% of raw usage for every replica copy – 35% MORE to make that searchable • Increased processing – Incoming data to indexer is streamed to indexing peers to satisfy required number of copies • More hosts – Need “replication factor” + 2 (search head, cluster master) 2
- 29. 29 Scaling - Storage SizingCalculator: http://splunk-sizing.appspot.com/
- 30. 30 Downsides of IndexerClustering • Increased Storage • Cluster master is required – use a VM. • Increased bandwidth Questions? 3
- 31. Copyright © 2015Splunk Inc. Search Heads
- 32. 32 Scaling the SearchHeads Splunk Search is critical, too! Scaling your search heads Scale to handle # of concurrent queries Dedicated Search heads for certain apps, scheduled alerts Remember – Search heads virtualize well!
- 33. Copyright © 2015Splunk Inc. Search Head Clustering
- 34. 34 SHP vs SHC SearchHead Clustering Seach Head Pooling • Available since v4.2 • Sharing configurations through NFS • Single point of failure • Performance issues • No shared storage requirement • Replication using local storage • Commodity hardware • OSes: Linux or Solaris NFS
- 35. 35 Search Head Clustering 1.Group search heads into a cluster 2. A captain gets elected dynamically 3. User created reports/dashboards automatically replicated to other search heads
- 36.
- 37. 37 Search Tier DesignBest Practices 37 • Minimum 3 nodes required • ES will still require a Separate Search Head or dedicated SHC • Use LDAP/AD/SSO for user Authentication • Load Balancer configured for sticky sessions • Must use deployer to push apps to search heads • Confirm your applications’ support for SHC! Questions?
- 38. Copyright © 2015Splunk Inc. The Final Stretch
- 39. 39 Load Balancer Search HeadCluster, Deployer Clustered Peer Node + Cluster master Deployment server Universal Forwarders on Servers Syslog, NetFlow data HFs for scheduled polling via API 39
- 40. 40 Hybrid Approach forrollout 40 • Add the existing Splunk instance as a search peer until the data retention period has expired • Disable scheduled searches on the old instance • Migrate any Summary Index data to new Indexers
- 41. 41 Additional considerations • Don’tshare hosts with other services – Not co-located with Exchange, Active Directory, Hypervisors • Don’t let anti-virus run on the Splunk partition on SH, Indexers • Some data collection apps require a heavy forwarder, see docs – VMware App – NetApp App – Checkpoint LEA 4
- 42. 42 Distributed Management Console ManageSplunk 6.2 environments Replaces Deployment Monitor App Incorporates SOS app prior to 6.2
- 43. 43 Cloud & Hybrid Scalewithout waiting for hardware
- 44. 44 Suggested Reading • DistributedDeployment Manual – http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Distributedoverv iew • Highlights – Reference hardware specs – How searches affect performance Dense / Rare / Sparse – App considerations – Summary table 4
- 45. 45 Top 5 thingsto Remember 45 • Indexers: Storage requirements, IOPS, RAID config • Indexer clustering: HA, DR, and site affinity! • SHC: Minimum buy-in for a SHC is 3 • When in doubt – add another Indexer • Excellent VM candidates: – Master Cluster Node (Indexer clustering) – Deployer (Search head clustering) – Deployment Server (Central Forwarder management) – License Master – Distributed Management Console
- 46.
Editor's Notes
- #9 By allowing Splunk Enterprise to be split into multiple roles, any portion of Splunk can be scaled as needed. Customers are using Splunk to index hundreds of TB/s a day and search over petabytes of data. Splunk can take a single search and query as many indexers as are needed to complete the job, allowing you to use inexpensive commodity hardware in massively parallel clusters. Besides achieve massive scale, splitting the roles enabled user to meet location and data segmentation requirements.
- #10 These are multiple logical roles, a Splunk instance can be one or more of the roles. The search head is what most users interact with. It is the webserver and app interpreting engine that provides the primary, web-based user interface. Since most of the data interpretation happens as-needed at search time, the role of the search head is to translate user and app requests into actionable searches for it’s indexer(s) and display the results. The Splunk web UI is highly customizable, either through our own view and app system, or by embedding Splunk searches in your own web apps or our API. Additional search heads can be deployed to scale with user or search load. The core of the Splunk infrastructure is indexing. An indexer does two things – it accepts and processes new data, adding it to the index and compressing it on disk. The indexer also services search requests, looking through the data it has via it’s indices and returning the appropriate results to the searcher over a secure compressed communication channel. Indexers scale out almost limitlessly and with almost no degradation in overall performance, allowing Splunk to scale from single-instance small deployments to truly massive Big Data challenges. The Splunk forwarder is an optional component that can be installed to forward data from servers, desktops, mainframes, and even ARM based devices. There are two types of forwarders; the full Splunk distribution or a dedicated “Universal Forwarder”. The full Splunk distribution can be configured to filter data before transmitting, execute scripts locally, or run SplunkWeb. This gives you several options depending on the footprint size your endpoints can tolerate. The universal forwarder is an ultra-lightweight agent designed to collect data in the smallest possible footprint. Both flavors of forwarder come with automatic load balancing, SSL encryption and data compression, and the ability to route data to multiple Splunk instances or third party systems. The Cluster Master coordinates which indexers have copies of which buckets to ensure we have met the proper number of replication and searchable copies of each bucket. All clustered Indexers check in with the Master to alert them of their status, and the status of each of their replicated indexes and buckets. It also manages the apps and configurations on clustered indexers. We will talk more about buckets later. The Deployment Server can be used to manage your Splunk forwarders, for centrally managed data collection. More on this to come. Listed for completeness are the license master and DMC roles, these typically coexist with other roles such as the Deployment server.
- #22 This slide shows the way we used to calculate data storage requirements.
- #23 For everything we’ve shown sizing-wize so far, the storage requirements were with respect to available disk AFTER factoring RAID, OS overhead for inodes, etc. Let’s talk about RAID for a sec.
- #24 This app makes it far easier to size the environment’s storage requirements. And it includes clustering configurations, which we’ll talk about in a sec.
- #26 Splunk’s clustering technology allows you to choose how many raw copies and searchable copies of your data you would like to keep. It also allows you to chose which indexers you want to store the copies on. This capability allows servers or even datacenters to go down without losing the ability to access the data. In addition, the search affinity capability allows users to fetch data from the closest or fastest location where there is a copy of the data which can not only save the time it takes to do a search but bandwidth by eliminating the need to use the WAN when there is a local copy.
- #29 Default 3/2 cluster uses 3*.15 + 2*.35 = 115% of license usage for that redudancy Processing : a little more CPU and more network this is much better in current versions, the indexed data (tsidx, etc) is streamed to the replica peer, rather than forcing the peer to re-index.
- #30 With Search factor / rep factor variables in the mix - what had been simple without clustering now becomes more challenging. Demo sizing calculator if time allows. Hot/Warm vs. Cold, in different RAID configurations. Sample indexes.conf is generated too.
- #31 As discussed – default parameters require *more than* original log size
- #36 Uniform user experience among pooled search heads No single point of failure Search job failure aware Does not require external storage such as NFS
- #37 Note the Deployer on this image. Deployer virtualizes very well. The deployer pushes apps & configurations to the search head cluster members.
- #40 Putting it all together
- #41 What’s the best way to roll out some of these features? It depends on customer environment. But one common method is shown here.