Scaling Splunk 101Quick Overview of Scaling Splunk with Commodity HardwareErik SwanOct, 09** Slides intentionally ugly, no designers were harmed during construction

Single Server InstallCommodity Architecture Simplest Splunk install is a single server that functions as both indexer and search head.A single box can easily index 100-200G per day, BUT for fast searching its best to use more than one box.Data from Splunk Forwarders, Syslog, Files, etc.Splunk (all in one)Users

Improving Search and Indexing PerformanceSplunk scales search and indexing performance horizontally by adding more indexers and in some cases scaling out a search tier. By spreading the incoming load across more indexers you index faster. Perhaps more importantly, by spreading the indexed data across more indexers your search performance improves linearly as well.Consider that every doubling of hardware will double your index and search performance and don’t be shy of adding 10’s of servers.RULE #1 – If your searches are slow, add another box!

Adding a Search HeadBy splitting out a Search Head, search performance is improved and load is taken off the indexer for faster indexing.Best to add sooner than later.Best for volumes between 5-100G p/day1 Indexer1 Search HeadData from Splunk Forwarders, Syslog, Files, etc.Spunk IndexerSplunk Search HeadUsers

Adding a second IndexerAs volume goes up beyond 100G OR you want to improve search performance its best to add a second Indexer. **Remember adding indexers improves search performance linearly as well.Best for volumes 20-200G p/day2 Indexers1 Search HeadData from Splunk Forwarders, Syslog, Files, etc.Spunk IndexerSpunk IndexerSplunk Search HeadUsers

Adding additional IndexersFor every new ~100G, or again to improve search performance add another indexer. RULE #1: If searches are slow, add an another indexer.For volumes from 200G-1T p/dayTBs/day from Splunk Forwarders and SyslogSpunk IndexerSpunk IndexerSpunk Indexer(n) Indexers Splunk Search HeadUsers

Adding additional IndexersFor every new ~100G, or again to improve search performance add another indexer. RULE #1: If searches are slow, add an another indexer.For volumes from 200G-1T p/dayTBs/day from Splunk Forwarders and SyslogAssume 100G p/day:Use Case : Log archival and some periodic troubleshooting 1 Commodity ServerUse Case #2 : Archival, troubleshooting and summary reporting 1 Index Server, 1 Search ServerUse Case #3: Archival, Trouble Shooting, and Reporting 2 Index Servers, 1 Search ServerUse Case #4: Many ( >2 ) users doing constant use 3+ Index Servers, 1 Search Server Spunk IndexerSpunk IndexerSpunk Indexer(n) Indexers Splunk Search HeadUsers

Adding additional Search HeadsTBs/day from Splunk Forwarders and SyslogAdding more Search Heads is a convenient way to improve search performance Add an additional Search Heads when:It makes sense to partition users.Too offload summary or scheduled searches. Spunk IndexerSpunk IndexerSpunk Indexer(n) Indexers Splunk Search HeadSplunk Search Head(n) Search Heads1~ 4T each p/dayLoad Bal.Users

Adding additional Search HeadsTBs/day from Splunk Forwarders and SyslogAssuming a load of 1T p/day:Use Case #1: Log archival and some periodic troubleshooting 4 Index Servers, 1 Search ServerUse Case #2: Archival, trouble shooting and some summary reporting 8+ Index Server, 1 Search ServerUse Case #3: Archival, Trouble Shooting, and Reporting 16+ Index Servers, 1 Search ServerUse Case #4: Many ( >2 ) users doing constant use 20+ Index Servers, 1 Search Server For every new ~TB p/day, add another search head.For volumes > 2T p/day(n) Indexers each <100G p/day(m) Search Heads for every ~1T p/daySpunk IndexerSpunk IndexerSpunk Indexer(n) Indexers Splunk Search HeadSplunk Search Head(n) Search Heads1~ 4T each p/dayLoad Bal.Users

Long term storage, add a SANTBs/day from Splunk Forwarders and SyslogLong term storage can not be kept on local commodity IO.If wanting to keep more than can be kept on local indexer disk, splunk can be configured to use SAN or other storage device.Best for keeping >30 day – multi year data.Spunk IndexerSpunk IndexerSpunk Indexer(n) Indexers Tier 1 SANSplunk Search HeadSplunk Search HeadLoad Bal.Users

Multi-datacenter or deploymentIf you have multiple data centers, it is often best to leave the data local and use distributed search between two deployments.If you have data that naturally partitions such that users would rarely search across the data, partitioning entire deployments can help.Obviously for DR as well.

Additional Scaling TopicsSummary Indexing – If your searches are slow consider using summary indexing: video - http://www.splunk.com/view/SP-CAAACZWdocs - http://www.splunk.com/base/Documentation/4.0.5/User/UseSummaryIndexingForIncreasedReportingEfficiencyRouting High Volume data to Separate Index – If you are searching or reporting on a source that is dwarfed by the volume of another source, you can partition data such that the high volume source is in its own index: docs - http://www.splunk.com/base/Documentation/latest/Admin/Setupmultipleindexes#Why_have_multiple_indexes.3F