SlideShare a Scribd company logo
1 of 45
Distributed Management Console
Bryan Schaefer
2
Personal Introduction
2
• Kamilo “Kam” Amir
• Works on the Splunk MidAtlantic Majors Team
• 4 years with Splunk, prior worked at BMC Software (BladeLogic) and Verizon
Business (Digex)
• Mike Wilson
• Works on Splunk Public Sector Team
• Yes, he works at Splunk for the last million years…
3
Agenda
• 6.4 DMC Recap
– Continuous Investment
– DMC Deployment Architectures
• So What’s Up With My Search Head Cluster?
• And that other Clustering thing, the Indexer Cluster?
• Indexes and Volumes Everywhere
• Forwarders (Really Everywhere)
• Oh, and One Other Thing…
3
4
Google
5
Blame
6
Try
7
Workaround
8
Answers
Distributed Management
Console 6.4 Recap
10
Continuous Investment in Management/Monitoring
• Started with Introspection in 6.1
• Items in 6.3 that will make Admins happy
– Data Integrity Control
– Forwarder Director
– Runaway Search Preventer
• The future
– Radically simplified setup/expansion
– Granular controls in distributed deployment
– Standard flows for common tasks in a distributed deployment
– Better App model for installation/management
1
11
History of Splunk Monitoring Tools
1
• index=_internal sourcetype=splunkd
– Go look at the logs!
• Splunkbase Tools
• Status/System Activity Dashboards
• Deployment Monitor
– License Usage Reporting!
– Alerting, Summarization
• S.o.S
– Developed by Splunk Support for Splunk Support and Customers
– Platform Resource Utilization collection with Technology Add-Ons
– Topology View
12
Distributed Management Console Architecture
1
Distributed Management Console Architecture
12
Search Heads/Search Head Cluster
Indexers/Index Cluster
Universal Forwarder
Distributed Search
Management
Data
DMC
Host
…
…
…
13
Setup Tasks
1
• Prerequisites
– Where does the DMC live?
– Topology Definition
– Forward all logs from all components back to the indexing tier
– All components must be Search Peers of the DMC Host
• Standalone vs Distributed Mode
– Server Roles
– Custom Groups
– Cluster Labels!
14 1
Search Head
Clustering Views
16
Search Head Clustering Views
1
• Motivation
– Plenty of data in logs/CLI
– Lots of customers deploying SHC
– What is going on in my Search Head
Cluster?
17 1
18 1
19 1
20 2
21 2
Index Clustering
Views
23
Indexer Clustering Views
2
• Motivation
– One layer deeper than originally
exposed
– Dealing with ever expanding
indexer counts
• Demo
24 2
25 2
Indexes & Volumes
View
27
Indexes and Volumes Views
2
• Motivation
– Customers love Fire Brigade
– Figuring out if you are meeting your
retention policies is tricky
• Demo
28 2
29 2
30 3
31 3
32 3
33 3
34 3
Forwarder
Monitoring Views
36
Forwarder Monitoring Views
3
• Motivation
– No Forwarder info in 6.2!
– Deployment Monitor no longer
improved/supported
– Some customers don’t use
Deployment Server
• Forwarder Monitoring Setup
– Runs a search against indexers
– Configurable period
– View reads from Asset Table
• Demo
37 3
38 3
39 3
Topology Views
41
Topology View
4
• Motivation
– Visual representation of
deployment
– Relationships between instances
– Deployment at-a-glance
– Troubleshooting
• Demo
42
KPI Overlays
4
43
Performance Overlays & Instance Details
4
44
SEPT 26-29, 2016
WALT DISNEY WORLD, ORLANDO
SWAN AND DOLPHIN RESORTS
• 5000+ IT & Business Professionals
• 3 days of technical content
• 165+ sessions
• 80+ Customer Speakers
• 35+ Apps in Splunk Apps Showcase
• 75+ Technology Partners
• 1:1 networking: Ask The Experts and Security
Experts, Birds of a Feather and Chalk Talks
• NEW hands-on labs!
• Expanded show floor, Dashboards Control
Room & Clinic, and MORE!
The 7th Annual Splunk Worldwide Users’ Conference
PLUS Splunk University
• Three days: Sept 24-26, 2016
• Get Splunk Certified for FREE!
• Get CPE credits for CISSP, CAP, SSCP
• Save thousands on Splunk education!
4
Thank You

More Related Content

What's hot

.conf Go 2022 - Observability Session
.conf Go 2022 - Observability Session.conf Go 2022 - Observability Session
.conf Go 2022 - Observability SessionSplunk
 
Splunk for IT Operations
Splunk for IT OperationsSplunk for IT Operations
Splunk for IT OperationsSplunk
 
Building Service Intelligence with Splunk IT Service Intelligence (ITSI)
Building Service Intelligence with Splunk IT Service Intelligence (ITSI) Building Service Intelligence with Splunk IT Service Intelligence (ITSI)
Building Service Intelligence with Splunk IT Service Intelligence (ITSI) Splunk
 
Splunk for Enterprise Security and User Behavior Analytics
 Splunk for Enterprise Security and User Behavior Analytics Splunk for Enterprise Security and User Behavior Analytics
Splunk for Enterprise Security and User Behavior AnalyticsSplunk
 
Integrating NiFi and Flink
Integrating NiFi and FlinkIntegrating NiFi and Flink
Integrating NiFi and FlinkBryan Bende
 
Getting Data into Splunk
Getting Data into SplunkGetting Data into Splunk
Getting Data into SplunkSplunk
 
Splunk IT Service Intelligence
Splunk IT Service IntelligenceSplunk IT Service Intelligence
Splunk IT Service IntelligenceGeorg Knon
 
Analytics Driven SIEM Workshop
Analytics Driven SIEM WorkshopAnalytics Driven SIEM Workshop
Analytics Driven SIEM WorkshopSplunk
 
SplunkLive! Splunk for Security
SplunkLive! Splunk for SecuritySplunkLive! Splunk for Security
SplunkLive! Splunk for SecuritySplunk
 
Splunk 6.4 Administration.pdf
Splunk 6.4 Administration.pdfSplunk 6.4 Administration.pdf
Splunk 6.4 Administration.pdfnitinscribd
 
Building Business Service Intelligence with ITSI
Building Business Service Intelligence with ITSIBuilding Business Service Intelligence with ITSI
Building Business Service Intelligence with ITSISplunk
 
Splunk Enterprise Security
Splunk Enterprise SecuritySplunk Enterprise Security
Splunk Enterprise SecuritySplunk
 
Threat Hunting Workshop
Threat Hunting WorkshopThreat Hunting Workshop
Threat Hunting WorkshopSplunk
 
Splunk workshop-Machine Data 101
Splunk workshop-Machine Data 101Splunk workshop-Machine Data 101
Splunk workshop-Machine Data 101Splunk
 
Threat Hunting Web Shells Using Splunk
Threat Hunting Web Shells Using SplunkThreat Hunting Web Shells Using Splunk
Threat Hunting Web Shells Using Splunkjamesmbower
 
Intro to open source observability with grafana, prometheus, loki, and tempo(...
Intro to open source observability with grafana, prometheus, loki, and tempo(...Intro to open source observability with grafana, prometheus, loki, and tempo(...
Intro to open source observability with grafana, prometheus, loki, and tempo(...LibbySchulze
 

What's hot (20)

.conf Go 2022 - Observability Session
.conf Go 2022 - Observability Session.conf Go 2022 - Observability Session
.conf Go 2022 - Observability Session
 
Splunk overview
Splunk overviewSplunk overview
Splunk overview
 
Splunk for IT Operations
Splunk for IT OperationsSplunk for IT Operations
Splunk for IT Operations
 
Building Service Intelligence with Splunk IT Service Intelligence (ITSI)
Building Service Intelligence with Splunk IT Service Intelligence (ITSI) Building Service Intelligence with Splunk IT Service Intelligence (ITSI)
Building Service Intelligence with Splunk IT Service Intelligence (ITSI)
 
Splunk for Enterprise Security and User Behavior Analytics
 Splunk for Enterprise Security and User Behavior Analytics Splunk for Enterprise Security and User Behavior Analytics
Splunk for Enterprise Security and User Behavior Analytics
 
Integrating NiFi and Flink
Integrating NiFi and FlinkIntegrating NiFi and Flink
Integrating NiFi and Flink
 
Getting Data into Splunk
Getting Data into SplunkGetting Data into Splunk
Getting Data into Splunk
 
Splunk IT Service Intelligence
Splunk IT Service IntelligenceSplunk IT Service Intelligence
Splunk IT Service Intelligence
 
Analytics Driven SIEM Workshop
Analytics Driven SIEM WorkshopAnalytics Driven SIEM Workshop
Analytics Driven SIEM Workshop
 
Splunk Architecture
Splunk ArchitectureSplunk Architecture
Splunk Architecture
 
SplunkLive! Splunk for Security
SplunkLive! Splunk for SecuritySplunkLive! Splunk for Security
SplunkLive! Splunk for Security
 
NetScaler ADC - Customer Overview
NetScaler ADC - Customer OverviewNetScaler ADC - Customer Overview
NetScaler ADC - Customer Overview
 
Splunk 6.4 Administration.pdf
Splunk 6.4 Administration.pdfSplunk 6.4 Administration.pdf
Splunk 6.4 Administration.pdf
 
Building Business Service Intelligence with ITSI
Building Business Service Intelligence with ITSIBuilding Business Service Intelligence with ITSI
Building Business Service Intelligence with ITSI
 
Splunk Enterprise Security
Splunk Enterprise SecuritySplunk Enterprise Security
Splunk Enterprise Security
 
Splunk
SplunkSplunk
Splunk
 
Threat Hunting Workshop
Threat Hunting WorkshopThreat Hunting Workshop
Threat Hunting Workshop
 
Splunk workshop-Machine Data 101
Splunk workshop-Machine Data 101Splunk workshop-Machine Data 101
Splunk workshop-Machine Data 101
 
Threat Hunting Web Shells Using Splunk
Threat Hunting Web Shells Using SplunkThreat Hunting Web Shells Using Splunk
Threat Hunting Web Shells Using Splunk
 
Intro to open source observability with grafana, prometheus, loki, and tempo(...
Intro to open source observability with grafana, prometheus, loki, and tempo(...Intro to open source observability with grafana, prometheus, loki, and tempo(...
Intro to open source observability with grafana, prometheus, loki, and tempo(...
 

Similar to Manage and Monitor Distributed Splunk Deployments

Distributed Management Console
Distributed Management ConsoleDistributed Management Console
Distributed Management ConsoleSplunk
 
Splunk Distributed Management Console
Splunk Distributed Management Console                                         Splunk Distributed Management Console
Splunk Distributed Management Console Splunk
 
Distributed Management Console Breakout Session
Distributed Management Console Breakout Session Distributed Management Console Breakout Session
Distributed Management Console Breakout Session Splunk
 
SenchaCon 2016: Creating a Flexible and Usable Industry Specific Solution - D...
SenchaCon 2016: Creating a Flexible and Usable Industry Specific Solution - D...SenchaCon 2016: Creating a Flexible and Usable Industry Specific Solution - D...
SenchaCon 2016: Creating a Flexible and Usable Industry Specific Solution - D...Sencha
 
Monitoring Splunk: S.o.S, DMC, and Beyond Breakout Session
Monitoring Splunk: S.o.S, DMC, and Beyond Breakout SessionMonitoring Splunk: S.o.S, DMC, and Beyond Breakout Session
Monitoring Splunk: S.o.S, DMC, and Beyond Breakout SessionSplunk
 
Introduction to SAFeMSIS CoreFall 2019Scenario –.docx
Introduction to SAFeMSIS CoreFall 2019Scenario –.docxIntroduction to SAFeMSIS CoreFall 2019Scenario –.docx
Introduction to SAFeMSIS CoreFall 2019Scenario –.docxvrickens
 
Monitoring Splunk: S.o.S, DMC, and Beyond
Monitoring Splunk: S.o.S, DMC, and BeyondMonitoring Splunk: S.o.S, DMC, and Beyond
Monitoring Splunk: S.o.S, DMC, and BeyondSplunk
 
Best Practices for a CoE
Best Practices for a CoEBest Practices for a CoE
Best Practices for a CoESplunk
 
SplunkSummit 2015 - A Quick Guide to Search Optimization
SplunkSummit 2015 - A Quick Guide to Search OptimizationSplunkSummit 2015 - A Quick Guide to Search Optimization
SplunkSummit 2015 - A Quick Guide to Search OptimizationSplunk
 
Understanding-Agile &Scrum.pdf
Understanding-Agile &Scrum.pdfUnderstanding-Agile &Scrum.pdf
Understanding-Agile &Scrum.pdfSwapnikaReddy6
 
Taking Splunk to the Next Level - Architecture
Taking Splunk to the Next Level - ArchitectureTaking Splunk to the Next Level - Architecture
Taking Splunk to the Next Level - ArchitectureSplunk
 
Taking Splunk to the Next Level - Architecture Breakout Session
Taking Splunk to the Next Level - Architecture Breakout SessionTaking Splunk to the Next Level - Architecture Breakout Session
Taking Splunk to the Next Level - Architecture Breakout SessionSplunk
 
Webinar: Ten Ways to Enhance Your Salesforce.com Application in 2013
Webinar: Ten Ways to Enhance Your Salesforce.com Application in 2013Webinar: Ten Ways to Enhance Your Salesforce.com Application in 2013
Webinar: Ten Ways to Enhance Your Salesforce.com Application in 2013Emtec Inc.
 
Future Proof Your DAM
Future Proof Your DAMFuture Proof Your DAM
Future Proof Your DAMjflorance
 
Customer Webinar: Scaling and Optimizing a Large e-Commerce Drupal Solution
Customer Webinar: Scaling and Optimizing a Large e-Commerce Drupal SolutionCustomer Webinar: Scaling and Optimizing a Large e-Commerce Drupal Solution
Customer Webinar: Scaling and Optimizing a Large e-Commerce Drupal SolutionSearchStax
 
chapter02-120827115348-phpapp01.pdf
chapter02-120827115348-phpapp01.pdfchapter02-120827115348-phpapp01.pdf
chapter02-120827115348-phpapp01.pdfAxmedMaxamuud6
 
5 Key Metrics to Release Better Software Faster
5 Key Metrics to Release Better Software Faster5 Key Metrics to Release Better Software Faster
5 Key Metrics to Release Better Software FasterDynatrace
 
Chapter 2 Analyzing the Business Case .pptx
Chapter 2 Analyzing the Business Case .pptxChapter 2 Analyzing the Business Case .pptx
Chapter 2 Analyzing the Business Case .pptxAxmedMaxamuudYoonis
 

Similar to Manage and Monitor Distributed Splunk Deployments (20)

Distributed Management Console
Distributed Management ConsoleDistributed Management Console
Distributed Management Console
 
Splunk Distributed Management Console
Splunk Distributed Management Console                                         Splunk Distributed Management Console
Splunk Distributed Management Console
 
Distributed Management Console Breakout Session
Distributed Management Console Breakout Session Distributed Management Console Breakout Session
Distributed Management Console Breakout Session
 
SenchaCon 2016: Creating a Flexible and Usable Industry Specific Solution - D...
SenchaCon 2016: Creating a Flexible and Usable Industry Specific Solution - D...SenchaCon 2016: Creating a Flexible and Usable Industry Specific Solution - D...
SenchaCon 2016: Creating a Flexible and Usable Industry Specific Solution - D...
 
Monitoring Splunk: S.o.S, DMC, and Beyond Breakout Session
Monitoring Splunk: S.o.S, DMC, and Beyond Breakout SessionMonitoring Splunk: S.o.S, DMC, and Beyond Breakout Session
Monitoring Splunk: S.o.S, DMC, and Beyond Breakout Session
 
Introduction to SAFeMSIS CoreFall 2019Scenario –.docx
Introduction to SAFeMSIS CoreFall 2019Scenario –.docxIntroduction to SAFeMSIS CoreFall 2019Scenario –.docx
Introduction to SAFeMSIS CoreFall 2019Scenario –.docx
 
Monitoring Splunk: S.o.S, DMC, and Beyond
Monitoring Splunk: S.o.S, DMC, and BeyondMonitoring Splunk: S.o.S, DMC, and Beyond
Monitoring Splunk: S.o.S, DMC, and Beyond
 
Best Practices for a CoE
Best Practices for a CoEBest Practices for a CoE
Best Practices for a CoE
 
SplunkSummit 2015 - A Quick Guide to Search Optimization
SplunkSummit 2015 - A Quick Guide to Search OptimizationSplunkSummit 2015 - A Quick Guide to Search Optimization
SplunkSummit 2015 - A Quick Guide to Search Optimization
 
ch11.ppt
ch11.pptch11.ppt
ch11.ppt
 
Understanding-Agile &Scrum.pdf
Understanding-Agile &Scrum.pdfUnderstanding-Agile &Scrum.pdf
Understanding-Agile &Scrum.pdf
 
Taking Splunk to the Next Level - Architecture
Taking Splunk to the Next Level - ArchitectureTaking Splunk to the Next Level - Architecture
Taking Splunk to the Next Level - Architecture
 
Taking Splunk to the Next Level - Architecture Breakout Session
Taking Splunk to the Next Level - Architecture Breakout SessionTaking Splunk to the Next Level - Architecture Breakout Session
Taking Splunk to the Next Level - Architecture Breakout Session
 
Webinar: Ten Ways to Enhance Your Salesforce.com Application in 2013
Webinar: Ten Ways to Enhance Your Salesforce.com Application in 2013Webinar: Ten Ways to Enhance Your Salesforce.com Application in 2013
Webinar: Ten Ways to Enhance Your Salesforce.com Application in 2013
 
Future Proof Your DAM
Future Proof Your DAMFuture Proof Your DAM
Future Proof Your DAM
 
Mis 9
Mis 9Mis 9
Mis 9
 
Customer Webinar: Scaling and Optimizing a Large e-Commerce Drupal Solution
Customer Webinar: Scaling and Optimizing a Large e-Commerce Drupal SolutionCustomer Webinar: Scaling and Optimizing a Large e-Commerce Drupal Solution
Customer Webinar: Scaling and Optimizing a Large e-Commerce Drupal Solution
 
chapter02-120827115348-phpapp01.pdf
chapter02-120827115348-phpapp01.pdfchapter02-120827115348-phpapp01.pdf
chapter02-120827115348-phpapp01.pdf
 
5 Key Metrics to Release Better Software Faster
5 Key Metrics to Release Better Software Faster5 Key Metrics to Release Better Software Faster
5 Key Metrics to Release Better Software Faster
 
Chapter 2 Analyzing the Business Case .pptx
Chapter 2 Analyzing the Business Case .pptxChapter 2 Analyzing the Business Case .pptx
Chapter 2 Analyzing the Business Case .pptx
 

More from Splunk

.conf Go 2023 - Data analysis as a routine
.conf Go 2023 - Data analysis as a routine.conf Go 2023 - Data analysis as a routine
.conf Go 2023 - Data analysis as a routineSplunk
 
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTVSplunk
 
.conf Go 2023 - Navegando la normativa SOX (Telefónica)
.conf Go 2023 - Navegando la normativa SOX (Telefónica).conf Go 2023 - Navegando la normativa SOX (Telefónica)
.conf Go 2023 - Navegando la normativa SOX (Telefónica)Splunk
 
.conf Go 2023 - Raiffeisen Bank International
.conf Go 2023 - Raiffeisen Bank International.conf Go 2023 - Raiffeisen Bank International
.conf Go 2023 - Raiffeisen Bank InternationalSplunk
 
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett .conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett Splunk
 
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär).conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)Splunk
 
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu....conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...Splunk
 
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever....conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...Splunk
 
.conf go 2023 - De NOC a CSIRT (Cellnex)
.conf go 2023 - De NOC a CSIRT (Cellnex).conf go 2023 - De NOC a CSIRT (Cellnex)
.conf go 2023 - De NOC a CSIRT (Cellnex)Splunk
 
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)Splunk
 
Splunk - BMW connects business and IT with data driven operations SRE and O11y
Splunk - BMW connects business and IT with data driven operations SRE and O11ySplunk - BMW connects business and IT with data driven operations SRE and O11y
Splunk - BMW connects business and IT with data driven operations SRE and O11ySplunk
 
Splunk x Freenet - .conf Go Köln
Splunk x Freenet - .conf Go KölnSplunk x Freenet - .conf Go Köln
Splunk x Freenet - .conf Go KölnSplunk
 
Splunk Security Session - .conf Go Köln
Splunk Security Session - .conf Go KölnSplunk Security Session - .conf Go Köln
Splunk Security Session - .conf Go KölnSplunk
 
Data foundations building success, at city scale – Imperial College London
 Data foundations building success, at city scale – Imperial College London Data foundations building success, at city scale – Imperial College London
Data foundations building success, at city scale – Imperial College LondonSplunk
 
Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...
Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...
Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...Splunk
 
SOC, Amore Mio! | Security Webinar
SOC, Amore Mio! | Security WebinarSOC, Amore Mio! | Security Webinar
SOC, Amore Mio! | Security WebinarSplunk
 
.conf Go Zurich 2022 - Keynote
.conf Go Zurich 2022 - Keynote.conf Go Zurich 2022 - Keynote
.conf Go Zurich 2022 - KeynoteSplunk
 
.conf Go Zurich 2022 - Platform Session
.conf Go Zurich 2022 - Platform Session.conf Go Zurich 2022 - Platform Session
.conf Go Zurich 2022 - Platform SessionSplunk
 
.conf Go Zurich 2022 - Security Session
.conf Go Zurich 2022 - Security Session.conf Go Zurich 2022 - Security Session
.conf Go Zurich 2022 - Security SessionSplunk
 
Inside SecOps at bet365
Inside SecOps at bet365 Inside SecOps at bet365
Inside SecOps at bet365 Splunk
 

More from Splunk (20)

.conf Go 2023 - Data analysis as a routine
.conf Go 2023 - Data analysis as a routine.conf Go 2023 - Data analysis as a routine
.conf Go 2023 - Data analysis as a routine
 
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
 
.conf Go 2023 - Navegando la normativa SOX (Telefónica)
.conf Go 2023 - Navegando la normativa SOX (Telefónica).conf Go 2023 - Navegando la normativa SOX (Telefónica)
.conf Go 2023 - Navegando la normativa SOX (Telefónica)
 
.conf Go 2023 - Raiffeisen Bank International
.conf Go 2023 - Raiffeisen Bank International.conf Go 2023 - Raiffeisen Bank International
.conf Go 2023 - Raiffeisen Bank International
 
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett .conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
 
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär).conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
 
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu....conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
 
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever....conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
 
.conf go 2023 - De NOC a CSIRT (Cellnex)
.conf go 2023 - De NOC a CSIRT (Cellnex).conf go 2023 - De NOC a CSIRT (Cellnex)
.conf go 2023 - De NOC a CSIRT (Cellnex)
 
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
 
Splunk - BMW connects business and IT with data driven operations SRE and O11y
Splunk - BMW connects business and IT with data driven operations SRE and O11ySplunk - BMW connects business and IT with data driven operations SRE and O11y
Splunk - BMW connects business and IT with data driven operations SRE and O11y
 
Splunk x Freenet - .conf Go Köln
Splunk x Freenet - .conf Go KölnSplunk x Freenet - .conf Go Köln
Splunk x Freenet - .conf Go Köln
 
Splunk Security Session - .conf Go Köln
Splunk Security Session - .conf Go KölnSplunk Security Session - .conf Go Köln
Splunk Security Session - .conf Go Köln
 
Data foundations building success, at city scale – Imperial College London
 Data foundations building success, at city scale – Imperial College London Data foundations building success, at city scale – Imperial College London
Data foundations building success, at city scale – Imperial College London
 
Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...
Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...
Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...
 
SOC, Amore Mio! | Security Webinar
SOC, Amore Mio! | Security WebinarSOC, Amore Mio! | Security Webinar
SOC, Amore Mio! | Security Webinar
 
.conf Go Zurich 2022 - Keynote
.conf Go Zurich 2022 - Keynote.conf Go Zurich 2022 - Keynote
.conf Go Zurich 2022 - Keynote
 
.conf Go Zurich 2022 - Platform Session
.conf Go Zurich 2022 - Platform Session.conf Go Zurich 2022 - Platform Session
.conf Go Zurich 2022 - Platform Session
 
.conf Go Zurich 2022 - Security Session
.conf Go Zurich 2022 - Security Session.conf Go Zurich 2022 - Security Session
.conf Go Zurich 2022 - Security Session
 
Inside SecOps at bet365
Inside SecOps at bet365 Inside SecOps at bet365
Inside SecOps at bet365
 

Recently uploaded

Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountBreaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountPuma Security, LLC
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreternaman860154
 
Hyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your Budget
Hyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your BudgetHyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your Budget
Hyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your BudgetEnjoy Anytime
 
Next-generation AAM aircraft unveiled by Supernal, S-A2
Next-generation AAM aircraft unveiled by Supernal, S-A2Next-generation AAM aircraft unveiled by Supernal, S-A2
Next-generation AAM aircraft unveiled by Supernal, S-A2Hyundai Motor Group
 
Pigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping ElbowsPigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping ElbowsPigging Solutions
 
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024BookNet Canada
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls
 
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024BookNet Canada
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationRidwan Fadjar
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge
 
Understanding the Laravel MVC Architecture
Understanding the Laravel MVC ArchitectureUnderstanding the Laravel MVC Architecture
Understanding the Laravel MVC ArchitecturePixlogix Infotech
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationMichael W. Hawkins
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Scott Keck-Warren
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls
 
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Patryk Bandurski
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonetsnaman860154
 

Recently uploaded (20)

Vulnerability_Management_GRC_by Sohang Sengupta.pptx
Vulnerability_Management_GRC_by Sohang Sengupta.pptxVulnerability_Management_GRC_by Sohang Sengupta.pptx
Vulnerability_Management_GRC_by Sohang Sengupta.pptx
 
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountBreaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path Mount
 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
 
Hyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your Budget
Hyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your BudgetHyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your Budget
Hyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your Budget
 
Next-generation AAM aircraft unveiled by Supernal, S-A2
Next-generation AAM aircraft unveiled by Supernal, S-A2Next-generation AAM aircraft unveiled by Supernal, S-A2
Next-generation AAM aircraft unveiled by Supernal, S-A2
 
Pigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping ElbowsPigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping Elbows
 
The transition to renewables in India.pdf
The transition to renewables in India.pdfThe transition to renewables in India.pdf
The transition to renewables in India.pdf
 
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
 
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 Presentation
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
Understanding the Laravel MVC Architecture
Understanding the Laravel MVC ArchitectureUnderstanding the Laravel MVC Architecture
Understanding the Laravel MVC Architecture
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food Manufacturing
 
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptxE-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 

Manage and Monitor Distributed Splunk Deployments

  • 2. 2 Personal Introduction 2 • Kamilo “Kam” Amir • Works on the Splunk MidAtlantic Majors Team • 4 years with Splunk, prior worked at BMC Software (BladeLogic) and Verizon Business (Digex) • Mike Wilson • Works on Splunk Public Sector Team • Yes, he works at Splunk for the last million years…
  • 3. 3 Agenda • 6.4 DMC Recap – Continuous Investment – DMC Deployment Architectures • So What’s Up With My Search Head Cluster? • And that other Clustering thing, the Indexer Cluster? • Indexes and Volumes Everywhere • Forwarders (Really Everywhere) • Oh, and One Other Thing… 3
  • 10. 10 Continuous Investment in Management/Monitoring • Started with Introspection in 6.1 • Items in 6.3 that will make Admins happy – Data Integrity Control – Forwarder Director – Runaway Search Preventer • The future – Radically simplified setup/expansion – Granular controls in distributed deployment – Standard flows for common tasks in a distributed deployment – Better App model for installation/management 1
  • 11. 11 History of Splunk Monitoring Tools 1 • index=_internal sourcetype=splunkd – Go look at the logs! • Splunkbase Tools • Status/System Activity Dashboards • Deployment Monitor – License Usage Reporting! – Alerting, Summarization • S.o.S – Developed by Splunk Support for Splunk Support and Customers – Platform Resource Utilization collection with Technology Add-Ons – Topology View
  • 12. 12 Distributed Management Console Architecture 1 Distributed Management Console Architecture 12 Search Heads/Search Head Cluster Indexers/Index Cluster Universal Forwarder Distributed Search Management Data DMC Host … … …
  • 13. 13 Setup Tasks 1 • Prerequisites – Where does the DMC live? – Topology Definition – Forward all logs from all components back to the indexing tier – All components must be Search Peers of the DMC Host • Standalone vs Distributed Mode – Server Roles – Custom Groups – Cluster Labels!
  • 14. 14 1
  • 16. 16 Search Head Clustering Views 1 • Motivation – Plenty of data in logs/CLI – Lots of customers deploying SHC – What is going on in my Search Head Cluster?
  • 17. 17 1
  • 18. 18 1
  • 19. 19 1
  • 20. 20 2
  • 21. 21 2
  • 23. 23 Indexer Clustering Views 2 • Motivation – One layer deeper than originally exposed – Dealing with ever expanding indexer counts • Demo
  • 24. 24 2
  • 25. 25 2
  • 27. 27 Indexes and Volumes Views 2 • Motivation – Customers love Fire Brigade – Figuring out if you are meeting your retention policies is tricky • Demo
  • 28. 28 2
  • 29. 29 2
  • 30. 30 3
  • 31. 31 3
  • 32. 32 3
  • 33. 33 3
  • 34. 34 3
  • 36. 36 Forwarder Monitoring Views 3 • Motivation – No Forwarder info in 6.2! – Deployment Monitor no longer improved/supported – Some customers don’t use Deployment Server • Forwarder Monitoring Setup – Runs a search against indexers – Configurable period – View reads from Asset Table • Demo
  • 37. 37 3
  • 38. 38 3
  • 39. 39 3
  • 41. 41 Topology View 4 • Motivation – Visual representation of deployment – Relationships between instances – Deployment at-a-glance – Troubleshooting • Demo
  • 43. 43 Performance Overlays & Instance Details 4
  • 44. 44 SEPT 26-29, 2016 WALT DISNEY WORLD, ORLANDO SWAN AND DOLPHIN RESORTS • 5000+ IT & Business Professionals • 3 days of technical content • 165+ sessions • 80+ Customer Speakers • 35+ Apps in Splunk Apps Showcase • 75+ Technology Partners • 1:1 networking: Ask The Experts and Security Experts, Birds of a Feather and Chalk Talks • NEW hands-on labs! • Expanded show floor, Dashboards Control Room & Clinic, and MORE! The 7th Annual Splunk Worldwide Users’ Conference PLUS Splunk University • Three days: Sept 24-26, 2016 • Get Splunk Certified for FREE! • Get CPE credits for CISSP, CAP, SSCP • Save thousands on Splunk education!

Editor's Notes

  1. Stela goes over the agenda
  2. Obvious questions about what can be co-hosted. What does Splunk look like when it gets big?
  3. A typical DMC setup page
  4. The Status and Configuration dashboard is an overview of your search head cluster. It is high-level information.
  5. The Configuration Replication dashboard provides insight into configurations that a user changes on any SHC member, and how these changes propagate through the cluster.
  6. The Artifact Replication dashboard contains several panels describing the cluster's "backlog" of search artifacts to replicate.
  7. Provides visibility into the captain’s role as a coordinator for scheduled searches in the cluster.
  8. In the Apps status panel, a persistent discrepancy indicates that the deployer has not finished deploying apps to its members.
  9. 2 indexes, 1 status view The status of several indexer clusters can now be consulted from a single location! No need to connect to several Cluster Master instances
  10. This view shows service tasks undertaken by the indexer clustering framework to meet data replication targets The marker shows a time when an indexer went down, requiring the surviving ones to start copying data buckets to repair the cluster We clearly see an initial peak of fix-up tasks identified, which slowly decreases over time as the cluster fixes itself In that manner, this view provides visibility into the progress of such unplanned reconfiguration events
  11. We’re looking at the _audit index on the ‘potato’ indexer cluster. We have a target time retention of 150 days for this index, which seems to be respected based on this ‘median data age’ metric.
  12. However, looking at the breakdown of data age per indexer, we can see that one indexer (svdev-centos6-006.sv.splunk.com) does not meet the target of 150 days of retention. To investigate further, we click on the table row corresponding to this index, which leads us to the Index Detail – Instance view.
  13. Looking in detail at the index that fails to meet the target retention for the _audit index, we see that: Data is not being deleted due to hitting the time-based retention policy (1st column) Data is not being deleted due to hitting the index-wide disk usage retention policy (2nd column) Data is not being deleted due to hitting directory-level (home & cold path) retention policies (3rd and 4th columns)
  14. Looking at how data age evolved over time, we can see a sharp drop-off on 09/08, indicating an incident on that day Furthermore, we see that on 09/08 we lost almost all cold buckets, indicating that something happened to the cold directory of this index on that day Let’s take a closer look at the settings for this index: Is this leveraging volumes?
  15. Indeed, both paths for this index are referencing volumes homePath (hot + warm buckets) is referencing a volume named “opt” coldPath (cold buckets) is referencing a volume named “cold” We should look at these volumes next, using the Volume Detail – Instance scoped to this indexer
  16. First let’s look at the ‘opt’ volume We see that this volume is _not_ full, so it’s not pushing data out We also see that the _audit index’s ‘home’ directory is hosted on this volume, with ~3GB worth of data Let’s move on to the ‘cold’ volume
  17. Looking at the ‘cold’ volume now This volume *is* full! It is pushing data out aggressively! All space in this volume is used by the ‘latex_imports’ index, representing only ~ 1 day’s worth of data Given that a full volume freezes older data first, the surge of recent data from ‘latex_imports’ has caused the volume to push out all data from the ‘_audit’ index Solution: separate indexes with different data density and target retention periods in different volumes
  18. Forwarder Monitoring – Deployment view can highlight missing forwarders Here we can clearly see two forwarders that have gone missing The first one – ‘atruong-mbpr15’ – hasn’t sent data to the indexers for ~ 3 hours The second one – ‘uf-dmcdemo’ – hasn’t sent data to the indexers for ~ 13 hours Let’s click on one of these missing forwarders for a drill-down to the Forwarder Monitoring – Instance view
  19. Forwarder Monitoring – Instance view We’re now looking in more detail at the history of forwarder ‘uf-dmcdemo’ connections to the indexers on the previous day We can clearly see a gap of several hours during which this forwarder did not connect to the indexers, which would have resulted in a “missing” status
  20. Missing forwarders can also be pro-actively detected using a built-in alert!
  21. We’re headed to the East Coast! 2 inspired Keynotes – General Session and Security Keynote + Super Sessions with Splunk Leadership in Cloud, IT Ops, Security and Business Analytics! 165+ Breakout sessions addressing all areas and levels of Operational Intelligence – IT, Business Analytics, Mobile, Cloud, IoT, Security…and MORE! 30+ hours of invaluable networking time with industry thought leaders, technologists, and other Splunk Ninjas and Champions waiting to share their business wins with you! Join the 50%+ of Fortune 100 companies who attended .conf2015 to get hands on with Splunk. You’ll be surrounded by thousands of other like-minded individuals who are ready to share exciting and cutting edge use cases and best practices. You can also deep dive on all things Splunk products together with your favorite Splunkers. Head back to your company with both practical and inspired new uses for Splunk, ready to unlock the unimaginable power of your data! Arrive in Orlando a Splunk user, leave Orlando a Splunk Ninja! REGISTRATION OPENS IN MARCH 2016 – STAY TUNED FOR NEWS ON OUR BEST REGISTRATION RATES – COMING SOON!