SlideShare a Scribd company logo
1 of 57
Download to read offline
Replacing NSClient++ for
Windows Monitoring
Sven Nierlein
09.11.2023
Beyond NSClient++ 2
whoami
• Name: Sven Nierlein
• E-Mail: sven@consol.de
• Location: Munich, Germany
• Work: consol.de
• We might have met before here in Nuremberg:
• 2022: Thruk
• 2021: Naemon password vault
• 2019: Naemon
• 2018: Thruk
• 2013: Thruk
• 2012: Mod-Gearman
Introduction
Beyond NSClient++ 4
Recap: NSClient++
• Windows Agent (and Linux)
• Supports several protocols
• NRPE
• REST
• and more …
• Lots of built-in checks
• Extendable
• Customisable
Beyond NSClient++ 5
Status Quo
• Basically all our customers use NSClient++ in some way
• Largest setup ~15k 250k installations
• Usually happy with NSClient++
Beyond NSClient++ 6
NSClient++ - So what’s the issue?
• TLS <= 1.2
• Last release was years ago
Beyond NSClient++ 7
NSClient++
• But besides that, it just works…
• Anyway, we needed a solution for the SSL issue…
Beyond NSClient++
• Just update SSL and rebuild, how hard could it be…
8
Going the easy way…
Beyond NSClient++ 9
Looking for alternatives
• There are a couple of options:
• https://github.com/infraweavers/monitoring-agent
• https://github.com/continentale/sma-monitoring-agent/
• https://github.com/continentale/monitoring-agent
• https://github.com/it-novum/openitcockpit-agent-go
• https://github.com/Icinga/icinga-powershell-framework
• Probably missed some more…
Alternative by Nick Youngson CC BY-SA 3.0 Alpha Stock Images
Beyond NSClient++ 10
Requirements
• Drop-In Replacement
• Support nsclient.ini Format
• Must work on Windows and Linux
• Modern SSL/TLS
• Checks format/arguments compatible to NSClient
• No Change to Naemon configuration
• Bonus Points
• Prometheus
SNClient+
Beyond NSClient++ 12
Introducing SNClient+
• Rewrite NSClient++ in Go
• Compatible with nsclient.ini
• Multiplatform
• Windows
• Linux
• OSX, Freebsd…
• Dual Stack
• NSClient++
• Prometheus
Beyond NSClient++ 13
What does SNClient mean?
• Let’s ask the chatbot of your choice:
• Security/Networking Client
• Software Name Client
• Server Ninja Client
• The actual meaning is:
• Secure Naemon Client
Beyond NSClient++ 14
SNClient Dual-Stack
NRPE
HTTP(S) /
REST
NSClient++ /
Naemon / Icinga /
Nagios
Prometheus
HTTP(S)
Checks
Protocol
Naemon Stack
Beyond NSClient++ 16
NSClient / Naemon Stack
NRPE HTTP(S) / REST
NSClient++ / Naemon / Icinga
Prometheus
Checks
Ext. Scripts
Plugins
Checks
Protocol
Beyond NSClient++
SNClient
Beyond NSClient++ 18
Installation
• .MSI Installer for windows
• DEB/RPM repository for linux:
https://labs.consol.de/repo/
• Single go binary
Beyond NSClient++ 19
Checks
• Built in check commands, ex.:
• check_cpu
• check_drivesize
• check_memory
• check_process
• check_service
• …
• Get full list with
• .snclient.exe run check_index
Beyond NSClient++ 20
Checks
• Checks can be executed via check_nsc_web utilising the REST API.
• NRPE is available for compatibility reasons, check_nsc_web is the recommended way.
https://github.com/ConSol-Monitoring/check_nsc_web
Beyond NSClient++
• Some checks need to be enabled in the [/modules]” section.
21
Checks
Beyond NSClient++
• Default snclient.ini contains all available values with their defaults
• Best practice: Put changes into a snclient_local.ini which is included by default
• Included files overwrite existing settings
22
Configuration
Beyond NSClient++
• The order of inheritance for the example to the right:
• /settings/sub1/other (most significant)
• /settings/sub1/default
• /settings/sub1
• /settings/default (least significant)
23
Configuration - Inheritance
[/settings/sub1/other]
key = value
[/settings/sub1/default]
; fallback if the above is not set
key = value
[/settings/sub1]
; fallback if the above is not set
key = value
[/settings/default]
; fallback if the above is not set
key = value
Beyond NSClient++
• Default macros:
• ${exe-path}
• ${shared-path}
• ${scripts}
• ${certificate-path}
• ${hostname}
• ${goos} and ${goarch}
• %(macroname) is also supported
• Example:
[/settings/log]
file name = ${shared-path}/snclient.log
24
Configuration - Macros
Beyond NSClient++
• On demand macros can be used to access values from other sections
• Example:
[/settings/external scripts/alias]
alias_allowed_hosts = check_dummy 0 "${/settings/WEB/server/allowed hosts}”
• Those macros are only valid in the .ini file, not as check arguments.
25
Configuration - On Demand Macros
Beyond NSClient++
• Use macro operators to alter existing values:
• Example:
[/settings/external scripts/alias]
alias_hostname = check_dummy 0 "host:${hostname:lc}"
26
Configuration - Macros Operators
https://omd.consol.de/docs/snclient/configuration/#macro-operators
Beyond NSClient++
• Each check builds an internal table for detail data:
.snclient.exe run -vvv check_service filter=“name=snclient”
OK: All 1 service(s) are ok.
• Filter are simple “<key> <op> <value>” sets combined by logical operators
27
Checks - Filter
classification cpu delayed desc name pid service start_type state
service-shared-process 0 % 0 AssignedAccessManager AssignedAccessManagerSvc 0 AssignedAccessManagerSvc demand stopped
service-own-process 1 % 0 Windows Audio Audiosrv 1700 Audiosrv auto running
service-own-process 0.2% 0 snclient snclient 8336 snclient Auto Running
…
https://omd.consol.de/docs/snclient/checks/filter/
Beyond NSClient++
• Thresholds work like filter but instead of removing an entry, the according state will be set.
.snclient.exe run check_service filter="name=snclient" "warn=cpu>90%" "crit=cpu>95" show-all
OK: snclient=running (auto) |'snclient'=4 'snclient rss'=32395264B 'snclient vms'=29831168B 'snclient cpu’=0.1%
• Use multiple warn/crit to create ranges
.snclient.exe run check_drivesize drive=c: warn="used > 10% && used < 30%"
OK: All 1 drive(s) are ok |… 'c: used %'=57.9%;@10:30;90;0;100
28
Checks - Thresholds
Beyond NSClient++ 29
External Scripts
• External scripts need to enabled in the “[/modules]” section first:
[/modules]
CheckExternalScripts = enabled
• Concise Format:
[/settings/external scripts]
my_check1 = check_custom.bat
my_check2 = myscriptscheck_custom.bat
• Verbose Format:
[/settings/external scripts/scripts/my_check1]
my_check1 = check_custom.bat
[/settings/external scripts/scripts/my_check2]
my_check2 = myscriptscheck_custom.bat
Beyond NSClient++ 30
External Scripts
• Don’t:
cmd /c echo scriptscheck_something.ps1 $ARG1$; exit($lastexitcode) | powershell.exe -command -
• Instead:
scriptscheck_something.ps1 $ARG1$
Beyond NSClient++ 31
Plugins
• Plugins are built in checks, but work more like the official monitoring-plugins
• Currently there are:
• check_nsc_web
• check_http
• check_tcp
• check_dns
• Plugins need to be enabled:
[/modules]
CheckBuiltinPlugins = enabled
https://github.com/mackerelio/go-check-plugins
Beyond NSClient++
• Example: use internal check_http to check reachability of services:
.snclient.exe run check_http -H company-service:8443 -S
HTTP OK: HTTP/1.1 307 Temporary Redirect - 202 bytes in 0.073 second response time | time=0.072689s...
.../check_nsc_web -p 123 -u https://hostname:8443 check_http -H labs.consol.de -S
• Example: use internal check_nsc_web to check other snclients:
.snclient.exe run check_nsc_web -p test -u https://192.123.123.2:8443 -k check_os_version
OK: Microsoft Windows 11 Pro 10.0.22631.2428 Build 22631.2428 (arch: amd64)
32
Plugins
Beyond NSClient++ 33
Security
• Native SSL/TLS Support
• Strong Encryption and Security Standards
• Secure Ciphers by Default
• Hashed Passwords:
[/settings/default]
password = SHA256:9f86d081...
• “Allowed Hosts” are used for the webserver as well (all listeners)
Prometheus Stack
Beyond NSClient++ 35
Prometheus Stack
NRPE HTTP(S) / REST
Prometheus
Protocol
Int. Metrics
Node Exp.
Windows E.
Exp Exp
Checks
NSClient++ /
Naemon / Icinga /
Nagios
Beyond NSClient++ 36
Prometheus Stack
• Exporter Exporter
• Basically a reverse proxy for prometheus exporters
Beyond NSClient++ 37
Prometheus Stack
• Exporter Exporter
• Uses .yaml format for compatibility reasons in the “modules dir”
• Example .exporter_modulesclient.yaml:
method: http
http:
scheme: http
port: 9999
path: /metrics
tls_insecure_skip_verify: true
• Does not verify metrics and just passes through
https://github.com/QubitProducts/exporter_exporter
Beyond NSClient++ 38
Prometheus Stack
• Managed Exporter
• Starts and controls exporters
• Controls memory usage
• Restarts exporter in case of errors
• Enable with
“ManagedExporterServer = enabled”
in the modules section
• URL Prefix /example usually results in
/example/metrics as path to metrics
Beyond NSClient++ 39
Prometheus Stack
• Windows Exporter
• Managed Exporter
• Started and controlled by SNClient+
• Checks memory usage and restarts the exporter
Beyond NSClient++ 40
Prometheus Stack
Beyond NSClient++ 41
Prometheus Stack
• Node Exporter
• Same as windows exporter but for linux
Beyond NSClient++ 42
Prometheus Stack
Beyond NSClient++ 43
Prometheus Stack
• Like every good tool in the prometheus eco system
• Provides internal metrics
• Must be enabled with “PrometheusServer” in the modules section.
Updates
Beyond NSClient++ 45
Updates
Examples
Beyond NSClient++ 47
Replace NRPE
Performance
Beyond NSClient++ 49
Performance
• Best practice:
• Naemon Core
• check_nsc_web
• Mod-Gearman (Go) Worker
• But everything else is fine as well
Beyond NSClient++ 50
Performance
Naemon check_nsc_web
exec
SNClient
http(s)
Classical Approach:
Naemon
tcp/aes
Mod-Gearman
Worker
SNClient
http(s)
Mod-Gearman:
The Mod-Gearman Go Worker uses a builtin check_nsc_web
Thruk Agents
Beyond NSClient++
• This is fine, but…
52
Thruk Agents
Beyond NSClient++ 53
Thruk Agents
Beyond NSClient++ 54
Thruk Agents
Roadmap
Beyond NSClient++ 56
Roadmap
• Improve defaults for builtin checks
• Add more builtin checks
• Add configuration validator
• Maybe add support for opentelemetry
Beyond NSClient++ 57
Thanks
• https://github.com/ConSol-Monitoring/snclient
• https://omd.consol.de/docs/snclient/
• mailto:sven.nierlein@consol.de
Slides: https://labs.consol.de/_slides/snclient-2023-11-09.pdf

More Related Content

Similar to SNClient+ - General purpose monitoring agent

Lateral Movement - Hacker Halted 2016
Lateral Movement - Hacker Halted 2016Lateral Movement - Hacker Halted 2016
Lateral Movement - Hacker Halted 2016
Xavier Ashe
 
What are your competitors doing seletskiy 10
What are your competitors doing seletskiy 10What are your competitors doing seletskiy 10
What are your competitors doing seletskiy 10
lsmichael
 

Similar to SNClient+ - General purpose monitoring agent (20)

OSSEC @ ISSA Jan 21st 2010
OSSEC @ ISSA Jan 21st 2010OSSEC @ ISSA Jan 21st 2010
OSSEC @ ISSA Jan 21st 2010
 
OSMC 2009 | Windows monitoring - Going where no man has gone before... by Mic...
OSMC 2009 | Windows monitoring - Going where no man has gone before... by Mic...OSMC 2009 | Windows monitoring - Going where no man has gone before... by Mic...
OSMC 2009 | Windows monitoring - Going where no man has gone before... by Mic...
 
Over-the-Air: How we Remotely Compromised the Gateway, BCM, and Autopilot ECU...
Over-the-Air: How we Remotely Compromised the Gateway, BCM, and Autopilot ECU...Over-the-Air: How we Remotely Compromised the Gateway, BCM, and Autopilot ECU...
Over-the-Air: How we Remotely Compromised the Gateway, BCM, and Autopilot ECU...
 
Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!
Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!
Puppet Camp NYC 2014: Build a Modern Infrastructure in 45 min!
 
Leveraging Cassandra for real-time multi-datacenter public cloud analytics
Leveraging Cassandra for real-time multi-datacenter public cloud analyticsLeveraging Cassandra for real-time multi-datacenter public cloud analytics
Leveraging Cassandra for real-time multi-datacenter public cloud analytics
 
iland Internet Solutions: Leveraging Cassandra for real-time multi-datacenter...
iland Internet Solutions: Leveraging Cassandra for real-time multi-datacenter...iland Internet Solutions: Leveraging Cassandra for real-time multi-datacenter...
iland Internet Solutions: Leveraging Cassandra for real-time multi-datacenter...
 
Distributed Automation(2018) - London Test Automation in Devops Meetup
Distributed Automation(2018) - London Test Automation in Devops MeetupDistributed Automation(2018) - London Test Automation in Devops Meetup
Distributed Automation(2018) - London Test Automation in Devops Meetup
 
Alexei Vladishev - Zabbix - Monitoring Solution for Everyone
Alexei Vladishev - Zabbix - Monitoring Solution for EveryoneAlexei Vladishev - Zabbix - Monitoring Solution for Everyone
Alexei Vladishev - Zabbix - Monitoring Solution for Everyone
 
"Ускорение сборки большого проекта на Objective-C + Swift" Иван Бондарь (Avito)
"Ускорение сборки большого проекта на Objective-C + Swift" Иван Бондарь (Avito)"Ускорение сборки большого проекта на Objective-C + Swift" Иван Бондарь (Avito)
"Ускорение сборки большого проекта на Objective-C + Swift" Иван Бондарь (Avito)
 
Automating Post Exploitation with PowerShell
Automating Post Exploitation with PowerShellAutomating Post Exploitation with PowerShell
Automating Post Exploitation with PowerShell
 
Linux Systems Performance 2016
Linux Systems Performance 2016Linux Systems Performance 2016
Linux Systems Performance 2016
 
Versioning for Developers
Versioning for DevelopersVersioning for Developers
Versioning for Developers
 
Lateral Movement: How attackers quietly traverse your Network
Lateral Movement: How attackers quietly traverse your NetworkLateral Movement: How attackers quietly traverse your Network
Lateral Movement: How attackers quietly traverse your Network
 
Lateral Movement - Hacker Halted 2016
Lateral Movement - Hacker Halted 2016Lateral Movement - Hacker Halted 2016
Lateral Movement - Hacker Halted 2016
 
What are your competitors doing seletskiy 10
What are your competitors doing seletskiy 10What are your competitors doing seletskiy 10
What are your competitors doing seletskiy 10
 
Consul administration at scale
Consul administration at scaleConsul administration at scale
Consul administration at scale
 
PaaSTA: Autoscaling at Yelp
PaaSTA: Autoscaling at YelpPaaSTA: Autoscaling at Yelp
PaaSTA: Autoscaling at Yelp
 
Cloud Platform Symantec Meetup Nov 2014
Cloud Platform Symantec Meetup Nov 2014Cloud Platform Symantec Meetup Nov 2014
Cloud Platform Symantec Meetup Nov 2014
 
CENTRAL MANAGEMENT OF NETWORK AND CALL SERVICES
CENTRAL MANAGEMENT OF NETWORK AND CALL SERVICESCENTRAL MANAGEMENT OF NETWORK AND CALL SERVICES
CENTRAL MANAGEMENT OF NETWORK AND CALL SERVICES
 
Exploring the Final Frontier of Data Center Orchestration: Network Elements -...
Exploring the Final Frontier of Data Center Orchestration: Network Elements -...Exploring the Final Frontier of Data Center Orchestration: Network Elements -...
Exploring the Final Frontier of Data Center Orchestration: Network Elements -...
 

More from Sven Nierlein (7)

Naemon 2019-11-06
Naemon 2019-11-06Naemon 2019-11-06
Naemon 2019-11-06
 
Thruk 2017
Thruk 2017Thruk 2017
Thruk 2017
 
Grafana & PNP4Nagios
Grafana & PNP4NagiosGrafana & PNP4Nagios
Grafana & PNP4Nagios
 
LMD - Livestatus Multitool Daemon
LMD - Livestatus Multitool DaemonLMD - Livestatus Multitool Daemon
LMD - Livestatus Multitool Daemon
 
Monitoring Workshop Kiel 2016 - Thruk II - Panorama Dashboards
Monitoring Workshop Kiel 2016 - Thruk II - Panorama DashboardsMonitoring Workshop Kiel 2016 - Thruk II - Panorama Dashboards
Monitoring Workshop Kiel 2016 - Thruk II - Panorama Dashboards
 
Naemon & Thruk
Naemon & ThrukNaemon & Thruk
Naemon & Thruk
 
Thruk Monitoring Gui
Thruk Monitoring GuiThruk Monitoring Gui
Thruk Monitoring Gui
 

Recently uploaded

Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
FIDO Alliance
 
CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)
Wonjun Hwang
 

Recently uploaded (20)

Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform Engineering
 
Event-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingEvent-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream Processing
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
 
CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)
 
ChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps ProductivityChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps Productivity
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch Tuesday
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoft
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
 
Introduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMIntroduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDM
 
Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdf
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptx
 
JohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptx
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
 
Vector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptxVector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptx
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
Design and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data ScienceDesign and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data Science
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 

SNClient+ - General purpose monitoring agent

  • 1. Replacing NSClient++ for Windows Monitoring Sven Nierlein 09.11.2023
  • 2. Beyond NSClient++ 2 whoami • Name: Sven Nierlein • E-Mail: sven@consol.de • Location: Munich, Germany • Work: consol.de • We might have met before here in Nuremberg: • 2022: Thruk • 2021: Naemon password vault • 2019: Naemon • 2018: Thruk • 2013: Thruk • 2012: Mod-Gearman
  • 4. Beyond NSClient++ 4 Recap: NSClient++ • Windows Agent (and Linux) • Supports several protocols • NRPE • REST • and more … • Lots of built-in checks • Extendable • Customisable
  • 5. Beyond NSClient++ 5 Status Quo • Basically all our customers use NSClient++ in some way • Largest setup ~15k 250k installations • Usually happy with NSClient++
  • 6. Beyond NSClient++ 6 NSClient++ - So what’s the issue? • TLS <= 1.2 • Last release was years ago
  • 7. Beyond NSClient++ 7 NSClient++ • But besides that, it just works… • Anyway, we needed a solution for the SSL issue…
  • 8. Beyond NSClient++ • Just update SSL and rebuild, how hard could it be… 8 Going the easy way…
  • 9. Beyond NSClient++ 9 Looking for alternatives • There are a couple of options: • https://github.com/infraweavers/monitoring-agent • https://github.com/continentale/sma-monitoring-agent/ • https://github.com/continentale/monitoring-agent • https://github.com/it-novum/openitcockpit-agent-go • https://github.com/Icinga/icinga-powershell-framework • Probably missed some more… Alternative by Nick Youngson CC BY-SA 3.0 Alpha Stock Images
  • 10. Beyond NSClient++ 10 Requirements • Drop-In Replacement • Support nsclient.ini Format • Must work on Windows and Linux • Modern SSL/TLS • Checks format/arguments compatible to NSClient • No Change to Naemon configuration • Bonus Points • Prometheus
  • 12. Beyond NSClient++ 12 Introducing SNClient+ • Rewrite NSClient++ in Go • Compatible with nsclient.ini • Multiplatform • Windows • Linux • OSX, Freebsd… • Dual Stack • NSClient++ • Prometheus
  • 13. Beyond NSClient++ 13 What does SNClient mean? • Let’s ask the chatbot of your choice: • Security/Networking Client • Software Name Client • Server Ninja Client • The actual meaning is: • Secure Naemon Client
  • 14. Beyond NSClient++ 14 SNClient Dual-Stack NRPE HTTP(S) / REST NSClient++ / Naemon / Icinga / Nagios Prometheus HTTP(S) Checks Protocol
  • 16. Beyond NSClient++ 16 NSClient / Naemon Stack NRPE HTTP(S) / REST NSClient++ / Naemon / Icinga Prometheus Checks Ext. Scripts Plugins Checks Protocol
  • 18. Beyond NSClient++ 18 Installation • .MSI Installer for windows • DEB/RPM repository for linux: https://labs.consol.de/repo/ • Single go binary
  • 19. Beyond NSClient++ 19 Checks • Built in check commands, ex.: • check_cpu • check_drivesize • check_memory • check_process • check_service • … • Get full list with • .snclient.exe run check_index
  • 20. Beyond NSClient++ 20 Checks • Checks can be executed via check_nsc_web utilising the REST API. • NRPE is available for compatibility reasons, check_nsc_web is the recommended way. https://github.com/ConSol-Monitoring/check_nsc_web
  • 21. Beyond NSClient++ • Some checks need to be enabled in the [/modules]” section. 21 Checks
  • 22. Beyond NSClient++ • Default snclient.ini contains all available values with their defaults • Best practice: Put changes into a snclient_local.ini which is included by default • Included files overwrite existing settings 22 Configuration
  • 23. Beyond NSClient++ • The order of inheritance for the example to the right: • /settings/sub1/other (most significant) • /settings/sub1/default • /settings/sub1 • /settings/default (least significant) 23 Configuration - Inheritance [/settings/sub1/other] key = value [/settings/sub1/default] ; fallback if the above is not set key = value [/settings/sub1] ; fallback if the above is not set key = value [/settings/default] ; fallback if the above is not set key = value
  • 24. Beyond NSClient++ • Default macros: • ${exe-path} • ${shared-path} • ${scripts} • ${certificate-path} • ${hostname} • ${goos} and ${goarch} • %(macroname) is also supported • Example: [/settings/log] file name = ${shared-path}/snclient.log 24 Configuration - Macros
  • 25. Beyond NSClient++ • On demand macros can be used to access values from other sections • Example: [/settings/external scripts/alias] alias_allowed_hosts = check_dummy 0 "${/settings/WEB/server/allowed hosts}” • Those macros are only valid in the .ini file, not as check arguments. 25 Configuration - On Demand Macros
  • 26. Beyond NSClient++ • Use macro operators to alter existing values: • Example: [/settings/external scripts/alias] alias_hostname = check_dummy 0 "host:${hostname:lc}" 26 Configuration - Macros Operators https://omd.consol.de/docs/snclient/configuration/#macro-operators
  • 27. Beyond NSClient++ • Each check builds an internal table for detail data: .snclient.exe run -vvv check_service filter=“name=snclient” OK: All 1 service(s) are ok. • Filter are simple “<key> <op> <value>” sets combined by logical operators 27 Checks - Filter classification cpu delayed desc name pid service start_type state service-shared-process 0 % 0 AssignedAccessManager AssignedAccessManagerSvc 0 AssignedAccessManagerSvc demand stopped service-own-process 1 % 0 Windows Audio Audiosrv 1700 Audiosrv auto running service-own-process 0.2% 0 snclient snclient 8336 snclient Auto Running … https://omd.consol.de/docs/snclient/checks/filter/
  • 28. Beyond NSClient++ • Thresholds work like filter but instead of removing an entry, the according state will be set. .snclient.exe run check_service filter="name=snclient" "warn=cpu>90%" "crit=cpu>95" show-all OK: snclient=running (auto) |'snclient'=4 'snclient rss'=32395264B 'snclient vms'=29831168B 'snclient cpu’=0.1% • Use multiple warn/crit to create ranges .snclient.exe run check_drivesize drive=c: warn="used > 10% && used < 30%" OK: All 1 drive(s) are ok |… 'c: used %'=57.9%;@10:30;90;0;100 28 Checks - Thresholds
  • 29. Beyond NSClient++ 29 External Scripts • External scripts need to enabled in the “[/modules]” section first: [/modules] CheckExternalScripts = enabled • Concise Format: [/settings/external scripts] my_check1 = check_custom.bat my_check2 = myscriptscheck_custom.bat • Verbose Format: [/settings/external scripts/scripts/my_check1] my_check1 = check_custom.bat [/settings/external scripts/scripts/my_check2] my_check2 = myscriptscheck_custom.bat
  • 30. Beyond NSClient++ 30 External Scripts • Don’t: cmd /c echo scriptscheck_something.ps1 $ARG1$; exit($lastexitcode) | powershell.exe -command - • Instead: scriptscheck_something.ps1 $ARG1$
  • 31. Beyond NSClient++ 31 Plugins • Plugins are built in checks, but work more like the official monitoring-plugins • Currently there are: • check_nsc_web • check_http • check_tcp • check_dns • Plugins need to be enabled: [/modules] CheckBuiltinPlugins = enabled https://github.com/mackerelio/go-check-plugins
  • 32. Beyond NSClient++ • Example: use internal check_http to check reachability of services: .snclient.exe run check_http -H company-service:8443 -S HTTP OK: HTTP/1.1 307 Temporary Redirect - 202 bytes in 0.073 second response time | time=0.072689s... .../check_nsc_web -p 123 -u https://hostname:8443 check_http -H labs.consol.de -S • Example: use internal check_nsc_web to check other snclients: .snclient.exe run check_nsc_web -p test -u https://192.123.123.2:8443 -k check_os_version OK: Microsoft Windows 11 Pro 10.0.22631.2428 Build 22631.2428 (arch: amd64) 32 Plugins
  • 33. Beyond NSClient++ 33 Security • Native SSL/TLS Support • Strong Encryption and Security Standards • Secure Ciphers by Default • Hashed Passwords: [/settings/default] password = SHA256:9f86d081... • “Allowed Hosts” are used for the webserver as well (all listeners)
  • 35. Beyond NSClient++ 35 Prometheus Stack NRPE HTTP(S) / REST Prometheus Protocol Int. Metrics Node Exp. Windows E. Exp Exp Checks NSClient++ / Naemon / Icinga / Nagios
  • 36. Beyond NSClient++ 36 Prometheus Stack • Exporter Exporter • Basically a reverse proxy for prometheus exporters
  • 37. Beyond NSClient++ 37 Prometheus Stack • Exporter Exporter • Uses .yaml format for compatibility reasons in the “modules dir” • Example .exporter_modulesclient.yaml: method: http http: scheme: http port: 9999 path: /metrics tls_insecure_skip_verify: true • Does not verify metrics and just passes through https://github.com/QubitProducts/exporter_exporter
  • 38. Beyond NSClient++ 38 Prometheus Stack • Managed Exporter • Starts and controls exporters • Controls memory usage • Restarts exporter in case of errors • Enable with “ManagedExporterServer = enabled” in the modules section • URL Prefix /example usually results in /example/metrics as path to metrics
  • 39. Beyond NSClient++ 39 Prometheus Stack • Windows Exporter • Managed Exporter • Started and controlled by SNClient+ • Checks memory usage and restarts the exporter
  • 41. Beyond NSClient++ 41 Prometheus Stack • Node Exporter • Same as windows exporter but for linux
  • 43. Beyond NSClient++ 43 Prometheus Stack • Like every good tool in the prometheus eco system • Provides internal metrics • Must be enabled with “PrometheusServer” in the modules section.
  • 49. Beyond NSClient++ 49 Performance • Best practice: • Naemon Core • check_nsc_web • Mod-Gearman (Go) Worker • But everything else is fine as well
  • 50. Beyond NSClient++ 50 Performance Naemon check_nsc_web exec SNClient http(s) Classical Approach: Naemon tcp/aes Mod-Gearman Worker SNClient http(s) Mod-Gearman: The Mod-Gearman Go Worker uses a builtin check_nsc_web
  • 52. Beyond NSClient++ • This is fine, but… 52 Thruk Agents
  • 56. Beyond NSClient++ 56 Roadmap • Improve defaults for builtin checks • Add more builtin checks • Add configuration validator • Maybe add support for opentelemetry
  • 57. Beyond NSClient++ 57 Thanks • https://github.com/ConSol-Monitoring/snclient • https://omd.consol.de/docs/snclient/ • mailto:sven.nierlein@consol.de Slides: https://labs.consol.de/_slides/snclient-2023-11-09.pdf