Jesse Olson - Nagios Log Server Architecture Overview - This presentation will provide a high-level introduction to Nagios Log Server.

1. Nagios Log Server
Architecture Overview
By Jesse Olson
jolson@nagios.com

2. Introducing: Myself
• Support Technician/Community Ambassador
• jolson

3. Topics Covered
•Elasticsearch, Logstash, Kibana
●
Subsystems (Jobs and Poller)
●
Backup architecture
●
Best practices

4. Elasticsearch
• Database
• JSON Object storage
• Java based
• RESTful HTTP API
• Scalable & Redundant

5. Elasticsearch Terminology
• Instance
• Cluster
• Shard
• Index

6. Index:
01012016
Instance 1 Instance 2
P5 R5
R4P4
R3P3
R2P2
P1 R1
Redundancy and Performance Through Shards
Cluster
Instance 3

7. It Just Works.
• Common Problems with Elasticsearch
• Out of memory
• Out of disk space
• High latency between instances
• Massive deployment
• Pitfalls aren't unique to Nagios Log Server

8. Logstash
• Log collection - inputs
• Log processing - filters
• Log exporting - outputs

9. Inputs
couchdb_changes
drupal_dblog
elasticsearch exec
eventlog file
ganglia gelf
generator graphite
github heartbeat
heroku http
http_poller irc
imap jdbc jmx
kafka log4j
lumberjack
•tcp
•udp
•syslog
meetup pipe
puppet_facter
relp rss rackspace
rabbitmq redis
snmptrap stdin
sqlite s3 sqs
stomp syslog
tcp twitter unix udp
varnishlog wmi
websocket xmpp
zeromq

10. •grok
•mutate
•geoip
Filters
aggregate alter
anonymize
collate csv cidr
clone cipher
checksum date
dns drop
elasticsearch
extractnumbers
environment
elapsed
fingerprint geoip
grok i18n json
json_encode kv
mutate metrics
multiline
metaevent
prune punct
ruby range
syslog_pri sleep
split throttle
translate uuid
urldecode
useragent xml
zeromq

11. Outputs
boundary circonus csv
cloudwatch
datadog_metrics email
elasticsearch exec file
google_bigquery
google_cloud_storage
ganglia gelf
graphtastic graphite
hipchat http irc
influxdb juggernaut jira
kafka lumberjack
librato loggly mongodb
• elasticsearch
• hipchat
• nagios
metriccatcher nagios
null nagios_nsca
opentsdb pagerduty
pipe riemann redmine
rackspace rabbitmq
redis riak s3 sqs
stomp statsd solr_http
sns syslog stdout tcp
udp webhdfs
websocket xmpp
zeromq

12. Outputs
elasticsearch
Inputs Filters
grok/mutate/geoip
NLS Cluster
DEV something.arkham.local IP 192.168.1.1
COUNTRY US CODE ERROR STATUS ON
Field Value
Device something.arkham.local
IP 192.168.1.1
Country US
Response Error
Status On
tcp/udp/syslog

13. Kibana

14. Nagios Log Server Community

15. How does Nagios Log Server
differ from the ELK Stack?
Nagios Log Server Literal ELK stack

16. Key Differences
• Users
• Alerting
• Backups
• Security
• Support
• Administration Time

17. Installation Differences
Nagios Log Server ELK Stack
cd /tmp
wget assets.nagios.com/downloads/nagios-
log-server/nagioslogserver-latest.tar.gz
tar xzf nagioslogserver-latest.tar.gz
cd nagioslogserver
./fullinstall
./upgrade
sudo add-apt-repository -y ppa:webupd8team/java
sudo apt-get update
sudo apt-get -y install oracle-java8-installer
wget -O - http://packages.elasticsearch.org/GPG-
KEY-elasticsearch | sudo apt-key add -
echo 'debhttp://packages.elasticsearch.org/elasticsearch/
1.4/debian stable main' | sudo tee
/etc/apt/sources.list.d/elasticsearch.list
sudo apt-get update
sudo apt-get -y install elasticsearch=1.4.4
sudo vi /etc/elasticsearch/elasticsearch.yml
ADD network.host: localhost
sudo service elasticsearch restart
sudo update-rc.d elasticsearch defaults 95 10
cd ~; wgethttps://download.elasticsearch.org/kibana/kibana
/kibana-4.0.1-linux-x64.tar.gz
tar xvf kibana-*.tar.gz
vi ~/kibana-4*/config/kibana.yml
ADD host: "localhost"
sudo mkdir -p /opt/kibana
sudo cp -R ~/kibana-4*/* /opt/kibana/
cd /etc/init.d && sudo wget
https://gist.githubusercontent.com/thisismitch/8
b15ac909aed214ad04a/raw/bce61d85643c2dcdfbc2728c
55a41dab444dca20/kibana4
sudo chmod +x /etc/init.d/kibana4
sudo update-rc.d kibana4 defaults 96 9
sudo service kibana4 start
sudo apt-get install nginx apache2-utils
sudo htpasswd -c /etc/nginx/htpasswd.users
kibanaadminsudo vi /etc/nginx/sites-available/default
sudo service nginx restart
echo 'debhttp://packages.elasticsearch.org/logstash/1.5/d
ebian stable main' | sudo tee
/etc/apt/sources.list.d/logstash.list
sudo apt-get update
sudo apt-get install logstash
sudo mkdir -p /etc/pki/tls/certs
sudo mkdir /etc/pki/tls/private
sudo vi /etc/ssl/openssl.cnf
cd /etc/pki/tls
sudo openssl req -config /etc/ssl/openssl.cnf
-x509 -days 3650 -batch -nodes -newkey rsa:2048
-keyout private/logstash-forwarder.key -out
certs/logstash-forwarder.crt
cd /etc/pki/tls; sudo openssl req
'/CN=logstash_server_fqd
-batch -nodes -n
private

18. Subsystems: Jobs and Poller
● Queue Based
● Automatic
● Cron Controlled

19. Jobs Subsystem
• Apply configuration
• Changing timezone
• Snapshots
• Start or stop services
• Alerts
• Backups
/usr/local/nagioslogserver/var/jobs.log

20. Poller Subsystem
• Keeps instances clustered
• Checks for updates
• Elasticsearch service status
• Logstash service status
• Instance IP address
• Instance hostname
/usr/local/nagioslogserver/var/poller.log

21. Backup Architecture
• Configuration Backup
• Snapshots
• Log backups
One reason you might need a backup server.

22. Configuration Backups
[jolson@localhost ~]#
ls -lh /store/backups/nagioslogserver
Sep 3 nagioslogserver.2015-09-03.1441308221.tar.gz
Sep 4 nagioslogserver.2015-09-04.1441394621.tar.gz
Sep 5 nagioslogserver.2015-09-05.1441481022.tar.gz
Sep 6 nagioslogserver.2015-09-06.1441567426.tar.gz
Sep 7 nagioslogserver.2015-09-07.1441653826.tar.gz

23. Snapshots
/usr/local/nagioslogserver/snapshots

24. Log Backups
NFS Server
Cluster
i4i2 i3i1

25. Best Practices
• 60GB Memory per instance
• Rotation Schedule
• Avoiding Split Brain

26. Avoiding Split Brain
Instance 1 Instance 2 Instance 3
Minimum Master Nodes: 2

27. Thank you!
Any Questions?