Comparison of four Day 2 tools for monitoring OpenStack: Nagios, Elastic, Monasca, Dynatrace

1. confidential
Day 2 operations
OpenStack Meetup, Ottawa, Sep 26, 2017

2. confidential
Dirk Wallerstorfer
 Cloud Technology Strategist @ Dynatrace
 Tech enthusiast
 Husband
 Father
 Son
 Austrian (no kangaroos)
 Never seen “Sound of music”
 Yes, I own a lederhosn
 No, I don’t know how to yodel
@wall_dirk
dirk.wallerstorfer@dynatrace.com

3. confidential

4. confidential
Day 0 Day 1 Day 2 Day 3

5. confidential

6. confidential

7. confidential
NRPE (Nagios Remote Plugin Executor)
...
server_port=5666
nrpe_user=nrpe
allowed_hosts=10.0.0.10
...
/etc/nagios/nrpe.cfg
1
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
controller
compute1
compute2

8. confidential
NRPE (Nagios Remote Plugin Executor)
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
1
2
controller
compute1
compute2
Configuration

9. confidential

10. confidential
NRPE (Nagios Remote Plugin Executor)
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
1
2
controller
compute1
compute2
define host {
address 10.0.0.3
...
}
...
define service {
...
check_command nova-list
}
etc/servers/controller.cfg
...
define command{
command_name nova-list
command_line /usr/local/nagios/plugins/nova-list
}
etc/objects/commands.cfg
#!/bin/env bash
export OS_USERNAME=admin
export OS_PASSWORD=admin
export OS_AUTH_URL=http://10.0.0.3:5000/v2.0
export OS_TENANT_NAME=admin
data=$(openstack server list 2>&1)
rv=$?
if [ "$rv" != "0" ] ; then
echo $data
exit $rv
fi
echo "$data" | grep -v -e '--------' -e '| Status ' -e '^$' | wc -l
/usr/local/nagios/plugins/nova-list

11. confidentialhttps://github.com/rochaporto/dashing-openstack

12. confidential
Nagios
 Monitoring IT infrastructure – and more ...
 Monitoring static entities
 Possibility to monitor dynamically created VMs
 Modify code of nova-api to reconfigure nagios after creation of new VM
 Otherwise shutdown instance will be displayed as outage
 Image that contains Nagios plugins || CAPS scripts to install plugins
 Alternative: Nagios XI – Enterprise
 Nagios Log Server
 Great talk from Nagios world 2014: Monitoring OpenStack
 https://www.youtube.com/watch?v=1U5fo6aPS-k

13. confidential

14. confidential
Elastic
 Previously: ELK Stack
 Now: Elastic Stack

15. confidential
Elastic Stack

16. confidential
Elastic Stack

17. confidential

18. confidential
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
controller
compute1
compute2
1 Setup & Configure Elasticsearch
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html
2 Setup & Configure Kibana
https://www.elastic.co/guide/en/kibana/current/setup.html
server.port: 5601
server.host: "10.0.0.10”
elasticsearch.url: "http://10.0.0.10:9200"
...
/etc/kibana/kibana.yml
network.host: 10.0.0.10
http.port: 9200
...
/etc/elasticsearch/elasticsearch.yml

19. confidential
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
controller
compute1
compute2
3 Setup & Configure Filebeats
filebeat.prospectors:
- input_type: log
paths:
- /var/log/keystone/*.log*
- /var/log/httpd/keystone*.log*
tags:
- keystone
exclude_files:
- ".gz$”
...
output.elasticsearch:
hosts: ["10.0.0.10:9200"]
#username: "elastic"
#password: "changeme"
...
filebeat.modules:
- module: apache2
access:
var.paths: ["/var/log/httpd/*.log*"]
/etc/filebeat/filebeat.yml

20. confidential
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
controller
compute1
compute2
4 Setup & Configure Metricbeats
metricbeat.modules:
- module: system
metricsets:
- cpu
- load
- filesystem
- fsstat
- memory
- network
- process
enabled: true
period: 10s
processes: ['.*']
output.elasticsearch:
hosts: ["10.0.0.10:9200"]
#username: "elastic"
#password: "changeme"
/etc/metricbeat/metricbeat.yml

21. confidential

22. confidential

23. confidential

24. confidential

25. confidential
influxdbelasticsearch mysql

26. confidential
Monasca Agent
The Monasca Agent supports collecting metrics from a variety of sources as follows:
 System metrics
 Nagios plugins
 Statsd
 Host alive (icmp/ssh)
 Process checks (# instances, memory, io, threads)
 Http Endpoint checks
 Service checks (mysql, rabbitmq)
 OpenStack process metrics
The Agent is extensible through configuration of additional plugins, written in Python.

27. confidential

28. confidential

29. confidential

30. confidential

31. confidential

32. confidential
OpenStack Dynatrace Cloud
Dynatrace
Mission Control
OpenStack
Services
Nova,
Neutron,
Keystone,
...
Dynatrace cluster
Security Gateway
WebUI
Access to
OpenStack APIs
Server nodes
Tenant 1
9999
Tenant 2
9999
Tenant 3
9999
443 https
8443
443 https

33. confidential
Setup Dynatrace Security Gateway
# wget -O sgw.sh https://zzv94586.live.dynatrace.com/....
# /bin/sh swg.sh
1
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
controller
compute1
compute2
SGW
Setup Dynatrace OneAgents
# wget -O agent.sh https://zzv94586.live.dynatrace.com/....
# /bin/sh agent.sh
2
Configure Keystone endpoint3

34. confidential

35. confidential
Correlation
 Correlation is a statistical measure that indicates the extent to which two or more variables fluctuate
together.

36. confidential
Correlation doesn‘t imply causation
B Z
Host CPU > 90%Service response time
increases by 2 seconds
B X
Z
C
A W
D
Y

37. confidential

38. confidential

39. confidential

40. confidential

41. confidential
Resource capacity and utilization
OpenStack service availability/performance
Supporting services
Log analytics
Applications running on top
Dependencies
Correlation of metrics/events/data
Real user monitoring, UX affects $
PaaS

42. confidential
This is NOT a
REGULAR SIZE
application
environment!
B X
Z
C
A W
D
Y

43. confidential

44. confidential

45. confidential

46. Confidential, Dynatrace, LLC
Thanks for
having me!