This document provides details on Day 2 operations for an OpenStack Meetup on September 27, 2017 in Montreal. It includes an agenda for the day which involves setting up and configuring monitoring tools like Nagios, Elastic Stack, Filebeat, Metricbeat and Monasca Agent to monitor an OpenStack cloud deployment. It also discusses using Dynatrace to monitor the OpenStack deployment, correlate metrics and events, and provide real user monitoring.

1. confidential
Day 2 operations
OpenStack Meetup, Montreal, Sep 27, 2017

2. confidential
Dirk Wallerstorfer
 Cloud Technology Strategist @ Dynatrace
 Tech enthusiast
 Husband
 Father
 Son
 Austrian (no kangaroos)
 Never seen “Sound of music”
 Yes, I own a lederhosn
 No, I don’t know how to yodel
@wall_dirk
dirk.wallerstorfer@dynatrace.com

3. confidential
Day 0 Day 1 Day 2 Day 3

4. confidential
Basic concept

5. confidential

6. confidential
NRPE (Nagios Remote Plugin Executor)
...
server_port=5666
nrpe_user=nrpe
allowed_hosts=10.0.0.10
...
/etc/nagios/nrpe.cfg
1
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
controller
compute1
compute2

7. confidential
NRPE (Nagios Remote Plugin Executor)
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
1
2
controller
compute1
compute2
Configuration

8. confidential
NRPE (Nagios Remote Plugin Executor)
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
1
2
controller
compute1
compute2
define host {
address 10.0.0.3
...
}
...
define service {
...
check_command nova-list
}
etc/servers/controller.cfg
...
define command{
command_name nova-list
command_line /usr/local/nagios/plugins/nova-list
}
etc/objects/commands.cfg
#!/bin/env bash
export OS_USERNAME=admin
export OS_PASSWORD=admin
export OS_AUTH_URL=http://10.0.0.3:5000/v2.0
export OS_TENANT_NAME=admin
data=$(openstack server list 2>&1)
rv=$?
if [ "$rv" != "0" ] ; then
echo $data
exit $rv
fi
echo "$data" | grep -v -e '--------' -e '| Status ' -e '^$' | wc -l
/usr/local/nagios/plugins/nova-list

9. confidential

10. confidentialhttps://github.com/rochaporto/dashing-openstack

11. confidential
Nagios
 Monitoring IT infrastructure – and more ...
 Monitoring static entities
 Alternative: Nagios XI – Enterprise
 Nagios Log Server
 Great talk from Nagios world 2014: Monitoring OpenStack
 https://www.youtube.com/watch?v=1U5fo6aPS-k

12. confidential

13. confidential
Elastic
 Previously: ELK Stack
 Now: Elastic Stack

14. confidential
Elastic Stack

15. confidential
Elastic Stack

16. confidential
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
controller
compute1
compute2
1 Setup & Configure Elasticsearch
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html
2 Setup & Configure Kibana
https://www.elastic.co/guide/en/kibana/current/setup.html
server.port: 5601
server.host: "10.0.0.10”
elasticsearch.url: "http://10.0.0.10:9200"
...
/etc/kibana/kibana.yml
network.host: 10.0.0.10
http.port: 9200
...
/etc/elasticsearch/elasticsearch.yml

17. confidential
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
controller
compute1
compute2
3 Setup & Configure Filebeats
filebeat.prospectors:
- input_type: log
paths:
- /var/log/keystone/*.log*
- /var/log/httpd/keystone*.log*
tags:
- keystone
exclude_files:
- ".gz$”
...
output.elasticsearch:
hosts: ["10.0.0.10:9200"]
#username: "elastic"
#password: "changeme"
...
filebeat.modules:
- module: apache2
access:
var.paths: ["/var/log/httpd/*.log*"]
/etc/filebeat/filebeat.yml

18. confidential
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
controller
compute1
compute2
4 Setup & Configure Metricbeats
metricbeat.modules:
- module: system
metricsets:
- cpu
- load
- filesystem
- fsstat
- memory
- network
- process
enabled: true
period: 10s
processes: ['.*']
output.elasticsearch:
hosts: ["10.0.0.10:9200"]
#username: "elastic"
#password: "changeme"
/etc/metricbeat/metricbeat.yml

19. confidential

20. confidential

21. confidential

22. confidential

23. confidential
influxdbelasticsearch mysql

24. confidential
Monasca Agent
The Monasca Agent supports collecting metrics from a variety of sources as follows:
 System metrics
 Nagios plugins
 Statsd
 Host alive (icmp/ssh)
 Process checks (# instances, memory, io, threads)
 Http Endpoint checks
 Service checks (mysql, rabbitmq)
 OpenStack process metrics
The Agent is extensible through configuration of additional plugins, written in Python.

25. confidential

26. confidential

27. confidential

28. confidential

29. confidential
OpenStack Dynatrace Cloud
Dynatrace
Mission Control
OpenStack
Services
Nova,
Neutron,
Keystone,
...
Dynatrace cluster
Security Gateway
WebUI
Access to
OpenStack APIs
Server nodes
Tenant 1
9999
Tenant 2
9999
Tenant 3
9999
443 https
8443
443 https

30. confidential
Setup Dynatrace Security Gateway
# wget -O sgw.sh https://zzv94586.live.dynatrace.com/....
# /bin/sh swg.sh
1
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
controller
compute1
compute2
SGW
Setup Dynatrace OneAgents
# wget -O agent.sh https://zzv94586.live.dynatrace.com/....
# /bin/sh agent.sh
2
Configure Keystone endpoint3

31. confidential

32. confidential
Correlation

33. confidential
Correlation

34. confidential
Correlation doesn‘t imply causation
B Z
Host CPU > 90%Service response time
increases by 2 seconds
B X
Z
C
A W
D
Y

35. confidential

36. confidential

37. confidential

38. confidential

39. confidential

40. confidential
Resource capacity and utilization
OpenStack service availability/performance
Supporting services
Log analytics
Applications running on top
Dependencies
Correlation of metrics/events/data
Real user monitoring, UX affects $
PaaS

41. confidential
This is NOT a
REGULAR SIZE
application
environment!
B X
Z
C
A W
D
Y

42. confidential

43. confidential

44. confidential

45. Confidential, Dynatrace, LLC
Thanks for
having me!