The document outlines the agenda for the Q3 OpenStack Meetup held on September 27, 2017, including check-in, speaker presentations on OpenStack Kolla, and day two operations. Key speakers, Stacy Véronneau and Dirk Wallerstorfer, discussed the latest OpenStack release (Pike) and improvements in various components such as Nova, Neutron, and Cinder. Additionally, the document highlights upcoming events and the role of Ormuco in deploying production-ready containers for scalable cloud operations.

1. Q3 MeetUp
September 27th 2017

2. Agenda
17:30-18:20 - Check-in, food, drinks, and networking
18:20-18:25 - Intros, infos and Reminders
18:25-19:05 - OpenStack Kolla
19:05-19:20 - Break
19:20-20:00 - Day 2 Operations

3. Let’s Say Thanks To Our Sponsors

4. Updates and Speakers

5. Stacy Véronneau
● Director of OpenStack Solutions and Lead OpenStack
Architect at CloudOps.
● Using public cloud resources since 2007
● Started ‘exploring’ OpenStack at Folsom and using at
Grizzly
● OpenStack MeetUp organizer
○ Montreal, Ottawa, Edmonton and Toronto (Co-Org)
● Speaker at OpenStack Days and Summit
○ Austin, Montreal, Barcelona, Boston
● OpenStack Mentor since August 2017

6. New MeetUp Co-Org
Mohammed Naser
PTL Puppet OpenStack

7. Our Speakers
● Gaetan Trellu and Farid Daencarnacao from Ormucco will be
talking about their use of the production-ready
containers and deployment tools for operating OpenStack
called Kolla.

8. Our Speakers
● Our second speaker, coming from Austria, will be will be
Dirk Wallerstorfer from Dynatrace. He will cover some day
2 tools (with demo) and the importance of being ready
once the key is turned on that new shiny cloud.

9. Hail PIKE !

10. Hail PIKE!
● 16th release of OpenStack
● Released on August 30th 2017
● Will be Followed by Queens and Rocky
● New in Nova
○ Nodes can remove themselves from service if they’re not
functioning properly.
○ Keep your instances from using all of the physical CPUs on
your host
○ More efficiently use resources with the PCIWeigher weigher
■ Prevent non-PCI workloads from being scheduled to
those hosts

11. Hail PIKE!
● New in Neutron
○ Zero-downtime upgrades from Ocata
○ Improvements to stability/performance
○ QoS and DVR improvements
○ Support for individual DNS domains set per Neutron port
● New in Cinder
○ Revert to snapshot feature
○ Under certain conditions, we now have support for
extending a volume that is in-use

12. Hail PIKE!
● New in Horizon
○ Download a clouds.yaml file
○ Create and delete ports in your networks
○ See which security groups apply to which Neutron ports
○ Domain dropdown
● New in Swift
○ Support for globally-distributed erasure codes
● New in Ceilometer
○ Support for Manilla and SDN Controllers

13. Hail PIKE!
● New in ...
○ Booting from Cinder volumes(Ironic)
○ Physical network awareness (Ironic)
○ Rolling upgrades (Ironic)
○ Shares backed by CephFS can now use the NFS protocol (Manilla)
○ Magnum, by default, Kubernetes clusters now Include the kubernetes
dashboard.
○ Magnum now includes a monitoring stack based on cAdvisor,
node-exporter, Prometheus and Grafana, but it must be enabled.
○ And many more at
■ https://lnkd.in/gg5raGm
■ https://www.openstack.org/software/pike/

14. Hello LUMINOUS!

15. Hello LUMINOUS!
● New LTS version of Ceph
● Released on August 29th 2017
● Will be Followed by Mimic
● New in Luminous
○ Bluestore engine is the new default
○ Introduction of new Manager Daemon (mgr)
○ New Dashboard
○ Improve performance (bluestore)
○ And many more at http://ceph.com/releases/v12-2-0-luminous-released/

16. OpenStack Days Canada

17. OpenStack Days Canada
● October 19th 2017 - Nature Museum
● https://www.openstackcanada.com/

18. Join us on Slack!
http://openstack-canada-slack-invite.herokuapp.com/

19. Kolla, our Journey...
Because we are Ansible and Docker fanboys !

20. Ormuco ?
● Deploy and manage Public and/or Private clouds
● Multiple federated cloud regions (Canada, Finland, UK)
● Custom user interface (not open source)
● Main offices in Montreal (CA) and Seattle (US)
● OpenStack user since 2015
● OpenStack Powered (RefStack Baby !)

21. A bit of context (1/2)
● Deploying OpenStack is a complicated task !
● Operating OpenStack is complex too…
○ RabbitMQ, Neutron, RIGHT ?????

22. A bit of context (2/2)
● Multiple deployment tools are existing

23. Mission statement
Kolla provides production-ready containers and deployment tools for operating
OpenStack clouds that are scalable, fast, reliable, and upgradable using
community best practices.

24. Always (Coca-)Kolla ?
● Easier deployment process than before (using Puppet, Bash, hands)
● Operational changes are easier to handle with Ansible
● Containerize the OpenStack components
○ Redeploy is faster
○ Consistency in Docker images
● Fast learning curve (even for Farid and Gaëtan)
● Don’t need to be a dev ( ) to understand the code

25. Some Kolla facts
● First commit: 2014-09-17
● Current version: 5.0.0 (Pike)
● “Stable” since Liberty release
● Contributors: ~240
● 3 main repositories
○ kolla
○ kolla-ansible
○ kolla-kubernetes
● Written in Python (should we mention that ?)
● Based on Ansible and Docker

26. Architecture (fancy word)
● Docker Registry
○ The place where Docker images are stored
● Operator node
○ The place where you could build your Docker images and/or run kolla-* commands
● OpenStack nodes
○ Controller, network, computes, etc...
● Ceph cluster
○ Kolla could deploy an entire Ceph cluster or use an external one

27. ● Build Docker images using kolla-build command
How does it work ? (1/8)
$ kolla-build --registry registry-uat:5000 --base ubuntu --base-tag xenial --type source
--threads 24 --nocache --push --namespace ormuco --tag 3.0.3 neutron octavia haproxy
$ kolla-build --tag 3.0.3 neutron octavia haproxy

28. How does it work ? (2/8)
● Generate Kolla passwords using kolla-genpwd command
aodh_database_password: MztMDTg3B60hrCouCouSTACYoPkLrB
aodh_keystone_password: KcuPPQq0jxP0PenStackForeverBAYBYoaW
barbican_crypto_key: Hi8okgW10myhel8IwouldliketobeapornstartycJRIY21AE=
barbican_database_password: aKKTNlVW6BFHelloMommy!1VDbHl5DDfdA
barbican_keystone_password: LookIamOnTVZCG1hlxpj3b6ukgR2ZR
barbican_p11_password: 8QNotReallyaTripleOfanboyANiNz
…

29. How does it work ? (3/8)
● Build the Ansible Kolla inventory based on original examples
○ allinone or multinodes
[control]
controller001
controller002
controller003
[network:children]
control
[compute]
compute001
compute002
[keystone:children]
control
…

30. How does it work ? (4/8)
● Configure Kolla environment file(s)
○ Enable/Disable OpenStack components that should (or not) be deployed
■ Manilla, Barbican, Octavia, Freezer, etc...
○ Add network information
■ FQDN, VIPs, interfaces, etc…
enable_manilla: "yes"
enable_barbican: "no"
kolla_external_fqdn: "big.donuts.com"
tunnel_interface: "bond0.154"
kolla_internal_vip_address: "10.128.150.100"

31. How does it work ? (5/8)
● Customize OpenStack configurations without change the upstream code
$ cat /etc/kolla/config/aodh.conf
[DEFAULT]
notification_topics = notifications,decker

32. How does it work ? (6/8)
● Deploy Kolla using kolla-ansible command (or something else)
$ kolla-ansible deploy -i /etc/ansible/hosts
…
$ kolla-ansible deploy -i /etc/ansible/hosts -t heat
…
$ ansible-playbook -i /etc/ansible/hosts -e @/etc/kolla/globals.yml -e
@/etc/kolla/ormuco_globals.yml -e @/etc/kolla/passwords.yml -e @/etc/kolla/ormuco_passwords.yml
-e CONFIG_DIR=/etc/kolla -e @/opt/kolla-ormuco/kolla/ansible/group_vars/all.yml ormuco.yml

33. How does it work ? (7/8)
● Reconfigure OpenStack components using kolla-ansible reconfigure
$ cat /etc/kolla/config/nova.conf
[DEFAULT]
allow_resize_to_same_host = true
$ kolla-ansible reconfigure -i /etc/ansible/hosts -t nova

34. How does it work ? (8/8)
● Deploying new Docker images using kolla-ansible upgrade
● Update haproxy_tag value in /etc/kolla/globals.yml
$ kolla-ansible upgrade -i /etc/ansible/hosts -t haproxy
$ kolla-build --tag 3.0.3 haproxy
$ docker images
10.128.152.150:4000/kollaglue/ubuntu-source-kolla-toolbox 3.0.2 589b34414729
2 weeks ago 832MB
10.128.152.150:4000/kollaglue/ubuntu-source-haproxy 3.0.3 7c5348a6fb6f
2 weeks ago 340MB

35. Things to improve
● Deploy doesn’t trigger the reconfigure of existing services
● Neutron L3 HA with Kolla could be a funny thing (like really funny...)
● Re-configuring infrastructure service (Galera) is not reliable
● Ansible merge_config module works only with OpenStack components :(
● Locked with Docker

36. Our vision

37. Demo / Q&A
THANKS (and sorry for our English accent)
Farid Da Encarnacao
● Mail: farid.daencarnacao@ormuco.com
● IRC: faridda
Gaëtan Trellu
● Mail: gaetan.trellu@ormuco.com
● IRC: goldyfruit
● Twitter: @goldyfruit

38. confidential
Day 2 operations
OpenStack Meetup, Ottawa, Sep 26, 2017

39. confidential
Dirk Wallerstorfer
 Cloud Technology Strategist @ Dynatrace
 Tech enthusiast
 Husband
 Father
 Son
 Austrian (no kangaroos)
 Never seen “Sound of music”
 Yes, I own a lederhosn
 No, I don’t know how to yodel
@wall_dirk
dirk.wallerstorfer@dynatrace.com

40. confidential

41. confidential
Day 0 Day 1 Day 2 Day 3

42. confidential

43. confidential

44. confidential
NRPE (Nagios Remote Plugin Executor)
...
server_port=5666
nrpe_user=nrpe
allowed_hosts=10.0.0.10
...
/etc/nagios/nrpe.cfg
1
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
controller
compute1
compute2

45. confidential
NRPE (Nagios Remote Plugin Executor)
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
1
2
controller
compute1
compute2
Configuration

46. confidential

47. confidential
NRPE (Nagios Remote Plugin Executor)
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
1
2
controller
compute1
compute2
define host {
address 10.0.0.3
...
}
...
define service {
...
check_command nova-list
}
etc/servers/controller.cfg
...
define command{
command_name nova-list
command_line /usr/local/nagios/plugins/nova-list
}
etc/objects/commands.cfg
#!/bin/env bash
export OS_USERNAME=admin
export OS_PASSWORD=admin
export OS_AUTH_URL=http://10.0.0.3:5000/v2.0
export OS_TENANT_NAME=admin
data=$(openstack server list 2>&1)
rv=$?
if [ "$rv" != "0" ] ; then
echo $data
exit $rv
fi
echo "$data" | grep -v -e '--------' -e '| Status ' -e '^$' | wc -l
/usr/local/nagios/plugins/nova-list

48. confidentialhttps://github.com/rochaporto/dashing-openstack

49. confidential
Nagios
 Monitoring IT infrastructure – and more ...
 Monitoring static entities
 Possibility to monitor dynamically created VMs
 Modify code of nova-api to reconfigure nagios after creation of new VM
 Otherwise shutdown instance will be displayed as outage
 Image that contains Nagios plugins || CAPS scripts to install plugins
 Alternative: Nagios XI – Enterprise
 Nagios Log Server
 Great talk from Nagios world 2014: Monitoring OpenStack
 https://www.youtube.com/watch?v=1U5fo6aPS-k

50. confidential

51. confidential
Elastic
 Previously: ELK Stack
 Now: Elastic Stack

52. confidential
Elastic Stack

53. confidential
Elastic Stack

54. confidential

55. confidential
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
controller
compute1
compute2
1 Setup & Configure Elasticsearch
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html
2 Setup & Configure Kibana
https://www.elastic.co/guide/en/kibana/current/setup.html
server.port: 5601
server.host: "10.0.0.10”
elasticsearch.url: "http://10.0.0.10:9200"
...
/etc/kibana/kibana.yml
network.host: 10.0.0.10
http.port: 9200
...
/etc/elasticsearch/elasticsearch.yml

56. confidential
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
controller
compute1
compute2
3 Setup & Configure Filebeats
filebeat.prospectors:
- input_type: log
paths:
- /var/log/keystone/*.log*
- /var/log/httpd/keystone*.log*
tags:
- keystone
exclude_files:
- ".gz$”
...
output.elasticsearch:
hosts: ["10.0.0.10:9200"]
#username: "elastic"
#password: "changeme"
...
filebeat.modules:
- module: apache2
access:
var.paths: ["/var/log/httpd/*.log*"]
/etc/filebeat/filebeat.yml

57. confidential
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
controller
compute1
compute2
4 Setup & Configure Metricbeats
metricbeat.modules:
- module: system
metricsets:
- cpu
- load
- filesystem
- fsstat
- memory
- network
- process
enabled: true
period: 10s
processes: ['.*']
output.elasticsearch:
hosts: ["10.0.0.10:9200"]
#username: "elastic"
#password: "changeme"
/etc/metricbeat/metricbeat.yml

58. confidential

59. confidential

60. confidential

61. confidential

62. confidential
influxdbelasticsearch mysql

63. confidential
Monasca Agent
The Monasca Agent supports collecting metrics from a variety of sources as follows:
 System metrics
 Nagios plugins
 Statsd
 Host alive (icmp/ssh)
 Process checks (# instances, memory, io, threads)
 Http Endpoint checks
 Service checks (mysql, rabbitmq)
 OpenStack process metrics
The Agent is extensible through configuration of additional plugins, written in Python.

64. confidential

65. confidential

66. confidential

67. confidential

68. confidential

69. confidential
OpenStack Dynatrace Cloud
Dynatrace
Mission Control
OpenStack
Services
Nova,
Neutron,
Keystone,
...
Dynatrace cluster
Security Gateway
WebUI
Access to
OpenStack APIs
Server nodes
Tenant 1
9999
Tenant 2
9999
Tenant 3
9999
443 https
8443
443 https

70. confidential
Setup Dynatrace Security Gateway
# wget -O sgw.sh https://zzv94586.live.dynatrace.com/....
# /bin/sh swg.sh
1
10.0.0.3
10.0.0.4
10.0.0.5
10.0.0.10
controller
compute1
compute2
SGW
Setup Dynatrace OneAgents
# wget -O agent.sh https://zzv94586.live.dynatrace.com/....
# /bin/sh agent.sh
2
Configure Keystone endpoint3

71. confidential

72. confidential
Correlation
 Correlation is a statistical measure that indicates the extent to which two or more variables fluctuate
together.

73. confidential
Correlation doesn‘t imply causation
B Z
Host CPU > 90%Service response time
increases by 2 seconds
B X
Z
C
A W
D
Y

74. confidential

75. confidential

76. confidential

77. confidential

78. confidential
Resource capacity and utilization
OpenStack service availability/performance
Supporting services
Log analytics
Applications running on top
Dependencies
Correlation of metrics/events/data
Real user monitoring, UX affects $
PaaS

79. confidential
This is NOT a
REGULAR SIZE
application
environment!
B X
Z
C
A W
D
Y

80. confidential

81. confidential

82. confidential

83. Confidential, Dynatrace, LLC
Thanks for
having me!