Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Copyright© 2015,NTT Software Corporation. All rights reserved. 1
2015.10.27
NTT Software Corporation
Cloud and Security Bu...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Agenda
• Introduction
• Server monitoring system
– Monitori...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Introduction
• NTT Software uses OpenStack from Essex versi...
Copyright© 2015,NTT Software Corporation. All rights reserved.
what do you need to monitoring
for the OpenStack.
Common
• ...
Copyright© 2015,NTT Software Corporation. All rights reserved.
what do you need to monitoring
for the OpenStack.
Common
• ...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Keyword
I think the OpenStack as one application.
Don’t thi...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Separate the physical and virtual
7
Physical Server
Middle
...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Server Monitoring System
8
Copyright© 2015,NTT Software Corporation. All rights reserved.
what do you need to monitoring
for the OpenStack.
Common
• ...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Server Monitoring(Physical)
• It need what you need in the ...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Physical Server Monitoring
(UserParameter)
11
How to Monito...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Physical Server Monitoring
(UserParameter)
• UserParameter ...
Copyright© 2015,NTT Software Corporation. All rights reserved.
what do you need to monitoring
for the OpenStack.
Common
• ...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Where is Zabbix-Server
14
Project A
VM VM
Hypervisor Zabbix...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Virtual Machine Monitoring
(auto-registration)
• You can au...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Virtual Machine Monitoring
(auto-registration)
16
WEB-serve...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Virtual Machine Monitoring
(network discovery)
• The networ...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Virtual Machine Monitoring
(network discovery)
18
Node1 Nod...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Fault of virtual resources
19
Compute
Node
Storage
Node
Nov...
Copyright© 2015,NTT Software Corporation. All rights reserved.
What do you monitor ?
• PingAlive
• ALL ProcessProcess
• CP...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Fault of virtual resources
• PingAlive
• ALL ProcessProcess...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Integrate multiple of Zabbix screen
22
Copyright© 2015,NTT Software Corporation. All rights reserved.
Where is Zabbix-Server
23
Project A
VM VM
Hypervisor Zabbix...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Too many tabs
24
Tenant A Tenant B
Tenant C Tenant D
Tenant...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Hatohol
25
Zabbix
1
Zabbix
2
Copyright© 2015,NTT Software Corporation. All rights reserved.
Summary of server monitoring
• Failure of the physical side...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Log Monitoring System
27
Copyright© 2015,NTT Software Corporation. All rights reserved.
what do you need to monitoring
for the OpenStack.
Common
• ...
Copyright© 2015,NTT Software Corporation. All rights reserved.
EFK+NZ Log Monitoring
Use Name
Search engine Elasticsearch
...
Copyright© 2015,NTT Software Corporation. All rights reserved.
EFK+NZ Log Monitoring
30
Copyright© 2015,NTT Software Corporation. All rights reserved.
EFK+NZ Log Monitoring
• What is Norikra
31
Schema-less Stre...
Copyright© 2015,NTT Software Corporation. All rights reserved.
EFK+NZ Log Monitoring
• Streaming log
Cut sometime→Analysis...
Copyright© 2015,NTT Software Corporation. All rights reserved.
EFK+NZ Structure
33
OpenStack
Fluentd
Monitoring
Server
Flu...
Copyright© 2015,NTT Software Corporation. All rights reserved.
EFK+NZ Log Monitoring
• Most important things,
how to write...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Detect the variation of the Kibana graph
• ex1) Detect the ...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Detect the variation of the Kibana graph
• keystone don't h...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Detect the variation of the Kibana graph
• keystone don't h...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Detect the variation of the Kibana graph
38
To Norikra of r...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Detect the variation of the Kibana graph
39
Keystone was
at...
Copyright© 2015,NTT Software Corporation. All rights reserved.
ERROR Analysis
• Detection of the patterned error
• Log AAA...
Copyright© 2015,NTT Software Corporation. All rights reserved.
ERROR Analysis
41
• Ex) Using a Flavor that
is too high mem...
Copyright© 2015,NTT Software Corporation. All rights reserved.
ERROR Analysis
42
what's happened.
Copyright© 2015,NTT Software Corporation. All rights reserved.
ERROR Analysis
43
Error cause is not
writing.
Copyright© 2015,NTT Software Corporation. All rights reserved.
ERROR Analysis
44
Zabbix was notified as
a error type 002
Copyright© 2015,NTT Software Corporation. All rights reserved.
Future Action
• Our experience and know-how from the
Essex ...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Announcement
• I will demonstrate the log monitoring system...
Copyright© 2015,NTT Software Corporation. All rights reserved.
ご清聴ありがとうございました。
THANK YOU FOR LISTENING.
47
Copyright© 2015,NTT Software Corporation. All rights reserved.
References Quotation
• Monasca
https://wiki.openstack.org/w...
Copyright© 2015,NTT Software Corporation. All rights reserved.
Trademark
OpenStackは、米国におけるOpenStack,LLCの登録商標です。
Zabbixはラトビ...
Upcoming SlideShare
Loading in …5
×

Monitoring system for OpenStack,using a OSS products

1,000 views

Published on

OpenStack summit japanese session

If you have question
contact me
E-mail:fukazu.satsuki@po.ntts.co.jp

Published in: Software
  • Login to see the comments

Monitoring system for OpenStack,using a OSS products

  1. 1. Copyright© 2015,NTT Software Corporation. All rights reserved. 1 2015.10.27 NTT Software Corporation Cloud and Security Business Department
  2. 2. Copyright© 2015,NTT Software Corporation. All rights reserved. Agenda • Introduction • Server monitoring system – Monitoring of the physical machine – Monitoring of the virtual machine – Integrate multiple of Zabbix screen • Log monitoring system 2
  3. 3. Copyright© 2015,NTT Software Corporation. All rights reserved. Introduction • NTT Software uses OpenStack from Essex version. • Monitoring and Log analysis has been an important issue during the development ,test and operation • I introduce our system that was resolved with a combination of OSS products. 3
  4. 4. Copyright© 2015,NTT Software Corporation. All rights reserved. what do you need to monitoring for the OpenStack. Common • Develop only a little. • One screen. • Automation physical • Resource Monitoring • Middleware Monitoring • Service Monitoring Log Monitoring • Collect • Visualize • Efficiency of error analysis. • Automatic analysis and notification Virtual • Automatic registration of monitoring • Automatic release of monitoring • Monitoring of virtual resources 4 1 2 4 3
  5. 5. Copyright© 2015,NTT Software Corporation. All rights reserved. what do you need to monitoring for the OpenStack. Common • Develop only a little. • One screen. • Automation physical • Resource Monitoring • Middleware Monitoring • Service Monitoring Log Monitoring • Collect • Visualize • Efficiency of error analysis. • Automatic analysis and notification Virtual • Automatic registration of monitoring • Automatic release of monitoring • Monitoring of virtual resources 5 1 2 4 3
  6. 6. Copyright© 2015,NTT Software Corporation. All rights reserved. Keyword I think the OpenStack as one application. Don’t think as a same layer , physical and virtual machine monitoring method . Failure of the physical side , detect by the physical side. Failure of the virtual side , detect by the virtual side. Log monitoring is EFK stack + Norikra + Zabbix 6
  7. 7. Copyright© 2015,NTT Software Corporation. All rights reserved. Separate the physical and virtual 7 Physical Server Middle ware OpenStack VM VM Physical side Virtual side
  8. 8. Copyright© 2015,NTT Software Corporation. All rights reserved. Server Monitoring System 8
  9. 9. Copyright© 2015,NTT Software Corporation. All rights reserved. what do you need to monitoring for the OpenStack. Common • Develop only a little. • One screen. • Automation physical • Resource Monitoring • Middleware Monitoring • Service Monitoring Log Monitoring • Collect • Visualize • Efficiency of error analysis. • Automatic analysis and notification Virtual • Automatic registration of monitoring • Automatic release of monitoring • Monitoring of virtual resources 9 1 2 4 3
  10. 10. Copyright© 2015,NTT Software Corporation. All rights reserved. Server Monitoring(Physical) • It need what you need in the general application monitoring. • It is particularly necessary for the monitoring of OpenStack. – Middleware Monitoring – Service Monitoring – Resource Monitoring 10
  11. 11. Copyright© 2015,NTT Software Corporation. All rights reserved. Physical Server Monitoring (UserParameter) 11 How to Monitor Middleware and Service • UserParameter • You get a script execution results as a monitoring items. • You can use plugin for Sensu, Nagios...and so on
  12. 12. Copyright© 2015,NTT Software Corporation. All rights reserved. Physical Server Monitoring (UserParameter) • UserParameter for nova service – It collects results of Nova hypervisor-show • (※) 引用:https://github.com/sensu-plugins/sensu-plugins-openstack 12 UserParameter=nova.hypervisor-state.running_vms,python /etc/zabbix/bin/nova-hypervisor- metrics.py -u admin -p admin -t admin -a http://192.168.0.10:35357/v2.0 | awk -F '[. t]' '$5=="running_vms" { x=x+$6 } END{print x}' $ python /etc/zabbix/bin/nova-hypervisor-metrics.py -u admin -p devstack -t admin -a http://192.168.0.5:35357/v2.0 | awk -F '[. t]' '$5=="running_vms" { x=x+$6 } END{print x}' 1
  13. 13. Copyright© 2015,NTT Software Corporation. All rights reserved. what do you need to monitoring for the OpenStack. Common • Develop only a little. • One screen. • Automation physical • Resource Monitoring • Middleware Monitoring • Service Monitoring Log Monitoring • Collect • Visualize • Efficiency of error analysis. • Automatic analysis and notification Virtual • Automatic registration of monitoring • Automatic release of monitoring • Monitoring of virtual resources 13 1 2 4 3
  14. 14. Copyright© 2015,NTT Software Corporation. All rights reserved. Where is Zabbix-Server 14 Project A VM VM Hypervisor Zabbix Project A Hypervisor VM Zabbi x Project B VM Zabbi x Pattern A Pattern B
  15. 15. Copyright© 2015,NTT Software Corporation. All rights reserved. Virtual Machine Monitoring (auto-registration) • You can automatically register the monitored host to zabbix-server. • If it is in set to zabbix-agent,zabbix-server's function ,the monitoring settings automatically. • ex) – zabbix-agent.conf metadata="<foo>" – Zabbix-server ‘s setting IF metadata=controller then template = controller 15
  16. 16. Copyright© 2015,NTT Software Corporation. All rights reserved. Virtual Machine Monitoring (auto-registration) 16 WEB-server WEB-server WEB-server metadata=web DB-server DB-server DB-server DB-server DB-server metadata=db APP server APP server metadata=app Zabbix Server IF metadata=web then template = webTemplate IF metadata=db then template = dbTemplate IF metadata=app then template = appTemplate
  17. 17. Copyright© 2015,NTT Software Corporation. All rights reserved. Virtual Machine Monitoring (network discovery) • The network discovery function that is used in automatic monitoring setting added, It use to release the monitoring settings. • It doesn’t distinguish between the power-off / delete / failure of VM. • If you don't restore , when event been abnormality notification. → No recovery plan → Safe to delete from the monitoring target → auto-registration after restoration 17
  18. 18. Copyright© 2015,NTT Software Corporation. All rights reserved. Virtual Machine Monitoring (network discovery) 18 Node1 Node2 Node3 Node4 Node5 zabbix Monitoring Network 192.168.100.0/24 Alert
  19. 19. Copyright© 2015,NTT Software Corporation. All rights reserved. Fault of virtual resources 19 Compute Node Storage Node Nova Cinder VM NW 192.168.0.0/24 VM VM VM VM VM
  20. 20. Copyright© 2015,NTT Software Corporation. All rights reserved. What do you monitor ? • PingAlive • ALL ProcessProcess • CPU • Memory • Disk Resource • HW • NWSNMP 20
  21. 21. Copyright© 2015,NTT Software Corporation. All rights reserved. Fault of virtual resources • PingAlive • ALL ProcessProcess • CPU • Memory • Disk Resource • HW • NWSNMP 21 Detectable
  22. 22. Copyright© 2015,NTT Software Corporation. All rights reserved. Integrate multiple of Zabbix screen 22
  23. 23. Copyright© 2015,NTT Software Corporation. All rights reserved. Where is Zabbix-Server 23 Project A VM VM Hypervisor Zabbix Project A Hypervisor VM Zabbi x Project B VM Zabbi x Pattern A Pattern B
  24. 24. Copyright© 2015,NTT Software Corporation. All rights reserved. Too many tabs 24 Tenant A Tenant B Tenant C Tenant D Tenant Hatohol Hatohol L3
  25. 25. Copyright© 2015,NTT Software Corporation. All rights reserved. Hatohol 25 Zabbix 1 Zabbix 2
  26. 26. Copyright© 2015,NTT Software Corporation. All rights reserved. Summary of server monitoring • Failure of the physical side , detect by the physical side. • Failure of the virtual side , detect by the virtual side. • You can use plugin for sensu, Nagios...etc. for middleware and OpenStack service monitoring. • Add to host「 auto-registration 」 • Del to host「 network discovery 」 • Integrate multiple of Zabbix screen that using hatohol. 26
  27. 27. Copyright© 2015,NTT Software Corporation. All rights reserved. Log Monitoring System 27
  28. 28. Copyright© 2015,NTT Software Corporation. All rights reserved. what do you need to monitoring for the OpenStack. Common • Develop only a little. • One screen. • Automation physical • Resource Monitoring • Middleware Monitoring • Service Monitoring Log Monitoring • Collect • Visualize • Efficiency of error analysis. • Automatic analysis and notification Virtual • Automatic registration of monitoring • Automatic release of monitoring • Monitoring of virtual resources 28 1 2 4 3
  29. 29. Copyright© 2015,NTT Software Corporation. All rights reserved. EFK+NZ Log Monitoring Use Name Search engine Elasticsearch Log collect Fluentd Log visualize Kibana Log analysis Norikra Notification Zabbix 29
  30. 30. Copyright© 2015,NTT Software Corporation. All rights reserved. EFK+NZ Log Monitoring 30
  31. 31. Copyright© 2015,NTT Software Corporation. All rights reserved. EFK+NZ Log Monitoring • What is Norikra 31 Schema-less Stream Processing with SQL Norikra is a open source server software provides "Stream Processing" with SQL, written in JRuby, runs on JVM, licensed under GPLv2. Quotation : http://norikra.github.io/
  32. 32. Copyright© 2015,NTT Software Corporation. All rights reserved. EFK+NZ Log Monitoring • Streaming log Cut sometime→Analysis→Loop 32 Quotation: Esper: Event Processing for Java http://www.espertech.com/products/esper
  33. 33. Copyright© 2015,NTT Software Corporation. All rights reserved. EFK+NZ Structure 33 OpenStack Fluentd Monitoring Server Fluentd Norikra Elastic search File Zabbix Kibana
  34. 34. Copyright© 2015,NTT Software Corporation. All rights reserved. EFK+NZ Log Monitoring • Most important things, how to write the rules of Norikra • Our experience and know-how will be rule of the Norikra. – develop・test・trouble – Log list • Every version. • It does not work if there is no rule after construction. 34
  35. 35. Copyright© 2015,NTT Software Corporation. All rights reserved. Detect the variation of the Kibana graph • ex1) Detect the variation of the Kibana graph. – Suspicious activity • ex2) Detect a failure as a pattern. – Error analysis 35
  36. 36. Copyright© 2015,NTT Software Corporation. All rights reserved. Detect the variation of the Kibana graph • keystone don't have mechanism to detect the dos attack. So, I want to detect it. • When under attack, 401 error log increases. • discover is possible if you look at the graph of Kibana. However, I must always watch. 36
  37. 37. Copyright© 2015,NTT Software Corporation. All rights reserved. Detect the variation of the Kibana graph • keystone don't have mechanism to detect the dos attack. So, I want to detect it. • When under attack, 401 error log increases. • discover is possible if you look at the graph of Kibana. However, I must always watch. 37
  38. 38. Copyright© 2015,NTT Software Corporation. All rights reserved. Detect the variation of the Kibana graph 38 To Norikra of rules it writes with "If from the same IP address per unit time 401 error in ○ times more access occurs." ,
  39. 39. Copyright© 2015,NTT Software Corporation. All rights reserved. Detect the variation of the Kibana graph 39 Keystone was attacked !!
  40. 40. Copyright© 2015,NTT Software Corporation. All rights reserved. ERROR Analysis • Detection of the patterned error • Log AAA+ Log BBB = Error Type 002 • Manually when you analyze. – Lock at log file nova-api.log nova-conductor.log nova-compute.log…and so on – Use grep command. – He requires a log analysis skills. 40
  41. 41. Copyright© 2015,NTT Software Corporation. All rights reserved. ERROR Analysis 41 • Ex) Using a Flavor that is too high memory. Failure to start.
  42. 42. Copyright© 2015,NTT Software Corporation. All rights reserved. ERROR Analysis 42 what's happened.
  43. 43. Copyright© 2015,NTT Software Corporation. All rights reserved. ERROR Analysis 43 Error cause is not writing.
  44. 44. Copyright© 2015,NTT Software Corporation. All rights reserved. ERROR Analysis 44 Zabbix was notified as a error type 002
  45. 45. Copyright© 2015,NTT Software Corporation. All rights reserved. Future Action • Our experience and know-how from the Essex version will continue efforts to expand the rule of Norikra . 45
  46. 46. Copyright© 2015,NTT Software Corporation. All rights reserved. Announcement • I will demonstrate the log monitoring system. – Place:NTT Group booth S14 – Date:29(Thu) 10:30~ 46
  47. 47. Copyright© 2015,NTT Software Corporation. All rights reserved. ご清聴ありがとうございました。 THANK YOU FOR LISTENING. 47
  48. 48. Copyright© 2015,NTT Software Corporation. All rights reserved. References Quotation • Monasca https://wiki.openstack.org/wiki/Monasca • Monasca/Monitoring Of Monasca https://wiki.openstack.org/wiki/Monasca/Monitoring_Of_Monasca • Monasca/Logging https://wiki.openstack.org/wiki/Monasca/Logging • Zabbix Documentation 2.2 https://www.zabbix.com/documentation/2.2/ • Elastic https://www.elastic.co/jp/ • Norikra http://norikra.github.io/ • EsperTech http://www.espertech.com/products/esper.php • Treasure Data Inc http://www.treasuredata.com/ 48
  49. 49. Copyright© 2015,NTT Software Corporation. All rights reserved. Trademark OpenStackは、米国におけるOpenStack,LLCの登録商標です。 Zabbixはラトビア共和国にあるZabbix LLCの商標です。 Erasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries. logstash is a trademark of Elasticsearch BV, registered in the U.S. and in other countries. Kibana is a trademark of Elasticsearch BV, registered in the U.S. and in other countries. その他、文中に記載されている商品・サービス名、および会社名は、それぞれ各社の商標また は登録商標です。 49

×