Monasca 를 이용한 cloud 모니터링 final
•Download as PPTX, PDF•
1 like•680 views
OpenStack 의 Monasca 서비스에 대한 소개
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Monasca 를 이용한 cloud 모니터링 final
Similar to Monasca 를 이용한 cloud 모니터링 final (20)
Monasca 를 이용한 cloud 모니터링 final
- 1. 변상욱 Cloud Consultant Helion OpenStack Product Management of Korea Hewlett Packard Enterprise Monasca 를 이용한 Cloud Monitoring
- 2. Agenda - Monitoring as a Service (Monasca) - Monaca Architecture - Helion OpenStack 의 Monasca - Helion Monitoring Console 소개
- 3. Monitoring Challenge Monitoring 의 필요성 Region Region A Region A Region B Region C Region XRegion B Zone Machine Instance Container• Scaling 의 증가 : 수백대의 서버, Multi-Region 의 Public/Private Cloud 의 Scaling • Cloud 내의 복잡한 구성 : 수많은 VM 들과 Container 들 • 미세 Metric data 추출 : Monitoring Data 의 홍수 • Dynamic 한 환경 : 변화 무쌍한 Infra
- 4. MONASCA MOnitoring At SCAle (Monitoring as a Service) Monasca 는 OpenStack 의 Project 중 하나로, Cloud 서비스의 장애와 VM 및 Cloud 의 성능을 Monitoring 하기 위한 Solution Monasca 의 주요 기능 및 소개 • Cloud 의 Infra 를 Monitoring 하고 Metric 을 이용한 Alarm 생성 • E-mail 등으로 Alarm 발생 시 notification • Java 와 Python 으로 작성 (초창기만 Java 사용) • Apache Kafka 가 주요 component • REST API 를 이용하여 Data 를 수집 • HPE, TWC, Rackspace, Cisco, IBM 등 참여 개발 • Kafka : Apache Kafka 는 분산 Messaging System • Storm : Apache Storm 은 분산 real-time Computing System
- 5. MONASCA Monasca 의 특징 • Monasca API 는 REST API 를 이용하여 Data 를 수집하고, 제공 • 모든 Component 는 HA 와 Scale Up/Out 을 고려하여 Design 됨 • Dedicated DB 를 사용하여, Metric 을 저장 o InfluxDB o Vertica • Infrastructure 의 변화에 대해 동적으로 처리, 사전에 미리 metric 을 정의해 놓을 필요가 없음 • 복합된 Alarm 설정을 통해, real-time 으로 metric 의 alarm 을 이용한 Trouble Shooting • Notification 의 이용 가능 (메일, 삐삐 등) • Apache License 하의 Open Source 로 구성되어 있음
- 6. MONASCA Monasca 의 Architecture • Monasca 는성능이 우수하고, 확장성과 장애 대응이 가능한 구조이며, Micro service message bus 기반의 architecture • REST API 를 사용하여, 빠른 metric 처리 • 모든 Major component 들은 Kafka 를 이용 • Large Scale System 의 모니터링을 위해 HA 와 Scale 이 가능한 Architecture • Kafka : Apache Kafka 는 분산 Messaging System • Storm : Apache Storm 은 분산 real-time Computing System
- 7. MONASCA Monasca 의 Architecture 의 세부 사항 Micro-services Message Bus Based Architecture • Load-Balancing 을 지원하고, 확장성(Scalability)과 시스템 관리를 위한 구조 • High-Availability 가 제공되며, 내구성이 우수하여 Data Loss 에 대한 우려가 없음 • 확장성 (Extensibility) 이 제공되어, Component 와 Service 를 쉽게 추가 할 수 있음 - Helion 의 경우, HP Operation Manager 와 연동하여, Monaca 의 기능 사용이 가능함 - Multi-Site 로 Data Replication 이 가능함 Threshold Engine • Real-Time 으로 Memory 내에서 Streaming 기능을 제공 • Apache Storm Base
- 8. MONASCA API Resources 1. /v2.0/versions 2. /v2.0/metrics 3. /v2.0/metrics/measurements 4. /v2.0/metrics/statistics 5. /v2.0/metrics/names 6. /v2.0/alarm-definitions 7. /v2.0/alarms 8. /v2.0/alarms/state-history 9. /v2.0/notification-methods https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md
- 9. MONASCA Metrics Monasca 를 위한 기본 측정 정의 • GET, POST /v2.0/metrics • GET /v2.0/metrics/measurements • GET /v2.0/metrics/statistics- avg, min, max, sum, count • GET /v2.0/metrics/names Metric 은 중복되지 않은 고유의 이름과 디멘션(Dimension) 으로 구분 짓는다. 디멘션(Dimensions) 은 일종의 Dictionary 로, metric 을 세분화 하여, Key 와 Value 의 쌍들을 정의한다.
- 10. MONASCA Metrics Example POST /v2.0/metrics { name: http_status, dimensions: { hostname: hlm001-cp1-c1-m2-mgmt, cluster: c1, control_plane: ccp, service: compute } timestamp: 0, /* milliseconds */ value: 0.0, value_meta: { status_code: 500, msg: Internal server error } } • Simple, concise, multi-dimensional flexible description • Name (string) • Dimensions: Dictionary of user-defined (key, value) pairs that are used to uniquely identify a metric • Optional dictionary of user-defined (key, value) pairs that can be used to describe a measurement • Normally used for errors and messages
- 11. MONASCA Alarm Definition - 일종의 Template 으로, metric name 과 dimension 이 match 되는 Alarm 을 생성 - 하나의 Alarm 정의는 다수의 Alarm 발생도 가능함 GET, POST /v2.0/alarm-definitions GET, PUT, PATCH, DELETE /v2.0/alarm-definitions{alarm-definition- id} • 복합 Alarm 구성 표현을 위한 간단한 표현식 예제: avg(cpu.user_perc{}) > 85 or avg(memory.system_perc{}) > 45 or avg(disk.read_ops(device=vda), 120) > 100 • Alarm 상태 표시 (OK, ALARM and UNDETERMINED) • Actions associated with alarms for state transitions • Severity (LOW, MEDIUM, HIGH, CRITICAL) 지정 • Thresholds 는 동적으로 수정 가능 Example: POST /v2.0/alarm-definitions { "name":”CPU percent greater than 10", "description":"The average CPU percent is greater than 85", "expression":"(avg(cpu,user_perc{region=uswest})> 85)", "match_by":[ "hostname" ], "severity":"LOW", "ok_actions":[ "c60ec47e-5038-4bf1-9f95-4046c6e9a759" ], "alarm_actions":[ "c60ec47e-5038-4bf1-9f95-4046c6e9a759" ], "undetermined_actions":[ "c60ec47e-5038-4bf1-9f95-4046c6e9a759“
- 12. MONASCA Alarms - Alarm 은 Threshold Engine 에 의해, 알람 정의에 일치하는 metric 이 발생 될 때 동작한다 GET /v2.0/alarms GET, PUT, PATH, DELETE /v2.0/alarms/{alarm-id} Query Parameters: • alarm_definition_id (string, optional) - Alarm definition ID to filter by. • metric_name (string(255), optional) - Name of metric to filter by. • metric_dimensions ({string(255): string(255)}, optional) - Dimensions of metrics to filter by specified as a comma separated array of (key, value) pairs as `key1:value1,key1:value1, ...` • state (string, optional) - State of alarm to filter by, either `OK`, `ALARM` or `UNDETERMINED`. • state_updated_start_time (string, optional) - The start time in ISO 8601 combined date and time format in UTC. Example: List alarms GET /v2.0/alarms?metric_name=cpu.user_perc&metric_di mensions=hostname:devstack&state=ALARM List alarm GET /v2.0/alarms/{alarm-id}
- 13. MONASCA Alarm History - OK, ALARM,UNDETERMINE 과 같은 상태 정보를 저장하여 기록함 GET /v2.0/alarms/state-history GET /v2.0/alarms/{alarm-id}/state-history
- 14. MONASCA Notification - Notification 방법 (Email, Pager Duty, WebHook) 에 대한 List - 다수의 Alarm 정의에 대해, 단일 Notification 방법 선택 가능 Examples: POST /v2.0/notification-methods { "name":"Name of notification method", "type":"EMAIL", "address":“sang-wook.byun@hpe.com" } POST /v2.0/notification-methods { "name":"Name of notification method", "type":”WEBHOOK", "address":”http://example.com/XXX" }
- 15. MONASCA Agent • Python 으로 개발 • Monitor 될 모든 System 에 배포 되어, 수행 • System metric 들 과 Service metric 들의 수집 • System up/down, http status 체크 등을 Active 하게 수행 • Default 30sec 주기로 수행 • Plug-in Architecture 로 되어 있어, 새로운 서비스 등의 추가가 가능 • Monasca 서비스는 component 나 node 장애에 대해 처리 가능 • Monasca API 는 Keystone 으로의 인증 요청을 줄이기 위해, in-memory 방식의 인증 token 을 저장
- 16. Helion OpenStack 2.1 Architecture 소개 Open Source (OpenStack Kilo) Plug-ins HPE Value-add (Open Source) UI UI Execution EnvironmentOperations Environment Infrastructure Services Identity Service (Keystone) Physical Infrastructure – Servers, Networking, Storage OperationalServices Deployment (Ansible) Service Deployment Artifacts Boot Images Service Playbooks Deployment Templates Sub Systems Object (Swift) Storage Service Image (Glance) Library Service Compute (Nova) Service Network (Neutron) Service Block Storage (Cinder) Service Linux for HPE Helion (Debian) Operations (OpsConsole) Dashboard KVM FC Local LDAP/AD Swift OpenStack Dashboard (Horizon) ESX iSCSI LHN 3PAR VMDK Storage (StoreVirtual Dashboard CMC) Sherpa Orchestration Service (Heat) DVR VXLAN VLAN Bare Metal (Cobbler) Provisioning Service Metering Service (Ceilometer) OVSvApp IPMI PXE Ceph ML2 Network Services DNS (DNSaaS) Service DNSaaS Recovery (Freezer) Management (Backup/Restore Scripts) Service Fail-over Management (HAProxy, Keepalived) MySQL Rabbit MQ Centralized Logging (Logstash, ElasticSearch) Infrastructure Monitoring Service (Monasca) HTTPS Termination (Stunnel) Logstash Monasca FW (FWaaS) Service VPN (VPNaaS) Service Federation Configuration Processor LB (LBaaS) Service Vertica Nova ESX (EON) Configuration Logging Search (Kibana) Dashboard HPE Value-add (HPE Assets) UEFI Day Zero Installer LBaaS VPNaaS FWaaS VSASwift Ceph InfluxDB
- 17. Helion OpenStack Cloud Monitoring Benefit 특징 내용 장점 (Benefit) Operations Excellence 자동으로 Deploy 되며, Configuration 은 관리 되는, 설치와 같이 바로 사용 가능한, Turnkey system level monitoring 초기 비용 절감 효과 Organic OpenStack Monitoring Helion 의 모든 OpenStack 배포판에서 이미 검증됨. 특별한 Plugin 과 별도의 agent 또는 특별한 network 구성등을 요구하지 않음. 구축 시간 단축 Simplifies start up experience Hybrid Interoperability API/CLI/Msg Bus 는 HP 관리 SW 제품과 표준으로 연동되며, 3rd Party tool 또한 HP 관리 SW 제품과 연동됨 초기 비용 절감 효과 도입 장벽 요소 감소 Prescribed Resolutions 일반적인 이슈에 대한 해결과 운영 community 로 부터의 Know-How 가용성 증가 문제 해결 시간 단축 Configurable alarms 단일 Host 를 위한 Alarm Level 조정 또는 기존의 정책과 Cluster 환경 부터, 각 각의 환경에 맞는 Alarm Level 의 조정 Easy tuning High Definition Metrics 단일 Alarm 으로 다양한 metric 을 이용하여, 미세한 문제의 발견과 분석이 가능함 빠른 문제 감지 Less down time Performance Tuning 쉽게 조절이 가능하여, 사용자들이 data 와 system 의 load level 을 조절하여, 불필요한 사항의 생성없이, 원하는 level 로 조절이 가능 Scalability and system performance Faster problem resolution Optimized resources Higher staff productivity
- 18. Helion OpenStack Cloud Monitoring Helion 에서의 Monasca Coverage Fully supported Partially supported Not Applicable Helion OpenStack Core Services Helion OpenStack Shared Services Nova Neutron Cinder Nuetron L3agent Glance Swift Ceilometer Horizon Heat Keystone Ops Console Logging Monasca BURA OVS Hlinux MySQL Rabbit Apache LogStash Beaver Elastic Kafka HAProxy Storm Service up? API up? Host up? Perf Resource Utilization Control Plane Cloud IaaS Compute Network Storage Cloud PaaS Application
- 19. Monasca Helion 에서의 Monitoring Factor • System (cpu, memory, network, file system, … ) • Service (MySQL, Kafka, nova, cinder, …. ) • Application Built-in Statsd daemon Python monasca-stats library : Adds support for dimensions • VM system metrics • Active checks HTTP status checks and respose times System up/down check (ping and ssh) • Runs any Nagios plugin or check_mk • Extensible/Pluggable : Additional services can be easily added - Host alive check on all systems using ping check - HTTP Status and response time on all OpenStack service endpoints - Process checks on all relevant processes - System Metrics: CPU, disk, IO, load, memory, process, network, NTP - Services: Elasticsearch, HAProxy, JVM, Kafka, MySQL, RabbitMQ, Zookeeper - OpenStack Services Swift and Monasca specific metrics - VM Metrics CPU, IO, Memory, Network and Host Alive See, http://monasca- agent.readthedocs.org/en/latest/Plugins/
- 20. Helion OpenStack Operation Console 둘러 보기 Administrator 를 위한 별도의 Portal • Cloud 운영을 위한 Center Dashboard • Cloud 의 가용성과 성능을 관리 • Severity, service, status 에 따른 최근 alarm 보기 • Log 확인 및 관리 • Key metric 을 추적하여, Graph 생성 • Real-time 으로 alarm 을 생성하고, 수정 • Notification 방법 관리 : e-mail, pager duty, web hooks
- 21. Helion OpenStack Operation Console 둘러 보기 Dashboard • Dashboard 에서 서비스 중인 Alarm 의 상태를 Monitoring 할 수 있음 • Monitoring 중인 서비스를 Click 하면, 현재의 Alarm 에 대한 세부 창이 나오며, 해당 창에서 Alarm 의 상태와 Dimension 을 확인 가능 • 또 다시, Alarm 을 Click 하면 Detail 한 내용과 Alarm 의 History 를 확인 하고, Comment 를 Update 가 가능
- 22. Helion OpenStack Operation Console 둘러 보기 Alarm Creation 1 1 Menu 의 Alarm Creation 선택 2 Create Alarm Definition 을 선택2 3 Parameter 입력 Ex. Name : Test Description : Test Severity : Low Function : AVG Metric : CPU.SYSTEM_PERC Dimension(s): hostname=hellion-c1 Relational Operator: > (Greater Than) Value : 75 3
- 23. Helion OpenStack Operation Console 둘러 보기 Alarm Creation 결과 조회 2 • 알림에 하나의 내용이 있으며, Click 시, Test 라는 알람이 정상적으로 생성되었다는 메시지를 확인할 수 있음 • Alarm 의 리스트에서, 생성한 Test 알람이 확인 가능 1
- 24. Helion OpenStack Operation Console 둘러 보기 Alarm Explorer • 모든 서비스와 Application 에 대한 alarm display • State, Alarm, Condition 등으로 sorting 이 가능 • Search Bar 를 통하여, key word 입력으로 filtering 하여, 원하는 alarm 을 볼수 있음 • 1번 처럼 alarm 의 check box 를 통해 선택을 하면, 2번의 SET CONDITION 버튼이 생기고, 해당 기능을 통해,Open, Resolved, Acknowledged 로 condition 설정이 가능 • 위 설정된 condition 을 통해 sorting 가능 1 2
- 25. Helion OpenStack Operation Console 둘러 보기 Time Series Graph 1 2 • 원하는 Data 를 Chart 형태로 생성 • 1번 메뉴에서 Time Series Graph 선택 후 Create Chart Click • 2번 Chart 생성 창에서, Chart 형태 (Bar, Line,..) 와 Chart 의 Size 및 Update Rate 을 선택 • 원하는 metric 을 선택 하여, 연계되는 dimension 의 값을 선택 • Data 를 Chart 에 Add 하고, Create Chart 를 선택하며, 해당 Chart 가 3번 처럼 생성됨 3
- 26. Helion OpenStack Operation Console 둘러 보기 Logging • Helion OpenStack 은 Central Log 관리를 위해 ELK (Elastic Search, LogStash, Kibana) 가 integrate 되어 있어, Operation Console 을 통해, Log 검색, Visual 한 Graph 생성이 가능 하도록 구성되어 있음
- 27. Monasca 마치며 - Monasca 는 Cloud 의 필수 project 로 자리 잡을 것으로 예상 됩니다. - VM instance 내의 Service 에 대한 Monitoring 의 need 에 따라, 해당 부분도 발전될 것으로 예상됩니다. • Monasca 는 Devstack 에 plugin 하여 Monasca 를 맛 볼수 있습니다. • Monsaca 는 Docker container 를 이용하여, Monasca Demo 를 보실 수 있습니다. https://github.com/openstack/monasca-api/tree/master/devstack https://hub.docker.com/r/monasca/demo/
- 28. Q & A
- 29. 감사합니다 29
Editor's Notes
- This is a sample Title Slide with Picture ideal for including a dark picture with a brief title and subtitle. A selection of pre-approved title slides are available in the HPE Title Slide Library. The location of the library will be communicated later. To insert a slide with a different picture from the HPE Title Slide Library: Open the file HPE_16x9_Title_Slide_Library.pptx From the Slide thumbnails pane, select the slide with the picture you would like to use in your presentation and click Copy (Ctrl+C) Open a copy of the new HPE 16x9 template (Standard or Events) or your current presentation In the Slide thumbnails pane, click Paste (Ctrl+V) A Paste Options clipboard icon will appear. Click the icon and select Keep Source Formatting. (Ctrl+K)
- Monasca Session 에서는 아래와 같은 내용을 다룰 예정입니다. Monasca 의 의미와, 그 Architecture, 그리고 Monasca 를 이용하고 있는 Helion OpenStack 의 Architecture 와, Helion OpenStack 에서의 Monitoring 사용 예제를 monitoring console 을 통해 살펴 보도록 하겠습니다.
- Monitoring solutions have been around for decades, but in many respects they fail to address the requirements of monitoring large-scale public and private clouds. Traditionally, performance, scalability and data retention have been limited to hundreds of systems. In a large-scale cloud service thousands of physical servers and hundreds of thousands of virtual machines (VMs) and containers need to be monitored, resulting in hundreds of terabytes of monitoring data. The original monitoring source data needs to be stored in an on-line, queryable, lossless form at data retention periods greater than thirteen months. Such long data retention periods are necessary for SLAs, business continuity, and analytics. Inventory elasticity is important because cloud infrastructure is constantly evolving with VMs and services continually being created and destroyed monitoring systems must be dynamic enough to understand the difference between a VM be purposely destroyed versus a VM that is in a failed state. Self-service models that empower teams to easily add new resources and monitor them independently of the monitoring teams involvement is necessary. Most solutions assume a static infrastructure that requires new services to be registered with the server prior to being monitored. This results in the monitoring team/server being the bottleneck. Extensibility is critical, but is often limited. Run-time configurability is necessary to be able to tune the system over time by allowing alarms to be dynamically adjusted, which in many systems is not supported. Generalization of alarm definitions/templates is necessary to describe and manage alarms in a one-to-many relationship in order to avoid having to manually declare each alarm even though they may share many common attributes and differ in only one, such as hostname. Spammy alerts and alert fatigue is a common short-coming of every thresholding system. Many operations teams receive thousands of alerts on a weekly basis. Improvements in run-time configurability and generalizing alarm definitions can help to address spammy alerts. Anomaly detection based on non-parametric statistics and machine learning is required as a more fundamental change.
- Monasca is a highly performant, scalable, fault-tolerant and extensible micro-services messages bus based architecture. It uses a REST API for high-speed metrics processing and querying and has a streaming alarm engine and notification engine. All of the major components are linked using Kafka. Every component in the system is built with High Availability (HA) in mind and can be scaled either horizontally or vertically to allow for monitoring of very large systems. The Monasca API is the gateway for all interaction with Monasca. In a typical scenario metrics are collected by the Monasca Agent running on a system and sent to the Monasca API. The API then published the metrics to the Kafka queue. From here the Monasca Persister (metric 과 Alarm 상태를 Kafka 에서 Read 하여, Metric DB 로 전환해주는 역할), consumes metrics and writes them to ourMetrics database. The Monasca Threshold Engine also consumes the metrics and uses them to evaluate alarms. At this point the metrics are in our system and can be queried using the Monasca API, either directly or through one of our other components, such as the Horizon plugin or the Monasca CLI. When the Threshold Engine evaluates the metrics against the alarms it can create alarm state transition events. These are published back to Kafka and are read by both the persister and Notification Engine. The Persister writes the alarm transitions to the DB for future retrieval. The notification engine will send a notification of the configured type for appropriate state transitions. In addition to the components discussed above we also have a configuration database used for storing information such as alarm definitions and notification methods. This database can be either MySQL or PostgreSQL.
- Advantages of message bus architecture Enables a micro-services foundation Load-balancing, scalability, system maintenance (new deploys) Handle different loads Extensibility: Easily add new components/services: HP Operations Manager i (OMi) BSM Connector for HP Helion Monasca Consumes alarm state transition messages from Kafka Multi-site replication of data And there is more... Pagination (책 등에 매긴 페이지 번호) is supported via offset and limit query parameters The Agent Forwarder buffers metrics for a short time to increase the size of the http request body (number of metrics) sent to the Monasca API. The Monasca API caches auth tokens in-memory to reduce the round-trip authorization requests to Keystone If network connectivity between the Agent and API occurs the Agent will buffer metrics and send when connectivity is restored Metrics are submitted using a “agent” role, which only allows metrics to be POST’d to the metrics endpoint Multi-site replication for metrics can be done by running two persisters simultaneously, sending to different metrics databases System can handle failure of any component or node
- Monasca-statsd daemon : statsd engine capable of handling dimensions associated with metrics submitted by a client that supports them. Also supports metrics from the standard statsd client. (udp/8125)
- The Helion platform provides a turnkey monitoring system that is ready to use immediately after cloud installation. This saves operators time and money by eliminating the need for a separate monitoring infrastructure and from having to manage complex network configurations. All aspects of the monitoring system are certified with HP Linux for Helion and Helion OpenStack and they are supported by HP. This saves operators set up time and lowers costs because operators do not need to stand up separate infrastructure or certify additional-plug ins with the Helion environment. HP Helion OpenStack monitoring ships with many integration points and can easily snap into existing data center management tooling and infrastructure. It ships with supported connectors with HPSW OMi and technical preview connectors for Ops A, Splunk, and ArcSight. The Helion OpenStack 2.0 documentation contains documented triage and resolution steps for common issues. This knowledge is based on years of OpenStack software operations experience from operating HP’s public cloud services. Reduces time to production Simplifies start up experaince
- We are monitoring all of the OpenStack core service availability and performance metrics. We are collecting log events from all OpenStack core services and most of the shared services. For a complete listing of alarms and monitored services please see the Helion documentation Monasca/alarms.