Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ELK and Monasca Crossing: Logging as an OpenStack Service


Published on

Our approach to integrating log management features (based on the ELK stack) into the OpenStack project Monasca.

The talk was given at the OpenStack Summit Tokyo 2015 (October 29). Video link:

Monitoring and log management are two closely related siblings, necessary to guarantee the stability and availability of an OpenStack system. Monasca is a scalable, multi-tenant, high performance, fault-tolerant OpenStack Monitoring as a Service (MONaaS) solution described at In this session we present our work on integrating additional log management functionality into Monasca, to bring the two siblings together in one tool - tailored for OpenStack'ers.

The technical basis of our work consists of Elasticsearch, Logstash and Kibana, aka ELK. The ELK stack has probably become the most widely used open source tool for centralized log management. It also finds frequent adoption among OpenStack users, as demonstrated in various sessions at previous summits. The goal of our work in the Monasca project is to:

1. give operators a unified turn-key solution to manage all logs from their multiple OpenStack systems;
2. provide OpenStack tenants a service that allows for managing logs from their applications running on OpenStack - accessible from Horizon.

As part of this talk we will dive into our architecture to show how to meet our primary design goals: scalability, multi-tenancy, resilience and extensibility. One important aspect is our integration with Monasca that uses common components, such as Kafka, and common architectural patterns. Another highlight that we will point out is our newly implemented REST API, that facilitates Logging as a Service in the spirit of Loggly. Here, authentication happens via Keystone, which extends the existing ELK stack by adding multi-tenancy, applying the tenant model of OpenStack.

Finally, we will give a demo of our solution.

Published in: Technology

ELK and Monasca Crossing: Logging as an OpenStack Service

  1. 1. ELK and Monasca Crossing: Logging as an OpenStack Service Witold Bedyk, Roland Hochmuth, Martin Roderus OpenStack Summit Tokyo, October 29, 2015
  3. 3. Logs in OpenStack • >50 log files on a single node → hundreds of files in a (HA) production OpenStack system • Need for centralized log management Component Service # log files Ceilometer Telemetry 8 Cinder Block Storage 5 Glance Images 2 Heat Orchestration 3 Horizon Dashboard 7 Keystone Identity 1 Neutron Networking 6 Nova Compute 8 Swift Object Storage 3 MongoDB, openvswitch, syslog Supporting services / components 5 etc. etc. etc.
  4. 4. Logging as an OpenStack Service • Multiple vendors offer logging solutions as add-ons – (Most of them based on Elasticsearch) • Our mission: consolidate vendor-specific solutions into a standardized OpenStack project • Analogy to Ironic: replaces multiple vendor-specific tools for bare metal deployment (Foreman, etc.) • Logging-as-a-Service: provide same functionality to OpenStack users – API (RESTful) – Authentication – Multi-tenancy
  5. 5. ELK and Monasca ELK Stack • Elasticsearch: search engine • Logstash: collection, parsing and transformation – Alternatives: Beaver, Fluentd • Kibana: graph dashboard • Competitive to proprietary solutions, such as Splunk Monasca • Metrics Monitoring-as-a- Service • Event processing coming soon • Highly performant, scalable and fault- tolerant
  6. 6. ELK and Monasca Crossing Why not start a separate project ``Logging‘‘? • Related topics: both metrics and logs... – ... indicate the health status of your infrastructure and services – ... let you analyze the root cause of an error • Functional (and nonfunctional) extension to ELK: – Multi-tenancy: Logging-as-a-Service – Performance/scalability – Alarms (future) • Correlation of metrics and logs (and OpenStack notifications)
  7. 7. ARCHITECTURE Metrics, Events and Logs
  8. 8. Metrics Monitoring Overview • Monitoring-as-a-Service – First class RESTfull API for monitoring – Authentication and multi-tenancy • Highly performant, scalable and fault-tolerant • Micro-services message bus architecture provides flexibility, extensibility and load-balancing • Built on Apache Kafka, Apache Storm, InfluxDB and the latest real- time streaming and big data infrastructure. • Metrics storage, retrieval, thresholding and notifications • Real-time streaming complex event processing in progress
  9. 9. Metrics Architecture Query Metrics Create Alarm Definitions Query/Delete Alarms Create Notification Methods system being monitored Monasca Agent Message Queue <<Kafka>> Metrics and Alarms Database Threshold Engine Notification Engine POST metrics Monasca API <<REST>> Horizon, Monitoring Dashboard Monasca Client (CLI) Config Database publishes metrics and domain events Query metrics Query Alarm History Store Alarms Definitions Store Notification Methods Persister
  10. 10. Complex Event Processing (in progress) • Real-time event stream processing • Events API – Store and query events. E.g. OpenStack Notifications – Specify event transforms – Define streams: Filter, group by and trigger on • Specific Events • Elapsed (expired) – Define stream/pipeline handlers • Code that runs when triggers occur. E.g. Evaluate the elapsed time between events. • Transform Engine: Transforms events to normalize and reduce data • Events Engine: Filters, groups, triggers and processes events
  11. 11. Events Processing Architecture Message Queue Events Engine Monasca Events API Events Database Events Transformer consumes raw events publishes transformed events consumes transformed events stores and queries events queries events publishes events sends events
  12. 12. Logging Overview • Built on Elasticsearch, Logstash & Kibana (ELK) • Added logging API + Logstash output plugin • Leverage proven technologies, architecture & design patterns and code in Monasca • Value-add to pure ELK: – Logging-as-a-Service – Greater scalability and performance – Alarms on logs (in progress)
  13. 13. Logging Architecture
  14. 14. Logging Alarms Sequence
  15. 15. Combined Architecture
  16. 16. DEMO
  17. 17. Contact Weekly meeting: Wednesdays at 15:00 UTC IRC: #openstack-meeting-3 on Introducing Using Monasca for Production OpenStack Monitoring Roland Hochmuth, Dan Dyer Aoba room 3:30pm - 4:10pm Monasca Team Meeting Sakura Tower, room S3. 4:40pm – 6:00pm
  18. 18. Logging API • POST methods: JSON. Single and bulk. • Authentication -> scoped queries • Future plan: allow direct Elasticsearch queries POST /v2.0/log/single HTTP/1.1 Content-Type: application/json X-Auth-Token: 27feed73a0ce4138934e30d619b415b0 X-Application-Type: apache X-Dimensions: applicationname:WebServer01,environment:production {"message":"Hello World!", "from":"hoover"}