In this session, Kiran demonstrates how to centralize all the CloudStack-related logs in one place using Elastic Search and generate beautiful dashboards in Grafana. This session simplifies the troubleshooting process involved with CloudStack and quickly helps to resolve the issue. ----------------------------------------- The CloudStack Collaboration Conference 2023 took place on 23-24th November. The conference, arranged by a group of volunteers from the Apache CloudStack Community, took place in the voco hotel, in Porte de Clichy, Paris. It hosted over 350 attendees, with 47 speakers holding technical talks, user stories, new features and integrations presentations and more.

1. Centralized logging for Cloudstack
using ELK and Grafana

2. About me
• QA Engineer – ShapeBlue
• Have a decade experience as
Support Engineer for Cloudstack
at Citrix, Persistent Systems
• Avid Podcast/Audiobooks listener

3. Agenda
• Why the need for the centralized logging
• Understanding ELK Concepts
• Processing of Cloudstack logs using Logstash
• Integrating ELK with Grafana
• Demo of the workflow
• Questions

4. Why the Need for Centralized Logging
• Mangement Server logs
• SSVM logs
• Console Proxy logs
• Virtual Router logs
• Cloudstack usage logs
• KVM Cloudstack agent logs
• Vsphere/Esxi logs
• Xenserver logs
• Various Cloudstack Plugin logs (Kubernetes, Quota)
If we can provide a single pane of glass view to analyze all the Logs and get insights, it’s can save a lot of time for
the end users

5. ELK Concepts
• ELK stack stands for Elastic Search, Logstash and Kibana
https://www.elastic.co/what-is/elk-stack
• Elastic search is based on Apache Lucene is a free and open-source
search engine software library, completely in Java. It’s main purpose is to
store, search and Analyze the data
• Logstash is a tool for parsing the logs and sending it to elastic search
• Kibana is the frontend for Elastic search mainly used for visualization and
querying the data
• Beats is a free and open platform for single-purpose data shippers. They
send data from hundreds or thousands of machines and systems to
Logstash or Elasticsearch

6. Search | View | Interact
Architecture of ELK stack

7. Architecture of ELK stack

8. Filebeat Architecutre
• Filebeat is designed to ship log files. Filebeat helps keep things simple by offering a lightweight way
(low memory footprint) to forward and centralize logs and files

9. Filebeat configuration (Filebeat.yaml )
• On Management server, the following path
can be specified
• Similary for System vm , Router, Hypervisor,
the respective log path can be mentioned

10. Logstash configuration

11. Logstash parsing

12. Data ingestion in ELK

13. Grafana
• Grafana is a popular visualization, monitoring , Alerting and observability Platform, it has integration with wide
variety of data sources (mysql, elk, etc)
https://grafana.com/

14. Sample Dashboard for Cloudstack Logs

15. Overall Architecture Workflow
• Cloudstack generates various logs
• We send the logs (Unstructured Data) to the Logstash tool to parse the data into key value paired
(JSON) (structured data)
• The structured data is stored in elastic search database(index)
• Grafana queries the elastic search to create visualization dashboards
• Single pane of glass to analyze the Logs and get insights and metrics

16. Demo
Code
• https://github.com/kiranchavala/logstash-cloudstack

17. Questions