Self-managing a highly scalable distributed system with Apache Kafka® at its core is not an easy feat. That’s why operators prefer tooling such as Confluent Control Center for administering and monitoring their deployments. However, sometimes, you might also like to import monitoring data into a third-party metrics aggregation platform for service correlations, consolidated dashboards, root cause analysis, or more fine-grained alerts. If you’ve ever asked a question along these lines: Can I export JMX data from Confluent clusters to my monitoring system with minimal configuration? What if I could correlate this service’s data spike with metrics from Confluent clusters in a single UI pane? Can I configure some Grafana dashboards for Confluent clusters? This talk will enable you on achieving the below: Monitoring Your Event Streams: Integrating Confluent with Prometheus and Grafana (this article) Monitoring Your Event Streams: Tutorial for Observability Into Apache Kafka Clients

1. Monitoring Event Streams:
Integrating Kafka on Kubernetes with Prometheus and Grafana
Geetha Anne - Sr Solutions Engineer

2. Data-Driven
Operations
Real-time,
Personalized,
Customer
Experiences
New Business
Models
Businesses are harnessing data in motion

3. Setting data in motion requires a platform that
spans across all of your environments
SELF-MANAGED SOFTWARE
Confluent Platform
The Enterprise Distribution of Apache Kafka
Deploy on-premises or in your private cloud
VM
FULLY MANAGED SERVICE
Confluent Cloud
Apache Kafka Re-Engineered for the Cloud
Available on the leading public clouds

4. Almost every enterprise is
on a journey to the cloud...
>41%
of enterprise workloads are in
public clouds in 2020
4
Source: Forbes

5. Global resiliency
3
Self-service and ease of
operations
2
These companies need a solution that brings the
benefits of cloud to on-premise environments
5
Elasticity to scale to demand
1

6. Kubernetes
The de facto standard for
building cloud-native
platforms for private
infrastructures

7. Confluent for Kubernetes
Introduces a declarative API-driven control plane to
deploy and manage Confluent in private infrastructures
Declarative API for operating
Confluent in production
Manage topics and RBAC
policies through
infrastructure-as-code
Integrates with cloud-native
ecosystem for security,
reliability, and DevOps
automation
Runs on Kubernetes: the infrastructure
runtime for cloud-native architectures

8. Enhanced Proactive Support
● Deploy with confidence
across market-leading
Kubernetes distributions
with a consistent
operational experience
● Build hybrid and
multi-cloud architectures
that span across different
regions and environments
● Become cloud-ready by
easily migrating workloads
to wherever your business
needs them
Intelligent Alerts
● Quickly scale to changing
business demands with
single-command elastic
scaling to meet any data in
motion workload
● Accelerate time-to-value
with infrastructure
-as-code approach,
combined with expert-
backed configs to
automatically deploy and
manage all your Kafka
infrastructure
Monitoring Dashboards
● Implement mission-critical
use cases end-to-end with
infinite storage, disaster
recovery, pre-built
connectors, and SQL-based
stream processing
● Protect sensitive data with
automated security and
cloud-native tooling
● Minimize business
disruption with automated
fault tolerance and rack
awareness
Confluent for K8s offers cloud-native benefits
with additional control and customization
8
Everywhere
Cloud-native Complete

9. Monitoring kafka - the traditional way

10. How does Prometheus work

11. Connecting Grafana to Prometheus

12. JMX Metrics
CFK deploys all Confluent components with JMX metrics enabled by default.
● JMX metrics are available on port 7203 of each pod.
● Jolokia (a REST interface for JMX metrics) is available on port 7777 of each pod.
● JMX Prometheus exporter is available on port 7778.
Authentication / encryption is not supported for Prometheus exporter.

13. Kafka Clusters + CFK
Install Confluent for Kubernetes. You may wish to start with the Quickstart Scenario.
Prerequisites:
● Helm v3.3.0+
● Prometheus Helm chart 13.8.0+ (app version 2.26.0+)
● Grafana Helm chart 6.7.4 (app version 7.5.3+)

14. Enabling Prometheus for Kafka Clusters
Install Prometheus
helm repo add stable https://charts.helm.sh/stable
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm upgrade --install demo-test stable/prometheus
--set alertmanager.persistentVolume.enabled=false
--set server.persistentVolume.enabled=false
--namespace default
Install Grafana
helm upgrade --install grafana grafana/grafana --namespace default

15. Open Grafana in your Browser
Start port-forwarding, so you can access Grafana in your browser with a localhost address:
kubectl port-forward
$(kubectl get pods -n default -l
app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana -o
name)
3000 --namespace default
Get your 'admin' user password:
kubectl get secret --namespace default grafana -o
jsonpath="{.data.admin-password}" | base64 --decode
Visit http://localhost:3000 in your browser, and login as the admin user with the decoded password.

16. Configure Grafana with a Prometheus Data Source

17. Configure Grafana with a Prometheus Data Source
Follow the in-browser instructions to configure a Prometheus data source for Grafana, You will be
asked to provide a URL. Enter the URL as shown below:
http://demo-test-prometheus-server.default.svc.cluster.local
Click "Save & Test". You should see a green alert at the bottom of the page saying "Data source is
working".

18. Import Grafana Dashboard Configuration
Follow the in-browser instructions to import a dashboard JSON configuration.
● Select the confluent-platform.json file located in this folder to load dashboard
for Confluent Platform, and then select the previously-configured Prometheus data
source.
● Select the confluent-operator.json file located in this folder to load dashboard
for Confluent Operator, and then select the previously-configured Prometheus data
source.

19. Setting up Grafana Dashboard

20. Monitor Zookeepers

21. Monitor Kafka Brokers

22. Monitor Kafka Connect Workers

23. Monitor Kafka Topic drill down

24. Conclusion
Scrape Kafka cluster metrics with Prometheus,
Grafana dashboards to chart and analyze cluster activity
Resources
https://docs.confluent.io/operator/current/co-monitor-cp.html#configure-c3-shor
t-to-monitor-ksqldb-kconnect-and-sr-clusters
https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/m
onitoring/grafana-dashboard