Monitoring Cockpit for OpenShift Clusters

The document discusses monitoring an OpenShift cluster with Prometheus. It describes what components need monitoring, including nodes, services, and pods. Prometheus is well-integrated for Kubernetes monitoring. The architecture proposed uses Prometheus to scrape metrics from targets like nodes and services, with alerting configured and dashboards built. It references existing Prometheus mixins for Kubernetes and OpenShift monitoring best practices. Special design choices like using remote write and a Blackbox exporter are highlighted.

Internet

Monitoring Cockpit
for OpenShift Clusters
Ulrike Klusik
25.11.2019, OpenShift Munich Meetup

Monitoring Cockpit 2
OpenShift Infrastructure
What needs to be monitored?
• From Monitoring perspective:
• Nodes as compute resources and
Node states
• central service URLs with high
availability and performance SLAs
• infrastructure Pods implementing
the services which can be
dynamically changing
• The API already provides meta data
about the cluster components =>
used to determine the metric targets
from https://docs.okd.io/3.11/architecture/index.html

Monitoring Cockpit 3
Monitoring with Prometheus
• Prometheus is good integrated in Kubernetes
ecosystem.
• Idea of Monitoring with Prometheus
• Monitored targets must provide their specific
metrics via http(s) endpoints
• Targets are typically determined dynamically
via service discovery and regularly scraped
• Alert rules are defined as conditions on metrics
• Alertmanager deduplicates alerts
and routes them to the incident handling tools
from https://prometheus.io/assets/architecture.png

Monitoring Cockpit 4
State of the Art Kubernetes Monitoring with Prometheus
• Prometheus Monitoring Mixin for Kubernetes
(https://github.com/kubernetes-monitoring/kubernetes-mixin)
• Provides for the standard Kubernetes services and nodes:
• Alert rules
• Dashboards
• Dashboard for resource usage of Pods/Containers
• Redhat’s OpenShift (a commercial Kubernetes implementation) includes an immutable
Prometheus Monitoring solution (https://github.com/openshift/cluster-monitoring-operator)
with fix alerts and dashboards. Also bases on the Mixins, plus some OpenShift specific
additions.
• What we were missing in these solutions:
• End user / application experience of cluster services
• Metrics volume too large for longer metric retention
• Cluster Overview over service and node availability

Monitoring Cockpit 5
namespace
Nodes
host
NODE-
EXPORTER
OMD server
INFLUXDB
ALERTMGR
(cluster possible)
Container
OMD-Service
Grafana
Monitoring Architecture
Kubelet +
cAdvisor
Openshift
metric target
HAProxy(Router)
infrastructure projects
remote write
(selected metrics)
Incident Mananagent
systems (e.g. Remedy,
Service Now)
custom webhook
api-servers
kube controllers
EFK Logging
(via Pods)
GlusterFS (via
Heketi-Route)
Project prometheus-infra-mon
PROMETHEUS
KSM/OSM
Kubernetes/OpenShift Cluster
• Kube-State-Metrics(KSM)/
OpenShift-State-Metrics(OSM):
metrics over objects and their
states
• Node-Exporter for operation
system metrics
• Blackbox Exporter: for test calls
to Service URLs
Blackbox
Exporter
Kubernetes
metric target

Monitoring Cockpit 7
Conclusion
• Special design decisions:
• remote write: The key metrics are stored in an external database for longer retention
• blackbox exporter: for active service availability tests

ConSol
Consulting & Solutions Software GmbH
St.-Cajetan-Straße 43
D-81669 München
Tel.: +49-89-45841-100
info@consol.de
www.consol.de
Twitter: @consol_de

This document discusses various tools for debugging applications running in Kubernetes clusters. It begins with an introduction to Kubernetes and provides an example of running a Java application in debug mode. It then summarizes several tools for debugging Kubernetes applications: kubectl port-forwarding, Telepresence, and Squash. Telepresence allows debugging local processes as if they are running in Kubernetes. Squash provides debugging integrated into IDEs. Port-forwarding can be used if other options are not available. The document also briefly mentions kubefwd and Rookout as bonus debugging tools.

"Using Automation Tools To Deploy And Operate Applications In Real World Scen...

Operator development made easy with helm

- Operators are applications that extend Kubernetes to manage complex stateful applications. They use custom resource definitions (CRDs) to configure and automate tasks. - Helm is a good starting point for creating operators as it is widely used and easy to learn. Operators created with Helm can later be used to manage resources in other operators. - The demo showed creating a Helm operator from a Nginx chart and combining two operators with ArgoCD to deploy example apps based on custom resources.

"Messaging with Quarkus"

Utilizing messaging systems grants us the capability to decouple services from each other: downtime of a service consuming messages does not impact the functionality of the service sending messages and vice-versa. In this talk we will discuss how to setup and use messaging systems. As practical examples, we use AMQP backed by ArtemisMQ, as well as kafka to send and receive messages, automatically as well as programmatically.

OpenShift Meetup - Red Hat OpenShift Container Storage explained

Red Hat OpenShift Container Storage (RHOCS) is software-defined storage optimized for container environments that addresses challenges around platform services, application portability, and persistence. It uses Red Hat Ceph Storage, the Rook.io operator, and NooBaa's Multicloud Object Gateway technology. RHOCS provides simplified deployment and management of storage and allows applications and storage to run on the same or different nodes in an OpenShift cluster, or to connect multiple OpenShift clusters to an external Ceph storage cluster. The presentation demonstrated a proof-of-concept using an external Red Hat Ceph storage cluster with OpenShift Container Platform 4.6.

K8s debugging talk

Olaf Meyer discusses various options for debugging Java applications running in Kubernetes clusters. He begins with an introduction to Kubernetes and demonstrates debugging with the kubectl command line tool and port forwarding. Meyer then covers using Telepresence and Squash for debugging, noting Squash integrates well with IDEs. While Telepresence is good for development containers, Squash has no overhead for Java applications. Additional debugging tools discussed include kubefwd and Rookout.

"Remote development of Quarkus applications"

A central focus of Quarkus is developer joy and productivity. As part of this focus, Quarkus provides a powerful development mode which, among other things, allows true hot compilation of code. In this talk we will take a look at this feature in detail. Aside from allowing hot reloading locally, we can also configure our application to allow hot reloading remotely. For example, the application could be running in a docker container or on a kubernetes-/openshift-cluster. This improves Dev/Prod-parity and thus enhances the development lifecycle.

"Kubernetes as Driver of Generic IT Automation"

Kubernetes as a platform is moving fast from being the "new IT" to standing right in the center of most companies infrastructure. What does that mean for IT Automation? For its own purposes, Kubernetes already comes with a well-engineered declarative model of managing computing resources that has proven to be very efficient. In classic IT, likewise proven automation solutions like Red Hat Ansible are established. This forms two automation silos, and as we all know: Silos are a bad thing. Is there a way to bridge this gap? In this session we will highlight the possibilities to use Kubernetes state management as backbone for IT automation by extending it with custom operators using Red Hat Ansible. Ansible with its focus on idempotency is a really great match for implementing Kubernetes-Operators and doing it to automate non-K8s resources, just like you would do with Ansible Tower, is easier than you might think. We will have a look at different use cases and provide a strategic outlook.

Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. It allows developers to easily scale containerized applications and provides features like deployments, monitoring, and a powerful API. Minikube allows developers to run a single-node Kubernetes cluster locally for testing and development, providing consistency between environments and making it easier to debug production issues.

Kubernetes Logging

Denys Havrysh

This document discusses implementing and testing a self-managed logging and visualization solution for a Kubernetes cluster. It considers tools like FluentD, Elasticsearch, Kibana, Helm, and Kops for collecting, processing, and visualizing logs. A turn-key deployment approach using Helm is recommended to install all stack components from a single chart and leverage dependencies. Concerns about authentication, capacity planning, and security hardening are noted for future improvement.

OSDC 2018 - Distributed monitoring

Gianluca Arbezzano

Modern software development is increasingly taking a “microservice” approach that has resulted in an explosion of complexity at the network level. We have more applications running distributed across different datacenters. Distributed tracing, events, and metrics are essential for observing and understanding modern microservice architectures. This talk is a deep dive on how to monitor your distributed system. You will get tools, methodologies, and experiences that will help you to realize what your applications expose and how to get value out from all these information. Gianluca Arbezzano, SRE at InfluxData will share how to monitor a distributed system, how to switch from a more traditional monitoring approach to observability. Stay focused on the server’s role and not on the hostname because it’s not really important anymore, our servers or containers are fast moving part and it’s easy to detach it from the right in case of trouble than call the server by name as a cute puppet. How to design a SLO for your core services and now to iterate on them. Instrument your services with tracing using tools like Zipkin or Jaeger to measure latency between in your network.

Five Lessons Learned from Large-scale Implementation of Kubernetes in the Ent...

DevOps.com

Learn what it takes to get a large-scale production Kubernetes cluster up and running in an on-premises environment. Get tips, best practices, and architectural approaches for the initial implementation. Understand the constraints, challenges, and how to solve for successful on-going day-2 operational issues. Join Sirish Raghuram, CEO of Platform9, and Ravi Ravichandran, VP cloud platform engineering at Juniper, for a fun conversational webinar where they will discuss how to avoid all the pitfalls of running production Kubernetes and ensuring high-reliability, scalability, and performance. They will also touch upon the implications of building and operating your own production clusters using your own resources and upstream Kubernetes versus commercial alternatives available in the market.

Deploy prometheus on kubernetes

Cloud Technology Experts

Kubernetes Helm: Why It Matters

Kubernetes Helm makes application deployment easy, standardized and reusable. Use of Kubernetes Helm leads to better developer productivity, reduced Kubernetes deployment complexity and enhanced enterprise production readiness. Enterprises using Kubernetes Helm can speed up the adoption of cloud native applications. These applications can be sourced from open-source community provided repositories, or from an organization’s internal repository of customized application blueprints. Developers can use Kubernetes Helm as a vehicle for packaging their applications and sharing them with the Kubernetes community. Kubernetes Helm also allows software vendors to offer their containerized applications at “the push of a button.” Through a single command or a few mouse clicks, users can install Kubernetes apps for dev-test or production environments.

Kubernetes Deployments: A "Hands-off" Approach

Rodrigo Reis

Cost-effective Compute Clusters with Spot and Pre-emptible Instances - KubeCo...

Kubernetes and Spot/Pre-emptible Instances (SPIs) are arguably a match made in heaven. Traditionally, the uncertainty of SPIs (they can be terminated at any time due to price fluctuations) have made managing them tricky, and restricted them to specific workloads and use cases. Kubernetes, in contrast, not only handles node failure very well, it has trained developers and architects to design applications to tolerate and even embrace failure. The prospect of Kubernetes abstracting the complexities of SPIs is now a reality, enabling applications to take advantage of low-cost compute across different clouds and possibly vendors. The purpose of this talk is to educate the audience on strategies for making the most out of this powerful combination. Specifically, we will discuss these topics: 1. What are spot bidding strategies, and what is their cost vs. predictability trade-off? 2. What class of Kubernetes applications would benefit the most from SPIs? 3. Available Kubernetes mechanisms (e.g taints/tolerations, affinity, availability zones) for placing applications based on their tolerance with SPIs 3. Implementation strategies (e.g. blending multiple autoscaling groups to satisfy both SPI-optimized applications vs. applications that are more mission-critical or stateful) 4. What out-of-the box solutions exist, either free or commercial? 5. How to take abstract away clouds from different regions and vendors, allowing workloads to always take advantage of the best available pricing? The talk concludes with real-world test results involving multiple use cases and configurations, giving the audience an idea of the potential cost savings and trade-offs (if any) of combining Kubernetes and SPIs.

Helm - Application deployment management for Kubernetes

Alexei Ledenev

Top 5 Considerations for Successful Deployment of Kubernetes

Thinking One Step Further with Time-saving DevOps Tools with Open Telekom Clo...

Bitnami

For on-demand recording of the webinar, click here: https://youtu.be/aN_DaNFfBx4 Why You Should Watch Application developers are well-advised to not only think about their actual programming, related time-saving tools and deployment schemes, but also about the specific needs of application operations - in particular with regards to data privacy requirements when dealing with clients in Europe and their customer data which are processed in applications. That's where the Open Telekom Cloud kicks in. As the the "new kid on the block", Open Telekom Cloud's public cloud offering features Bitnami's vast range of open source applications. Join Bitnami as we host our featured speaker, Max Guhl, from Deutsche Telekom. He will showcase the Open Telekom Cloud's intuitive user interface and how this Public Cloud does not only smoothly integrate Bitnami's application catalog, but also provides answers on how to comply with the upcoming European General Data Protection Regulation already today Register now to watch and learn: What the Open Telekom Cloud is How to launch and manage DevOps tools and instances on Open Telekom Cloud Actions to keep the new European General Data Protection Regulation (GDPR) in mind The benefits of using Bitnami with Open Telekom Cloud

Getting Started with Kafka on k8s

VMware Tanzu

This document provides an agenda and overview of Kafka on Kubernetes. It begins with an introduction to Kafka fundamentals and messaging systems. It then discusses key ideas behind Kafka's architecture like data parallelism and batching. The rest of the document explains various Kafka concepts in detail like topics, partitions, producers, consumers, and replication. It also introduces Kubernetes concepts relevant for running Kafka like StatefulSets, StorageClasses and the operator pattern. The goal is to help understand how to build event-driven systems using Kafka and deploy it on Kubernetes.

Kubernetes 1.16 and rancher 2.3 enhancements

Saiyam Pathak

Kubernetes 1.16 includes several new features such as IPV4/IPV6 dual stack support, PVC cloning, custom resource definition enhancements, and server-side apply moving to the API server. Rancher 2.3 provides the ability to reuse Kubernetes configurations across clusters and is the first to deliver general availability support for Windows containers and Kubernetes with Windows worker nodes. The document discusses these new features in Kubernetes 1.16 and enhancements in Rancher 2.3.

Kubernetes fundamentals

Victor Morales

Introduction to container mangement

Martin Marcher

This document provides an overview of container management and Kubernetes concepts. It discusses delivery and deployment methods like classic deployment, containers, virtualization, and container orchestration. It then covers Kubernetes components like etcd, the control plane, and nodes. It outlines cluster administration tasks and best practices for cluster usage. Finally, it provides examples of Kubernetes resource types like pods, replica sets, and deployments.

Serverless stream processing of Debezium data change events with Knative | De...

Come and join us for an (almost) no-slides session around the terrific trio of Debezium, Apache Kafka Streams, and Knative Eventing! Leveraging Apache Kafka as the de-facto standard for event-driven data pipelines, these open-source technologies allow you to ingest data changes from relational and NoSQL databases, process and enrich them, and consume them serverless-style. In a live demo, you’ll see how Debezium, Apache Kafka, Quarkus, and Knative are the dream-team for building serverless, cloud-native stream processing pipelines. You will learn: How to stream change events out of your database using Debezium How to use the Quarkus extension for Kafka Streams to build cloud-native stream processing applications, running either on the JVM or GraalVM How to consume and distribute Kafka messages with Knative Eventing, allowing you to manage modern serverless workloads on Kubernetes.

Docker - Build, Ship and Run Any App, Anywhere Hollywood edition

Brian Christner

Kubernetes: The evolution of distributed systems | DevNation Tech Talk

Kubernetes has evolved to provide capabilities for managing the lifecycle of distributed applications such as deployment, scaling, configuration, and isolation of resources. It addresses needs such as service discovery, networking, bindings to APIs, and state management through controllers, custom resources, and extensions like operators, service meshes, and serverless platforms like Knative. Emerging technologies are exploring hybrid deployments, edge computing, improved state abstractions, and integration across runtimes and clouds.

DevOps: Kubernetes + Helm with Azure

Jessica Deen

This document discusses DevOps concepts like Kubernetes and Helm. It begins with an introduction and agenda. It then provides overviews of Kubernetes and what it is used for. Next, it discusses Helm, including what it is, its architecture, how to install and use it, and how Helm charts are structured. It concludes with information on dependencies, lifecycle management, and configuration management capabilities of Helm.

Democratizing machine learning on kubernetes

Docker, Inc.

One of the largest challenges facing the machine learning community today is understanding how to build a platform to run common open-source machine learning libraries such as Tensorflow. Both Joy and Lachie are both passionate about making machine learning accessible to the masses using Kubernetes. In this session they'll share how to deploy a distributed Tensorflow training cluster complete with GPU scheduling on Kubernetes. We'll also share how distributed Tensorflow training works, various options for distributed training, and when to choose what option. We'll also share some best practices on using distributed Tensorflow on top of Kubernetes, based on our latest performance tests performed on public cloud providers. All work presented in this session will be accessible via a public Github repository.

OSMC 2019 | Monitoring Cockpit for Kubernetes Clusters by Ulrike Klusik

NETWAYS

Monitoring Kubernetes Clusters with Prometheus is state of the art. The difficulty is to find the significant metrics from the vast amount of available metrics. This talk shows a Monitoring Cockpit defined to get a quick overview of the cluster health and usage. It uses the Standard Metrics available for Kubernetes/OpenShift Clusters and their standard services. The monitoring solution is based on Prometheus, using InfluxDB for central long term storage and Grafana.

Monitoring klassisch oder Cloud

1) Monitoring cloud applications presents unique challenges compared to traditional on-premise applications due to the dynamic and scalable nature of cloud infrastructure. 2) Prometheus is an open source monitoring solution suited for containerized and cloud-native applications like Kubernetes due to its ability to dynamically discover targets and collect many metrics. 3) OMD Labs integrates tools like Prometheus and Grafana to monitor complex and dynamically changing infrastructures like OpenShift clusters, providing dashboards that visualize the health and performance of cluster components from nodes to pods and services.

What's hot

A mini adventure in Minikube devopsdays CT 2017

Whitney Tennant

Kubernetes Logging

Denys Havrysh

OSDC 2018 - Distributed monitoring

Gianluca Arbezzano

Five Lessons Learned from Large-scale Implementation of Kubernetes in the Ent...

DevOps.com

Deploy prometheus on kubernetes

Cloud Technology Experts

Kubernetes Helm: Why It Matters

Kubernetes Deployments: A "Hands-off" Approach

Rodrigo Reis

Cost-effective Compute Clusters with Spot and Pre-emptible Instances - KubeCo...

Helm - Application deployment management for Kubernetes

Alexei Ledenev

Top 5 Considerations for Successful Deployment of Kubernetes

Thinking One Step Further with Time-saving DevOps Tools with Open Telekom Clo...

Bitnami

Getting Started with Kafka on k8s

VMware Tanzu

Kubernetes 1.16 and rancher 2.3 enhancements

Saiyam Pathak

Kubernetes fundamentals

Victor Morales

Introduction to container mangement

Martin Marcher

Serverless stream processing of Debezium data change events with Knative | De...

Docker - Build, Ship and Run Any App, Anywhere Hollywood edition

Brian Christner

Kubernetes: The evolution of distributed systems | DevNation Tech Talk

DevOps: Kubernetes + Helm with Azure

Jessica Deen

Democratizing machine learning on kubernetes

Docker, Inc.

What's hot (20)

A mini adventure in Minikube devopsdays CT 2017

Kubernetes Logging

OSDC 2018 - Distributed monitoring

Five Lessons Learned from Large-scale Implementation of Kubernetes in the Ent...

Deploy prometheus on kubernetes

Kubernetes Helm: Why It Matters

Kubernetes Deployments: A "Hands-off" Approach

Cost-effective Compute Clusters with Spot and Pre-emptible Instances - KubeCo...

Helm - Application deployment management for Kubernetes

Top 5 Considerations for Successful Deployment of Kubernetes

Thinking One Step Further with Time-saving DevOps Tools with Open Telekom Clo...

Getting Started with Kafka on k8s

Kubernetes 1.16 and rancher 2.3 enhancements

Kubernetes fundamentals

Introduction to container mangement

Serverless stream processing of Debezium data change events with Knative | De...

Docker - Build, Ship and Run Any App, Anywhere Hollywood edition

Kubernetes: The evolution of distributed systems | DevNation Tech Talk

DevOps: Kubernetes + Helm with Azure

Democratizing machine learning on kubernetes

Similar to Monitoring Cockpit for OpenShift Clusters

OSMC 2019 | Monitoring Cockpit for Kubernetes Clusters by Ulrike Klusik

NETWAYS

Monitoring klassisch oder Cloud