Modern cloud-native applications are incredibly complex systems. Keeping the systems healthy and meeting SLAs for our customers is crucial for long-term success. In this session, we will dive into the three pillars of observability - metrics, logs, tracing - the foundation of successful troubleshooting in distributed systems. You'll learn the gotchas and pitfalls of rolling out the OpenTelemetry stack on Kubernetes to effectively collect all your signals without worrying about a vendor lock in. Additionally we will replace parts of the Prometheus stack to scrape metrics with OpenTelemetry collector and operator.
The monolith to cloud-native, microservices evolution has driven a shift from monitoring to observability. OpenTelemetry, a merger of the OpenTracing and OpenCensus projects, is enabling Observability 2.0. This talk gives an overview of the OpenTelemetry project and then outlines some production-proven architectures for improving the observability of your applications and systems.
The monolith to cloud-native, microservices evolution has driven a shift from monitoring to observability. OpenTelemetry, a merger of the OpenTracing and OpenCensus projects, is enabling Observability 2.0. This talk covers the fundamental concepts of observability and then demonstrates how to instrument your applications using the OpenTelemetry libraries.
Are you interested in dipping your toes in the cloud native observability waters, but as an engineer you are not sure where to get started with tracing problems through your microservices and application landscapes on Kubernetes? Then this is the session for you, where we take you on your first steps in an active open-source project that offers a buffet of languages, challenges, and opportunities for getting started with telemetry data.
The project is called openTelemetry, but before diving into the specifics, we’ll start with de-mystifying key concepts and terms such as observability, telemetry, instrumentation, cardinality, percentile to lay a foundation. After understanding the nuts and bolts of observability and distributed traces, we’ll explore the openTelemetry community; its Special Interest Groups (SIGs), repositories, and how to become not only an end-user, but possibly a contributor.We will wrap up with an overview of the components in this project, such as the Collector, the OpenTelemetry protocol (OTLP), its APIs, and its SDKs.
Attendees will leave with an understanding of key observability concepts, become grounded in distributed tracing terminology, be aware of the components of openTelemetry, and know how to take their first steps to an open-source contribution!
Key Takeaways: Open source, vendor neutral instrumentation is an exciting new reality as the industry standardizes on openTelemetry for observability. OpenTelemetry is on a mission to enable effective observability by making high-quality, portable telemetry ubiquitous. The world of observability and monitoring today has a steep learning curve and in order to achieve ubiquity, the project would benefit from growing our contributor community.
My contribution to the "Grafana & Friends" Meetup.
This presentation goes into the context in the Observability landscape, the basics of OpenTelemetry with its signals and lookout what to expect next.
Everyone wants observability into their system, but find themselves with too many vendors and tools, each with its own API, SDK, agent and collectors.
In this talk I will present OpenTelemetry, an ambitious open source project with the promise of a unified framework for collecting observability data. With OpenTelemetry you could instrument your application in a vendor-agnostic way, and then analyze the telemetry data in your backend tool of choice, whether Prometheus, Jaeger, Zipkin, or others.
I will cover the current state of the various projects of OpenTelemetry (across programming languages, exporters, receivers, protocols), some of which not even GA yet, and provide useful guidance on how to get started with it.
The monolith to cloud-native, microservices evolution has driven a shift from monitoring to observability. OpenTelemetry, a merger of the OpenTracing and OpenCensus projects, is enabling Observability 2.0. This talk gives an overview of the OpenTelemetry project and then outlines some production-proven architectures for improving the observability of your applications and systems.
The monolith to cloud-native, microservices evolution has driven a shift from monitoring to observability. OpenTelemetry, a merger of the OpenTracing and OpenCensus projects, is enabling Observability 2.0. This talk covers the fundamental concepts of observability and then demonstrates how to instrument your applications using the OpenTelemetry libraries.
Are you interested in dipping your toes in the cloud native observability waters, but as an engineer you are not sure where to get started with tracing problems through your microservices and application landscapes on Kubernetes? Then this is the session for you, where we take you on your first steps in an active open-source project that offers a buffet of languages, challenges, and opportunities for getting started with telemetry data.
The project is called openTelemetry, but before diving into the specifics, we’ll start with de-mystifying key concepts and terms such as observability, telemetry, instrumentation, cardinality, percentile to lay a foundation. After understanding the nuts and bolts of observability and distributed traces, we’ll explore the openTelemetry community; its Special Interest Groups (SIGs), repositories, and how to become not only an end-user, but possibly a contributor.We will wrap up with an overview of the components in this project, such as the Collector, the OpenTelemetry protocol (OTLP), its APIs, and its SDKs.
Attendees will leave with an understanding of key observability concepts, become grounded in distributed tracing terminology, be aware of the components of openTelemetry, and know how to take their first steps to an open-source contribution!
Key Takeaways: Open source, vendor neutral instrumentation is an exciting new reality as the industry standardizes on openTelemetry for observability. OpenTelemetry is on a mission to enable effective observability by making high-quality, portable telemetry ubiquitous. The world of observability and monitoring today has a steep learning curve and in order to achieve ubiquity, the project would benefit from growing our contributor community.
My contribution to the "Grafana & Friends" Meetup.
This presentation goes into the context in the Observability landscape, the basics of OpenTelemetry with its signals and lookout what to expect next.
Everyone wants observability into their system, but find themselves with too many vendors and tools, each with its own API, SDK, agent and collectors.
In this talk I will present OpenTelemetry, an ambitious open source project with the promise of a unified framework for collecting observability data. With OpenTelemetry you could instrument your application in a vendor-agnostic way, and then analyze the telemetry data in your backend tool of choice, whether Prometheus, Jaeger, Zipkin, or others.
I will cover the current state of the various projects of OpenTelemetry (across programming languages, exporters, receivers, protocols), some of which not even GA yet, and provide useful guidance on how to get started with it.
Installation of Grafana on linux ; connectivity with Prometheus database , installation of Prometheus ; Installation of node_exporter ,Tomcat-exporter ; installation and configuration of alert manager .. Detailed step by step installation and working
OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdfNETWAYS
Everyone wants observability into their system, but find themselves with too many vendors and tools, each with its own API, SDK, agent, and collectors. In this talk I will present OpenTelemetry, an ambitious open source project with the promise of a unified framework for collecting observability data. With OpenTelemetry you could instrument your application in a vendor-agnostic way, and then analyse the telemetry data in your backend tool of choice, whether Prometheus, Jaeger, Zipkin, or others. I will cover the current state of the various projects of OpenTelemetry (across programming languages, exporters, receivers, protocols), some of which not even GA yet, and provide useful guidance on how to get started with it.
Designing a complete ci cd pipeline using argo events, workflow and cd productsJulian Mazzitelli
https://www.youtube.com/watch?v=YmIAatr3Who
Presented at Cloud and AI DevFest GDG Montreal on September 27, 2019.
Are you looking to get more flexibility out of your CICD platform? Interested how GitOps fits into the mix? Learn how Argo CD, Workflows, and Events can be combined to craft custom CICD flows. All while staying Kubernetes native, enabling you to leverage existing observability tooling.
Observability on Kubernetes - High Availability on Prometheus with Thanos. An introduction to new Thanos (project from CNCF) what uses a sidecar on Prometheus to bring needed features to monitoring with Prometheus. Presentation for Kubernetes Bogota Group.
Observability in Java: Getting Started with OpenTelemetryDevOps.com
Our software is more complex than ever: applications must be reliable, predictable, and easy to use to meet modern expectations. As developers, this means our responsibilities have grown while the things we can control have stayed the same. In order to better understand our systems and create truly modern software, we need observability.
This workshop will walk through what observability means for Java developers and how to achieve it in our systems with the least amount of work using the open source observability project OpenTelemetry.
MeetUp Monitoring with Prometheus and Grafana (September 2018)Lucas Jellema
This presentation introduces the concept of monitoring - focusing on why and how and finally on the tools to use. It introduces Prometheus (metrics gathering, processing, alerting), application instrumentation and Prometheus exporters and finally it introduces Grafana as a common companion for dashboarding, alerting and notifications. This presentations also introduces the handson workshop - for which materials are available from https://github.com/lucasjellema/monitoring-workshop-prometheus-grafana
Presentation in IBM Cloud Meet-up of Toronto
https://www.meetup.com/IBM-Cloud-Toronto/events/253903913/?_xtd=gatlbWFpbF9jbGlja9oAJGU3NmM3ZjdmLWE2NzgtNGVlNC1iNGZiLTBlZGE5ZWM0NDZjOQ
The monolith to cloud-native, microservices evolution has driven a shift from monitoring to observability. OpenTelemetry, a merger of the OpenTracing and OpenCensus projects, is enabling Observability 2.0. This talk covers the latest concepts in observability and then demonstrates how to configure and deploy various OpenTelemetry components to effectively meet your SLO's.
Presented at GDG Devfest Ukraine 2018.
Prometheus has become the defacto monitoring system for cloud native applications, with systems like Kubernetes and Etcd natively exposing Prometheus metrics. In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin. Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.
This is a talk on how you can monitor your microservices architecture using Prometheus and Grafana. This has easy to execute steps to get a local monitoring stack running on your local machine using docker.
OSMC 2023 | What’s new with Grafana Labs’s Open Source Observability stack by...NETWAYS
Open source is at the heart of what we do at Grafana Labs and there is so much happening! The intent of this talk to update everyone on the latest development when it comes to Grafana, Pyroscope, Faro, Loki, Mimir, Tempo and more. Everyone has had at least heard about Grafana but maybe some of the other projects mentioned above are new to you? Welcome to this talk 😉 Beside the update what is new we will also quickly introduce them during this talk.
Combining logs, metrics, and traces for unified observabilityElasticsearch
Learn how Elasticsearch efficiently combines data in a single store and how Kibana is used to analyze it. Plus, see how recent developments help identify, troubleshoot, and resolve operational issues faster.
Everything You wanted to Know About Distributed TracingAmuhinda Hungai
In the age of microservices, understanding how applications are executing in a highly distributed environment can be complicated. Looking at log files only gives a snapshot of the whole story and looking at a single service in isolation simply does not give enough information. Each service is just one side of a bigger story. Distributed tracing has emerged as an invaluable technique that succeeds in summarizing all sides of the story into a shared timeline. Yet deploying it can be quite challenging, especially in the large scale, polyglot environments of modern companies that mix together many different technologies. During this session, we will take a look at patterns and means to implement Tracing for services. After introducing the basic concepts we will cover how the tracing model works, and how to safely use it in production to troubleshoot and diagnose issues.
These are the slides for a talk/workshop delivered to the Cloud Native Wales user group (@CloudNativeWal) on 2019-01-10.
In these slides, we go over some principles of gitops and a hands on session to apply these to manage a microservice.
You can find out more about GitOps online https://www.weave.works/technologies/gitops/
No production system is complete without a way to monitor it. In software, we define observability as the ability to understand how our system is performing. This talk dives into capabilities and tools that are recommended for implementing observability when running K8s in production as the main platform today for deploying and maintaining containers with cloud-native solutions.
We start by introducing the concept of observability in the context of distributed systems such as K8s and the difference with monitoring. We continue by reviewing the observability stack in K8s and the main functionalities. Finally, we will review the tools K8s provides for monitoring and logging, and get metrics from applications and infrastructure.
Between the points to be discussed we can highlight:
-Introducing the concept of observability
-Observability stack in K8s
-Tools and apps for implementing Kubernetes observability
-Integrating Prometheus with OpenMetrics
Installation of Grafana on linux ; connectivity with Prometheus database , installation of Prometheus ; Installation of node_exporter ,Tomcat-exporter ; installation and configuration of alert manager .. Detailed step by step installation and working
OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdfNETWAYS
Everyone wants observability into their system, but find themselves with too many vendors and tools, each with its own API, SDK, agent, and collectors. In this talk I will present OpenTelemetry, an ambitious open source project with the promise of a unified framework for collecting observability data. With OpenTelemetry you could instrument your application in a vendor-agnostic way, and then analyse the telemetry data in your backend tool of choice, whether Prometheus, Jaeger, Zipkin, or others. I will cover the current state of the various projects of OpenTelemetry (across programming languages, exporters, receivers, protocols), some of which not even GA yet, and provide useful guidance on how to get started with it.
Designing a complete ci cd pipeline using argo events, workflow and cd productsJulian Mazzitelli
https://www.youtube.com/watch?v=YmIAatr3Who
Presented at Cloud and AI DevFest GDG Montreal on September 27, 2019.
Are you looking to get more flexibility out of your CICD platform? Interested how GitOps fits into the mix? Learn how Argo CD, Workflows, and Events can be combined to craft custom CICD flows. All while staying Kubernetes native, enabling you to leverage existing observability tooling.
Observability on Kubernetes - High Availability on Prometheus with Thanos. An introduction to new Thanos (project from CNCF) what uses a sidecar on Prometheus to bring needed features to monitoring with Prometheus. Presentation for Kubernetes Bogota Group.
Observability in Java: Getting Started with OpenTelemetryDevOps.com
Our software is more complex than ever: applications must be reliable, predictable, and easy to use to meet modern expectations. As developers, this means our responsibilities have grown while the things we can control have stayed the same. In order to better understand our systems and create truly modern software, we need observability.
This workshop will walk through what observability means for Java developers and how to achieve it in our systems with the least amount of work using the open source observability project OpenTelemetry.
MeetUp Monitoring with Prometheus and Grafana (September 2018)Lucas Jellema
This presentation introduces the concept of monitoring - focusing on why and how and finally on the tools to use. It introduces Prometheus (metrics gathering, processing, alerting), application instrumentation and Prometheus exporters and finally it introduces Grafana as a common companion for dashboarding, alerting and notifications. This presentations also introduces the handson workshop - for which materials are available from https://github.com/lucasjellema/monitoring-workshop-prometheus-grafana
Presentation in IBM Cloud Meet-up of Toronto
https://www.meetup.com/IBM-Cloud-Toronto/events/253903913/?_xtd=gatlbWFpbF9jbGlja9oAJGU3NmM3ZjdmLWE2NzgtNGVlNC1iNGZiLTBlZGE5ZWM0NDZjOQ
The monolith to cloud-native, microservices evolution has driven a shift from monitoring to observability. OpenTelemetry, a merger of the OpenTracing and OpenCensus projects, is enabling Observability 2.0. This talk covers the latest concepts in observability and then demonstrates how to configure and deploy various OpenTelemetry components to effectively meet your SLO's.
Presented at GDG Devfest Ukraine 2018.
Prometheus has become the defacto monitoring system for cloud native applications, with systems like Kubernetes and Etcd natively exposing Prometheus metrics. In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin. Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.
This is a talk on how you can monitor your microservices architecture using Prometheus and Grafana. This has easy to execute steps to get a local monitoring stack running on your local machine using docker.
OSMC 2023 | What’s new with Grafana Labs’s Open Source Observability stack by...NETWAYS
Open source is at the heart of what we do at Grafana Labs and there is so much happening! The intent of this talk to update everyone on the latest development when it comes to Grafana, Pyroscope, Faro, Loki, Mimir, Tempo and more. Everyone has had at least heard about Grafana but maybe some of the other projects mentioned above are new to you? Welcome to this talk 😉 Beside the update what is new we will also quickly introduce them during this talk.
Combining logs, metrics, and traces for unified observabilityElasticsearch
Learn how Elasticsearch efficiently combines data in a single store and how Kibana is used to analyze it. Plus, see how recent developments help identify, troubleshoot, and resolve operational issues faster.
Everything You wanted to Know About Distributed TracingAmuhinda Hungai
In the age of microservices, understanding how applications are executing in a highly distributed environment can be complicated. Looking at log files only gives a snapshot of the whole story and looking at a single service in isolation simply does not give enough information. Each service is just one side of a bigger story. Distributed tracing has emerged as an invaluable technique that succeeds in summarizing all sides of the story into a shared timeline. Yet deploying it can be quite challenging, especially in the large scale, polyglot environments of modern companies that mix together many different technologies. During this session, we will take a look at patterns and means to implement Tracing for services. After introducing the basic concepts we will cover how the tracing model works, and how to safely use it in production to troubleshoot and diagnose issues.
These are the slides for a talk/workshop delivered to the Cloud Native Wales user group (@CloudNativeWal) on 2019-01-10.
In these slides, we go over some principles of gitops and a hands on session to apply these to manage a microservice.
You can find out more about GitOps online https://www.weave.works/technologies/gitops/
No production system is complete without a way to monitor it. In software, we define observability as the ability to understand how our system is performing. This talk dives into capabilities and tools that are recommended for implementing observability when running K8s in production as the main platform today for deploying and maintaining containers with cloud-native solutions.
We start by introducing the concept of observability in the context of distributed systems such as K8s and the difference with monitoring. We continue by reviewing the observability stack in K8s and the main functionalities. Finally, we will review the tools K8s provides for monitoring and logging, and get metrics from applications and infrastructure.
Between the points to be discussed we can highlight:
-Introducing the concept of observability
-Observability stack in K8s
-Tools and apps for implementing Kubernetes observability
-Integrating Prometheus with OpenMetrics
Kubernetes @ Squarespace (SRE Portland Meetup October 2017)Kevin Lynch
In this presentation I talk about our motivation to converting our microservices to run on Kubernetes. I discuss many of the technical challenges we encountered along the way, including networking issues, Java issues, monitoring and alerting, and managing all of our resources!
Kubernetes is a solid leader among different cloud orchestration engines and its adoption rate is growing on a daily basis. Naturally people want to run both their applications and databases on the same infrastructure.
There are a lot of ways to deploy and run PostgreSQL on Kubernetes, but most of them are not cloud-native. Around one year ago Zalando started to run HA setup of PostgreSQL on Kubernetes managed by Patroni. Those experiments were quite successful and produced a Helm chart for Patroni. That chart was useful, albeit a single problem: Patroni depended on Etcd, ZooKeeper or Consul.
Few people look forward to deploy two applications instead of one and support them later on. In this talk I would like to introduce Kubernetes-native Patroni. I will explain how Patroni uses Kubernetes API to run a leader election and store the cluster state. I’m going to live-demo a deployment of HA PostgreSQL cluster on Minikube and share our own experience of running more than 130 clusters on Kubernetes.
Patroni is a Python open-source project developed by Zalando in cooperation with other contributors on GitHub: https://github.com/zalando/patroni
How to do a LIVE-demo with minikube:
1. git clone https://github.com/zalando/patroni
2. cd patroni
3. git checkout feature/demo
4. cd kubernetes
5. open demo.sh and edit line #4 (specify the minikube context )
6. docker build -t patroni .
7. may be docker push patroni
8. may be edit patroni_k8s.yaml line #22 and put the name of patroni image you build there
9. install tmux
10. run tmux in one terminal
11. run bash demo.sh in another terminal and press Enter from time to time
For the Docker users out there, Sematext's DevOps Evangelist, Stefan Thies, goes through a number of different Docker monitoring options, points out their pros and cons, and offers solutions for Docker monitoring. Webinar contains actionable content, diagrams and how-to steps.
Using eBPF to Measure the k8s Cluster HealthScyllaDB
As a k8s cluster-admin your app teams have a certain expectation of your cluster to be available to deploy services at any time without problems. While there is no shortage on metrics in k8s its important to have the right metrics to alert on issues and giving you enough data to react to potential availability issues. Prometheus has become a standard and sheds light on the inner behaviour of Kubernetes clusters and workloads. Lots of KPIs (CPU, IO, network. Etc) in our On-Premise environment are less precise when we start to work in a Cloud environment. Ebpf is the perfect technology that fulfills that requirement as it gives us information down to the kernel level. In 2018 Cloudflare shared an opensource project to expose custom ebpf metrics in Prometheus. Join this session and learn about: • What is ebpf? • What type of metrics we can collect? • How to expose those metrics in a K8s environment. This session will try to deliver a step-by-step guide on how to take advantage of the ebpf exporter.
Speaker: Alexander Kukushkin
Kubernetes is a solid leader among different cloud orchestration engines and its adoption rate is growing on a daily basis. Naturally people want to run both their applications and databases on the same infrastructure.
There are a lot of ways to deploy and run PostgreSQL on Kubernetes, but most of them are not cloud-native. Around one year ago Zalando started to run HA setup of PostgreSQL on Kubernetes managed by Patroni. Those experiments were quite successful and produced a Helm chart for Patroni. That chart was useful, albeit a single problem: Patroni depended on Etcd, ZooKeeper or Consul.
Few people look forward to deploy two applications instead of one and support them later on. In this talk I would like to introduce Kubernetes-native Patroni. I will explain how Patroni uses Kubernetes API to run a leader election and store the cluster state. I’m going to live-demo a deployment of HA PostgreSQL cluster on Minikube and share our own experience of running more than 130 clusters on Kubernetes.
Patroni is a Python open-source project developed by Zalando in cooperation with other contributors on GitHub: https://github.com/zalando/patroni
Kubernetes for java developers - Tutorial at Oracle Code One 2018Anthony Dahanne
You’re a Java developer? Already familiar with Docker? Want to know more about Kubernetes and its ecosystem for developers? During this session, you’ll get familiar with core Kubernetes concepts (pods, deployments, services, volumes, and so on) before seeing the most-popular and most-productive Kubernetes tools in action, with a special focus on Java development. By the end of the session, you’ll have a better understanding of how you can leverage Kubernetes to speed up your Java deployments on-premises or to any cloud.
Functioning incessantly of Data Science Platform with Kubeflow - Albert Lewan...GetInData
Did you like it? Check out our blog to stay up to date: https://getindata.com/blog
The talk is focused on administration, development and monitoring platform with Apache Spark, Apache Flink and Kubeflow in which the monitoring stack is based on Prometheus stack.
Author: Albert Lewandowski
Linkedin: https://www.linkedin.com/in/albert-lewandowski/
___
Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets.
Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries.
https://getindata.com
Sebastien Thomas, System Architect at Coyote Amerique, gave a presentation on operator frameworks. His talk covered how Operator SDK can be used to create Kubernetes Operators with Go.
Kubernetes @ Squarespace: Kubernetes in the DatacenterKevin Lynch
This talk was presented at SRE NYC Meetup on August 16, 2017 at Squarespace HQ.
https://www.youtube.com/watch?v=UJ1QAKprVr4
As the engineering teams at Squarespace grow, we have been building more and more microservices. However, this has added operational strain as we try to shoehorn a growing, complex dynamic environment into our static data center infrastructure. We needed to rethink how we handle deployments, dependency management, resource allocation, monitoring, and alerting. Docker containerization and Kubernetes orchestration helps us tackle many of these problems, but the journey has been challenging. In this talk, we’ll discuss the challenges of running Kubernetes in a datacenter and how we switched to a more SLA-focused alert structure than per instance health with Prometheus and AlertManager.
It's been said that open source software is eating the world. In the observability space, the project making this possible is OpenTelemetry. It's quickly becoming the standard for instrumentation and data collection of observability data. Understanding what data to collect and how to collect it properly is fundamental to ensuring users can quickly address availability and performance issues. Steve Flanders, Director of Engineering at Splunk, discusses the components of the project, its current status, and how you can get started integrating it into your modern app infrastructure.
Speakers:
Steve Flanders
Managing Container Clusters in OpenStack Native WayQiming Teng
This is a presentation from the OpenStack Austin Summit. It talks about managing containers in an OpenStack native way where containers are treated as first class citizens.
20180503 kube con eu kubernetes metrics deep diveBob Cotton
Kubernetes generates a wealth of metrics. Some explicitly within the Kubernetes API server, the Kublet, and cAdvisor or implicitly by observing events such as the kube-state-metrics project. A subset of these metrics are used within Kubernetes itself to make scheduling decisions, however, other metrics can be used to determine the overall health of the system or for capacity planning purposes.
Kubernetes exposes metrics from several places, some available internally, others through add-on projects. In this session you will learn about:
- Node level metrics, as exposed from the node_exporter
- Kublet metrics
- API server metrics
- etcd metrics
- cAdvisor metrics
- Metrics exposed from kube-state-metrics
Monitoring in Big Data Platform - Albert Lewandowski, GetInDataGetInData
Did you like it? Check out our blog to stay up to date: https://getindata.com/blog
The webinar was organized by GetinData on 2020. During the webinar we explaned the concept of monitoring and observability with focus on data analytics platforms.
Watch more here: https://www.youtube.com/watch?v=qSOlEN5XBQc
Whitepaper - Monitoring ang Observability for Data Platform: https://getindata.com/blog/white-paper-big-data-monitoring-observability-data-platform/
Speaker: Albert Lewandowski
Linkedin: https://www.linkedin.com/in/albert-lewandowski/
___
Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets.
Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries.
https://getindata.com
Kubernetes (K8s) is a powerful, flexible and portable open source framework for distributed containerized applications delivery and management. An important part of the services provided by most Kubernetes clusters is the containers’ networking stack. In most cases and for many applications it “just works”, but this seeming simplicity is backed by a complex stack of technologies that provide many capabilities beyond the basics.
This presentation accompanies the meetup and webinar where Oleg Chunikhin, CTO at Kublr, shows how Kubernetes networking stack works, describes main components, interfaces and extensibility options.
What is covered:
- general notions of Kubernetes networking - Pods and Network Policies
- implementation of Kubernetes networking - CNI, CNI plugins, and Linux network namespaces
- some Kubernetes CNI providers: Calico, Weave, Flanel, and Canal
- K8S networking extensibility for advanced and “exotic” use-cases with Multus CNI plugin as an example
Integrating Puppet and Gitolite for sysadmins cooperationsLuca Mazzaferro
In this slides is presented a light solution based on the integration between Puppet-Foreman and Gitolite to the problem: How to enable many sysadmins to work together on one work environment without interfering with each other?
Make Your Containers Faster: Linux Container Performance ToolsKernel TLV
If you look under the hood, Linux containers are just processes with some isolation features and resource quotas sprinkled on top. In this talk, we will apply modern Linux performance tools to container analysis: get high-level resource utilization on running containers with docker stats, htop, and nsenter; dig into high-CPU issues with perf; detect slow filesystem latency with BPF-based tools; and generate flame graphs of interesting event call stacks.
Sasha Goldshtein is the CTO of Sela Group, a Microsoft MVP and Regional Director, Pluralsight and O'Reilly author, and international consultant and trainer. Sasha is the author of two books and multiple online courses, and a prolific blogger. He is also an active open source contributor to projects focused on system diagnostics, performance monitoring, and tracing -- across multiple operating systems and runtimes. Sasha authored and delivered training courses on Linux performance optimization, event tracing, production debugging, mobile application development, and modern C++. Between his consulting engagements, Sasha speaks at international conferences world-wide.
You can find more details on the meetup page - https://www.meetup.com/Tel-Aviv-Yafo-Linux-Kernel-Meetup/events/245319189/
Similar to Exploring the power of OpenTelemetry on Kubernetes (20)
In 2022 we heard your GitOps questions at meetups and gatherings, big stages and local panels and one question was often top of mind: how do I get started? The benefits of GitOps are calling your name, but getting started isn’t that straightforward.
Red Hat is excited to kick off 2023 with a DevNation TechTalk, focused on GitOps to help you sift through your questions. At DevNation you’ll hear from passionate GitOps practitioners about the pitfalls to avoid and hurdles to jump while kicking off or evolving your GitOps practices. This event is aimed at audiences that are new to GitOps or early in their practice development within a cloud native environment.
During this live session you’ll learn:
Upcoming updates and key milestones in the ArgoCD roadmap and how Red Hat will support them
How to simplify the delivery GitOps across multi-cloud environments
GitOps best practices from experts at:
PostNord Strålfors: Filip Jansson
Arbetsförmedlingen: Misho Kmetovski & Richard Hermansson
Swiss Railways (SBB): Manuel Wallrapp & Thomas Bruederli
Plus stick around for an “Ask me Anything” segment to ask any outstanding questions live.
GitHub plays a key role in the everyday work of thousands of developers and is a central piece of the open-source software ecosystem. Even though it is getting better and better every day, it still misses some key features that we need. If you want a better way of reviewing PRs, navigating through the code or better yet - writing the code without leaving the browser - this talk is for you!
This talk will be demo driven, and as the title suggests, we will start with the aesthetic revamp. But we definitely won’t stop there! You will also learn a few cool things about interacting with GitHub through the command line. So not only your UI will be officially revamped, but you will also gain a productivity boost.
Quinoa: A modern Quarkus UI with no hassles | DevNation tech TalkRed Hat Developers
The Quarkus Quinoa extension takes care of all the web UI build/wiring/dev-mode hassles and lets you focus on your web application logic. In this tech talk, we’ll bring a shopping list app to life with Quarkus, Hibernate as a backend, and React as a frontend. Quinoa will be the glue that makes it all work seamlessly from dev to production.
Extra micrometer practices with Quarkus | DevNation Tech TalkRed Hat Developers
How do you know you have improved the performance of your portfolio of applications? By measuring it, of course! The ultimate goal of gathering application metrics is to have a standardized practice that is applicable across multiple microservices and that adds minimal runtime overhead. Join this session to discuss what metrics are available out of the box with the Quarkus micrometer extensions, what (and, more importantly, why) you should customize, and how those pieces of information will impact your development outcomes.
Event-driven autoscaling through KEDA and Knative Integration | DevNation Tec...Red Hat Developers
This talk will teach you how to redesign an event-driven autoscaling architecture for cloud-native microservices by utilizing Apache Kafka, Knative, and KEDA infrastructure. You will also learn how to deploy serverless applications (Quarkus) using a Knative service. Finally, KEDA will enable you to autoscale Knative Eventing components (KafkaSource) through events consumption over standard resources (CPU, memory).
Loom is among the most highly anticipated projects in the Java world. It promises to address concurrency and Java execution model issues by providing virtual threads. Thus, there is no need to write concurrent programs using asynchronous or reactive APIs; it will be possible to use the traditional imperative model and let Loom handle the rest. The JVM will execute the program and leverage non-blocking APIs automatically!
Sounds good, doesn't it? How does it work, though? Are there any hidden costs? What is Loom going to change in modern Java frameworks? We will answer these questions in this talk. Starting with the integration of Loom in Quarkus, we will compare the different approaches we considered, discuss their respective pros and cons, and show how Loom might change the Java world.
Quarkus Renarde 🦊♥: an old-school Web framework with today's touch | DevNatio...Red Hat Developers
Quarkus Renarde 🦊♥ is a new Web framework based on Quarkus. This framework focuses not on microservices but web applications and makes Quarkus even easier to use for web apps: - Endpoints based on convention, even easier than RESTEasy Reactive and JAX-RS - Server-side templating with Qute - Validation with Hibernate Validation - Data with Hibernate ORM or Reactive with Panache - Simple authentication with OpenID Connect or WebAuthn Quarkus Renarde 🦊♥ can deliver all this while still providing the joy of developing with Quarkus, with live reload, continuous testing, the Dev, and more.
Recent changes in one desktop product generated many doubts in developer communities regarding containers. Can we still use or create them? Do we have alternatives to docker? We have some answers! Join us in this session to learn more about some popular docker alternatives. You can create containers without docker, and you can also run and publish them. There's life after docker, and containers are here to stay.
Distributed deployment of microservices across multiple OpenShift clusters | ...Red Hat Developers
Hybrid-cloud and multi-cloud patterns are the next application deployment architectures, and Kubernetes is the de facto container orchestration engine. 50% of production Kubernetes workloads involve some form of microservices applications. How can we manage this inter-cluster application connectivity? Meet Skupper: an open-source project that solves multi-cloud communication for Kubernetes. In this Tech Talk, you will briefly learn about Skupper and watch a live demo of an e-commerce application with 10 microservices spanning three OpenShift clusters running on three different public cloud providers.
DevNation Workshop: Object detection with Red Hat OpenShift Data Science [Mar...Red Hat Developers
In this workshop, you’ll learn an easy way to incorporate data science and AI/ML into an OpenShift development workflow. As an example, you’ll use an object detection model to detect ‘dog(s)’ in an image.
You will:
Use Jupyter Notebooks and TensorFlow to explore a pre-trained object detection model
Serve the model in a REST API as a Flask App
Use Source-to-Image (S2I) to build and deploy the Flask app
Explore Kafka streams from Notebooks
Deploy a Kafka consumer with the same object detection model
You’ll be able to do all of this without having to install anything on your own computer, thanks to Red Hat OpenShift Data Science and Red Hat OpenShift Streams for Apache Kafka.
Note: Beginner data handling and Python skills are required for this workshop.
Dear security, compliance, and auditing: We’re sorry. Love, DevOps | DevNatio...Red Hat Developers
DevOps solved the conflict between development and operations, but other essential aspects of the delivery lifecycle—security, compliance, and audit—were left out. DevSecOps is an excellent reminder that security must be DevOps’d, but compliance and audit are still missing. There’s no need for a new DevSecAuditComplianceOps buzzword; instead, let’s talk about continuous authorization, which applies Zero Trust principles to continuous monitoring. In this tech talk, Bill Bensing will discuss practical ways to start with continuous authorization for the software delivery lifecycle using Ploigos.
11 CLI tools every developer should know | DevNation Tech TalkRed Hat Developers
What's your favorite IDE? VS Code? IDEA? Eclipse? Visual Studio? The right IDE is fundamental to your productivity as a developer, but you might need something else to become more outstanding. Why don't we take a look at your terminal? Come to this session to learn eleven CLI tools that will boost your developer productivity.
A Microservices approach with Cassandra and Quarkus | DevNation Tech TalkRed Hat Developers
We will dissect the world famous todo app that provides a REST API (which is the foundation of microservices) with data backed by Apache Cassandra. We will leverage the TODO MVC and the TODO backend projects with the back end that we will build with Quarkus and Cassandra. Attendees will get an overview of Cassandra, including the driver for Quarkus. Through live coding (that attendees can try out later) in a cloud-based environment, primarily in Quarkus and Cassandra, attendees will understand how to implement and connect the APIs to the backend and leverage the generic client(s)provided. After attending this session attendees will walk away with a good understanding of implementing microservices using Cassandra and Quarkus. They will also get a working knowledge of how Astra (Cassandra as a service) can be leveraged in other solutions.
GitHub Actions and OpenShift: Supercharging your software development loops...Red Hat Developers
Every software developer wants more productivity. What if the only commands you needed to deploy were "git commit" and "git push"? Join us as we walk you through a live demonstration of how you can ship your lovely application code from your local machine to a free OpenShift cluster, fully automated through GitHub Actions. By the end of this session, you'll have a sound understanding of building a GitHub Action workflow for your codebase that leverages OpenShift to deploy your application.
To the moon and beyond with Java 17 APIs! | DevNation Tech TalkRed Hat Developers
Since moving to a 6 monthly release cadence, the Java platform is evolving more dynamically than ever before. It can be quite a challenge to stay on top of all the changes and new features. In this talk we're going to explore the most important developments in the Java API: which classes have been added, and what has been removed? Join Duke, the Java mascot, for a trip to space and learn which exciting new APIs provided by the Java platform can help him with his journey:
The Java Vector API for utilizing the SIMD capabilities of modern CPU architectures
The Foreign Linker API for integrating with native code
The JFR Event Streaming API for publishing JDK Flight Recorder Events
We'll also take a look at some useful changes to the Java runtime, such as CDS archives for a faster spaceship..., uhm, application launch!
Profile your Java apps in production on Red Hat OpenShift with Cryostat | Dev...Red Hat Developers
Did you know that OpenJDK comes with Java Flight Recorder (JFR), an embedded production time profiler? Cryostat provides easy and secure access to JFR across container boundaries so you can profile that performance bottleneck, or find that annoying bug. Join this session to learn about using Cryostat to profile Java applications in production on OpenShift.
Kafka at the Edge: an IoT scenario with OpenShift Streams for Apache Kafka | ...Red Hat Developers
Apache Kafka is taking the world by storm and is rapidly becoming the de-facto event bus for event-driven and streaming applications that respond to events and data in real time. OpenShift Streams for Apache Kafka is Red Hat's fully hosted and managed Apache Kafka service targeting development teams that want to incorporate streaming data and scalable messaging in their applications, without the burden of setting up and maintaining a Kafka cluster infrastructure.
In this session you will discover how Apache Kafka can be used in an IoT scenario to ingest data from devices and make them available in real-time to other applications.
More specifically you will learn how to:
Simulate devices that send MQTT messages to a MQTT broker
Use Apache Camel and Camel-K to bridge MQTT with Apache Kafka
Use Kafka Streams in a Quarkus application to process the device messages
Query the state of the devices using GraphQ
Kubernetes configuration and security policies with KubeLinter | DevNation Te...Red Hat Developers
With Kubernetes, implementing security policies can be challenging. First, developers, administrators, and security teams need to understand security policies in collaboration to have the best chance of successful adoption. Next, policy enforcement needs to integrate with developer workflows. Lastly, policies need to contain corrective action that is as close to the developer as possible. KubeLinter solves these problems by linting Kubernetes YAML files and Helm charts at the source: the developer.
In this session, we will evaluate KubeLinter by moving through a hands-on demo of the application, showing a use case for local machines and CI pipeline integration, and chatting about how best to integrate it into your organization:
KubeLinter, and its default checks
How you can leverage the application in your day-to-day operations
The open source StackRox community
Level-up your gaming telemetry using Kafka Streams | DevNation Tech TalkRed Hat Developers
Many modern video games are constantly evolving post-release. New maps, game modes, and game balancing adjustments are rolled out, often on a weekly basis. This continuous iteration to improve player engagement and satisfaction requires data-driven decision making based on events and telemetry captured during gameplay, and from community forums and discussions.
In this session you will learn how OpenShift Streams for Apache Kafka and Kafka Streams can be used to analyze real-time events and telemetry reported by a game server, using a practical example that encourages audience participation. Specifically you’ll learn how to:
Provision Kafka clusters on OpenShift Streams for Apache Kafka.
Develop a Java application that uses Kafka Streams and Quarkus to process event data.
Deploy the application locally, or on OpenShift and connect it to your OpenShift Streams for Apache Kafka Cluster.
Friends don't let friends do dual writes: Outbox pattern with OpenShift Strea...Red Hat Developers
Dual writes are a common source of issues in distributed event-driven applications. A dual write occurs when an application has to change data in two different systems - for instance, when an application needs to persist data in the database and send a Kafka message to notify other systems. If one of these two operations fail, you might end up with inconsistent data which can be hard to detect and fix.
OpenShift Streams for Apache Kafka is Red Hat's fully hosted and managed Apache Kafka service targeting development teams that want to incorporate streaming data and scalable messaging in their applications, without the burden of setting up and maintaining a Kafka cluster infrastructure. Debezium is an open source distributed platform for change data capture. Built on top of Apache Kafka, it allows applications to react to inserts, updates, and deletes in your databases.
In this session you will learn how you can leverage OpenShift Streams for Apache Kafka and Debezium to avoid the dual write issue in an event-driven application using the outbox pattern. More specifically, we will show you how to:
Provision a Kafka cluster on OpenShift Streams for Apache Kafka.
Deploy and configure Debezium to use OpenShift Streams for Apache Kafka.
Refactor an application to leverage Debezium and OpenShift Streams for Apache Kafka to avoid the dual write problem.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
39. 39
● For metrics and traces OpenTelemetry takes the approach of a
clean-sheet design, specifies a new API and provides full
implementations of this API in multiple languages.
● Our approach with logs is somewhat different. For OpenTelemetry to be
successful in logging space we need to support existing legacy of logs
and logging libraries, while offering improvements and better integration
with the rest of observability world where possible.
OpenTelemetry logs
40. 40
OpenTelemetry Operator - collecting logs
● File log receiver
○ Available in the “contrib” docker image
○ Example config - opentelemetry-collector-contrib/otel-collector-config.yml
○ Includes: /var/log/pods/*/*/*.log
○ Set of operators to parse data: json_parser, regex_parser, move…
○ Collectors resource attributes: namespace, pod name/UID/restarts, container name
● Fluentforwardreceiver - receive data from
46. ● OpenTelemetry Operator
○ Auto-instrumentation for Webservers
● Profiling vision see 0212-profiling-vision.md
● OpAMP: Open Agent Management Protocol see open-telemetry/opamp-spec
46
What is next in OpenTelemetry?