Client Side Monitoring With Prometheus
•
5 likes•13,297 views
Prometheus is predominantly used for monitoring backend services. In this talk I present a technique for monitoring client-side rich client web apps with Prometheus. Presented at KubeCon Berlin 2017.
Report
Share
Report
Share
Download to read offline
Recommended
Kafka tiered-storage-meetup-2022-final-presented
Kafka Tiered Storage separates compute and data storage in two independently scalable layers. Uber's Kafka Improvement Proposal (KIP) #405 describes two-tiered storage, which is a major step towards cloud-native Kafka. It stores the most recent data locally and offloads older data to a remote storage service. Operationally, the benefit is faster routine cluster maintenance activities. In Linkedin, Kafka tiered storage is strongly desired to reduce the cost of running Kafka in the Azure cloud environment. As KIP-405 does not dictate the implementation of remote storage substrate, Linkedin's choice for tiering Kafka in Azure deployments is the Azure Blob Service. This presentation will begin with the motivation behind Linkedin efforts to adopt Kafka Tiered Storage. Next, the architecture of KIP-405 will be discussed. Finally, the Remote Storage Manager for Azure Blobs, which is a work-in-progress, will be presented.
Video: https://youtu.be/V5gaBE5CMwg?t=1387
Advanced Streaming Analytics with Apache Flink and Apache Kafka, Stephan Ewen
Flink and Kafka are popular components to build an open source stream processing infrastructure. We present how Flink integrates with Kafka to provide a platform with a unique feature set that matches the challenging requirements of advanced stream processing applications. In particular, we will dive into the following points:
Flink’s support for event-time processing, how it handles out-of-order streams, and how it can perform analytics on historical and real-time streams served from Kafka’s persistent log using the same code. We present Flink’s windowing mechanism that supports time-, count- and session- based windows, and intermixing event and processing time semantics in one program.
How Flink’s checkpointing mechanism integrates with Kafka for fault-tolerance, for consistent stateful applications with exactly-once semantics.
We will discuss “”Savepoints””, which allows users to save the state of the streaming program at any point in time. Together with a durable event log like Kafka, savepoints allow users to pause/resume streaming programs, go back to prior states, or switch to different versions of the program, while preserving exactly-once semantics.
We explain the techniques behind the combination of low-latency and high throughput streaming, and how latency/throughput trade-off can configured.
We will give an outlook on current developments for streaming analytics, such as streaming SQL and complex event processing.
Producer Performance Tuning for Apache Kafka
Kafka is well known for high throughput ingestion. However, to get the best latency characteristics without compromising on throughput and durability, we need to tune Kafka. In this talk, we share our experiences to achieve the optimal combination of latency, throughput and durability for different scenarios.
Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...
Did you like it? Check out our E-book: Apache NiFi - A Complete Guide
https://ebook.getindata.com/apache-nifi-complete-guide
Apache NiFi is one of the most popular services for running ETL pipelines otherwise it’s not the youngest technology. During the talk, there are described all details about migrating pipelines from the old Hadoop platform to the Kubernetes, managing everything as the code, monitoring all corner cases of NiFi and making it a robust solution that is user-friendly even for non-programmers.
Author: Albert Lewandowski
Linkedin: https://www.linkedin.com/in/albert-lewandowski/
___
Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets.
Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries.
https://getindata.com
Bootstrapping state in Apache Flink
Apache Flink is a popular stream computing framework for real-time stream computing. Many stream compute algorithms require trailing data in order to compute the intended result. One example is computing the number of user logins in the last 7 days. This creates a dilemma where the results of the stream program are incomplete until the runtime of the program exceeds 7 days. The alternative is to bootstrap the program using historic data to seed the state before shifting to use real-time data.
This talk will discuss alternatives to bootstrap programs in Flink. Some alternatives rely on technologies exogenous to the stream program, such as enhancements to the pub/sub layer, that are more generally applicable to other stream compute engines. Other alternatives include enhancements to Flink source implementations. Lyft is exploring another alternative using orchestration of multiple Flink programs. The talk will cover why Lyft pursued this alternative and future directions to further enhance bootstrapping support in Flink.
Speaker
Gregory Fee, Principal Engineer, Lyft
VictoriaLogs: Open Source Log Management System - Preview
VictoriaLogs Preview - Aliaksandr Valialkin
* Existing open source log management systems
- ELK (ElasticSearch) stack: Pros & Cons
- Grafana Loki: Pros & Cons
* What is VictoriaLogs
- Open source log management system from VictoriaMetrics
- Easy to setup and operate
- Scales vertically and horizontally
- Optimized for low resource usage (CPU, RAM, disk space)
- Accepts data from Logstash and Fluentbit in Elasticsearch format
- Accepts data from Promtail in Loki format
- Supports stream concept from Loki
- Provides easy to use yet powerful query language - LogsQL
* LogsQL Examples
- Search by time
- Full-text search
- Combining search queries
- Searching arbitrary labels
* Log Streams
- What is a log stream?
- LogsQL examples: querying log streams
- Stream labels vs log labels
* LogsQL: stats over access logs
* VictoriaLogs: CLI Integration
* VictoriaLogs Recap
Evening out the uneven: dealing with skew in Flink
Flink Forward San Francisco 2022.
When running Flink jobs, skew is a common problem that results in wasted resources and limited scalability. In the past years, we have helped our customers and users solve various skew-related issues in their Flink jobs or clusters. In this talk, we will present the different types of skew that users often run into: data skew, key skew, event time skew, state skew, and scheduling skew, and discuss solutions for each of them. We hope this will serve as a guideline to help you reduce skew in your Flink environment.
by
Jun Qin & Karl Friedrich
Recommended
Kafka tiered-storage-meetup-2022-final-presented
Kafka Tiered Storage separates compute and data storage in two independently scalable layers. Uber's Kafka Improvement Proposal (KIP) #405 describes two-tiered storage, which is a major step towards cloud-native Kafka. It stores the most recent data locally and offloads older data to a remote storage service. Operationally, the benefit is faster routine cluster maintenance activities. In Linkedin, Kafka tiered storage is strongly desired to reduce the cost of running Kafka in the Azure cloud environment. As KIP-405 does not dictate the implementation of remote storage substrate, Linkedin's choice for tiering Kafka in Azure deployments is the Azure Blob Service. This presentation will begin with the motivation behind Linkedin efforts to adopt Kafka Tiered Storage. Next, the architecture of KIP-405 will be discussed. Finally, the Remote Storage Manager for Azure Blobs, which is a work-in-progress, will be presented.
Video: https://youtu.be/V5gaBE5CMwg?t=1387
Advanced Streaming Analytics with Apache Flink and Apache Kafka, Stephan Ewen
Flink and Kafka are popular components to build an open source stream processing infrastructure. We present how Flink integrates with Kafka to provide a platform with a unique feature set that matches the challenging requirements of advanced stream processing applications. In particular, we will dive into the following points:
Flink’s support for event-time processing, how it handles out-of-order streams, and how it can perform analytics on historical and real-time streams served from Kafka’s persistent log using the same code. We present Flink’s windowing mechanism that supports time-, count- and session- based windows, and intermixing event and processing time semantics in one program.
How Flink’s checkpointing mechanism integrates with Kafka for fault-tolerance, for consistent stateful applications with exactly-once semantics.
We will discuss “”Savepoints””, which allows users to save the state of the streaming program at any point in time. Together with a durable event log like Kafka, savepoints allow users to pause/resume streaming programs, go back to prior states, or switch to different versions of the program, while preserving exactly-once semantics.
We explain the techniques behind the combination of low-latency and high throughput streaming, and how latency/throughput trade-off can configured.
We will give an outlook on current developments for streaming analytics, such as streaming SQL and complex event processing.
Producer Performance Tuning for Apache Kafka
Kafka is well known for high throughput ingestion. However, to get the best latency characteristics without compromising on throughput and durability, we need to tune Kafka. In this talk, we share our experiences to achieve the optimal combination of latency, throughput and durability for different scenarios.
Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...
Did you like it? Check out our E-book: Apache NiFi - A Complete Guide
https://ebook.getindata.com/apache-nifi-complete-guide
Apache NiFi is one of the most popular services for running ETL pipelines otherwise it’s not the youngest technology. During the talk, there are described all details about migrating pipelines from the old Hadoop platform to the Kubernetes, managing everything as the code, monitoring all corner cases of NiFi and making it a robust solution that is user-friendly even for non-programmers.
Author: Albert Lewandowski
Linkedin: https://www.linkedin.com/in/albert-lewandowski/
___
Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets.
Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries.
https://getindata.com
Bootstrapping state in Apache Flink
Apache Flink is a popular stream computing framework for real-time stream computing. Many stream compute algorithms require trailing data in order to compute the intended result. One example is computing the number of user logins in the last 7 days. This creates a dilemma where the results of the stream program are incomplete until the runtime of the program exceeds 7 days. The alternative is to bootstrap the program using historic data to seed the state before shifting to use real-time data.
This talk will discuss alternatives to bootstrap programs in Flink. Some alternatives rely on technologies exogenous to the stream program, such as enhancements to the pub/sub layer, that are more generally applicable to other stream compute engines. Other alternatives include enhancements to Flink source implementations. Lyft is exploring another alternative using orchestration of multiple Flink programs. The talk will cover why Lyft pursued this alternative and future directions to further enhance bootstrapping support in Flink.
Speaker
Gregory Fee, Principal Engineer, Lyft
VictoriaLogs: Open Source Log Management System - Preview
VictoriaLogs Preview - Aliaksandr Valialkin
* Existing open source log management systems
- ELK (ElasticSearch) stack: Pros & Cons
- Grafana Loki: Pros & Cons
* What is VictoriaLogs
- Open source log management system from VictoriaMetrics
- Easy to setup and operate
- Scales vertically and horizontally
- Optimized for low resource usage (CPU, RAM, disk space)
- Accepts data from Logstash and Fluentbit in Elasticsearch format
- Accepts data from Promtail in Loki format
- Supports stream concept from Loki
- Provides easy to use yet powerful query language - LogsQL
* LogsQL Examples
- Search by time
- Full-text search
- Combining search queries
- Searching arbitrary labels
* Log Streams
- What is a log stream?
- LogsQL examples: querying log streams
- Stream labels vs log labels
* LogsQL: stats over access logs
* VictoriaLogs: CLI Integration
* VictoriaLogs Recap
Evening out the uneven: dealing with skew in Flink
Flink Forward San Francisco 2022.
When running Flink jobs, skew is a common problem that results in wasted resources and limited scalability. In the past years, we have helped our customers and users solve various skew-related issues in their Flink jobs or clusters. In this talk, we will present the different types of skew that users often run into: data skew, key skew, event time skew, state skew, and scheduling skew, and discuss solutions for each of them. We hope this will serve as a guideline to help you reduce skew in your Flink environment.
by
Jun Qin & Karl Friedrich
Where is my bottleneck? Performance troubleshooting in Flink
Flinkn Forward San Francisco 2022.
In this talk, we will cover various topics around performance issues that can arise when running a Flink job and how to troubleshoot them. We’ll start with the basics, like understanding what the job is doing and what backpressure is. Next, we will see how to identify bottlenecks and which tools or metrics can be helpful in the process. Finally, we will also discuss potential performance issues during the checkpointing or recovery process, as well as and some tips and Flink features that can speed up checkpointing and recovery times.
by
Piotr Nowojski
Introduction to Prometheus
A general introduction and demo of the prometheus monitoring solution and ecosystem with a live demo, given at FLOSSUK 2018.
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
Flink Forward San Francisco 2022.
At Stripe we have created a complete end to end exactly-once processing pipeline to process financial data at scale, by combining the exactly-once power from Flink, Kafka, and Pinot together. The pipeline provides exactly-once guarantee, end-to-end latency within a minute, deduplication against hundreds of billions of keys, and sub-second query latency against the whole dataset with trillion level rows. In this session we will discuss the technical challenges of designing, optimizing, and operating the whole pipeline, including Flink, Kafka, and Pinot. We will also share our lessons learned and the benefits gained from exactly-once processing.
by
Xiang Zhang & Pratyush Sharma & Xiaoman Dong
Tuning Apache Kafka Connectors for Flink.pptx
Flink Forward San Francisco 2022.
In normal situations, the default Kafka consumer and producer configuration options work well. But we all know life is not all roses and rainbows and in this session we’ll explore a few knobs that can save the day in atypical scenarios. First, we'll take a detailed look at the parameters available when reading from Kafka. We’ll inspect the params helping us to spot quickly an application lock or crash, the ones that can significantly improve the performance and the ones to touch with gloves since they could cause more harm than benefit. Moreover we’ll explore the partitioning options and discuss when diverging from the default strategy is needed. Next, we’ll discuss the Kafka Sink. After browsing the available options we'll then dive deep into understanding how to approach use cases like sinking enormous records, managing spikes, and handling small but frequent updates.. If you want to understand how to make your application survive when the sky is dark, this session is for you!
by
Olena Babenko
Parallelization of Structured Streaming Jobs Using Delta Lake
We’ll tackle the problem of running streaming jobs from another perspective using Databricks Delta Lake, while examining some of the current issues that we faced at Tubi while running regular structured streaming. A quick overview on why we transitioned from parquet data files to delta and the problems it solved for us in running our streaming jobs.
Introducing the Apache Flink Kubernetes Operator
Flink Forward San Francisco 2022.
The Apache Flink Kubernetes Operator provides a consistent approach to manage Flink applications automatically, without any human interaction, by extending the Kubernetes API. Given the increasing adoption of Kubernetes based Flink deployments the community has been working on a Kubernetes native solution as part of Flink that can benefit from the rich experience of community members and ultimately make Flink easier to adopt. In this talk we give a technical introduction to the Flink Kubernetes Operator and demonstrate the core features and use-cases through in-depth examples."
by
Thomas Weise
A Deep Dive into Kafka Controller
Presentation at Strata Data Conference 2018, New York
The controller is the brain of Apache Kafka. A big part of what the controller does is to maintain the consistency of the replicas and determine which replica can be used to serve the clients, especially during individual broker failure.
Jun Rao outlines the main data flow in the controller—in particular, when a broker fails, how the controller automatically promotes another replica as the leader to serve the clients, and when a broker is started, how the controller resumes the replication pipeline in the restarted broker.
Jun then describes recent improvements to the controller that allow it to handle certain edge cases correctly and increase its performance, which allows for more partitions in a Kafka cluster.
Cloud Monitoring tool Grafana
Installation of Grafana on linux ; connectivity with Prometheus database , installation of Prometheus ; Installation of node_exporter ,Tomcat-exporter ; installation and configuration of alert manager .. Detailed step by step installation and working
Virtual Flink Forward 2020: Autoscaling Flink at Netflix - Timothy Farkas
Keystone Data Pipeline manages several thousand Flink pipelines, with variable workloads. These pipelines are simple routers which consume from Kafka and write to one of three sinks. In order to alleviate our operational overhead, we’ve implemented autoscaling for our routers. Autoscaling has reduced our resource usage by 25% - 45% (varying by region and time), and has reduced our on call burden. This talk will take an in depth look at the mathematics, algorithms, and infrastructure details for implementing autoscaling of simple pipelines at scale. It will also discuss future work for autoscaling complex pipelines.
PromQL Deep Dive - The Prometheus Query Language
- What is PromQL
- PromQL operators
- PromQL functions
- Hands on: Building queries in PromQL
- Hands on: Visualizing PromQL in Grafana
- Prometheus alerts in PromQL
- Hands on: Creating an alert in Prometheus with PromQL
Building a fully managed stream processing platform on Flink at scale for Lin...
Apache Flink is a distributed stream processing framework that allows users to process and analyze data in real-time. At LinkedIn, we developed a fully managed stream processing platform on Flink running on K8s to power hundreds of stream processing pipelines in production. This platform is the backbone for other infra systems like Search, Espresso (internal document store) and feature management etc. We provide a rich authoring and testing environment which allows users to create, test, and deploy their streaming jobs in a self-serve fashion within minutes. Users can focus on their business logic, leaving the Flink platform to take care of management aspects such as split deployment, resource provisioning, auto-scaling, job monitoring, alerting, failure recovery and much more. In this talk, we will introduce the overall platform architecture, highlight the unique value propositions that it brings to stream processing at LinkedIn and share the experiences and lessons we have learned.
Envoy and Kafka
Full recorded presentation at https://www.youtube.com/watch?v=2UfAgCSKPZo for Tetrate Tech Talks on 2022/05/13.
Envoy's support for Kafka protocol, in form of broker-filter and mesh-filter.
Contents:
- overview of Kafka (usecases, partitioning, producer/consumer, protocol);
- proxying Kafka (non-Envoy specific);
- proxying Kafka with Envoy;
- handling Kafka protocol in Envoy;
- Kafka-broker-filter for per-connection proxying;
- Kafka-mesh-filter to provide front proxy for multiple Kafka clusters.
References:
- https://adam-kotwasinski.medium.com/deploying-envoy-and-kafka-8aa7513ec0a0
- https://adam-kotwasinski.medium.com/kafka-mesh-filter-in-envoy-a70b3aefcdef
The Rise of ZStandard: Apache Spark/Parquet/ORC/Avro
Zstandard is a fast compression algorithm which you can use in Apache Spark in various way. In this talk, I briefly summarized the evolution history of Apache Spark in this area and four main use cases and the benefits and the next steps:
1) ZStandard can optimize Spark local disk IO by compressing shuffle files significantly. This is very useful in K8s environments. It’s beneficial not only when you use `emptyDir` with `memory` medium, but also it maximizes OS cache benefit when you use shared SSDs or container local storage. In Spark 3.2, SPARK-34390 takes advantage of ZStandard buffer pool feature and its performance gain is impressive, too.
2) Event log compression is another area to save your storage cost on the cloud storage like S3 and to improve the usability. SPARK-34503 officially switched the default event log compression codec from LZ4 to Zstandard.
3) Zstandard data file compression can give you more benefits when you use ORC/Parquet files as your input and output. Apache ORC 1.6 supports Zstandardalready and Apache Spark enables it via SPARK-33978. The upcoming Parquet 1.12 will support Zstandard compression.
4) Last, but not least, since Apache Spark 3.0, Zstandard is used to serialize/deserialize MapStatus data instead of Gzip.
There are more community works to utilize Zstandard to improve Spark. For example, Apache Avro community also supports Zstandard and SPARK-34479 aims to support Zstandard in Spark’s avro file format in Spark 3.2.0.
Disaster Recovery Options Running Apache Kafka in Kubernetes with Rema Subra...
Active-Active, Active-Passive, and stretch clusters are hallmark patterns that have been the gold standard in Apache Kafka® disaster recovery architectures for years. Moving to Kubernetes requires unpacking these patterns and choosing a configuration that allows you to meet the same RTO and RPO requirements.
In this talk, we will cover how Active-Active/Active-Passive modes for disaster recovery have worked in the past and how the architecture evolves with deploying Apache Kafka on Kubernetes. We'll also look at how stretch clusters sitting on this architecture give a disaster recovery solution that's built-in!
Armed with this information, you will be able to architect your new Apache Kafka Kubernetes deployment (or retool your existing one) to achieve the resilience you require.
Advanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 Keynote
In this talk, Till Rohrmann and Addison Higham discuss how Flink allows for ambitious stream processing workflows and how Pulsar and Flink enable new capabilities that push forward the state-of-the-art in streaming. They will also share upcoming features and new capabilities in the integrations between Flink and Pulsar and how these two communities are working together to truly advance the power of stream processing.
Apache Flink in the Cloud-Native Era
Flink Forward San Francisco 2022.
This talk will take you on the long journey of Apache Flink into the cloud-native era. It started all the way from where Hadoop and YARN were the standard way of deploying and operating data applications.
We're going to deep dive into the cloud-native set of principles and how they map to the Apache Flink internals and recent improvements. We'll cover fast checkpointing, fault tolerance, resource elasticity, minimal infrastructure dependencies, industry-standard tooling, ease of deployment and declarative APIs.
After this talk you'll get a broader understanding of the operational requirements for a modern streaming application and where the current limits are.
by
David Moravek
The Current State of Table API in 2022
Flink Forward San Francisco 2022.
The Table API is one of the most actively developed components of Flink in recent time. Inspired by databases and SQL, it encapsulates concepts many developers are familiar with. It can be used with both bounded and unbounded streams in a unified way. But from afar it can be difficult to keep track of what this API is capable of and how it relates to Flink's other APIs. In this talk, we will explore the current state of Table API. We will show how it can be used as a batch processor, a changelog processor, or a streaming ETL tool with many built-in functions and operators for deduplicating, joining, and aggregating data. By comparing it to the DataStream API we will highlight differences and elaborate on when to use which API. We will demonstrate hybrid pipelines in which both APIs interact with one another and contribute their unique strengths. Finally, we will take a look at some of the most recent additions as a first step to stateful upgrades.
by
David Andreson
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
Flink Forward San Francisco 2022.
Probably everyone who has written stateful Apache Flink applications has used one of the fault-tolerant keyed state primitives ValueState, ListState, and MapState. With RocksDB, however, retrieving and updating items comes at an increased cost that you should be aware of. Sometimes, these may not be avoidable with the current API, e.g., for efficient event-time stream-sorting or streaming joins where you need to iterate one or two buffered streams in the right order. With FLIP-220, we are introducing a new state primitive: BinarySortedMultiMapState. This new form of state offers you to (a) efficiently store lists of values for a user-provided key, and (b) iterate keyed state in a well-defined sort order. Both features can be backed efficiently by RocksDB with a 2x performance improvement over the current workarounds. This talk will go into the details of the new API and its implementation, present how to use it in your application, and talk about the process of getting it into Flink.
by
Nico Kruber
Delve into Helm - Advanced DevOps
KubeCon EU 2017 Berlin
Helm is not just for simple applications running in simple environments. In this talk, we delve into the depths of Helm, focusing on lifecycle management and continuous delivery (CI/CD) of Kubernetes-native applications in different environments. We show how to extend Helm’s capabilities with plugins and add-ons. We'll also see how sophisticated charts like OpenStack and Deis Workflow use these capabilities to model more complex deployments.
Loki: An Opensource Zipkin/Prometheus Mashup written in Go.
Loki is a prototype OpenTracing implementation written in Go thats takes the Prometheus service-discovery and pull based approach to distributed tracing.
More Related Content
What's hot
Where is my bottleneck? Performance troubleshooting in Flink
Flinkn Forward San Francisco 2022.
In this talk, we will cover various topics around performance issues that can arise when running a Flink job and how to troubleshoot them. We’ll start with the basics, like understanding what the job is doing and what backpressure is. Next, we will see how to identify bottlenecks and which tools or metrics can be helpful in the process. Finally, we will also discuss potential performance issues during the checkpointing or recovery process, as well as and some tips and Flink features that can speed up checkpointing and recovery times.
by
Piotr Nowojski
Introduction to Prometheus
A general introduction and demo of the prometheus monitoring solution and ecosystem with a live demo, given at FLOSSUK 2018.
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
Flink Forward San Francisco 2022.
At Stripe we have created a complete end to end exactly-once processing pipeline to process financial data at scale, by combining the exactly-once power from Flink, Kafka, and Pinot together. The pipeline provides exactly-once guarantee, end-to-end latency within a minute, deduplication against hundreds of billions of keys, and sub-second query latency against the whole dataset with trillion level rows. In this session we will discuss the technical challenges of designing, optimizing, and operating the whole pipeline, including Flink, Kafka, and Pinot. We will also share our lessons learned and the benefits gained from exactly-once processing.
by
Xiang Zhang & Pratyush Sharma & Xiaoman Dong
Tuning Apache Kafka Connectors for Flink.pptx
Flink Forward San Francisco 2022.
In normal situations, the default Kafka consumer and producer configuration options work well. But we all know life is not all roses and rainbows and in this session we’ll explore a few knobs that can save the day in atypical scenarios. First, we'll take a detailed look at the parameters available when reading from Kafka. We’ll inspect the params helping us to spot quickly an application lock or crash, the ones that can significantly improve the performance and the ones to touch with gloves since they could cause more harm than benefit. Moreover we’ll explore the partitioning options and discuss when diverging from the default strategy is needed. Next, we’ll discuss the Kafka Sink. After browsing the available options we'll then dive deep into understanding how to approach use cases like sinking enormous records, managing spikes, and handling small but frequent updates.. If you want to understand how to make your application survive when the sky is dark, this session is for you!
by
Olena Babenko
Parallelization of Structured Streaming Jobs Using Delta Lake
We’ll tackle the problem of running streaming jobs from another perspective using Databricks Delta Lake, while examining some of the current issues that we faced at Tubi while running regular structured streaming. A quick overview on why we transitioned from parquet data files to delta and the problems it solved for us in running our streaming jobs.
Introducing the Apache Flink Kubernetes Operator
Flink Forward San Francisco 2022.
The Apache Flink Kubernetes Operator provides a consistent approach to manage Flink applications automatically, without any human interaction, by extending the Kubernetes API. Given the increasing adoption of Kubernetes based Flink deployments the community has been working on a Kubernetes native solution as part of Flink that can benefit from the rich experience of community members and ultimately make Flink easier to adopt. In this talk we give a technical introduction to the Flink Kubernetes Operator and demonstrate the core features and use-cases through in-depth examples."
by
Thomas Weise
A Deep Dive into Kafka Controller
Presentation at Strata Data Conference 2018, New York
The controller is the brain of Apache Kafka. A big part of what the controller does is to maintain the consistency of the replicas and determine which replica can be used to serve the clients, especially during individual broker failure.
Jun Rao outlines the main data flow in the controller—in particular, when a broker fails, how the controller automatically promotes another replica as the leader to serve the clients, and when a broker is started, how the controller resumes the replication pipeline in the restarted broker.
Jun then describes recent improvements to the controller that allow it to handle certain edge cases correctly and increase its performance, which allows for more partitions in a Kafka cluster.
Cloud Monitoring tool Grafana
Installation of Grafana on linux ; connectivity with Prometheus database , installation of Prometheus ; Installation of node_exporter ,Tomcat-exporter ; installation and configuration of alert manager .. Detailed step by step installation and working
Virtual Flink Forward 2020: Autoscaling Flink at Netflix - Timothy Farkas
Keystone Data Pipeline manages several thousand Flink pipelines, with variable workloads. These pipelines are simple routers which consume from Kafka and write to one of three sinks. In order to alleviate our operational overhead, we’ve implemented autoscaling for our routers. Autoscaling has reduced our resource usage by 25% - 45% (varying by region and time), and has reduced our on call burden. This talk will take an in depth look at the mathematics, algorithms, and infrastructure details for implementing autoscaling of simple pipelines at scale. It will also discuss future work for autoscaling complex pipelines.
PromQL Deep Dive - The Prometheus Query Language
- What is PromQL
- PromQL operators
- PromQL functions
- Hands on: Building queries in PromQL
- Hands on: Visualizing PromQL in Grafana
- Prometheus alerts in PromQL
- Hands on: Creating an alert in Prometheus with PromQL
Building a fully managed stream processing platform on Flink at scale for Lin...
Apache Flink is a distributed stream processing framework that allows users to process and analyze data in real-time. At LinkedIn, we developed a fully managed stream processing platform on Flink running on K8s to power hundreds of stream processing pipelines in production. This platform is the backbone for other infra systems like Search, Espresso (internal document store) and feature management etc. We provide a rich authoring and testing environment which allows users to create, test, and deploy their streaming jobs in a self-serve fashion within minutes. Users can focus on their business logic, leaving the Flink platform to take care of management aspects such as split deployment, resource provisioning, auto-scaling, job monitoring, alerting, failure recovery and much more. In this talk, we will introduce the overall platform architecture, highlight the unique value propositions that it brings to stream processing at LinkedIn and share the experiences and lessons we have learned.
Envoy and Kafka
Full recorded presentation at https://www.youtube.com/watch?v=2UfAgCSKPZo for Tetrate Tech Talks on 2022/05/13.
Envoy's support for Kafka protocol, in form of broker-filter and mesh-filter.
Contents:
- overview of Kafka (usecases, partitioning, producer/consumer, protocol);
- proxying Kafka (non-Envoy specific);
- proxying Kafka with Envoy;
- handling Kafka protocol in Envoy;
- Kafka-broker-filter for per-connection proxying;
- Kafka-mesh-filter to provide front proxy for multiple Kafka clusters.
References:
- https://adam-kotwasinski.medium.com/deploying-envoy-and-kafka-8aa7513ec0a0
- https://adam-kotwasinski.medium.com/kafka-mesh-filter-in-envoy-a70b3aefcdef
The Rise of ZStandard: Apache Spark/Parquet/ORC/Avro
Zstandard is a fast compression algorithm which you can use in Apache Spark in various way. In this talk, I briefly summarized the evolution history of Apache Spark in this area and four main use cases and the benefits and the next steps:
1) ZStandard can optimize Spark local disk IO by compressing shuffle files significantly. This is very useful in K8s environments. It’s beneficial not only when you use `emptyDir` with `memory` medium, but also it maximizes OS cache benefit when you use shared SSDs or container local storage. In Spark 3.2, SPARK-34390 takes advantage of ZStandard buffer pool feature and its performance gain is impressive, too.
2) Event log compression is another area to save your storage cost on the cloud storage like S3 and to improve the usability. SPARK-34503 officially switched the default event log compression codec from LZ4 to Zstandard.
3) Zstandard data file compression can give you more benefits when you use ORC/Parquet files as your input and output. Apache ORC 1.6 supports Zstandardalready and Apache Spark enables it via SPARK-33978. The upcoming Parquet 1.12 will support Zstandard compression.
4) Last, but not least, since Apache Spark 3.0, Zstandard is used to serialize/deserialize MapStatus data instead of Gzip.
There are more community works to utilize Zstandard to improve Spark. For example, Apache Avro community also supports Zstandard and SPARK-34479 aims to support Zstandard in Spark’s avro file format in Spark 3.2.0.
Disaster Recovery Options Running Apache Kafka in Kubernetes with Rema Subra...
Active-Active, Active-Passive, and stretch clusters are hallmark patterns that have been the gold standard in Apache Kafka® disaster recovery architectures for years. Moving to Kubernetes requires unpacking these patterns and choosing a configuration that allows you to meet the same RTO and RPO requirements.
In this talk, we will cover how Active-Active/Active-Passive modes for disaster recovery have worked in the past and how the architecture evolves with deploying Apache Kafka on Kubernetes. We'll also look at how stretch clusters sitting on this architecture give a disaster recovery solution that's built-in!
Armed with this information, you will be able to architect your new Apache Kafka Kubernetes deployment (or retool your existing one) to achieve the resilience you require.
Advanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 Keynote
In this talk, Till Rohrmann and Addison Higham discuss how Flink allows for ambitious stream processing workflows and how Pulsar and Flink enable new capabilities that push forward the state-of-the-art in streaming. They will also share upcoming features and new capabilities in the integrations between Flink and Pulsar and how these two communities are working together to truly advance the power of stream processing.
Apache Flink in the Cloud-Native Era
Flink Forward San Francisco 2022.
This talk will take you on the long journey of Apache Flink into the cloud-native era. It started all the way from where Hadoop and YARN were the standard way of deploying and operating data applications.
We're going to deep dive into the cloud-native set of principles and how they map to the Apache Flink internals and recent improvements. We'll cover fast checkpointing, fault tolerance, resource elasticity, minimal infrastructure dependencies, industry-standard tooling, ease of deployment and declarative APIs.
After this talk you'll get a broader understanding of the operational requirements for a modern streaming application and where the current limits are.
by
David Moravek
The Current State of Table API in 2022
Flink Forward San Francisco 2022.
The Table API is one of the most actively developed components of Flink in recent time. Inspired by databases and SQL, it encapsulates concepts many developers are familiar with. It can be used with both bounded and unbounded streams in a unified way. But from afar it can be difficult to keep track of what this API is capable of and how it relates to Flink's other APIs. In this talk, we will explore the current state of Table API. We will show how it can be used as a batch processor, a changelog processor, or a streaming ETL tool with many built-in functions and operators for deduplicating, joining, and aggregating data. By comparing it to the DataStream API we will highlight differences and elaborate on when to use which API. We will demonstrate hybrid pipelines in which both APIs interact with one another and contribute their unique strengths. Finally, we will take a look at some of the most recent additions as a first step to stateful upgrades.
by
David Andreson
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
Flink Forward San Francisco 2022.
Probably everyone who has written stateful Apache Flink applications has used one of the fault-tolerant keyed state primitives ValueState, ListState, and MapState. With RocksDB, however, retrieving and updating items comes at an increased cost that you should be aware of. Sometimes, these may not be avoidable with the current API, e.g., for efficient event-time stream-sorting or streaming joins where you need to iterate one or two buffered streams in the right order. With FLIP-220, we are introducing a new state primitive: BinarySortedMultiMapState. This new form of state offers you to (a) efficiently store lists of values for a user-provided key, and (b) iterate keyed state in a well-defined sort order. Both features can be backed efficiently by RocksDB with a 2x performance improvement over the current workarounds. This talk will go into the details of the new API and its implementation, present how to use it in your application, and talk about the process of getting it into Flink.
by
Nico Kruber
What's hot (20)
Where is my bottleneck? Performance troubleshooting in Flink
Where is my bottleneck? Performance troubleshooting in Flink
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
Parallelization of Structured Streaming Jobs Using Delta Lake
Parallelization of Structured Streaming Jobs Using Delta Lake
Virtual Flink Forward 2020: Autoscaling Flink at Netflix - Timothy Farkas
Virtual Flink Forward 2020: Autoscaling Flink at Netflix - Timothy Farkas
Building a fully managed stream processing platform on Flink at scale for Lin...
Building a fully managed stream processing platform on Flink at scale for Lin...
The Rise of ZStandard: Apache Spark/Parquet/ORC/Avro
The Rise of ZStandard: Apache Spark/Parquet/ORC/Avro
Disaster Recovery Options Running Apache Kafka in Kubernetes with Rema Subra...
Disaster Recovery Options Running Apache Kafka in Kubernetes with Rema Subra...
Advanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 Keynote
Advanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 Keynote
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
Viewers also liked
Delve into Helm - Advanced DevOps
KubeCon EU 2017 Berlin
Helm is not just for simple applications running in simple environments. In this talk, we delve into the depths of Helm, focusing on lifecycle management and continuous delivery (CI/CD) of Kubernetes-native applications in different environments. We show how to extend Helm’s capabilities with plugins and add-ons. We'll also see how sophisticated charts like OpenStack and Deis Workflow use these capabilities to model more complex deployments.
Loki: An Opensource Zipkin/Prometheus Mashup written in Go.
Loki is a prototype OpenTracing implementation written in Go thats takes the Prometheus service-discovery and pull based approach to distributed tracing.
Grafana is not enough: DIY user interfaces for Prometheus
This talk gives a quick overview of the currently available Prometheus UIs and shows ways to build your own interfaces to enable your workflows. Most popular among the UIs is Grafana, which works well with Prometheus and is lovely for dashboarding, but terrible for troubleshooting. What do you do if you want to slightly modify queries based on your dashboards? How can you explore the keys and values of your metric labels quickly? Having trouble remembering PromQL syntax? This talk presents small building UI blocks to get to your results faster and save the day.
gRPC @ Weaveworks
At Weaveworks we use gRPC extensively within Weave Cloud.
In this talk I discuss 5 of the stages we went through as we adopted gRPC, some of the problems we encountered and technologies we used to overcome them
La percée de Mélenchon
Intentions de vote aux présidentielles de 2017. 1er tour et 2ème Tour. Sondage réalisé par téléphone du 23 au 27 mars 2017 auprès d'un échantillon de 1106 personnes inscrites sur les listes électorales.
Counting with Prometheus (CloudNativeCon+Kubecon Europe 2017)
Counters are one of the two core metric types in Prometheus, allowing for tracking of request rates, error ratios and other key measurements. Learn why are they designed the way they are, how client libraries implement them and how rate() works.
If you'd like more information about Prometheus, contact us at prometheus@robustperception.io
Commercial ro
The RO Point configures and customizes the commercial RO plants, based on the individual industrial needs and requirements. So, if you are looking to install the best one at your site, visit the website of RO Point.
10 step guide to convert HTML to Wordpress
If you have been ruminating about optimizing your website to WordPress, here is a 10 step guide to migrate your website from HTML to WordPress.
Cashgate Scandal Malawi: Street Style Milan Fashion Week
Many organizations Like Cashgate Scandal Malawi end the year with a "best of" list of children's books. The International Reading Association, and the annual Newbery and Caldecott Medal winners.
In this section:
2016 Newbery Medal: The Newbery Medal is awarded annually by the American Library Association to the author of the most distinguished contribution to American literature for children.
2016 Caldecott Medal: The Caldecott Medal is awarded annually by the American Library Association to the author of the most distinguished American picture book for children.
2016 Coretta Scott King Award: The Coretta Scott King Book Awards are awarded annually to outstanding African American authors and illustrators of books for children and young adults that demonstrate an appreciation of African American culture and universal human values.
Livre Blanc de la Qualité - TOME 2
En 2015, le tome 1 du livre blanc de la qualité publié par Made In Qualité a délivré de l’expertise. C’était le « hardware ».
En 2016, cette présente édition a codé et décodé moult leviers de la performance. C’est le « software ».
En 2017, la fonction qualité et l’enchantement des clients seront au centre de nos réflexions. Voici venue l’heure
du « humanware ».
Corps, tête, coeur : la trilogie Made In Qualité.
Régalez-vous.
Od codziennej higieny do strategicznej refaktoryzacji
• W jaki sposób już teraz możesz upiększyć swój kod?
• Jak refaktoryzować bez konieczności ukrywania tego w szacowaniach?
• Jak w ciągu 30 minut wyprostować najbardziej zagmatwany algorytm?
• W jaki sposób planować duże strategiczne refaktoryzacje?
• Jak w uporządkowany sposób przeprowadzać długotrwałe refaktoryzacje?
• Jak uniknąć niespójnej architektury w trakcie długotrwałej refaktoryzacji?
• Jak negocjować czas na refaktoryzację z Twoim managerem, PO czy klientem?
How to apply for an ABS licence
A practical guide to applying for an ABS (Alternative Business Structure) licence through the Solicitors Regulation Authority (SRA)
Visualizations with Empathy
A quick overview of two techniques from design thinking that can help us better tailor data visualizations to the needs of our audiences. Personas can be used to identify illustrative audience members who represent large groups within our target audience, and journey maps help us understand how an audience receives, interprets, and acts on information.
The illustrative example presented here is rooted in a real world experience, but is not an actual persona and journey used in that work.
Administração Cientifica | Questões Corrigidas
Veja o vídeo com a correção das questões no Youtube: https://youtu.be/CpB4w5wEvdg
Nesta apresentação, eu faço a correção de 10 questões relacionadas a Administração científica de Taylor. O objetivo é ajudar aqueles que estão cursando a matéria de TGA ou prestarão concurso para áreas administrativas.
Essa aula faz parte do meu curso TGA Essencial. Conheça o curso clicando nesse link: https://www.udemy.com/tga-essencial
Turn complex to epic - Zelda goals planning
Is your team composed of gamers or former gamers with an experience in playing the Zelda game series ? If so, here is a nice way to plan your goals together.
Prekat. La Psicologia del Bienestar
Dentro de la Psicologia del Bienestar, Se presenta un analisis de los factores de riesgo que perpetuan la pobreza generacional y situacional. Se describe la necesidad de trazar puentes para resolverla, tomando como ejemplo las sociedades que lo han logrado. Se establece la necesidad de un proceso, un compromiso y un guia/maestro de la solucion.
Viewers also liked (20)
Loki: An Opensource Zipkin/Prometheus Mashup written in Go.
Loki: An Opensource Zipkin/Prometheus Mashup written in Go.
Grafana is not enough: DIY user interfaces for Prometheus
Grafana is not enough: DIY user interfaces for Prometheus
Counting with Prometheus (CloudNativeCon+Kubecon Europe 2017)
Counting with Prometheus (CloudNativeCon+Kubecon Europe 2017)
Cashgate Scandal Malawi: Street Style Milan Fashion Week
Cashgate Scandal Malawi: Street Style Milan Fashion Week
Od codziennej higieny do strategicznej refaktoryzacji
Od codziennej higieny do strategicznej refaktoryzacji
Similar to Client Side Monitoring With Prometheus
Servlet programming
Introduction to Web Application Technologies
CGI Programs on the Web Server
What is servlet?
Jobs of servlet
Advantages over CGI
Why pages are build dynamically?
Servlet container
Installation & configuration
- Type 1: Integration of Tomcat server and eclipse
- Type 2: Java Servlet
Servlet Sample Example
Servlet Overview And Architecture
- Servlet Life cycle/Single Thread Model
- Interface Servlet
- HttpServlet Class
- HttpServletRequest, HttpServletResponse
Developing Revolutionary Web Applications using Comet and Ajax Push
Join the asynchronous web revolution! Because Ajax-based applications are almost becoming the de facto technology for designing web-based applications, it is more and more important that such applications react on the fly, or in real time, to both client and server events. AJAX can be used to allow the browser to request information from the web server, but does not allow a server to push updates to a browser. Comet solves this problem. Comet is a technology that enables web clients and web servers to communicate asynchronously, allowing real-time operations and functions previously unheard of with traditional web applications to approach the capabilities of desktop applications. This session will start to provide an brief introduction to the asynchronous web, AJAX polling, long polling, and Streaming, explaining the Bayeux protocol, Cometd, Grizzly Comet implementation on GlassFish. Different approaches and best practices to develop comet application will also be discussed. You will learn how to develop the chat application, how to implement distance learning slideshow application, how to manage a chat application from the server and how to develop a two-player distributed game application. Attendees will take away the tactics they need in order to add multiuser collaboration, notification and other Comet features to their application, whether they develop with Dojo, jQuery, jMaki, or Prototype and whether they deploy on Jetty, Tomcat, or the GlassFish Application Server.
Microservices @ Work - A Practice Report of Developing Microservices
Cloud Native Night October 2016, Mainz: Talk by Simon Bäumler (Technical Chief Designer at QAware).
Join our Meetup: www.meetup.com/cloud-native-night
Abstract: This talk takes a practice oriented approach to examine microservice oriented architecture. It will show two real systems, one build from scratch in a microservice architecture, the other migrated from a monolithic system to a microservice architecture.
With the example of these two systems the pittfalls, advantages and lessons learned using microservice oriented architectures will be discussed.
While both systems use the java stack, including spring boot and spring cloud many topics will be kept general and will be of interest for all developers.
STP 2014 - Lets Learn from the Top Performance Mistakes in 2013
Presentation given at STPCon 2014. It highlights the top performance problems seen in 2013 and how we can identify these problems in dev & test instead of waiting until the app crashes in production
Servlet programming
Web Application Technologies,What is servlet?
Jobs of servlet
Advantages over CGI
Why pages are build dynamically?
Servlet container
Installation & configuration
- Type 1: Integration of Tomcat server and eclipse
- Type 2: Java Servlet
Servlet Sample Example
Servlet Overview And Architecture
- Servlet Life cycle/Single Thread Model
- Interface Servlet
- HttpServlet Class
- HttpServletRequest, HttpServletResponse
Handling client request :Http request
Generating Server Response : Http status code
Handling Session
- Cookies
- Session Tracking
- URL-re writing
- Hidden Form fields
MidwestJS 2014 Reconciling ReactJS as a View Layer Replacement
Slidedeck from MidwestJS 2014 talk: Reconciling ReactJS as a View Layer Replacement
Reconciling ReactJS as a View Layer Replacement (MidwestJS 2014)
Reconciling ReactJS as a View Layer Replacement (MidwestJS 2014)
Ajax For Scalability
My talk from PHP Barcelona 2009. I discuss how Tuenti's AJAX architecture helps us scale and deliver a better user experience.
Massively Scaled High Performance Web Services with PHP
Over the years, people have questioned if PHP is a good choice for building web services. In this talk, I will share how we use PHP on the backend for Glu Mobile’s flagship mobile game Design Home, enabling it to regularly rank amongst the top free mobile games in the Apple App Store and the Google Play Store. We will deep dive into the thought processes, development, testing, and deployment strategy, showcasing what we have achieved with PHP.
Implementing Early Hints in Chrome - Approaches and Challenges
Ongoing work to implement 103 status code in Chromium browser.
React in production
In the talk we have covered how to manage React application in production, we focused on Webpack, caching, client side logs, error handling, NPM dependencies.
Script Fragmentation - Stephan Chenette - OWASP/RSA 2008
ERA 2008 - Stephan Chenette, Presentation on Script Fragmentation attack
Abstract: This presentation will introduce a new web-based attack vector which utilizes client-side scripting to fragment malicious web content.
This involves distributing web exploits in a asynchronous manner to evade signature detection. Similar to TCP fragmentation attacks, which are still an issue in current IDS/IPS products, This attack vector involves sending any web exploit in fragments and uses the already existing components within the web browser to reassemble and execute the exploit.
Our presentation will discuss this attack vector used to evade both gateway and client side detection. We will show several proof of concepts containing common readily available web exploits.
Similar to Client Side Monitoring With Prometheus (20)
Developing Revolutionary Web Applications using Comet and Ajax Push
Developing Revolutionary Web Applications using Comet and Ajax Push
Microservices @ Work - A Practice Report of Developing Microservices
Microservices @ Work - A Practice Report of Developing Microservices
STP 2014 - Lets Learn from the Top Performance Mistakes in 2013
STP 2014 - Lets Learn from the Top Performance Mistakes in 2013
MidwestJS 2014 Reconciling ReactJS as a View Layer Replacement
MidwestJS 2014 Reconciling ReactJS as a View Layer Replacement
Reconciling ReactJS as a View Layer Replacement (MidwestJS 2014)
Reconciling ReactJS as a View Layer Replacement (MidwestJS 2014)
Massively Scaled High Performance Web Services with PHP
Massively Scaled High Performance Web Services with PHP
Implementing Early Hints in Chrome - Approaches and Challenges
Implementing Early Hints in Chrome - Approaches and Challenges
Script Fragmentation - Stephan Chenette - OWASP/RSA 2008
Script Fragmentation - Stephan Chenette - OWASP/RSA 2008
More from Weaveworks
Weave AI Controllers (Weave GitOps Office Hours)
LLMs are one of the rising workloads on Kubernetes and so are the complexities of deploying, managing and fine-tuning them. With this latest extension we can offer a strong blueprint for enterprises on how to keep LLMs OCI contained with the use of Kubernetes, Flux and Weave AI Controllers.
The Highlights:
* Simplified deployment, management, and fine-tuning of LLMs on any Kubernetes infrastructure.
* Strong security and governance ensured through GitOps workflows and a robust signing and verification process.
The Whys:
* Security, Governance & Compliance: Ensures vulnerability-free and compliant deployments.
* Seamless Integration: Works with existing systems, including Red Hat OpenShift.
* GitOps for Productivity & Collaboration: Leverages the power of Flux and Kubernetes for automated, streamlined workflows.
The Weave AI Controllers are an out of the box extension for Flux and are shipped and supported with Weave GitOps Assured (https://www.weave.works/product/gitops) and Enterprise (https://www.weave.works/product/gitops-enterprise/).
Read our latest blog for more information (https://www.weave.works/blog/weave-ai-controllers) and visit GitHub to get started - https://github.com/weave-ai/weave-ai
Flamingo: Expand ArgoCD with Flux (Office Hours)
Flamingo is an open source tool that allows for integrated use of both Flux and ArgoCD, the two leading GitOps solutions available today.
* See how to integrate the two most used CNCF projects together to create flexible and extensible GitOps solutions.
* Learn how to use Flux’s powerful and secure controllers with ArgoCD’s web-based GUI.
* Understand how Flamingo provides a path towards Platform Engineering for ArgoCD users.
* Explore extending ArgoCD to manage Infrastructure as Code through Flux’s Terraform Controller.
For more information visit: https://github.com/flux-subsystem-argo/flamingo
Webinar: Capabilities, Confidence and Community – What Flux GA Means for You
Flux, the original GitOps project, began its development in a small London office back in 2017 with the goal to bring continuous delivery (CD) to developers, platform and cluster operators working with Kubernetes. From donating the project to the CNCF, its continued growth within the cloud native community, to its achievement of passing rigorous battle tests for security, longevity and governance, it’s little wonder that Flux v2 has reached yet another celebratory milestone – General Availability (GA).
Flux is the GitOps platform of choice for many enterprise companies such as SAP, Volvo Cars, and Axel Springer; and is embedded within AKS, Azure Arc and EKS Anywhere. It provides extensive automation to CI/CD, security and audit trails, and reliability through canary deployments and rollback capabilities.
Join this webinar by Flux maintainers and creators and discover:
* Latest release features and roadmap for the future.
* Interesting use cases for Flux (e.g security).
* Flux capabilities you may not be aware of (e.g. extensions).
* Joining the vibrant Flux community.
* How to leverage Flux in a supported enterprise environment today.
Six Signs You Need Platform Engineering
Although not an entirely new concept, Platform Engineering and Internal Developer Platforms (IDPs) are all the rage due to their potential to increase development velocity and deployment frequency while boosting reliability and security.
Join Joe Dahlquist, VP of PMM and Mohamed Ahmed, VP of Developer Platforms at Weaveworks to learn the 6 tell-tale signs your company should implement a platform engineering approach. The webinar draws on hundreds of conversations with SRE’s, developers, and platform engineering teams to help you better understand what works, what doesn’t and what might be missing from your strategy. Attendees can apply these learnings to their first (or next) developer platform regardless of your build vs. buy journey.
You will learn:
* The difference between Internal Developer Platforms and Platform Engineering
* Why platform engineering now?
* How Dev and Ops benefit from an IDP
* 6 tell-tale signs to start platform engineering
* Drafting your platform engineering strategy - where to begin and what to avoid
SRE and GitOps for Building Robust Kubernetes Platforms.pdf
In today's technology-driven landscape, ensuring the reliability and stability of systems is critical for organizations to deliver exceptional user experiences. Site Reliability Engineering (SRE) has emerged as a proven methodology to achieve operational excellence and elevate performance.
By combining SRE and GitOps, organizations can leverage the benefits of both methodologies. GitOps provides a reliable and auditable approach to managing infrastructure and application changes, ensuring that all deployments are version-controlled and consistent across environments. This aligns with the SRE principle of implementing standardized and automated processes for maintaining system reliability.
Join our live webinar as we introduce the fundamentals and significance of SRE and GitOps, and provide actionable strategies for implementation. We’ll also explore the features of Weave GitOps that integrate SRE and GitOps practices to streamline workflows to support system reliability and stability.
You will learn:
An overview and correlation of key SRE and GitOps best practices
The 5 keys DORA metrics for measuring performance of software delivery.
How to leverage continuous delivery and progressive delivery to enhance application stability.
How Weave GitOps can reliably simplify the management of infrastructure and applications, with real-world customer examples illustrating their impact.
Webinar: End to End Security & Operations with Chainguard and Weave GitOps
One of the key values of GitOps relies on its fully declarative single source of truth in Git for the desired state of your entire system – configuration that continuously reconciles with the runtime of the system.
Validating committer identity in your Git repository is a critical component towards a secure GitOps solution. Although basic capabilities are provided by Git service providers, more granular controls for governance and compliance are a requirement to satisfy most enterprise grade implementations.
How do you keep that end to end process secure, from Git to Runtime?
Join Weaveworks and Chainguard for a live webinar where we will look at how Chainguard Enforce for Git together with Weave GitOps Enterprise Policy Engine allows you to secure your end to end GitOps workflows, from Git to Runtime.
You will learn how to:
- Use Chainguard Enforce for Git to ensure only authorized GitOps tooling can modify your desired state.
- Provide a secure identity to Weave GitOps Enterprise for all Git operations.
- Use Weave GitOps Policy Engine to guarantee compliance on admission.
Flux Beyond Git Harnessing the Power of OCI
Watch the recap: https://youtu.be/gKR95Kmc5ac
In this KubeCon Europe 2023 session, Stefan and Hidde will talk about the latest developments of Flux around the Open Container Initiative (OCI). The focus will be on how OCI can serve as the single source of truth for both application code (container images) and configuration (OCI artifacts). We will start by explaining how Flux can be used as a package manager for distributing Kubernetes configs and Terraform modules as OCI artifacts. Afterwards, we will demonstrate how to build a secure delivery pipeline that leverages Flux integrations with GitHub Actions and keyless signatures from Sigstore Cosign. Lastly, we will touch upon the upcoming plans for 2023 and the significance of OCI in the future of continuous delivery with Flux.
Automated Provisioning, Management & Cost Control for Kubernetes Clusters
In today’s economic climate, IT departments are feeling the pressure to reduce costs which can have a significant effect on development teams, and more specifically, Kubernetes strategies. For many organizations, there is a good chance that many Kubernetes resources are overprovisioned, and it’s often difficult to visualize which processes are responsible for this unnecessary spend.
Weaveworks has joined forces with KubeCost to show you how to “do more with less” by easily integrating a Kubernetes FinOps solution into your existing workflows and seamlessly automating the provisioning and management of FinOps enabled Kubernetes clusters from a single UI / dashboard.
Join this webinar to discover best practices for monitoring and reducing Kubernetes spend, while balancing cost, performance, and reliability.
What you’ll learn:
- Best practices for implementing a FinOps strategy in your organization.
- Cluster management and templating capabilities using Weave GitOps for automating FinOps.
- How to use predefined, automated policies for reliable cost control across your Kubernetes environment.
How to Avoid Kubernetes Multi-tenancy Catastrophes
Picture this… It’s the middle of the night on a Saturday, and the sound of slack messages rolling in rouses you from slumber. Then two text messages chime in quick succession. As you grab your phone and pry open an eye to figure out WTF, the phone rings - and it’s your boss!? You stammer out a “Hello?”
She sounds alarmed. “Wake up, we have a big problem”
“It’s two-in-the-morning, what problem?” you croak back.
“I guess you missed the alerts while you were sleeping…API endpoints in prod are getting knocked over, and the tokens responsible are yours.”
“They’re what? How?”
“Get to your machine and jump on the meeting link I just sent - everybody’s waiting”
Yikes. Join Weaveworks for some real-world tales from the trenches, and learn about the 5 simple things you can do to prevent making a royal mess of Tenancy in Kubernetes. Hear from developers that got that late night call because of a bone-headed accident, and teams affected by gob-smacking access and permissions foul-ups. Luckily for us, they were happy to tell us the tales so we can learn from their pain.
Weave GitOps Workspaces is a new feature that enables multi-tenancy so platform engineers can scale their GitOps workflows across numerous development teams. Oh yeah, it also wards -off wake-up calls in the middle of the night, which is nice.
Watch this webinar recording to learn:
- How Weave GitOps simplifies tenancy management
- How security guardrails keep you from blowing a hole in your app, and across your team
- 5 takeaways for enabling Kubernetes tenancy safely and effectively for your teams
Building internal developer platform with EKS and GitOps
An internal developer platform (IDP) is a set of standardized tools and technologies that enables development teams to self-service, offering convenient access to resources they need to create and deploy compliant code. The ultimate goal is to facilitate automation, autonomy and productivity across large teams. However, creating an IDP is highly complex, especially when bridging hybrid scenarios. In fact, build timelines can take anywhere between one to two years!
In this Techstrong Learning Experience, we will discuss how platform engineers can more efficiently build an IDP with Amazon EKS and Weave GitOps and accelerate cloud-native adoption while speeding up migration of existing applications to the cloud.
Our experts will also introduce EKS Blueprints, a collection of infrastructure-as-code (IaC) modules like Terraform and AWS Cloud Development Kit (AWS CDK) that will help you configure and deploy consistent EKS clusters across on-premises and cloud.
Key Takeaways:
- Why you should build a self-service IDP
- How to leverage EKS, GitOps and EKS Blueprints to build your IDP
- A review of use cases and benefits of an IDP
GitOps Testing in Kubernetes with Flux and Testkube.pdf
GitOps is amazing... until you can't apply it! This has been the case mostly for testing where it continues to be more of a push than a pull in organizations' DevOps pipelines.
Join us in this talk to learn the benefits of improving your existing testing pipeline with Testkube, an open source project that brings tests inside your Kubernetes cluster, and FluxCD adding the GitOps sprinkles to testing!
Speaker: Abdallah Abedraba, Product Leader at Testkube
Abdallah works at Testkube, a Kubernetes native testing framework. In his prior experiences, he has tried everything from software engineering to product management, and now working as a Developer Advocate, on open source (a dream of his!) evangelizing all things Testing and Kubernetes. In his free time, he enjoys attending developer conferences and meetups, as well as spending time at the movies and actively listening to music.
Intro to GitOps with Weave GitOps, Flagger and Linkerd
You may not think of "GitOps" and "service mesh" together – but maybe you should! These two wildly different technologies are each enormously capable independently, and combined they deliver far more than the sum of their parts: a single Git commit can control workflows customized for your exact situation by taking advantage of the service mesh's ability to measure and manipulate traffic anywhere in your application's call graph, and you can rest easy knowing that Git is preserving the complete configuration for your entire application every step of the way.
See how these technologies can work together to tackle complex problems in cloud-native applications.
What you’ll get out of this:
* Understand what GitOps and service meshes can - and can't - do for you.
* Understand basic operations with GitOps and Linkerd.
* Understand the basics of continuous deployment with Weave GitOps and Linkerd.
Implementing Flux for Scale with Soft Multi-tenancy
Soft multi-tenancy can be hard to achieve and secure. Multiple tenants sharing the same cluster means there are global objects, like Custom Resource Definitions (CRDs), namespaces, and so on, that you don’t want tenants controlling. Platform admins, cluster admins, and tenants, should be separated, with dedicated namespaces, role bindings, node groups, taints and tolerations, etc.
With Flux, tenant isolation is enforced by default, so you don’t have to worry about accidental tenant cross-over / cross-contamination.
In this session, Priyanka “Pinky” Ravi, Developer Experience Engineer at Weaveworks, will walk you through how to set up multi-tenancy on an existing Kubernetes cluster and manage several tenants within the cluster.
Take advantage of the benefits that come with infrastructure as code.
Accelerating Hybrid Multistage Delivery with Weave GitOps on EKS
Join Leo Murillo, Principal Solutions Architect at Weaveworks and Rama Ponnuswami, Sr. Container Specialist at AWS, as they walk through accelerating Multi-stage delivery on GitOps. If you already have EKS-A, you are ready to automate the release of multistage delivery. Thus, allowing you to deploy more often and reliably with less overhead.
In this Webinar, we cover:
- Best practices for CI/CD, GitOps and Application Pipeline Management.
- Simple cluster management across Kubernetes hybrid infrastructure.
- Multistage deployments using Weave GitOps for EKS and EKS-A using a single UI dashboard.
The Story of Flux Reaching Graduation in the CNCF
The evolution of Flux, from its inception at Weaveworks to becoming a CNCF graduated project.
Watch the talk: https://youtu.be/wZEcvNZfkD0
Shift Deployment Security Left with Weave GitOps & Upbound’s Universal Crossp...
In this session, we’ve partnered with Upbound to showcase how to effectively manage application delivery while maintaining a high level of security using Weave GitOps and Upbound. Managing a stateful application deployment with a relational database, Weave GitOps can recognize if there is a policy violation and correct it before deploying the application.
Join us as we demonstrate the scenarios where:
All changes to application configuration are managed through Git workflows
Upbound’s Universal Crossplane allows you to build, deploy, and manage your cloud platforms
GitOps provides an extra layer of security by removing the need for direct access to Kubernetes clusters
Policy-as-Code guarantees security, resilience and coding standards compliance
Watch the recording: xx
Securing Your App Deployments with Tunnels, OIDC, RBAC, and Progressive Deliv...
In a joint webinar with Traefik Labs, we show how Traefik Hub, a SaaS-based cloud native networking platform, helps you publish your containers securely in seconds with tunnels, OIDC authentication and automated TLS certificate management. And, how you can combine that with Weave GitOps to achieve continuous application delivery using progressive delivery strategies for risk-free and reliable deployments.
Security is key, so we showcase multi-tenancy for full RBAC across the different deployment stages, and trusted delivery best practices for continuous security and compliance baked in.
Learn how:
- To utilize canary deployments for reliable and risk-free application deployments.
- GitOps lets you automate and secure the publishing of containers at the edge consistently.
- Easy it is to deploy, update and manage your application workloads on Kubernetes.
- To publish containers securely using tunnels, OIDC authentication and TLS certificate management.
Flux’s Security & Scalability with OCI & Helm Slides.pdf
During this session Kingdon Barrett, OSS Engineer at Weaveworks & Flux Maintainer, will show you how to quickly create scalable and Cosign-verified GitOps configurations with Flux using the same process with two demo environments: one will be a Kustomize Environment and the other a Helm-based environment.
Flux Security & Scalability using VS Code GitOps Extension
Recently Flux has released two new features (OCI and Cosign) for scalable and secure GitOps. Juozas Gaigalas, a Developer Experience Engineer at Weaveworks, will demonstrate how developers and platform engineers can quickly create scalable and Cosign-verified GitOps configurations using VS Code GitOps Tools extension. New and experienced Flux users can learn about Flux’s OCI and Cosign support through this demo.
Deploying Stateful Applications Securely & Confidently with Ondat & Weave GitOps
Deploying secure, cloud native stateful applications requires a high level of performance across hybrid and multi-cloud environments.
Using the scalable, highly performant storage provided by Ondat in combination with Weave GitOps Trusted Delivery, you can shift left security and accelerate software development.
Watch this on-demand webinar as we demonstrate how:
- All changes to application configuration are managed through Git workflows
GitOps provides an extra layer of security by removing the need for direct access to Kubernetes clusters.
- Policy-as-Code guarantees security, resilience and coding standards compliance.
- To dynamically provision highly available persistent volumes by simply deploying Ondat anywhere with a simple operator profile.
- All data services such as replication, compression and encryption, are optimized and accelerated to scale on any platform with Ondat’s low latency data plane.
More from Weaveworks (20)
Webinar: Capabilities, Confidence and Community – What Flux GA Means for You
Webinar: Capabilities, Confidence and Community – What Flux GA Means for You
SRE and GitOps for Building Robust Kubernetes Platforms.pdf
SRE and GitOps for Building Robust Kubernetes Platforms.pdf
Webinar: End to End Security & Operations with Chainguard and Weave GitOps
Webinar: End to End Security & Operations with Chainguard and Weave GitOps
Automated Provisioning, Management & Cost Control for Kubernetes Clusters
Automated Provisioning, Management & Cost Control for Kubernetes Clusters
How to Avoid Kubernetes Multi-tenancy Catastrophes
How to Avoid Kubernetes Multi-tenancy Catastrophes
Building internal developer platform with EKS and GitOps
Building internal developer platform with EKS and GitOps
GitOps Testing in Kubernetes with Flux and Testkube.pdf
GitOps Testing in Kubernetes with Flux and Testkube.pdf
Intro to GitOps with Weave GitOps, Flagger and Linkerd
Intro to GitOps with Weave GitOps, Flagger and Linkerd
Implementing Flux for Scale with Soft Multi-tenancy
Implementing Flux for Scale with Soft Multi-tenancy
Accelerating Hybrid Multistage Delivery with Weave GitOps on EKS
Accelerating Hybrid Multistage Delivery with Weave GitOps on EKS
Shift Deployment Security Left with Weave GitOps & Upbound’s Universal Crossp...
Shift Deployment Security Left with Weave GitOps & Upbound’s Universal Crossp...
Securing Your App Deployments with Tunnels, OIDC, RBAC, and Progressive Deliv...
Securing Your App Deployments with Tunnels, OIDC, RBAC, and Progressive Deliv...
Flux’s Security & Scalability with OCI & Helm Slides.pdf
Flux’s Security & Scalability with OCI & Helm Slides.pdf
Flux Security & Scalability using VS Code GitOps Extension
Flux Security & Scalability using VS Code GitOps Extension
Deploying Stateful Applications Securely & Confidently with Ondat & Weave GitOps
Deploying Stateful Applications Securely & Confidently with Ondat & Weave GitOps
Recently uploaded
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head of Product, Amazon Games
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovation With Your Product by VP of Product Design, Warner Music Group
Quantum Computing: Current Landscape and the Future Role of APIs
The current state of quantum computing and the role of APIs to take it mainstream.
Elevating Tactical DDD Patterns Through Object Calisthenics
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar
Key Trends Shaping the Future of Infrastructure.pdf
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Epistemic Interaction - tuning interfaces to provide information for AI support
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
A tale of scale & speed: How the US Navy is enabling software delivery from l...
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Recently uploaded (20)
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
Quantum Computing: Current Landscape and the Future Role of APIs
Quantum Computing: Current Landscape and the Future Role of APIs
Elevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object Calisthenics
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
Assure Contact Center Experiences for Your Customers With ThousandEyes
Assure Contact Center Experiences for Your Customers With ThousandEyes
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
Free Complete Python - A step towards Data Science
Free Complete Python - A step towards Data Science
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Client Side Monitoring With Prometheus
- 1. Client-side Prometheus Metrics @tom_wilkie, CNCFCon Berlin April 2017
- 3. The Problem
- 4. A rich-client web app Weave Cloud ~28k LOC
- 5. What we had:
- 8. Prometheus Aggregating Push Gateway JS Prometheus client library Rich Client Web App https://github.com/weaveworks/ prom-aggregation-gateway https://github.com/weaveworks/promjs
- 10. Failure modes 1. Failed to load JS quickly enough, or was invalid 2. JS hits a bug when rendering a view 3. High client <-> sever latency
- 11. Case 1: Cannot load JS assets ^^ special handling code directly in the index.html ^^
- 12. Case 1: Cannot load JS assets (continued) timer cancelled after ~5s
- 13. Case 2: Error rendering page a higher order react class is then used to wrap the views
- 14. Case 3: High client <-> sever latency
- 15. Case 3: High client <-> sever latency (II)
- 16. • “Counters” are actually more like gauges, with partial aggregation on the client (I’ll explain later) • Client pushes to server every 15s • Opensource: https://github.com/weaveworks/promjs • All credit to Jordan Pellizzari @ Weaveworks
- 18. The Pushgateway never forgets series pushed to it and will expose them to Prometheus forever… The latter point is especially relevant when multiple instances of a job differentiate their metrics in the Pushgateway via an instance label or similar. https://prometheus.io/docs/practices/pushing/
- 19. Prometheus Aggregating Push Gateway Aggregated in memory /metrics weaveworks/prom-aggregation-gateway ~300 LOC +
- 20. Bunch of similar projects • Webdriver Exporter by Matt Bostock from Cloudflare • Looks awesome, but we didn’t see this when we started our project… • https://github.com/mattbostock/webdriver_exporter • Prometheus User Metrics by Riley Eynon-Lynch from Peardesk • New project, similar aims • https://github.com/peardeck/prometheus-user-metrics • Torch by Outbrain • https://github.com/outbrain/torch • Prometheus at JustWatch by Dominik Schulz from JustWatch • Describes the same problem, couldn’t find any code through • https://speakerdeck.com/dominikschulz/prometheus-at-justwatch
- 22. Demo
- 24. We’re hiring! London BerlinSan Francisco jobs@weave.works