Grafana Mimir and VictoriaMetrics_ Performance Tests.pptx
•Download as PPTX, PDF•
0 likes•605 views
VictoriaMetrics and Grafana Mimir are time series databases with support of mostly the same protocols and APIs. However, they have different architectures and components, which makes the comparison more complicated. In the talk, we'll go through the details of the benchmark where I compared both solutions. We'll see how VictoriaMetrics and Mimir are dealing with identical workloads and how efficient they’re with using the allocated resources. The talk will cover design and architectural details, weak and strong points, trade-offs, and maintenance complexity of both solutions.
Report
Share
Report
Share
Recommended
Introduction to Prometheus
A general introduction and demo of the prometheus monitoring solution and ecosystem with a live demo, given at FLOSSUK 2018.
OSMC 2022 | VictoriaMetrics: scaling to 100 million metrics per second by Ali...
The growth of observability trends and Kubernetes adoption generates more demanding requirements for monitoring systems. Volumes of time series data increase exponentially, and old solutions just can’t keep up with the pace. The talk will cover how and why we created a new open source time series database from scratch. Which architectural decisions, which trade-offs we had to take in order to match the new expectations and handle 100 million metrics per second with VictoriaMetrics. The talk will be interesting for software engineers and DevOps familiar with observability and modern monitoring systems, or for those who’re interested in building scalable high performant databases for time series.
Monitoring_with_Prometheus_Grafana_Tutorial
This document provides an overview of setting up monitoring for MySQL and MongoDB servers using Prometheus and Grafana. It discusses installing and configuring Prometheus, Grafana, exporters for collecting metrics from MySQL, MongoDB and systems, and dashboards for visualizing the metrics in Grafana. The tutorial hands-on sets up Prometheus and Grafana in two virtual machines to monitor a MySQL master-slave replication setup and MongoDB cluster.
Prometheus - Intro, CNCF, TSDB,PromQL,Grafana
https://www.youtube.com/playlist?list=PLAiEy9H6ItrKC5PbH7KiELiSEIKv3tuov
-What is Prometheus?
-Difference Between Nagios vs Prometheus
-Architecture
-Alertmanager
-Time series DB
-PromQL (Prometheus Query Language)
-Live Demo
-Grafana
VictoriaLogs: Open Source Log Management System - Preview
VictoriaLogs Preview - Aliaksandr Valialkin
* Existing open source log management systems
- ELK (ElasticSearch) stack: Pros & Cons
- Grafana Loki: Pros & Cons
* What is VictoriaLogs
- Open source log management system from VictoriaMetrics
- Easy to setup and operate
- Scales vertically and horizontally
- Optimized for low resource usage (CPU, RAM, disk space)
- Accepts data from Logstash and Fluentbit in Elasticsearch format
- Accepts data from Promtail in Loki format
- Supports stream concept from Loki
- Provides easy to use yet powerful query language - LogsQL
* LogsQL Examples
- Search by time
- Full-text search
- Combining search queries
- Searching arbitrary labels
* Log Streams
- What is a log stream?
- LogsQL examples: querying log streams
- Stream labels vs log labels
* LogsQL: stats over access logs
* VictoriaLogs: CLI Integration
* VictoriaLogs Recap
Monitoring Kubernetes with Prometheus
Prometheus has become the defacto monitoring system for cloud native applications, with systems like Kubernetes and Etcd natively exposing Prometheus metrics. In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOSs Prometheus Operator, or using Prometheus Ksonnet Mixin. Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.
VictoriaMetrics 2023 Roadmap
* Grafana datasource plugin
* OpenTelemetry ingestion protocol support
* Streaming aggregation
* vmalert: UI for rules management
* vmalert hysteresis support
* vmui explore tab
* VictoriaLogs
Monitoring with Prometheus
This document provides an overview of using Prometheus for monitoring and alerting. It discusses using Node Exporters and other exporters to collect metrics, storing metrics in Prometheus, querying metrics using PromQL, and configuring alert rules and the Alertmanager for notifications. Key aspects covered include scraping configs, common exporters, data types and selectors in PromQL, operations and functions, and setting up alerts and the Alertmanager for routing alerts.
Recommended
Introduction to Prometheus
A general introduction and demo of the prometheus monitoring solution and ecosystem with a live demo, given at FLOSSUK 2018.
OSMC 2022 | VictoriaMetrics: scaling to 100 million metrics per second by Ali...
The growth of observability trends and Kubernetes adoption generates more demanding requirements for monitoring systems. Volumes of time series data increase exponentially, and old solutions just can’t keep up with the pace. The talk will cover how and why we created a new open source time series database from scratch. Which architectural decisions, which trade-offs we had to take in order to match the new expectations and handle 100 million metrics per second with VictoriaMetrics. The talk will be interesting for software engineers and DevOps familiar with observability and modern monitoring systems, or for those who’re interested in building scalable high performant databases for time series.
Monitoring_with_Prometheus_Grafana_Tutorial
This document provides an overview of setting up monitoring for MySQL and MongoDB servers using Prometheus and Grafana. It discusses installing and configuring Prometheus, Grafana, exporters for collecting metrics from MySQL, MongoDB and systems, and dashboards for visualizing the metrics in Grafana. The tutorial hands-on sets up Prometheus and Grafana in two virtual machines to monitor a MySQL master-slave replication setup and MongoDB cluster.
Prometheus - Intro, CNCF, TSDB,PromQL,Grafana
https://www.youtube.com/playlist?list=PLAiEy9H6ItrKC5PbH7KiELiSEIKv3tuov
-What is Prometheus?
-Difference Between Nagios vs Prometheus
-Architecture
-Alertmanager
-Time series DB
-PromQL (Prometheus Query Language)
-Live Demo
-Grafana
VictoriaLogs: Open Source Log Management System - Preview
VictoriaLogs Preview - Aliaksandr Valialkin
* Existing open source log management systems
- ELK (ElasticSearch) stack: Pros & Cons
- Grafana Loki: Pros & Cons
* What is VictoriaLogs
- Open source log management system from VictoriaMetrics
- Easy to setup and operate
- Scales vertically and horizontally
- Optimized for low resource usage (CPU, RAM, disk space)
- Accepts data from Logstash and Fluentbit in Elasticsearch format
- Accepts data from Promtail in Loki format
- Supports stream concept from Loki
- Provides easy to use yet powerful query language - LogsQL
* LogsQL Examples
- Search by time
- Full-text search
- Combining search queries
- Searching arbitrary labels
* Log Streams
- What is a log stream?
- LogsQL examples: querying log streams
- Stream labels vs log labels
* LogsQL: stats over access logs
* VictoriaLogs: CLI Integration
* VictoriaLogs Recap
Monitoring Kubernetes with Prometheus
Prometheus has become the defacto monitoring system for cloud native applications, with systems like Kubernetes and Etcd natively exposing Prometheus metrics. In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOSs Prometheus Operator, or using Prometheus Ksonnet Mixin. Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.
VictoriaMetrics 2023 Roadmap
* Grafana datasource plugin
* OpenTelemetry ingestion protocol support
* Streaming aggregation
* vmalert: UI for rules management
* vmalert hysteresis support
* vmui explore tab
* VictoriaLogs
Monitoring with Prometheus
This document provides an overview of using Prometheus for monitoring and alerting. It discusses using Node Exporters and other exporters to collect metrics, storing metrics in Prometheus, querying metrics using PromQL, and configuring alert rules and the Alertmanager for notifications. Key aspects covered include scraping configs, common exporters, data types and selectors in PromQL, operations and functions, and setting up alerts and the Alertmanager for routing alerts.
Prometheus
Prometheus is an open-source monitoring system started in 2012 by former Google engineers. It uses a pull-based architecture to easily scale and features a powerful multi-dimensional data model and query language. Prometheus scrapes metrics from instrumented jobs like node exporters and stores time series data which can then be queried and graphed.
Prometheus (Prometheus London, 2016)
Brian Brazil is an engineer passionate about reliable software operations. He worked at Google SRE for 7 years and is the founder of Prometheus, an open source time series database designed for monitoring system and service metrics. Prometheus supports metric labeling, unified alerting and graphing, and is efficient, decentralized, reliable, and opinionated in how it encourages good monitoring practices.
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...
Kubernetes has the concept of resource requests and limits. Pods get scheduled on the nodes based on their requests and optionally limited in how much of the resource they can consume. Understanding and optimizing resource requests/limits is crucial both for reducing resource "slack" and ensuring application performance/low-latency. This talk shows our approach to monitoring and optimizing Kubernetes resources for 80+ clusters to achieve cost-efficiency and reducing impact for latency-critical applications. All shown tools are Open Source and can be applied to most Kubernetes deployments.
Monitoring with prometheus
A short presentation of Prometheus given at the CodeU Automation Night meetup in Aarhus at November 1st 2016.
Docker Swarm Introduction
Docker Swarm allows managing Docker clusters remotely. The key components are swarm managers, swarm nodes, and a scheduler. Swarm managers oversee nodes in the cluster using Docker APIs. The scheduler uses strategies and filters to determine where to place containers on nodes. Discovery services help register and discover nodes in the cluster.
Introduction to Kafka Cruise Control
An introduction to Kafka cruise control which performs dynamic workload balancing for Kafka clusters.
Grafana introduction
Grafana is an open source analytics and monitoring tool that uses InfluxDB to store time series data and provide visualization dashboards. It collects metrics like application and server performance from Telegraf every 10 seconds, stores the data in InfluxDB using the line protocol format, and allows users to build dashboards in Grafana to monitor and get alerts on metrics. An example scenario is using it to collect and display load time metrics from a QA whitelist VM.
Cruise Control: Effortless management of Kafka clusters
Kafka has become the de facto standard for streaming data with high-throughput, low-latency, and fault-tolerance. However, its rising adoption raises new challenges. In particular, the growing cluster sizes, increasing volume and diversity of user traffic, and aging network and server components induce an overhead in managing the system. This overhead makes it infeasible for human operators to constantly monitor, identify, and mitigate issues. The resulting utilization imbalance across brokers leads to unpredictable client performance due to the high variation in their throughput and latency. Finally, properly expanding, shrinking, or upgrading clusters also incurs a management overhead. Hence, adopting a principled approach to manage Kafka clusters is integral to the sustainability of the infrastructure.
This talk will describe how LinkedIn alleviates the management overhead of large-scale Kafka clusters using Cruise Control. To this end, first, we will discuss the reactive and proactive techniques that Cruise Control uses to support admin operations for cluster maintenance, enable anomaly detection with self-healing, and provide real-time monitoring for Kafka clusters. Next, we will examine how Cruise Control performs in production. Finally, we will conclude with questions and further discussion.
Prometheus and Grafana
Prometheus is an open-source monitoring system that collects metrics from configured targets, stores time-series data, and allows users to query and visualize the data. It works by scraping metrics over HTTP from applications and servers, storing the data in its time-series database, and providing a UI and query language to analyze the data. Prometheus is useful for monitoring system metrics like CPU usage and memory as well as application metrics like HTTP requests and errors.
Monitoring With Prometheus
Prometheus: Monitoring by "Pravin Magdum" from "Crevise". The presentation was done at #doppa17 DevOps++ Global Summit 2017. All the copyrights are reserved with the author
Storing 16 Bytes at Scale
The document discusses techniques for storing time series data at scale in a time series database (TSDB). It describes storing 16 bytes of data per sample by compressing timestamps and values. It proposes organizing data into blocks, chunks, and files to handle high churn rates. An index structure uses unique IDs and sorted label mappings to enable efficient queries over millions of time series and billions of samples. Benchmarks show the TSDB can handle over 100,000 samples/second while keeping memory, CPU and disk usage low.
Grafana optimization for Prometheus
This document discusses using Grafana to optimize visualization of metrics from Prometheus in a dynamic environment. It describes deploying multiple Prometheus instances to monitor over 100 instances per service across various services running on EC2. Key Grafana features discussed include templating to dynamically filter dashboards, panel repetition to show multiple graphs, and scripted dashboards to generate dashboards from JSON definitions. The document provides examples of using these features to create service trend dashboards, dynamically refresh dashboards based on time range changes, switch data sources, and generate alert dashboards from Prometheus alert views.
Prometheus 101
Prometheus is an open-source monitoring system that collects metrics from instrumented systems and applications and allows for querying and alerting on metrics over time. It is designed to be simple to operate, scalable, and provides a powerful query language and multidimensional data model. Key features include no external dependencies, metrics collection by scraping endpoints, time-series storage, and alerting handled by the AlertManager with support for various integrations.
Infrastructure & System Monitoring using Prometheus
The document introduces infrastructure and system monitoring using Prometheus. It discusses the importance of monitoring, common things to monitor like services, applications, and OS metrics. It provides an overview of Prometheus including its main components and data format. The document demonstrates setting up Prometheus, adding host metrics using Node Exporter, configuring Grafana, monitoring Docker containers using cAdvisor, configuring alerting in Prometheus and Alertmanager, instrumenting application code, and integrating Consul for service discovery. Live code demos are provided for key concepts.
Under the Hood of a Shard-per-Core Database Architecture
Most databases are based on architectures that pre-date advances to modern hardware. This results in performance issues, the need to overprovision, and a high total cost of ownership. In this webinar we will discuss the advances to modern server technology and take a deep dive into Scylla’s shard-per-core architecture and our asynchronous engine, the Seastar framework.
Join us to learn how Seastar (and Scylla):
Avoid locks and contention on the CPU level
Bypass kernel bottlenecks
Implement its per-core shared-nothing autosharding mechanism
Utilize modern storage hardware
Leverage NUMA to get the best RAM performance
Balance your data across CPUs and nodes for best and smoothest performance
Plus we’ll cover the advantages of unlocking vertical scalability.
Monitoring With Prometheus
In this session, we will start with the importance of monitoring of services and infrastructure. We will discuss about Prometheus an opensource monitoring tool. We will discuss the architecture of Prometheus. We will also discuss some visualization tools which can be used over Prometheus. Then we will have a quick demo for Prometheus and Grafana.
PromQL Deep Dive - The Prometheus Query Language
- What is PromQL
- PromQL operators
- PromQL functions
- Hands on: Building queries in PromQL
- Hands on: Visualizing PromQL in Grafana
- Prometheus alerts in PromQL
- Hands on: Creating an alert in Prometheus with PromQL
Deploying Flink on Kubernetes - David Anderson
Kubernetes has rapidly established itself as the de facto standard for orchestrating containerized infrastructures. And with the recent completion of the refactoring of Flink's deployment and process model known as FLIP-6, Kubernetes has become a natural choice for Flink deployments. In this talk we will walk through how to get Flink running on Kubernetes
Introduction to helm
Helm is a package manager for Kubernetes that allows for easy installation, upgrade, and management of Kubernetes applications. It provides repeatability, reliability, and simplifies deploying applications across multiple Kubernetes environments. Helm originated from an internal hackathon at Deis and was jointly developed by Google and Deis. It is now maintained by the Cloud Native Computing Foundation. Helm consists of a client that interacts with the Tiller server running inside the Kubernetes cluster to manage application lifecycles using charts, which are packages containing Kubernetes resource definitions.
cLoki: Like Loki but for ClickHouse
Presentation by Lorenzo Mangani of QXIP at the October 26 SF Bay Area ClickHouse meetup
https://www.meetup.com/San-Francisco-Bay-Area-ClickHouse-Meetup
https://qxip.net/
High Performance Erlang - Pitfalls and Solutions
Presented at Erlang Factory 2016, San Francisco, CA.
Erlang is widely used for building concurrent applications. However, when we push the performance of our Erlang based application to handle millions of concurrent clients, some Erlang scalability issues begin to show and some conventional programming paradigm of Erlang no longer hold. We would like to share some of these issue and how we address them. In addition, we share some of our experience on how to profile an Erlang application to identify bottlenecks.
We will take a deep look at some of the basic mechanisms of Erlang and show how they behave under high load and parallelism, which includes message delivery, process management and shared data structures such as maps and ETS tables. We will demonstrate their limitations and propose techniques to alleviate the issues.
We will also share profiling techniques on how to find those bottlenecks in Erlang applications across different levels. We will share techniques for writing highly performant Erlang applications.
Microservice monitoring
This document discusses various techniques for monitoring microservices including Circuit Breaker pattern, bulkheads, back pressure, Hystrix, Hystrix Dashboard, thread pools, Dynatrace, and memory analysis. Hystrix is a latency and fault tolerance library used for monitoring, controlling concurrency, and isolating points of access to remote systems. The Hystrix Dashboard provides real-time monitoring. Dynatrace allows deep profiling of applications through memory dumps and other techniques to identify performance issues and root causes. The workshops agenda covers setting up Hystrix commands, the dashboard, and connecting applications to Dynatrace for memory and GC monitoring.
More Related Content
What's hot
Prometheus
Prometheus is an open-source monitoring system started in 2012 by former Google engineers. It uses a pull-based architecture to easily scale and features a powerful multi-dimensional data model and query language. Prometheus scrapes metrics from instrumented jobs like node exporters and stores time series data which can then be queried and graphed.
Prometheus (Prometheus London, 2016)
Brian Brazil is an engineer passionate about reliable software operations. He worked at Google SRE for 7 years and is the founder of Prometheus, an open source time series database designed for monitoring system and service metrics. Prometheus supports metric labeling, unified alerting and graphing, and is efficient, decentralized, reliable, and opinionated in how it encourages good monitoring practices.
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...
Kubernetes has the concept of resource requests and limits. Pods get scheduled on the nodes based on their requests and optionally limited in how much of the resource they can consume. Understanding and optimizing resource requests/limits is crucial both for reducing resource "slack" and ensuring application performance/low-latency. This talk shows our approach to monitoring and optimizing Kubernetes resources for 80+ clusters to achieve cost-efficiency and reducing impact for latency-critical applications. All shown tools are Open Source and can be applied to most Kubernetes deployments.
Monitoring with prometheus
A short presentation of Prometheus given at the CodeU Automation Night meetup in Aarhus at November 1st 2016.
Docker Swarm Introduction
Docker Swarm allows managing Docker clusters remotely. The key components are swarm managers, swarm nodes, and a scheduler. Swarm managers oversee nodes in the cluster using Docker APIs. The scheduler uses strategies and filters to determine where to place containers on nodes. Discovery services help register and discover nodes in the cluster.
Introduction to Kafka Cruise Control
An introduction to Kafka cruise control which performs dynamic workload balancing for Kafka clusters.
Grafana introduction
Grafana is an open source analytics and monitoring tool that uses InfluxDB to store time series data and provide visualization dashboards. It collects metrics like application and server performance from Telegraf every 10 seconds, stores the data in InfluxDB using the line protocol format, and allows users to build dashboards in Grafana to monitor and get alerts on metrics. An example scenario is using it to collect and display load time metrics from a QA whitelist VM.
Cruise Control: Effortless management of Kafka clusters
Kafka has become the de facto standard for streaming data with high-throughput, low-latency, and fault-tolerance. However, its rising adoption raises new challenges. In particular, the growing cluster sizes, increasing volume and diversity of user traffic, and aging network and server components induce an overhead in managing the system. This overhead makes it infeasible for human operators to constantly monitor, identify, and mitigate issues. The resulting utilization imbalance across brokers leads to unpredictable client performance due to the high variation in their throughput and latency. Finally, properly expanding, shrinking, or upgrading clusters also incurs a management overhead. Hence, adopting a principled approach to manage Kafka clusters is integral to the sustainability of the infrastructure.
This talk will describe how LinkedIn alleviates the management overhead of large-scale Kafka clusters using Cruise Control. To this end, first, we will discuss the reactive and proactive techniques that Cruise Control uses to support admin operations for cluster maintenance, enable anomaly detection with self-healing, and provide real-time monitoring for Kafka clusters. Next, we will examine how Cruise Control performs in production. Finally, we will conclude with questions and further discussion.
Prometheus and Grafana
Prometheus is an open-source monitoring system that collects metrics from configured targets, stores time-series data, and allows users to query and visualize the data. It works by scraping metrics over HTTP from applications and servers, storing the data in its time-series database, and providing a UI and query language to analyze the data. Prometheus is useful for monitoring system metrics like CPU usage and memory as well as application metrics like HTTP requests and errors.
Monitoring With Prometheus
Prometheus: Monitoring by "Pravin Magdum" from "Crevise". The presentation was done at #doppa17 DevOps++ Global Summit 2017. All the copyrights are reserved with the author
Storing 16 Bytes at Scale
The document discusses techniques for storing time series data at scale in a time series database (TSDB). It describes storing 16 bytes of data per sample by compressing timestamps and values. It proposes organizing data into blocks, chunks, and files to handle high churn rates. An index structure uses unique IDs and sorted label mappings to enable efficient queries over millions of time series and billions of samples. Benchmarks show the TSDB can handle over 100,000 samples/second while keeping memory, CPU and disk usage low.
Grafana optimization for Prometheus
This document discusses using Grafana to optimize visualization of metrics from Prometheus in a dynamic environment. It describes deploying multiple Prometheus instances to monitor over 100 instances per service across various services running on EC2. Key Grafana features discussed include templating to dynamically filter dashboards, panel repetition to show multiple graphs, and scripted dashboards to generate dashboards from JSON definitions. The document provides examples of using these features to create service trend dashboards, dynamically refresh dashboards based on time range changes, switch data sources, and generate alert dashboards from Prometheus alert views.
Prometheus 101
Prometheus is an open-source monitoring system that collects metrics from instrumented systems and applications and allows for querying and alerting on metrics over time. It is designed to be simple to operate, scalable, and provides a powerful query language and multidimensional data model. Key features include no external dependencies, metrics collection by scraping endpoints, time-series storage, and alerting handled by the AlertManager with support for various integrations.
Infrastructure & System Monitoring using Prometheus
The document introduces infrastructure and system monitoring using Prometheus. It discusses the importance of monitoring, common things to monitor like services, applications, and OS metrics. It provides an overview of Prometheus including its main components and data format. The document demonstrates setting up Prometheus, adding host metrics using Node Exporter, configuring Grafana, monitoring Docker containers using cAdvisor, configuring alerting in Prometheus and Alertmanager, instrumenting application code, and integrating Consul for service discovery. Live code demos are provided for key concepts.
Under the Hood of a Shard-per-Core Database Architecture
Most databases are based on architectures that pre-date advances to modern hardware. This results in performance issues, the need to overprovision, and a high total cost of ownership. In this webinar we will discuss the advances to modern server technology and take a deep dive into Scylla’s shard-per-core architecture and our asynchronous engine, the Seastar framework.
Join us to learn how Seastar (and Scylla):
Avoid locks and contention on the CPU level
Bypass kernel bottlenecks
Implement its per-core shared-nothing autosharding mechanism
Utilize modern storage hardware
Leverage NUMA to get the best RAM performance
Balance your data across CPUs and nodes for best and smoothest performance
Plus we’ll cover the advantages of unlocking vertical scalability.
Monitoring With Prometheus
In this session, we will start with the importance of monitoring of services and infrastructure. We will discuss about Prometheus an opensource monitoring tool. We will discuss the architecture of Prometheus. We will also discuss some visualization tools which can be used over Prometheus. Then we will have a quick demo for Prometheus and Grafana.
PromQL Deep Dive - The Prometheus Query Language
- What is PromQL
- PromQL operators
- PromQL functions
- Hands on: Building queries in PromQL
- Hands on: Visualizing PromQL in Grafana
- Prometheus alerts in PromQL
- Hands on: Creating an alert in Prometheus with PromQL
Deploying Flink on Kubernetes - David Anderson
Kubernetes has rapidly established itself as the de facto standard for orchestrating containerized infrastructures. And with the recent completion of the refactoring of Flink's deployment and process model known as FLIP-6, Kubernetes has become a natural choice for Flink deployments. In this talk we will walk through how to get Flink running on Kubernetes
Introduction to helm
Helm is a package manager for Kubernetes that allows for easy installation, upgrade, and management of Kubernetes applications. It provides repeatability, reliability, and simplifies deploying applications across multiple Kubernetes environments. Helm originated from an internal hackathon at Deis and was jointly developed by Google and Deis. It is now maintained by the Cloud Native Computing Foundation. Helm consists of a client that interacts with the Tiller server running inside the Kubernetes cluster to manage application lifecycles using charts, which are packages containing Kubernetes resource definitions.
cLoki: Like Loki but for ClickHouse
Presentation by Lorenzo Mangani of QXIP at the October 26 SF Bay Area ClickHouse meetup
https://www.meetup.com/San-Francisco-Bay-Area-ClickHouse-Meetup
https://qxip.net/
What's hot (20)
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...
Cruise Control: Effortless management of Kafka clusters
Cruise Control: Effortless management of Kafka clusters
Infrastructure & System Monitoring using Prometheus
Infrastructure & System Monitoring using Prometheus
Under the Hood of a Shard-per-Core Database Architecture
Under the Hood of a Shard-per-Core Database Architecture
Similar to Grafana Mimir and VictoriaMetrics_ Performance Tests.pptx
High Performance Erlang - Pitfalls and Solutions
Presented at Erlang Factory 2016, San Francisco, CA.
Erlang is widely used for building concurrent applications. However, when we push the performance of our Erlang based application to handle millions of concurrent clients, some Erlang scalability issues begin to show and some conventional programming paradigm of Erlang no longer hold. We would like to share some of these issue and how we address them. In addition, we share some of our experience on how to profile an Erlang application to identify bottlenecks.
We will take a deep look at some of the basic mechanisms of Erlang and show how they behave under high load and parallelism, which includes message delivery, process management and shared data structures such as maps and ETS tables. We will demonstrate their limitations and propose techniques to alleviate the issues.
We will also share profiling techniques on how to find those bottlenecks in Erlang applications across different levels. We will share techniques for writing highly performant Erlang applications.
Microservice monitoring
This document discusses various techniques for monitoring microservices including Circuit Breaker pattern, bulkheads, back pressure, Hystrix, Hystrix Dashboard, thread pools, Dynatrace, and memory analysis. Hystrix is a latency and fault tolerance library used for monitoring, controlling concurrency, and isolating points of access to remote systems. The Hystrix Dashboard provides real-time monitoring. Dynatrace allows deep profiling of applications through memory dumps and other techniques to identify performance issues and root causes. The workshops agenda covers setting up Hystrix commands, the dashboard, and connecting applications to Dynatrace for memory and GC monitoring.
Microservices with Micronaut
JCON Online 2021, International Java Community Conference, 07.10.21, Moritz Kammerer (@Moritz Kammerer, Expert Software Engineer at QAware).
== Please download slides in case they are blurred! ===
In his talk we have had a look at how Microservices can be developed with Micronaut. In our slides you can find out if it kept its promise.
Robotics technical Presentation
HIGH PERFORMANCE EDGE DATA PROCESSING SOFTWARE FOR ROBOTIC APPLICATIONS
Robotic edge autonomous systems is a fast growing market with as fast growing challenges:
Sophisticated applications,
Increasing demand in complexity:
Communications with other systems,
Heavy algorithms: AI, vision navigation, etc. Hardware has improved substantially:
- Better embedded computers
- Improved sensors (more and faster data)
https://klepsydra.com/demos-media/#earth-observation-throughput
Breach and attack simulation tools
With the focus on security, most organisations test the security defenses via pen-testing. But what about after the network has been compromised. Is there an Advance Persistent Threat (APT) sitting on the network? Will the defenses be able to detect this?
This talk will discuss some of the open source tools that can help simulate this threat. So as to test the security defenses if an APT makes it onto the network.
RedisConf17 - Redis in High Traffic Adtech Stack
Rahul Babbar presented on Times Internet's use of Redis in their high-traffic adtech stack. Some key points:
- Times Internet is India's largest digital company with 240+ million unique visitors per month.
- Their adtech stack powers ads for 150+ publishers and handles 9 billion ad impressions per month.
- Redis is used for caching, user profiles, analytics and more due to its low latency and ability to handle high volumes.
- Their Redis clusters consist of over 160 nodes across 4 clusters, serving over 2 million queries per second with 99% under 2ms response times.
Sista: Improving Cog’s JIT performance
Title: Sista: Improving Cog’s JIT performance
Speaker: Clément Béra
Thu, August 21, 9:45am – 10:30am
Video Part1
https://www.youtube.com/watch?v=X4E_FoLysJg
Video Part2
https://www.youtube.com/watch?v=gZOk3qojoVE
Description
Abstract: Although recent improvements of the Cog VM performance made it one of the fastest available Smalltalk virtual machine, the overhead compared to optimized C code remains important. Efficient industrial object oriented virtual machine, such as Javascript V8's engine for Google Chrome and Oracle Java Hotspot can reach on many benchs the performance of optimized C code thanks to adaptive optimizations performed their JIT compilers. The VM becomes then cleverer, and after executing numerous times the same portion of codes, it stops the code execution, looks at what it is doing and recompiles critical portion of codes in code faster to run based on the current environment and previous executions.
Bio: Clément Béra and Eliot Miranda has been working together on Cog's JIT performance for the last year. Clément Béra is a young engineer and has been working in the Pharo team for the past two years. Eliot Miranda is a Smalltalk VM expert who, among others, has implemented Cog's JIT and the Spur Memory Manager for Cog.
Pragmatic model checking: from theory to implementations
Invited presentation given by Niels Lohmann on December 3, 2013 in Potsdam, Germany as invited lecture at the Business Process Compliance course at the Hasso-Plattner-Institute.
Micrometrics to forecast performance tsunamis
Tsunami waves travel at the speed of 500 - 600 miles/hr. Normal waves travel at the speed of 5 - 60 miles/hr. Due to technical limitations, even massive Tsunamis are hard to forecast and detect beforehand. In recent times, hyper sensitive micro-metrics measuring technologies are employed to forecast Tsunamis. Similarly, it’s hard to forecast production performance problems beforehand. In this session you will learn the micro-metrics to be measured in dev/test environments that can forecast production performance problems with a fair level of accuracy.
How to Design a Program Repair Bot? Insights from the Repairnator Project
Repairnator is a program repair bot that analyzes failed CI builds from TravisCI projects on GitHub written in Java and using Maven. It reproduces bugs locally and applies program repair tools like Nopol and Astor to synthesize patches. The document discusses the design choices for Repairnator, steps it takes to analyze builds, reproduce bugs, synthesize patches, and results obtained. It finds bug reproduction and patch generation are difficult with current tools only achieving a valid patch for 1 of over 14,000 builds analyzed. Future work could involve directly integrating with TravisCI and reproducing failures faster.
Common Pitfalls of Functional Programming and How to Avoid Them: A Mobile Gam...
This material is presented on CUFP 2013.
Functional programming is already an established technology is many areas. However, the lack of skilled developers has been a challenging hurdle in the adoption of such languages. It is easy for an inexperienced programmer to fall into the many traps of functional programming, resulting in a loss of productivity and bad software quality. Resource leaks caused by Haskell's lazy evaluation, for instance, are only the tip of the iceberg. Knowledge sharing and a mature tool-assisted development process are ways to avoid such pitfalls. At GREE, one of the largest mobile gaming companies, we use Haskell and Scala to develop major components of our platform, such as a distributed NoSQL solution, or an image storage infrastructure. However, only 11 programmers use functional programming on their daily task. In this talk, we will describe some unexpected functional programming issues we ran into, how we solved them and how we hope to avoid them in the future. We have developed a system testing framework to enhance regression testing, spent lots of time documenting pitfalls and introduced technical reviews. Recently, we even started holding lunchtime presentations about functional programming in order to attract beginners and prevent them from falling into the same traps.
Microservices with Micronaut
Java Day 2021, WeAreDevelopers, 2021-09-01, online: Moritz Kammerer (@Moritz Kammerer, Expert Software Engineer at QAware).
== Please download slides in case they are blurred! ===
In this talk, we took a look at how Microservices can be developed with Micronaut. Have a look if it has kept its promises.
Embedded Mirror Maker
An embedded mirror maker is being prototyped to address the large number of dedicated machines currently used for mirroring. The proposed approach would embed the mirroring logic directly in the Kafka brokers to reduce latency, load, and number of machines. It would use idempotent producers, dynamic configuration via Zookeeper, and handle scenarios like leader movement. Challenges include tighter broker/mirror coupling and ensuring message ordering across clusters.
LCA14: LCA14-412: GPGPU on ARM SoC session
This document summarizes a meeting about accelerating SQLite with OpenCL on ARM SoCs. It discusses porting the Clover OpenCL implementation ("Shamrock") to ARM and newer LLVM versions to enable CPU-only OpenCL. Initial SQLite performance testing on ARM showed potential gains from parallelizing operations. The group's goals are to run SQLite's operations on the GPU using OpenCL kernels to achieve significant performance improvements over the CPU. Their current status and next steps involve porting the Khronos OpenCL conformance tests to ARM and updating Shamrock and SQLite to newer OpenCL specifications.
Citi TechTalk Session 2: Kafka Deep Dive
Purpose of the session is to have a dive into Apache, Kafka, Data Streaming and Kafka in the cloud
- Dive into Apache Kafka
- Data Streaming
- Kafka in the cloud
Pinot: Realtime OLAP for 530 Million Users - Sigmod 2018
Pinot is a real-time OLAP data store that can support multiple analytics use cases like interactive dashboards, site facing queries, and anomaly detection in a single system. It achieves this through features like configurable indexes, dynamic query planning and execution, smart data partitioning and routing, and pre-materialized indexes like star-trees that optimize for latency and throughput across different workloads. The document discusses Pinot's architecture and optimizations that enable it to meet the performance requirements of these different use cases.
The Quest for the Perfect API
Discusses my 25-year journey for finding the perfect operating-system interface, covering our work on the Mungi single-address-space operating system (SASOS), early work on L4 microkernels, and now the seL4 microkernel, its evolution and verification.
Talk originally given at a seminar series hosted by VMware Research on occasion of the company's 20st anniversary.
Trends in Systems and How to Get Efficient Performance
In this video from Switzerland HPC Conference, Martin Hilgeman from Dell presents: HPC Workload Efficiency and the Challenges for System Builders.
"With all the advances in massively parallel and multi-core computing with CPUs and accelerators it is often overlooked whether the computational work is being done in an efficient manner. This efficiency is largely being determined at the application level and therefore puts the responsibility of sustaining a certain performance trajectory into the hands of the user. It is observed that the adoption rate of new hardware capabilities is decreasing and lead to a feeling of diminishing returns. This presentation shows the well-known laws of parallel performance from the perspective of a system builder. It also covers through the use of real case studies, examples of how to program for energy efficient parallel application performance."
Watch the video: http://wp.me/p3RLHQ-gIS
Learn more: http://dell.com
and
http://www.hpcadvisorycouncil.com/events/2017/swiss-workshop/agenda.php
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
JVM Memory Model - Yoav Abrahami, Wix
This document discusses the Java Virtual Machine (JVM) memory model and just-in-time (JIT) compilation. It explains that the JVM uses dynamic compilation via a JIT to optimize bytecode at runtime. The JIT profiles code and performs optimizations like inlining, loop unrolling, and escape analysis. It also discusses how the JVM memory model allows for instruction reordering and caching but ensures sequential consistency through happens-before rules and volatile variables. The document provides examples of anomalies that can occur without synchronization and how tools like synchronized, locks, and atomic operations can be used to prevent issues.
Chronix Poster for the Poster Session FAST 2017
Chronix is a domain specific time series database designed for anomaly detection in operational data. It is optimized for the needs of anomaly detection by supporting domain specific data types, analysis algorithms, data models, and query languages. It aims to address limitations of general purpose time series databases by exploiting characteristics of operational data through features like optional pre-computation of extras, timestamp compression, domain specific records and compression techniques, and multi-dimensional storage. An evaluation using data from five industry projects found that Chronix has significantly smaller memory and storage footprints and faster data retrieval and analysis times compared to other time series databases.
Similar to Grafana Mimir and VictoriaMetrics_ Performance Tests.pptx (20)
Pragmatic model checking: from theory to implementations
Pragmatic model checking: from theory to implementations
How to Design a Program Repair Bot? Insights from the Repairnator Project
How to Design a Program Repair Bot? Insights from the Repairnator Project
Common Pitfalls of Functional Programming and How to Avoid Them: A Mobile Gam...
Common Pitfalls of Functional Programming and How to Avoid Them: A Mobile Gam...
Pinot: Realtime OLAP for 530 Million Users - Sigmod 2018
Pinot: Realtime OLAP for 530 Million Users - Sigmod 2018
Trends in Systems and How to Get Efficient Performance
Trends in Systems and How to Get Efficient Performance
Recently uploaded
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
ここ3000字までしか入らないけどタイトルの方がたくさん文字入ると思います。
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/
Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit.
In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing.
van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
Building Production Ready Search Pipelines with Spark and Milvus
Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.
National Security Agency - NSA mobile device best practices
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
Full-RAG: A modern architecture for hyper-personalization
Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.
20240607 QFM018 Elixir Reading List May 2024
Everything I found interesting about the Elixir programming ecosystem in May 2024
Microsoft - Power Platform_G.Aspiotis.pdf
Revolutionizing Application Development
with AI-powered low-code, presentation by George Aspiotis, Sr. Partner Development Manager, Microsoft
TrustArc Webinar - 2024 Global Privacy Survey
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
Mind map of terminologies used in context of Generative AI
Mind map of common terms used in context of Generative AI.
Best 20 SEO Techniques To Improve Website Visibility In SERP
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
UiPath Test Automation using UiPath Test Suite series, part 6
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency.
During the hour, we’ll take you through:
Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board.
Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes.
Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI.
We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI.
This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!
Recently uploaded (20)
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Grafana Mimir and VictoriaMetrics_ Performance Tests.pptx
- 1. Grafana Mimir and VictoriaMetrics Performance Tests Roman Khavronenko | github.com/hagen1778
- 2. Roman Khavronenko Co-founder of VictoriaMetrics Software engineer with experience in distributed systems, monitoring and high-performance services. https://github.com/hagen1778 https://twitter.com/hagen1778
- 3. The High Performance Open Source Time Series Database & Monitoring Solution 120M+ downloads 11k stars in total 200+ contributors Grammarly, CERN, Open Cosmos, Wix Semrush, Roblox
- 5. Stats: ● Active time series: 1Bil ● Ingestion: 50Mil/s Resources: ● CPUs: 7000 ● RAM: 30TiB ● Replicas: 1500
- 7. Disclaimer: Do not simply rely on benchmarks. Run your own tests.
- 8. Methodology
- 14. Mimir allocated resources: grafana/mimir:2.2.0 Component Replicas CPU Mem (GiB) compactor 1 1.2 2 distributor 5 2 6 ingester 5 4 25 querier 4 2 24 query-frontend 1 2 6 store-gateway 1 1 6 various caches 1* 18 Total 43.2 283
- 18. VictoriaMetrics resources: VictoriaMetrics:1.80.0-cluster Component Replicas CPU Mem (GiB) vminsert 4 2 4 vmselect 2 8 16 vmstorage 10 2 16 Total 44 (+1 CPU) 208 (-80 GiB)
- 23. Benchmark Stats: ● Duration: 24h ● Ingestion: 360K samples/s ● Unique time series: 5.5Mil ● Churn rate: 1% each 10 min ● Total number of series over 24h: 13.6Mil ● Total number of samples over 24h: 31Bil
- 24. Samples per second ingested
- 27. Disk space used: local filesystem
- 28. Disk space used: object storage
- 29. Disk space used Mimir: Replication: x3 Local FS: 300 GiB Object storage: 150 GiB -blocks-storage.tsdb.retention-period To reduce disk size at local FS compactor runs compaction jobs at: 2h, 12h and 24h compactor deduplicates replicated data VictoriaMetrics: Replication: x2 Local FS: 50 GiB Downsampling -downsampling.period=30d:5m,180d:1h Retention filters -retentionFilter='{env="dev"}:3d' -retentionFilter='{env="staging"}:30d' -retentionPeriod=1y No object storage support Stores all replicated data
- 30. VictoriaMetrics and Mimir CPU
- 31. VictoriaMetrics and Mimir Memory
- 32. Memory utilization for Mimir components
- 33. Memory utilization for VictoriaMetrics components
- 34. Mimir Read Query latency
- 35. VictoriaMetrics Read Query latency
- 36. Summary after 24h benchmark Mimir (x3 replication): CPU avg used: 20 / 43 cores Mem max used: 120 GiB / 283 GiB Read latency: 50th - 95ms 99th - 35s VictoriaMetrics (x2 replication): CPU avg used: 12 / 44 cores Mem max used: 19 GiB / 208 GiB Read latency: 50th - 165ms 99th - 17s
- 38. Benchmark - x5 increase in load Stats: ● Duration: 3h ● Ingestion: 1.8Mil samples/s (x5) ● Unique time series: 29 Mil (x5) ● Churn rate: 1% each 10 min ● Total number of series over 3h: 32 Mil ● Total number of samples over 3h: 19 Bil
- 39. Samples per second ingested
- 41. CPU utilization for VictoriaMetrics components
- 42. Memory utilization for VictoriaMetrics components
- 43. Summary after 3h benchmark VictoriaMetrics (x2 replication): CPU avg used: 26 / 32 cores Mem max used: 69 GiB / 208 GiB Read latency: 50th - 17s 99th - 120s
- 44. Additional materials ● Benchmark config params ● List of executed alerts on read path ● Mimir helm chart overrides ● VictoriaMetrics helm chart overrides ● Exploring limits benchmark config ● Grafana dashboards snapshots: ○ Prometheus-benchmark writer ○ Mimir and VictoriaMetrics ○ Mimir CPU usage detailed ○ Mimir and VictoriaMetrics: exploring limits ○ VictoriaMetrics cluster dashboard
- 48. Questions? ● https://victoriametrics.com/blog ● https://victoriametrics.com/blog/mimir-benchmark/ ● VictoriaMetrics scaling to 100M samples/s ● The journey of observability infrastructure at Roblox (warn: requires email to share) ● https://github.com/VictoriaMetrics ● https://github.com/hagen1778