Prometheus Multi Tenancy

•

0 likes•779 views

Natan Yellin discusses options for gathering Prometheus metrics from multiple Kubernetes tenants in a multi-tenant environment. There are three main approaches: solving it outside of Prometheus using other tools; using multiple Prometheus instances with a centralized Prometheus; or using a single Prometheus instance with built-in multi-tenancy. The most mature option currently is to use multiple Prometheus instances with a central Prometheus for long-term storage and unified queries. Tools like Thanos, Cortex, and Mimir provide ways to implement this approach.

Technology

Multi-tenant Kubernetes
observability with Prometheus
robusta-dev Natan Yellin aantn
Natan Yellin, robusta.dev

$ whoami
Co-founder of robusta.dev
Multi-cluster Kubernetes observability
Add-on to Prometheus
Substack newsletter: Why this Kubernetes thing?
Natan Yellin aantn
robusta-dev

How should I gather
Prometheus metrics from
all my tenants?
Natan Yellin aantn
robusta-dev

Assumptions
Natan Yellin aantn
Clusters
Namespaces
Virtual clusters (e.g. capsule, kamaji, vcluster)
etc...
1. Many Kubernetes tenants
2. Tenants need some form of isolation
3. We want to monitor with Prometheus
robusta-dev

What should I use?
Natan Yellin aantn
robusta-dev

In the beginning there was one
Natan Yellin aantn
robusta-dev

In the beginning there was one
Natan Yellin aantn
Simple
No security isolation/RBAC
No performance isolation
If tenants are clusters, discovery is
annoying
Advantages:
Disadvantages:
"One team broke Prometheus for
everyone else"
robusta-dev

Then there were many
Natan Yellin aantn
robusta-dev

Then there were many
Natan Yellin aantn
Simple
Security isolation
Performance isolation
Scalable?
No unified queries
No unified management
More resources?
Advantages:
Major Disadvantage:
Minor Disadvantages:
"If you break it, it only breaks for your
product line."
robusta-dev

What we want
Natan Yellin aantn
Isolation
Scalability
Decentralized:
Query all Prometheuses at once
Centralized:
robusta-dev

What else we want?
Natan Yellin aantn
Scalability
Long term storage of metrics
1.
2.
robusta-dev

Three approaches
Natan Yellin aantn
robusta-dev

Solve it outside Prometheus
Natan Yellin aantn
robusta-dev

Solve it outside Prometheus
Natan Yellin aantn
Doesn't touch Prometheus itself
Delegates problem to other tool
Queries need to address one
Prometheus at a time
Key advantages:
Key disadvantage:
robusta-dev

Multiple + Centralized (take 1)
Natan Yellin aantn
robusta-dev

Multiple + central (take 1)
Natan Yellin aantn
Reuses existing Prometheus
Federated can do roll-up
Federated can selectively scrape
With roll-up/selective you can't
actually query all Prometheuses
Scaling
Key advantages:
Key disadvantages:
robusta-dev

Natan Yellin aantn
Disclaimer: Thanos has lots of options, I'm simplifying a little
robusta-dev

Multiple + central (take 2)
Natan Yellin aantn
robusta-dev

Multiple Prometheuses + central Prometheus (take 2)
Natan Yellin aantn
Super scalable!
Reuses existing Prometheus
Very common solution, lots of tooling
No RBAC built-in
Key advantages:
Key disadvantages:
"Most mature option" - most people
robusta-dev

One Prometheus to Rule them All
Natan Yellin aantn
robusta-dev

One Prometheus to Rule them All
Natan Yellin aantn
robusta-dev
Cortex
Grafana Mimir
VictoriaMetrics
TimescaleDB
M3DB
Options:
...

Grafana Mimir
Natan Yellin aantn
robusta-dev
Native multi-tenancy!
Backed by Grafana
Complexity
Key advantages:
Key disadvantages:

Other useful tools
Natan Yellin aantn
Add prom-label-proxy to Thanos
(and others) to enforce RBAC
robusta-dev

Thank you!
Natan Yellin aantn
A special thank you to Shalom Cohen and Evgeny Uklist + Racoons team for
providing inputs
robusta-dev

Questions?
Natan Yellin aantn
robusta-dev

What's hot

Room 1 - 4 - Phạm Tường Chiến & Trần Văn Thắng - Deliver managed Kubernetes C...

Vietnam Open Infrastructure User Group

왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요

Jo Hoon

Author: Oleg Chunikhin, www.eastbanctech.com Kubernetes is a portable open source system for managing and orchestrating containerized cluster applications. Kubernetes solves a number of DevOps related problems out of the box in a simple and unified way – rolling updates and update rollback, canary deployment and other complicated deployment scenarios, scaling, load balancing, service discovery, logging, monitoring, persistent storage management, and much more. You will learn how in less than 30 minutes a reliable self-healing production-ready Kubernetes cluster may be deployed on AWS and used to host and operate multiple environments and applications.

DevOps with Kubernetes

EastBanc Tachnologies

Get these visually appealing Kubernetes Concepts And Architecture PowerPoint Presentation Slides to discuss the process of operating containerized applications. You can display the need for containers by the company with the help of an open-source architecture PPT slideshow. The architecture of containers can be demonstrated with the help of a visually appealing PPT slideshow. The reasons for opting for Kubernetes by an organization can be explained to your teammates with the help of containers PowerPoint infographics. Highlight the roadmap for installing Kubernetes in the organization by using content-ready PPT slides. Take the assistance of visually appealing PPT templates to depict the major advantages of Kubernetes such as improving productivity, the stability of application run, and many more. After that, display 30 60 90 days plan to implement Kubernetes in the organization. Display the key components of Kubernetes with the help of a diagram using this professionally designed cluster architecture PPT layouts. Describe the functionality of each components of Kubernetes. Hence, download Kubernetes architecture PPT slides to easily and efficiently manage the clusters. https://bit.ly/34DWa7x

Kubernetes Concepts And Architecture Powerpoint Presentation Slides

SlideTeam

Introduction to Kubernetes Workshop

Bob Killen

Introduction to kubernetes

Gabriel Carro

Introduction of CCE and DevCloud

Opsta

Kubernetes Basics

Eueung Mulyana

OpenShift is Red Hat's Platform-as-a-Service (PaaS) that lets developers quickly develop, host, and scale Docker container-based applications. OpenShift enables a uniform and standardised approach to container management across all hosting options including AWS/EC2 and other private/public cloud and on/off-premise variants. At this session, you will learn how Red Hat's enterprise clients are using OpenShift to enable their digital transformation initiatives. Examples will cover how realising a hybrid cloud strategy can simplify and reduce the risk of migrating and transitioning application workloads to containers in the cloud. Alex Smith, Solutions Architect, Amazon Web Services, ASEAN Stephen Bylo, Senior Solution Architect, Red Hat Asia Pacific Pte Ltd

Containers Anywhere with OpenShift by Red Hat

Amazon Web Services

https://www.youtube.com/watch?v=YmIAatr3Who Presented at Cloud and AI DevFest GDG Montreal on September 27, 2019. Are you looking to get more flexibility out of your CICD platform? Interested how GitOps fits into the mix? Learn how Argo CD, Workflows, and Events can be combined to craft custom CICD flows. All while staying Kubernetes native, enabling you to leverage existing observability tooling.

Designing a complete ci cd pipeline using argo events, workflow and cd products

Julian Mazzitelli

Istio service mesh introduction

Kyohei Mizumoto

Red Hat OpenShift Container Platform Overview

James Falkner

Kubernetes

Meng-Ze Lee

K8s security best practices

Sharon Vendrov

Introduction to kubernetes

Raffaele Di Fazio

Introduction to the Container Network Interface (CNI)

Weaveworks

SlideTeam presents Kubernetes Docker Container Implementation Ppt PowerPoint Presentation Slide Templates. This PPT slideshow is an ideal virtual expression of the fundamentals of Kubernetes. The smart data-visualizations make this PowerPoint presentation easy-to-understand and perfect to introduce your audience to the container orchestration system. Use our PPT theme to communicate the definition and need for containers or virtual private servers. Communicate the container, and microservices architecture using cutting-edge graphics. Explain the need for and benefits of Kubernetes for an organization. Elucidate the features, architecture, use cases, installation roadmap, and the 30-60-90 day plan in Kubernetes. Use the neat tabular format to compare Kubernetes with docker swarm based on various parameters. Familiarize your viewers with the various components of Kubernetes. Elaborate on what is Kubelet, Kubectl, and Kubeadm with the help of labeled diagrams. This presentation acquaints your audience with the significance of Kubernetes in management, scaling, automating, and deploying computer applications. Hit the download icon and start personalization. https://bit.ly/2L0Ojdu

Kubernetes Docker Container Implementation Ppt PowerPoint Presentation Slide ...

SlideTeam

Introducing An Architectural Deep Dive With Kubernetes And Containers PowerPoint Presentation Slides. Present the need for the containers in an organization with the help of a readily available PPT slideshow. Discuss container architecture, use cases details to make your presentation elaborative. Showcase the features, architecture, installation roadmap, and the 30-60-90 day plan in Kubernetes with the help of modern-designed PPT infographics. Familiarize your viewers with the various components of Kubernetes with the help of content-ready Kubernetes Docker PPT visuals. Make full use of high-quality icons to make your presentation attention-grabbing and meaningful. Compare and contrast Kubernetes with docker swarm based on various parameters with the help of this attention-grabbing PPT slideshow. Elaborate on Kubelet, Kubectl, and Kubeadm with the help of labeled diagrams. Showcase the networking model of Kubernetes, security measures, and the development process with this easy-to-use docker Architecture PowerPoint template. Therefore, hit the download button now to grab this amazing presentation. https://bit.ly/3vtLeFb

An Architectural Deep Dive With Kubernetes And Containers Powerpoint Presenta...

SlideTeam

OpenTelemetry For Architects

Kevin Brockhoff

Cloud Native Landscape (CNCF and OCI)

Chris Aniszczyk

What's hot (20)

Room 1 - 4 - Phạm Tường Chiến & Trần Văn Thắng - Deliver managed Kubernetes C...

왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요

DevOps with Kubernetes

Kubernetes Concepts And Architecture Powerpoint Presentation Slides

Introduction to Kubernetes Workshop

Introduction to kubernetes

Introduction of CCE and DevCloud

Kubernetes Basics

Containers Anywhere with OpenShift by Red Hat

Designing a complete ci cd pipeline using argo events, workflow and cd products

Istio service mesh introduction

Red Hat OpenShift Container Platform Overview

Kubernetes

K8s security best practices

Introduction to kubernetes

Introduction to the Container Network Interface (CNI)

Kubernetes Docker Container Implementation Ppt PowerPoint Presentation Slide ...

An Architectural Deep Dive With Kubernetes And Containers Powerpoint Presenta...

OpenTelemetry For Architects

Cloud Native Landscape (CNCF and OCI)

Similar to Prometheus Multi Tenancy

Oscon 2012 tdd_cassandra

zznate

In this presentation, we will talk about the tools we leveraged and developed, the processes we established in our CI/CD in order to give to our developers fully isolated environments per their needs and run automated tests in a Kubernetes cluster. Presented by Spiros Economakis, a Senior DevOps Engineer and Cloud Integration Lead at Lenses.io Email him at: spiros@lenses.io Follow him on Twitter: @spirosoik

Creating an effective developer experience on Kubernetes

Lenses.io

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1W22OMy. Jerome Petazzoni explains in detail the advantages of immutable servers, then how to implement them with containers in general, and Docker in particular. Filmed at qconnewyork.com. Jerome Petazzoni is a senior engineer at Docker, where he helps others to containerize all the things. In another life he built and operated Xen clouds when EC2 was just the name of a plane, developed a GIS to deploy fiber interconnects through the French subway, managed commando deployments of large-scale video streaming systems in bandwidth-constrained environments such as conference centers.

Easier, Better, Faster, Safer Deployment with Docker and Immutable Containers

C4Media

Speaker: Remco Overdijk Genre & level: Backend, Way of working, Medior Familiar tools like Statsd, Graphite, Nagios, etc. are no longer used in the Cloud, meaning we’ve hitched a new ride: Prometheus, and it’s all about Metrics! “A Metric, The Hitchhiker’s Guide to Prometheus says, is about the most massively useful thing someone doing Monitoring can have. It has great practical value. You can wave your Metric in emergencies as a distress signal, and produce pretty Graphs at the same time.” Don’t Panic, this talk is not about deploying Prometheus, Kubernetes or Vogon Poetry, but all about YOU! How exactly would that work, using metrics for monitoring purposes? Is it really that different from having separate stacks? Can I export 42 as a Metric? How do I migrate from Statsd/Nagios to this new world? What do I do when metrics seem to be insufficient to monitor something? Like a Babel Fish, this talk translates your questions into hands-on tips and tricks on working with Prometheus. Not only for the cloud, but all applications/services in general.

The hitchhiker’s guide to Prometheus

Bol.com Techlab

Prometheus monitoring

Hien Nguyen Van

The hitchhiker’s guide to Prometheus

Bol.com Techlab

OWF: Xen - Open Source Hypervisor Designed for Clouds

The Linux Foundation

An Introduction to Maven

Vadym Lotar

Maven overview

Samuel Langlois

OpenStack Tempest and REST API testing

openstackindia

Breaking down your monolith into microservices — or starting fresh with a microservice architecture on a new project — is only half the battle. The next step is making your containerized microservice application fault tolerant and highly available, but how do you get there? How do you go from your local development machine to a production-grade cluster? See how container orchestration platforms like Kubernetes and Swarm can be essential assets for your production workloads, and learn how to replace bottlenecks and single points of failure with highly available, scalable microservice deployments.

Scalable and Available Services with Docker and Kubernetes

Laura Frank Tacho

This talk describes about the history of testing framework in Ruby. Ruby 2.2 bundles two testing frameworks. They are minitest and test-unit. Do you know why two testing frameworks are bundled? Do you know that test-unit was bundled and then removed from Ruby distribution? If you can't answer these questions and you're interested in testing framework, this talk will help you. This talk describes about the history of bundled testing framework in Ruby and some major testing frameworks for Ruby in chronological order.

The history of testing framework in Ruby

Kouhei Sutou

Recent vulnerabilities like Heartbleed and Shellshock have brought the security practices and security track record of open-source projects into the spotlight. A project’s response to security issues can have a major impact on how much risk end users are exposed to and how the project is perceived in the technology industry. In this talk, we will compare the security practices of key infrastructure projects such as Linux, Xen Project, Ceph, OVS, OpeNebula and others. We will explore the trade-offs of different security practices, such as community trust, competing stakeholder interests, fairness and media coverage of vulnerabilities. Finally, we will explore the evolution of the Xen Project’s security process over the past 3 years as a case study. We will illustrate the trade-offs, pain points and unexpected issues we have experienced, to help other projects and their users understand the potential pit-falls in designing robust security processes. Author Biography Lars Kurth had his first contact with the open source community in 1997 when he worked on various parts of the ARM toolchain. This experience led Lars to become a passionate open source enthusiast who worked with and for many open source communities over the past 17 years. Lars contributed to projects such as GCC, Eclipse, Symbian and Xen and became the open source community manager for Xen.org in 2011 and later chairman of the Xen Project Advisory Board. Lars is an IT generalist with a wide range of skills in software development and methodology. He is experienced in leading and building engineering teams and communities, as well as constructing marketing, product management, and change programs impacting 1000s of users. He has 19 years of industry experience in the infrastructure, tools and mobile sectors working at ARM, Citrix, Nokia and the Symbian Foundation.

OpenNebulaConf2015 1.14 Are Today’s FOSS Security Practices Robust Enough in ...

OpenNebula Project

Upgrade Kubernetes the boring way

Oleksandr Slynko

Robusta -Tool Presentation (DevOps).pptx

Knoldus Inc.

Android Mobile Continuous Integration. UA Mobile 2016.

UA Mobile

Test driven Infrastructure development with Ansible and Molecule

Serena Lorenzini

7 Habits of Highly Effective Jenkins Users

Andrew Bayer

Securing OpenStack and Beyond with Ansible

Major Hayden

Continuous Kernel Integration

Major Hayden

Similar to Prometheus Multi Tenancy (20)

Oscon 2012 tdd_cassandra

Creating an effective developer experience on Kubernetes

Easier, Better, Faster, Safer Deployment with Docker and Immutable Containers

The hitchhiker’s guide to Prometheus

Prometheus monitoring

The hitchhiker’s guide to Prometheus

OWF: Xen - Open Source Hypervisor Designed for Clouds

An Introduction to Maven

Maven overview

OpenStack Tempest and REST API testing

Scalable and Available Services with Docker and Kubernetes

The history of testing framework in Ruby

OpenNebulaConf2015 1.14 Are Today’s FOSS Security Practices Robust Enough in ...

Upgrade Kubernetes the boring way

Robusta -Tool Presentation (DevOps).pptx

Android Mobile Continuous Integration. UA Mobile 2016.

Test driven Infrastructure development with Ansible and Molecule

7 Habits of Highly Effective Jenkins Users

Securing OpenStack and Beyond with Ansible

Continuous Kernel Integration

Recently uploaded

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf

FIDO Alliance

Intrigued by why some of the world's largest companies (Netflix, Google, Cisco, Twitter, Uber etc) are using gRPC? In this demo based talk we delve into the world of gRPC in .Net, what it does and why we should use it. We compare the interface with both Rest and graphQL. We will show you how to implement grpc server-side in .net and in the web. Finally, I will show you how the tooling helps you deliver powerful interfaces and interact with them quickly and simply.

Demystifying gRPC in .Net by John Staveley

John Staveley

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams. Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...

Jeffrey Haguewood

Mission to Decommission: Importance of Decommissioning Products to Increase E...

Product School

Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to: Create a campaign using Mailchimp with merge tags/fields Send an interactive Slack channel message (using buttons) Have the message received by managers and peers along with a test email for review But there’s more: In a second workflow supporting the same use case, you’ll see: Your campaign sent to target colleagues for approval If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team But—if the “Reject” button is pushed, colleagues will be alerted via Slack message Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors. And... Speakers: Akshay Agnihotri, Product Manager Charlie Greenberg, Host

Connector Corner: Automate dynamic content and events by pushing a button

DianaGray10

Assuring Contact Center Experiences for Your Customers With ThousandEyes

ThousandEyes

Speed Wins: From Kafka to APIs in Minutes

confluent

Key Trends Shaping the Future of Infrastructure.pdf

Cheryl Hung

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf

FIDO Alliance

Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”. All of this illustrated with link prediction over knowledge graphs, but the argument is general.

Neuro-symbolic is not enough, we need neuro-*semantic*

Frank van Harmelen

I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.

"Impact of front-end architecture on development cost", Viktor Turskyi

Fwdays

IoT Analytics Company Presentation May 2024

IoTAnalytics

UiPath Test Automation using UiPath Test Suite series, part 3

DianaGray10

AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...

Product School

Knowledge engineering: from people to machines and back

Elena Simperl

ODC, Data Fabric and Architecture User Group

CatarinaPereira64715

The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more. Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/ Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

BookNet Canada

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...

Thierry Lestable

From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...

Product School

Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place. Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects. Here’s what you’ll gain: - Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows. - Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy. - Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency. - Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity. We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic. Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.

Essentials of Automations: Optimizing FME Workflows with Parameters

Safe Software

Recently uploaded (20)

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf

Demystifying gRPC in .Net by John Staveley

Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...

Mission to Decommission: Importance of Decommissioning Products to Increase E...

Connector Corner: Automate dynamic content and events by pushing a button

Assuring Contact Center Experiences for Your Customers With ThousandEyes

Speed Wins: From Kafka to APIs in Minutes

Key Trends Shaping the Future of Infrastructure.pdf

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf

Neuro-symbolic is not enough, we need neuro-*semantic*

"Impact of front-end architecture on development cost", Viktor Turskyi

IoT Analytics Company Presentation May 2024

UiPath Test Automation using UiPath Test Suite series, part 3

AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...

Knowledge engineering: from people to machines and back

ODC, Data Fabric and Architecture User Group

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...

From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...

Essentials of Automations: Optimizing FME Workflows with Parameters

Prometheus Multi Tenancy

1. Multi-tenant Kubernetes observability with Prometheus robusta-dev Natan Yellin aantn Natan Yellin, robusta.dev

2. $ whoami Co-founder of robusta.dev Multi-cluster Kubernetes observability Add-on to Prometheus Substack newsletter: Why this Kubernetes thing? Natan Yellin aantn robusta-dev

3. How should I gather Prometheus metrics from all my tenants? Natan Yellin aantn robusta-dev

4. Assumptions Natan Yellin aantn Clusters Namespaces Virtual clusters (e.g. capsule, kamaji, vcluster) etc... 1. Many Kubernetes tenants 2. Tenants need some form of isolation 3. We want to monitor with Prometheus robusta-dev

5. What should I use? Natan Yellin aantn robusta-dev

6. In the beginning there was one Natan Yellin aantn robusta-dev

7. In the beginning there was one Natan Yellin aantn Simple No security isolation/RBAC No performance isolation If tenants are clusters, discovery is annoying Advantages: Disadvantages: "One team broke Prometheus for everyone else" robusta-dev

8. Then there were many Natan Yellin aantn robusta-dev

9. Then there were many Natan Yellin aantn Simple Security isolation Performance isolation Scalable? No unified queries No unified management More resources? Advantages: Major Disadvantage: Minor Disadvantages: "If you break it, it only breaks for your product line." robusta-dev

10. What we want Natan Yellin aantn Isolation Scalability Decentralized: Query all Prometheuses at once Centralized: robusta-dev

11. What else we want? Natan Yellin aantn Scalability Long term storage of metrics 1. 2. robusta-dev

12. Three approaches Natan Yellin aantn robusta-dev

13. Solve it outside Prometheus Natan Yellin aantn robusta-dev

14. Solve it outside Prometheus Natan Yellin aantn Doesn't touch Prometheus itself Delegates problem to other tool Queries need to address one Prometheus at a time Key advantages: Key disadvantage: robusta-dev

15. Multiple + Centralized (take 1) Natan Yellin aantn robusta-dev

16. Multiple + central (take 1) Natan Yellin aantn Reuses existing Prometheus Federated can do roll-up Federated can selectively scrape With roll-up/selective you can't actually query all Prometheuses Scaling Key advantages: Key disadvantages: robusta-dev

17. Natan Yellin aantn Disclaimer: Thanos has lots of options, I'm simplifying a little robusta-dev

18. Multiple + central (take 2) Natan Yellin aantn robusta-dev

19. Multiple Prometheuses + central Prometheus (take 2) Natan Yellin aantn Super scalable! Reuses existing Prometheus Very common solution, lots of tooling No RBAC built-in Key advantages: Key disadvantages: "Most mature option" - most people robusta-dev

20. One Prometheus to Rule them All Natan Yellin aantn robusta-dev

21. One Prometheus to Rule them All Natan Yellin aantn robusta-dev Cortex Grafana Mimir VictoriaMetrics TimescaleDB M3DB Options: ...

22. Grafana Mimir Natan Yellin aantn robusta-dev Native multi-tenancy! Backed by Grafana Complexity Key advantages: Key disadvantages:

23. Other useful tools Natan Yellin aantn Add prom-label-proxy to Thanos (and others) to enforce RBAC robusta-dev

24. Thank you! Natan Yellin aantn A special thank you to Shalom Cohen and Evgeny Uklist + Racoons team for providing inputs robusta-dev

25. Questions? Natan Yellin aantn robusta-dev

Prometheus Multi Tenancy

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Prometheus Multi Tenancy

Similar to Prometheus Multi Tenancy (20)

Recently uploaded

Recently uploaded (20)

Prometheus Multi Tenancy