SlideShare a Scribd company logo
1 of 30
Download to read offline
Grafana Loki: Like Prometheus, but for logs.
Tom Wilkie, Feb 2019
Demo
Tom Wilkie VP Product, Grafana Labs
Previously: Kausal, Weaveworks, Google, Acunu, Xensource
Prometheus & Cortex maintainer, mixins authors etc
Twitter: @tom_wilkie Email: tom@grafana.com
Loki is a horizontally-scalable, highly-available, multi-
tenant log aggregation system inspired by Prometheus.
03/18 Project started
12/18 Launched at KubeCon
12/18 #1 on HN for ~12hrs!
01/19 ~5k GitHub stars
https://github.com/grafana/loki
goo.gl/5DEVH6
#0 Simple and cost effective to operate
#1 Integrated with existing observability tools
#2 Cloud Native and Airplane Friendly
#0 Simple to scale
Existing log aggregation systems do full text indexing and support complex queries
DEwMGIwZ => {
time: “2018-01-31 15:41:04”,
job: “frontend”,
env: “dev”,
line: “POST /api/prom/push...”
}
(“time", “2018-01-31 15:41:04”) -> “DEwMGIwZ”
(“job”, “frontend”) -> “DEwMGIwZ”
(“env”, “dev”) -> “DEwMGIwZ”
(“line”, “POST”) -> “DEwMGIwZ”
(“line”, “/api/prom/push”) -> “DEwMGIwZ”
(“line”, “HTTP/1.1”) -> “DEwMGIwZ”
(“line”, “502”) -> “DEwMGIwZ”
(“time", “2018-01-31 15:41:04”) -> “DEwMGIwZ”
(“job”, “frontend”) -> “DEwMGIwZ”
(“env”, “dev”) -> “DEwMGIwZ”
(“line”, “POST”) -> “DEwMGIwZ”
(“line”, “/api/prom/push”) -> “DEwMGIwZ”
(“line”, “HTTP/1.1”) -> “DEwMGIwZ”
(“line”, “502”) -> “DEwMGIwZ”
NodeN
…
Node1
Node0
Existing log aggregation systems do full text indexing and support complex queries
{job=“frontend”, env=“dev”} => {
time: “2018-01-31 15:41:04”,
line: “POST /api/prom/push HTTP/1.1 502 0"
}
Loki doesn’t index the text of the logs, instead grouping entries
into “streams” and indexing those with labels.
#1 Integrated with
existing tools
1. Alert 2. Dashboard 3. Adhoc Query
4. Log Aggregation
5. Distributed Tracing
Fix!
Prometheus’ data model is very simple:
<identifier> → [ (t0, v0), (t1, v1), ... ]
Timestamps are millisecond int64, values are float64
Identifiers are bags of (label, value) pairs:
{job=“foo”, instance=“bar”, ... }
https://www.slideshare.net/Docker/monitoring-the-prometheus-way-julius-voltz-prometheus
Apps
Apps
Apps
apps
k8s
#0 Prometheus talks to k8s to discover list of targets
#1 Target information is “relabelled” to build labels
#2 Metrics are pulled from apps
#3 Target labels added to series labels
What is Relabelling?
Loki’s data model is very similar:
<identifier> → [ (t0, v0), (t1, v1), ... ]
Timestamps are nanosecond floats, values are byte arrays.
Identifiers are the same - label sets.
https://www.slideshare.net/Docker/monitoring-the-prometheus-way-julius-voltz-prometheus
prom
tail
Apps
Apps
Apps
apps
k8s
1. Alert 2. Dashboard 3. Adhoc Query
4. Log Aggregation
5. Distributed Tracing
Fix!
#2 Cloud Native and
Airplane Friendly
Airplane Friendly
prom
tail
Apps
Apps
Apps
Scale out
prom
tail
Apps
Apps
Apps
prom
tail
Apps
Apps
Apps
prom
tail
Apps
Apps
Apps
prom
tail
Apps
Apps
Apps
prom
tail
Apps
Apps
Apps
Microservices
promtail
Containerised Kubernetes Native Cloud Storage
#0 Simple and cost effective to operate
#1 Integrated with existing observability tools
#2 Cloud Native and Airplane Friendly
Demo
Whats next?
LogQL
https://goo.gl/8h58zu
rate(({job=”app”} | “/foo” ! “/foo/bar”)[1m])
extract({job=”default/nginx”}, “code=(d+)”, “co
|> {code >= 500}
sum(extract({job=”app”}, “code=(d+)”))
Launch first beta in ~April
Add Alerts & Rules off logs
Make it easier to get context, ad hoc filtering
Improve clustering & durability
Thanks! Questions?
(we’re hiring)

More Related Content

Similar to Grafana_Loki.pdf

OWASP WTE - Now in the Cloud!
OWASP WTE - Now in the Cloud!OWASP WTE - Now in the Cloud!
OWASP WTE - Now in the Cloud!Matt Tesauro
 
Analyzing data with docker v4
Analyzing data with docker   v4Analyzing data with docker   v4
Analyzing data with docker v4Andreas Dewes
 
A DevOps guide to Kubernetes
A DevOps guide to KubernetesA DevOps guide to Kubernetes
A DevOps guide to KubernetesPaul Czarkowski
 
Kubernetes Java Operator
Kubernetes Java OperatorKubernetes Java Operator
Kubernetes Java OperatorAnthony Dahanne
 
Building Your Docker Tech Stack
Building Your Docker Tech StackBuilding Your Docker Tech Stack
Building Your Docker Tech StackBret Fisher
 
Building your production tech stack for docker container platform
Building your production tech stack for docker container platformBuilding your production tech stack for docker container platform
Building your production tech stack for docker container platformDocker, Inc.
 
UniK - a unikernel compiler and runtime
UniK - a unikernel compiler and runtimeUniK - a unikernel compiler and runtime
UniK - a unikernel compiler and runtimeLee Calcote
 
Containerize! Between Docker and Jube.
Containerize! Between Docker and Jube.Containerize! Between Docker and Jube.
Containerize! Between Docker and Jube.Henryk Konsek
 
From CoreOS to Kubernetes and Concourse CI
From CoreOS to Kubernetes and Concourse CIFrom CoreOS to Kubernetes and Concourse CI
From CoreOS to Kubernetes and Concourse CIDenis Izmaylov
 
Lightweight coding in powerful Cloud Development Environments (DigitalXchange...
Lightweight coding in powerful Cloud Development Environments (DigitalXchange...Lightweight coding in powerful Cloud Development Environments (DigitalXchange...
Lightweight coding in powerful Cloud Development Environments (DigitalXchange...Lucas Jellema
 
The challenge of application distribution - Introduction to Docker (2014 dec ...
The challenge of application distribution - Introduction to Docker (2014 dec ...The challenge of application distribution - Introduction to Docker (2014 dec ...
The challenge of application distribution - Introduction to Docker (2014 dec ...Sébastien Portebois
 
Brainlunch Docker.io
Brainlunch Docker.ioBrainlunch Docker.io
Brainlunch Docker.iocyon GmbH
 
Kubernetes for Java Developers
Kubernetes for Java DevelopersKubernetes for Java Developers
Kubernetes for Java DevelopersAnthony Dahanne
 
Apache Storm 0.9 basic training - Verisign
Apache Storm 0.9 basic training - VerisignApache Storm 0.9 basic training - Verisign
Apache Storm 0.9 basic training - VerisignMichael Noll
 
CoreOS fest 2016 Summary - DevOps BP 2016 June
CoreOS fest 2016 Summary - DevOps BP 2016 JuneCoreOS fest 2016 Summary - DevOps BP 2016 June
CoreOS fest 2016 Summary - DevOps BP 2016 JuneZsolt Molnar
 
Docker + App Container = ocp
Docker + App Container = ocpDocker + App Container = ocp
Docker + App Container = ocpApcera
 
What's New in Docker - February 2017
What's New in Docker - February 2017What's New in Docker - February 2017
What's New in Docker - February 2017Patrick Chanezon
 
Extending DevOps to Big Data Applications with Kubernetes
Extending DevOps to Big Data Applications with KubernetesExtending DevOps to Big Data Applications with Kubernetes
Extending DevOps to Big Data Applications with KubernetesNicola Ferraro
 
DockerCon 15 Keynote - Day 2
DockerCon 15 Keynote - Day 2DockerCon 15 Keynote - Day 2
DockerCon 15 Keynote - Day 2Docker, Inc.
 

Similar to Grafana_Loki.pdf (20)

OWASP WTE - Now in the Cloud!
OWASP WTE - Now in the Cloud!OWASP WTE - Now in the Cloud!
OWASP WTE - Now in the Cloud!
 
Analyzing data with docker v4
Analyzing data with docker   v4Analyzing data with docker   v4
Analyzing data with docker v4
 
A DevOps guide to Kubernetes
A DevOps guide to KubernetesA DevOps guide to Kubernetes
A DevOps guide to Kubernetes
 
Kubernetes Java Operator
Kubernetes Java OperatorKubernetes Java Operator
Kubernetes Java Operator
 
Building Your Docker Tech Stack
Building Your Docker Tech StackBuilding Your Docker Tech Stack
Building Your Docker Tech Stack
 
Building your production tech stack for docker container platform
Building your production tech stack for docker container platformBuilding your production tech stack for docker container platform
Building your production tech stack for docker container platform
 
UniK - a unikernel compiler and runtime
UniK - a unikernel compiler and runtimeUniK - a unikernel compiler and runtime
UniK - a unikernel compiler and runtime
 
Containerize! Between Docker and Jube.
Containerize! Between Docker and Jube.Containerize! Between Docker and Jube.
Containerize! Between Docker and Jube.
 
From CoreOS to Kubernetes and Concourse CI
From CoreOS to Kubernetes and Concourse CIFrom CoreOS to Kubernetes and Concourse CI
From CoreOS to Kubernetes and Concourse CI
 
Lightweight coding in powerful Cloud Development Environments (DigitalXchange...
Lightweight coding in powerful Cloud Development Environments (DigitalXchange...Lightweight coding in powerful Cloud Development Environments (DigitalXchange...
Lightweight coding in powerful Cloud Development Environments (DigitalXchange...
 
The challenge of application distribution - Introduction to Docker (2014 dec ...
The challenge of application distribution - Introduction to Docker (2014 dec ...The challenge of application distribution - Introduction to Docker (2014 dec ...
The challenge of application distribution - Introduction to Docker (2014 dec ...
 
Brainlunch Docker.io
Brainlunch Docker.ioBrainlunch Docker.io
Brainlunch Docker.io
 
Kubernetes for Java Developers
Kubernetes for Java DevelopersKubernetes for Java Developers
Kubernetes for Java Developers
 
Apache Storm 0.9 basic training - Verisign
Apache Storm 0.9 basic training - VerisignApache Storm 0.9 basic training - Verisign
Apache Storm 0.9 basic training - Verisign
 
CoreOS fest 2016 Summary - DevOps BP 2016 June
CoreOS fest 2016 Summary - DevOps BP 2016 JuneCoreOS fest 2016 Summary - DevOps BP 2016 June
CoreOS fest 2016 Summary - DevOps BP 2016 June
 
Docker practical solutions
Docker practical solutionsDocker practical solutions
Docker practical solutions
 
Docker + App Container = ocp
Docker + App Container = ocpDocker + App Container = ocp
Docker + App Container = ocp
 
What's New in Docker - February 2017
What's New in Docker - February 2017What's New in Docker - February 2017
What's New in Docker - February 2017
 
Extending DevOps to Big Data Applications with Kubernetes
Extending DevOps to Big Data Applications with KubernetesExtending DevOps to Big Data Applications with Kubernetes
Extending DevOps to Big Data Applications with Kubernetes
 
DockerCon 15 Keynote - Day 2
DockerCon 15 Keynote - Day 2DockerCon 15 Keynote - Day 2
DockerCon 15 Keynote - Day 2
 

Recently uploaded

办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一
办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一
办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一F La
 
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming PipelinesConf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming PipelinesTimothy Spann
 
Multiple time frame trading analysis -brianshannon.pdf
Multiple time frame trading analysis -brianshannon.pdfMultiple time frame trading analysis -brianshannon.pdf
Multiple time frame trading analysis -brianshannon.pdfchwongval
 
Heart Disease Classification Report: A Data Analysis Project
Heart Disease Classification Report: A Data Analysis ProjectHeart Disease Classification Report: A Data Analysis Project
Heart Disease Classification Report: A Data Analysis ProjectBoston Institute of Analytics
 
9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort service
9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort service9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort service
9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort servicejennyeacort
 
科罗拉多大学波尔得分校毕业证学位证成绩单-可办理
科罗拉多大学波尔得分校毕业证学位证成绩单-可办理科罗拉多大学波尔得分校毕业证学位证成绩单-可办理
科罗拉多大学波尔得分校毕业证学位证成绩单-可办理e4aez8ss
 
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...Boston Institute of Analytics
 
Generative AI for Social Good at Open Data Science East 2024
Generative AI for Social Good at Open Data Science East 2024Generative AI for Social Good at Open Data Science East 2024
Generative AI for Social Good at Open Data Science East 2024Colleen Farrelly
 
Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Cathrine Wilhelmsen
 
Predictive Analysis for Loan Default Presentation : Data Analysis Project PPT
Predictive Analysis for Loan Default  Presentation : Data Analysis Project PPTPredictive Analysis for Loan Default  Presentation : Data Analysis Project PPT
Predictive Analysis for Loan Default Presentation : Data Analysis Project PPTBoston Institute of Analytics
 
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024thyngster
 
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...Thomas Poetter
 
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...Boston Institute of Analytics
 
办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree
办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree
办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degreeyuu sss
 
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档208367051
 
IMA MSN - Medical Students Network (2).pptx
IMA MSN - Medical Students Network (2).pptxIMA MSN - Medical Students Network (2).pptx
IMA MSN - Medical Students Network (2).pptxdolaknnilon
 
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一F La
 
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...Amil Baba Dawood bangali
 
Semantic Shed - Squashing and Squeezing.pptx
Semantic Shed - Squashing and Squeezing.pptxSemantic Shed - Squashing and Squeezing.pptx
Semantic Shed - Squashing and Squeezing.pptxMike Bennett
 
Thiophen Mechanism khhjjjjjjjhhhhhhhhhhh
Thiophen Mechanism khhjjjjjjjhhhhhhhhhhhThiophen Mechanism khhjjjjjjjhhhhhhhhhhh
Thiophen Mechanism khhjjjjjjjhhhhhhhhhhhYasamin16
 

Recently uploaded (20)

办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一
办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一
办理(Vancouver毕业证书)加拿大温哥华岛大学毕业证成绩单原版一比一
 
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming PipelinesConf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
 
Multiple time frame trading analysis -brianshannon.pdf
Multiple time frame trading analysis -brianshannon.pdfMultiple time frame trading analysis -brianshannon.pdf
Multiple time frame trading analysis -brianshannon.pdf
 
Heart Disease Classification Report: A Data Analysis Project
Heart Disease Classification Report: A Data Analysis ProjectHeart Disease Classification Report: A Data Analysis Project
Heart Disease Classification Report: A Data Analysis Project
 
9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort service
9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort service9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort service
9711147426✨Call In girls Gurgaon Sector 31. SCO 25 escort service
 
科罗拉多大学波尔得分校毕业证学位证成绩单-可办理
科罗拉多大学波尔得分校毕业证学位证成绩单-可办理科罗拉多大学波尔得分校毕业证学位证成绩单-可办理
科罗拉多大学波尔得分校毕业证学位证成绩单-可办理
 
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
 
Generative AI for Social Good at Open Data Science East 2024
Generative AI for Social Good at Open Data Science East 2024Generative AI for Social Good at Open Data Science East 2024
Generative AI for Social Good at Open Data Science East 2024
 
Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)
 
Predictive Analysis for Loan Default Presentation : Data Analysis Project PPT
Predictive Analysis for Loan Default  Presentation : Data Analysis Project PPTPredictive Analysis for Loan Default  Presentation : Data Analysis Project PPT
Predictive Analysis for Loan Default Presentation : Data Analysis Project PPT
 
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024
Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024
 
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...
 
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
 
办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree
办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree
办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree
 
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
原版1:1定制南十字星大学毕业证(SCU毕业证)#文凭成绩单#真实留信学历认证永久存档
 
IMA MSN - Medical Students Network (2).pptx
IMA MSN - Medical Students Network (2).pptxIMA MSN - Medical Students Network (2).pptx
IMA MSN - Medical Students Network (2).pptx
 
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一
办理(UWIC毕业证书)英国卡迪夫城市大学毕业证成绩单原版一比一
 
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...
NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...
 
Semantic Shed - Squashing and Squeezing.pptx
Semantic Shed - Squashing and Squeezing.pptxSemantic Shed - Squashing and Squeezing.pptx
Semantic Shed - Squashing and Squeezing.pptx
 
Thiophen Mechanism khhjjjjjjjhhhhhhhhhhh
Thiophen Mechanism khhjjjjjjjhhhhhhhhhhhThiophen Mechanism khhjjjjjjjhhhhhhhhhhh
Thiophen Mechanism khhjjjjjjjhhhhhhhhhhh
 

Grafana_Loki.pdf