SlideShare a Scribd company logo

Histograms at scale - Monitorama 2019

Evan Chan
Evan Chan

Slides for my talk at Monitorama PDX 2019. Histograms have the potential to give us tools to meet SLO/SLAs, quantile measurements, and very rich heatmap displays for debugging. Their promise has not been fulfilled by TSDB backends however. This talk talks about the concept of histograms as first class citizens in storage. What does accuracy mean for histograms? How can we store and compress rich histograms for evaluation and querying at massive scale? How can we fix some of the issues with histograms in Prometheus, such as proper aggregation, bucketing, avoiding clipping, etc.?

1 of 36
Download to read offline
Rich Histograms at Scale:
A New Hope
Evan Chan
@evanfchan
http://github.com/filodb/FiloDB
This is not a contribution
This is not a contribution@evanfchan
What do we do with
Histograms?
This is not a contribution@evanfchan
The Evolution of Histograms
• Pre-aggregated percentiles
Prometheus
InfluxDB
???
Statsd
Graphite
OpenTSDB• Histogram with buckets
• Prometheus histograms
• HDRHistogram
• T-Digests
This is not a contribution@evanfchan
Overlaid Latency Quantiles
This is not a contribution@evanfchan
Now an incident happens…
Ad

Recommended

Degrading Performance? You Might be Suffering From the Small Files Syndrome
Degrading Performance? You Might be Suffering From the Small Files SyndromeDegrading Performance? You Might be Suffering From the Small Files Syndrome
Degrading Performance? You Might be Suffering From the Small Files SyndromeDatabricks
 
InfluxDB IOx Tech Talks: Intro to the InfluxDB IOx Read Buffer - A Read-Optim...
InfluxDB IOx Tech Talks: Intro to the InfluxDB IOx Read Buffer - A Read-Optim...InfluxDB IOx Tech Talks: Intro to the InfluxDB IOx Read Buffer - A Read-Optim...
InfluxDB IOx Tech Talks: Intro to the InfluxDB IOx Read Buffer - A Read-Optim...InfluxData
 
Impacts of Sharding, Partitioning, Encoding, and Sorting on Distributed Query...
Impacts of Sharding, Partitioning, Encoding, and Sorting on Distributed Query...Impacts of Sharding, Partitioning, Encoding, and Sorting on Distributed Query...
Impacts of Sharding, Partitioning, Encoding, and Sorting on Distributed Query...InfluxData
 
Iceberg: a fast table format for S3
Iceberg: a fast table format for S3Iceberg: a fast table format for S3
Iceberg: a fast table format for S3DataWorks Summit
 
Modeling Data and Queries for Wide Column NoSQL
Modeling Data and Queries for Wide Column NoSQLModeling Data and Queries for Wide Column NoSQL
Modeling Data and Queries for Wide Column NoSQLScyllaDB
 
The Proxy Wars - MySQL Router, ProxySQL, MariaDB MaxScale
The Proxy Wars - MySQL Router, ProxySQL, MariaDB MaxScaleThe Proxy Wars - MySQL Router, ProxySQL, MariaDB MaxScale
The Proxy Wars - MySQL Router, ProxySQL, MariaDB MaxScaleColin Charles
 
Scylla Summit 2022: How to Migrate a Counter Table for 68 Billion Records
Scylla Summit 2022: How to Migrate a Counter Table for 68 Billion RecordsScylla Summit 2022: How to Migrate a Counter Table for 68 Billion Records
Scylla Summit 2022: How to Migrate a Counter Table for 68 Billion RecordsScyllaDB
 

More Related Content

What's hot

RocksDB compaction
RocksDB compactionRocksDB compaction
RocksDB compactionMIJIN AN
 
Survey of High Performance NoSQL Systems
Survey of High Performance NoSQL SystemsSurvey of High Performance NoSQL Systems
Survey of High Performance NoSQL SystemsScyllaDB
 
How Prometheus Store the Data
How Prometheus Store the DataHow Prometheus Store the Data
How Prometheus Store the DataHao Chen
 
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...Databricks
 
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the CloudAmazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the CloudNoritaka Sekiyama
 
Goでヤフーの分散オブジェクトストレージを作った話 Go Conference 2017 Spring
Goでヤフーの分散オブジェクトストレージを作った話 Go Conference 2017 SpringGoでヤフーの分散オブジェクトストレージを作った話 Go Conference 2017 Spring
Goでヤフーの分散オブジェクトストレージを作った話 Go Conference 2017 SpringYahoo!デベロッパーネットワーク
 
Some Iceberg Basics for Beginners (CDP).pdf
Some Iceberg Basics for Beginners (CDP).pdfSome Iceberg Basics for Beginners (CDP).pdf
Some Iceberg Basics for Beginners (CDP).pdfMichael Kogan
 
RocksDB detail
RocksDB detailRocksDB detail
RocksDB detailMIJIN AN
 
Dawid Weiss- Finite state automata in lucene
 Dawid Weiss- Finite state automata in lucene Dawid Weiss- Finite state automata in lucene
Dawid Weiss- Finite state automata in luceneLucidworks (Archived)
 
Scylla Summit 2022: Scylla 5.0 New Features, Part 2
Scylla Summit 2022: Scylla 5.0 New Features, Part 2Scylla Summit 2022: Scylla 5.0 New Features, Part 2
Scylla Summit 2022: Scylla 5.0 New Features, Part 2ScyllaDB
 
ORC: 2015 Faster, Better, Smaller
ORC: 2015 Faster, Better, SmallerORC: 2015 Faster, Better, Smaller
ORC: 2015 Faster, Better, SmallerDataWorks Summit
 
Spring Boot Microservices vs Akka Actor Cluster
Spring Boot Microservices vs Akka Actor Cluster Spring Boot Microservices vs Akka Actor Cluster
Spring Boot Microservices vs Akka Actor Cluster OpenCredo
 
iceberg introduction.pptx
iceberg introduction.pptxiceberg introduction.pptx
iceberg introduction.pptxDori Waldman
 
Everything you always wanted to know about Redis but were afraid to ask
Everything you always wanted to know about Redis but were afraid to askEverything you always wanted to know about Redis but were afraid to ask
Everything you always wanted to know about Redis but were afraid to askCarlos Abalde
 
Deep Dive on Amazon Aurora MySQL Performance Tuning (DAT429-R1) - AWS re:Inve...
Deep Dive on Amazon Aurora MySQL Performance Tuning (DAT429-R1) - AWS re:Inve...Deep Dive on Amazon Aurora MySQL Performance Tuning (DAT429-R1) - AWS re:Inve...
Deep Dive on Amazon Aurora MySQL Performance Tuning (DAT429-R1) - AWS re:Inve...Amazon Web Services
 
HyperLogLog in Hive - How to count sheep efficiently?
HyperLogLog in Hive - How to count sheep efficiently?HyperLogLog in Hive - How to count sheep efficiently?
HyperLogLog in Hive - How to count sheep efficiently?bzamecnik
 
ClickHouse Introduction by Alexander Zaitsev, Altinity CTO
ClickHouse Introduction by Alexander Zaitsev, Altinity CTOClickHouse Introduction by Alexander Zaitsev, Altinity CTO
ClickHouse Introduction by Alexander Zaitsev, Altinity CTOAltinity Ltd
 
HBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBaseHBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBaseenissoz
 
SQLアンチパターン(インデックスショットガン)
SQLアンチパターン(インデックスショットガン)SQLアンチパターン(インデックスショットガン)
SQLアンチパターン(インデックスショットガン)Tomoaki Uchida
 
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of Facebook
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookTech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of Facebook
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookThe Hive
 

What's hot (20)

RocksDB compaction
RocksDB compactionRocksDB compaction
RocksDB compaction
 
Survey of High Performance NoSQL Systems
Survey of High Performance NoSQL SystemsSurvey of High Performance NoSQL Systems
Survey of High Performance NoSQL Systems
 
How Prometheus Store the Data
How Prometheus Store the DataHow Prometheus Store the Data
How Prometheus Store the Data
 
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
 
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the CloudAmazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
 
Goでヤフーの分散オブジェクトストレージを作った話 Go Conference 2017 Spring
Goでヤフーの分散オブジェクトストレージを作った話 Go Conference 2017 SpringGoでヤフーの分散オブジェクトストレージを作った話 Go Conference 2017 Spring
Goでヤフーの分散オブジェクトストレージを作った話 Go Conference 2017 Spring
 
Some Iceberg Basics for Beginners (CDP).pdf
Some Iceberg Basics for Beginners (CDP).pdfSome Iceberg Basics for Beginners (CDP).pdf
Some Iceberg Basics for Beginners (CDP).pdf
 
RocksDB detail
RocksDB detailRocksDB detail
RocksDB detail
 
Dawid Weiss- Finite state automata in lucene
 Dawid Weiss- Finite state automata in lucene Dawid Weiss- Finite state automata in lucene
Dawid Weiss- Finite state automata in lucene
 
Scylla Summit 2022: Scylla 5.0 New Features, Part 2
Scylla Summit 2022: Scylla 5.0 New Features, Part 2Scylla Summit 2022: Scylla 5.0 New Features, Part 2
Scylla Summit 2022: Scylla 5.0 New Features, Part 2
 
ORC: 2015 Faster, Better, Smaller
ORC: 2015 Faster, Better, SmallerORC: 2015 Faster, Better, Smaller
ORC: 2015 Faster, Better, Smaller
 
Spring Boot Microservices vs Akka Actor Cluster
Spring Boot Microservices vs Akka Actor Cluster Spring Boot Microservices vs Akka Actor Cluster
Spring Boot Microservices vs Akka Actor Cluster
 
iceberg introduction.pptx
iceberg introduction.pptxiceberg introduction.pptx
iceberg introduction.pptx
 
Everything you always wanted to know about Redis but were afraid to ask
Everything you always wanted to know about Redis but were afraid to askEverything you always wanted to know about Redis but were afraid to ask
Everything you always wanted to know about Redis but were afraid to ask
 
Deep Dive on Amazon Aurora MySQL Performance Tuning (DAT429-R1) - AWS re:Inve...
Deep Dive on Amazon Aurora MySQL Performance Tuning (DAT429-R1) - AWS re:Inve...Deep Dive on Amazon Aurora MySQL Performance Tuning (DAT429-R1) - AWS re:Inve...
Deep Dive on Amazon Aurora MySQL Performance Tuning (DAT429-R1) - AWS re:Inve...
 
HyperLogLog in Hive - How to count sheep efficiently?
HyperLogLog in Hive - How to count sheep efficiently?HyperLogLog in Hive - How to count sheep efficiently?
HyperLogLog in Hive - How to count sheep efficiently?
 
ClickHouse Introduction by Alexander Zaitsev, Altinity CTO
ClickHouse Introduction by Alexander Zaitsev, Altinity CTOClickHouse Introduction by Alexander Zaitsev, Altinity CTO
ClickHouse Introduction by Alexander Zaitsev, Altinity CTO
 
HBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBaseHBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBase
 
SQLアンチパターン(インデックスショットガン)
SQLアンチパターン(インデックスショットガン)SQLアンチパターン(インデックスショットガン)
SQLアンチパターン(インデックスショットガン)
 
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of Facebook
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookTech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of Facebook
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of Facebook
 

Similar to Histograms at scale - Monitorama 2019

FiloDB: Reactive, Real-Time, In-Memory Time Series at Scale
FiloDB: Reactive, Real-Time, In-Memory Time Series at ScaleFiloDB: Reactive, Real-Time, In-Memory Time Series at Scale
FiloDB: Reactive, Real-Time, In-Memory Time Series at ScaleEvan Chan
 
A Production Quality Sketching Library for the Analysis of Big Data
A Production Quality Sketching Library for the Analysis of Big DataA Production Quality Sketching Library for the Analysis of Big Data
A Production Quality Sketching Library for the Analysis of Big DataDatabricks
 
Online statistical analysis using transducers and sketch algorithms
Online statistical analysis using transducers and sketch algorithmsOnline statistical analysis using transducers and sketch algorithms
Online statistical analysis using transducers and sketch algorithmsSimon Belak
 
Index conf sparkml-feb20-n-pentreath
Index conf sparkml-feb20-n-pentreathIndex conf sparkml-feb20-n-pentreath
Index conf sparkml-feb20-n-pentreathChester Chen
 
Avoiding big data antipatterns
Avoiding big data antipatternsAvoiding big data antipatterns
Avoiding big data antipatternsgrepalex
 
2013 06-03 berlin buzzwords
2013 06-03 berlin buzzwords2013 06-03 berlin buzzwords
2013 06-03 berlin buzzwordsNitay Joffe
 
2013.09.10 Giraph at London Hadoop Users Group
2013.09.10 Giraph at London Hadoop Users Group2013.09.10 Giraph at London Hadoop Users Group
2013.09.10 Giraph at London Hadoop Users GroupNitay Joffe
 
2014.02.13 (Strata) Graph Analysis with One Trillion Edges on Apache Giraph
2014.02.13 (Strata) Graph Analysis with One Trillion Edges on Apache Giraph2014.02.13 (Strata) Graph Analysis with One Trillion Edges on Apache Giraph
2014.02.13 (Strata) Graph Analysis with One Trillion Edges on Apache GiraphAvery Ching
 
HBaseCon2017 Improving HBase availability in a multi tenant environment
HBaseCon2017 Improving HBase availability in a multi tenant environmentHBaseCon2017 Improving HBase availability in a multi tenant environment
HBaseCon2017 Improving HBase availability in a multi tenant environmentHBaseCon
 
Thorny Path to the Large Scale Graph Processing, Алексей Зиновьев (Тамтэк)
Thorny Path to the Large Scale Graph Processing, Алексей Зиновьев (Тамтэк)Thorny Path to the Large Scale Graph Processing, Алексей Зиновьев (Тамтэк)
Thorny Path to the Large Scale Graph Processing, Алексей Зиновьев (Тамтэк)Ontico
 
Thorny path to the Large-Scale Graph Processing (Highload++, 2014)
Thorny path to the Large-Scale Graph Processing (Highload++, 2014)Thorny path to the Large-Scale Graph Processing (Highload++, 2014)
Thorny path to the Large-Scale Graph Processing (Highload++, 2014)Alexey Zinoviev
 
Sketch algorithms
Sketch algorithmsSketch algorithms
Sketch algorithmsSimon Belak
 
Faster Faster Faster! Datamarts with Hive at Yahoo
Faster Faster Faster! Datamarts with Hive at YahooFaster Faster Faster! Datamarts with Hive at Yahoo
Faster Faster Faster! Datamarts with Hive at YahooMithun Radhakrishnan
 
Faster, Faster, Faster: The True Story of a Mobile Analytics Data Mart on Hive
Faster, Faster, Faster: The True Story of a Mobile Analytics Data Mart on HiveFaster, Faster, Faster: The True Story of a Mobile Analytics Data Mart on Hive
Faster, Faster, Faster: The True Story of a Mobile Analytics Data Mart on HiveDataWorks Summit/Hadoop Summit
 
Scaling ingest pipelines with high performance computing principles - Rajiv K...
Scaling ingest pipelines with high performance computing principles - Rajiv K...Scaling ingest pipelines with high performance computing principles - Rajiv K...
Scaling ingest pipelines with high performance computing principles - Rajiv K...SignalFx
 
Search at Twitter: Presented by Michael Busch, Twitter
Search at Twitter: Presented by Michael Busch, TwitterSearch at Twitter: Presented by Michael Busch, Twitter
Search at Twitter: Presented by Michael Busch, TwitterLucidworks
 
Everything I Ever Learned About JVM Performance Tuning @Twitter
Everything I Ever Learned About JVM Performance Tuning @TwitterEverything I Ever Learned About JVM Performance Tuning @Twitter
Everything I Ever Learned About JVM Performance Tuning @TwitterAttila Szegedi
 
hbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at Alibaba
hbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at Alibabahbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at Alibaba
hbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at AlibabaMichael Stack
 

Similar to Histograms at scale - Monitorama 2019 (20)

FiloDB: Reactive, Real-Time, In-Memory Time Series at Scale
FiloDB: Reactive, Real-Time, In-Memory Time Series at ScaleFiloDB: Reactive, Real-Time, In-Memory Time Series at Scale
FiloDB: Reactive, Real-Time, In-Memory Time Series at Scale
 
PraveenBOUT++
PraveenBOUT++PraveenBOUT++
PraveenBOUT++
 
A Production Quality Sketching Library for the Analysis of Big Data
A Production Quality Sketching Library for the Analysis of Big DataA Production Quality Sketching Library for the Analysis of Big Data
A Production Quality Sketching Library for the Analysis of Big Data
 
Online statistical analysis using transducers and sketch algorithms
Online statistical analysis using transducers and sketch algorithmsOnline statistical analysis using transducers and sketch algorithms
Online statistical analysis using transducers and sketch algorithms
 
Index conf sparkml-feb20-n-pentreath
Index conf sparkml-feb20-n-pentreathIndex conf sparkml-feb20-n-pentreath
Index conf sparkml-feb20-n-pentreath
 
Avoiding big data antipatterns
Avoiding big data antipatternsAvoiding big data antipatterns
Avoiding big data antipatterns
 
2013 06-03 berlin buzzwords
2013 06-03 berlin buzzwords2013 06-03 berlin buzzwords
2013 06-03 berlin buzzwords
 
2013.09.10 Giraph at London Hadoop Users Group
2013.09.10 Giraph at London Hadoop Users Group2013.09.10 Giraph at London Hadoop Users Group
2013.09.10 Giraph at London Hadoop Users Group
 
2014.02.13 (Strata) Graph Analysis with One Trillion Edges on Apache Giraph
2014.02.13 (Strata) Graph Analysis with One Trillion Edges on Apache Giraph2014.02.13 (Strata) Graph Analysis with One Trillion Edges on Apache Giraph
2014.02.13 (Strata) Graph Analysis with One Trillion Edges on Apache Giraph
 
HBaseCon2017 Improving HBase availability in a multi tenant environment
HBaseCon2017 Improving HBase availability in a multi tenant environmentHBaseCon2017 Improving HBase availability in a multi tenant environment
HBaseCon2017 Improving HBase availability in a multi tenant environment
 
Thorny Path to the Large Scale Graph Processing, Алексей Зиновьев (Тамтэк)
Thorny Path to the Large Scale Graph Processing, Алексей Зиновьев (Тамтэк)Thorny Path to the Large Scale Graph Processing, Алексей Зиновьев (Тамтэк)
Thorny Path to the Large Scale Graph Processing, Алексей Зиновьев (Тамтэк)
 
Thorny path to the Large-Scale Graph Processing (Highload++, 2014)
Thorny path to the Large-Scale Graph Processing (Highload++, 2014)Thorny path to the Large-Scale Graph Processing (Highload++, 2014)
Thorny path to the Large-Scale Graph Processing (Highload++, 2014)
 
Sketch algorithms
Sketch algorithmsSketch algorithms
Sketch algorithms
 
L6.sp17.pptx
L6.sp17.pptxL6.sp17.pptx
L6.sp17.pptx
 
Faster Faster Faster! Datamarts with Hive at Yahoo
Faster Faster Faster! Datamarts with Hive at YahooFaster Faster Faster! Datamarts with Hive at Yahoo
Faster Faster Faster! Datamarts with Hive at Yahoo
 
Faster, Faster, Faster: The True Story of a Mobile Analytics Data Mart on Hive
Faster, Faster, Faster: The True Story of a Mobile Analytics Data Mart on HiveFaster, Faster, Faster: The True Story of a Mobile Analytics Data Mart on Hive
Faster, Faster, Faster: The True Story of a Mobile Analytics Data Mart on Hive
 
Scaling ingest pipelines with high performance computing principles - Rajiv K...
Scaling ingest pipelines with high performance computing principles - Rajiv K...Scaling ingest pipelines with high performance computing principles - Rajiv K...
Scaling ingest pipelines with high performance computing principles - Rajiv K...
 
Search at Twitter: Presented by Michael Busch, Twitter
Search at Twitter: Presented by Michael Busch, TwitterSearch at Twitter: Presented by Michael Busch, Twitter
Search at Twitter: Presented by Michael Busch, Twitter
 
Everything I Ever Learned About JVM Performance Tuning @Twitter
Everything I Ever Learned About JVM Performance Tuning @TwitterEverything I Ever Learned About JVM Performance Tuning @Twitter
Everything I Ever Learned About JVM Performance Tuning @Twitter
 
hbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at Alibaba
hbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at Alibabahbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at Alibaba
hbaseconasia2019 Phoenix Improvements and Practices on Cloud HBase at Alibaba
 

More from Evan Chan

Porting a Streaming Pipeline from Scala to Rust
Porting a Streaming Pipeline from Scala to RustPorting a Streaming Pipeline from Scala to Rust
Porting a Streaming Pipeline from Scala to RustEvan Chan
 
Designing Stateful Apps for Cloud and Kubernetes
Designing Stateful Apps for Cloud and KubernetesDesigning Stateful Apps for Cloud and Kubernetes
Designing Stateful Apps for Cloud and KubernetesEvan Chan
 
Building a High-Performance Database with Scala, Akka, and Spark
Building a High-Performance Database with Scala, Akka, and SparkBuilding a High-Performance Database with Scala, Akka, and Spark
Building a High-Performance Database with Scala, Akka, and SparkEvan Chan
 
700 Updatable Queries Per Second: Spark as a Real-Time Web Service
700 Updatable Queries Per Second: Spark as a Real-Time Web Service700 Updatable Queries Per Second: Spark as a Real-Time Web Service
700 Updatable Queries Per Second: Spark as a Real-Time Web ServiceEvan Chan
 
Building Scalable Data Pipelines - 2016 DataPalooza Seattle
Building Scalable Data Pipelines - 2016 DataPalooza SeattleBuilding Scalable Data Pipelines - 2016 DataPalooza Seattle
Building Scalable Data Pipelines - 2016 DataPalooza SeattleEvan Chan
 
FiloDB - Breakthrough OLAP Performance with Cassandra and Spark
FiloDB - Breakthrough OLAP Performance with Cassandra and SparkFiloDB - Breakthrough OLAP Performance with Cassandra and Spark
FiloDB - Breakthrough OLAP Performance with Cassandra and SparkEvan Chan
 
Breakthrough OLAP performance with Cassandra and Spark
Breakthrough OLAP performance with Cassandra and SparkBreakthrough OLAP performance with Cassandra and Spark
Breakthrough OLAP performance with Cassandra and SparkEvan Chan
 
Productionizing Spark and the Spark Job Server
Productionizing Spark and the Spark Job ServerProductionizing Spark and the Spark Job Server
Productionizing Spark and the Spark Job ServerEvan Chan
 
Akka in Production - ScalaDays 2015
Akka in Production - ScalaDays 2015Akka in Production - ScalaDays 2015
Akka in Production - ScalaDays 2015Evan Chan
 
MIT lecture - Socrata Open Data Architecture
MIT lecture - Socrata Open Data ArchitectureMIT lecture - Socrata Open Data Architecture
MIT lecture - Socrata Open Data ArchitectureEvan Chan
 
OLAP with Cassandra and Spark
OLAP with Cassandra and SparkOLAP with Cassandra and Spark
OLAP with Cassandra and SparkEvan Chan
 
Spark Summit 2014: Spark Job Server Talk
Spark Summit 2014:  Spark Job Server TalkSpark Summit 2014:  Spark Job Server Talk
Spark Summit 2014: Spark Job Server TalkEvan Chan
 
Spark Job Server and Spark as a Query Engine (Spark Meetup 5/14)
Spark Job Server and Spark as a Query Engine (Spark Meetup 5/14)Spark Job Server and Spark as a Query Engine (Spark Meetup 5/14)
Spark Job Server and Spark as a Query Engine (Spark Meetup 5/14)Evan Chan
 
Cassandra Day 2014: Interactive Analytics with Cassandra and Spark
Cassandra Day 2014: Interactive Analytics with Cassandra and SparkCassandra Day 2014: Interactive Analytics with Cassandra and Spark
Cassandra Day 2014: Interactive Analytics with Cassandra and SparkEvan Chan
 
Real-time Analytics with Cassandra, Spark, and Shark
Real-time Analytics with Cassandra, Spark, and SharkReal-time Analytics with Cassandra, Spark, and Shark
Real-time Analytics with Cassandra, Spark, and SharkEvan Chan
 

More from Evan Chan (15)

Porting a Streaming Pipeline from Scala to Rust
Porting a Streaming Pipeline from Scala to RustPorting a Streaming Pipeline from Scala to Rust
Porting a Streaming Pipeline from Scala to Rust
 
Designing Stateful Apps for Cloud and Kubernetes
Designing Stateful Apps for Cloud and KubernetesDesigning Stateful Apps for Cloud and Kubernetes
Designing Stateful Apps for Cloud and Kubernetes
 
Building a High-Performance Database with Scala, Akka, and Spark
Building a High-Performance Database with Scala, Akka, and SparkBuilding a High-Performance Database with Scala, Akka, and Spark
Building a High-Performance Database with Scala, Akka, and Spark
 
700 Updatable Queries Per Second: Spark as a Real-Time Web Service
700 Updatable Queries Per Second: Spark as a Real-Time Web Service700 Updatable Queries Per Second: Spark as a Real-Time Web Service
700 Updatable Queries Per Second: Spark as a Real-Time Web Service
 
Building Scalable Data Pipelines - 2016 DataPalooza Seattle
Building Scalable Data Pipelines - 2016 DataPalooza SeattleBuilding Scalable Data Pipelines - 2016 DataPalooza Seattle
Building Scalable Data Pipelines - 2016 DataPalooza Seattle
 
FiloDB - Breakthrough OLAP Performance with Cassandra and Spark
FiloDB - Breakthrough OLAP Performance with Cassandra and SparkFiloDB - Breakthrough OLAP Performance with Cassandra and Spark
FiloDB - Breakthrough OLAP Performance with Cassandra and Spark
 
Breakthrough OLAP performance with Cassandra and Spark
Breakthrough OLAP performance with Cassandra and SparkBreakthrough OLAP performance with Cassandra and Spark
Breakthrough OLAP performance with Cassandra and Spark
 
Productionizing Spark and the Spark Job Server
Productionizing Spark and the Spark Job ServerProductionizing Spark and the Spark Job Server
Productionizing Spark and the Spark Job Server
 
Akka in Production - ScalaDays 2015
Akka in Production - ScalaDays 2015Akka in Production - ScalaDays 2015
Akka in Production - ScalaDays 2015
 
MIT lecture - Socrata Open Data Architecture
MIT lecture - Socrata Open Data ArchitectureMIT lecture - Socrata Open Data Architecture
MIT lecture - Socrata Open Data Architecture
 
OLAP with Cassandra and Spark
OLAP with Cassandra and SparkOLAP with Cassandra and Spark
OLAP with Cassandra and Spark
 
Spark Summit 2014: Spark Job Server Talk
Spark Summit 2014:  Spark Job Server TalkSpark Summit 2014:  Spark Job Server Talk
Spark Summit 2014: Spark Job Server Talk
 
Spark Job Server and Spark as a Query Engine (Spark Meetup 5/14)
Spark Job Server and Spark as a Query Engine (Spark Meetup 5/14)Spark Job Server and Spark as a Query Engine (Spark Meetup 5/14)
Spark Job Server and Spark as a Query Engine (Spark Meetup 5/14)
 
Cassandra Day 2014: Interactive Analytics with Cassandra and Spark
Cassandra Day 2014: Interactive Analytics with Cassandra and SparkCassandra Day 2014: Interactive Analytics with Cassandra and Spark
Cassandra Day 2014: Interactive Analytics with Cassandra and Spark
 
Real-time Analytics with Cassandra, Spark, and Shark
Real-time Analytics with Cassandra, Spark, and SharkReal-time Analytics with Cassandra, Spark, and Shark
Real-time Analytics with Cassandra, Spark, and Shark
 

Recently uploaded

INTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHI
INTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHIINTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHI
INTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHIKiranKandhro1
 
MAXIMUM POWER POINT TRACKING ALGORITHMS APPLIED TO WIND-SOLAR HYBRID SYSTEM
MAXIMUM POWER POINT TRACKING ALGORITHMS APPLIED TO WIND-SOLAR HYBRID SYSTEMMAXIMUM POWER POINT TRACKING ALGORITHMS APPLIED TO WIND-SOLAR HYBRID SYSTEM
MAXIMUM POWER POINT TRACKING ALGORITHMS APPLIED TO WIND-SOLAR HYBRID SYSTEMArunkumar Tulasi
 
Introduction to Machine Learning Unit-1 Notes for II-II Mechanical Engineerin...
Introduction to Machine Learning Unit-1 Notes for II-II Mechanical Engineerin...Introduction to Machine Learning Unit-1 Notes for II-II Mechanical Engineerin...
Introduction to Machine Learning Unit-1 Notes for II-II Mechanical Engineerin...C Sai Kiran
 
Center Enamel is the leading bolted steel tanks manufacturer in China.docx
Center Enamel is the leading bolted steel tanks manufacturer in China.docxCenter Enamel is the leading bolted steel tanks manufacturer in China.docx
Center Enamel is the leading bolted steel tanks manufacturer in China.docxsjzzztc
 
Metrology Measurements and All units PPT
Metrology Measurements and  All units PPTMetrology Measurements and  All units PPT
Metrology Measurements and All units PPTdinesh babu
 
nervous system ppt pptx anatomy system of nerves
nervous system ppt pptx anatomy system of nervesnervous system ppt pptx anatomy system of nerves
nervous system ppt pptx anatomy system of nervesPhebeLois1
 
Nexus - Final Day 12th February 2024.pptx
Nexus - Final Day 12th February 2024.pptxNexus - Final Day 12th February 2024.pptx
Nexus - Final Day 12th February 2024.pptxRohanAgarwal340656
 
Laser And its Application's - Engineering Physics
Laser And its Application's - Engineering PhysicsLaser And its Application's - Engineering Physics
Laser And its Application's - Engineering PhysicsPurva Nikam
 
Microservices: Benefits, drawbacks and are they for me?
Microservices: Benefits, drawbacks and are they for me?Microservices: Benefits, drawbacks and are they for me?
Microservices: Benefits, drawbacks and are they for me?Marian Marinov
 
CDE_Sustainability Performance_20240214.pdf
CDE_Sustainability Performance_20240214.pdfCDE_Sustainability Performance_20240214.pdf
CDE_Sustainability Performance_20240214.pdf8-koi
 
PM24_Oral_Presentation_Template_Guidelines.pptx
PM24_Oral_Presentation_Template_Guidelines.pptxPM24_Oral_Presentation_Template_Guidelines.pptx
PM24_Oral_Presentation_Template_Guidelines.pptxnissamant
 
chap. 3. lipid deterioration oil and fat processign
chap. 3. lipid deterioration oil and fat processignchap. 3. lipid deterioration oil and fat processign
chap. 3. lipid deterioration oil and fat processignteddymebratie
 
GDSC Web Bootcamp - Day - 2 - JavaScript
GDSC Web Bootcamp -  Day - 2   - JavaScriptGDSC Web Bootcamp -  Day - 2   - JavaScript
GDSC Web Bootcamp - Day - 2 - JavaScriptSahithiGurlinka
 
GDSC Google Cloud Study jam Web Bootcamp - Day-4 Session 4
GDSC  Google Cloud Study jam Web Bootcamp - Day-4  Session 4GDSC  Google Cloud Study jam Web Bootcamp - Day-4  Session 4
GDSC Google Cloud Study jam Web Bootcamp - Day-4 Session 4SahithiGurlinka
 
ExtraordinAIre Monthly Newsletter Jan 2024
ExtraordinAIre Monthly Newsletter Jan 2024ExtraordinAIre Monthly Newsletter Jan 2024
ExtraordinAIre Monthly Newsletter Jan 2024Savipriya Raghavendra
 
Sample Case Study of industry 4.0 and its Outcome
Sample Case Study of industry 4.0 and its OutcomeSample Case Study of industry 4.0 and its Outcome
Sample Case Study of industry 4.0 and its OutcomeHarshith A S
 
Introduction and replication to DragonflyDB
Introduction and replication to DragonflyDBIntroduction and replication to DragonflyDB
Introduction and replication to DragonflyDBMarian Marinov
 
Microstrip Bandpass Filter Design using EDA Tolol such as keysight ADS and An...
Microstrip Bandpass Filter Design using EDA Tolol such as keysight ADS and An...Microstrip Bandpass Filter Design using EDA Tolol such as keysight ADS and An...
Microstrip Bandpass Filter Design using EDA Tolol such as keysight ADS and An...GauravBhartie
 
Get start with Machine Learning and Vertexai
Get start with Machine Learning and VertexaiGet start with Machine Learning and Vertexai
Get start with Machine Learning and VertexaiAshishChanchal1
 
【文凭定制】坎特伯雷大学毕业证学历认证
【文凭定制】坎特伯雷大学毕业证学历认证【文凭定制】坎特伯雷大学毕业证学历认证
【文凭定制】坎特伯雷大学毕业证学历认证muvgemo
 

Recently uploaded (20)

INTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHI
INTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHIINTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHI
INTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHI
 
MAXIMUM POWER POINT TRACKING ALGORITHMS APPLIED TO WIND-SOLAR HYBRID SYSTEM
MAXIMUM POWER POINT TRACKING ALGORITHMS APPLIED TO WIND-SOLAR HYBRID SYSTEMMAXIMUM POWER POINT TRACKING ALGORITHMS APPLIED TO WIND-SOLAR HYBRID SYSTEM
MAXIMUM POWER POINT TRACKING ALGORITHMS APPLIED TO WIND-SOLAR HYBRID SYSTEM
 
Introduction to Machine Learning Unit-1 Notes for II-II Mechanical Engineerin...
Introduction to Machine Learning Unit-1 Notes for II-II Mechanical Engineerin...Introduction to Machine Learning Unit-1 Notes for II-II Mechanical Engineerin...
Introduction to Machine Learning Unit-1 Notes for II-II Mechanical Engineerin...
 
Center Enamel is the leading bolted steel tanks manufacturer in China.docx
Center Enamel is the leading bolted steel tanks manufacturer in China.docxCenter Enamel is the leading bolted steel tanks manufacturer in China.docx
Center Enamel is the leading bolted steel tanks manufacturer in China.docx
 
Metrology Measurements and All units PPT
Metrology Measurements and  All units PPTMetrology Measurements and  All units PPT
Metrology Measurements and All units PPT
 
nervous system ppt pptx anatomy system of nerves
nervous system ppt pptx anatomy system of nervesnervous system ppt pptx anatomy system of nerves
nervous system ppt pptx anatomy system of nerves
 
Nexus - Final Day 12th February 2024.pptx
Nexus - Final Day 12th February 2024.pptxNexus - Final Day 12th February 2024.pptx
Nexus - Final Day 12th February 2024.pptx
 
Laser And its Application's - Engineering Physics
Laser And its Application's - Engineering PhysicsLaser And its Application's - Engineering Physics
Laser And its Application's - Engineering Physics
 
Microservices: Benefits, drawbacks and are they for me?
Microservices: Benefits, drawbacks and are they for me?Microservices: Benefits, drawbacks and are they for me?
Microservices: Benefits, drawbacks and are they for me?
 
CDE_Sustainability Performance_20240214.pdf
CDE_Sustainability Performance_20240214.pdfCDE_Sustainability Performance_20240214.pdf
CDE_Sustainability Performance_20240214.pdf
 
PM24_Oral_Presentation_Template_Guidelines.pptx
PM24_Oral_Presentation_Template_Guidelines.pptxPM24_Oral_Presentation_Template_Guidelines.pptx
PM24_Oral_Presentation_Template_Guidelines.pptx
 
chap. 3. lipid deterioration oil and fat processign
chap. 3. lipid deterioration oil and fat processignchap. 3. lipid deterioration oil and fat processign
chap. 3. lipid deterioration oil and fat processign
 
GDSC Web Bootcamp - Day - 2 - JavaScript
GDSC Web Bootcamp -  Day - 2   - JavaScriptGDSC Web Bootcamp -  Day - 2   - JavaScript
GDSC Web Bootcamp - Day - 2 - JavaScript
 
GDSC Google Cloud Study jam Web Bootcamp - Day-4 Session 4
GDSC  Google Cloud Study jam Web Bootcamp - Day-4  Session 4GDSC  Google Cloud Study jam Web Bootcamp - Day-4  Session 4
GDSC Google Cloud Study jam Web Bootcamp - Day-4 Session 4
 
ExtraordinAIre Monthly Newsletter Jan 2024
ExtraordinAIre Monthly Newsletter Jan 2024ExtraordinAIre Monthly Newsletter Jan 2024
ExtraordinAIre Monthly Newsletter Jan 2024
 
Sample Case Study of industry 4.0 and its Outcome
Sample Case Study of industry 4.0 and its OutcomeSample Case Study of industry 4.0 and its Outcome
Sample Case Study of industry 4.0 and its Outcome
 
Introduction and replication to DragonflyDB
Introduction and replication to DragonflyDBIntroduction and replication to DragonflyDB
Introduction and replication to DragonflyDB
 
Microstrip Bandpass Filter Design using EDA Tolol such as keysight ADS and An...
Microstrip Bandpass Filter Design using EDA Tolol such as keysight ADS and An...Microstrip Bandpass Filter Design using EDA Tolol such as keysight ADS and An...
Microstrip Bandpass Filter Design using EDA Tolol such as keysight ADS and An...
 
Get start with Machine Learning and Vertexai
Get start with Machine Learning and VertexaiGet start with Machine Learning and Vertexai
Get start with Machine Learning and Vertexai
 
【文凭定制】坎特伯雷大学毕业证学历认证
【文凭定制】坎特伯雷大学毕业证学历认证【文凭定制】坎特伯雷大学毕业证学历认证
【文凭定制】坎特伯雷大学毕业证学历认证
 

Histograms at scale - Monitorama 2019

  • 1. Rich Histograms at Scale: A New Hope Evan Chan @evanfchan http://github.com/filodb/FiloDB
  • 2. This is not a contribution
  • 3. This is not a contribution@evanfchan What do we do with Histograms?
  • 4. This is not a contribution@evanfchan The Evolution of Histograms • Pre-aggregated percentiles Prometheus InfluxDB ??? Statsd Graphite OpenTSDB• Histogram with buckets • Prometheus histograms • HDRHistogram • T-Digests
  • 5. This is not a contribution@evanfchan Overlaid Latency Quantiles
  • 6. This is not a contribution@evanfchan Now an incident happens…
  • 7. This is not a contribution@evanfchan Heatmaps: Rich Visuals
  • 8. This is not a contribution@evanfchan Grafana Heatmaps • Buckets are scalable to much more input data but needs TSDB support for histogram buckets • Time series: flexible, but Grafana needs to read ALL the raw data
  • 9. This is not a contribution@evanfchan Useful Histograms • Should be aggregatable • Supports quantiles, distributions, other f(x) • Heatmaps - histograms over time • Should be accurate • Should scale and be efficient
  • 10. This is not a contribution@evanfchan Buckets and Accuracy • Max quantile error = bucket width / lowerBound • Exponential buckets = consistent max quantile errors (Good!) • Linear almost never makes sense • Your custom Prom histogram buckets likely have >100% error Histogram Type Max Error % # Buckets Linear 100% 60,000,000 Exponential 99.1% 26 Linear 10% 600,000,000 Exponential 10.0% 188 Example: (1000, 6E10) value range
  • 11. This is not a contribution@evanfchan Configuring your Histograms • Start with the range of values you need: (min, max) • Pick the desired max quantile error % • Think about trading off publish freq for accuracy • # buckets = log(max/min) / log(1 + max_error) • Example: Max error=50%, (1000 to 6E10): numBuckets = Math.log(6E10/1000) / Math.log(1 + 0.50)
 exponentialBuckets(1000, 1 + 0.50, numBuckets)
  • 12. This is not a contribution Histograms at Scale
  • 13. This is not a contribution@evanfchan Histograms as First-Class Citizen • Modeling, transporting, and storing histograms holistically offers many benefits • Scalability — much better storage, network, query speed • Proper aggregations • Better accuracy and features • Adaptable to better histogram designs in the future • Almost nobody is doing this yet
  • 14. This is not a contribution@evanfchan Prometheus Histogram Schema __name__ metric_sum 5 buckets, sum, count per histogram __name__ metric_count __name__ metric_bucket __name__ metric_bucket __name__ metric_bucket __name__ metric_bucket __name__ metric_bucket le 0.5 le 2.0 le 5.0 le 10. le 25. 44 5 0 2 3 5 5 35 6 1 4 6 6 6 50 10 1 5 8 9 10 60 11 2 6 10 11 11 Series1 Series2 Series3 Series4 Series5 Series6 Series7
  • 15. This is not a contribution@evanfchan The Scale Problem with Histograms • My app: 100 metrics, 20 histograms • Assume range of (1000, 6E10). • Notice how histograms dominate the time series! Max error % Num buckets Histogram Series Other Series Total Series 50% 44 882 80 962 10% 188 3762 80 3842 2% 905 18102 80 18182
  • 16. This is not a contribution@evanfchan Mama we got a problem • Actual system: hundreds of millions of metrics, each one has histogram with 64 buckets • Using Prometheus would lead to tens of billions of series
  • 17. This is not a contribution@evanfchan Prometheus: Raw Data __name__ metric_sum __name__ metric_count __name__ metric_bucket __name__ metric_bucket __name__ metric_bucket __name__ metric_bucket __name__ metric_bucket le 0.5 le 2.0 le 5.0 le 10. le 25. Zone Us-west Zone Us-west Zone Us-west Zone Us-west Zone Us-west Zone Us-west Zone Us-west 44 5 0 2 3 5 5
  • 18. This is not a contribution@evanfchan Atomicity Issues • Prom export, scrape does not guarantee grouping of histogram buckets. • Easy to only get part of a histogram • FiloDB is a distributed database. 7 records might end up in 7 different nodes! • Calculating histogram_quantile: talk to 7 nodes for every query!
  • 19. This is not a contribution@evanfchan Single Histogram Schema 5 buckets, sum, count per histogram __name__ metric Sum Count Hist 0.5 2.0 5.0 10. 25. 44 5 0 2 3 5 5 35 6 1 4 6 6 6 50 10 1 5 8 9 10 60 11 2 6 10 11 11 Series1
  • 20. This is not a contribution@evanfchan Single Histogram Raw Data __name__ MetricZone Us-west 44 5 0 2 3 5 5 Sum Count Hist (0.5, 2, 5, 10, 25) • One record, not (n + 2). No distribution problem! • Labels only appear once • Savings proportional to # of histogram buckets • 50x savings for 64 histogram buckets
  • 21. This is not a contribution@evanfchan Much smaller network and disk usage • One time series vs 66 -> 50x network I/O reduction • Single histogram schema in FiloDB uses < 0.2 bytes per histogram bucket Network I/O Bytesper histogram 0 3500 7000 10500 14000 Series/bucket Series/histo Storage cost Bytesperbucket 0 0.4 0.8 1.2 1.6 Series/bucket Series/histo
  • 22. This is not a contribution@evanfchan Optimizing Histograms: Compression • Delta encoding of increasing bucket values 0 2 3 5 5 0 2 1 2 0 1 4 6 6 6 1 3 2 0 0 • Compressed size about 4x-10x better than 1 time series per bucket (64 buckets; FiloDB) • 0.18 bytes/histogram bucket (range: 0.16 - 0.61) FiloDB SingleHistogram 0.18 bytes/bucket Prometheus 1.5 bytes/bucket Raw data 8 bytes/bucket
  • 23. This is not a contribution@evanfchan Optimizing Histograms: Querying (64 Buckets) • histogram_quantile() is more than 100x faster than series-per-bucket • No need for group-by • Localized computation vs needing to jump across 64 bucket time series histogram_quantile() QPS 0 7500 15000 22500 30000 Series/Bucket Series/Histo
  • 24. This is not a contribution Rich Histograms Usability and Correctness
  • 25. This is not a contribution@evanfchan Changing buckets…. sum() • sum(rate(http_req_latency{…..}[5m])) by (le) • Different buckets lead to incorrect sums 2.5 5 10 50 +Infle= 25 100
  • 26. This is not a contribution@evanfchan Holistic Histograms: 
 Correct Sums • Adding histograms holistically allows us to track bucket changes and correctly sum them 2.5 5 10 50 +Infle= 25 100
  • 27. This is not a contribution@evanfchan histogram_quantile clipping • At 20:00, quantile is clipped at 2nd-last bucket of 10.0
  • 28. This is not a contribution@evanfchan histogram_max_quantile • Client sends a max value at each time interval
  • 29. This is not a contribution@evanfchan histogram_max_quantile • Having a known max allows us to interpolate in last bucket • Cannot interpolate to +Inf • https://github.com/filodb/FiloDB/pull/361 2.5 5 10 25 +Infle= 40 0.9
  • 30. This is not a contribution@evanfchan Ad-Hoc Histograms • Just the quantile, min, max from gauges is not that useful • Get heat map for CPU use across k8s containers • histogram(2, 8, container_cpu_usage_seconds_total{….}) • Aggregate histogram across gauges using new histogram() function • Yes Grafana can do heat maps from raw series - but you can only read so many raw time series. :)
  • 31. This is not a contribution@evanfchan Summary: Rich Histograms at Scale • Treating histograms as a first class citizen • Massive savings in storage and network I/O • Solve aggregation and other correctness issues • Move towards T-Digests and future formats
  • 32. Thank you very much! Please reach out to help make useful histograms
 at scale a reality! @evanfchan http://github.com/filodb/FiloDB Monitorama slack: #talk-evan-chan
  • 33. This is not a contribution@evanfchan Example 2: Write size
  • 34. This is not a contribution@evanfchan Heatmap 2: Write Size
  • 35. This is not a contribution@evanfchan Histogram aggregation: Prometheus • Group by is needed for summing histogram buckets due to data model - leak of abstraction • What if dev changes the histogram scheme? (# of buckets, etc.) • Not possible to resolve scheme differences in Prom, since aggregation knows nothing about histograms sum(rate(histogram_bucket{app="foo")[5m])) by (le)
  • 36. This is not a contribution@evanfchan Histogram aggregation: FiloDB • No need for _bucket, but need to select histogram column • No need for group by. Histograms are natively understood and correct aggregations happen sum(rate(histogram{app=“foo”,__col__=“h”)[5m]))