SlideShare a Scribd company logo
1 of 40
Download to read offline
Chronix as long term storage for Prometheus
Florian Lautenschlager, Moritz Kammerer
@flolaut, @phxql
Prometheus
Cloud Native Application
Cloud Native Application
Cloud Native Application
Cloud Native Application
Cloud Native Application
Cloud Native Application
Real-time monitoring and alerting for cloud native apps to detect
anomalies close to their occurrence and to initiate measures.
TIMENOW 14 Days
Beyond real-time monitoring of cloud native apps?
Nothing more to do?
Prometheus
Cloud Native Application
Cloud Native Application
Cloud Native Application
Cloud Native Application
Cloud Native Application
Cloud Native Application
TIMENOW THEN
Real-time monitoring and alerting for cloud native apps to detect
anomalies close to their occurrence and to initiate measures.
Lossless long term storage to store
data forever allowing analyses
beyond real-time monitoring!
Chronix
Agenda
■ Some words about Chronix, its Architecture, its Features, and its Performance.
■ How did we built the integration with Prometheus.
■ Showcase: Prometheus, Chronix Ingester, Chronix, and Grafana
Chronix is more than just a simple time series database. It’s a
time series processing tool stack for all purposes.
Time Series Database: What’s that?
■ Definition 1: “A sample s is a tuple of {timestamp, value}, where the
value could be any kind of object.”
■ Definition 2: “A time series T is an arbitrary list of chronological
ordered samples of one value type”.
■ Definition 3: “A chunk C is a chronological ordered part of a time
series.”
■ Definition 4: “A time series database TSDB is a specialized database
for storing and retrieving time series in an efficient and optimized
way”.
s
{t,v}
1
T
{s1,s2}
T
CT
T1
C1,1
C1,2
TSDB
T3C2,2
T1 C2,1
Chronix’ architecture enables both efficient storage of time
series and millisecond range queries.
(1)
Semantic Transformation
(2)
Attributes and Chunks
(3)
Basic Compression
(4)
Multi-Dimensional
Storage
Record
data:<chunk>
attributes
Record
data:compressed
<chunk>
attributes
Record Storage
68 Billion Points
1 Mio. Chunks *
68.000 Points
~ 96% Compression
Optional
The key data type of Chronix is called a record.
It stores a compressed time series chunk and its attributes.
record{
data:compressed{<chunk>}
//technical fields
id: 3dce1de0−...−93fb2e806d19
version: 1501692859622883300
start: 1427457011238
end: 1427471159292
//optional attributes
host: prodI5
process: scheduler
group: jmx
metric: heapMemory.Usage.Used
max: 896.571
}
Data:compressed{<chunk of time series data>}
■ Time Series: timestamp, numeric value
■ Traces: calls, exceptions, …
■ Logs: access, method runtimes
■ Complex data: models, test coverage,
anything else…
Optional attributes
■ Arbitrary attributes for the time series
■ Attributes are indexed
■ Make the chunk searchable
■ Can contain pre-calculated values
Chronix provides specialized aggregations, transformations,
and analyses for time series that are commonly used.
Aggregations
■ Min / Max / Average / Sum / Count
■ Percentile
■ Standard Deviation
■ First / Last
■ Range
Analyses
■ Trend Analysis
Using a linear regression model
■ Outlier Analysis
Using the IQR
■ Frequency Analysis
Check occurrence within a time range
■ Fast Dynamic Time Warping
Time series similarity search
■ Symbolic Aggregate Approximation
Similarity and pattern search
Transformations
■ Bottom/Top n-values
■ Moving average
■ Divide / Scale
■ Downsampling
Many more
Many more
Only scalar values? One size fits all? No! What about logs,
traces, and others? No problem – Just do it yourself!
■ Chronix Time Series
■Time Series framework that is used by Chronix.
■Time Series Types:
■Numeric: Doubles (the time series known to be the default)
■More to come.
public interface TimeSeriesConverter<T> {
/**
* Shall create an object of type T from the given binary time series.
*/
T from(BinaryTimeSeries binaryTimeSeriesChunk, long queryStart, long queryEnd);
/**
* Shall do the conversation of the custom time series T into the binary time series that is
stored.
*/
BinaryTimeSeries to(T timeSeriesChunk);
}
That‘s the easiest way to play with Chronix. A single instance of
Chronix on a single node.
Java 8 (JRE)
Chronix - 0.4
Solr - 6.2.1
Lucene
Solr plugins
8983
Your Computer
Chronix-Query-Handler
Chronix-Ingestion-Handler
Chronix-Retention
OpenTSDB
Prometheus
KairosDB
HTTP
Chronix-Compaction-Handler
Chronix Client
InfluxDB
Graphite
Go
Java
Code-Slide: How to set up Chronix, ask for time series data, and
call some server-side aggregations in Java.
■ Create a connection to Solr and set up Chronix
■ Define and range query and stream its results
■ Call some aggregations
solr = new HttpSolrClient("http://localhost:8913/solr/chronix/")
chronix = new ChronixClient(new MetricTimeSeriesConverter<>(),
new ChronixSolrStorage(200, groupBy, reduce))
query = new SolrQuery("metric:*Load*")
chronix.stream(solr,query)
query.addFilterQuery("function=max,min,count,sdiff")
stream = chronix.stream(solr,query) Signed Difference:
First=20, Last=-100
 -80
Group chunks on a combination
of attributes and reduce them to
a time series.
Get all time series whose
metric contains Load
Compared to other time series databases Chronix‘ results for
our use case are outstanding.
■ We have evaluated Chronix with:
■InfluxDB, OpenTSDB, and KairosDB
■All databases are configured as single node
■ Storage demand for 108 GB of raw csv time
series data.
■Chronix (8.7 GB) saves 20% – 84% of the space
other time series databases.
■ Query times on imported data.
■73% – 92% faster on data retrieval.
■80% – 97% faster on a mix of analyses.
■ Memory footprint: after start, max during
import, max during query mix
■Chronix takes 1.6 times less memory than
the best alternative.
The hard facts. For more details I suggest you to read our
research paper about Chronix.
Florian Lautenschlager, Michael Philippsen, Andreas Kumlehn, Josef Adersberger
Chronix: Long Term Storage and Retrieval Technology for Anomaly Detection in
Operational Data
FAST 2017 (submitted)
17
Let‘s dig into Chronix Ingesters’ internals.
Image Credit: http://www.taringa.net/posts/ciencia-educacion/12656540/La-Filosofia-del-Dr-House-2.html
Big Picture. It’s a simply and scalable architecture.
Prometheus
Standard Prometheus
Installation
Chronix ServerChronix Ingester
• Collects metrics from
various services.
• Writes them to its
default storage
• Writes them using the
standard remote write
interface to Chronix
Ingester
• Collects samples in
batches and writes
them later to Chronix
with an ideal batch size
• Writes checkpoints to
disk to avoid loss of
data.
• Scales easily
• Lossless long term
storage
• Data distribution
(Apache Solr)
• Rich set of analyses
functions for data
analytics beyond real-
time monitoring.
Chronix Chronix
Single Host
Prometheus Chronix ServerChronix Ingester
In-Memory
Everything runs on a single machine. Small. Simple. Beautiful.
S S S B B B
S Sample: {t,v}
B Batch: [{t,v},{t,v},{t,v}]
Single Host
Prometheus
Chronix Server
Chronix Ingester
In-Memory
Once per Prometheus on a single host.
Chronix Ingester
In-Memory
Prometheus
S Sample: {t,v}
B Batch: [{t,v},{t,v},{t,v}]
Single Host
Prometheus
Chronix ServerChronix Ingester
In-Memory
Chronix Ingester Singleton ;-)
Prometheus
S Sample: {t,v}
B Batch: [{t,v},{t,v},{t,v}]
B B B
Single Host
Prometheus
Chronix Server
Chronix Ingester
In-Memory
Chronix Ingester Cloud behind a proxy to serve multiple
Prometheus servers.
Prometheus
S Sample: {t,v}
B Batch: [{t,v},{t,v},{t,v}]
N
G
I
N
X
Chronix Ingester
In-Memory
Prometheus
Prometheus
Single Host
Single Host
Single HostSingle Host
Prometheus
Chronix ServerChronix Ingester
In-Memory
Cloud Mode: Multiple Prometheus Servers, One Chronix Ingester
per Host, A Chronix Server Cloud
Prometheus
N
G
I
N
X
Chronix Ingester
In-Memory
Prometheus
Prometheus Chronix Server Cloud
M
a
s
t
e
r
Architectural Key Factor: The Chronix Ingestor
■ Small Go Program
■Binary Size: 8.5 MB
■Lines of Code: ~ 720 LoC
■Scales easily: Copy, Execute
■ Handles writes from Prometheus
■Just a small configuration:
remote_write: url:
http://<host>:<port>/ingest
■ Batches samples in memory
■Prometheus sends single samples.
■Chronix needs large chunks (n single
samples) to work well
■Max Batch Age
■5M, 12H, ..
■ Crash and restart resilience
■In-memory is dangerous. The Ingester
holds some amount of transient state
■Regularly writes checkpoints of the entire
in-memory state to disk
■Latest checkpoint is loaded on restart
Chronix loves Chunks. Hence the Ingester batches samples.
The data models for Prometheus and Chronix are similar.
■ Prometheus
■Uses so called labels (key-value pairs) to store dimensional values
■Are added dynamically
■Stores samples (pairs of timestamp and scalar value)
■ Chronix
■Uses attributes (key-value pairs) to store dimensional values
■Schema, Schema less, Dynamic Fields, etc.
■Stores samples of timestamp an any value type: scalar, trace, string, etc.
An example Chronix schema to define the available fields.
<?xml version="1.0" encoding="UTF-8" ?>
<schema name="Chronix" version="1.5">
<types>
<fieldType name="int" class="solr.TrieIntField" precisionStep="0" positionIncrementGap="0"/>
<fieldType name="long" class="solr.TrieLongField" precisionStep="0" positionIncrementGap="0"/>
<fieldType name="double" class="solr.TrieDoubleField" precisionStep="0" positionIncrementGap="0"/>
<fieldType name="string" class="solr.StrField" sortMissingLast="true" omitNorms="true"/>
<fieldType name="binary" class="solr.BinaryField"/>
</types>
<fields>
<!-- The required fields -->
<field name="id" type="string" indexed="true" stored="true" required="true"/>
<field name="_version_" type="long" indexed="true" stored="true"/>
<field name="start" type="long" indexed="true" stored="true" required="true"/>
<field name="end" type="long" indexed="true" stored="true" required="true"/>
<field name="data" type="binary" indexed="true" stored="true" required="false"/>
<field name="metric" type="string" indexed="true" stored="true" required="true"/>
<!-- Dynamic field for tags -->
<dynamicField name="*_s" type="string" indexed="true" stored="true"/>
</fields>
<uniqueKey>id</uniqueKey>
<solrQueryParser defaultOperator="OR"/>
</schema>
Definition of types
Available Fields
Prometheus labels are strings. Chronix Ingester creates them in
Chronix Server dynamically using the dynamicField *_s.
Prometheus_Label -> Chronix_Label
host -> host_s
Showcase: Prometheus, Chronix Ingester, Chronix and Grafana
Prometheus Chronix ServerChronix Ingester
In-Memory
S S S
Grafana
B B B
Disk usage: 11 Days of Data
112,815,835 Samples
Prometheus: ~ 786 MB (whole data directory)
Chronix: ~ 265 MB (without compaction)
A few words about performance in our showcase.
Compaction Effects.
Compaction
Points per
Chunk
Amount of
Records
Disk Usage in
MB
Compaction Time in
Seconds
no -1 610355 265 0
yes 100 1422369 357 134
yes 500 284815 187 75
yes 1000 142573 160 93
yes 5000 28850 131 69
yes 10000 14797 126 61
yes 25000 6408 123 61
yes 100000 2051 121 60
yes 500000 920 119 63
Contains about 112 points per chunk without compaction!
A few words about performance in our showcase.
CPU usage: 4 Cores available (= 400 % Max)
A few words about performance in our showcase.
Memory consumption (max. 8 G)
Ingester
Prometheus
Prometheus Configuration
Chronix Default Web-UI
Using the data source plugins for Chronix and Prometheus.
Ingester Health: Everything Green!
Short Term Data in Prometheus.
Long Term Data in Chronix.
See the difference?
Everything is open source and free to everyone.
The code is the truth.
Chronix Website: www.chronix.io
Chronix Github: https://github.com/ChronixDB
- Ingester: https://github.com/ChronixDB/chronix.ingester
Questions?
- Twitter: @ChronixDB, @flolaut, @phxql
- Slack: https://qaware.slack.com/messages/chronix/
Now it’s your turn.
Now it’s your turn.

More Related Content

What's hot

Time Series Data in a Time Series World
Time Series Data in a Time Series WorldTime Series Data in a Time Series World
Time Series Data in a Time Series WorldMapR Technologies
 
Anatomy of an action
Anatomy of an actionAnatomy of an action
Anatomy of an actionGordon Chung
 
Monitoring with Prometheus
Monitoring with PrometheusMonitoring with Prometheus
Monitoring with PrometheusShiao-An Yuan
 
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...Flink Forward
 
Stabilising the jenga tower
Stabilising the jenga towerStabilising the jenga tower
Stabilising the jenga towerGordon Chung
 
Monitoring infrastructure with prometheus
Monitoring infrastructure with prometheusMonitoring infrastructure with prometheus
Monitoring infrastructure with prometheusShahnawaz Saifi
 
Azure Functions - Get rid of your servers, use functions!
Azure Functions - Get rid of your servers, use functions!Azure Functions - Get rid of your servers, use functions!
Azure Functions - Get rid of your servers, use functions!QAware GmbH
 
Cassandra and Storm at Health Market Sceince
Cassandra and Storm at Health Market SceinceCassandra and Storm at Health Market Sceince
Cassandra and Storm at Health Market SceinceP. Taylor Goetz
 
Real Time Data Streaming using Kafka & Storm
Real Time Data Streaming using Kafka & StormReal Time Data Streaming using Kafka & Storm
Real Time Data Streaming using Kafka & StormRan Silberman
 
Need for Time series Database
Need for Time series DatabaseNeed for Time series Database
Need for Time series DatabasePramit Choudhary
 
Build a Complex, Realtime Data Management App with Postgres 14!
Build a Complex, Realtime Data Management App with Postgres 14!Build a Complex, Realtime Data Management App with Postgres 14!
Build a Complex, Realtime Data Management App with Postgres 14!Jonathan Katz
 
Collect distributed application logging using fluentd (EFK stack)
Collect distributed application logging using fluentd (EFK stack)Collect distributed application logging using fluentd (EFK stack)
Collect distributed application logging using fluentd (EFK stack)Marco Pas
 
Monitoring using Prometheus and Grafana
Monitoring using Prometheus and GrafanaMonitoring using Prometheus and Grafana
Monitoring using Prometheus and GrafanaArvind Kumar G.S
 
Breaking Prometheus (Promcon Berlin '16)
Breaking Prometheus (Promcon Berlin '16)Breaking Prometheus (Promcon Berlin '16)
Breaking Prometheus (Promcon Berlin '16)Matthew Campbell
 
Monitoring Kubernetes with Prometheus
Monitoring Kubernetes with PrometheusMonitoring Kubernetes with Prometheus
Monitoring Kubernetes with PrometheusGrafana Labs
 
Virtual training Intro to InfluxDB & Telegraf
Virtual training  Intro to InfluxDB & TelegrafVirtual training  Intro to InfluxDB & Telegraf
Virtual training Intro to InfluxDB & TelegrafInfluxData
 
Introduction to Twitter Storm
Introduction to Twitter StormIntroduction to Twitter Storm
Introduction to Twitter StormUwe Printz
 
Distributed real time stream processing- why and how
Distributed real time stream processing- why and howDistributed real time stream processing- why and how
Distributed real time stream processing- why and howPetr Zapletal
 
Gnocchi v3 brownbag
Gnocchi v3 brownbagGnocchi v3 brownbag
Gnocchi v3 brownbagGordon Chung
 
Scaling Up Logging and Metrics
Scaling Up Logging and MetricsScaling Up Logging and Metrics
Scaling Up Logging and MetricsRicardo Lourenço
 

What's hot (20)

Time Series Data in a Time Series World
Time Series Data in a Time Series WorldTime Series Data in a Time Series World
Time Series Data in a Time Series World
 
Anatomy of an action
Anatomy of an actionAnatomy of an action
Anatomy of an action
 
Monitoring with Prometheus
Monitoring with PrometheusMonitoring with Prometheus
Monitoring with Prometheus
 
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...
 
Stabilising the jenga tower
Stabilising the jenga towerStabilising the jenga tower
Stabilising the jenga tower
 
Monitoring infrastructure with prometheus
Monitoring infrastructure with prometheusMonitoring infrastructure with prometheus
Monitoring infrastructure with prometheus
 
Azure Functions - Get rid of your servers, use functions!
Azure Functions - Get rid of your servers, use functions!Azure Functions - Get rid of your servers, use functions!
Azure Functions - Get rid of your servers, use functions!
 
Cassandra and Storm at Health Market Sceince
Cassandra and Storm at Health Market SceinceCassandra and Storm at Health Market Sceince
Cassandra and Storm at Health Market Sceince
 
Real Time Data Streaming using Kafka & Storm
Real Time Data Streaming using Kafka & StormReal Time Data Streaming using Kafka & Storm
Real Time Data Streaming using Kafka & Storm
 
Need for Time series Database
Need for Time series DatabaseNeed for Time series Database
Need for Time series Database
 
Build a Complex, Realtime Data Management App with Postgres 14!
Build a Complex, Realtime Data Management App with Postgres 14!Build a Complex, Realtime Data Management App with Postgres 14!
Build a Complex, Realtime Data Management App with Postgres 14!
 
Collect distributed application logging using fluentd (EFK stack)
Collect distributed application logging using fluentd (EFK stack)Collect distributed application logging using fluentd (EFK stack)
Collect distributed application logging using fluentd (EFK stack)
 
Monitoring using Prometheus and Grafana
Monitoring using Prometheus and GrafanaMonitoring using Prometheus and Grafana
Monitoring using Prometheus and Grafana
 
Breaking Prometheus (Promcon Berlin '16)
Breaking Prometheus (Promcon Berlin '16)Breaking Prometheus (Promcon Berlin '16)
Breaking Prometheus (Promcon Berlin '16)
 
Monitoring Kubernetes with Prometheus
Monitoring Kubernetes with PrometheusMonitoring Kubernetes with Prometheus
Monitoring Kubernetes with Prometheus
 
Virtual training Intro to InfluxDB & Telegraf
Virtual training  Intro to InfluxDB & TelegrafVirtual training  Intro to InfluxDB & Telegraf
Virtual training Intro to InfluxDB & Telegraf
 
Introduction to Twitter Storm
Introduction to Twitter StormIntroduction to Twitter Storm
Introduction to Twitter Storm
 
Distributed real time stream processing- why and how
Distributed real time stream processing- why and howDistributed real time stream processing- why and how
Distributed real time stream processing- why and how
 
Gnocchi v3 brownbag
Gnocchi v3 brownbagGnocchi v3 brownbag
Gnocchi v3 brownbag
 
Scaling Up Logging and Metrics
Scaling Up Logging and MetricsScaling Up Logging and Metrics
Scaling Up Logging and Metrics
 

Similar to Chronix as Long-Term Storage for Prometheus

OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...NETWAYS
 
Chronix: A fast and efficient time series storage based on Apache Solr
Chronix: A fast and efficient time series storage based on Apache SolrChronix: A fast and efficient time series storage based on Apache Solr
Chronix: A fast and efficient time series storage based on Apache SolrFlorian Lautenschlager
 
Monitoring, the Prometheus Way - Julius Voltz, Prometheus
Monitoring, the Prometheus Way - Julius Voltz, Prometheus Monitoring, the Prometheus Way - Julius Voltz, Prometheus
Monitoring, the Prometheus Way - Julius Voltz, Prometheus Docker, Inc.
 
Time Series Processing with Apache Spark
Time Series Processing with Apache SparkTime Series Processing with Apache Spark
Time Series Processing with Apache SparkJosef Adersberger
 
Time Series Processing with Apache Spark
Time Series Processing with Apache SparkTime Series Processing with Apache Spark
Time Series Processing with Apache SparkQAware GmbH
 
Riga dev day: Lambda architecture at AWS
Riga dev day: Lambda architecture at AWSRiga dev day: Lambda architecture at AWS
Riga dev day: Lambda architecture at AWSAntons Kranga
 
Prometheus - basics
Prometheus - basicsPrometheus - basics
Prometheus - basicsJuraj Hantak
 
Infrastructure & System Monitoring using Prometheus
Infrastructure & System Monitoring using PrometheusInfrastructure & System Monitoring using Prometheus
Infrastructure & System Monitoring using PrometheusMarco Pas
 
[OpenInfra Days Korea 2018] Day 2 - E6 - OpenInfra monitoring with Prometheus
[OpenInfra Days Korea 2018] Day 2 - E6 - OpenInfra monitoring with Prometheus[OpenInfra Days Korea 2018] Day 2 - E6 - OpenInfra monitoring with Prometheus
[OpenInfra Days Korea 2018] Day 2 - E6 - OpenInfra monitoring with PrometheusOpenStack Korea Community
 
Time Series Analysis
Time Series AnalysisTime Series Analysis
Time Series AnalysisQAware GmbH
 
Time Series Processing with Solr and Spark: Presented by Josef Adersberger, Q...
Time Series Processing with Solr and Spark: Presented by Josef Adersberger, Q...Time Series Processing with Solr and Spark: Presented by Josef Adersberger, Q...
Time Series Processing with Solr and Spark: Presented by Josef Adersberger, Q...Lucidworks
 
Новый InterSystems: open-source, митапы, хакатоны
Новый InterSystems: open-source, митапы, хакатоныНовый InterSystems: open-source, митапы, хакатоны
Новый InterSystems: open-source, митапы, хакатоныTimur Safin
 
Monitoring Weave Cloud with Prometheus
Monitoring Weave Cloud with PrometheusMonitoring Weave Cloud with Prometheus
Monitoring Weave Cloud with PrometheusWeaveworks
 
Testing kubernetes and_open_shift_at_scale_20170209
Testing kubernetes and_open_shift_at_scale_20170209Testing kubernetes and_open_shift_at_scale_20170209
Testing kubernetes and_open_shift_at_scale_20170209mffiedler
 
Exploiting Multicore CPUs Now: Scalability and Reliability for Off-the-shelf ...
Exploiting Multicore CPUs Now: Scalability and Reliability for Off-the-shelf ...Exploiting Multicore CPUs Now: Scalability and Reliability for Off-the-shelf ...
Exploiting Multicore CPUs Now: Scalability and Reliability for Off-the-shelf ...Emery Berger
 
Predictive Maintenance with Deep Learning and Apache Flink
Predictive Maintenance with Deep Learning and Apache FlinkPredictive Maintenance with Deep Learning and Apache Flink
Predictive Maintenance with Deep Learning and Apache FlinkDongwon Kim
 
Monitoring Kubernetes with Prometheus
Monitoring Kubernetes with PrometheusMonitoring Kubernetes with Prometheus
Monitoring Kubernetes with PrometheusGrafana Labs
 
Monitoring in Motion: Monitoring Containers and Amazon ECS
Monitoring in Motion: Monitoring Containers and Amazon ECSMonitoring in Motion: Monitoring Containers and Amazon ECS
Monitoring in Motion: Monitoring Containers and Amazon ECSAmazon Web Services
 
Flink Forward Berlin 2017: Dongwon Kim - Predictive Maintenance with Apache F...
Flink Forward Berlin 2017: Dongwon Kim - Predictive Maintenance with Apache F...Flink Forward Berlin 2017: Dongwon Kim - Predictive Maintenance with Apache F...
Flink Forward Berlin 2017: Dongwon Kim - Predictive Maintenance with Apache F...Flink Forward
 
Where is my cache architectural patterns for caching microservices by example
Where is my cache architectural patterns for caching microservices by exampleWhere is my cache architectural patterns for caching microservices by example
Where is my cache architectural patterns for caching microservices by exampleRafał Leszko
 

Similar to Chronix as Long-Term Storage for Prometheus (20)

OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
 
Chronix: A fast and efficient time series storage based on Apache Solr
Chronix: A fast and efficient time series storage based on Apache SolrChronix: A fast and efficient time series storage based on Apache Solr
Chronix: A fast and efficient time series storage based on Apache Solr
 
Monitoring, the Prometheus Way - Julius Voltz, Prometheus
Monitoring, the Prometheus Way - Julius Voltz, Prometheus Monitoring, the Prometheus Way - Julius Voltz, Prometheus
Monitoring, the Prometheus Way - Julius Voltz, Prometheus
 
Time Series Processing with Apache Spark
Time Series Processing with Apache SparkTime Series Processing with Apache Spark
Time Series Processing with Apache Spark
 
Time Series Processing with Apache Spark
Time Series Processing with Apache SparkTime Series Processing with Apache Spark
Time Series Processing with Apache Spark
 
Riga dev day: Lambda architecture at AWS
Riga dev day: Lambda architecture at AWSRiga dev day: Lambda architecture at AWS
Riga dev day: Lambda architecture at AWS
 
Prometheus - basics
Prometheus - basicsPrometheus - basics
Prometheus - basics
 
Infrastructure & System Monitoring using Prometheus
Infrastructure & System Monitoring using PrometheusInfrastructure & System Monitoring using Prometheus
Infrastructure & System Monitoring using Prometheus
 
[OpenInfra Days Korea 2018] Day 2 - E6 - OpenInfra monitoring with Prometheus
[OpenInfra Days Korea 2018] Day 2 - E6 - OpenInfra monitoring with Prometheus[OpenInfra Days Korea 2018] Day 2 - E6 - OpenInfra monitoring with Prometheus
[OpenInfra Days Korea 2018] Day 2 - E6 - OpenInfra monitoring with Prometheus
 
Time Series Analysis
Time Series AnalysisTime Series Analysis
Time Series Analysis
 
Time Series Processing with Solr and Spark: Presented by Josef Adersberger, Q...
Time Series Processing with Solr and Spark: Presented by Josef Adersberger, Q...Time Series Processing with Solr and Spark: Presented by Josef Adersberger, Q...
Time Series Processing with Solr and Spark: Presented by Josef Adersberger, Q...
 
Новый InterSystems: open-source, митапы, хакатоны
Новый InterSystems: open-source, митапы, хакатоныНовый InterSystems: open-source, митапы, хакатоны
Новый InterSystems: open-source, митапы, хакатоны
 
Monitoring Weave Cloud with Prometheus
Monitoring Weave Cloud with PrometheusMonitoring Weave Cloud with Prometheus
Monitoring Weave Cloud with Prometheus
 
Testing kubernetes and_open_shift_at_scale_20170209
Testing kubernetes and_open_shift_at_scale_20170209Testing kubernetes and_open_shift_at_scale_20170209
Testing kubernetes and_open_shift_at_scale_20170209
 
Exploiting Multicore CPUs Now: Scalability and Reliability for Off-the-shelf ...
Exploiting Multicore CPUs Now: Scalability and Reliability for Off-the-shelf ...Exploiting Multicore CPUs Now: Scalability and Reliability for Off-the-shelf ...
Exploiting Multicore CPUs Now: Scalability and Reliability for Off-the-shelf ...
 
Predictive Maintenance with Deep Learning and Apache Flink
Predictive Maintenance with Deep Learning and Apache FlinkPredictive Maintenance with Deep Learning and Apache Flink
Predictive Maintenance with Deep Learning and Apache Flink
 
Monitoring Kubernetes with Prometheus
Monitoring Kubernetes with PrometheusMonitoring Kubernetes with Prometheus
Monitoring Kubernetes with Prometheus
 
Monitoring in Motion: Monitoring Containers and Amazon ECS
Monitoring in Motion: Monitoring Containers and Amazon ECSMonitoring in Motion: Monitoring Containers and Amazon ECS
Monitoring in Motion: Monitoring Containers and Amazon ECS
 
Flink Forward Berlin 2017: Dongwon Kim - Predictive Maintenance with Apache F...
Flink Forward Berlin 2017: Dongwon Kim - Predictive Maintenance with Apache F...Flink Forward Berlin 2017: Dongwon Kim - Predictive Maintenance with Apache F...
Flink Forward Berlin 2017: Dongwon Kim - Predictive Maintenance with Apache F...
 
Where is my cache architectural patterns for caching microservices by example
Where is my cache architectural patterns for caching microservices by exampleWhere is my cache architectural patterns for caching microservices by example
Where is my cache architectural patterns for caching microservices by example
 

More from QAware GmbH

50 Shades of K8s Autoscaling #JavaLand24.pdf
50 Shades of K8s Autoscaling #JavaLand24.pdf50 Shades of K8s Autoscaling #JavaLand24.pdf
50 Shades of K8s Autoscaling #JavaLand24.pdfQAware GmbH
 
Make Agile Great - PM-Erfahrungen aus zwei virtuellen internationalen SAFe-Pr...
Make Agile Great - PM-Erfahrungen aus zwei virtuellen internationalen SAFe-Pr...Make Agile Great - PM-Erfahrungen aus zwei virtuellen internationalen SAFe-Pr...
Make Agile Great - PM-Erfahrungen aus zwei virtuellen internationalen SAFe-Pr...QAware GmbH
 
Fully-managed Cloud-native Databases: The path to indefinite scale @ CNN Mainz
Fully-managed Cloud-native Databases: The path to indefinite scale @ CNN MainzFully-managed Cloud-native Databases: The path to indefinite scale @ CNN Mainz
Fully-managed Cloud-native Databases: The path to indefinite scale @ CNN MainzQAware GmbH
 
Down the Ivory Tower towards Agile Architecture
Down the Ivory Tower towards Agile ArchitectureDown the Ivory Tower towards Agile Architecture
Down the Ivory Tower towards Agile ArchitectureQAware GmbH
 
"Mixed" Scrum-Teams – Die richtige Mischung macht's!
"Mixed" Scrum-Teams – Die richtige Mischung macht's!"Mixed" Scrum-Teams – Die richtige Mischung macht's!
"Mixed" Scrum-Teams – Die richtige Mischung macht's!QAware GmbH
 
Make Developers Fly: Principles for Platform Engineering
Make Developers Fly: Principles for Platform EngineeringMake Developers Fly: Principles for Platform Engineering
Make Developers Fly: Principles for Platform EngineeringQAware GmbH
 
Der Tod der Testpyramide? – Frontend-Testing mit Playwright
Der Tod der Testpyramide? – Frontend-Testing mit PlaywrightDer Tod der Testpyramide? – Frontend-Testing mit Playwright
Der Tod der Testpyramide? – Frontend-Testing mit PlaywrightQAware GmbH
 
Was kommt nach den SPAs
Was kommt nach den SPAsWas kommt nach den SPAs
Was kommt nach den SPAsQAware GmbH
 
Cloud Migration mit KI: der Turbo
Cloud Migration mit KI: der Turbo Cloud Migration mit KI: der Turbo
Cloud Migration mit KI: der Turbo QAware GmbH
 
Migration von stark regulierten Anwendungen in die Cloud: Dem Teufel die See...
 Migration von stark regulierten Anwendungen in die Cloud: Dem Teufel die See... Migration von stark regulierten Anwendungen in die Cloud: Dem Teufel die See...
Migration von stark regulierten Anwendungen in die Cloud: Dem Teufel die See...QAware GmbH
 
Aus blau wird grün! Ansätze und Technologien für nachhaltige Kubernetes-Cluster
Aus blau wird grün! Ansätze und Technologien für nachhaltige Kubernetes-Cluster Aus blau wird grün! Ansätze und Technologien für nachhaltige Kubernetes-Cluster
Aus blau wird grün! Ansätze und Technologien für nachhaltige Kubernetes-Cluster QAware GmbH
 
Endlich gute API Tests. Boldly Testing APIs Where No One Has Tested Before.
Endlich gute API Tests. Boldly Testing APIs Where No One Has Tested Before.Endlich gute API Tests. Boldly Testing APIs Where No One Has Tested Before.
Endlich gute API Tests. Boldly Testing APIs Where No One Has Tested Before.QAware GmbH
 
Kubernetes with Cilium in AWS - Experience Report!
Kubernetes with Cilium in AWS - Experience Report!Kubernetes with Cilium in AWS - Experience Report!
Kubernetes with Cilium in AWS - Experience Report!QAware GmbH
 
50 Shades of K8s Autoscaling
50 Shades of K8s Autoscaling50 Shades of K8s Autoscaling
50 Shades of K8s AutoscalingQAware GmbH
 
Kontinuierliche Sicherheitstests für APIs mit Testkube und OWASP ZAP
Kontinuierliche Sicherheitstests für APIs mit Testkube und OWASP ZAPKontinuierliche Sicherheitstests für APIs mit Testkube und OWASP ZAP
Kontinuierliche Sicherheitstests für APIs mit Testkube und OWASP ZAPQAware GmbH
 
Service Mesh Pain & Gain. Experiences from a client project.
Service Mesh Pain & Gain. Experiences from a client project.Service Mesh Pain & Gain. Experiences from a client project.
Service Mesh Pain & Gain. Experiences from a client project.QAware GmbH
 
50 Shades of K8s Autoscaling
50 Shades of K8s Autoscaling50 Shades of K8s Autoscaling
50 Shades of K8s AutoscalingQAware GmbH
 
Blue turns green! Approaches and technologies for sustainable K8s clusters.
Blue turns green! Approaches and technologies for sustainable K8s clusters.Blue turns green! Approaches and technologies for sustainable K8s clusters.
Blue turns green! Approaches and technologies for sustainable K8s clusters.QAware GmbH
 
Per Anhalter zu Cloud Nativen API Gateways
Per Anhalter zu Cloud Nativen API GatewaysPer Anhalter zu Cloud Nativen API Gateways
Per Anhalter zu Cloud Nativen API GatewaysQAware GmbH
 
Aus blau wird grün! Ansätze und Technologien für nachhaltige Kubernetes-Cluster
Aus blau wird grün! Ansätze und Technologien für nachhaltige Kubernetes-Cluster Aus blau wird grün! Ansätze und Technologien für nachhaltige Kubernetes-Cluster
Aus blau wird grün! Ansätze und Technologien für nachhaltige Kubernetes-Cluster QAware GmbH
 

More from QAware GmbH (20)

50 Shades of K8s Autoscaling #JavaLand24.pdf
50 Shades of K8s Autoscaling #JavaLand24.pdf50 Shades of K8s Autoscaling #JavaLand24.pdf
50 Shades of K8s Autoscaling #JavaLand24.pdf
 
Make Agile Great - PM-Erfahrungen aus zwei virtuellen internationalen SAFe-Pr...
Make Agile Great - PM-Erfahrungen aus zwei virtuellen internationalen SAFe-Pr...Make Agile Great - PM-Erfahrungen aus zwei virtuellen internationalen SAFe-Pr...
Make Agile Great - PM-Erfahrungen aus zwei virtuellen internationalen SAFe-Pr...
 
Fully-managed Cloud-native Databases: The path to indefinite scale @ CNN Mainz
Fully-managed Cloud-native Databases: The path to indefinite scale @ CNN MainzFully-managed Cloud-native Databases: The path to indefinite scale @ CNN Mainz
Fully-managed Cloud-native Databases: The path to indefinite scale @ CNN Mainz
 
Down the Ivory Tower towards Agile Architecture
Down the Ivory Tower towards Agile ArchitectureDown the Ivory Tower towards Agile Architecture
Down the Ivory Tower towards Agile Architecture
 
"Mixed" Scrum-Teams – Die richtige Mischung macht's!
"Mixed" Scrum-Teams – Die richtige Mischung macht's!"Mixed" Scrum-Teams – Die richtige Mischung macht's!
"Mixed" Scrum-Teams – Die richtige Mischung macht's!
 
Make Developers Fly: Principles for Platform Engineering
Make Developers Fly: Principles for Platform EngineeringMake Developers Fly: Principles for Platform Engineering
Make Developers Fly: Principles for Platform Engineering
 
Der Tod der Testpyramide? – Frontend-Testing mit Playwright
Der Tod der Testpyramide? – Frontend-Testing mit PlaywrightDer Tod der Testpyramide? – Frontend-Testing mit Playwright
Der Tod der Testpyramide? – Frontend-Testing mit Playwright
 
Was kommt nach den SPAs
Was kommt nach den SPAsWas kommt nach den SPAs
Was kommt nach den SPAs
 
Cloud Migration mit KI: der Turbo
Cloud Migration mit KI: der Turbo Cloud Migration mit KI: der Turbo
Cloud Migration mit KI: der Turbo
 
Migration von stark regulierten Anwendungen in die Cloud: Dem Teufel die See...
 Migration von stark regulierten Anwendungen in die Cloud: Dem Teufel die See... Migration von stark regulierten Anwendungen in die Cloud: Dem Teufel die See...
Migration von stark regulierten Anwendungen in die Cloud: Dem Teufel die See...
 
Aus blau wird grün! Ansätze und Technologien für nachhaltige Kubernetes-Cluster
Aus blau wird grün! Ansätze und Technologien für nachhaltige Kubernetes-Cluster Aus blau wird grün! Ansätze und Technologien für nachhaltige Kubernetes-Cluster
Aus blau wird grün! Ansätze und Technologien für nachhaltige Kubernetes-Cluster
 
Endlich gute API Tests. Boldly Testing APIs Where No One Has Tested Before.
Endlich gute API Tests. Boldly Testing APIs Where No One Has Tested Before.Endlich gute API Tests. Boldly Testing APIs Where No One Has Tested Before.
Endlich gute API Tests. Boldly Testing APIs Where No One Has Tested Before.
 
Kubernetes with Cilium in AWS - Experience Report!
Kubernetes with Cilium in AWS - Experience Report!Kubernetes with Cilium in AWS - Experience Report!
Kubernetes with Cilium in AWS - Experience Report!
 
50 Shades of K8s Autoscaling
50 Shades of K8s Autoscaling50 Shades of K8s Autoscaling
50 Shades of K8s Autoscaling
 
Kontinuierliche Sicherheitstests für APIs mit Testkube und OWASP ZAP
Kontinuierliche Sicherheitstests für APIs mit Testkube und OWASP ZAPKontinuierliche Sicherheitstests für APIs mit Testkube und OWASP ZAP
Kontinuierliche Sicherheitstests für APIs mit Testkube und OWASP ZAP
 
Service Mesh Pain & Gain. Experiences from a client project.
Service Mesh Pain & Gain. Experiences from a client project.Service Mesh Pain & Gain. Experiences from a client project.
Service Mesh Pain & Gain. Experiences from a client project.
 
50 Shades of K8s Autoscaling
50 Shades of K8s Autoscaling50 Shades of K8s Autoscaling
50 Shades of K8s Autoscaling
 
Blue turns green! Approaches and technologies for sustainable K8s clusters.
Blue turns green! Approaches and technologies for sustainable K8s clusters.Blue turns green! Approaches and technologies for sustainable K8s clusters.
Blue turns green! Approaches and technologies for sustainable K8s clusters.
 
Per Anhalter zu Cloud Nativen API Gateways
Per Anhalter zu Cloud Nativen API GatewaysPer Anhalter zu Cloud Nativen API Gateways
Per Anhalter zu Cloud Nativen API Gateways
 
Aus blau wird grün! Ansätze und Technologien für nachhaltige Kubernetes-Cluster
Aus blau wird grün! Ansätze und Technologien für nachhaltige Kubernetes-Cluster Aus blau wird grün! Ansätze und Technologien für nachhaltige Kubernetes-Cluster
Aus blau wird grün! Ansätze und Technologien für nachhaltige Kubernetes-Cluster
 

Recently uploaded

DATA ANALYSIS using various data sets like shoping data set etc
DATA ANALYSIS using various data sets like shoping data set etcDATA ANALYSIS using various data sets like shoping data set etc
DATA ANALYSIS using various data sets like shoping data set etclalithasri22
 
Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Cathrine Wilhelmsen
 
Decoding Patterns: Customer Churn Prediction Data Analysis Project
Decoding Patterns: Customer Churn Prediction Data Analysis ProjectDecoding Patterns: Customer Churn Prediction Data Analysis Project
Decoding Patterns: Customer Churn Prediction Data Analysis ProjectBoston Institute of Analytics
 
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdfEnglish-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdfblazblazml
 
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...Jack Cole
 
Digital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfDigital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfNicoChristianSunaryo
 
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...Boston Institute of Analytics
 
knowledge representation in artificial intelligence
knowledge representation in artificial intelligenceknowledge representation in artificial intelligence
knowledge representation in artificial intelligencePriyadharshiniG41
 
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...Boston Institute of Analytics
 
Digital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing worksDigital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing worksdeepakthakur548787
 
Non Text Magic Studio Magic Design for Presentations L&P.pdf
Non Text Magic Studio Magic Design for Presentations L&P.pdfNon Text Magic Studio Magic Design for Presentations L&P.pdf
Non Text Magic Studio Magic Design for Presentations L&P.pdfPratikPatil591646
 
What To Do For World Nature Conservation Day by Slidesgo.pptx
What To Do For World Nature Conservation Day by Slidesgo.pptxWhat To Do For World Nature Conservation Day by Slidesgo.pptx
What To Do For World Nature Conservation Day by Slidesgo.pptxSimranPal17
 
World Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdf
World Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdfWorld Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdf
World Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdfsimulationsindia
 
modul pembelajaran robotic Workshop _ by Slidesgo.pptx
modul pembelajaran robotic Workshop _ by Slidesgo.pptxmodul pembelajaran robotic Workshop _ by Slidesgo.pptx
modul pembelajaran robotic Workshop _ by Slidesgo.pptxaleedritatuxx
 
Introduction to Mongo DB-open-­‐source, high-­‐performance, document-­‐orient...
Introduction to Mongo DB-open-­‐source, high-­‐performance, document-­‐orient...Introduction to Mongo DB-open-­‐source, high-­‐performance, document-­‐orient...
Introduction to Mongo DB-open-­‐source, high-­‐performance, document-­‐orient...boychatmate1
 
IBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaIBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaManalVerma4
 
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBoston Institute of Analytics
 

Recently uploaded (20)

2023 Survey Shows Dip in High School E-Cigarette Use
2023 Survey Shows Dip in High School E-Cigarette Use2023 Survey Shows Dip in High School E-Cigarette Use
2023 Survey Shows Dip in High School E-Cigarette Use
 
Data Analysis Project: Stroke Prediction
Data Analysis Project: Stroke PredictionData Analysis Project: Stroke Prediction
Data Analysis Project: Stroke Prediction
 
DATA ANALYSIS using various data sets like shoping data set etc
DATA ANALYSIS using various data sets like shoping data set etcDATA ANALYSIS using various data sets like shoping data set etc
DATA ANALYSIS using various data sets like shoping data set etc
 
Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)
 
Insurance Churn Prediction Data Analysis Project
Insurance Churn Prediction Data Analysis ProjectInsurance Churn Prediction Data Analysis Project
Insurance Churn Prediction Data Analysis Project
 
Decoding Patterns: Customer Churn Prediction Data Analysis Project
Decoding Patterns: Customer Churn Prediction Data Analysis ProjectDecoding Patterns: Customer Churn Prediction Data Analysis Project
Decoding Patterns: Customer Churn Prediction Data Analysis Project
 
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdfEnglish-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf
 
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
 
Digital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfDigital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdf
 
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...
 
knowledge representation in artificial intelligence
knowledge representation in artificial intelligenceknowledge representation in artificial intelligence
knowledge representation in artificial intelligence
 
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
 
Digital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing worksDigital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing works
 
Non Text Magic Studio Magic Design for Presentations L&P.pdf
Non Text Magic Studio Magic Design for Presentations L&P.pdfNon Text Magic Studio Magic Design for Presentations L&P.pdf
Non Text Magic Studio Magic Design for Presentations L&P.pdf
 
What To Do For World Nature Conservation Day by Slidesgo.pptx
What To Do For World Nature Conservation Day by Slidesgo.pptxWhat To Do For World Nature Conservation Day by Slidesgo.pptx
What To Do For World Nature Conservation Day by Slidesgo.pptx
 
World Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdf
World Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdfWorld Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdf
World Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdf
 
modul pembelajaran robotic Workshop _ by Slidesgo.pptx
modul pembelajaran robotic Workshop _ by Slidesgo.pptxmodul pembelajaran robotic Workshop _ by Slidesgo.pptx
modul pembelajaran robotic Workshop _ by Slidesgo.pptx
 
Introduction to Mongo DB-open-­‐source, high-­‐performance, document-­‐orient...
Introduction to Mongo DB-open-­‐source, high-­‐performance, document-­‐orient...Introduction to Mongo DB-open-­‐source, high-­‐performance, document-­‐orient...
Introduction to Mongo DB-open-­‐source, high-­‐performance, document-­‐orient...
 
IBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaIBEF report on the Insurance market in India
IBEF report on the Insurance market in India
 
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
 

Chronix as Long-Term Storage for Prometheus

  • 1. Chronix as long term storage for Prometheus Florian Lautenschlager, Moritz Kammerer @flolaut, @phxql
  • 2. Prometheus Cloud Native Application Cloud Native Application Cloud Native Application Cloud Native Application Cloud Native Application Cloud Native Application Real-time monitoring and alerting for cloud native apps to detect anomalies close to their occurrence and to initiate measures. TIMENOW 14 Days
  • 3. Beyond real-time monitoring of cloud native apps? Nothing more to do?
  • 4. Prometheus Cloud Native Application Cloud Native Application Cloud Native Application Cloud Native Application Cloud Native Application Cloud Native Application TIMENOW THEN Real-time monitoring and alerting for cloud native apps to detect anomalies close to their occurrence and to initiate measures. Lossless long term storage to store data forever allowing analyses beyond real-time monitoring! Chronix
  • 5.
  • 6. Agenda ■ Some words about Chronix, its Architecture, its Features, and its Performance. ■ How did we built the integration with Prometheus. ■ Showcase: Prometheus, Chronix Ingester, Chronix, and Grafana
  • 7. Chronix is more than just a simple time series database. It’s a time series processing tool stack for all purposes.
  • 8. Time Series Database: What’s that? ■ Definition 1: “A sample s is a tuple of {timestamp, value}, where the value could be any kind of object.” ■ Definition 2: “A time series T is an arbitrary list of chronological ordered samples of one value type”. ■ Definition 3: “A chunk C is a chronological ordered part of a time series.” ■ Definition 4: “A time series database TSDB is a specialized database for storing and retrieving time series in an efficient and optimized way”. s {t,v} 1 T {s1,s2} T CT T1 C1,1 C1,2 TSDB T3C2,2 T1 C2,1
  • 9. Chronix’ architecture enables both efficient storage of time series and millisecond range queries. (1) Semantic Transformation (2) Attributes and Chunks (3) Basic Compression (4) Multi-Dimensional Storage Record data:<chunk> attributes Record data:compressed <chunk> attributes Record Storage 68 Billion Points 1 Mio. Chunks * 68.000 Points ~ 96% Compression Optional
  • 10. The key data type of Chronix is called a record. It stores a compressed time series chunk and its attributes. record{ data:compressed{<chunk>} //technical fields id: 3dce1de0−...−93fb2e806d19 version: 1501692859622883300 start: 1427457011238 end: 1427471159292 //optional attributes host: prodI5 process: scheduler group: jmx metric: heapMemory.Usage.Used max: 896.571 } Data:compressed{<chunk of time series data>} ■ Time Series: timestamp, numeric value ■ Traces: calls, exceptions, … ■ Logs: access, method runtimes ■ Complex data: models, test coverage, anything else… Optional attributes ■ Arbitrary attributes for the time series ■ Attributes are indexed ■ Make the chunk searchable ■ Can contain pre-calculated values
  • 11. Chronix provides specialized aggregations, transformations, and analyses for time series that are commonly used. Aggregations ■ Min / Max / Average / Sum / Count ■ Percentile ■ Standard Deviation ■ First / Last ■ Range Analyses ■ Trend Analysis Using a linear regression model ■ Outlier Analysis Using the IQR ■ Frequency Analysis Check occurrence within a time range ■ Fast Dynamic Time Warping Time series similarity search ■ Symbolic Aggregate Approximation Similarity and pattern search Transformations ■ Bottom/Top n-values ■ Moving average ■ Divide / Scale ■ Downsampling Many more Many more
  • 12. Only scalar values? One size fits all? No! What about logs, traces, and others? No problem – Just do it yourself! ■ Chronix Time Series ■Time Series framework that is used by Chronix. ■Time Series Types: ■Numeric: Doubles (the time series known to be the default) ■More to come. public interface TimeSeriesConverter<T> { /** * Shall create an object of type T from the given binary time series. */ T from(BinaryTimeSeries binaryTimeSeriesChunk, long queryStart, long queryEnd); /** * Shall do the conversation of the custom time series T into the binary time series that is stored. */ BinaryTimeSeries to(T timeSeriesChunk); }
  • 13. That‘s the easiest way to play with Chronix. A single instance of Chronix on a single node. Java 8 (JRE) Chronix - 0.4 Solr - 6.2.1 Lucene Solr plugins 8983 Your Computer Chronix-Query-Handler Chronix-Ingestion-Handler Chronix-Retention OpenTSDB Prometheus KairosDB HTTP Chronix-Compaction-Handler Chronix Client InfluxDB Graphite Go Java
  • 14. Code-Slide: How to set up Chronix, ask for time series data, and call some server-side aggregations in Java. ■ Create a connection to Solr and set up Chronix ■ Define and range query and stream its results ■ Call some aggregations solr = new HttpSolrClient("http://localhost:8913/solr/chronix/") chronix = new ChronixClient(new MetricTimeSeriesConverter<>(), new ChronixSolrStorage(200, groupBy, reduce)) query = new SolrQuery("metric:*Load*") chronix.stream(solr,query) query.addFilterQuery("function=max,min,count,sdiff") stream = chronix.stream(solr,query) Signed Difference: First=20, Last=-100  -80 Group chunks on a combination of attributes and reduce them to a time series. Get all time series whose metric contains Load
  • 15. Compared to other time series databases Chronix‘ results for our use case are outstanding. ■ We have evaluated Chronix with: ■InfluxDB, OpenTSDB, and KairosDB ■All databases are configured as single node ■ Storage demand for 108 GB of raw csv time series data. ■Chronix (8.7 GB) saves 20% – 84% of the space other time series databases. ■ Query times on imported data. ■73% – 92% faster on data retrieval. ■80% – 97% faster on a mix of analyses. ■ Memory footprint: after start, max during import, max during query mix ■Chronix takes 1.6 times less memory than the best alternative.
  • 16. The hard facts. For more details I suggest you to read our research paper about Chronix. Florian Lautenschlager, Michael Philippsen, Andreas Kumlehn, Josef Adersberger Chronix: Long Term Storage and Retrieval Technology for Anomaly Detection in Operational Data FAST 2017 (submitted)
  • 17. 17 Let‘s dig into Chronix Ingesters’ internals. Image Credit: http://www.taringa.net/posts/ciencia-educacion/12656540/La-Filosofia-del-Dr-House-2.html
  • 18. Big Picture. It’s a simply and scalable architecture. Prometheus Standard Prometheus Installation Chronix ServerChronix Ingester • Collects metrics from various services. • Writes them to its default storage • Writes them using the standard remote write interface to Chronix Ingester • Collects samples in batches and writes them later to Chronix with an ideal batch size • Writes checkpoints to disk to avoid loss of data. • Scales easily • Lossless long term storage • Data distribution (Apache Solr) • Rich set of analyses functions for data analytics beyond real- time monitoring. Chronix Chronix
  • 19. Single Host Prometheus Chronix ServerChronix Ingester In-Memory Everything runs on a single machine. Small. Simple. Beautiful. S S S B B B S Sample: {t,v} B Batch: [{t,v},{t,v},{t,v}]
  • 20. Single Host Prometheus Chronix Server Chronix Ingester In-Memory Once per Prometheus on a single host. Chronix Ingester In-Memory Prometheus S Sample: {t,v} B Batch: [{t,v},{t,v},{t,v}]
  • 21. Single Host Prometheus Chronix ServerChronix Ingester In-Memory Chronix Ingester Singleton ;-) Prometheus S Sample: {t,v} B Batch: [{t,v},{t,v},{t,v}] B B B
  • 22. Single Host Prometheus Chronix Server Chronix Ingester In-Memory Chronix Ingester Cloud behind a proxy to serve multiple Prometheus servers. Prometheus S Sample: {t,v} B Batch: [{t,v},{t,v},{t,v}] N G I N X Chronix Ingester In-Memory Prometheus Prometheus
  • 23. Single Host Single Host Single HostSingle Host Prometheus Chronix ServerChronix Ingester In-Memory Cloud Mode: Multiple Prometheus Servers, One Chronix Ingester per Host, A Chronix Server Cloud Prometheus N G I N X Chronix Ingester In-Memory Prometheus Prometheus Chronix Server Cloud M a s t e r
  • 24. Architectural Key Factor: The Chronix Ingestor ■ Small Go Program ■Binary Size: 8.5 MB ■Lines of Code: ~ 720 LoC ■Scales easily: Copy, Execute ■ Handles writes from Prometheus ■Just a small configuration: remote_write: url: http://<host>:<port>/ingest ■ Batches samples in memory ■Prometheus sends single samples. ■Chronix needs large chunks (n single samples) to work well ■Max Batch Age ■5M, 12H, .. ■ Crash and restart resilience ■In-memory is dangerous. The Ingester holds some amount of transient state ■Regularly writes checkpoints of the entire in-memory state to disk ■Latest checkpoint is loaded on restart
  • 25. Chronix loves Chunks. Hence the Ingester batches samples.
  • 26. The data models for Prometheus and Chronix are similar. ■ Prometheus ■Uses so called labels (key-value pairs) to store dimensional values ■Are added dynamically ■Stores samples (pairs of timestamp and scalar value) ■ Chronix ■Uses attributes (key-value pairs) to store dimensional values ■Schema, Schema less, Dynamic Fields, etc. ■Stores samples of timestamp an any value type: scalar, trace, string, etc.
  • 27. An example Chronix schema to define the available fields. <?xml version="1.0" encoding="UTF-8" ?> <schema name="Chronix" version="1.5"> <types> <fieldType name="int" class="solr.TrieIntField" precisionStep="0" positionIncrementGap="0"/> <fieldType name="long" class="solr.TrieLongField" precisionStep="0" positionIncrementGap="0"/> <fieldType name="double" class="solr.TrieDoubleField" precisionStep="0" positionIncrementGap="0"/> <fieldType name="string" class="solr.StrField" sortMissingLast="true" omitNorms="true"/> <fieldType name="binary" class="solr.BinaryField"/> </types> <fields> <!-- The required fields --> <field name="id" type="string" indexed="true" stored="true" required="true"/> <field name="_version_" type="long" indexed="true" stored="true"/> <field name="start" type="long" indexed="true" stored="true" required="true"/> <field name="end" type="long" indexed="true" stored="true" required="true"/> <field name="data" type="binary" indexed="true" stored="true" required="false"/> <field name="metric" type="string" indexed="true" stored="true" required="true"/> <!-- Dynamic field for tags --> <dynamicField name="*_s" type="string" indexed="true" stored="true"/> </fields> <uniqueKey>id</uniqueKey> <solrQueryParser defaultOperator="OR"/> </schema> Definition of types Available Fields Prometheus labels are strings. Chronix Ingester creates them in Chronix Server dynamically using the dynamicField *_s. Prometheus_Label -> Chronix_Label host -> host_s
  • 28. Showcase: Prometheus, Chronix Ingester, Chronix and Grafana Prometheus Chronix ServerChronix Ingester In-Memory S S S Grafana B B B
  • 29. Disk usage: 11 Days of Data 112,815,835 Samples Prometheus: ~ 786 MB (whole data directory) Chronix: ~ 265 MB (without compaction) A few words about performance in our showcase.
  • 30. Compaction Effects. Compaction Points per Chunk Amount of Records Disk Usage in MB Compaction Time in Seconds no -1 610355 265 0 yes 100 1422369 357 134 yes 500 284815 187 75 yes 1000 142573 160 93 yes 5000 28850 131 69 yes 10000 14797 126 61 yes 25000 6408 123 61 yes 100000 2051 121 60 yes 500000 920 119 63 Contains about 112 points per chunk without compaction!
  • 31. A few words about performance in our showcase. CPU usage: 4 Cores available (= 400 % Max)
  • 32. A few words about performance in our showcase. Memory consumption (max. 8 G) Ingester
  • 36. Using the data source plugins for Chronix and Prometheus.
  • 38. Short Term Data in Prometheus. Long Term Data in Chronix. See the difference?
  • 39. Everything is open source and free to everyone. The code is the truth. Chronix Website: www.chronix.io Chronix Github: https://github.com/ChronixDB - Ingester: https://github.com/ChronixDB/chronix.ingester Questions? - Twitter: @ChronixDB, @flolaut, @phxql - Slack: https://qaware.slack.com/messages/chronix/
  • 40. Now it’s your turn. Now it’s your turn.