Storing 16 Bytes at Scale

1.
Storing 16 Bytesat Scale Fabian Reinartz Software Engineer @fabxc

2.
Time Series time

3.
Time Series time series

4.
requests_total{path="/status", method="GET", instance="10.0.0.1:80"} requests_total{path="/status",method="POST", instance="10.0.0.3:80"} requests_total{path="/", method="GET", instance="10.0.0.2:80"} ... Time Series

5.
requests_total{path="/status", method="GET", instance="10.0.0.1:80"} requests_total{path="/status",method="POST", instance="10.0.0.3:80"} requests_total{path="/", method="GET", instance="10.0.0.2:80"} Select: requests_total Time Series

6.
requests_total{path="/status", method="GET", instance="10.0.0.1:80"} requests_total{path="/status",method="POST", instance="10.0.0.3:80"} requests_total{path="/", method="GET", instance="10.0.0.2:80"} Select: requests_total{method="GET"} Time Series

7.

8.

9.

10.

11.
Order of Scale 5million active time series 30 second scrape interval 1 month of retention 166,000 samples/second 432 billion samples 2,000 - 15,000 targets

12.
Order of Scale 5million active time series 30 second scrape interval 1 month of retention 166,000 samples/second 432 billion samples 8 byte timestamp + 8 byte value ⇒ 7 TB on disk 2,000 - 15,000 targets seems OK

13.
Order of Scale 5million active time series 30 second scrape interval 6 month of retention 166,000 samples/second 2592 billion samples 8 byte timestamp + 8 byte value ⇒ 42 TB on disk 2,000 - 15,000 targets

14.
Sample Compression –Timestamps 1496163646 1496163676 1496163706 1496163735 1496163765

15.
Sample Compression –Timestamps 1496163646 1496163676 1496163706 1496163735 1496163765 1496163646 +30 +30 +29 +30Δ

16.
Sample Compression –Timestamps 1496163646 1496163676 1496163706 1496163735 1496163765 1496163646 +30 +30 +29 +30 1496163646 +30 +0 -1 +1 Δ ΔΔ

17.
Sample Compression –Values Source: http://www.vldb.org/pvldb/vol8/p1816-teller.pdf

18.
Sample Compression Raw: 16bytes/sample Compressed: 1.37 bytes/sample @beorn7 @dgryski

19.
Order of Scale 5million active time series 30 second scrape interval 1 month of retention 166,000 samples/second 432 billion samples 8 byte timestamp + 8 byte value ⇒ 7 TB on disk 0.8 TB 2,000 - 15,000 targets

20.
Storing 16 Bytesat Scale Fabian Reinartz Software Engineer, CoreOS @fabxc 1.37

21.
Time Series –Chunks time series 1 file/series filled with chunks

22.
Time Series –Chunks time series 1 file/series filled with chunks

23.
Churn time series

24.
Churn time series

25.
Order of Scale– with Churn 5 million active time series 150 million total time series 30 second scrape interval 1 month of retention 166,000 samples/second 432 billion samples

26.
Churn time series

27.
Blocks time series

28.
Blocks time series

29.
Blocks time series

30.
Blocks data ├── 01BX6V6H5JQN8RWVXM5ABYXY9N │ ├──chunks │ │ ├── 000001 │ │ ├── 000002 │ │ └── 000003 │ ├── index │ ├── meta.json │ └── tombstones ├── 01BX6V6TY06G5MFQ0GPH7EMXRH │ ├── chunks │ │ └── 000001 │ ├── index │ ├── meta.json │ └── tombstones └── wal ├── 000004 ├── 000005 └── 000006

31.
Compaction time series

32.
Compaction time series

33.
Retention time series

34.
Retention time series

35.
Tree Block 1 chunk 1chunk 2 chunk 3 time Block 2 Block 3 Block 4 Block 5

36.
Tree Block 1 chunk 1chunk 2 chunk 3 chunk 4 chunk 5 time Block 1 Block 2 Block 2 Block 3

37.
Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, }

38.
Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● In eachblock, a series has a unique ID ID: 5

39.
Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } ● In eachblock, a series has a unique ID ● Maintain sorted lists from label pair to IDs ID: 5 ... status=”200”: 1 2 5 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... ...

40.
Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } status=”200”: 1 25 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... ● In each block, a series has a unique ID ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations (merge, intersect)

41.
Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } status=”200”: 1 25 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: ● In each block, a series has a unique ID ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations (merge, intersect)

42.
Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } status=”200”: 1 25 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: ● In each block, a series has a unique ID ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations (merge, intersect)

43.
Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } status=”200”: 1 25 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 ● In each block, a series has a unique ID ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations (merge, intersect)

44.

45.

46.
Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } status=”200”: 1 25 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 5 ● In each block, a series has a unique ID ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations (merge, intersect)

47.

48.

49.

50.
Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } status=”200”: 1 25 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 5 1502 ● In each block, a series has a unique ID ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations (merge, intersect)

51.
Index { __name__=”requests_total”, pod=”nginx-34534242-abc723 job=”nginx”, path=”/api/v1/status”, status=”200”, method=”GET”, } status=”200”: 1 25 99 1000 1001 1500 1502 500000 method=”GET”: 2 3 4 5 6 9 10 1502 999999 ... Intersect: 2 5 1502 ● In each block, a series has a unique ID ● Maintain sorted lists from label pair to IDs ● Efficient k-way set operations (merge, intersect)

52.
Benchmarks

53.
Benchmarks ● Kubernetes cluster+ dedicated Prometheus nodes ● 800 microservice instances + Kubernetes components ● 120,000 samples/second ● 300,000 active time series ● Swap out 50% of pods every 10 minutes

54.
Benchmarks (memory)

55.
Benchmarks (CPU)

56.
Benchmarks (disk writes)

57.
Benchmarks (on-disk size)

58.
Benchmarks (query latency)

59.
Try it out! We’rehiring: coreos.com/careers

60.
● https://fabxc.org/blog/2017-04-10-writing-a-tsdb/ ● https://github.com/prometheus/tsdb More

Storing 16 Bytes at Scale

More Related Content

What's hot

Similar to Storing 16 Bytes at Scale

Recently uploaded

Storing 16 Bytes at Scale