In this session, you will learn downsampling strategies and techniques for your InfluxDB data.

2. Agenda: New Practitioners Track
WORKSHOPAGENDA
8:00 AM – 9:00 AM Breakfast
9:00 AM – 10:00 AM Installing the TICK Stack and Your First Query Katy Farmer
10:00 AM – 10:50 AM Chronograf and Dashboarding Margo Schaedel
10:50 AM – 11:20 AM Break
11:20 AM – 12:10 PM Writing Queries (InfluxQL and TICK) Katy Farmer
12:10 PM – 1:10 PM Lunch
1:10 PM – 2:00 PM Architecting InfluxEnterprise for Success Ed Bernier
2:00 PM – 2:10 PM Break
2:10 PM – 3:10 PM Optimizing the TICK Stack Dave Patton
3:10 PM – 4:00 PM Downsampling Data Michael DeSa
4:00 PM Pizza and Beer

3. Michael DeSa
Software Engineer
@mjdesa
Downsampling Data
Michael DeSa is a Software Engineer at InfluxData who focuses
on increasing the performance capabilities of InfluxDB. He has
led the InfluxDB training course across the US, providing
students with an in depth understanding of how InfluxDB works
as well as sharing best practices. He has a degree in Math, with
a focus on Abstract Algebra, from the University of California, at
Berkeley and was an Instructor of the Web Development
Immersive series at General Assembly in San Francisco.

4. Downsampling Data
Michael Desa
02/14/18

5. Agenda
• What is Downsampling
• Why would I want to do it
• How to Downsample Data with Influx
– Continuous Queries
– Kapacitor
• Future Work

6. © 2018 InfluxData. All rights reserved.6
What is Downsampling - Textbook Definition
The process of reducing the sampling rate of a signal.

7. © 2018 InfluxData. All rights reserved.7
Full Resolution Data

8. © 2018 InfluxData. All rights reserved.8
Downsampling - every nth value

9. © 2018 InfluxData. All rights reserved.9
Downsampling - random subset of N points

10. © 2018 InfluxData. All rights reserved.10
Downsampling - time-windowed average

11. Why
Downsample?
• full resolution data:
– more disk
– more network
– more cpu
– more $$$
• full resolution data + browser = sadness

12. Smart
Downsampling
• Data relevance decreases with age
– new data - high resolution
– old data - low resolution
• As new data becomes old, decrease the
resolution

13. © 2018 InfluxData. All rights reserved.13
What is Downsampling - Common Definition
The process of expiring older high resolution data and replacing it with a
decimated (lower resolution) version.
(also called rollups)

14. © 2018 InfluxData. All rights reserved.14
Downsampling with Influx
Downsampling
• Continuous Queries (CQs)
• Kapacitor
Expiry
• Retention Policies

15. Continuous Queries

16. CQ Syntax
CREATE CONTINUOUS QUERY <cq_name>
ON <database_name>
BEGIN
SELECT <function>(<stuff>)
INTO <destination_measurement>
FROM <source_measurement>
GROUP BY time(<interval>), [tags]
END

17. CQ Example
CREATE CONTINUOUS QUERY mycq
ON telegraf
BEGIN
SELECT mean(usage_user)
INTO telegraf.rollup_5m.cpu
FROM telegraf.autogen.cpu
GROUP BY time(5m), *
END

18. CQ Example
CREATE CONTINUOUS QUERY mycq
ON telegraf
BEGIN
SELECT mean(usage_user) AS usage_user
INTO telegraf.rollup_5m.cpu
FROM telegraf.autogen.cpu
GROUP BY time(5m), *
END

19. CQ Example
CREATE CONTINUOUS QUERY mycq
ON telegraf
BEGIN
SELECT mean(*)
INTO telegraf.rollup_5m.:MEASUREMENT:
FROM telegraf.autogen./.*/
GROUP BY time(5m), *
END

20. Issues with the
General CQ
CREATE CONTINUOUS QUERY mycq
ON telegraf
BEGIN
SELECT mean(*)
INTO telegraf.rollup_5m.:MEASUREMENT:
FROM telegraf.autogen./.*/
GROUP BY time(5m), *
END

21. Issues with CQs
• Places Extra Load on your InfluxDB instance
• Limited to things expressible in InfluxQL
– No rolling windows
– No cross measurement joins
• All CQ processing and execution runs on a
single node in the cluster
• Can’t have consistent field names without
having many CQs
– field_a -> mean_field_a

22. Kapacitor

23. © 2018 InfluxData. All rights reserved.23
Batching Streaming
● Queries InfluxDB periodically
● Does not buffer as much of data
in RAM
● Places additional query load on
InfluxDB
● Writes are mirrored onto the
InfluxDB instance
● Buffers all data in RAM
● Places additional write load on
Kapacitor

24. Batch TICKscript
batch
|query('''
SELECT mean("usage_user") AS
usage_user
FROM "telegraf"."autogen"."cpu"
''')
.period(5m)
.every(5m)
|influxDBOut()
.database('telegraf')
.retentionPolicy('rollup_5m')

25. Stream
TICKscript
stream
|from()
.measurement('cpu')
.groupBy(*)
|window()
.period(5m)
.every(5m)
|mean('usage_user')
.as('usage_user')
|influxDBOut()
.database('telegraf')
.retentionPolicy('rollup_5m')

26. Rolling Window
stream
|from()
.measurement('cpu')
.groupBy(*)
|window()
.period(25m)
.every(5m)
|mean('usage_user')
.as('usage_user')
|influxDBOut()
.database('telegraf')
.retentionPolicy('rollup_5m')

27. Issues with the
TICKscript
stream
|from()
.measurement('cpu')
.groupBy(*)
|window()
.period(5m)
.every(5m)
|mean('usage_user')
.as('usage_user')
|influxDBOut()
.database('telegraf')
.retentionPolicy('rollup_5m')

28. Issues with
Kapacitor
• Stream tasks currently aren’t feasible in high
throughput InfluxDB instances
– All write load gets mirrored onto a single
Kapacitor instance
• TICKscript doesn’t have support for
wildcard operations
– Only way to downsample all data is to have
(#measurement x #fields) tasks

29. © 2018 InfluxData. All rights reserved.29
CQs Kapacitor
● Built into InfluxDB
● Uses InfluxQL
● Often enough for simple
downsampling
● (-) Places extra load on InfluxDB
● Runs externally to InfluxDB
● Uses TICKscript
● Supports more complex
operations
● (-) Buffers streaming data in RAM

30. © 2018 InfluxData. All rights reserved.30
What is a Retention Policy
A policy used to expire aged data

31. Retention Policy
Syntax
CREATE RETENTION POLICY <rp>
ON <db>
DURATION <duration>
REPLICATION <n>
[DEFAULT]

32. Retention Policy
Example
CREATE RETENTION POLICY autogen
ON telegraf
DURATION 10d
REPLICATION 1
DEFAULT

33. Retention Policy
Example
CREATE RETENTION POLICY rollup_5m
ON telegraf
DURATION 0
REPLICATION 1

34. Demo

35. Thank You