In this session, you will learn downsampling strategies and techniques for your InfluxDB data.

1. Michael DeSa - Software Engineer
Downsampling your data

2. © 2017 InfluxData. All rights reserved.2 © 2017 InfluxData. All rights reserved.2
✓ Continuous Queries (CQs)
✓ Create custom CQs
✓ Retention Policies (RPs)
✓ Create custom RPs
✓ Combine CQs and RPs to manage
downsampling and data retention
✓ Common issues with CQs and RPs
Agenda

3. © 2017 InfluxData. All rights reserved.3
Continuous Queries
¨ What are Continuous Queries (CQs)?
¨ InfluxQL queries that run automatically and periodically on real-time data and store
query results in a specified measurement
¨ Why would I use CQs?
¨ Automatic downsampling and pre-calculating expensive queries
An Overview

4. © 2017 InfluxData. All rights reserved.4
Continuous Queries
The Basic Syntax
CREATE CONTINUOUS QUERY <cq_name>
ON <database_name>
BEGIN
SELECT <function>(<stuff>)
INTO <destination_measurement>
FROM <source_measurement>
GROUP BY time(<interval>)
END
1
2
3

5. © 2017 InfluxData. All rights reserved.5
Continuous Queries
The Basic Syntax: In Practice
CREATE CONTINUOUS QUERY "average_cpu_usage" ON "telegraf" BEGIN
SELECT MEAN("usage_idle") INTO "ave_cpu" FROM "cpu" GROUP BY time( 1m)
END
> SELECT "usage_idle" FROM "cpu"
name: cpu
time usage_idle
---- ----------
2017-02-07T23:14:00Z 99.599599599
2017-02-07T23:14:10Z 99.699398797
2017-02-07T23:14:20Z 99.699699699
2017-02-07T23:14:30Z 99.600000000
2017-02-07T23:14:40Z 99.500000000
2017-02-07T23:14:50Z 99.799599198
2017-02-07T23:15:00Z 99.600000000
2017-02-07T23:15:10Z 99.400599400
2017-02-07T23:15:20Z 99.600000000
2017-02-07T23:15:30Z 99.699699699
2017-02-07T23:15:40Z 99.500000000
2017-02-07T23:15:50Z 99.699699699
> SELECT "mean" FROM "ave_cpu"
name: ave_cpu
time mean
---- ----
2017-02-07T23:14:00Z 99.649716215
2017-02-07T23:15:00Z 99.583333133

6. © 2017 InfluxData. All rights reserved.6
Continuous Queries
The Basic Syntax: In Practice
Feb 07 23:15:00 executing continuous query
average_cpu_usage
SELECT […] WHERE time >= '2017-02-07T23:14:00Z' AND
time < '2016-08-29T23:15:00Z' GROUP BY time(1h)
Feb 07 23:16:00 executing continuous query
average_cpu_usage
SELECT […] WHERE time >= '2017-02-07T23:15:00Z' AND
time < '2016-08-29T23:16:00Z' GROUP BY time(1h)
The basic CQ:
● Executes at the same interval as the GROUP BY time() interval.
● Executes a single query that covers the time range between
now() and now() minus the GROUP BY time() interval.

7. © 2017 InfluxData. All rights reserved.7
Continuous Queries
The Advanced Syntax
CREATE CONTINUOUS QUERY <cq_name>
ON <database_name>
RESAMPLE
EVERY <interval>
FOR <interval>
BEGIN
SELECT <function>(<stuff>)
INTO <destination_measurement>
FROM <source_measurement>
GROUP BY time(<interval>)
END
1
2
3
4

8. © 2017 InfluxData. All rights reserved.8
Continuous Queries
The Advanced Syntax: In Practice
CREATE CONTINUOUS QUERY "average_cpu_usage" ON "telegraf"
RESAMPLE EVERY 2m FOR 3m BEGIN
SELECT MEAN("usage_idle") INTO "ave_cpu" FROM “cpu" GROUP BY time(1m)
END
> SELECT "usage_idle" FROM "cpu"
name: cpu
time usage_idle
---- ----------
2017-02-07T23:14:00Z 99.599599599
2017-02-07T23:14:10Z 99.699398797
2017-02-07T23:14:20Z 99.699699699
2017-02-07T23:14:30Z 99.600000000
2017-02-07T23:14:40Z 99.500000000
2017-02-07T23:14:50Z 99.799599198
2017-02-07T23:15:00Z 99.600000000
2017-02-07T23:15:10Z 99.400599400
2017-02-07T23:15:20Z 99.600000000
2017-02-07T23:15:30Z 99.699699699
2017-02-07T23:15:40Z 99.500000000
2017-02-07T23:15:50Z 99.699699699
> SELECT "mean" FROM "ave_cpu"
name: ave_cpu
time mean
---- ----
2017-02-07T23:14:00Z 99.649716215
2017-02-07T23:15:00Z 99.583333133

9. © 2017 InfluxData. All rights reserved.9
Continuous Queries
The Advanced Syntax: In Practice
Feb 07 23:16:00 executing continuous query
average_cpu_usage
SELECT […] WHERE time >= '2017-02-07T23:13:00Z' AND
time < '2016-08-29T23:16:00Z' GROUP BY time(1h)
Feb 07 23:18:00 executing continuous query
average_cpu_usage
SELECT […] WHERE time >= '2017-02-07T23:15:00Z' AND
time < '2016-08-29T23:18:00Z' GROUP BY time(1h)
In the advanced syntax:
The EVERY interval determines how often InfluxDB executes the CQ.
The FOR interval determines the time range over which the CQ runs
queries.

10. © 2017 InfluxData. All rights reserved.10
Retention Policies (RPs)
¨ What What are RPs?
¨ The part of InfluxDB’s data structure that describe for how long InfluxDB keeps
data.
¨ Why would I use RPs?
¨ Expire unneeded data.
¨
An Overview

11. © 2017 InfluxData. All rights reserved.11
Retention Policies
The Syntax
CREATE RETENTION POLICY <retention_policy_name>
ON <database_name>
DURATION <duration>
REPLICATION <n>
[SHARD DURATION <duration>] [DEFAULT]
DURATION units:
u microseconds
ms milliseconds
s seconds
m minutes
h hours
d days
w weeks
INF
REPLICATION
settings:
Single instance:
no effect
Clustering: RF
<= data nodes
SHARD DURATION
settings:
https://docs.inf
luxdata.com/infl
uxdb/latest/conc
epts/schema_and_
data_layout/#sha
rd-group-duratio
n-management

12. © 2017 InfluxData. All rights reserved.12
Retention Policies
The Syntax: In Practice
> SHOW RETENTION POLICIES ON "telegraf"
name duration shardGroupDuration replicaN default
---- -------- ------------------ -------- -------
autogen 0s 168h0m0s 1 true
other 24h0m0s 1h0m0s 1 false
CREATE RETENTION POLICY "other" ON "telegraf" DURATION
1d REPLICATION 1

13. © 2017 InfluxData. All rights reserved.13
Retention Policies
The Syntax: In Practice
Rebalance a cluster:
https://docs.influxdata.com/enterprise/latest/guides/r
ebalance/
ALTER RETENTION POLICY <retention_policy_name>
ON <database_name>
[DURATION <duration> |
REPLICATION <n> |
SHARD DURATION <duration> |
DEFAULT]

14. © 2017 InfluxData. All rights reserved.14
Retention Policies
The Syntax: In Practice
> SHOW RETENTION POLICIES ON "telegraf"
name duration shardGroupDuration replicaN default
---- -------- ------------------ -------- -------
autogen 0s 168h0m0s 1 true
other 48h0m0s 1h0m0s 1 false
ALTER RETENTION POLICY "other" ON "telegraf" DURATION
2d

15. © 2017 InfluxData. All rights reserved.15
Continuous Queries & Retention Policies: A Case Study
¨ Downsample 10-second resolution Telegraf data to 5-minute resolution data
¨ Store the 10-second resolution data for one week
¨ Store the 5-minute resolution data for four weeks
¨ What you need
a. A working InfluxDB instance

16. © 2017 InfluxData. All rights reserved.16
Continuous Queries & Retention Policies
¨ Step 1: Create a database
A Case Study
> SHOW DATABASES
name: databases
name
----
telegraf
_internal
> CREATE DATABASE "telegraf"
>

17. © 2017 InfluxData. All rights reserved.17
Continuous Queries & Retention Policies
¨ Step 2: Create a one-week retention policy
A Case Study
> SHOW RETENTION POLICIES ON "telegraf"
name duration shardGroupDuration replicaN default
---- -------- ------------------ -------- -------
autogen 0s 168h0m0s 1 false
one_week 168h0m0s 1h0m0s 1 true
> CREATE RETENTION POLICY "one_week" ON "telegraf"
DURATION 1w REPLICATION 1 DEFAULT

18. © 2017 InfluxData. All rights reserved.18
Continuous Queries & Retention Policies
¨ Step 3: Create a four-week retention policy
A Case Study
> SHOW RETENTION POLICIES ON "telegraf"
name duration shardGroupDuration replicaN default
---- -------- ------------------ -------- -------
autogen 0s 168h0m0s 1 false
one_week 168h0m0s 24h0m0s 1 true
four_week 672h0m0s 24h0m0s 1 false
> CREATE RETENTION POLICY "four_week" ON "telegraf"
DURATION 4w REPLICATION 1
>

19. © 2017 InfluxData. All rights reserved.19
Continuous Queries & Retention Policies
¨ Step 4: Create a continuous query
A Case Study
Fully-qualify a measurement:
"<database>"."<retention_policy>"."<measurement>"
> CREATE CONTINUOUS QUERY "ave_usage" ON "telegraf"
BEGIN
SELECT MEAN("usage_idle")
INTO "telegraf"."four_week"."ave_cpu"
FROM "telegraf"."one_week"."cpu"
WHERE "cpu" = 'cpu-total' GROUP BY time(5m)
END

20. © 2017 InfluxData. All rights reserved.20
Continuous Queries & Retention Policies
¨ Step 5: Install Telegraf
A Case Study
$ service telegraf start
$ wget
https://dl.influxdata.com/telegraf/releases/telegraf_1
.2.1_amd64.deb
$ sudo dpkg -i telegraf_1.2.1_amd64.deb

21. © 2017 InfluxData. All rights reserved.21
Continuous Queries & Retention Policies
¨ Step 6: Wait a bit… (about five minutes)
A Case Study

22. © 2017 InfluxData. All rights reserved.22
Continuous Queries & Retention Policies
¨ Step 7: Confirm your downsample/data expiration
A CaseStudy
> SELECT "usage_idle" FROM
"telegraf"."one_week"."cpu" LIMIT 3
name: cpu
time usage_idle
---- ----------
2017-02-08T17:01:20Z 99.49949949949996
2017-02-08T17:01:20Z 99.49949949949996
2017-02-08T17:01:30Z 99.59879638916786
> SELECT "mean" FROM
"telegraf"."four_week"."ave_cpu" LIMIT 3
name: ave_cpu
time mean
---- ----
2017-02-08T17:30:00Z 99.20285887894339
2017-02-08T17:35:00Z 99.31273835165075
2017-02-08T17:40:00Z 99.29954474534229
> USE "telegraf"
Using database telegraf
> SHOW MEASUREMENTS
name: measurements
name
----
ave_cpu
cpu
disk
diskio
kernel
mem
processes
swap
system

23. © 2017 InfluxData. All rights reserved.23
Continuous Queries
¨ Issue 1: Working with historical data
Some Common Issues
> SELECT MAX("water_level")
INTO "maximums" FROM "h2o_feet"
WHERE time >= '2015-08-18T00:00:00Z' AND time <=
'2015-08-18T00:18:00Z'
GROUP BY time(12m)
name: result
time written
---- -------
1970-01-01T00:00:00Z 2
> SELECT * FROM "maximums"
name: maximums
time max
---- ---
2015-08-18T00:00:00Z 8.12
2015-08-18T00:12:00Z 7.887
> SELECT "water_level","location" FROM "h2o_feet"
LIMIT 8
name: h2o_feet
time water_level location
---- ----------- --------
2015-08-18T00:00:00Z 8.12 coyote_creek
2015-08-18T00:00:00Z 2.064 santa_monica
2015-08-18T00:06:00Z 8.005 coyote_creek
2015-08-18T00:06:00Z 2.116 santa_monica
2015-08-18T00:12:00Z 7.887 coyote_creek
2015-08-18T00:12:00Z 2.028 santa_monica
2015-08-18T00:18:00Z 7.762 coyote_creek
2015-08-18T00:18:00Z 2.126 santa_monica

24. © 2017 InfluxData. All rights reserved.24
Continuous Queries
¨ Issue 2: Missing data in CQ results
Some Common Issues
> SELECT * FROM "french_bulldogs"
name: french_bulldogs
---------------------
time color name
2016-06-26T20:17:59Z grey rumpelstiltskin
2016-06-26T20:17:59Z peach princess
> SELECT "name","color" INTO "dogs" FROM
"french_bulldogs"
name: result
------------
time written
1970-01-01T00:00:00Z 2
> SELECT * FROM "dogs"
name: dogs
----------
time color name
2016-06-26T20:17:59Z peach princess
1 2
3

25. © 2017 InfluxData. All rights reserved.25
Continuous Queries
¨ Issue 2: Missing data in CQ results
Some Common Issues
> SELECT * FROM "french_bulldogs"
name: french_bulldogs
---------------------
time color name
2016-06-26T20:17:59Z grey rumpelstiltskin
2016-06-26T20:17:59Z peach princess
> SELECT "name" INTO "dogs" FROM "french_bulldogs"
GROUP BY "color"
name: result
------------
time written
1970-01-01T00:00:00Z 2
> SELECT * FROM "dogs"
name: dogs
----------
time color name
2016-06-26T20:17:59Z grey rumpelstiltskin
2016-06-26T20:17:59Z peach princess
1 2
3

26. © 2017 InfluxData. All rights reserved.26
Continuous Queries
¨ Issue 3: Configuring the CQ schedule
Some Common Issues
SELECT MEAN("sunflowers")
FROM "flower_orders"
WHERE time >= '2016-08-29T18:00:00Z' AND time <=
'2016-08-29T19:45:00Z' GROUP BY time(1h)
name: flower_orders
-------------------
time sunflowers
2016-08-29T18:00:00Z 34
2016-08-29T18:15:00Z 28
2016-08-29T18:30:00Z 19
2016-08-29T18:45:00Z 20
2016-08-29T19:00:00Z 56
2016-08-29T19:15:00Z 76
2016-08-29T19:30:00Z 29
2016-08-29T19:45:00Z 90
2016-08-29T20:00:00Z 70
name: flower_orders
-------------------
time mean
2016-08-29T18:00:00Z 25.25
2016-08-29T19:00:00Z 62.75

27. © 2017 InfluxData. All rights reserved.27
Continuous Queries
¨ Issue 3: Configuring the CQ schedule
Some Common Issues
SELECT MEAN("sunflowers")
FROM "flower_orders"
WHERE time >= '2016-08-29T18:15:00Z' AND time <=
'2016-08-29T19:45:00Z' GROUP BY time(1h)
name: flower_orders
-------------------
time sunflowers
2016-08-29T18:00:00Z 34
2016-08-29T18:15:00Z 28
2016-08-29T18:30:00Z 19
2016-08-29T18:45:00Z 20
2016-08-29T19:00:00Z 56
2016-08-29T19:15:00Z 76
2016-08-29T19:30:00Z 29
2016-08-29T19:45:00Z 90
2016-08-29T20:00:00Z 70
name: flower_orders
-------------------
time mean
2016-08-29T18:00:00Z 25.25
2016-08-29T19:00:00Z 62.75

28. © 2017 InfluxData. All rights reserved.28
Continuous Queries
¨ Issue 3: Configuring the CQ schedule
Some Common Issues
SELECT MEAN("sunflowers")
FROM "flower_orders"
WHERE time >= '2016-08-29T18:15:00Z' AND time <=
'2016-08-29T19:45:00Z' GROUP BY time(1h,15m)
name: flower_orders
-------------------
time sunflowers
2016-08-29T18:00:00Z 34
2016-08-29T18:15:00Z 28
2016-08-29T18:30:00Z 19
2016-08-29T18:45:00Z 20
2016-08-29T19:00:00Z 56
2016-08-29T19:15:00Z 76
2016-08-29T19:30:00Z 29
2016-08-29T19:45:00Z 90
2016-08-29T20:00:00Z 70
name: flower_orders
-------------------
time mean
2016-08-29T18:15:00Z 30.75
2016-08-29T19:15:00Z 65

29. © 2017 InfluxData. All rights reserved.29
Retention Policies
¨ Issue 1: Writing to Retention Policies
Some Common Issues
HTTP API
$ curl -i -XPOST "http://localhost:8086/write?db=telegraf&rp=four_week" --data-binary
'mymeas,mytag=1 myfield=90'
CLI Option 1
> USE "telegraf"."four_week"
Using database telegraf
Using retention policy four_week
CLI Option 2
> INSERT INTO "four_week" mymeas,mytag=1 myfield=90

30. © 2017 InfluxData. All rights reserved.30
Retention Policies
¨ Issue 1: Writing to Retention Policies
Some Common Issues
HTTP API
$ curl -G "http://localhost:8086/query?db=telegraf" --data-urlencode "q=SELECT * FROM
telegraf.four_week.ave_cpu"
CLI Option 1
> USE "telegraf"."four_week"
Using database telegraf
Using retention policy four_week
CLI Option 2
> SELECT * FROM "telegraf"."four_week"."ave_cpu"

31. © 2017 InfluxData. All rights reserved.31 © 2017 InfluxData. All rights reserved.31
Using Kapacitor for Downsampling

32. © 2017 InfluxData. All rights reserved.32
¨ Implement if CQs are using too many host resources
¨ Kapacitor accepts either stream or batch tasks
¨ Writes data back into InfluxDB
¨
Downsample using Kapacitor
Offload computation to a separate host

33. © 2017 InfluxData. All rights reserved.33
Example
// batch_cpu.tick
batch
|query('''
SELECT mean("usage_user") AS usage_user
FROM "telegraf"."autogen"."cpu"
''')
.period(5m)
.every(5m)
|influxDBOut()
.database('telegraf')
.retenionPolicy('5m')
.tag('source', 'kapacitor')
● Downsample the data into 5m windows
● Store that data back into a the 5m retention
policy in the telegraf database

34. © 2017 InfluxData. All rights reserved.34
• https://docs.influxdata.com/influxd
b/latest/guides/downsampling_an
d_retention/
• community.influxdata.com
OtherKapacitor & TelegrafCQs & RPs
• https://docs.influxdata.com/influxd
b/latest/query_language/continuou
s_queries/
• https://docs.influxdata.com/influxd
b/latest/query_language/database
_management/#create-retention-p
olicies-with-create-retention-policy
• https://docs.influxdata.com/kap
acitor/latest/examples/continuo
us_queries/
• https://docs.influxdata.com/tele
graf/latest/concepts/aggregator
_processor_plugins/

35. Thank You
community.influxdata.com