Time-series workloads (i.e. data from sensors, IoT devices, finance, or even satellites) are generally insert-mostly, and data typically arrives in time order (at regular or irregular intervals). Given the high velocity and continuous workload of writing time-series, insert performance is paramount. But what is the use of inserting a significant amount of data if you can't analyze, visualize, and act on it effectively? Unlike many OLTP workloads, you often don't need the granularity of each data point, but rather reports on aggregates over significant periods of time and other analysis are the key to making good decisions with the data you store. This talk describes how TimescaleDB (a time-series database packaged as an extension of PostgreSQL) has implemented the infrastructure for creating continuously up-to-date aggregates without write amplification using features of Postgres (partial aggregates, invalidation triggers, proper locking/transaction safety, background workers, union views, query planner etc.), as well as how/when to use these pre-calculated results to speed your queries.

1. Creating Continuously Up to
Date Materialized Aggregates
David Kohn
Software Engineer, Timescale
david@timescale.com · github.com/timescale

2. A Use Case: Stock Trades
(Mostly because they’re familiar, and reasonably
representative)

4. SELECT time_bucket('15 min', trade_time),
symbol,
first(price, trade_time) as open,
max(price) as high,
min(price) as low,
last(price, trade_time) as close,
sum(num_shares) as total_volume
FROM trade_data
WHERE trade_time > now() - '30 days'::interval
GROUP BY time_bucket('15 min', trade_time), symbol
ORDER BY time_bucket('15 min', trade_time) DESC;
Typical OHLC Query

5. Typical OHLC Query

6. How do we make it into an
interactive dashboard?

7. Let’s try a Materialized View:
CREATE MATERIALIZED VIEW ohlc AS
SELECT time_bucket('15 min', trade_time),
symbol,
first(price, trade_time) as open,
max(price) as high,
min(price) as low,
last(price, trade_time) as close,
sum(num_shares) as total_volume
FROM trade_data
GROUP BY time_bucket('15 min', trade_time), symbol;

8. But…
• Need to run REFRESH MATERIALIZED VIEW manually or it will
quickly be out of date.
• Recalculates the entire view on every refresh
• Will REFRESH CONCURRENTLY help? (Spoiler, not really…)
• Very inefficient for insert mostly, time-ordered workloads

9. What about triggers?
raw
data in
15 min
trigger

10. What about triggers?
raw
data in
15 min
trigger
1 day
trigger

11. • Triggers can keep materializations up to date
• But they cause write amplification every row inserted (or updated/
deleted) means a modification to each materialization a.k.a.
• Triggers can cause lock conflicts that further slow writes
What about triggers?

12. The data industry is undergoing a
generational shift

13. 1970s-1990s
The relational database era
for transactional processing
Oracle, DB2, SQL Server

14. 2000s-2010s
The big data and non-
relational era for analytics
Hadoop, Cassandra, MongoDB

15. The Rise of Machine Data
44ZB
data collected from IoT devices
by 2020 (IDC)
71%
of global businesses now
collecting IoT data (451 Research)
75%
of IoT data goes unused today
by 92% of businesses (Verizon)
25GB
data collected per hour by
connected cars (McKinsey)

16. Time-series
• Primarily UPDATEs
• Writes randomly distributed
• Transactions to multiple
primary keys
• Primarily INSERTs
• Writes to recent time interval
• Writes primarily associated
with a timestamp
OLTP

17. Chunks
Hypertable
Automatic Space-Time Partitioning

18. Query Planner
Time-Series Queries Standard SQL Queries
Storage Layer
Data Management
The Extensibility of PostgreSQL
9.6, 10, 11, …

19. Okay, okay, how about a cron job?
Already Materialized Data
Next
15 Min
Inserting
Here
cron job
15 min

20. Okay, okay, how about a cron job?
Already Materialized Data
Last
15 Min
Inserting
Here
wait 15 min…

21. • Deals with most inserts as they are usually to the most recent period
• Reduced write amplification because aggregates are computed
once per period
• BUT… what about late writes? Deletes? Updates?
• Synchronization issues often arise
Okay, okay, how about a cron job?

22. TimescaleDB Continuous Aggregates
raw
data in scheduled
aggregation
15 min
invalidation
events
log

23. TimescaleDB Continuous Aggregates
Materialized Data
Next
15 Min
Inserting
Here
scheduled
aggregation
15 min

24. TimescaleDB Continuous Aggregates
invalidation
events
log scheduled
aggregation
15 min
Non
Materialized
Data
Materialized Data
Next
15 Min
Inserting
Here

25. • Designed for high volume, mostly ordered, insert-mostly workloads
• Minimal write amplification, while maintaining correctness
• None for writes above threshold
• One row per-statement invalidation overhead below threshold
• Meticulously avoid locking issues using PG transactional guarantees
• Maintained consistently without user intervention
TimescaleDB Continuous Aggregates

26. CREATE VIEW ohlc_continuous
WITH (timescaledb.continuous)
AS SELECT time_bucket('15 min', trade_time),
symbol,
first(price, trade_time) as open,
max(price) as high,
min(price) as low,
last(price, trade_time) as close,
sum(num_shares) as total_volume
FROM trade_data
GROUP BY time_bucket('15 min', trade_time), symbol;
TimescaleDB Continuous Aggregates

27. Single View Across Aggregated & Raw Data
C
om
ing
Soon!
raw 15 min
SELECT * FROM ohlc_continuous;

28. Vision: Materialized View as an Index
In
D
evelopm
ent
SELECT symbol,
max(price) as max_price
FROM trade_data
GROUP BY symbol;
SELECT symbol,
max(high) as max_price
FROM ohlc_continuous
GROUP BY symbol;
• Why should your app need to know that the view exists?
• Indexes are transparent, why can’t a materialization be?

29. CREATE VIEW ohlc_continuous
WITH (timescaledb.continuous)
AS SELECT time_bucket('15 min', trade_time),
symbol,
first(price, trade_time) as open,
max(price) as high,
min(price) as low,
last(price, trade_time) as close,
sum(num_shares) as total_volume,
avg(price) as avg_price
FROM trade_data
GROUP BY time_bucket('15 min', trade_time), symbol;
The Problem With Average

30. Partial Aggregation
• Example: for average store the sum and the count
• Combine by summing each of them
• Finalize by dividing the sum by the count.
• All parallelizable aggregates Postgres must have partial aggregation, combine and
finalize functions defined
• Instead of storing the final state of the aggregate, we store partials and then combine
and finalize at run time

31. Data Retention
C
om
ing
Soon!
raw 15 minGranularity
Retention 2 weeks 3 years

32. SELECT time_bucket_regroup('2 hours', trade_time),
symbol,
avg(price) as avg_price
FROM ohlc_continuous
GROUP BY time_bucket_regroup(‘2 hours', trade_time), symbol;
Re-Grouping With Exact Results
SELECT time_bucket_regroup('15 min', trade_time),
avg(price) as avg_price
FROM ohlc_continuous
GROUP BY time_bucket_regroup('15 min', trade_time);
In
D
evelopm
ent

33. Source code
• github.com/timescale/timescaledb
Join the Community
• slack.timescale.com

34. timescale.com/cloud-promo