pgday.seoul 2019: TimescaleDB

Chan Shik Lim
Chan Shik LimSenior Software Engineer
TimescaleDB:
Building a scalable
time-series
database on
PostgreSQL
Chanshik Lim
Developer at NexCloud
chanshik@gmail.com
Agenda
• Time-series Data?
• TimescaleDB Overview
• Using TimescaleDB
• Q & A
Time-series
Data?
Time-series Data? (1)
timestamp device_id cpu_1m_avg free_mem temperature location_id dev_type
2017-01-01 01:02:00 abc123 80 500MB 72 335 field
2017-01-01 01:02:23 def456 90 400MB 64 335 roof
2017-01-01 01:02:30 ghi789 120 0MB 56 77 roof
2017-01-01 01:03:12 abc123 80 500MB 72 335 field
2017-01-01 01:03:35 def456 95 350MB 64 335 roof
2017-01-01 01:03:42 ghi789 100 100MB 56 77 roof
Time-series Data? (2)
• Time-centric
• Data records always have a timestamp
• Append-only
• Data is almost solely append-only (INSERTs)
• Recent
• New data is typically about recent time intervals
Time-series Data? (3)
• Monitoring computer systems
• VM, server, container metrics (CPU, free memory, net/disk IOPS)
• Service and application metrics (request rates, request latency)
• Financial trading systems
• Classic securities, newer cryptocurrencies, payments, transaction events
• Internet of Things
• Data from sensors on industrial machines and equipment
• Eventing applications
• User/customer interaction data like clickstreams, pageviews, logins, singups
• Environmental monitoring
• Temperature, humidity, pressure, pH, pollen count, air flow, …
TimescaleDB
Overview
Easy to Use
• Full SQL interface for all SQL natively supported by PostgreSQL
• Secondary indexes
• Non time-based aggregates
• Sub-queries
• Window functions
• Connects to any client or tool that speaks PostgresSQL
• Time-oriented features
• Robust support for Data retention policies
Scalable
• Transparent time/space partitioning
• Scaling up (single node)
• Scaling out (private beta)
• High data write rates
• Right-sized chunks
• Parallelized operations across chunks and servers
Reliable
• Engineered up from PostgreSQL, packaged as an extension
• Proven foundations
• From 20+ years of PostgreSQL research
• Streaming replication
• Backups
• Flexible management options
• Compatible with existing PostgreSQL ecosystem and tooling
Architecture
• Hypertables
• Abstraction of a single continuous table across all space and time
intervals
• Chunks
• Each chunk corresponds to a specific time interval and a region of
partition key’s space
Using
TimescaleDB
Installing
• https://docs.timescale.com/latest/getting-started/installation
• Using Docker Image
• shm-size: set /dev/shm partition size
• Mapping /var/lib/postgresql/data to host directory
$ docker run -d --name timescaledb -p 5432:5432 
-e POSTGRES_PASSWORD=password 
-v /mnt/timescaledb:/var/lib/postgresql/data 
--shm-size 1G 
timescale/timescaledb:1.5.1-pg11
Setting up
$ psql -U postgres -h localhost
postgres=# create database tutorial;
CREATE DATABASE
postgres=# c tutorial
You are now connected to database "tutorial" as user "postgres".
tutorial=# create extension if not exists timescaledb cascade;
NOTICE: extension "timescaledb" already exists, skipping
CREATE EXTENSION
Creating a Hypertable
tutorial=# CREATE TABLE conditions (
tutorial(# time TIMESTAMPTZ NOT NULL,
tutorial(# location TEXT NOT NULL,
tutorial(# temperature DOUBLE PRECISION NULL,
tutorial(# humidity DOUBLE PRECISION NULL
tutorial(# );
CREATE TABLE
tutorial=# SELECT create_hypertable('conditions', 'time’,
chunk_time_interval => interval '1 day');
create_hypertable
-------------------------
(1,public,conditions,t)
(1 row)
Inserting
tutorial=# INSERT INTO conditions
tutorial-# VALUES
tutorial-# (NOW(), 'office', 70.0, 50.0),
tutorial-# (NOW(), 'basement', 66.5, 60.0),
tutorial-# (NOW(), 'garage', 77.0, 65.2);
INSERT 0 3
tutorial=# select * from conditions;
time | location | temperature | humidity
-------------------------------+----------+-------------+----------
2019-12-06 20:12:06.987648+00 | office | 70 | 50
2019-12-06 20:12:06.987648+00 | basement | 66.5 | 60
2019-12-06 20:12:06.987648+00 | garage | 77 | 65.2
(3 rows)
Querying
tutorial=# SELECT time_bucket('15 minutes', time) AS fifteen_min,
tutorial-# location, COUNT(*),
tutorial-# MAX(temperature) AS max_temp,
tutorial-# MAX(humidity) AS max_hum
tutorial-# FROM conditions
tutorial-# WHERE time > NOW() - interval '3 hours'
tutorial-# GROUP BY fifteen_min, location
tutorial-# ORDER BY fifteen_min DESC, max_temp DESC;
fifteen_min | location | count | max_temp | max_hum
------------------------+----------+-------+----------+---------
2019-12-06 20:00:00+00 | garage | 1 | 77 | 65.2
2019-12-06 20:00:00+00 | office | 1 | 70 | 50
2019-12-06 20:00:00+00 | basement | 1 | 66.5 | 60
(3 rows)
Q & A
References
• https://docs.timescale.com/latest/introduction
• https://www.youtube.com/watch?v=F-UGFSGlzsk
• https://blog.timescale.com/blog/building-columnar-compression-in-a-row-oriented-
database/
1 of 19

Recommended

Introduction to ML with Apache Spark MLlib by
Introduction to ML with Apache Spark MLlibIntroduction to ML with Apache Spark MLlib
Introduction to ML with Apache Spark MLlibTaras Matyashovsky
5.8K views88 slides
MLlib: Spark's Machine Learning Library by
MLlib: Spark's Machine Learning LibraryMLlib: Spark's Machine Learning Library
MLlib: Spark's Machine Learning Libraryjeykottalam
9.9K views33 slides
Security Best Practice: Oracle passwords, but secure! by
Security Best Practice: Oracle passwords, but secure!Security Best Practice: Oracle passwords, but secure!
Security Best Practice: Oracle passwords, but secure!Stefan Oehrli
8.1K views48 slides
Introduction to Apache ActiveMQ Artemis by
Introduction to Apache ActiveMQ ArtemisIntroduction to Apache ActiveMQ Artemis
Introduction to Apache ActiveMQ ArtemisYoshimasa Tanabe
4.7K views24 slides
Intro to Apache Spark by
Intro to Apache SparkIntro to Apache Spark
Intro to Apache SparkRobert Sanders
4.5K views45 slides
Oracle MAA (Maximum Availability Architecture) 18c - An Overview by
Oracle MAA (Maximum Availability Architecture) 18c - An OverviewOracle MAA (Maximum Availability Architecture) 18c - An Overview
Oracle MAA (Maximum Availability Architecture) 18c - An OverviewMarkus Michalewicz
1.6K views40 slides

More Related Content

What's hot

Building a Scalable Record Linkage System with Apache Spark, Python 3, and Ma... by
Building a Scalable Record Linkage System with Apache Spark, Python 3, and Ma...Building a Scalable Record Linkage System with Apache Spark, Python 3, and Ma...
Building a Scalable Record Linkage System with Apache Spark, Python 3, and Ma...Databricks
2.6K views27 slides
Powering Custom Apps at Facebook using Spark Script Transformation by
Powering Custom Apps at Facebook using Spark Script TransformationPowering Custom Apps at Facebook using Spark Script Transformation
Powering Custom Apps at Facebook using Spark Script TransformationDatabricks
777 views56 slides
14- Tumbling Window Trigger dependency in Azure Data Factory.pptx by
14- Tumbling Window Trigger dependency in Azure Data Factory.pptx14- Tumbling Window Trigger dependency in Azure Data Factory.pptx
14- Tumbling Window Trigger dependency in Azure Data Factory.pptxBRIJESH KUMAR
102 views8 slides
Splunking the JVM by
Splunking the JVMSplunking the JVM
Splunking the JVMDamien Dallimore
3.7K views19 slides
Working With a Real-World Dataset in Neo4j: Import and Modeling by
Working With a Real-World Dataset in Neo4j: Import and ModelingWorking With a Real-World Dataset in Neo4j: Import and Modeling
Working With a Real-World Dataset in Neo4j: Import and ModelingNeo4j
3.2K views56 slides
Tanel Poder - Performance stories from Exadata Migrations by
Tanel Poder - Performance stories from Exadata MigrationsTanel Poder - Performance stories from Exadata Migrations
Tanel Poder - Performance stories from Exadata MigrationsTanel Poder
6.3K views28 slides

What's hot(20)

Building a Scalable Record Linkage System with Apache Spark, Python 3, and Ma... by Databricks
Building a Scalable Record Linkage System with Apache Spark, Python 3, and Ma...Building a Scalable Record Linkage System with Apache Spark, Python 3, and Ma...
Building a Scalable Record Linkage System with Apache Spark, Python 3, and Ma...
Databricks2.6K views
Powering Custom Apps at Facebook using Spark Script Transformation by Databricks
Powering Custom Apps at Facebook using Spark Script TransformationPowering Custom Apps at Facebook using Spark Script Transformation
Powering Custom Apps at Facebook using Spark Script Transformation
Databricks777 views
14- Tumbling Window Trigger dependency in Azure Data Factory.pptx by BRIJESH KUMAR
14- Tumbling Window Trigger dependency in Azure Data Factory.pptx14- Tumbling Window Trigger dependency in Azure Data Factory.pptx
14- Tumbling Window Trigger dependency in Azure Data Factory.pptx
BRIJESH KUMAR102 views
Working With a Real-World Dataset in Neo4j: Import and Modeling by Neo4j
Working With a Real-World Dataset in Neo4j: Import and ModelingWorking With a Real-World Dataset in Neo4j: Import and Modeling
Working With a Real-World Dataset in Neo4j: Import and Modeling
Neo4j3.2K views
Tanel Poder - Performance stories from Exadata Migrations by Tanel Poder
Tanel Poder - Performance stories from Exadata MigrationsTanel Poder - Performance stories from Exadata Migrations
Tanel Poder - Performance stories from Exadata Migrations
Tanel Poder6.3K views
Building fast interpreters in Rust by Ingvar Stepanyan
Building fast interpreters in RustBuilding fast interpreters in Rust
Building fast interpreters in Rust
Ingvar Stepanyan4.7K views
Apache Camel K - Copenhagen v2 by Claus Ibsen
Apache Camel K - Copenhagen v2Apache Camel K - Copenhagen v2
Apache Camel K - Copenhagen v2
Claus Ibsen1.8K views
PySpark in practice slides by Dat Tran
PySpark in practice slidesPySpark in practice slides
PySpark in practice slides
Dat Tran3K views
Oracle Client Failover - Under The Hood by Ludovico Caldara
Oracle Client Failover - Under The HoodOracle Client Failover - Under The Hood
Oracle Client Failover - Under The Hood
Ludovico Caldara1.9K views
Zero-Copy Event-Driven Servers with Netty by Daniel Bimschas
Zero-Copy Event-Driven Servers with NettyZero-Copy Event-Driven Servers with Netty
Zero-Copy Event-Driven Servers with Netty
Daniel Bimschas22.5K views
Oracle GoldenGate 21c New Features and Best Practices by Bobby Curtis
Oracle GoldenGate 21c New Features and Best PracticesOracle GoldenGate 21c New Features and Best Practices
Oracle GoldenGate 21c New Features and Best Practices
Bobby Curtis2.1K views
A simple introduction to redis by Zhichao Liang
A simple introduction to redisA simple introduction to redis
A simple introduction to redis
Zhichao Liang2.9K views
Proxysql use case scenarios fosdem17 by Alkin Tezuysal
Proxysql use case scenarios    fosdem17Proxysql use case scenarios    fosdem17
Proxysql use case scenarios fosdem17
Alkin Tezuysal2.1K views
Deep Dive into the New Features of Apache Spark 3.1 by Databricks
Deep Dive into the New Features of Apache Spark 3.1Deep Dive into the New Features of Apache Spark 3.1
Deep Dive into the New Features of Apache Spark 3.1
Databricks661 views
BI, Reporting and Analytics on Apache Cassandra by Victor Coustenoble
BI, Reporting and Analytics on Apache CassandraBI, Reporting and Analytics on Apache Cassandra
BI, Reporting and Analytics on Apache Cassandra
Victor Coustenoble27K views
Apache Spark 2.0: A Deep Dive Into Structured Streaming - by Tathagata Das by Databricks
Apache Spark 2.0: A Deep Dive Into Structured Streaming - by Tathagata Das Apache Spark 2.0: A Deep Dive Into Structured Streaming - by Tathagata Das
Apache Spark 2.0: A Deep Dive Into Structured Streaming - by Tathagata Das
Databricks22.1K views
MongoDB - Aggregation Pipeline by Jason Terpko
MongoDB - Aggregation PipelineMongoDB - Aggregation Pipeline
MongoDB - Aggregation Pipeline
Jason Terpko1.2K views

Similar to pgday.seoul 2019: TimescaleDB

Re-Engineering PostgreSQL as a Time-Series Database by
Re-Engineering PostgreSQL as a Time-Series DatabaseRe-Engineering PostgreSQL as a Time-Series Database
Re-Engineering PostgreSQL as a Time-Series DatabaseAll Things Open
2.1K views74 slides
Benchmarking Solr Performance at Scale by
Benchmarking Solr Performance at ScaleBenchmarking Solr Performance at Scale
Benchmarking Solr Performance at Scalethelabdude
18.2K views30 slides
Owning time series with team apache Strata San Jose 2015 by
Owning time series with team apache   Strata San Jose 2015Owning time series with team apache   Strata San Jose 2015
Owning time series with team apache Strata San Jose 2015Patrick McFadin
4.7K views151 slides
iland Internet Solutions: Leveraging Cassandra for real-time multi-datacenter... by
iland Internet Solutions: Leveraging Cassandra for real-time multi-datacenter...iland Internet Solutions: Leveraging Cassandra for real-time multi-datacenter...
iland Internet Solutions: Leveraging Cassandra for real-time multi-datacenter...DataStax Academy
1.6K views63 slides
Leveraging Cassandra for real-time multi-datacenter public cloud analytics by
Leveraging Cassandra for real-time multi-datacenter public cloud analyticsLeveraging Cassandra for real-time multi-datacenter public cloud analytics
Leveraging Cassandra for real-time multi-datacenter public cloud analyticsJulien Anguenot
2.3K views63 slides
Plazma - Treasure Data’s distributed analytical database - by
Plazma - Treasure Data’s distributed analytical database -Plazma - Treasure Data’s distributed analytical database -
Plazma - Treasure Data’s distributed analytical database -Treasure Data, Inc.
12.6K views31 slides

Similar to pgday.seoul 2019: TimescaleDB(20)

Re-Engineering PostgreSQL as a Time-Series Database by All Things Open
Re-Engineering PostgreSQL as a Time-Series DatabaseRe-Engineering PostgreSQL as a Time-Series Database
Re-Engineering PostgreSQL as a Time-Series Database
All Things Open2.1K views
Benchmarking Solr Performance at Scale by thelabdude
Benchmarking Solr Performance at ScaleBenchmarking Solr Performance at Scale
Benchmarking Solr Performance at Scale
thelabdude18.2K views
Owning time series with team apache Strata San Jose 2015 by Patrick McFadin
Owning time series with team apache   Strata San Jose 2015Owning time series with team apache   Strata San Jose 2015
Owning time series with team apache Strata San Jose 2015
Patrick McFadin4.7K views
iland Internet Solutions: Leveraging Cassandra for real-time multi-datacenter... by DataStax Academy
iland Internet Solutions: Leveraging Cassandra for real-time multi-datacenter...iland Internet Solutions: Leveraging Cassandra for real-time multi-datacenter...
iland Internet Solutions: Leveraging Cassandra for real-time multi-datacenter...
DataStax Academy1.6K views
Leveraging Cassandra for real-time multi-datacenter public cloud analytics by Julien Anguenot
Leveraging Cassandra for real-time multi-datacenter public cloud analyticsLeveraging Cassandra for real-time multi-datacenter public cloud analytics
Leveraging Cassandra for real-time multi-datacenter public cloud analytics
Julien Anguenot2.3K views
Plazma - Treasure Data’s distributed analytical database - by Treasure Data, Inc.
Plazma - Treasure Data’s distributed analytical database -Plazma - Treasure Data’s distributed analytical database -
Plazma - Treasure Data’s distributed analytical database -
Treasure Data, Inc.12.6K views
Developing with Cassandra by Sperasoft
Developing with CassandraDeveloping with Cassandra
Developing with Cassandra
Sperasoft2.3K views
Speed up R with parallel programming in the Cloud by Revolution Analytics
Speed up R with parallel programming in the CloudSpeed up R with parallel programming in the Cloud
Speed up R with parallel programming in the Cloud
Revolution Analytics13.7K views
Cassandra and Spark by nickmbailey
Cassandra and SparkCassandra and Spark
Cassandra and Spark
nickmbailey1.7K views
Databases Have Forgotten About Single Node Performance, A Wrongheaded Trade Off by Timescale
Databases Have Forgotten About Single Node Performance, A Wrongheaded Trade OffDatabases Have Forgotten About Single Node Performance, A Wrongheaded Trade Off
Databases Have Forgotten About Single Node Performance, A Wrongheaded Trade Off
Timescale533 views
Infrastructure review - Shining a light on the Black Box by Miklos Szel
Infrastructure review - Shining a light on the Black BoxInfrastructure review - Shining a light on the Black Box
Infrastructure review - Shining a light on the Black Box
Miklos Szel8 views
Speeding up R with Parallel Programming in the Cloud by Revolution Analytics
Speeding up R with Parallel Programming in the CloudSpeeding up R with Parallel Programming in the Cloud
Speeding up R with Parallel Programming in the Cloud
Product Update: EDB Postgres Platform 2017 by EDB
Product Update: EDB Postgres Platform 2017Product Update: EDB Postgres Platform 2017
Product Update: EDB Postgres Platform 2017
EDB1.1K views
Speed Up Your Existing Relational Databases with Hazelcast and Speedment by Hazelcast
Speed Up Your Existing Relational Databases with Hazelcast and SpeedmentSpeed Up Your Existing Relational Databases with Hazelcast and Speedment
Speed Up Your Existing Relational Databases with Hazelcast and Speedment
Hazelcast1.6K views
Александр Соловьев "Cassandra in-e commerce". Выступление на Cassandra conf 2013 by it-people
Александр Соловьев "Cassandra in-e commerce". Выступление на Cassandra conf 2013Александр Соловьев "Cassandra in-e commerce". Выступление на Cassandra conf 2013
Александр Соловьев "Cassandra in-e commerce". Выступление на Cassandra conf 2013
it-people1.2K views
Apache cassandra & apache spark for time series data by Patrick McFadin
Apache cassandra & apache spark for time series dataApache cassandra & apache spark for time series data
Apache cassandra & apache spark for time series data
Patrick McFadin48.1K views
Overview of data analytics service: Treasure Data Service by SATOSHI TAGOMORI
Overview of data analytics service: Treasure Data ServiceOverview of data analytics service: Treasure Data Service
Overview of data analytics service: Treasure Data Service
SATOSHI TAGOMORI2.6K views

More from Chan Shik Lim

FPV Streaming Server with ffmpeg by
FPV Streaming Server with ffmpegFPV Streaming Server with ffmpeg
FPV Streaming Server with ffmpegChan Shik Lim
737 views21 slides
Improving monitoring systems Interoperability with OpenMetrics by
Improving monitoring systems Interoperability with OpenMetricsImproving monitoring systems Interoperability with OpenMetrics
Improving monitoring systems Interoperability with OpenMetricsChan Shik Lim
444 views30 slides
Kubernetes on Premise Practical Guide by
Kubernetes on Premise Practical GuideKubernetes on Premise Practical Guide
Kubernetes on Premise Practical GuideChan Shik Lim
4.1K views64 slides
Kubernetes on Premise by
Kubernetes on PremiseKubernetes on Premise
Kubernetes on PremiseChan Shik Lim
504 views45 slides
Hadoop High Availability Summary by
Hadoop High Availability SummaryHadoop High Availability Summary
Hadoop High Availability SummaryChan Shik Lim
170 views24 slides
Python Programming: Tuning and Optimization by
Python Programming: Tuning and OptimizationPython Programming: Tuning and Optimization
Python Programming: Tuning and OptimizationChan Shik Lim
1.1K views19 slides

More from Chan Shik Lim(10)

FPV Streaming Server with ffmpeg by Chan Shik Lim
FPV Streaming Server with ffmpegFPV Streaming Server with ffmpeg
FPV Streaming Server with ffmpeg
Chan Shik Lim737 views
Improving monitoring systems Interoperability with OpenMetrics by Chan Shik Lim
Improving monitoring systems Interoperability with OpenMetricsImproving monitoring systems Interoperability with OpenMetrics
Improving monitoring systems Interoperability with OpenMetrics
Chan Shik Lim444 views
Kubernetes on Premise Practical Guide by Chan Shik Lim
Kubernetes on Premise Practical GuideKubernetes on Premise Practical Guide
Kubernetes on Premise Practical Guide
Chan Shik Lim4.1K views
Hadoop High Availability Summary by Chan Shik Lim
Hadoop High Availability SummaryHadoop High Availability Summary
Hadoop High Availability Summary
Chan Shik Lim170 views
Python Programming: Tuning and Optimization by Chan Shik Lim
Python Programming: Tuning and OptimizationPython Programming: Tuning and Optimization
Python Programming: Tuning and Optimization
Chan Shik Lim1.1K views
Python Programming: Data Structure by Chan Shik Lim
Python Programming: Data StructurePython Programming: Data Structure
Python Programming: Data Structure
Chan Shik Lim337 views
Python Programming: Class and Object Oriented Programming by Chan Shik Lim
Python Programming: Class and Object Oriented ProgrammingPython Programming: Class and Object Oriented Programming
Python Programming: Class and Object Oriented Programming
Chan Shik Lim519 views
Python Programming: Function by Chan Shik Lim
Python Programming: FunctionPython Programming: Function
Python Programming: Function
Chan Shik Lim599 views
Python Programming: Type and Object by Chan Shik Lim
Python Programming: Type and ObjectPython Programming: Type and Object
Python Programming: Type and Object
Chan Shik Lim668 views

Recently uploaded

Unit 1_Lecture 2_Physical Design of IoT.pdf by
Unit 1_Lecture 2_Physical Design of IoT.pdfUnit 1_Lecture 2_Physical Design of IoT.pdf
Unit 1_Lecture 2_Physical Design of IoT.pdfStephenTec
12 views36 slides
PRODUCT LISTING.pptx by
PRODUCT LISTING.pptxPRODUCT LISTING.pptx
PRODUCT LISTING.pptxangelicacueva6
14 views1 slide
STPI OctaNE CoE Brochure.pdf by
STPI OctaNE CoE Brochure.pdfSTPI OctaNE CoE Brochure.pdf
STPI OctaNE CoE Brochure.pdfmadhurjyapb
14 views1 slide
handbook for web 3 adoption.pdf by
handbook for web 3 adoption.pdfhandbook for web 3 adoption.pdf
handbook for web 3 adoption.pdfLiveplex
22 views16 slides
Transcript: The Details of Description Techniques tips and tangents on altern... by
Transcript: The Details of Description Techniques tips and tangents on altern...Transcript: The Details of Description Techniques tips and tangents on altern...
Transcript: The Details of Description Techniques tips and tangents on altern...BookNet Canada
136 views15 slides
HTTP headers that make your website go faster - devs.gent November 2023 by
HTTP headers that make your website go faster - devs.gent November 2023HTTP headers that make your website go faster - devs.gent November 2023
HTTP headers that make your website go faster - devs.gent November 2023Thijs Feryn
22 views151 slides

Recently uploaded(20)

Unit 1_Lecture 2_Physical Design of IoT.pdf by StephenTec
Unit 1_Lecture 2_Physical Design of IoT.pdfUnit 1_Lecture 2_Physical Design of IoT.pdf
Unit 1_Lecture 2_Physical Design of IoT.pdf
StephenTec12 views
STPI OctaNE CoE Brochure.pdf by madhurjyapb
STPI OctaNE CoE Brochure.pdfSTPI OctaNE CoE Brochure.pdf
STPI OctaNE CoE Brochure.pdf
madhurjyapb14 views
handbook for web 3 adoption.pdf by Liveplex
handbook for web 3 adoption.pdfhandbook for web 3 adoption.pdf
handbook for web 3 adoption.pdf
Liveplex22 views
Transcript: The Details of Description Techniques tips and tangents on altern... by BookNet Canada
Transcript: The Details of Description Techniques tips and tangents on altern...Transcript: The Details of Description Techniques tips and tangents on altern...
Transcript: The Details of Description Techniques tips and tangents on altern...
BookNet Canada136 views
HTTP headers that make your website go faster - devs.gent November 2023 by Thijs Feryn
HTTP headers that make your website go faster - devs.gent November 2023HTTP headers that make your website go faster - devs.gent November 2023
HTTP headers that make your website go faster - devs.gent November 2023
Thijs Feryn22 views
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas... by Bernd Ruecker
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...
Bernd Ruecker37 views
Five Things You SHOULD Know About Postman by Postman
Five Things You SHOULD Know About PostmanFive Things You SHOULD Know About Postman
Five Things You SHOULD Know About Postman
Postman33 views
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N... by James Anderson
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...
James Anderson85 views
The details of description: Techniques, tips, and tangents on alternative tex... by BookNet Canada
The details of description: Techniques, tips, and tangents on alternative tex...The details of description: Techniques, tips, and tangents on alternative tex...
The details of description: Techniques, tips, and tangents on alternative tex...
BookNet Canada127 views
Voice Logger - Telephony Integration Solution at Aegis by Nirmal Sharma
Voice Logger - Telephony Integration Solution at AegisVoice Logger - Telephony Integration Solution at Aegis
Voice Logger - Telephony Integration Solution at Aegis
Nirmal Sharma39 views
SAP Automation Using Bar Code and FIORI.pdf by Virendra Rai, PMP
SAP Automation Using Bar Code and FIORI.pdfSAP Automation Using Bar Code and FIORI.pdf
SAP Automation Using Bar Code and FIORI.pdf
STKI Israeli Market Study 2023 corrected forecast 2023_24 v3.pdf by Dr. Jimmy Schwarzkopf
STKI Israeli Market Study 2023   corrected forecast 2023_24 v3.pdfSTKI Israeli Market Study 2023   corrected forecast 2023_24 v3.pdf
STKI Israeli Market Study 2023 corrected forecast 2023_24 v3.pdf
Data Integrity for Banking and Financial Services by Precisely
Data Integrity for Banking and Financial ServicesData Integrity for Banking and Financial Services
Data Integrity for Banking and Financial Services
Precisely21 views

pgday.seoul 2019: TimescaleDB

  • 1. TimescaleDB: Building a scalable time-series database on PostgreSQL Chanshik Lim Developer at NexCloud chanshik@gmail.com
  • 2. Agenda • Time-series Data? • TimescaleDB Overview • Using TimescaleDB • Q & A
  • 4. Time-series Data? (1) timestamp device_id cpu_1m_avg free_mem temperature location_id dev_type 2017-01-01 01:02:00 abc123 80 500MB 72 335 field 2017-01-01 01:02:23 def456 90 400MB 64 335 roof 2017-01-01 01:02:30 ghi789 120 0MB 56 77 roof 2017-01-01 01:03:12 abc123 80 500MB 72 335 field 2017-01-01 01:03:35 def456 95 350MB 64 335 roof 2017-01-01 01:03:42 ghi789 100 100MB 56 77 roof
  • 5. Time-series Data? (2) • Time-centric • Data records always have a timestamp • Append-only • Data is almost solely append-only (INSERTs) • Recent • New data is typically about recent time intervals
  • 6. Time-series Data? (3) • Monitoring computer systems • VM, server, container metrics (CPU, free memory, net/disk IOPS) • Service and application metrics (request rates, request latency) • Financial trading systems • Classic securities, newer cryptocurrencies, payments, transaction events • Internet of Things • Data from sensors on industrial machines and equipment • Eventing applications • User/customer interaction data like clickstreams, pageviews, logins, singups • Environmental monitoring • Temperature, humidity, pressure, pH, pollen count, air flow, …
  • 8. Easy to Use • Full SQL interface for all SQL natively supported by PostgreSQL • Secondary indexes • Non time-based aggregates • Sub-queries • Window functions • Connects to any client or tool that speaks PostgresSQL • Time-oriented features • Robust support for Data retention policies
  • 9. Scalable • Transparent time/space partitioning • Scaling up (single node) • Scaling out (private beta) • High data write rates • Right-sized chunks • Parallelized operations across chunks and servers
  • 10. Reliable • Engineered up from PostgreSQL, packaged as an extension • Proven foundations • From 20+ years of PostgreSQL research • Streaming replication • Backups • Flexible management options • Compatible with existing PostgreSQL ecosystem and tooling
  • 11. Architecture • Hypertables • Abstraction of a single continuous table across all space and time intervals • Chunks • Each chunk corresponds to a specific time interval and a region of partition key’s space
  • 13. Installing • https://docs.timescale.com/latest/getting-started/installation • Using Docker Image • shm-size: set /dev/shm partition size • Mapping /var/lib/postgresql/data to host directory $ docker run -d --name timescaledb -p 5432:5432 -e POSTGRES_PASSWORD=password -v /mnt/timescaledb:/var/lib/postgresql/data --shm-size 1G timescale/timescaledb:1.5.1-pg11
  • 14. Setting up $ psql -U postgres -h localhost postgres=# create database tutorial; CREATE DATABASE postgres=# c tutorial You are now connected to database "tutorial" as user "postgres". tutorial=# create extension if not exists timescaledb cascade; NOTICE: extension "timescaledb" already exists, skipping CREATE EXTENSION
  • 15. Creating a Hypertable tutorial=# CREATE TABLE conditions ( tutorial(# time TIMESTAMPTZ NOT NULL, tutorial(# location TEXT NOT NULL, tutorial(# temperature DOUBLE PRECISION NULL, tutorial(# humidity DOUBLE PRECISION NULL tutorial(# ); CREATE TABLE tutorial=# SELECT create_hypertable('conditions', 'time’, chunk_time_interval => interval '1 day'); create_hypertable ------------------------- (1,public,conditions,t) (1 row)
  • 16. Inserting tutorial=# INSERT INTO conditions tutorial-# VALUES tutorial-# (NOW(), 'office', 70.0, 50.0), tutorial-# (NOW(), 'basement', 66.5, 60.0), tutorial-# (NOW(), 'garage', 77.0, 65.2); INSERT 0 3 tutorial=# select * from conditions; time | location | temperature | humidity -------------------------------+----------+-------------+---------- 2019-12-06 20:12:06.987648+00 | office | 70 | 50 2019-12-06 20:12:06.987648+00 | basement | 66.5 | 60 2019-12-06 20:12:06.987648+00 | garage | 77 | 65.2 (3 rows)
  • 17. Querying tutorial=# SELECT time_bucket('15 minutes', time) AS fifteen_min, tutorial-# location, COUNT(*), tutorial-# MAX(temperature) AS max_temp, tutorial-# MAX(humidity) AS max_hum tutorial-# FROM conditions tutorial-# WHERE time > NOW() - interval '3 hours' tutorial-# GROUP BY fifteen_min, location tutorial-# ORDER BY fifteen_min DESC, max_temp DESC; fifteen_min | location | count | max_temp | max_hum ------------------------+----------+-------+----------+--------- 2019-12-06 20:00:00+00 | garage | 1 | 77 | 65.2 2019-12-06 20:00:00+00 | office | 1 | 70 | 50 2019-12-06 20:00:00+00 | basement | 1 | 66.5 | 60 (3 rows)
  • 18. Q & A
  • 19. References • https://docs.timescale.com/latest/introduction • https://www.youtube.com/watch?v=F-UGFSGlzsk • https://blog.timescale.com/blog/building-columnar-compression-in-a-row-oriented- database/