PGConf.ASIA 2019 Bali - 10 September 2019 Speaker: Koichi Suzuki Room: SQL Title: IoT and PostgreSQL

1. https://www.2ndQuadrant.com
Koichi Suzuki
PGConf.Asia 2019

2. https://www.2ndQuadrant.com
Standard in IoT
oneM2M: Cross-application service spec.
https://portail-qualite.public.lu/dam-assets/publications/normalisation/2018/workshop-etsi/4-xavier-piednoir-onem2m-workshop-ilnas-etsi.pdf

3. https://www.2ndQuadrant.com
OneM2M: Layer and interfaces
https://portail-qualite.public.lu/dam-assets/publications/normalisation/2018/workshop-etsi/4-xavier-piednoir-onem2m-workshop-ilnas-etsi.pdf
Underlying database design looks to be left to us

4. https://www.2ndQuadrant.com
Characteristics/Requirement of IoT
● Analytic queries log and time series
○ Data is stati, not updated
● Scalability
○ Geographical distribution
○ Huge number of devices
● Integration with other platforms
○ Streaming query
○ KAFKA, AMQP (RabbitMQ etc)

5. https://www.2ndQuadrant.com
Analytic Queries
● Aggregate functions
● Window functions
● CTE (WITH clause)

6. https://www.2ndQuadrant.com
Example of IoT data table

7. https://www.2ndQuadrant.com
Aggregate function
min/max reading of the measurement

8. https://www.2ndQuadrant.com
Which device read MIN?

9. https://www.2ndQuadrant.com
Ranking of the reading (window function)

10. https://www.2ndQuadrant.com
Other kind of ranking

11. https://www.2ndQuadrant.com
Reading difference from previous rank

12. https://www.2ndQuadrant.com
Maximum reading difference

13. https://www.2ndQuadrant.com
Other useful window functions
● row_number()
● percent_rank()
● cume_dist()
● ntile(num_buckets integer)
● lead(value anyelement [,
offset integer [, default anyelement ]])
● first_value(value any)
● last_value(value any)
● nth_value(value any, nth integer)

14. https://www.2ndQuadrant.com
Log and time series data
● Ideal for partitioning
○ Timestamp based partitioning
○ Region/device type based partitioning
○ Multi-dimensional partitioning
○ PostgreSQL native partitioning
○ Pg_partman extension

15. https://www.2ndQuadrant.com
Partition over Time

16. https://www.2ndQuadrant.com
Table definition

17. https://www.2ndQuadrant.com
Partition definitions

18. https://www.2ndQuadrant.com
See partitions

19. https://www.2ndQuadrant.com
Write rows to partitioned table
same as usual table

20. https://www.2ndQuadrant.com
Read as parent table

21. https://www.2ndQuadrant.com
Rows are stored in partition

22. https://www.2ndQuadrant.com
Multi Dimensional Partitioning

23. https://www.2ndQuadrant.com
Parent table definition

24. https://www.2ndQuadrant.com
Partitions for individual devices

25. https://www.2ndQuadrant.com
Partition of each device ‘a’ over time

26. https://www.2ndQuadrant.com
Partition of each device ‘b’ over time

27. https://www.2ndQuadrant.com
Read different level of partitions

28. https://www.2ndQuadrant.com
Pg_partman for easy partition maintenance
create extension

29. https://www.2ndQuadrant.com
Create parent table

30. https://www.2ndQuadrant.com
Create partition

31. https://www.2ndQuadrant.com
Subpartition with pg_partman

32. https://www.2ndQuadrant.com
Subpartition with pg_partman (2)

33. https://www.2ndQuadrant.com
Scale out with BDR
BDR intruduction
● Multi-Master PostgreSQL extension from 2nd
Quadrant
● Based upon pglogical and logical replication
○ Logical level WAL
○ Extension for multi-master infrastructure

34. https://www.2ndQuadrant.com
2ndQuadrant
PostgreSQL Solutions for the Enterprise
● AlwaysOn Architecture
○ Up to 5 nines of availability
● Rolling Upgrades
○ One-by-one upgrades with zero downtime
● Geographically Distributed Clusters
○ ‘Eager’ and ‘Efficient’ modes to suit your
latency tolerance
● Point-in-Time Recovery
○ Guaranteed full disaster recovery
Postgres-BDR
AlwaysOn Multi-master Replication for Distributed PostgreSQL

35. https://www.2ndQuadrant.com
Scale out with BDR
Local device
writes to local
database
Site 1
Site 3
Site 2
Site N
Table-wise logial bi-directional replication
Any writes propagate to all the otherGlobal N/W

36. https://www.2ndQuadrant.com
Thank you very much!

37. https://www.2ndQuadrant.com
OneM2M: Process
https://www.w3.org/2017/05/wot-f2f/slides/20170516-W3C-oneM2M.pdf