Some basic thoughts about BigData and RDBMS

1. BigData and open-source RDBMSs
Alexander Tokarev

2. Who am I
1. Cloud architect:
- Solution design
- Best practices
- Performance tuning
2. Enterprise databases Oracle - 2001, Oracle 8.1.7 – 19c
3. Cloud databases: Redshift, Athena, BigQuery, SnowFlake

3. Agenda
1.Big Data
2.Postgres ecosystem
3.Others vendors
4.Conclusion
5.Q&A

4. Big Data
1.Velocity
2.Volume
3.Variety
4.…………….
5.…………….
6.Vxxxxxxxxx

5. Big Data in RDBMS
1.Velocity
2.Volume

6. Big Data ingestion nature
1.batch/micro-batch
2.realtime

7. Big Data nature
1.Time series data: date, numeric metrics
• Append-only mostly
• Huge volume of inserts
2.Analytical data: date, dimensions, measures
• A lot of dimensions
• Dimensions are subjects for update
• Moderate ingestion speed

8. Big Data data models
1.Star-style normalized
2.Wide tables
3.Nested structures

9. Postgres eco-system
1.Postgres
2.Citus
3.Timescale
4.Greenplum

10. Postgres
1.Classical RDBMS
2.Very good for OLTP
3.Our experience – max DB size 5 TB
4.Partitioning is fine from PG 12
5.Row-based database
6.Search conditions predefined

11. Postgres improvements types
1.Fork
+ more control
- major version faster
2.Extension
+ in sync with main version
- not all changes are possible

12. Timescale
1.Postgres extension
2.Time-series optimized database (like Prometheus)
3.Ingestion 10x faster than Postgres
4.Own partitioning implementation
5.Time-series-oriented SQL
6.Attempts to create stream processing with SQL
7.SQL is perfect to extract by dates

13. MPP databases
+
Very fast
Horizontally scalable
-
Different modeling approach
Like good disks
Hard to fined managed databases

14. Citus
1.Postgres extension
2.Owned by Microsoft
3.MPP tailored for OLTP – sharding for fast ingestion
4.Vendor states for OLAP as well – not true
5.SQL is perfect to extract data by shard key

15. Greenplum
1.Postgres fork
2.OLAP workload optimized MPP database
3.Perfect partitioning with automatic data lifecycle management
4.A lot of compression algorithms
5.Up to 100 TB clusters very fast
6.A lot of integrations: files, HDFS
7.Perfect SQL support – perfect for analytical queries

16. Big data platforms
1.Set of opensource components
2.Components for storage, processing, management
3.Storage components different by tiers:
- in-memory for hot
- MPP for warm
- Hadoop for cold

17. Big data platform Arenadata DB

18. Clickhouse
1. DBMS written from scratch
2. Written by developers for developers – Yandex.Metrica
3. Wide table optimized MPP columnar database
4. Extremely fast batch ingestion + InMemory buffer tables
5. Very fast queries for wide tables
6.Aggregations during ingestion + real-time materialized views
7. Hundreds TB clusters
8. Huge number of integrations: files, HDFS, Kafka, JDBC
9. Self-sufficient for ELT/ETL
10. Own SQL dialect

19. Summary
Postgres Citus Timescale Greenplum Clickhouse
Columnar storage - - - + +
SQL richness ++ + + + -
Integrations - - - + +++
In-flight aggregations - - -+ - +++
Ingestion style Per row Per row Per row Bulk Huge
batches
Ingestion performance + + ++++ ++ +++
Wide table oriented - - + + +++
Star-schema oriented - - - + -
Horizontal scaling - + ++ ++ +++
Managed in cloud +++ + + - -+

20. More questions?