Why use a columnar database for analytical workloads by Shane Johnson, Senior Director, MariaDB In this session, we’re going to discuss how columnar databases the improve performance and efficiency of analytical workloads. We’ll begin by explaining why transactional queries (e.g., return every column in a single row) benefit from row-based storage whereas analytical queries (e.g., return the aggregate of a single column in every row) benefit from column-based storage. We will walk through the storage and query processing architecture of MariaDB AX, an open source columnar database, to show how columnar databases work. In addition, we will show how massively parallel processing, combined with column-based storage, not only improves the performance and efficiency of analytical workloads, but scales to support interactive, ad hoc analytical queries on terabytes of data and billions of rows in real time.

1. Why use a columnar database
for analytical workloads
Shane Johnson
Senior Director of Product Marketing

2. Introducing MariaDB AX
Why use a columnar database
for analytical workloads

3. MariaDB Server
InnoDB
MariaDB TX MariaDB AX
Optimized for OLTP
Row-based storage
Optimized for OLAP
Columnar storage

4. MariaDB AX and columnar database use cases
– agenda –
MariaDB AX
Introduction
Architecture
Use cases
Financial services
Healthcare
Telecommunications
Digital advertising

5. A database platform for
modern analytics and data
warehousing.
MariaDB AX and columnar database use cases
– introduction –
Distributed data
Columnar storage
Parallel processing
Data adapters
Connectors (Spark & Kafka)
Open source
Standard SQL

6. MariaDB AX architecture
Why use a columnar database
for analytical workloads

7. MariaDB Server
InnoDB
customer month year spend
1 01 2018 100.00
2 02 2018 50.00
3 03 2018 75.00
10000000 03 2018 75.00

8. MariaDB Server
InnoDB
customer month year spend
1 01 2018 100.00
2 02 2018 50.00
3 03 2018 75.00
10000000 03 2018 75.00
SELECT AVG(spend)
FROM tbl_purchases

9. MariaDB Server
InnoDB
ColumnStore
ColumnStore
Storage
customer month year spend
1 01 2018 100.00
2 02 2018 50.00
3 03 2018 75.00
10000000 03 2018 75.00

10. MariaDB Server
InnoDB
ColumnStore
ColumnStore
Storage
customer month year spend
1 01 2018 100.00
2 02 2018 50.00
3 03 2018 75.00
10000000 03 2018 75.00
SELECT AVG(spend)
FROM tbl_purchases

11. MariaDB Server
InnoDB
ColumnStore
ColumnStore
Storage
customer month year spend
1 01 2018 100.00
2 02 2018 50.00
3 03 2018 75.00
10000000 03 2018 75.00
1. high compression (65-95%)
2. supports spare columns (NULL)
3. supports many columns
4. no need for indexes

12. MariaDB Server
InnoDB
ColumnStore
ColumnStore
Storage
customer month year spend
1 01 2018 100.00
2 02 2018 50.00
3 03 2018 75.00
10000000 03 2018 75.00
1. columns stored as segments (file)
2. segments have extents (logical)
3. extents have 8 million rows

14. Performance Module (PM)
User Module (UM)
MariaDB Server
InnoDB
ColumnStore
ColumnStore
Storage

15. S1 (UM + PM)
MariaDB Server
InnoDB
ColumnStore
ColumnStore
Storage

16. S1 (UM)
MariaDB Server
InnoDB
ColumnStore
ColumnStore
Storage
S2 (PM)
Rows 1 to 300,000

17. S4 (PM)S2 (PM)
S1 (UM)
MariaDB Server
InnoDB
ColumnStore
ColumnStore
Storage
S3 (PM)
ColumnStore
Storage
ColumnStore
Storage
Rows 1 to 100,000 Rows 100,001 to 200,000 Rows 200,001 to 300,000

18. S4 (PM)S2 (PM)
S1 (UM)
MariaDB Server
InnoDB
ColumnStore
ColumnStore
Storage
S3 (PM)
ColumnStore
Storage
ColumnStore
Storage
T1: 00,001 to 30,000
T2: 30,001 to 60,000
T4: 100,001 to 130,000
T5: 130,001 to 160,000
T7: 200,001 to 230,000
T8: 230,001 to 260,000

19. S6 (PM)S4 (PM)
S2 (UM)
MariaDB Server
InnoDB
ColumnStore
ColumnStore
Storage
S5 (PM)
ColumnStore
Storage
S1 (UM)
MariaDB Server
InnoDB
ColumnStore
S3 (UM)
MariaDB Server
InnoDB
ColumnStore
ColumnStore
Storage
Query
throughput
Query
latency

20. S2 (PM)
S1 (UM)
MariaDB Server
InnoDB
ColumnStore
ColumnStore
Storage
SQL
S4 (PM)
ColumnStore
Storage
S6 (PM)
ColumnStore
Storage

21. S1 (UM)
S2 (PM)
ColumnStore
Storage
Import (CLI)
S4 (PM)
ColumnStore
Storage
S6 (PM)
ColumnStore
Storage
File

22. S2 (PM)
ColumnStore
Storage
S4 (PM)
ColumnStore
Storage
S6 (PM)
ColumnStore
Storage
Import (CLI)
File (2/3)
Import (CLI)
File (1/3)
Import (CLI)
File (3/3)

23. Application/Service/Script
S2 (PM)
ColumnStore
Storage
Bulk data
adapter
S4 (PM)
ColumnStore
Storage
S6 (PM)
ColumnStore
Storage

24. Spark job/task
S2 (PM)
ColumnStore
Storage
Spark
Connector
S4 (PM)
ColumnStore
Storage
S6 (PM)
ColumnStore
Storage

25. Bulk adapters
S1 (UM)
MariaDB Server
InnoDB
ColumnStore
ColumnStore
C
Java
Python
Stream adapters
Kafka
MaxScale CDCSpark
Storage
SQL
Import (CSV)S2 (PM)

26. MariaDB AX
customer use cases
Why use a columnar database
for analytical workloads

27. MariaDB AX and columnar database use cases
– financial services –
Drivers
Become customer-centric
Facilitate regulatory compliance
Create competitive advantages
Goals
Improve customer satisfaction
Mitigate financial risks
Predict market changes
Use cases
Fraud detection: identify patterns + detect anomalies in financial transactions
Compliance archiving: store financial trade history for long-term retention
Investment forecasting: analyze financial markets + securities to predict ROI

28. MariaDB AX and columnar database use cases
– OTC Markets Group –
Data
10TB of rolling data (5 years)
10,000 U.S. and global securities
100,000 trades
24 million quotes
Use cases
Subscribers analyze quote and trading
data
Regulatory agencies build compliance
reports on demand

29. MariaDB AX and columnar database use cases
– healthcare –
Drivers
Digital transformation
Electronic health records (EHRs)
Value-base care (VBC)
Goals
Improved population health
Better patient experiences
Reduced cost of care
Use cases
Population health mgt: analyze claims/surveys to recommend interventins
Evidence-based medicine: improve diagnostic accuracy by analyzing EHRs
Precision medicine: identify targeted treatments by analyzing genomes

30. MariaDB AX and columnar database use cases
– Institute for Health Metrics and Evaluation –
Data
30TB of data
100 billion data points
Multi-billion row tables
Use cases
Enable the public to analyze global
health population data via online data
visualization tools

32. MariaDB AX and columnar database use cases
– telecommunications –
Drivers
Improve sales, marketing and
operational efficiency
Goals
High customer retention
Better network optimizations
New services and revenue
Use cases
Churn prevention: analyze customer plans/usage to create retention programs
Cross-selling: identify opportunities by analyzing call detail records (CDRs)
Network optimization: analyze traffic/cell tower data to optimize capacity

33. MariaDB AX and columnar database use cases
– Pinger, Inc. –
About
30 million texts
3 million phone calls
1.5 billions logs a day
24 months’ worth of data
Use cases
To support customer behavioural
analysis based on historical data and
usage

34. MariaDB AX and columnar database use cases
– digital advertising –
Drivers
Granularity of data
Demographic and behavioral
Social and location
Goals
Deliver the right ad to the right person at
the right time, in the right location and
through the right medium
Use cases
Audience segmentation: improve ad relevance via fine-grained visitor profiles
Ad placement: choose where to show ads based on click and conversion data
Real-time bidding: analyze big request/response history to optimize prices

35. MariaDB AX and columnar database use cases
– digital advertising vendor –
About
300 million impressions a month
70 million rows a day
60TB of uncompressed data
Use cases
Enable customers to create a custom
report on up to 30 columns on
demand

36. Why use a columnar database
for analytical workloads
Questions?

37. Thank you