Big Data Analytics with MariaDB ColumnStore provides an overview of MariaDB ColumnStore. Key points include: - MariaDB ColumnStore is an open source columnar storage engine that provides high performance analytics on large datasets in a scalable distributed environment using standard SQL. - Columnar storage organizes data by columns rather than rows, improving query performance by only accessing relevant columns. It supports workloads from terabytes to petabytes of data. - Common use cases include data warehousing, financial services, healthcare, telecom, and any workload requiring analysis of millions to billions of rows. - The architecture employs a distributed query processing model with horizontal partitioning and parallel query execution across nodes for high scalability

1. Big Data Analytics with
MariaDB ColumnStore

2. Agenda
Overview
Why Columnar?
Use Cases
Architecture and Features
What is new in 1.1?

3. MariaDB ColumnStore
• GPLv2 Open Source
• Columnar, Massively Parallel
MariaDB Storage Engine
• Scalable, high-performance
analytics platform
• Built in redundancy and
high availability
• Runs on premise, on AWS cloud
• Full SQL syntax and capabilities
regardless of platformBig Data Sources Analytics Insight
MariaDB ColumnStore
. . .
Node 1 Node 2 Node 3 Node N
Local / AWS®
/ GlusterFS
®
ELT
Tools
BI
Tools
Latest GA Version: 1.0.11
Current Beta Version: 1.1.0

4. MariaDB ColumnStore
High performance columnar storage engine that support wide variety of
analytical use cases with SQL in a highly scalable distributed environments
Parallel query
processing for
distributed
environments
Faster, More
Efficient Queries
Single SQL Interface
for OLTP and
analytics
Easier Enterprise
Analytics
Power of SQL and
Freedom of Open
Source to Big Data
Analytics
Better Price
Performance

5. Why Columnar ?
• Row oriented
– Rows stored
sequentially in a file
– Scans through every
record row by row
• Column oriented:
– Each column is stored
in a separate file
– Scans the only
relevant column
ID Fname Lname State Zip Phone Age Sex
1 Bugs Bunny NY 11217 (718) 938-3235 34 M
2 Yosemite Sam CA 95389 (209) 375-6572 52 M
3 Daffy Duck NY 10013 (212) 227-1810 35 M
4 Elmer Fudd ME 04578 (207) 882-7323 43 M
5 Witch Hazel MA 01970 (978) 744-0991 57 F
ID
1
2
3
4
5
Fname
Bugs
Yosemite
Daffy
Elmer
Witch
Lname
Bunny
Sam
Duck
Fudd
Hazel
State
NY
CA
NY
ME
MA
Zip
11217
95389
10013
04578
01970
Phone
(718) 938-3235
(209) 375-6572
(212) 227-1810
(207) 882-7323
(978) 744-0991
Age
34
52
35
43
57
Sex
M
M
M
M
F
SELECT Fname FROM Table 1 WHERE State = 'NY'

6. OLTP/NoSQL
Workloads
Suited for reporting or analysis of millions-billions of rows from data sets containing millions-trillions of rows.
OLAP/Analytic/
Reporting Workloads
Workload – Query Vision/Scope
1 100 10,000
10-100GB
10,000,000,000
1-10TB
1,000,000 100,000,000
100-1,000GB
InnoDB, MyRocks, MyISAM ColumnStore

7. Use Cases

8. Data Warehousing
Selective column
based queries
Large number of
dimensions
High Performance
Analytics On Large
Volume Of Data
Reporting and analysis
on millions or billions
of rows
From datasets
containing millions to
trillions of rows
Terabytes to Petabytes
of datasets
Analytics Require
Complex Joins,
Windowing Functions
Technical Use Cases

9. Financial
Services
Trade Analytics
• Analyze 20-30 million quotes per day
• Identify trade patterns and predict the outcome
Fraud Detection
• Fraudulent or anomaly trade detection among millions of transactions per day
• Proactively identify risks and prevent billions of loss due to fraud
Regulatory Compliance
• Archive historic transactional data
• FINRA, Dodd Frank Act, SEC, SOX

10. Health care /
Life Science
Genome analysis
• In-depth genome research for the dairy industry to improve production of milk and
protein.
• Fast data load for large amount of genome dataset (DNA data for 7billion cows in US -
20GB per load)
• SQL based analytics
Health care spending analysis
• Data size: 3TB
• Analyze US health care spending for 155 conditions with 7 years of
historical data
• Used sankey diagram, treemap, and pyramid chart to analyze trends by
age, sex, type of care, and condition
Viral disease analysis
• Used geospatial techniques with interactive map to identify Ebola disease
spread
• The map displays not only the existing transmission of Ebola virus, but also
the probability of occurence

11. Telecom
Customer behavior analysis
• Analyze call data record to segment customers based on their behavior
• Data-driven analysis for customer satisfaction
• Create behavioral based up-sell or cross-sell opportunity
Call data analysis
• Data size: 6TB
• Ingest 1.5 million rows of logs per day with 30million texts and 3million
calls
• Call and network quality analysis
• Provide higher quality customer services based on data

12. Architecture

13. MariaDB ColumnStore Architecture
Columnar Distributed Data Storage
User Connections
User Module nUser Module 1
Performance
Module n
Performance
Module 2
Performance
Module 1
MariaDB
Front End
Query Engine
User Module
Processes SQL Requests
Performance Module
Distributed Processing Engine

14. MariaDB ColumnStore
MariaDB ColumnStore
uses standard
“Engine=columnstore”
syntax
mysql> use tpcds_djoshi
Database changed
mysql> select count(*) from store_sales;
+----------+
| count(*) |
+----------+
| 2880404 |
+----------+
1 row in set (1.68 sec)
mysql> describe warehouse;
+-------------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------------+--------------+------+-----+---------+-------+
| w_warehouse_sk | int(11) | NO | | NULL | |
| w_warehouse_id | char(16) | NO | | NULL | |
| w_warehouse_name | varchar(20) | YES | | NULL | |
| w_warehouse_sq_ft | int(11) | YES | | NULL | |
| w_street_number | char(10) | YES | | NULL | |
| w_street_name | varchar(60) | YES | | NULL | |
| w_street_type | char(15) | YES | | NULL | |
| w_suite_number | char(10) | YES | | NULL | |
| w_city | varchar(60) | YES | | NULL | |
| w_county | varchar(30) | YES | | NULL | |
| w_state | char(2) | YES | | NULL | |
| w_zip | char(10) | YES | | NULL | |
| w_country | varchar(20) | YES | | NULL | |
| w_gmt_offset | decimal(5,2) | YES | | NULL | |
+-------------------+--------------+------+-----+---------+-------+
14 rows in set (0.05 sec)
CREATE TABLE `game_warehouse`.`dim_title` (
`id` INT,
`name` VARCHAR(45),
`publisher` VARCHAR(45),
`release_date` DATE,
`language` INT,
`platform_name` VARCHAR(45),
`version` VARCHAR(45)
) ENGINE=columnstore;
Uses custom scalable
columnar architecture

15. MariaDB ColumnStore
mysql> use tpcds_djoshi
Database changed
mysql> select count(*) from store_sales;
+----------+
| count(*) |
+----------+
| 2880404 |
+----------+
1 row in set (1.68 sec)
mysql> describe warehouse;
+-------------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------------+--------------+------+-----+---------+-------+
| w_warehouse_sk | int(11) | NO | | NULL | |
| w_warehouse_id | char(16) | NO | | NULL | |
| w_warehouse_name | varchar(20) | YES | | NULL | |
| w_warehouse_sq_ft | int(11) | YES | | NULL | |
| w_street_number | char(10) | YES | | NULL | |
| w_street_name | varchar(60) | YES | | NULL | |
| w_street_type | char(15) | YES | | NULL | |
| w_suite_number | char(10) | YES | | NULL | |
| w_city | varchar(60) | YES | | NULL | |
| w_county | varchar(30) | YES | | NULL | |
| w_state | char(2) | YES | | NULL | |
| w_zip | char(10) | YES | | NULL | |
| w_country | varchar(20) | YES | | NULL | |
| w_gmt_offset | decimal(5,2) | YES | | NULL | |
+-------------------+--------------+------+-----+---------+-------+
14 rows in set (0.05 sec)
MariaDB Front End
Standard ANSI SQL

16. Storage Architecture
Column 1
Extent 1 (8 million rows, 8MB～64MB)
Extent 2 (8 million rows)
Extent M (8 million rows)
Column 2 Column 3 ... Column N
Data automatically arranged by
• Column – Acts as Vertical Partitioning
• Extents – Acts as horizontal partition
...
Table
Logical View
8 million rows
• Columnar storage
– Each column stored as separate file
– No index management for query
performance tuning
– Online Schema changes: Add new column
without impacting running queries
• Automatic horizontal partitioning
– Logical partition every 8 Million rows
– In memory metadata of partition min and max
– No partition management for query performance
tuning
• Compression
– Default ON
– Accelerate decompression rate
– Reduce I/O for compressed blocks

17. Horizontal
Partition:
8 Million Rows
Extent 2
Horizontal
Partition:
8 Million Rows
Extent 3
Horizontal
Partition:
8 Million Rows
Extent 1
Storage Architecture reduces I/O
• Only touch column files
that are in projection, filter
and join conditions
• Eliminate disk block touches
to partitions outside filter
and join conditions
Extent 1:
Min State: CA, Max State: NY
Extent 2:
Min State: OR, Max State: WY
Extent 3:
Min State: IA, Max State: TN
SELECT Fname FROM Table 1 WHERE State = ‘NY’
High Performance Query Processing
ID
1
2
3
4
...
8M
8M+1
...
16M
16M+1
...
24M
Fname
Bugs
Yosemite
Daffy
Hazel
...
...
Jane
...
Elmer
Lname
Bunny
Sam
Duck
Fudd
...
...
...
State
NY
CA
NY
ME
...
MN
WY
TX
OR
...
VA
TN
IA
NY
...
PA
Zip
11217
95389
10013
04578
...
...
...
Phone
(718) 938-3235
(209) 375-6572
(212) 227-1810
(207) 882-7323
...
...
...
Age
34
52
35
43
...
...
...
Sex
M
M
M
F
...
...
...
Vertical
Partition
Vertical
Partition
Vertical
Partition
Vertical
Partition
Vertical
Partition
…
ELIMINATED PARTITION

18. SQL Features
Cross Engine
Joins
UDF
DML
Aggregation
DDL
Disk Based
Joins
Windowing
Functions
SELECT
QUERY

19. MAX RANK
MIN DENSE_RANK
COUNT PERCENT_RANK
SUM NTH_VALUE
AVG FIRST_VALUE
VARIANCE LAST_VALUE
VAR_POP CUME_DIST
VAR_SAMP LAG
STD LEAD
STDDEV NTILE
STDDEV_POP PERCENTILE_CONT
STDDEV_SAMP PERCENTILE_DISC
ROW_NUMBER MEDIAN
• Aggregate over a series of related rows
• Simplified function for complex statistical
analytics over sliding window per row
- Cumulative, moving or centered aggregates
- Simple Statistical functions like rank, max, min,
average, median
- More complex functions such as distribution,
percentile, lag, lead
- Without running complex sub-queries
Windowing Functions

20. Top N Visitors for each Month
Window Function Example
Total for Each
Visitor by Month
Top 1 :
Time_rank = 1
Top 2 :
Time_rank <= 2
Top N :
Time_rank <= N

21. High Performance Data Ingestion
• Fully parallel high speed data load
– Parallel data loads on all PMs simultaneously
– Multiple tables in can be loaded simultaneously
– Read queries continue without being blocked
• Micro-batch loading for real-time
data flow
Column 1
Extent 1 (8 million rows, 8MB～64MB)
Extent 2 (8 million rows)
Extent M (8 million rows)
Column 2 ... Column N
Horizontal
Partition
...
Horizontal
Partition
Horizontal
Partition
High Water Mark
New Data being loaded
Dataaccessedby
runningqueries

22. Enterprise Grade
• Enterprise grade security
– SSL, role based access, auditability
• Flexibility of Platform
– Run on on-premise using commodity
Linux servers
– Run on AWS
• High Availability
– Automatic UM failover
– Automatic PM failover with distributed
data attachment across all PMs in SAN
and EBS environment
User Module
Performance Module
Columnar Distributed Data Storage

23. MariaDB ColumnStore 1.0
Data Engine ● Columnar Engine based on MariaDB 10.1
Scale
● Columnar, Massively Parallel
● Linear scalability with automatic data partitioning
● Data compression designed to accelerate decompression rate, reducing disk I/O
Performance
● High performance analytics
● Columnar optimized, massively parallel, distributed query processing on commodity servers
Data Ingestion ● High speed parallel data load and extract without blocking reads
Analytics
● In database analytics with complex joins, windowing functions
● ACID Compliant
● Extensible User Defined Functions (UDF) for custom analytics
● Out of box BI Tools connectivity, Analytics integration with R
Enterprise Grade
● Cross join tables between MariaDB and ColumnStore for full insight
● SSL support, Auditability, Role Based Access
● Built-in High availability for UM and PM
Ease of Use
● Automatic horizontal partitioning
● No index, views or manual partition tuning needed
● Online schema changes while read queries continue
● Deploy anywhere on premise or cloud

24. What is new in 1.1 ?

25. ColumnStore 1.1 Features
Data Engine:
Streaming / API :
High Availability:
Analytics:
Data Types:
Ease of Use:
Performance:
Security:
Certifications:
Columnar Storage engine based on MariaDB Server 10.2
Bulk import API to support programmatic and streaming writes
Integrated GlusterFS support to provide storage HA for local disk
User Defined Aggregate / Window Functions
Text and Blob support
Backup and Restore Tool
Improved query and memory handling (5% faster than 1.0)
Audit Plugin integration
Tableau certification

26. Data Streaming: ColumnStore Data API
What:
• C++ API to directly write to PM nodes
• Per table write
• Input data is C++ data structure in API calls
• Can run remotely from UM and PM servers
• Bindings for Python, Go, and Java in progress (and other
languages as long as supported by SWIG).
Benefits:
● Real-time streaming directly into distributed data store
● No need to move large CSV data files to UM/PM
● Enable non-CSV data sources for columnstore
● Run outside UM/PM. Build custom ETL applications
https://mariadb.com/kb/en/library/columnstore-bulk-write-
sdk/
…
PM Node
Write
Engine
PM Node
Write
Engine
PM Node
Write
Engine
syslog Data Sources
Data Streaming
Application
CS Data API
Library

27. ColumnStore Data Adapters 1.1
What ?
• Pre-packaged data adapters written using CS data API
• Convert from a specific data source into MariaDB
ColumnStore
Benefits
● Out of box real time data streaming into CS
● No need to move large CSV data files to UM/PM
● Enable non-CSV data sources for columnstore
● Run outside UM/PM. Build custom ETL applications
MaxScale CDC
Adapter
…
PM Node
Write
Engine
PM Node
Write
Engine
PM Node
Write
Engine
CS Data API
Library
MaxScale CDC
API
Avro Adapter
CS Data API
Library
Kafka Consumer
Interface
MaxScale
MDB Master

28. MariaDB AX
● MariaDB ColumnStore releases
● MariaDB database proxy, MaxScale
● MariaDB Connectors
● 24x7x365 support
● 30-minute emergency response time
● Mission-critical patching
● Guaranteed version support
● Management and monitoring tools
● Installers
Modern data warehousing solution for large scale analytics
MariaDB ColumnStore
MariaDB MaxScale
MariaDB Connectors

29. Getting Started
• https://mariadb.com/kb/en/mariadb-columnstore/
• https://mariadb.com/downloads/mariadb-ax

30. Thank you