SlideShare a Scribd company logo
1 of 40
MARIADB
ColumnStore
Understanding the Architecture
Andrew Hutchings
Lead Software Engineer, MariaDB ColumnStore
MariaDB Corporation
Who Am I?
● Andrew Hutchings, aka “LinuxJedi”
● Lead Software Engineer for MariaDB’s ColumnStore
● Previous worked for:
○ NGINX - Senior Developer Advocate / Technical Product
Manager
○ HP - Principal Software Engineer (HP Cloud / ATG)
○ SkySQL - Senior Sustaining Engineer
○ Rackspace - Senior Software Engineer
○ Sun/Oracle - MySQL Senior Support Engineer
● Co-author of MySQL 5.1 Plugin Development
● IRC/Twitter: LinuxJedi
● EMail: linuxjedi@mariadb.com
Overview
● History of MariaDB ColumnStore
● Technical Use Case
● Components of MariaDB ColumnStore
● Disk Storage
● Writing Data
● Querying Data
● Optimizing for MariaDB ColumnStore
● Closing Notes
● Questions
History of MariaDB ColumnStore
● March 2010 - Calpont launches InfiniDB
● September 2014 - Calpont (now itself called InfiniDB) closes down
○ MariaDB (then SkySQL) supports InfiniDB customers
● April 2016 - MariaDB announces development of MariaDB ColumnStore
● August 2016 - I joined MariaDB and jumped straight into ColumnStore
● December 2016 - MariaDB ColumnStore 1.0 GA
○ InfiniDB + MariaDB 10.1 + Many fixes and improvements
● November 2017 - MariaDB ColumnStore 1.1 GA
○ MariaDB 10.2 + APIs + Even more improvements
● December 2018 - MariaDB ColumnStore 1.2 GA
○ MariaDB 10.3 + TIME, microseconds, UDAFs + Lots more
Technical Use Case
Technical Use Case
● Very large data sets
○ Many columns
○ Many millions of rows
● Complex joins and aggregates
● Rapid bulk data insertion
○ The larger the batch the better
MariaDB ColumnStore
● Smaller data sets
● Basic queries
● Lots of DML queries
● Complex data types
Traditional OLTP Engines
Data Types
● INT types - range is 2 less from max unsigned or min signed
● CHAR† - max 255 bytes
● VARCHAR† - max 8000 bytes
● DECIMAL - max 18 digits
● DOUBLE/FLOAT
● DATETIME - with sub-seconds in 1.2
● DATE
● TIME
● BLOB/TEXT†
† Empty string is the same as NULL
Other DDL Differences
● No indexes
○ Columns are somewhat self-indexing
● Auto increment is handled differently (a table comment)
● No constraints
● PARTITION syntax not supported
○ Columns are partitioned automatically
Row-oriented vs. Column-oriented Format
ID Fname Lname State Zip Phone Age Sex
1 Bugs Bunny NY 11217 (718) 938-3235 34 M
2 Yosemite Sam CA 95389 (209) 375-6572 52 M
3 Daffy Duck NY 10013 (212) 227-1810 35 M
4 Elmer Fudd ME 04578 (207) 882-7323 43 M
5 Witch Hazel MA 01970 (978) 744-0991 57 F
ID
1
2
3
4
5
Fname
Bugs
Yosemite
Daffy
Elmer
Witch
Lname
Bunny
Sam
Duck
Fudd
Hazel
State
NY
CA
NY
ME
MA
Zip
11217
95389
10013
04578
01970
Phone
(718) 938-3235
(209) 375-6572
(212) 227-1810
(207) 882-7323
(978) 744-0991
Age
34
52
35
43
57
Sex
M
M
M
M
F
SELECT Fname FROM People WHERE State = 'NY'
Components of
MariaDB
ColumnStore
ColumnStore Modules
● User Module (UM)
○ MariaDB Server / Storage Engine Plugin
○ ExeMgr
○ DMLProc, DDLProc
● Performance Module (PM)
○ PrimProc
○ WriteEngine
○ ProcMgr / ProcMon
○ DBRM
Query Processing
Shared Nothing Distributed Data Storage
SQL
Column
Primitives
User
Module
Performance
Module
UM
PM
Primitives ↓↓↓↓
Intermediate
↑↑Results↑↑
Hardware Requirements
● Lots of RAM
○ minimum 32GB for UM, 16GB for PM
○ minimum 4GB for trying single server out on a VM
● Optimised for HDD spindles, will still work with SSD
○ We are looking into SSD optimisation soon
● More cores typically better
○ 8 core minimum recommendation
● For AWS m4.4xlarge is the recommended minimum
Disk Storage
Column Types
• 8-byte fixed length token (pointer).
• A variable length value stored at the
location identified by the pointer.
1-byte Field
with 8192 values
per 8k block
2-byte Field
with 4096 values
per 8k block
4-byte Field
with 2048 values
per 8k block
8-byte Field
with 1024 values
per 8k block
Dictionary structure
made up of 2
files/extents with:
Extent Map
Metadata Definition
Object ID The ID for the column (or dictionary)
Object Type Column or Dictionary
LBID Start / End Start / End Logical Block Pointer
Minimum Value Lowest value in the extent
Maximum Value Highest value in the extent
Width Column Width
DBRoot DBRoot (disk partition) number
Partition ID / Segment ID / Block Offset The extent number
High Water Mark Atomic last block pointer
Disk Storage
Blocks (8KB)
Extent1
(8MB~64MB
8 million rows)
Logical
Layer
Segment File1
(maps to an Extent)
Physical
Layer
Compression
Chunks (4MB)
Writing Data
Inserting Data
● Multiple methods
○ Single INSERTs
○ INSERT...SELECT
○ LOAD DATA INFILE
○ cpimport
○ Bulk Write API
● Designed for large bulk inserts
● Inserts are appended at the end of extents (or new extents created)
○ This means reads are not affected
○ A High Water Mark pointing to the last block is moved at the end of the insert
cpimport
● Uses CSV files or piped CSV data
● Fastest way to get data into ColumnStore
● Does minimal data conversion and pipes it straight into the PMs
○ Works by appending new blocks to the table and moving an atomic block pointer
(HWM)
○ No UNDO log needed (atomic pointer not moved on rollback)
○ Therefore can cause a gap of 0-64KB in a column
● Can load multiple tables simultaneously
● Can load into multiple PMs for the same table simultaneously
● Can load into specific PMs for physical partitioning by PM
Bulk Write API
● A simple C++ API to inject data into the PMs
○ Bindings in Python and Java available
● Works in a similar way to cpimport
○ Append new blocks and an atomic block pointer (HWM)
● LGPL licensed
DML Writes
● Regular INSERT / UPDATE / DELETE
○ Also INSERT...SELECT and LOAD DATA INFILE when autocommit is off
● Slow compared to other engines
○ INSERT is very slow compared to cpimport
● Requires the use of a version buffer for an undo log
○ But INSERT appends to data blocks so no wasted storage
● Data sent to DMLProc to process
A Note About DELETE
● Need to touch every column and the undo log
○ So very slow
● Also leaves a gap in the column that won’t be filled
● Having a column that is marked using an UPDATE query is faster
● Dropping entire partitions is instantaneous
○ Partitions can be disabled first
INSERT...SELECT / LOAD DATA INFILE
● Injects the binary row data from MariaDB into cpimport
● Good for backwards compatibility with tools and remote loading
● cpimport then injects this data into the column extent files
○ In 1.2 it will use the write API instead
● If autocommit is turned off this will behave like regular DML instead (slow)
Querying Data
Physical Execution Layout
Round Robin
MariaDB
Client
MariaDB
Server
ExeMgr
ExeMgr
PrimProc
PrimProc
PrimProc
PrimProc
Extent Elimination
Horizontal
Partition:
8 Million Rows
Extent 2
Horizontal
Partition:
8 Million Rows
Extent 3
Horizontal
Partition:
8 Million Rows
Extent 1
Storage Architecture reduces I/O
• Only touch column files
that are in filter, projection, group by, and
join conditions
• Eliminate disk block touches
to partitions outside filter
and join conditions
Extent 1:
ShipDate: 2016-01-12 - 2016-03-05
Extent 2:
ShipDate: 2016-03-05 - 2016-09-23
Extent 3:
ShipDate: 2016-09-24 - 2017-01-06
SELECT Item, sum(Quantity) FROM Orders
WHERE ShipDate between ‘2016-01-01’ and ‘2016-01-31’
GROUP BY Item
Id OrderId Line Item Quantity Price Supplier ShipDate ShipMode
1 1 1 Laptop 5 1000 Dell 2016-01-12 G
2 1 2 Monitor 5 200 LG 2016-01-13 G
3 2 1 Mouse 1 20 Logitech 2016-02-05 M
4 3 1 Laptop 3 1600 Apple 2016-01-31 P
... ... ... ... ... ... ... ... ...
8M 2016-03-05
8M+1 2016-03-05
... ... ... ... ... ... ... ... ...
16M 2016-09-23
16M+1 2016-09-24
... ... ... ... ... ... ... ... ...
24M 2017-01-06
ELIMINATED PARTITION
ELIMINATED PARTITION
Query Analysis
MariaDB [tpch1]> select calsettrace(1);
...
MariaDB [tpch1]> select c_count, count(*) as custdist
-> from ( select c_custkey, count(o_orderkey) c_count
-> from v_customer left outer join v_orders on c_custkey = o_custkey
-> and o_comment not like '%special%requests%'
-> group by c_custkey ) c_orders
-> group by c_count
-> order by custdist desc, c_count desc;
...
42 rows in set, 1 warning (9.07 sec)
MariaDB [tpch1]> select calgetstats()G
*************************** 1. row ***************************
calgetstats(): Query Stats: MaxMemPct-4; NumTempFiles-0; TempFileSpace-0B; ApproxPhyI/O-0; CacheI/O-12503;
BlocksTouched-12503; PartitionBlocksEliminated-812; MsgBytesIn-102MB; MsgBytesOut-3KB; Mode-Distributed
1 row in set (0.00 sec)
Query Analysis
MariaDB [tpch1]> select calgettrace()G
*************************** 1. row ***************************
calgettrace():
Desc Mode Table TableOID ReferencedColumns PIO LIO PBE Elapsed Rows
BPS PM customer 7254 (c_custkey) 0 75 0 0.032 150000
TNS UM - - - - - - 0.045 150000
BPS PM customer 7254 (c_custkey) 0 0 75 0.000 0
TNS UM - - - - - - 0.000 0
TUS UM - - - - - - 0.303 150000
BPS PM orders 7268 (o_comment,o_custkey,o_orderkey) 0 12428 0 2.293 1500000
TNS UM - - - - - - 2.967 1500000
BPS PM orders 7268 (o_comment,o_custkey,o_orderkey) 0 0 737 0.000 0
TNS UM - - - - - - 0.000 0
TUS UM - - - - - - 3.796 1500000
HJS UM v_customer-v_orders - - - - - ----- -
TAS UM - - - - - - 1.658 150000
TNS UM - - - - - - 0.044 150000
TAS UM - - - - - - 0.050 42
1 row in set (0.01 sec)
Cross Engine Joins
● Allows non-ColumnStore tables to join with
ColumnStore
● The whole query is processed by
ColumnStore
● Cross Engine makes new MariaDB
connections to retrieve data from
non-ColumnStore tables Original
Query
Non-ColumnStore Query
(Cross Engine)
MariaDB
Server
Optimizing for
MariaDB
ColumnStore
Data Modeling
● Star-schema optimizations are generally a good idea
● Conservative data typing is very important
○ Especially around fixed-length vs. dictionary boundary (8 bytes)
○ IP Address vs. IP Number
● Break down compound fields into individual fields:
○ Trivializes searching for sub-fields
○ Can avoid dictionary overhead
○ Cost to re-assemble is generally small
Data Insertion
● Order data as best you can before inserting
○ Helps extent elimination when min/max range for an extent is small
● Insert in large batches using cpimport or bulk write API
Improving Your Queries
● Avoid filtering on a >= 8byte VARCHAR/CHAR column where possible
○ Two extents need to be read per column, no extent elimination
● Use extent map elimination where possible
● Don’t use a function to filter
○ Extent elimination won’t happen
● Only reference required columns, avoid “SELECT *”
● Use the smallest possible data type for your data
● Avoid large ORDER BY
● Read https://mariadb.com/kb/en/mariadb/columnstore-performance-tuning/
Tuning
● Generally self-tuning
○ Uses as much RAM as possible automatically
○ Uses all CPU cores
● More RAM in PMs = more LRU data cache
● More RAM in UMs = ability to process aggregates / joins on bigger data sets
○ Disk joins are possible
Order By / Limit
● Order by and limit are applied on the final vtable result by mariadb using the
worst sort engine and it's single threaded.
● Keep result sets lean if you use order by.
● A workaround for performance is if you move order by limit into an artifical
subquery then it uses the ExeMgr multi threaded sort engine which is faster
(but not distributed).
Example:
SELECT a, b FROM t1 WHERE a = 2000 ORDER BY b LIMIT 50;
Convert to:
SELECT * FROM (SELECT a, b FROM t1 WHERE a = 2000 ORDER BY b LIMIT 50;) sq;
Recommended Settings
● In general most out of the box settings are fine. The main core tunable settings
(there are more but may or may not work)
● Sometimes reduce / tweak if hitting memory limits with high memory queries:
○ NumBlocksPct (default 80 separate, 50 combined/single)
○ TotalUmMemory (default 50 separate, 25 combined/single)
● Tune down for more concurrency or tune up for larger core counts.
○ MaxOutstandingRequests (default 20)
● Increase substantially (e.g. 2048mb) to enable more joins to be pushed down
○ PmMaxMemorySmallSide (default 64mb)
Closing Notes
MariaDB ColumnStore 1.3
● MariaDB 10.4 base
● Convergence
● Ease of Use
THANK YOU!

More Related Content

What's hot

A Day in the Life of a ClickHouse Query Webinar Slides
A Day in the Life of a ClickHouse Query Webinar Slides A Day in the Life of a ClickHouse Query Webinar Slides
A Day in the Life of a ClickHouse Query Webinar Slides Altinity Ltd
 
M|18 Deep Dive: InnoDB Transactions and Write Paths
M|18 Deep Dive: InnoDB Transactions and Write PathsM|18 Deep Dive: InnoDB Transactions and Write Paths
M|18 Deep Dive: InnoDB Transactions and Write PathsMariaDB plc
 
Presto query optimizer: pursuit of performance
Presto query optimizer: pursuit of performancePresto query optimizer: pursuit of performance
Presto query optimizer: pursuit of performanceDataWorks Summit
 
MySQL Architecture and Engine
MySQL Architecture and EngineMySQL Architecture and Engine
MySQL Architecture and EngineAbdul Manaf
 
MariaDB Galera Cluster presentation
MariaDB Galera Cluster presentationMariaDB Galera Cluster presentation
MariaDB Galera Cluster presentationFrancisco Gonçalves
 
Big Data Analytics with MariaDB ColumnStore
Big Data Analytics with MariaDB ColumnStoreBig Data Analytics with MariaDB ColumnStore
Big Data Analytics with MariaDB ColumnStoreMariaDB plc
 
Navigating Transactions: ACID Complexity in Modern Databases
Navigating Transactions: ACID Complexity in Modern DatabasesNavigating Transactions: ACID Complexity in Modern Databases
Navigating Transactions: ACID Complexity in Modern DatabasesShivji Kumar Jha
 
InnoDB Internal
InnoDB InternalInnoDB Internal
InnoDB Internalmysqlops
 
MariaDB MaxScale
MariaDB MaxScaleMariaDB MaxScale
MariaDB MaxScaleMariaDB plc
 
MySQL Timeout Variables Explained
MySQL Timeout Variables Explained MySQL Timeout Variables Explained
MySQL Timeout Variables Explained Mydbops
 
[Pgday.Seoul 2017] 3. PostgreSQL WAL Buffers, Clog Buffers Deep Dive - 이근오
[Pgday.Seoul 2017] 3. PostgreSQL WAL Buffers, Clog Buffers Deep Dive - 이근오[Pgday.Seoul 2017] 3. PostgreSQL WAL Buffers, Clog Buffers Deep Dive - 이근오
[Pgday.Seoul 2017] 3. PostgreSQL WAL Buffers, Clog Buffers Deep Dive - 이근오PgDay.Seoul
 
MySQL topology healing at OLA.
MySQL topology healing at OLA.MySQL topology healing at OLA.
MySQL topology healing at OLA.Mydbops
 
The Full MySQL and MariaDB Parallel Replication Tutorial
The Full MySQL and MariaDB Parallel Replication TutorialThe Full MySQL and MariaDB Parallel Replication Tutorial
The Full MySQL and MariaDB Parallel Replication TutorialJean-François Gagné
 
Dd and atomic ddl pl17 dublin
Dd and atomic ddl pl17 dublinDd and atomic ddl pl17 dublin
Dd and atomic ddl pl17 dublinStåle Deraas
 
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdf
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdfDeep Dive on ClickHouse Sharding and Replication-2202-09-22.pdf
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdfAltinity Ltd
 
High Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouseHigh Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouseAltinity Ltd
 
Optimizing MariaDB for maximum performance
Optimizing MariaDB for maximum performanceOptimizing MariaDB for maximum performance
Optimizing MariaDB for maximum performanceMariaDB plc
 
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)Altinity Ltd
 
Batch and Stream Graph Processing with Apache Flink
Batch and Stream Graph Processing with Apache FlinkBatch and Stream Graph Processing with Apache Flink
Batch and Stream Graph Processing with Apache FlinkVasia Kalavri
 

What's hot (20)

A Day in the Life of a ClickHouse Query Webinar Slides
A Day in the Life of a ClickHouse Query Webinar Slides A Day in the Life of a ClickHouse Query Webinar Slides
A Day in the Life of a ClickHouse Query Webinar Slides
 
M|18 Deep Dive: InnoDB Transactions and Write Paths
M|18 Deep Dive: InnoDB Transactions and Write PathsM|18 Deep Dive: InnoDB Transactions and Write Paths
M|18 Deep Dive: InnoDB Transactions and Write Paths
 
Presto query optimizer: pursuit of performance
Presto query optimizer: pursuit of performancePresto query optimizer: pursuit of performance
Presto query optimizer: pursuit of performance
 
MySQL Architecture and Engine
MySQL Architecture and EngineMySQL Architecture and Engine
MySQL Architecture and Engine
 
MariaDB Galera Cluster presentation
MariaDB Galera Cluster presentationMariaDB Galera Cluster presentation
MariaDB Galera Cluster presentation
 
Big Data Analytics with MariaDB ColumnStore
Big Data Analytics with MariaDB ColumnStoreBig Data Analytics with MariaDB ColumnStore
Big Data Analytics with MariaDB ColumnStore
 
Navigating Transactions: ACID Complexity in Modern Databases
Navigating Transactions: ACID Complexity in Modern DatabasesNavigating Transactions: ACID Complexity in Modern Databases
Navigating Transactions: ACID Complexity in Modern Databases
 
InnoDB Internal
InnoDB InternalInnoDB Internal
InnoDB Internal
 
MariaDB MaxScale
MariaDB MaxScaleMariaDB MaxScale
MariaDB MaxScale
 
Log Structured Merge Tree
Log Structured Merge TreeLog Structured Merge Tree
Log Structured Merge Tree
 
MySQL Timeout Variables Explained
MySQL Timeout Variables Explained MySQL Timeout Variables Explained
MySQL Timeout Variables Explained
 
[Pgday.Seoul 2017] 3. PostgreSQL WAL Buffers, Clog Buffers Deep Dive - 이근오
[Pgday.Seoul 2017] 3. PostgreSQL WAL Buffers, Clog Buffers Deep Dive - 이근오[Pgday.Seoul 2017] 3. PostgreSQL WAL Buffers, Clog Buffers Deep Dive - 이근오
[Pgday.Seoul 2017] 3. PostgreSQL WAL Buffers, Clog Buffers Deep Dive - 이근오
 
MySQL topology healing at OLA.
MySQL topology healing at OLA.MySQL topology healing at OLA.
MySQL topology healing at OLA.
 
The Full MySQL and MariaDB Parallel Replication Tutorial
The Full MySQL and MariaDB Parallel Replication TutorialThe Full MySQL and MariaDB Parallel Replication Tutorial
The Full MySQL and MariaDB Parallel Replication Tutorial
 
Dd and atomic ddl pl17 dublin
Dd and atomic ddl pl17 dublinDd and atomic ddl pl17 dublin
Dd and atomic ddl pl17 dublin
 
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdf
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdfDeep Dive on ClickHouse Sharding and Replication-2202-09-22.pdf
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdf
 
High Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouseHigh Performance, High Reliability Data Loading on ClickHouse
High Performance, High Reliability Data Loading on ClickHouse
 
Optimizing MariaDB for maximum performance
Optimizing MariaDB for maximum performanceOptimizing MariaDB for maximum performance
Optimizing MariaDB for maximum performance
 
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
 
Batch and Stream Graph Processing with Apache Flink
Batch and Stream Graph Processing with Apache FlinkBatch and Stream Graph Processing with Apache Flink
Batch and Stream Graph Processing with Apache Flink
 

Similar to Understanding the architecture of MariaDB ColumnStore

M|18 Understanding the Architecture of MariaDB ColumnStore
M|18 Understanding the Architecture of MariaDB ColumnStoreM|18 Understanding the Architecture of MariaDB ColumnStore
M|18 Understanding the Architecture of MariaDB ColumnStoreMariaDB plc
 
What to expect from MariaDB Platform X5, part 2
What to expect from MariaDB Platform X5, part 2What to expect from MariaDB Platform X5, part 2
What to expect from MariaDB Platform X5, part 2MariaDB plc
 
Best Practices for Migrating your Data Warehouse to Amazon Redshift
Best Practices for Migrating your Data Warehouse to Amazon RedshiftBest Practices for Migrating your Data Warehouse to Amazon Redshift
Best Practices for Migrating your Data Warehouse to Amazon RedshiftAmazon Web Services
 
What’s new in MariaDB ColumnStore
What’s new in MariaDB ColumnStoreWhat’s new in MariaDB ColumnStore
What’s new in MariaDB ColumnStoreMariaDB plc
 
Best Practices for Migrating your Data Warehouse to Amazon Redshift
Best Practices for Migrating your Data Warehouse to Amazon RedshiftBest Practices for Migrating your Data Warehouse to Amazon Redshift
Best Practices for Migrating your Data Warehouse to Amazon RedshiftAmazon Web Services
 
A Brief Introduction of TiDB (Percona Live)
A Brief Introduction of TiDB (Percona Live)A Brief Introduction of TiDB (Percona Live)
A Brief Introduction of TiDB (Percona Live)PingCAP
 
Best Practices for Migrating your Data Warehouse to Amazon Redshift
Best Practices for Migrating your Data Warehouse to Amazon Redshift Best Practices for Migrating your Data Warehouse to Amazon Redshift
Best Practices for Migrating your Data Warehouse to Amazon Redshift Amazon Web Services
 
Deep Dive into DynamoDB
Deep Dive into DynamoDBDeep Dive into DynamoDB
Deep Dive into DynamoDBAWS Germany
 
TiDB vs Aurora.pdf
TiDB vs Aurora.pdfTiDB vs Aurora.pdf
TiDB vs Aurora.pdfssuser3fb50b
 
Big Data LDN 2017: Big Data Analytics with MariaDB ColumnStore
Big Data LDN 2017: Big Data Analytics with MariaDB ColumnStoreBig Data LDN 2017: Big Data Analytics with MariaDB ColumnStore
Big Data LDN 2017: Big Data Analytics with MariaDB ColumnStoreMatt Stubbs
 
Argus Production Monitoring at Salesforce
Argus Production Monitoring at SalesforceArgus Production Monitoring at Salesforce
Argus Production Monitoring at SalesforceHBaseCon
 
Argus Production Monitoring at Salesforce
Argus Production Monitoring at Salesforce Argus Production Monitoring at Salesforce
Argus Production Monitoring at Salesforce HBaseCon
 
Maryna Popova "Deep dive AWS Redshift"
Maryna Popova "Deep dive AWS Redshift"Maryna Popova "Deep dive AWS Redshift"
Maryna Popova "Deep dive AWS Redshift"Lviv Startup Club
 
Big data should be simple
Big data should be simpleBig data should be simple
Big data should be simpleDori Waldman
 
How to build TiDB
How to build TiDBHow to build TiDB
How to build TiDBPingCAP
 
[db tech showcase OSS 2017] A23: Analytics with MariaDB ColumnStore by MariaD...
[db tech showcase OSS 2017] A23: Analytics with MariaDB ColumnStore by MariaD...[db tech showcase OSS 2017] A23: Analytics with MariaDB ColumnStore by MariaD...
[db tech showcase OSS 2017] A23: Analytics with MariaDB ColumnStore by MariaD...Insight Technology, Inc.
 
[db tech showcase OSS 2017] A25: Replacing Oracle Database at DBS Bank by Mar...
[db tech showcase OSS 2017] A25: Replacing Oracle Database at DBS Bank by Mar...[db tech showcase OSS 2017] A25: Replacing Oracle Database at DBS Bank by Mar...
[db tech showcase OSS 2017] A25: Replacing Oracle Database at DBS Bank by Mar...Insight Technology, Inc.
 
Building an Amazon Datawarehouse and Using Business Intelligence Analytics Tools
Building an Amazon Datawarehouse and Using Business Intelligence Analytics ToolsBuilding an Amazon Datawarehouse and Using Business Intelligence Analytics Tools
Building an Amazon Datawarehouse and Using Business Intelligence Analytics ToolsAmazon Web Services
 
Introducing TiDB [Delivered: 09/27/18 at NYC SQL Meetup]
Introducing TiDB [Delivered: 09/27/18 at NYC SQL Meetup]Introducing TiDB [Delivered: 09/27/18 at NYC SQL Meetup]
Introducing TiDB [Delivered: 09/27/18 at NYC SQL Meetup]Kevin Xu
 
The Parquet Format and Performance Optimization Opportunities
The Parquet Format and Performance Optimization OpportunitiesThe Parquet Format and Performance Optimization Opportunities
The Parquet Format and Performance Optimization OpportunitiesDatabricks
 

Similar to Understanding the architecture of MariaDB ColumnStore (20)

M|18 Understanding the Architecture of MariaDB ColumnStore
M|18 Understanding the Architecture of MariaDB ColumnStoreM|18 Understanding the Architecture of MariaDB ColumnStore
M|18 Understanding the Architecture of MariaDB ColumnStore
 
What to expect from MariaDB Platform X5, part 2
What to expect from MariaDB Platform X5, part 2What to expect from MariaDB Platform X5, part 2
What to expect from MariaDB Platform X5, part 2
 
Best Practices for Migrating your Data Warehouse to Amazon Redshift
Best Practices for Migrating your Data Warehouse to Amazon RedshiftBest Practices for Migrating your Data Warehouse to Amazon Redshift
Best Practices for Migrating your Data Warehouse to Amazon Redshift
 
What’s new in MariaDB ColumnStore
What’s new in MariaDB ColumnStoreWhat’s new in MariaDB ColumnStore
What’s new in MariaDB ColumnStore
 
Best Practices for Migrating your Data Warehouse to Amazon Redshift
Best Practices for Migrating your Data Warehouse to Amazon RedshiftBest Practices for Migrating your Data Warehouse to Amazon Redshift
Best Practices for Migrating your Data Warehouse to Amazon Redshift
 
A Brief Introduction of TiDB (Percona Live)
A Brief Introduction of TiDB (Percona Live)A Brief Introduction of TiDB (Percona Live)
A Brief Introduction of TiDB (Percona Live)
 
Best Practices for Migrating your Data Warehouse to Amazon Redshift
Best Practices for Migrating your Data Warehouse to Amazon Redshift Best Practices for Migrating your Data Warehouse to Amazon Redshift
Best Practices for Migrating your Data Warehouse to Amazon Redshift
 
Deep Dive into DynamoDB
Deep Dive into DynamoDBDeep Dive into DynamoDB
Deep Dive into DynamoDB
 
TiDB vs Aurora.pdf
TiDB vs Aurora.pdfTiDB vs Aurora.pdf
TiDB vs Aurora.pdf
 
Big Data LDN 2017: Big Data Analytics with MariaDB ColumnStore
Big Data LDN 2017: Big Data Analytics with MariaDB ColumnStoreBig Data LDN 2017: Big Data Analytics with MariaDB ColumnStore
Big Data LDN 2017: Big Data Analytics with MariaDB ColumnStore
 
Argus Production Monitoring at Salesforce
Argus Production Monitoring at SalesforceArgus Production Monitoring at Salesforce
Argus Production Monitoring at Salesforce
 
Argus Production Monitoring at Salesforce
Argus Production Monitoring at Salesforce Argus Production Monitoring at Salesforce
Argus Production Monitoring at Salesforce
 
Maryna Popova "Deep dive AWS Redshift"
Maryna Popova "Deep dive AWS Redshift"Maryna Popova "Deep dive AWS Redshift"
Maryna Popova "Deep dive AWS Redshift"
 
Big data should be simple
Big data should be simpleBig data should be simple
Big data should be simple
 
How to build TiDB
How to build TiDBHow to build TiDB
How to build TiDB
 
[db tech showcase OSS 2017] A23: Analytics with MariaDB ColumnStore by MariaD...
[db tech showcase OSS 2017] A23: Analytics with MariaDB ColumnStore by MariaD...[db tech showcase OSS 2017] A23: Analytics with MariaDB ColumnStore by MariaD...
[db tech showcase OSS 2017] A23: Analytics with MariaDB ColumnStore by MariaD...
 
[db tech showcase OSS 2017] A25: Replacing Oracle Database at DBS Bank by Mar...
[db tech showcase OSS 2017] A25: Replacing Oracle Database at DBS Bank by Mar...[db tech showcase OSS 2017] A25: Replacing Oracle Database at DBS Bank by Mar...
[db tech showcase OSS 2017] A25: Replacing Oracle Database at DBS Bank by Mar...
 
Building an Amazon Datawarehouse and Using Business Intelligence Analytics Tools
Building an Amazon Datawarehouse and Using Business Intelligence Analytics ToolsBuilding an Amazon Datawarehouse and Using Business Intelligence Analytics Tools
Building an Amazon Datawarehouse and Using Business Intelligence Analytics Tools
 
Introducing TiDB [Delivered: 09/27/18 at NYC SQL Meetup]
Introducing TiDB [Delivered: 09/27/18 at NYC SQL Meetup]Introducing TiDB [Delivered: 09/27/18 at NYC SQL Meetup]
Introducing TiDB [Delivered: 09/27/18 at NYC SQL Meetup]
 
The Parquet Format and Performance Optimization Opportunities
The Parquet Format and Performance Optimization OpportunitiesThe Parquet Format and Performance Optimization Opportunities
The Parquet Format and Performance Optimization Opportunities
 

More from MariaDB plc

MariaDB Paris Workshop 2023 - MaxScale 23.02.x
MariaDB Paris Workshop 2023 - MaxScale 23.02.xMariaDB Paris Workshop 2023 - MaxScale 23.02.x
MariaDB Paris Workshop 2023 - MaxScale 23.02.xMariaDB plc
 
MariaDB Paris Workshop 2023 - Newpharma
MariaDB Paris Workshop 2023 - NewpharmaMariaDB Paris Workshop 2023 - Newpharma
MariaDB Paris Workshop 2023 - NewpharmaMariaDB plc
 
MariaDB Paris Workshop 2023 - Cloud
MariaDB Paris Workshop 2023 - CloudMariaDB Paris Workshop 2023 - Cloud
MariaDB Paris Workshop 2023 - CloudMariaDB plc
 
MariaDB Paris Workshop 2023 - MariaDB Enterprise
MariaDB Paris Workshop 2023 - MariaDB EnterpriseMariaDB Paris Workshop 2023 - MariaDB Enterprise
MariaDB Paris Workshop 2023 - MariaDB EnterpriseMariaDB plc
 
MariaDB Paris Workshop 2023 - Performance Optimization
MariaDB Paris Workshop 2023 - Performance OptimizationMariaDB Paris Workshop 2023 - Performance Optimization
MariaDB Paris Workshop 2023 - Performance OptimizationMariaDB plc
 
MariaDB Paris Workshop 2023 - MaxScale
MariaDB Paris Workshop 2023 - MaxScale MariaDB Paris Workshop 2023 - MaxScale
MariaDB Paris Workshop 2023 - MaxScale MariaDB plc
 
MariaDB Paris Workshop 2023 - novadys presentation
MariaDB Paris Workshop 2023 - novadys presentationMariaDB Paris Workshop 2023 - novadys presentation
MariaDB Paris Workshop 2023 - novadys presentationMariaDB plc
 
MariaDB Paris Workshop 2023 - DARVA presentation
MariaDB Paris Workshop 2023 - DARVA presentationMariaDB Paris Workshop 2023 - DARVA presentation
MariaDB Paris Workshop 2023 - DARVA presentationMariaDB plc
 
MariaDB Tech und Business Update Hamburg 2023 - MariaDB Enterprise Server
MariaDB Tech und Business Update Hamburg 2023 - MariaDB Enterprise Server MariaDB Tech und Business Update Hamburg 2023 - MariaDB Enterprise Server
MariaDB Tech und Business Update Hamburg 2023 - MariaDB Enterprise Server MariaDB plc
 
MariaDB SkySQL Autonome Skalierung, Observability, Cloud-Backup
MariaDB SkySQL Autonome Skalierung, Observability, Cloud-BackupMariaDB SkySQL Autonome Skalierung, Observability, Cloud-Backup
MariaDB SkySQL Autonome Skalierung, Observability, Cloud-BackupMariaDB plc
 
Einführung : MariaDB Tech und Business Update Hamburg 2023
Einführung : MariaDB Tech und Business Update Hamburg 2023Einführung : MariaDB Tech und Business Update Hamburg 2023
Einführung : MariaDB Tech und Business Update Hamburg 2023MariaDB plc
 
Hochverfügbarkeitslösungen mit MariaDB
Hochverfügbarkeitslösungen mit MariaDBHochverfügbarkeitslösungen mit MariaDB
Hochverfügbarkeitslösungen mit MariaDBMariaDB plc
 
Die Neuheiten in MariaDB Enterprise Server
Die Neuheiten in MariaDB Enterprise ServerDie Neuheiten in MariaDB Enterprise Server
Die Neuheiten in MariaDB Enterprise ServerMariaDB plc
 
Global Data Replication with Galera for Ansell Guardian®
Global Data Replication with Galera for Ansell Guardian®Global Data Replication with Galera for Ansell Guardian®
Global Data Replication with Galera for Ansell Guardian®MariaDB plc
 
Introducing workload analysis
Introducing workload analysisIntroducing workload analysis
Introducing workload analysisMariaDB plc
 
Under the hood: SkySQL monitoring
Under the hood: SkySQL monitoringUnder the hood: SkySQL monitoring
Under the hood: SkySQL monitoringMariaDB plc
 
Introducing the R2DBC async Java connector
Introducing the R2DBC async Java connectorIntroducing the R2DBC async Java connector
Introducing the R2DBC async Java connectorMariaDB plc
 
MariaDB Enterprise Tools introduction
MariaDB Enterprise Tools introductionMariaDB Enterprise Tools introduction
MariaDB Enterprise Tools introductionMariaDB plc
 
The architecture of SkySQL
The architecture of SkySQLThe architecture of SkySQL
The architecture of SkySQLMariaDB plc
 
What to expect from MariaDB Platform X5, part 1
What to expect from MariaDB Platform X5, part 1What to expect from MariaDB Platform X5, part 1
What to expect from MariaDB Platform X5, part 1MariaDB plc
 

More from MariaDB plc (20)

MariaDB Paris Workshop 2023 - MaxScale 23.02.x
MariaDB Paris Workshop 2023 - MaxScale 23.02.xMariaDB Paris Workshop 2023 - MaxScale 23.02.x
MariaDB Paris Workshop 2023 - MaxScale 23.02.x
 
MariaDB Paris Workshop 2023 - Newpharma
MariaDB Paris Workshop 2023 - NewpharmaMariaDB Paris Workshop 2023 - Newpharma
MariaDB Paris Workshop 2023 - Newpharma
 
MariaDB Paris Workshop 2023 - Cloud
MariaDB Paris Workshop 2023 - CloudMariaDB Paris Workshop 2023 - Cloud
MariaDB Paris Workshop 2023 - Cloud
 
MariaDB Paris Workshop 2023 - MariaDB Enterprise
MariaDB Paris Workshop 2023 - MariaDB EnterpriseMariaDB Paris Workshop 2023 - MariaDB Enterprise
MariaDB Paris Workshop 2023 - MariaDB Enterprise
 
MariaDB Paris Workshop 2023 - Performance Optimization
MariaDB Paris Workshop 2023 - Performance OptimizationMariaDB Paris Workshop 2023 - Performance Optimization
MariaDB Paris Workshop 2023 - Performance Optimization
 
MariaDB Paris Workshop 2023 - MaxScale
MariaDB Paris Workshop 2023 - MaxScale MariaDB Paris Workshop 2023 - MaxScale
MariaDB Paris Workshop 2023 - MaxScale
 
MariaDB Paris Workshop 2023 - novadys presentation
MariaDB Paris Workshop 2023 - novadys presentationMariaDB Paris Workshop 2023 - novadys presentation
MariaDB Paris Workshop 2023 - novadys presentation
 
MariaDB Paris Workshop 2023 - DARVA presentation
MariaDB Paris Workshop 2023 - DARVA presentationMariaDB Paris Workshop 2023 - DARVA presentation
MariaDB Paris Workshop 2023 - DARVA presentation
 
MariaDB Tech und Business Update Hamburg 2023 - MariaDB Enterprise Server
MariaDB Tech und Business Update Hamburg 2023 - MariaDB Enterprise Server MariaDB Tech und Business Update Hamburg 2023 - MariaDB Enterprise Server
MariaDB Tech und Business Update Hamburg 2023 - MariaDB Enterprise Server
 
MariaDB SkySQL Autonome Skalierung, Observability, Cloud-Backup
MariaDB SkySQL Autonome Skalierung, Observability, Cloud-BackupMariaDB SkySQL Autonome Skalierung, Observability, Cloud-Backup
MariaDB SkySQL Autonome Skalierung, Observability, Cloud-Backup
 
Einführung : MariaDB Tech und Business Update Hamburg 2023
Einführung : MariaDB Tech und Business Update Hamburg 2023Einführung : MariaDB Tech und Business Update Hamburg 2023
Einführung : MariaDB Tech und Business Update Hamburg 2023
 
Hochverfügbarkeitslösungen mit MariaDB
Hochverfügbarkeitslösungen mit MariaDBHochverfügbarkeitslösungen mit MariaDB
Hochverfügbarkeitslösungen mit MariaDB
 
Die Neuheiten in MariaDB Enterprise Server
Die Neuheiten in MariaDB Enterprise ServerDie Neuheiten in MariaDB Enterprise Server
Die Neuheiten in MariaDB Enterprise Server
 
Global Data Replication with Galera for Ansell Guardian®
Global Data Replication with Galera for Ansell Guardian®Global Data Replication with Galera for Ansell Guardian®
Global Data Replication with Galera for Ansell Guardian®
 
Introducing workload analysis
Introducing workload analysisIntroducing workload analysis
Introducing workload analysis
 
Under the hood: SkySQL monitoring
Under the hood: SkySQL monitoringUnder the hood: SkySQL monitoring
Under the hood: SkySQL monitoring
 
Introducing the R2DBC async Java connector
Introducing the R2DBC async Java connectorIntroducing the R2DBC async Java connector
Introducing the R2DBC async Java connector
 
MariaDB Enterprise Tools introduction
MariaDB Enterprise Tools introductionMariaDB Enterprise Tools introduction
MariaDB Enterprise Tools introduction
 
The architecture of SkySQL
The architecture of SkySQLThe architecture of SkySQL
The architecture of SkySQL
 
What to expect from MariaDB Platform X5, part 1
What to expect from MariaDB Platform X5, part 1What to expect from MariaDB Platform X5, part 1
What to expect from MariaDB Platform X5, part 1
 

Recently uploaded

[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypseTomasz Kowalczewski
 
Your Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | EvmuxYour Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | Evmuxevmux96
 
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...Neo4j
 
A Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdfA Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdfICS
 
OpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCAOpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCAShane Coughlan
 
GraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4jGraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4jNeo4j
 
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaUNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaNeo4j
 
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-CloudAlluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-CloudAlluxio, Inc.
 
Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024Henry Schreiner
 
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Lisi Hocke
 
Community is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletCommunity is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletAndrea Goulet
 
BusinessGPT - Security and Governance for Generative AI
BusinessGPT  - Security and Governance for Generative AIBusinessGPT  - Security and Governance for Generative AI
BusinessGPT - Security and Governance for Generative AIAGATSoftware
 

Recently uploaded (20)

[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
 
Your Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | EvmuxYour Ultimate Web Studio for Streaming Anywhere | Evmux
Your Ultimate Web Studio for Streaming Anywhere | Evmux
 
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
 
A Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdfA Deep Dive into Secure Product Development Frameworks.pdf
A Deep Dive into Secure Product Development Frameworks.pdf
 
Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...
Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...
Abortion Pill Prices Mthatha (@](+27832195400*)[ 🏥 Women's Abortion Clinic In...
 
OpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCAOpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCA
 
GraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4jGraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4j
 
Abortion Pill Prices Polokwane ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
Abortion Pill Prices Polokwane ](+27832195400*)[ 🏥 Women's Abortion Clinic in...Abortion Pill Prices Polokwane ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
Abortion Pill Prices Polokwane ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
 
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaUNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
 
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-CloudAlluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud
 
Abortion Clinic In Stanger ](+27832195400*)[ 🏥 Safe Abortion Pills In Stanger...
Abortion Clinic In Stanger ](+27832195400*)[ 🏥 Safe Abortion Pills In Stanger...Abortion Clinic In Stanger ](+27832195400*)[ 🏥 Safe Abortion Pills In Stanger...
Abortion Clinic In Stanger ](+27832195400*)[ 🏥 Safe Abortion Pills In Stanger...
 
Abortion Pill Prices Rustenburg [(+27832195400*)] 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Rustenburg [(+27832195400*)] 🏥 Women's Abortion Clinic i...Abortion Pill Prices Rustenburg [(+27832195400*)] 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Rustenburg [(+27832195400*)] 🏥 Women's Abortion Clinic i...
 
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
 
Abortion Pill Prices Aliwal North ](+27832195400*)[ 🏥 Women's Abortion Clinic...
Abortion Pill Prices Aliwal North ](+27832195400*)[ 🏥 Women's Abortion Clinic...Abortion Pill Prices Aliwal North ](+27832195400*)[ 🏥 Women's Abortion Clinic...
Abortion Pill Prices Aliwal North ](+27832195400*)[ 🏥 Women's Abortion Clinic...
 
Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024
 
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
 
微信号购买
微信号购买微信号购买
微信号购买
 
Community is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletCommunity is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea Goulet
 
BusinessGPT - Security and Governance for Generative AI
BusinessGPT  - Security and Governance for Generative AIBusinessGPT  - Security and Governance for Generative AI
BusinessGPT - Security and Governance for Generative AI
 
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
 

Understanding the architecture of MariaDB ColumnStore

  • 1. MARIADB ColumnStore Understanding the Architecture Andrew Hutchings Lead Software Engineer, MariaDB ColumnStore MariaDB Corporation
  • 2. Who Am I? ● Andrew Hutchings, aka “LinuxJedi” ● Lead Software Engineer for MariaDB’s ColumnStore ● Previous worked for: ○ NGINX - Senior Developer Advocate / Technical Product Manager ○ HP - Principal Software Engineer (HP Cloud / ATG) ○ SkySQL - Senior Sustaining Engineer ○ Rackspace - Senior Software Engineer ○ Sun/Oracle - MySQL Senior Support Engineer ● Co-author of MySQL 5.1 Plugin Development ● IRC/Twitter: LinuxJedi ● EMail: linuxjedi@mariadb.com
  • 3. Overview ● History of MariaDB ColumnStore ● Technical Use Case ● Components of MariaDB ColumnStore ● Disk Storage ● Writing Data ● Querying Data ● Optimizing for MariaDB ColumnStore ● Closing Notes ● Questions
  • 4. History of MariaDB ColumnStore ● March 2010 - Calpont launches InfiniDB ● September 2014 - Calpont (now itself called InfiniDB) closes down ○ MariaDB (then SkySQL) supports InfiniDB customers ● April 2016 - MariaDB announces development of MariaDB ColumnStore ● August 2016 - I joined MariaDB and jumped straight into ColumnStore ● December 2016 - MariaDB ColumnStore 1.0 GA ○ InfiniDB + MariaDB 10.1 + Many fixes and improvements ● November 2017 - MariaDB ColumnStore 1.1 GA ○ MariaDB 10.2 + APIs + Even more improvements ● December 2018 - MariaDB ColumnStore 1.2 GA ○ MariaDB 10.3 + TIME, microseconds, UDAFs + Lots more
  • 6. Technical Use Case ● Very large data sets ○ Many columns ○ Many millions of rows ● Complex joins and aggregates ● Rapid bulk data insertion ○ The larger the batch the better MariaDB ColumnStore ● Smaller data sets ● Basic queries ● Lots of DML queries ● Complex data types Traditional OLTP Engines
  • 7. Data Types ● INT types - range is 2 less from max unsigned or min signed ● CHAR† - max 255 bytes ● VARCHAR† - max 8000 bytes ● DECIMAL - max 18 digits ● DOUBLE/FLOAT ● DATETIME - with sub-seconds in 1.2 ● DATE ● TIME ● BLOB/TEXT† † Empty string is the same as NULL
  • 8. Other DDL Differences ● No indexes ○ Columns are somewhat self-indexing ● Auto increment is handled differently (a table comment) ● No constraints ● PARTITION syntax not supported ○ Columns are partitioned automatically
  • 9. Row-oriented vs. Column-oriented Format ID Fname Lname State Zip Phone Age Sex 1 Bugs Bunny NY 11217 (718) 938-3235 34 M 2 Yosemite Sam CA 95389 (209) 375-6572 52 M 3 Daffy Duck NY 10013 (212) 227-1810 35 M 4 Elmer Fudd ME 04578 (207) 882-7323 43 M 5 Witch Hazel MA 01970 (978) 744-0991 57 F ID 1 2 3 4 5 Fname Bugs Yosemite Daffy Elmer Witch Lname Bunny Sam Duck Fudd Hazel State NY CA NY ME MA Zip 11217 95389 10013 04578 01970 Phone (718) 938-3235 (209) 375-6572 (212) 227-1810 (207) 882-7323 (978) 744-0991 Age 34 52 35 43 57 Sex M M M M F SELECT Fname FROM People WHERE State = 'NY'
  • 11. ColumnStore Modules ● User Module (UM) ○ MariaDB Server / Storage Engine Plugin ○ ExeMgr ○ DMLProc, DDLProc ● Performance Module (PM) ○ PrimProc ○ WriteEngine ○ ProcMgr / ProcMon ○ DBRM
  • 12. Query Processing Shared Nothing Distributed Data Storage SQL Column Primitives User Module Performance Module UM PM Primitives ↓↓↓↓ Intermediate ↑↑Results↑↑
  • 13. Hardware Requirements ● Lots of RAM ○ minimum 32GB for UM, 16GB for PM ○ minimum 4GB for trying single server out on a VM ● Optimised for HDD spindles, will still work with SSD ○ We are looking into SSD optimisation soon ● More cores typically better ○ 8 core minimum recommendation ● For AWS m4.4xlarge is the recommended minimum
  • 15. Column Types • 8-byte fixed length token (pointer). • A variable length value stored at the location identified by the pointer. 1-byte Field with 8192 values per 8k block 2-byte Field with 4096 values per 8k block 4-byte Field with 2048 values per 8k block 8-byte Field with 1024 values per 8k block Dictionary structure made up of 2 files/extents with:
  • 16. Extent Map Metadata Definition Object ID The ID for the column (or dictionary) Object Type Column or Dictionary LBID Start / End Start / End Logical Block Pointer Minimum Value Lowest value in the extent Maximum Value Highest value in the extent Width Column Width DBRoot DBRoot (disk partition) number Partition ID / Segment ID / Block Offset The extent number High Water Mark Atomic last block pointer
  • 17. Disk Storage Blocks (8KB) Extent1 (8MB~64MB 8 million rows) Logical Layer Segment File1 (maps to an Extent) Physical Layer Compression Chunks (4MB)
  • 19. Inserting Data ● Multiple methods ○ Single INSERTs ○ INSERT...SELECT ○ LOAD DATA INFILE ○ cpimport ○ Bulk Write API ● Designed for large bulk inserts ● Inserts are appended at the end of extents (or new extents created) ○ This means reads are not affected ○ A High Water Mark pointing to the last block is moved at the end of the insert
  • 20. cpimport ● Uses CSV files or piped CSV data ● Fastest way to get data into ColumnStore ● Does minimal data conversion and pipes it straight into the PMs ○ Works by appending new blocks to the table and moving an atomic block pointer (HWM) ○ No UNDO log needed (atomic pointer not moved on rollback) ○ Therefore can cause a gap of 0-64KB in a column ● Can load multiple tables simultaneously ● Can load into multiple PMs for the same table simultaneously ● Can load into specific PMs for physical partitioning by PM
  • 21. Bulk Write API ● A simple C++ API to inject data into the PMs ○ Bindings in Python and Java available ● Works in a similar way to cpimport ○ Append new blocks and an atomic block pointer (HWM) ● LGPL licensed
  • 22. DML Writes ● Regular INSERT / UPDATE / DELETE ○ Also INSERT...SELECT and LOAD DATA INFILE when autocommit is off ● Slow compared to other engines ○ INSERT is very slow compared to cpimport ● Requires the use of a version buffer for an undo log ○ But INSERT appends to data blocks so no wasted storage ● Data sent to DMLProc to process
  • 23. A Note About DELETE ● Need to touch every column and the undo log ○ So very slow ● Also leaves a gap in the column that won’t be filled ● Having a column that is marked using an UPDATE query is faster ● Dropping entire partitions is instantaneous ○ Partitions can be disabled first
  • 24. INSERT...SELECT / LOAD DATA INFILE ● Injects the binary row data from MariaDB into cpimport ● Good for backwards compatibility with tools and remote loading ● cpimport then injects this data into the column extent files ○ In 1.2 it will use the write API instead ● If autocommit is turned off this will behave like regular DML instead (slow)
  • 26. Physical Execution Layout Round Robin MariaDB Client MariaDB Server ExeMgr ExeMgr PrimProc PrimProc PrimProc PrimProc
  • 27. Extent Elimination Horizontal Partition: 8 Million Rows Extent 2 Horizontal Partition: 8 Million Rows Extent 3 Horizontal Partition: 8 Million Rows Extent 1 Storage Architecture reduces I/O • Only touch column files that are in filter, projection, group by, and join conditions • Eliminate disk block touches to partitions outside filter and join conditions Extent 1: ShipDate: 2016-01-12 - 2016-03-05 Extent 2: ShipDate: 2016-03-05 - 2016-09-23 Extent 3: ShipDate: 2016-09-24 - 2017-01-06 SELECT Item, sum(Quantity) FROM Orders WHERE ShipDate between ‘2016-01-01’ and ‘2016-01-31’ GROUP BY Item Id OrderId Line Item Quantity Price Supplier ShipDate ShipMode 1 1 1 Laptop 5 1000 Dell 2016-01-12 G 2 1 2 Monitor 5 200 LG 2016-01-13 G 3 2 1 Mouse 1 20 Logitech 2016-02-05 M 4 3 1 Laptop 3 1600 Apple 2016-01-31 P ... ... ... ... ... ... ... ... ... 8M 2016-03-05 8M+1 2016-03-05 ... ... ... ... ... ... ... ... ... 16M 2016-09-23 16M+1 2016-09-24 ... ... ... ... ... ... ... ... ... 24M 2017-01-06 ELIMINATED PARTITION ELIMINATED PARTITION
  • 28. Query Analysis MariaDB [tpch1]> select calsettrace(1); ... MariaDB [tpch1]> select c_count, count(*) as custdist -> from ( select c_custkey, count(o_orderkey) c_count -> from v_customer left outer join v_orders on c_custkey = o_custkey -> and o_comment not like '%special%requests%' -> group by c_custkey ) c_orders -> group by c_count -> order by custdist desc, c_count desc; ... 42 rows in set, 1 warning (9.07 sec) MariaDB [tpch1]> select calgetstats()G *************************** 1. row *************************** calgetstats(): Query Stats: MaxMemPct-4; NumTempFiles-0; TempFileSpace-0B; ApproxPhyI/O-0; CacheI/O-12503; BlocksTouched-12503; PartitionBlocksEliminated-812; MsgBytesIn-102MB; MsgBytesOut-3KB; Mode-Distributed 1 row in set (0.00 sec)
  • 29. Query Analysis MariaDB [tpch1]> select calgettrace()G *************************** 1. row *************************** calgettrace(): Desc Mode Table TableOID ReferencedColumns PIO LIO PBE Elapsed Rows BPS PM customer 7254 (c_custkey) 0 75 0 0.032 150000 TNS UM - - - - - - 0.045 150000 BPS PM customer 7254 (c_custkey) 0 0 75 0.000 0 TNS UM - - - - - - 0.000 0 TUS UM - - - - - - 0.303 150000 BPS PM orders 7268 (o_comment,o_custkey,o_orderkey) 0 12428 0 2.293 1500000 TNS UM - - - - - - 2.967 1500000 BPS PM orders 7268 (o_comment,o_custkey,o_orderkey) 0 0 737 0.000 0 TNS UM - - - - - - 0.000 0 TUS UM - - - - - - 3.796 1500000 HJS UM v_customer-v_orders - - - - - ----- - TAS UM - - - - - - 1.658 150000 TNS UM - - - - - - 0.044 150000 TAS UM - - - - - - 0.050 42 1 row in set (0.01 sec)
  • 30. Cross Engine Joins ● Allows non-ColumnStore tables to join with ColumnStore ● The whole query is processed by ColumnStore ● Cross Engine makes new MariaDB connections to retrieve data from non-ColumnStore tables Original Query Non-ColumnStore Query (Cross Engine) MariaDB Server
  • 32. Data Modeling ● Star-schema optimizations are generally a good idea ● Conservative data typing is very important ○ Especially around fixed-length vs. dictionary boundary (8 bytes) ○ IP Address vs. IP Number ● Break down compound fields into individual fields: ○ Trivializes searching for sub-fields ○ Can avoid dictionary overhead ○ Cost to re-assemble is generally small
  • 33. Data Insertion ● Order data as best you can before inserting ○ Helps extent elimination when min/max range for an extent is small ● Insert in large batches using cpimport or bulk write API
  • 34. Improving Your Queries ● Avoid filtering on a >= 8byte VARCHAR/CHAR column where possible ○ Two extents need to be read per column, no extent elimination ● Use extent map elimination where possible ● Don’t use a function to filter ○ Extent elimination won’t happen ● Only reference required columns, avoid “SELECT *” ● Use the smallest possible data type for your data ● Avoid large ORDER BY ● Read https://mariadb.com/kb/en/mariadb/columnstore-performance-tuning/
  • 35. Tuning ● Generally self-tuning ○ Uses as much RAM as possible automatically ○ Uses all CPU cores ● More RAM in PMs = more LRU data cache ● More RAM in UMs = ability to process aggregates / joins on bigger data sets ○ Disk joins are possible
  • 36. Order By / Limit ● Order by and limit are applied on the final vtable result by mariadb using the worst sort engine and it's single threaded. ● Keep result sets lean if you use order by. ● A workaround for performance is if you move order by limit into an artifical subquery then it uses the ExeMgr multi threaded sort engine which is faster (but not distributed). Example: SELECT a, b FROM t1 WHERE a = 2000 ORDER BY b LIMIT 50; Convert to: SELECT * FROM (SELECT a, b FROM t1 WHERE a = 2000 ORDER BY b LIMIT 50;) sq;
  • 37. Recommended Settings ● In general most out of the box settings are fine. The main core tunable settings (there are more but may or may not work) ● Sometimes reduce / tweak if hitting memory limits with high memory queries: ○ NumBlocksPct (default 80 separate, 50 combined/single) ○ TotalUmMemory (default 50 separate, 25 combined/single) ● Tune down for more concurrency or tune up for larger core counts. ○ MaxOutstandingRequests (default 20) ● Increase substantially (e.g. 2048mb) to enable more joins to be pushed down ○ PmMaxMemorySmallSide (default 64mb)
  • 39. MariaDB ColumnStore 1.3 ● MariaDB 10.4 base ● Convergence ● Ease of Use