MariaDB ColumnStore is a high performance columnar storage engine that provides fast and efficient analytics on large datasets in distributed environments. It stores data column-by-column for high compression and read performance. Queries are processed in parallel across nodes for scalability. MariaDB ColumnStore is used for real-time analytics use cases in industries like healthcare, life sciences, and telecommunications to gain insights from large datasets for applications like customer behavior analysis, genome research, and call data monitoring.

1. Analytics with
MariaDB ColumnStore
The Whys, Whats and
Hows

2. Agenda
• The Task - Analytics – Why and what
• The Requirements – What do we need for analytics
• The Solution – Column Based Storage
• The Product – MariaDB AX and MariaDB ColumnStore
• The Uses – MariaDB ColumnStore in action

3. Why Analytics and what do you get
A high level view on analytics

4. Why Analytics ?
• Get the most value of your data asset
• Faster Better decision making process
• Cost reduction
• New products and services

5. What is likely
to happen?
Why is it
happening?
Types of analytics
What is
happening?
What should I
do about it?

6. Descriptive: What happened ?
● Reports
○ Sales Report
○ Expense summary
● Ad-hoc requests to analyst

7. Diagnostics: Why did it happen
• Aggregates: aggregate measure over one or
more dimension
– Find total sales
– Top five product ranked by sales
• Roll-ups: Aggregate at different levels of
dimension hierarchy
– given total sales by city, roll-up to get sales by
state
• Drill-down: Inverse of roll-ups
– given total sales by state, drill-down to get
total by city
• Slicing and Dicing:
– Equality and range selections on one or more
dimensions

8. Predictive: What is likely to happen
• Sales Prediction
– Analyze data to identify trends, spot
weakness or determine conditions
among broader data sets for making
decisions about the future
• Targeted marketing
– what is likelihood of a customer buying
a particular product based on past
buying behavior

9. Real World Example - Visualization

10. Prescriptive: What is the best course of action?
Paradox of choices
With too many choices, which one is the best?

11. Big Data Analytics Use Cases
By industry
Finance
Identify trade patterns
Detect fraud and anomolies
Predict trading outcomes
Manufacturing
Simulations to improve design/yield
Detect production anomolies
Predict machine failures (sensor data)
Telecom
Behavioral analysis of customer calls
Network analysis (perf and reliability)
Healthcare
Find genetic profiles/matches
Analyze health vs spending
Predict viral oubreaks

12. Analytics Database requirements
Why this is different from OLTP
and why indexes are not helpful

13. What is an OLTP workload?
• OLTP applications are represents the most common database workload
• OLTP applications has a read / write ratio of maybe 50/50
– Web apps / E-commerce has more reads, ending with maybe 90/10
• OLTP applications deals with data on a row by row level
– Customer data, product data, order items etc.
– Single rows are selected, inserted, updated and deleted, one by one or in small groups
• OLTP data structures is somewhat of a representation of the business or the
applications that manage the data
– An order reference a customer, and order item is linked to an order
– Typically 3rd normal form or higher
– Sometimes individual aspects break the normal form, for performance reasons
• Transactions and ACID properties are required

14. The analytics workload
• Deals with data from a high level perspective
• Handles data in large groups of rows
– SELECTs data by date, customer location, product id etc.
– Data is loaded in batch or streamed in
– Data is mostly just INSERTed
• Dealing with individual data items is usually ineffective
• Data structures are optimized for analytics use and performance
• Data is sometimes purged, but just as often not
• Contains structured, semi-structured and sometimes unstructured data
• Data often comes from many different sources, internal and external
• Queries are ad-hoc, largely
• Transactions and ACID requirements are relaxed

15. Analytics database requirements
• Fast access to large amounts of data
• Scalable as data grows over time
– Analytics requirements increasing
– Regulatory requirements
– New data sources are added
• Load performance must be fast, scalable and predictable
• Data loading should be very flexible due to the different sources of data
– Some data loaded in batch, other is streamed
• Query performance also need to be scalable
• Data compression is a requirement
– Data size constraints, as well as read performance from disk

16. B-tree indexes
The good
B-tree indexes
The bad
• Well known technology
• Works with most types of data
• Scales reasonably well
• Really good for OLTP
transactional data
• Really bad for unbalanced data
• Index modifications can be really
slow
• Index modifications are largely single
threaded
• Slows down with the amount of data
• Really not scalable with large
amount of data

17. In summary, what do we need
• Something that can compress data A LOT
• Something that can be written to with fast and predictable performance
• Something that doesn't necessarily support transactions
– It doesn't hurt, but performance is so much more important
• Something that can support analytics queries
– Ad-hoc queries
– Aggregate queries
• Something that can scale as data grows
• Something that can still have a level of high availability
• Something that works with analytics tools, like Tableau, R etc.

18. The Solution
Distributed Column based storage

19. Existing Approaches
Limited real time analytics
Slow releases of product innovation
Expensive hardware and software
Data Warehouses
Hadoop / NoSQL
LIMITED SQL
SUPPORT
DIFFICULT TO
INSTALL/MANAGE
LIMITED TALENT POOL
DATA LAKE W/ NO DATA
MANAGEMENT
Hard to use

20. To the rescue – Column Based Storage
• Data is stored column by column
• Each column is stored in one or more extents
– Each extent is represented by 1 file
• Each extent is arranged in fixed size blocks
• Extents are compressed (using Snappy)
• Data is one of
– Fixed size (1, 2, 4 or 8 bytes)
– Dictionary based with a fixed size pointer
• Meta data is in an extent map
– Extent map is in memory
– Extent map contains meta data on each
extent, like min and max values
Table
Column1 Column N
Extent 1
(8MB～64MB
8 million rows)
Extent N
(8MB～64MB
8 million rows)

21. To the rescue – Distributed data processing
• Clients connect to a User Module
• The User Module optimizes and
controls the execution
• Data is distributed among the
Performance Modules
• Data is stored, processed and
managed by Performance Modules
• Performance Modules process
query primitives in parallel
• The User Module combines the
results from the Performance
Modules
User Modules
Performance
Module 1 ... Performance
Module N
Performance
Module 2
Performance
Module 3
Clients
User Connections

22. MariaDB Analytics
MariaDB ColumnStore and MariaDB AX

23. MariaDB ColumnStore
High performance columnar storage engine that supports a wide variety
of analytical use cases in highly scalable distributed environments
Parallel query
processing for distributed
environments
Faster, More
Efficient Queries
Single Interface for
OLTP and analytics
Easy to Manage and
Scale
Easier Enterprise
Analytics
Power of SQL and
Freedom of Open
Source to Big Data
Analytics
Better Price
Performance

24. MariaDB AX
MariaDB Server
MariaDB MaxScale
MariaDB ColumnStore
Parallel queries
Distributed storage
No indexes
Automatic partitioning
Read optimized
High compression
Low disk IO
ColumnStore
Storage
ColumnStore
Storage
ColumnStore
Storage
MariaDB Server
ColumnStore
MariaDB Server
ColumnStore
MariaDB MaxScale
MariaDB Server
ColumnStore
ColumnStore
Storage
MariaDB MaxScale

25. Easier Enterprise
Analytics
ANSI SQL
Single SQL Front-end
• Use a single SQL interface for analytics and OLTP
• Leverage MariaDB Security features - Encryption for
data in motion , role based access and auditability
Full ANSI SQL
• No more SQL “like” query
• Support complex join, aggregation and window
function
Easy to manage and scale
• Eliminate needs for indexes and views
• Automated horizontal/vertical partitioning
• Linear scalable by adding new nodes as data grows
• Out of box connection with BI tools

26. Faster, More
Efficient Queries
Optimized for Columnar storage
• Columnar storage reduces disk I/O
• Blazing fast read-intensive workload
• Ultra fast data import
Parallel distributed query execution
• Distributed queries into series of parallel operations
• Fully parallel high speed data ingestion
Highly available analytic environment
• Built-in Redundancy
• Automatic fail-over
Parallel
Query Processing

27. MariaDB ColumnStore
Analytics Use Cases

28. Healthcare / Life Science Industry
Genome analysis
• In-depth genome research for the dairy industry to improve production of milk and protein.
• Fast data load for large amount of genome dataset (DNA data for 7billion cows in US - 20GB per load)
Healthcare spending analysis
• Analyze 3TB of US health care spending for 155 conditions with 7 years of historical data
• Used sankey diagram, treemap, and pyramid chart to analyze trends by age, sex, type of care, and condition
Why MariaDB ColumnStore
• Strong security features including role based data access and audit plug in
• MPP architecture handles analytics on big data with high speed
• Easy to analyze archived data with SQL based analytics
• Does not require DBA to index or partition data

29. Telecommunication Industry
Customer behavior analysis
• Analyze call data record to segment customers based on their behavior
• Data-driven analysis for customer satisfaction
• Create behavioral based upsell or cross-sell opportunity
Call data analysis
• Data size: 6TB
• Ingest 1.5 million rows of logs per day with 30million texts and 3million calls
• Call and network quality analysis
• Provide higher quality customer services based on data
Why MariaDB ColumnStore
• ColumnStore support time based partitioning and time-series analysis
• Fast data load for real-time analytics
• MPP architecture handles analytics on big data with high speed
• Easy to analyze the archived data with SQL based analytics

30. In Conclusion
• Analytics require a different technology to be able to cope with
– Different types of data
– Different types of data access
• OLTP databases has different requirements compared to Analytics
• Column Based storage allows high compression
• Metadata can replace indexing
• Distributed processing allows for performance and scalability
• MariaDB ColumnStore implement a fast an efficient distributed database for
analytics
• MariaDB AX is the subscription for professional use of MariaDB ColumnStore
• MariaDB ColumnStore is gaining wide acceptance

31. Thank you