SkySQL
MariaDB
CONNECT Storage Engine
Serge Frezefond
http://serge.frezefond.com
@sfrezefond

SkySQL Ab 2012 Confidential
Goal of the CONNECT Storage Engine :
BI on various file formats
Most of the data in companies is in various external
datas...
Behind the scene
Traditional BI
Data is processed by an ETL
– Change in the data model(denormalization...)

Agregates are ...
MariaDB CONNECT Storage Engine : created
by Olivier Bertrand
IBM database researcher
– Independant, 50 years expertise pro...
How did the CONNECT Storage Engine move to
MariaDB?
Olivier met Monty creator of MySQL and MariaDB a few years
ago (2004 f...
The CONNECT Storage Engine
Uses the MySQL Plugin Architecture
•
•
•

Plugin Architectureis a major differentiator of MySQL...
The CONNECT Storage Engine
implements advanced features
Support of external data sources :


–

Support multi files table...
The CONNECT Storage Engine
implements advanced features


Add indexing to files
– index optimized for read



Condition ...
The CONNECT Storage Engine
implements advanced features
Catalog table :
– For Example Describe for odbc table
– No need to...
XML Table Type
<?xml version="1.0" encoding="ISO-8859-1"?>
<BIBLIO SUBJECT="XML">
<BOOK ISBN="9782212090819" LANG="fr" SUB...
XML Table Type
create table xsampall (
isbn char(15) field_format='@ISBN',
language char(2) field_format='@LANG',
authorln...
XMLTable Type
query result
select isbn, subject, title, publisher from xsamp2;

ISBN

SUBJEC

TTITLE

PUBLISHER

978221209...
XCOL Table Type
Name
Sophie
Valentine

childlist
Manon, Alice, Antoine
Arthur, Sidonie, Prune

CREATE TABLE xchild (
mothe...
XCOL Table Type
select * from xchild;
mother
child
Sophie
Manon
Sophie
Alice
…
select count(child) from xchild;

October 2...
OCCUR Table Type
Name
John
Bill
Mary
…

dog cat rabbit bird fish
2
0 0
0 0
0
1 0
0 0
1
1 0
0 0

create table xpet ( name v...
OCCUR Table Type
select * from xpet;
Name
John
Mary
Mary
Lisbeth
Kevin
Kevin

October 2012

race number
dog 2
dog 1
cat
1
...
PIVOT Table Type

Who Week What Amount
Joe
3
Beer 18.00
Beth 4
Food 17.00
Janet 5
Beer 14.00
Joe
3
Food 12.00
…
create tab...
PIVOT Table Type

select * from pivex;
Who
Beth
Beth
Beth
Janet
…

October 2012

Week
3
4
5
3

Beer
16.00
15.00
20.00
18.0...
Connect Storage Engine
VEC table / Column store
col1

col1

-1 or per column file
- Indexes work
- Fixed size record

col3...
CONNECT Storage Engine
ODBC table type
Allow to access to any datasource accessible through
ODBC.
–
–
–
–
–
–

Excel
Acces...
ODBC table type
Access db example
create table customers engine=connect table_type=ODBC
block_size=10 tabname='Customers'
...
ODBC database access
From a linux box
•

UnixODBC must be used as an ODBC Driver manager.

•

The ODBC driver of the targe...
ODBC access database
can pass any command to ODBC target
create table crlite ( command varchar(128) not null,
number int(5...
ODBC access database
can pass any command to ODBC target
select * from crlite where command =
'update lite set birth = ''2...
Connect Storage Engine
MYSQL table type (a proxy table)
- same syntax as federatedx :

create Table lineitem1
ENGINE=CONNE...
MYSQL table Type
agregation on the remote server
create Table lineitem1
ENGINE=CONNECT TABLE_TYPE=MYSQL
SRCDEF='select l_s...
CONNECT Storage Engine
MYSQL table type vs. Federated(X)
•
•
•
•

support the limit clause
Implements condition push down
...
Connect Storage Engine
TBL table type (// Merge)
col1
col1

col2

col2

col3
ODBC table

col1

col2 col3

MySQL table

Mut...
Table List Table (// Merge)
works with a distributed configuration
Node 1
col1 col2

Node 0

ODBC table

MySQL table
Node ...
Parallel execution on distributed sharded
tables
Node 1
col1 col2

Node 0

ODBC table

Node 2
col1 col2 col3

TBL

MYSQL /...
Connect Storage Engine vs.
MySQL Merge tables
Table list table :
- support non MyISAM tables
- no need to the exact same s...
Importing /exporting MySQL data
in various formats
Importing file data into MySQL tables
– Here for example from an XML fi...
Ideas / Roadmap








ODBC type improvement
MySQL table type improvement
Batch key access
Adaptative query ( // M...
Where is the MariaDB Connect Storage Engine
available ?







100 % open source on launchpad
Binary packages on Mar...
How you can help
•
•
•
•

Adopt it / Test it.
Bugs : report bugs / propose fixes
Documentation : help improve it
Sharing :...
Conclusion


The MariaDB Connect Storage Engine :





Open MariaDB to BI and data analysis
Brings real value to Mari...
Thank You
Documentation:
https://mariadb.com/kb/en/connect/
Serge Frezefond
@sfrezefond
http://serge.frezefond.com

SkySQL...
Upcoming SlideShare
Loading in …5
×

MariaDB CONNECT Storage Engine

3,458 views

Published on

This webinar is a technical overview of the MariaDB CONNECT Storage Engine
The MariaDB CONNECT Storage Engine allows to access various file formats (CSV, XML, Excel, etc). It give access to any ODBC data sources (Oracle, DB2, SQLServer, etc). It also allows to access remote MySQL tables or ODBC tables.

A CONNECT table itself can be a set of remote MySQL tables. This opens the door to interesting distributed architectures that can help to address big data.

SQL requests can be executed in parallel against these CONNECT distributed tables

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,458
On SlideShare
0
From Embeds
0
Number of Embeds
293
Actions
Shares
0
Downloads
19
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • If column given . Eliminate, reorder type conversion
  • MariaDB CONNECT Storage Engine

    1. 1. SkySQL MariaDB CONNECT Storage Engine Serge Frezefond http://serge.frezefond.com @sfrezefond SkySQL Ab 2012 Confidential
    2. 2. Goal of the CONNECT Storage Engine : BI on various file formats Most of the data in companies is in various external datasources (many in non relational database format) : – – – – – – – Dbase, Firebird, SQlite DOS,FIX,BIN,CSV XML stored per column... Microsoft Access & Excel Distributed mysql servers Non MySQL relational databases: Oracle, SQL Server… Targeting BI data access on these formats. Not targeted for OLTP SkySQL Ab 2012 Confidential
    3. 3. Behind the scene Traditional BI Data is processed by an ETL – Change in the data model(denormalization...) Agregates are computed – Need to be defined and maintained Might need to move data out of RDBMS to other kind of datastore – OLAP, Collumn store, Hadoop/Hbase ... Specific tools are used to query the data IT is involved to maintain this machinery SkySQL Ab 2012 Confidential
    4. 4. MariaDB CONNECT Storage Engine : created by Olivier Bertrand IBM database researcher – Independant, 50 years expertise programing Very experienced on databases – Worked on system-R, DB2, natural language query ... Discovered MySQL when looking for friendly place to test new concepts.(2004) – Decided to go open source – Started to appreciate the MariaDB openess and friendlyness SkySQL Ab 2012 Confidential
    5. 5. How did the CONNECT Storage Engine move to MariaDB? Olivier met Monty creator of MySQL and MariaDB a few years ago (2004 for other concepts) MariaDB team helped Olivier to work with them: – First access to launchpad, go to linux, Olivier start working with MariaDB team : – Testing, bug fixes, security, test cases, doc ... MariaDB and Olivier agreed that is was ready to be released and supported under GPL – MariaDB flexibility ease integration SkySQL Ab 2012 Confidential
    6. 6. The CONNECT Storage Engine Uses the MySQL Plugin Architecture • • • Plugin Architectureis a major differentiator of MySQL Datastores can interact with the MySQL sql layer Allow advanced interaction – Specific Create Table parameters(MariaDB) – Auto-discovery of table structure (MariaDB) – Condition push down • Allow join with other storage engines – InnoDB / MyISAM tables SkySQL Ab 2012 Confidential
    7. 7. The CONNECT Storage Engine implements advanced features Support of external data sources :  – Support multi files tables Support Big File Table > 2G Support virtual tables (DIR) Add autocreate of tables :     –   Odbc, MySQL, WMI, INI ... The structure is discovered from the data source Use MariaDB create table new parameters capability (avoid comments polution) Support compressed tables SkySQL Ab 2012 Confidential
    8. 8. The CONNECT Storage Engine implements advanced features  Add indexing to files – index optimized for read  Condition Push down – Used with ODBC and MySQL to push condition to the target database. Big perf gain.   Support MariaDB virtuals columns Support of special columns : – Rowid, fileid, tabid, servid  Muti tables table (like merge) – Different structure, not myisam only, remotely distributed tables SkySQL Ab 2012 Confidential
    9. 9. The CONNECT Storage Engine implements advanced features Catalog table : – For Example Describe for odbc table – No need to do create table – Access to data / column metadata Memory file maping – For file type table (not xml) Table format .ini Multiple CONNECT tables can be created on the same underlying file – Indexes can be shared between tables SkySQL Ab 2012 Confidential
    10. 10. XML Table Type <?xml version="1.0" encoding="ISO-8859-1"?> <BIBLIO SUBJECT="XML"> <BOOK ISBN="9782212090819" LANG="fr" SUBJECT="applications"> <AUTHOR> <FIRSTNAME>Jean-Christophe</FIRSTNAME> <LASTNAME>Bernadac</LASTNAME> </AUTHOR> <TITLE>Construire une application XML</TITLE> <PUBLISHER> <NAME>Eyrolles</NAME> <PLACE>Paris</PLACE> </PUBLISHER> <DATEPUB>1999</DATEPUB> </BOOK> </BIBLIO> October 2012 SkySQL Ab 2012 Confidential
    11. 11. XML Table Type create table xsampall ( isbn char(15) field_format='@ISBN', language char(2) field_format='@LANG', authorln char(20) field_format='AUTHOR/LASTNAME', title char(32) field_format='TITLE', translated char(32) field_format='TRANSLATOR/@PREFIX', tranln char(20) field_format='TRANSLATOR/LASTNAME', publisher char(20) field_format='PUBLISHER/NAME', year int(4) field_format='DATEPUB') engine=CONNECT table_type=XML file_name='Xsample.xml' tabname='BIBLIO' option_list='rownode=BOOK,skipnull=1'; October 2012 SkySQL Ab 2012 Confidential
    12. 12. XMLTable Type query result select isbn, subject, title, publisher from xsamp2; ISBN SUBJEC TTITLE PUBLISHER 9782212090819 applications Construire une application XML Eyrolles Paris 9782840825685 applications XML en Action Microsoft Press Can also generate HTML October 2012 SkySQL Ab 2012 Confidential
    13. 13. XCOL Table Type Name Sophie Valentine childlist Manon, Alice, Antoine Arthur, Sidonie, Prune CREATE TABLE xchild ( mother char(12) NOT NULL flag=1, child varchar(30) DEFAULT NULL flag=2 ) ENGINE=CONNECT table_type=XCOL tabname='children' option_list='colname=child'; October 2012 SkySQL Ab 2012 Confidential
    14. 14. XCOL Table Type select * from xchild; mother child Sophie Manon Sophie Alice … select count(child) from xchild; October 2012 SkySQL Ab 2012 Confidential returns 10
    15. 15. OCCUR Table Type Name John Bill Mary … dog cat rabbit bird fish 2 0 0 0 0 0 1 0 0 0 1 1 0 0 0 create table xpet ( name varchar(12) not null, race char(6) not null, number int not null) engine=connect table_type=occur tabname=pets option_list='OccurCol=number,RankCol=race' Colist='dog,cat,rabbit,bird,fish'; October 2012 SkySQL Ab 2012 Confidential
    16. 16. OCCUR Table Type select * from xpet; Name John Mary Mary Lisbeth Kevin Kevin October 2012 race number dog 2 dog 1 cat 1 rabbit 2 cat 2 bird 6 SkySQL Ab 2012 Confidential
    17. 17. PIVOT Table Type Who Week What Amount Joe 3 Beer 18.00 Beth 4 Food 17.00 Janet 5 Beer 14.00 Joe 3 Food 12.00 … create table pivex Engine=connect table_type=pivot tabname=expenses; October 2012 SkySQL Ab 2012 Confidential
    18. 18. PIVOT Table Type select * from pivex; Who Beth Beth Beth Janet … October 2012 Week 3 4 5 3 Beer 16.00 15.00 20.00 18.00 Car 0.00 0.00 0.00 19.00 Food 0.00 17.00 12.00 18.00 SkySQL Ab 2012 Confidential
    19. 19. Connect Storage Engine VEC table / Column store col1 col1 -1 or per column file - Indexes work - Fixed size record col3 row1 row2 col1 col2 free col1 col3 free free col3 col2 free row3 free SkySQL Ab 2012 Confidential free
    20. 20. CONNECT Storage Engine ODBC table type Allow to access to any datasource accessible through ODBC. – – – – – – Excel Access Firebird SQLite SQL Server, Oracle, DB2 ... Possibility to do multifiles ODBC – To query consolidated monthly excel datasheet SkySQL Ab 2012 Confidential
    21. 21. ODBC table type Access db example create table customers engine=connect table_type=ODBC block_size=10 tabname='Customers' Connection='DSN=MS Access Database;DBQ=C:/Program Files/Microsoft Office/Office/1033/FPNWIND.MDB;'; SkySQL Ab 2012 Confidential
    22. 22. ODBC database access From a linux box • UnixODBC must be used as an ODBC Driver manager. • The ODBC driver of the target database must be installed – For Oracle, DB2 – install Oracle Database instant Client with ODBC suplement SkySQL Ab 2012 Confidential
    23. 23. ODBC access database can pass any command to ODBC target create table crlite ( command varchar(128) not null, number int(5) not null flag=1, message varchar(255) flag=2) engine=connect table_type=odbc connection='Driver=SQLite3 ODBC Driver;Database=test.sqlite3;NoWCHAR=yes' option_list='Execsrc=1'; SkySQL Ab 2012 Confidential
    24. 24. ODBC access database can pass any command to ODBC target select * from crlite where command = 'update lite set birth = ''2012-07-14'' where ID = 2'; Can be wrapped in a procedure : create procedure send_cmd(cmd varchar(255)) MODIFIES SQL DATA select * from crlite where command = cmd; call send_cmd('drop tlite'); SkySQL Ab 2012 Confidential
    25. 25. Connect Storage Engine MYSQL table type (a proxy table) - same syntax as federatedx : create Table lineitem1 ENGINE=CONNECT TABLE_TYPE=MYSQL connection='mysql://proxy:pwd1@node1:3306/dbt3/lineitem3'; Server 1 node1 SkySQL Ab 2012 Confidential
    26. 26. MYSQL table Type agregation on the remote server create Table lineitem1 ENGINE=CONNECT TABLE_TYPE=MYSQL SRCDEF='select l_suppkey, sum(l_quantity) qt from dbt3.lineitem3 group by l_suppkey' connection='mysql://proxy:pwd1@node1:3306/dbt3/lineitem3’; Node 0 Node 1 SkySQL Ab 2012 Confidential
    27. 27. CONNECT Storage Engine MYSQL table type vs. Federated(X) • • • • support the limit clause Implements condition push down Autodiscovery of table structure Can define the columns we want to see SkySQL Ab 2012 Confidential
    28. 28. Connect Storage Engine TBL table type (// Merge) col1 col1 col2 col2 col3 ODBC table col1 col2 col3 MySQL table Muti tables table (like merge) – Different structure, not myisam only, – remotely distributed tables col1 col2 col3 SkySQL Ab 2012 Confidential col4
    29. 29. Table List Table (// Merge) works with a distributed configuration Node 1 col1 col2 Node 0 ODBC table MySQL table Node 2 col1 col2 col3 TBL MYSQL / ODBC Node 3 col1 col2 col3 SkySQL Ab 2012 Confidential col4
    30. 30. Parallel execution on distributed sharded tables Node 1 col1 col2 Node 0 ODBC table Node 2 col1 col2 col3 TBL MYSQL / ODBC Node 3 col1 col2 col3 SkySQL Ab 2012 Confidential MySQL table col4
    31. 31. Connect Storage Engine vs. MySQL Merge tables Table list table : - support non MyISAM tables - no need to the exact same structure for table - underlying tables can be remote – Distributed architecture SkySQL Ab 2012 Confidential
    32. 32. Importing /exporting MySQL data in various formats Importing file data into MySQL tables – Here for example from an XML file : create table biblio select * from xsampall2; Exporting data from MySQ: Here f we export to XML format : create table handout engine=CONNECT table_type=XML file_name='handout.htm' header=yes option_list='name=TABLE,coltype=HTML,attribute=border=1;cellpadding=5' select plugin_name handler, plugin_version version, plugin_author author, plugin_description description, plugin_maturity maturity from information_schema.plugins where plugin_type = 'STORAGE ENGINE'; SkySQL Ab 2012 Confidential
    33. 33. Ideas / Roadmap        ODBC type improvement MySQL table type improvement Batch key access Adaptative query ( // MySQL Cluster) ? Partition based TBL type(Like Spider) Transactional / XA support JSON File format SkySQL Ab 2012 Confidential
    34. 34. Where is the MariaDB Connect Storage Engine available ?       100 % open source on launchpad Binary packages on MariaDB.org Open Bug database Public Roadmap Released test cases Improvement request / worklog SkySQL Ab 2012 Confidential
    35. 35. How you can help • • • • Adopt it / Test it. Bugs : report bugs / propose fixes Documentation : help improve it Sharing : test it, blog about it, – Share your experience about interesting usages. SkySQL Ab 2012 Confidential
    36. 36. Conclusion  The MariaDB Connect Storage Engine :     Open MariaDB to BI and data analysis Brings real value to MariaDB users Illustrates openess of MariaDB community Supported by SkySQL / MariaDB SkySQL Ab 2012 Confidential
    37. 37. Thank You Documentation: https://mariadb.com/kb/en/connect/ Serge Frezefond @sfrezefond http://serge.frezefond.com SkySQL Ab 2012 Confidential 37

    ×