SkySQLMariaDBCONNECT Storage EngineSerge FrezefondSkySQL Ab 2012 Confidential
SkySQL Ab 2012 ConfidentialGoal of the CONNECT Storage Engine :BI on various file formatsMost of the data in companies is ...
SkySQL Ab 2012 ConfidentialBehind the sceneTraditional BIData is processed by an ETL– Change in the data model(denormaliza...
SkySQL Ab 2012 ConfidentialMariaDB CONNECT Storage Engine :created by Olivier BertrandIBM database researcher– Now retired...
SkySQL Ab 2012 ConfidentialHow did the CONNECT Storage Enginemove to MariaDB?Olivier met Monty creator of MySQL and MariaD...
SkySQL Ab 2012 ConfidentialThe CONNECT Storage EngineUses the MySQL Plugin ArchitectureA major differentiator of MySQLAllo...
SkySQL Ab 2012 ConfidentialThe CONNECT Storage Engineimplements advanced features●Support of external data sources :–Odbc,...
SkySQL Ab 2012 ConfidentialThe CONNECT Storage Engineimplements advanced features●Add indexing to files– index optimized f...
SkySQL Ab 2012 ConfidentialCONNECT Storage Engine vs.Federated(X) storage Engine- support ODBC– With muti files ODBC- supp...
SkySQL Ab 2012 ConfidentialCONNECT Storage EngineODBC componentAllow to access to any datasource accessiblethrough ODBC.– ...
SkySQL Ab 2012 ConfidentialThe CONNECT Storage Engineimplements advanced featuresCatalog table :– For Example Describe for...
SkySQL Ab 2012 ConfidentialThe CONNECT Storage Engineimplements advanced featuresMultiple CONNECT tables can be created on...
SkySQL Ab 2012 ConfidentialConnect Storage EngineVEC table / Column storecol1col1col2col1col3col2col1row3row2row1col3col1f...
SkySQL Ab 2012 ConfidentialConnect Storage EngineTable List Table (// Merge)col1col1 col2col1 col2col3col3col1 col2col3col...
SkySQL Ab 2012 ConfidentialConnect Storage Engine vs.MySQL Merge tablesTable list table :- support non MyISAM tables- no n...
SkySQL Ab 2012 ConfidentialImporting /exporting MySQL datain various formatsImporting file data into MySQL tables– Here fo...
SkySQL Ab 2012 ConfidentialWhere is the MariaDB Connect StorageEngine available ?●It is 100 % open source●It is available ...
SkySQL Ab 2012 ConfidentialHow you can helpAdopt it / Test it.Bugs : report bugs / propose fixesDocumentation : help impro...
SkySQL Ab 2012 ConfidentialConclusion●The MariaDB Connect Storage Engine :●Open MariaDB to BI and data analysis●Brings rea...
SkySQL Ab 2012 Confidential20Thank YouSerge Frezefondwww.skysql.comlwww.facebook.com/skysqlwww.linkedin.com/company/skysql
SkySQL Ab 2012 ConfidentialEnhancements consideredfor MariaDB connect storage engine- parallel access to multitables table-
Upcoming SlideShare
Loading in …5
×

MariaDB CONNECT Storage Engine

4,019 views

Published on

New CONNECT Storage Engine now in MariaDB 10.0.2

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,019
On SlideShare
0
From Embeds
0
Number of Embeds
354
Actions
Shares
0
Downloads
30
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

MariaDB CONNECT Storage Engine

  1. 1. SkySQLMariaDBCONNECT Storage EngineSerge FrezefondSkySQL Ab 2012 Confidential
  2. 2. SkySQL Ab 2012 ConfidentialGoal of the CONNECT Storage Engine :BI on various file formatsMost of the data in companies is in variousexternal datasources (many in non relationaldatabase format) :– Dbase, Firebird, SQlite– csv,– xml,– stored per column...– Microsoft Access & Excel– Distributed mysql serversTargeting BI data access on these formats.Not targeted for OLTP
  3. 3. SkySQL Ab 2012 ConfidentialBehind the sceneTraditional BIData is processed by an ETL– Change in the data model(denormalization...)Agregates are computed– Need to be defined and maintainedMight need to move data out of RDBMS toother kind of datastore– OLAP, Collumn store, Hadoop/Hbase ...Specific tools are used to query the dataIT is involved to maintain this machinery
  4. 4. SkySQL Ab 2012 ConfidentialMariaDB CONNECT Storage Engine :created by Olivier BertrandIBM database researcher– Now retired, 50 years expertise programingVery experienced on databases– Worked on system-R, DB2, natural languagequery ...Discovered MySQL when looking for friendlyplace to test new concepts.(2004)– Decided to go open source– Started to appreciate the MariaDB openessand friendlyness
  5. 5. SkySQL Ab 2012 ConfidentialHow did the CONNECT Storage Enginemove to MariaDB?Olivier met Monty creator of MySQL and MariaDB afew years ago (2004 for other concepts)SkySQL met Olivier and we start working together :– First access to launchpad, go to linux, test,Olivier start working with MariaDB team :– Testing, bug fixes, security, test cases ...SkySQL / MariaDB and Olivier agreed that is wasready to be released and supported under GPL– MariaDB flexibility ease integration
  6. 6. SkySQL Ab 2012 ConfidentialThe CONNECT Storage EngineUses the MySQL Plugin ArchitectureA major differentiator of MySQLAllow very specific datastore to interact withthe MySQL sql layerAllow advanced interaction– Specific Create Table parameters(MariaDB)– Condition push downAllow join with other storage engines– InnoDB / MyISAM tables
  7. 7. SkySQL Ab 2012 ConfidentialThe CONNECT Storage Engineimplements advanced features●Support of external data sources :–Odbc, MySQL, WMI ...●Support multi files tables●Support Big File Table > 2G●Support virtual tables (DIR)●Add autocreate of tables :–The structure is discovered from the datasource●Use MariaDB create table new parameterscapability (avoid comments polution)●Support compressed tables
  8. 8. SkySQL Ab 2012 ConfidentialThe CONNECT Storage Engineimplements advanced features●Add indexing to files– index optimized for read●Condition Push down– Used with ODBC and MySQL to push conditionto the target database. Big perf gain.●Support MariaDB virtuals columns●Support of special columns :– Rowid, fileid, tabid, servid●Muti tables table (like merge)– Different structure, not myisam only, remotelydistributed tables
  9. 9. SkySQL Ab 2012 ConfidentialCONNECT Storage Engine vs.Federated(X) storage Engine- support ODBC– With muti files ODBC- support access to MySQL table- support the limit clause
  10. 10. SkySQL Ab 2012 ConfidentialCONNECT Storage EngineODBC componentAllow to access to any datasource accessiblethrough ODBC.– Excel– Access– Firebird– SQLite– SQL Server, Oracle, DB2– ...Possibility to do multifiles ODBC– To query consolidated monthly excel datasheet
  11. 11. SkySQL Ab 2012 ConfidentialThe CONNECT Storage Engineimplements advanced featuresCatalog table :– For Example Describe for odbc table– No need to do create table– Access to data / column metadataMemory file maping– For file type table (not xml)Table format .ini
  12. 12. SkySQL Ab 2012 ConfidentialThe CONNECT Storage Engineimplements advanced featuresMultiple CONNECT tables can be created on thesame underlying file– Indexes can be shared between tables
  13. 13. SkySQL Ab 2012 ConfidentialConnect Storage EngineVEC table / Column storecol1col1col2col1col3col2col1row3row2row1col3col1freefreefreefreefreecol3free-1 or per column file- Indexes work- Fixed size record
  14. 14. SkySQL Ab 2012 ConfidentialConnect Storage EngineTable List Table (// Merge)col1col1 col2col1 col2col3col3col1 col2col3col1 col2ODBC tableMySQL tablecol4Muti tables table (like merge)– Different structure, not myisamonly, remotely distributed tables
  15. 15. SkySQL Ab 2012 ConfidentialConnect Storage Engine vs.MySQL Merge tablesTable list table :- support non MyISAM tables- no need to the exact same structure for table- underlying tables can be remote– Distributed architecture
  16. 16. SkySQL Ab 2012 ConfidentialImporting /exporting MySQL datain various formatsImporting file data into MySQL tables– Here for example from an XML file :• create table biblio select * from xsampall2;Exporting data from MySQL– Here for example we export to XML format :create table handout engine=CONNECT table_type=XMLfile_name=handout.htm header=yesoption_list=name=TABLE,coltype=HTML,attribute=border=1;cellpadding=5select plugin_name handler, plugin_version version, plugin_authorauthor, plugin_description description, plugin_maturity maturityfrom information_schema.plugins where plugin_type = STORAGE ENGINE;
  17. 17. SkySQL Ab 2012 ConfidentialWhere is the MariaDB Connect StorageEngine available ?●It is 100 % open source●It is available on MariaDB launchpad●Open Bug database●Public Roadmap●Released test cases●Improvement request / worklog
  18. 18. SkySQL Ab 2012 ConfidentialHow you can helpAdopt it / Test it.Bugs : report bugs / propose fixesDocumentation : help improve itSharing : test it, blog about it,– Share your experience about interestingusages.
  19. 19. SkySQL Ab 2012 ConfidentialConclusion●The MariaDB Connect Storage Engine :●Open MariaDB to BI and data analysis●Brings real value to MariaDB users●Illustrates openess of MariaDB community●Supported by SkySQL / MariaDB
  20. 20. SkySQL Ab 2012 Confidential20Thank YouSerge Frezefondwww.skysql.comlwww.facebook.com/skysqlwww.linkedin.com/company/skysql
  21. 21. SkySQL Ab 2012 ConfidentialEnhancements consideredfor MariaDB connect storage engine- parallel access to multitables table-

×