SlideShare a Scribd company logo
1 of 27
MySQL vs. MonetDB
A benchmark comparison between in-memory and out-of-memory
databases
Derek Aikins
Advisor: Dr. Feng Yu
Overview
• History of MySql.
• What is a relational database.
• History of Monetdb.
• What is a column-store database.
• Relational vs. Column-Store databases operations
• Goal of Project
• TPC-H
• Installing and compiling TPC-H
• Generating data using TPC-H
• Generating queries using TPC-H
• Queries
• Results of Tests
• Problems Encountered
• Conclusion
History of MySQL
• The world’s most popular open source relational database.
• Leading database choice for web-based applications and used by high
profile web properties including Facebook, Twitter, and Youtube.
• Created by a Swedish company, MySQL AB, originally developed by
David Axmark and Micheal Widenius in 1994.
• First Version release on May 23, 1995.
• MySQL AB was acquired by Sun Microsystems in 2008
• Oracle acquired Sun Microsystems on January 27, 2010
What is a Relational Database?
• MySql is a relational database
• A relational database is a digital database that organizes data into one
or more tables of columns and rows.
• Tables are known as relations
• Each table represents one “entity type” such as customer or product
• Rows (records) represent instances of that type of entity such as
“Lee” or “chair”
• Columns represent values to that instance such as address or price.
Examples of a database
Region Table
R_REGIONKEY R_NAME R_COMMENT
0 AFRICA
lar deposits. blithely final packages cajole. regular waters are
final requests. regular accounts are
1 AMERICA hs use ironic, even requests. s
2 ASIA ges. thinly even pinto beans ca
3 EUROPE ly final courts cajole furiously final excuse
4 MIDDLE EAST
uickly special accounts cajole carefully blithely close requests.
carefully final asymptotes
Nation Table
N_NATION
KEY
N_NAME N_REGIONKE
Y
N_COMMENT
0 ALGERIA 0 haggle. carefully final
1 ARGINTINA 1 al foxes promise slyly
2 BRAZIL 1 y alongside of the pending
3 CANADA 1 eas hang ironic,
4 EGYPT 4 y above the carefully
History of Monetdb
• An open source column-store database
• Developed in the Netherlands at the Centrum Wiskunde & Informatica (CWI)
• Data mining project in the 1990s required improved database support which
resulted in a CWI spin-off called Data Distilleries, which used early
implementations in its analytical suite
• Data Distilleries became a subsidiary of SPSS in 2003, which was later acquired
by IBM in 2009
• MonetDB in its current form was first created by Peter A. Boncz and Martin L.
Kerstein at the University of Amsterdam
• Please refer to Dr. Boncz’s thesis for more details
Monet: a next-Generation DBMS Kernel For Query-Intensive Applications
• The first version was released on September 30, 2004
What is a Column-Store Database
• Column-store databases store data as columns rather than rows
• By storing data in columns rather than rows, the database can access
the data it needs to answer a query rather than scanning and
discarding unwanted data in rows
• Query performance is often increased as a result, particularly in very
large data sets
• In most cases Column-Store databases store data in-memory (RAM)
unlike most row based databases that store their data on the
harddrive
Relational vs. Column-Store database
operation
Cust_ID Name Address City State Zip code Area
Code
Phone # Rent/Own Annual
Income
1 Jack 12 A St. Howland OH 44481 330 369-3597 Rent 74,000
2 Brian 13 B St. Howland OH 44481 330 856-1534 Rent 58,000
3 Mike 8 K St Warren OH 44483 330 373-1215 Own 92,000
4 Anna 62 Main St. Sharon PA 16101 724 654-0893 Own 110,000
5 Tasha 546 1st St. Stow OH 44752 216 849-5775 Rent 52,000
6 Sidney 84 Third St. Gilbert AZ 76534 480 758-6549 Own 90,000
7 Tyler 846 Wick Rd. Las Vegas NV 65487 231 654-5473 Own 60,000
8 Aaron 213 Maple St. Daytona FL 32547 519 159-3425 Rent 66,000
9 Beth 8749 Trump St. Detriot MI 87945 375 325-1849 Own 50,000
Goal of this Project
• Take a standard dataset and a standard set of queries and run the test
on two different databases, MySQL and Monetdb.
• By doing so, I intend to demonstrate the efficiency and speed that a
column-store database has over a traditional relational database.
• To do this I will be using a data generator, TPC-H, for benchmarking
databases which can also generate the queries for the data.
• Then push all data generated by TPC-H into both databases and run
each queries multiple times to get average times on both databases.
• After all runs are complete, I will gather all results generated and
compare how the two databases performed.
TPC-H
• A decision support benchmark
• Consists of a suite of business oriented ad-hoc queries and
concurrent data modifications
• The queries and the data populating the database have been chosen
to have broad industry-wide relevance
• This benchmark illustrates decision support systems that examine
large volumes of data, execute queries with a high degree of
complexity, and give answers to critical business questions
Installing and Compiling TPC-H
• The program used to generate the data from TPC-H is called dbgen.
• To install dbgen first I need to download the file from the TPC-H site using the
following command cd Downloads/tpch_2_16_0/tpch_2_15_0/dbgen/
• Then I had to create a makefile and go in and change the lines to CC = gcc
DATABASE=SQLSERVER Machine=LINUX
Workload=TPCH
• Next in the dbgen folder I had to find the tpcd.h file and edit the lines
define START_TRAN "BEGIN WORK;“ define
END_TRAN "COMMIT WORK;“
• Then I ran the make command.
Generating the data- 100 Mb
• After installing and setup TPC-H, I generated the data using dbgen
• Using the command ./dbgen -s 0.1 to generate the data where the 0.1 in the
command dictates the amount of data to be generated in this case I use 100 Mb
• Once the data was generated I created the database in MySQL using the CREATE
DATABASE tpch; command and then chose the database to load the data in the
tables.
• I then created each table with the CREATE TABLE command and set all the
descriptions for each column
• Once the tables were created it was time to load the data into each table using
the LOAD DATA LOCAL INFILE 'customer.tbl' INTO TABLE CUSTOMER FIELDS
TERMINATED BY '|‘ and changing the table name to each table
Query 1
mysql> select
s_acctbal,
s_name,
n_name,
p_partkey, where
p_mfgr, p_partkey = ps_partkey
s_address, and s_suppkey = ps_suppkey
s_phone, and s_nationkey = n_nationkey
s_comment and n_regionkey = r_regionkey
from and r_name = 'ASIA'
part, order by
supplier, s_acctbal desc,
partsupp, n_name,
nation, s_name
region p_partkey:
where
p_partkey = ps_partkey
and s_suppkey = ps_suppkey
and p_size = 19
and p_type like 'PROMO ANODIZED BRASS'
and s_nationkey = n_nationkey
and n_regionkey = r_regionkey
and r_name = 'ASIA'
and ps_supplycost = (
select
min(ps_supplycost)
from
partsupp,
supplier,
nation,
region
)
Query 2
select
-> l_orderkey,
-> sum(l_extendedprice * (1 - l_discount)) as revenue,
-> o_orderdate,
-> o_shippriority
-> from
-> customer,
-> orders,
-> lineitem
-> where
-> c_mktsegment = 'AUTOMOBILE'
-> and c_custkey = o_custkey
-> and l_orderkey = o_orderkey
-> and o_orderdate < date '1995-08-15'
-> and l_shipdate > date '1995-08-27'
-> group by
-> l_orderkey,
-> o_orderdate,
-> o_shippriority
-> order by
-> revenue desc,
-> o_orderdate;
Query 3
select
o_orderpriority,
count(*) as order_count
from
orders
where
o_orderdate >= date '1996-10-29'
and o_orderdate < date '1996-10-29' + interval '3' month
and exists (
select
*
from
lineitem
where
where
l_orderkey = o_orderkey
and l_commitdate < l_receiptdate
)
)
group by
o_orderpriority
order by
o_orderpriority;
Query 4
select
sum(l_extendedprice * l_discount) as revenue
from
lineitem
where
l_shipdate >= date '1993-03-05'
and l_shipdate < date '1993-03-06' + interval '1' year
and l_discount between .03 and .06
and l_quantity < 3;
Query 5
select
sum(l_extendedprice) / 7.0 as avg_yearly
from
lineitem,
part
where
p_partkey = l_partkeyt
and p_brand = 'Brand#11'
and p_container = 'MED JAR'
and l_quantity < (
select
0.2 * avg(l_quantity)
from
lineitem
where
l_partkey = p_partkey
);
Results for Queries 1 & 2
Query 1 Results
26.156667 25.97666667
0.0120647
0
5
10
15
20
25
30
Query 1
TIMEINSECONDS
Chart Title
MySQL (InnoDB) MySQL (MyISAM) Monetdb
Query 2 Results
645.31667 652.2966667
0.0201565
0
100
200
300
400
500
600
700
Query 2
TIMEINSECONDS
Chart Title
MySQL (InnoDB) MySQL (MyISAM) Monetdb
Results for Query 3 & 4
853.25
2086.01
0.014732667
0
500
1000
1500
2000
2500
Query 3
TIMEINSECONDS
QUERY 3
MySQL (InnoDB) MySQL (MyISAM) Monetdb
0.483333333
0.29333333
0.011313667
0
0.1
0.2
0.3
0.4
0.5
0.6
Query 4
TIMEINSECONDS
QUERY 4
MySQL (InnoDB) MySQL (MyISAM) Monetdb
Result for Query 5
137.1533333
337.2766667
0.018765667
0
50
100
150
200
250
300
350
400
Query 5
TIMEINSECONDS
QUERY 5
MySQL (InnoDB) MySQL (MyISAM) Monetdb
Total Numerical Results
MySQL (InnoDB) MySQL (MyISAM) Monetdb
QUERY TIME TIME (in seconds) QUERY TIME TIME (in seconds) Query TIME TIME (in seconds)
Query 1 Query 1 Query 1
run 1 26.14 sec 26.14 run 1 25.83 sec 25.83 run 1 12.854 ms 0.012854
run 2 26.20 sec 26.2 run 2 26.06 sec 26.06 run 2 12.141 ms 0.012141
run 3 26.13 sec 26.13 run 3 26.04 sec 26.04 run 3 11.199 ms 0.011199
Average Time 26.15666667 Average Time 25.977 Average Time 0.012064667
Total Numerical Results
MySQL (InnoDB) MySQL (MyISAM) Monetdb
Query 2 TIME TIME (in seconds) Query 2 TIME TIME (in seconds) Query 2 TIME TIME (in seconds)
run 1 10 min 42.03 sec 642.03 run 1 10 min 52.36 sec 652.36 run 1 23.548 ms 0.023548
run 2 10 min 46.84 sec 646.84 run 2 10 min 51.56 sec 652.56 run 2 16.765 ms 0.016765
run 3 10 min 47.08 sec 647.08 run 3 10 min 51.97 sec 651.97 run 3 24.834 ms 0.024834
Average time 645.3166667 Average time 652.2966667 Average time 0.0201565
Query 3 Query 3 Query 3
run 1 14 min 0.65 sec 840.65 run 1 35 min 53.17 sec 2153.17 run 1 15.332 ms 0.015332
run 2 14 min 12.67 sec 852.67 run 2 34 min 11.52 sec 2051.52 run 2 15.168 ms 0.015168
run 3 14 min 26.43 sec 866.43 run 3 34 min 13.34 sec 2053.34 run 3 13.698 ms 0.013698
Average time 853.25 Average time 2086.01 Average time 0.014732667
Total Numerical Results
MySQL (InnoDB) MySQL (MyISAM) Monetdb
QUERY TIME TIME (in seconds) QUERY TIME TIME (in seconds) Query TIME TIME (in seconds)
Query 4 Query 4 Query 4
run 1 .48 sec 0.48 run 1 0.29 sec 0.29 run 1 12.992 ms 0.012992
run 2 .49 sec 0.49 run 2 0.30 sec 0.30 run 2 10.641 ms 0.010641
run 3 .48 sec 0.48 run 3 0.29 sec 0.29 run 3 10.308 ms 0.010308
Average time 0.483333333 Average time 0.293333333 Average time 0.011313667
Query 5 Query 5 Query 5
run 1 2 min 18.08 sec 138.08 run 1 5 min 36.59 sec 336.59 run 1 18.776 ms 0.018776
run 2 2 min 19.34 sec 139.34 run 2 5 min 39.10 sec 339.1 run 2 21.746 ms 0.021746
run 3 2 min 14.04 sec 134.04 run 3 5 min 36.14 sec 336.14 run 3 15.775 ms 0.015775
Average time 137.1533333 Average time 337.2766667 Average time 0.018765667
Comparison Results
Queries MySQL
(InnoDB)
MySQL
(MyISAM)
Monetdb InnoDB/
MyISAM
InnoDB/
Monetdb
MyISAM/
Monetdb
Average time Average time Average time
Query 1 26.157 25.977 0.0121 0.0069 times faster
than InnoDB
2,168 times faster
than InnoDB
2,153 times faster
than MyISAM
Query 2 645.317 652.297 0.0201 0.0108 times slower
than InnoDB
32,015 times faster
than InnoDB
32,361 times faster
than MyISAM
Query 3 853.25 2086.01 0.0147 2.445 times slower
than InnoDB
57,915 times faster
than InnoDB
141,591 times faster
than MyISAM
Query 4 0.0483 0.293 0.0113 1.648 times faster
than InnoDB
43 times faster than
InnoDB
26 times faster than
MyISAM
Query 5 137.153 337.277 0.0188 2.459 times slower
than InnoDB
7309 times faster than
InnoDB
17,973 times faster
than MyISAM
Challenges Encountered
• Throughout this project I encountered several challenges:
• The first difficulty encountered was the installation of the several programs
used for this project
• Once all programs were installed the next challenge was the uploading of the
data to the databases
• After all data was loaded into the database tables, one of the largest
challenges was to examine each query and fill in areas that needed exact
information from the tables for the query to even run
• The largest challenge I faced through this entire project was learning to use
the command line to do everything as I have not had much experience with
this.
Summary
• A relational database is a digital database that organizes data into one
or more tables of columns and rows.
• Column-store databases store data as columns rather than rows
• TPC-H is a decision support benchmark that examine large volumes of
data, execute queries with a high degree of complexity, and give
answers to critical business questions
• As the data shows from the tests conducted on the two different
databases, column-store databases such as Monetdb are considerably
faster in run time compared to traditional relational databases such
as MySQL.
ALL Results and Test Queries can be found at:
https://github.com/Djaikins/MySQL-vs-Monetdb
or
By searching djaikins on Github

More Related Content

What's hot

Column oriented database
Column oriented databaseColumn oriented database
Column oriented database
Kanike Krishna
 

What's hot (20)

Bigtable
BigtableBigtable
Bigtable
 
In-memory database
In-memory databaseIn-memory database
In-memory database
 
Fractal Tree Indexes : From Theory to Practice
Fractal Tree Indexes : From Theory to PracticeFractal Tree Indexes : From Theory to Practice
Fractal Tree Indexes : From Theory to Practice
 
IBM DB2 LUW/UDB DBA Training by www.etraining.guru
IBM DB2 LUW/UDB DBA Training by www.etraining.guruIBM DB2 LUW/UDB DBA Training by www.etraining.guru
IBM DB2 LUW/UDB DBA Training by www.etraining.guru
 
Data Organization in InnoDB
Data Organization in InnoDBData Organization in InnoDB
Data Organization in InnoDB
 
Big table
Big tableBig table
Big table
 
Get More Out of MySQL with TokuDB
Get More Out of MySQL with TokuDBGet More Out of MySQL with TokuDB
Get More Out of MySQL with TokuDB
 
What’s new in MariaDB ColumnStore
What’s new in MariaDB ColumnStoreWhat’s new in MariaDB ColumnStore
What’s new in MariaDB ColumnStore
 
SQL Server In-Memory OLTP introduction (Hekaton)
SQL Server In-Memory OLTP introduction (Hekaton)SQL Server In-Memory OLTP introduction (Hekaton)
SQL Server In-Memory OLTP introduction (Hekaton)
 
Google Bigtable paper presentation
Google Bigtable paper presentationGoogle Bigtable paper presentation
Google Bigtable paper presentation
 
Column oriented database
Column oriented databaseColumn oriented database
Column oriented database
 
MySQL database
MySQL databaseMySQL database
MySQL database
 
cPanelCon 2015: InnoDB Alchemy
cPanelCon 2015: InnoDB AlchemycPanelCon 2015: InnoDB Alchemy
cPanelCon 2015: InnoDB Alchemy
 
Log Structured Merge Tree
Log Structured Merge TreeLog Structured Merge Tree
Log Structured Merge Tree
 
Performance tuning ColumnStore
Performance tuning ColumnStorePerformance tuning ColumnStore
Performance tuning ColumnStore
 
Big table presentation-final
Big table presentation-finalBig table presentation-final
Big table presentation-final
 
Dd and atomic ddl pl17 dublin
Dd and atomic ddl pl17 dublinDd and atomic ddl pl17 dublin
Dd and atomic ddl pl17 dublin
 
Google BigTable
Google BigTableGoogle BigTable
Google BigTable
 
google Bigtable
google Bigtablegoogle Bigtable
google Bigtable
 
Bigtable
BigtableBigtable
Bigtable
 

Viewers also liked

Lofar python meetup jan9 2013
Lofar python meetup jan9 2013Lofar python meetup jan9 2013
Lofar python meetup jan9 2013
Gijs Molenaar
 
Revoca concessione sai_8_-_documento_-consorzio_ato_delib_121
Revoca concessione sai_8_-_documento_-consorzio_ato_delib_121Revoca concessione sai_8_-_documento_-consorzio_ato_delib_121
Revoca concessione sai_8_-_documento_-consorzio_ato_delib_121
Stefano Battiato
 
Reglamento dgpmpt 10-05-2013
Reglamento dgpmpt 10-05-2013Reglamento dgpmpt 10-05-2013
Reglamento dgpmpt 10-05-2013
CECY50
 
Ejercicios
EjerciciosEjercicios
Ejercicios
CECY50
 

Viewers also liked (20)

Lofar python meetup jan9 2013
Lofar python meetup jan9 2013Lofar python meetup jan9 2013
Lofar python meetup jan9 2013
 
Tpc h benchmarking no mysql
Tpc h benchmarking no mysqlTpc h benchmarking no mysql
Tpc h benchmarking no mysql
 
TPC-H in MongoDB
TPC-H in MongoDBTPC-H in MongoDB
TPC-H in MongoDB
 
AssociaBrasil
AssociaBrasilAssociaBrasil
AssociaBrasil
 
Vinda da Família Real ao Brasil
Vinda da Família Real ao BrasilVinda da Família Real ao Brasil
Vinda da Família Real ao Brasil
 
Why are you coding like this?
Why are you coding like this?Why are you coding like this?
Why are you coding like this?
 
Revoca concessione sai_8_-_documento_-consorzio_ato_delib_121
Revoca concessione sai_8_-_documento_-consorzio_ato_delib_121Revoca concessione sai_8_-_documento_-consorzio_ato_delib_121
Revoca concessione sai_8_-_documento_-consorzio_ato_delib_121
 
Miguel Maestripieri video 5 amigos
Miguel Maestripieri video 5 amigosMiguel Maestripieri video 5 amigos
Miguel Maestripieri video 5 amigos
 
Reglamento dgpmpt 10-05-2013
Reglamento dgpmpt 10-05-2013Reglamento dgpmpt 10-05-2013
Reglamento dgpmpt 10-05-2013
 
EDUCATIVO
EDUCATIVOEDUCATIVO
EDUCATIVO
 
My last vacation
My last vacationMy last vacation
My last vacation
 
Proyecto integrador
Proyecto integradorProyecto integrador
Proyecto integrador
 
Man
ManMan
Man
 
From ExactSource- Rules Regatding Admission of Expert Witness Testimony
From ExactSource- Rules Regatding Admission of Expert Witness TestimonyFrom ExactSource- Rules Regatding Admission of Expert Witness Testimony
From ExactSource- Rules Regatding Admission of Expert Witness Testimony
 
Dona guidinha do poço
Dona guidinha do poçoDona guidinha do poço
Dona guidinha do poço
 
PostgreSQL and Benchmarks
PostgreSQL and BenchmarksPostgreSQL and Benchmarks
PostgreSQL and Benchmarks
 
Ensayo historia iv
Ensayo historia ivEnsayo historia iv
Ensayo historia iv
 
Vitaminas
VitaminasVitaminas
Vitaminas
 
Ejercicios
EjerciciosEjercicios
Ejercicios
 
Etica profissional
Etica profissionalEtica profissional
Etica profissional
 

Similar to MySQL vs. MonetDB

Tutorial(release)
Tutorial(release)Tutorial(release)
Tutorial(release)
Oshin Hung
 
Extending Apache Spark SQL Data Source APIs with Join Push Down with Ioana De...
Extending Apache Spark SQL Data Source APIs with Join Push Down with Ioana De...Extending Apache Spark SQL Data Source APIs with Join Push Down with Ioana De...
Extending Apache Spark SQL Data Source APIs with Join Push Down with Ioana De...
Databricks
 
Apache Solr as a compressed, scalable, and high performance time series database
Apache Solr as a compressed, scalable, and high performance time series databaseApache Solr as a compressed, scalable, and high performance time series database
Apache Solr as a compressed, scalable, and high performance time series database
Florian Lautenschlager
 
Secrets of highly_avail_oltp_archs
Secrets of highly_avail_oltp_archsSecrets of highly_avail_oltp_archs
Secrets of highly_avail_oltp_archs
Tarik Essawi
 
How Azure Databricks helped make IoT Analytics a Reality with Janath Manohara...
How Azure Databricks helped make IoT Analytics a Reality with Janath Manohara...How Azure Databricks helped make IoT Analytics a Reality with Janath Manohara...
How Azure Databricks helped make IoT Analytics a Reality with Janath Manohara...
Databricks
 

Similar to MySQL vs. MonetDB (20)

MySQL vs MonetDB Bencharmarks
MySQL vs MonetDB BencharmarksMySQL vs MonetDB Bencharmarks
MySQL vs MonetDB Bencharmarks
 
Data Science Lab Meetup: Cassandra and Spark
Data Science Lab Meetup: Cassandra and SparkData Science Lab Meetup: Cassandra and Spark
Data Science Lab Meetup: Cassandra and Spark
 
Tutorial(release)
Tutorial(release)Tutorial(release)
Tutorial(release)
 
Cassandra Day Denver 2014: Using Cassandra to Support Crisis Informatics Rese...
Cassandra Day Denver 2014: Using Cassandra to Support Crisis Informatics Rese...Cassandra Day Denver 2014: Using Cassandra to Support Crisis Informatics Rese...
Cassandra Day Denver 2014: Using Cassandra to Support Crisis Informatics Rese...
 
Big Data Warehousing Meetup: Real-time Trade Data Monitoring with Storm & Cas...
Big Data Warehousing Meetup: Real-time Trade Data Monitoring with Storm & Cas...Big Data Warehousing Meetup: Real-time Trade Data Monitoring with Storm & Cas...
Big Data Warehousing Meetup: Real-time Trade Data Monitoring with Storm & Cas...
 
Scaling Elasticsearch at Synthesio
Scaling Elasticsearch at SynthesioScaling Elasticsearch at Synthesio
Scaling Elasticsearch at Synthesio
 
Deploying your Data Warehouse on AWS
Deploying your Data Warehouse on AWSDeploying your Data Warehouse on AWS
Deploying your Data Warehouse on AWS
 
Calum McCrea, Software Engineer at Kx Systems, "Kx: How Wall Street Tech can ...
Calum McCrea, Software Engineer at Kx Systems, "Kx: How Wall Street Tech can ...Calum McCrea, Software Engineer at Kx Systems, "Kx: How Wall Street Tech can ...
Calum McCrea, Software Engineer at Kx Systems, "Kx: How Wall Street Tech can ...
 
Best Practices for Supercharging Cloud Analytics on Amazon Redshift
Best Practices for Supercharging Cloud Analytics on Amazon RedshiftBest Practices for Supercharging Cloud Analytics on Amazon Redshift
Best Practices for Supercharging Cloud Analytics on Amazon Redshift
 
Sizing MongoDB Clusters
Sizing MongoDB Clusters Sizing MongoDB Clusters
Sizing MongoDB Clusters
 
What You Need To Know About The Top Database Trends
What You Need To Know About The Top Database TrendsWhat You Need To Know About The Top Database Trends
What You Need To Know About The Top Database Trends
 
Extending Apache Spark SQL Data Source APIs with Join Push Down with Ioana De...
Extending Apache Spark SQL Data Source APIs with Join Push Down with Ioana De...Extending Apache Spark SQL Data Source APIs with Join Push Down with Ioana De...
Extending Apache Spark SQL Data Source APIs with Join Push Down with Ioana De...
 
Paradigmas de procesamiento en Big Data: estado actual, tendencias y oportu...
Paradigmas de procesamiento en  Big Data: estado actual,  tendencias y oportu...Paradigmas de procesamiento en  Big Data: estado actual,  tendencias y oportu...
Paradigmas de procesamiento en Big Data: estado actual, tendencias y oportu...
 
Apache Solr as a compressed, scalable, and high performance time series database
Apache Solr as a compressed, scalable, and high performance time series databaseApache Solr as a compressed, scalable, and high performance time series database
Apache Solr as a compressed, scalable, and high performance time series database
 
Curriculum Associates Strata NYC 2017
Curriculum Associates Strata NYC 2017Curriculum Associates Strata NYC 2017
Curriculum Associates Strata NYC 2017
 
Curriculum Associates Strata NYC 2017
Curriculum Associates Strata NYC 2017Curriculum Associates Strata NYC 2017
Curriculum Associates Strata NYC 2017
 
Curriculum Associates Strata NYC 2017
Curriculum Associates Strata NYC 2017Curriculum Associates Strata NYC 2017
Curriculum Associates Strata NYC 2017
 
AWS re:Invent 2016| DAT318 | Migrating from RDBMS to NoSQL: How Sony Moved fr...
AWS re:Invent 2016| DAT318 | Migrating from RDBMS to NoSQL: How Sony Moved fr...AWS re:Invent 2016| DAT318 | Migrating from RDBMS to NoSQL: How Sony Moved fr...
AWS re:Invent 2016| DAT318 | Migrating from RDBMS to NoSQL: How Sony Moved fr...
 
Secrets of highly_avail_oltp_archs
Secrets of highly_avail_oltp_archsSecrets of highly_avail_oltp_archs
Secrets of highly_avail_oltp_archs
 
How Azure Databricks helped make IoT Analytics a Reality with Janath Manohara...
How Azure Databricks helped make IoT Analytics a Reality with Janath Manohara...How Azure Databricks helped make IoT Analytics a Reality with Janath Manohara...
How Azure Databricks helped make IoT Analytics a Reality with Janath Manohara...
 

Recently uploaded

Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Peter Udo Diehl
 

Recently uploaded (20)

Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
 
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfThe Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
 
AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101
 
Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
 
ECS 2024 Teams Premium - Pretty Secure
ECS 2024   Teams Premium - Pretty SecureECS 2024   Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty Secure
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeFree and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
 
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties ReimaginedEasier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
 
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxUnpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through Observability
 
A Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyA Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System Strategy
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
 
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. Startups
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at Comcast
 

MySQL vs. MonetDB

  • 1. MySQL vs. MonetDB A benchmark comparison between in-memory and out-of-memory databases Derek Aikins Advisor: Dr. Feng Yu
  • 2. Overview • History of MySql. • What is a relational database. • History of Monetdb. • What is a column-store database. • Relational vs. Column-Store databases operations • Goal of Project • TPC-H • Installing and compiling TPC-H • Generating data using TPC-H • Generating queries using TPC-H • Queries • Results of Tests • Problems Encountered • Conclusion
  • 3. History of MySQL • The world’s most popular open source relational database. • Leading database choice for web-based applications and used by high profile web properties including Facebook, Twitter, and Youtube. • Created by a Swedish company, MySQL AB, originally developed by David Axmark and Micheal Widenius in 1994. • First Version release on May 23, 1995. • MySQL AB was acquired by Sun Microsystems in 2008 • Oracle acquired Sun Microsystems on January 27, 2010
  • 4. What is a Relational Database? • MySql is a relational database • A relational database is a digital database that organizes data into one or more tables of columns and rows. • Tables are known as relations • Each table represents one “entity type” such as customer or product • Rows (records) represent instances of that type of entity such as “Lee” or “chair” • Columns represent values to that instance such as address or price.
  • 5. Examples of a database Region Table R_REGIONKEY R_NAME R_COMMENT 0 AFRICA lar deposits. blithely final packages cajole. regular waters are final requests. regular accounts are 1 AMERICA hs use ironic, even requests. s 2 ASIA ges. thinly even pinto beans ca 3 EUROPE ly final courts cajole furiously final excuse 4 MIDDLE EAST uickly special accounts cajole carefully blithely close requests. carefully final asymptotes Nation Table N_NATION KEY N_NAME N_REGIONKE Y N_COMMENT 0 ALGERIA 0 haggle. carefully final 1 ARGINTINA 1 al foxes promise slyly 2 BRAZIL 1 y alongside of the pending 3 CANADA 1 eas hang ironic, 4 EGYPT 4 y above the carefully
  • 6. History of Monetdb • An open source column-store database • Developed in the Netherlands at the Centrum Wiskunde & Informatica (CWI) • Data mining project in the 1990s required improved database support which resulted in a CWI spin-off called Data Distilleries, which used early implementations in its analytical suite • Data Distilleries became a subsidiary of SPSS in 2003, which was later acquired by IBM in 2009 • MonetDB in its current form was first created by Peter A. Boncz and Martin L. Kerstein at the University of Amsterdam • Please refer to Dr. Boncz’s thesis for more details Monet: a next-Generation DBMS Kernel For Query-Intensive Applications • The first version was released on September 30, 2004
  • 7. What is a Column-Store Database • Column-store databases store data as columns rather than rows • By storing data in columns rather than rows, the database can access the data it needs to answer a query rather than scanning and discarding unwanted data in rows • Query performance is often increased as a result, particularly in very large data sets • In most cases Column-Store databases store data in-memory (RAM) unlike most row based databases that store their data on the harddrive
  • 8. Relational vs. Column-Store database operation Cust_ID Name Address City State Zip code Area Code Phone # Rent/Own Annual Income 1 Jack 12 A St. Howland OH 44481 330 369-3597 Rent 74,000 2 Brian 13 B St. Howland OH 44481 330 856-1534 Rent 58,000 3 Mike 8 K St Warren OH 44483 330 373-1215 Own 92,000 4 Anna 62 Main St. Sharon PA 16101 724 654-0893 Own 110,000 5 Tasha 546 1st St. Stow OH 44752 216 849-5775 Rent 52,000 6 Sidney 84 Third St. Gilbert AZ 76534 480 758-6549 Own 90,000 7 Tyler 846 Wick Rd. Las Vegas NV 65487 231 654-5473 Own 60,000 8 Aaron 213 Maple St. Daytona FL 32547 519 159-3425 Rent 66,000 9 Beth 8749 Trump St. Detriot MI 87945 375 325-1849 Own 50,000
  • 9. Goal of this Project • Take a standard dataset and a standard set of queries and run the test on two different databases, MySQL and Monetdb. • By doing so, I intend to demonstrate the efficiency and speed that a column-store database has over a traditional relational database. • To do this I will be using a data generator, TPC-H, for benchmarking databases which can also generate the queries for the data. • Then push all data generated by TPC-H into both databases and run each queries multiple times to get average times on both databases. • After all runs are complete, I will gather all results generated and compare how the two databases performed.
  • 10. TPC-H • A decision support benchmark • Consists of a suite of business oriented ad-hoc queries and concurrent data modifications • The queries and the data populating the database have been chosen to have broad industry-wide relevance • This benchmark illustrates decision support systems that examine large volumes of data, execute queries with a high degree of complexity, and give answers to critical business questions
  • 11. Installing and Compiling TPC-H • The program used to generate the data from TPC-H is called dbgen. • To install dbgen first I need to download the file from the TPC-H site using the following command cd Downloads/tpch_2_16_0/tpch_2_15_0/dbgen/ • Then I had to create a makefile and go in and change the lines to CC = gcc DATABASE=SQLSERVER Machine=LINUX Workload=TPCH • Next in the dbgen folder I had to find the tpcd.h file and edit the lines define START_TRAN "BEGIN WORK;“ define END_TRAN "COMMIT WORK;“ • Then I ran the make command.
  • 12. Generating the data- 100 Mb • After installing and setup TPC-H, I generated the data using dbgen • Using the command ./dbgen -s 0.1 to generate the data where the 0.1 in the command dictates the amount of data to be generated in this case I use 100 Mb • Once the data was generated I created the database in MySQL using the CREATE DATABASE tpch; command and then chose the database to load the data in the tables. • I then created each table with the CREATE TABLE command and set all the descriptions for each column • Once the tables were created it was time to load the data into each table using the LOAD DATA LOCAL INFILE 'customer.tbl' INTO TABLE CUSTOMER FIELDS TERMINATED BY '|‘ and changing the table name to each table
  • 13. Query 1 mysql> select s_acctbal, s_name, n_name, p_partkey, where p_mfgr, p_partkey = ps_partkey s_address, and s_suppkey = ps_suppkey s_phone, and s_nationkey = n_nationkey s_comment and n_regionkey = r_regionkey from and r_name = 'ASIA' part, order by supplier, s_acctbal desc, partsupp, n_name, nation, s_name region p_partkey: where p_partkey = ps_partkey and s_suppkey = ps_suppkey and p_size = 19 and p_type like 'PROMO ANODIZED BRASS' and s_nationkey = n_nationkey and n_regionkey = r_regionkey and r_name = 'ASIA' and ps_supplycost = ( select min(ps_supplycost) from partsupp, supplier, nation, region )
  • 14. Query 2 select -> l_orderkey, -> sum(l_extendedprice * (1 - l_discount)) as revenue, -> o_orderdate, -> o_shippriority -> from -> customer, -> orders, -> lineitem -> where -> c_mktsegment = 'AUTOMOBILE' -> and c_custkey = o_custkey -> and l_orderkey = o_orderkey -> and o_orderdate < date '1995-08-15' -> and l_shipdate > date '1995-08-27' -> group by -> l_orderkey, -> o_orderdate, -> o_shippriority -> order by -> revenue desc, -> o_orderdate;
  • 15. Query 3 select o_orderpriority, count(*) as order_count from orders where o_orderdate >= date '1996-10-29' and o_orderdate < date '1996-10-29' + interval '3' month and exists ( select * from lineitem where where l_orderkey = o_orderkey and l_commitdate < l_receiptdate ) ) group by o_orderpriority order by o_orderpriority;
  • 16. Query 4 select sum(l_extendedprice * l_discount) as revenue from lineitem where l_shipdate >= date '1993-03-05' and l_shipdate < date '1993-03-06' + interval '1' year and l_discount between .03 and .06 and l_quantity < 3;
  • 17. Query 5 select sum(l_extendedprice) / 7.0 as avg_yearly from lineitem, part where p_partkey = l_partkeyt and p_brand = 'Brand#11' and p_container = 'MED JAR' and l_quantity < ( select 0.2 * avg(l_quantity) from lineitem where l_partkey = p_partkey );
  • 18. Results for Queries 1 & 2 Query 1 Results 26.156667 25.97666667 0.0120647 0 5 10 15 20 25 30 Query 1 TIMEINSECONDS Chart Title MySQL (InnoDB) MySQL (MyISAM) Monetdb Query 2 Results 645.31667 652.2966667 0.0201565 0 100 200 300 400 500 600 700 Query 2 TIMEINSECONDS Chart Title MySQL (InnoDB) MySQL (MyISAM) Monetdb
  • 19. Results for Query 3 & 4 853.25 2086.01 0.014732667 0 500 1000 1500 2000 2500 Query 3 TIMEINSECONDS QUERY 3 MySQL (InnoDB) MySQL (MyISAM) Monetdb 0.483333333 0.29333333 0.011313667 0 0.1 0.2 0.3 0.4 0.5 0.6 Query 4 TIMEINSECONDS QUERY 4 MySQL (InnoDB) MySQL (MyISAM) Monetdb
  • 20. Result for Query 5 137.1533333 337.2766667 0.018765667 0 50 100 150 200 250 300 350 400 Query 5 TIMEINSECONDS QUERY 5 MySQL (InnoDB) MySQL (MyISAM) Monetdb
  • 21. Total Numerical Results MySQL (InnoDB) MySQL (MyISAM) Monetdb QUERY TIME TIME (in seconds) QUERY TIME TIME (in seconds) Query TIME TIME (in seconds) Query 1 Query 1 Query 1 run 1 26.14 sec 26.14 run 1 25.83 sec 25.83 run 1 12.854 ms 0.012854 run 2 26.20 sec 26.2 run 2 26.06 sec 26.06 run 2 12.141 ms 0.012141 run 3 26.13 sec 26.13 run 3 26.04 sec 26.04 run 3 11.199 ms 0.011199 Average Time 26.15666667 Average Time 25.977 Average Time 0.012064667
  • 22. Total Numerical Results MySQL (InnoDB) MySQL (MyISAM) Monetdb Query 2 TIME TIME (in seconds) Query 2 TIME TIME (in seconds) Query 2 TIME TIME (in seconds) run 1 10 min 42.03 sec 642.03 run 1 10 min 52.36 sec 652.36 run 1 23.548 ms 0.023548 run 2 10 min 46.84 sec 646.84 run 2 10 min 51.56 sec 652.56 run 2 16.765 ms 0.016765 run 3 10 min 47.08 sec 647.08 run 3 10 min 51.97 sec 651.97 run 3 24.834 ms 0.024834 Average time 645.3166667 Average time 652.2966667 Average time 0.0201565 Query 3 Query 3 Query 3 run 1 14 min 0.65 sec 840.65 run 1 35 min 53.17 sec 2153.17 run 1 15.332 ms 0.015332 run 2 14 min 12.67 sec 852.67 run 2 34 min 11.52 sec 2051.52 run 2 15.168 ms 0.015168 run 3 14 min 26.43 sec 866.43 run 3 34 min 13.34 sec 2053.34 run 3 13.698 ms 0.013698 Average time 853.25 Average time 2086.01 Average time 0.014732667
  • 23. Total Numerical Results MySQL (InnoDB) MySQL (MyISAM) Monetdb QUERY TIME TIME (in seconds) QUERY TIME TIME (in seconds) Query TIME TIME (in seconds) Query 4 Query 4 Query 4 run 1 .48 sec 0.48 run 1 0.29 sec 0.29 run 1 12.992 ms 0.012992 run 2 .49 sec 0.49 run 2 0.30 sec 0.30 run 2 10.641 ms 0.010641 run 3 .48 sec 0.48 run 3 0.29 sec 0.29 run 3 10.308 ms 0.010308 Average time 0.483333333 Average time 0.293333333 Average time 0.011313667 Query 5 Query 5 Query 5 run 1 2 min 18.08 sec 138.08 run 1 5 min 36.59 sec 336.59 run 1 18.776 ms 0.018776 run 2 2 min 19.34 sec 139.34 run 2 5 min 39.10 sec 339.1 run 2 21.746 ms 0.021746 run 3 2 min 14.04 sec 134.04 run 3 5 min 36.14 sec 336.14 run 3 15.775 ms 0.015775 Average time 137.1533333 Average time 337.2766667 Average time 0.018765667
  • 24. Comparison Results Queries MySQL (InnoDB) MySQL (MyISAM) Monetdb InnoDB/ MyISAM InnoDB/ Monetdb MyISAM/ Monetdb Average time Average time Average time Query 1 26.157 25.977 0.0121 0.0069 times faster than InnoDB 2,168 times faster than InnoDB 2,153 times faster than MyISAM Query 2 645.317 652.297 0.0201 0.0108 times slower than InnoDB 32,015 times faster than InnoDB 32,361 times faster than MyISAM Query 3 853.25 2086.01 0.0147 2.445 times slower than InnoDB 57,915 times faster than InnoDB 141,591 times faster than MyISAM Query 4 0.0483 0.293 0.0113 1.648 times faster than InnoDB 43 times faster than InnoDB 26 times faster than MyISAM Query 5 137.153 337.277 0.0188 2.459 times slower than InnoDB 7309 times faster than InnoDB 17,973 times faster than MyISAM
  • 25. Challenges Encountered • Throughout this project I encountered several challenges: • The first difficulty encountered was the installation of the several programs used for this project • Once all programs were installed the next challenge was the uploading of the data to the databases • After all data was loaded into the database tables, one of the largest challenges was to examine each query and fill in areas that needed exact information from the tables for the query to even run • The largest challenge I faced through this entire project was learning to use the command line to do everything as I have not had much experience with this.
  • 26. Summary • A relational database is a digital database that organizes data into one or more tables of columns and rows. • Column-store databases store data as columns rather than rows • TPC-H is a decision support benchmark that examine large volumes of data, execute queries with a high degree of complexity, and give answers to critical business questions • As the data shows from the tests conducted on the two different databases, column-store databases such as Monetdb are considerably faster in run time compared to traditional relational databases such as MySQL.
  • 27. ALL Results and Test Queries can be found at: https://github.com/Djaikins/MySQL-vs-Monetdb or By searching djaikins on Github