MapReduce and DBMS Hybrids

  • 533 views
Uploaded on

Workshop conducted for Teradata, Islamabad

Workshop conducted for Teradata, Islamabad

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
533
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
17
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. 12: MapReduce and DBMS HybridsZubair Nabizubair.nabi@itu.edu.pkMay 26, 2013Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 1 / 37
  • 2. Outline1 Hive2 HadoopDB3 nCluster4 SummaryZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 2 / 37
  • 3. Outline1 Hive2 HadoopDB3 nCluster4 SummaryZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 3 / 37
  • 4. IntroductionData warehousing solution built atop Hadoop by Facebook1https://www.facebook.com/note.php?note_id=89508453919Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 4 / 37
  • 5. IntroductionData warehousing solution built atop Hadoop by FacebookNow an Apache open source project1https://www.facebook.com/note.php?note_id=89508453919Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 4 / 37
  • 6. IntroductionData warehousing solution built atop Hadoop by FacebookNow an Apache open source projectQueries are expressed in SQL-like HiveQL, which are compiled intomap-reduce jobs1https://www.facebook.com/note.php?note_id=89508453919Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 4 / 37
  • 7. IntroductionData warehousing solution built atop Hadoop by FacebookNow an Apache open source projectQueries are expressed in SQL-like HiveQL, which are compiled intomap-reduce jobsAlso contains a type system for describing RDBMS-like tables1https://www.facebook.com/note.php?note_id=89508453919Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 4 / 37
  • 8. IntroductionData warehousing solution built atop Hadoop by FacebookNow an Apache open source projectQueries are expressed in SQL-like HiveQL, which are compiled intomap-reduce jobsAlso contains a type system for describing RDBMS-like tablesA system catalog, Hive-Metastore, which contains schemas andstatistics is used for data exploration and query optimization1https://www.facebook.com/note.php?note_id=89508453919Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 4 / 37
  • 9. IntroductionData warehousing solution built atop Hadoop by FacebookNow an Apache open source projectQueries are expressed in SQL-like HiveQL, which are compiled intomap-reduce jobsAlso contains a type system for describing RDBMS-like tablesA system catalog, Hive-Metastore, which contains schemas andstatistics is used for data exploration and query optimizationStores 2PB of uncompressed data at Facebook and is heavily used forsimple summarization, business intelligence, machine learning, amongmany other applications11https://www.facebook.com/note.php?note_id=89508453919Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 4 / 37
  • 10. IntroductionData warehousing solution built atop Hadoop by FacebookNow an Apache open source projectQueries are expressed in SQL-like HiveQL, which are compiled intomap-reduce jobsAlso contains a type system for describing RDBMS-like tablesA system catalog, Hive-Metastore, which contains schemas andstatistics is used for data exploration and query optimizationStores 2PB of uncompressed data at Facebook and is heavily used forsimple summarization, business intelligence, machine learning, amongmany other applications1Also used by Digg, Grooveshark, hi5, Last.fm, Scribd, etc.1https://www.facebook.com/note.php?note_id=89508453919Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 4 / 37
  • 11. Data ModelTables:Similar to RDBMS tablesZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 5 / 37
  • 12. Data ModelTables:Similar to RDBMS tablesEach table has a corresponding HDFS directoryZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 5 / 37
  • 13. Data ModelTables:Similar to RDBMS tablesEach table has a corresponding HDFS directoryThe contents of the table are serialized and stored in files within thatdirectoryZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 5 / 37
  • 14. Data ModelTables:Similar to RDBMS tablesEach table has a corresponding HDFS directoryThe contents of the table are serialized and stored in files within thatdirectorySerialization can be both system provided or user definedZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 5 / 37
  • 15. Data ModelTables:Similar to RDBMS tablesEach table has a corresponding HDFS directoryThe contents of the table are serialized and stored in files within thatdirectorySerialization can be both system provided or user definedSerialization information of each table is also stored in theHive-Metastore for query optimizationZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 5 / 37
  • 16. Data ModelTables:Similar to RDBMS tablesEach table has a corresponding HDFS directoryThe contents of the table are serialized and stored in files within thatdirectorySerialization can be both system provided or user definedSerialization information of each table is also stored in theHive-Metastore for query optimizationTables can also be defined for data stored in external sources such asHDFS, NFS, and local FSZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 5 / 37
  • 17. Data Model (2)Partitions:Determine the distribution of data within sub-directories of the maintable directoryZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 6 / 37
  • 18. Data Model (2)Partitions:Determine the distribution of data within sub-directories of the maintable directoryFor instance, for a table T stored in /wh/T and partitioned on columnsds and ctryZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 6 / 37
  • 19. Data Model (2)Partitions:Determine the distribution of data within sub-directories of the maintable directoryFor instance, for a table T stored in /wh/T and partitioned on columnsds and ctryData with ds value 20090101 and ctry value US,Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 6 / 37
  • 20. Data Model (2)Partitions:Determine the distribution of data within sub-directories of the maintable directoryFor instance, for a table T stored in /wh/T and partitioned on columnsds and ctryData with ds value 20090101 and ctry value US,Will be stored in files within /wh/T/ds=20090101/ctry=USZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 6 / 37
  • 21. Data Model (2)Partitions:Determine the distribution of data within sub-directories of the maintable directoryFor instance, for a table T stored in /wh/T and partitioned on columnsds and ctryData with ds value 20090101 and ctry value US,Will be stored in files within /wh/T/ds=20090101/ctry=USBuckets:Data within partitions is divided into bucketsZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 6 / 37
  • 22. Data Model (2)Partitions:Determine the distribution of data within sub-directories of the maintable directoryFor instance, for a table T stored in /wh/T and partitioned on columnsds and ctryData with ds value 20090101 and ctry value US,Will be stored in files within /wh/T/ds=20090101/ctry=USBuckets:Data within partitions is divided into bucketsBuckets are calculated based on the hash of a column within thepartitionZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 6 / 37
  • 23. Data Model (2)Partitions:Determine the distribution of data within sub-directories of the maintable directoryFor instance, for a table T stored in /wh/T and partitioned on columnsds and ctryData with ds value 20090101 and ctry value US,Will be stored in files within /wh/T/ds=20090101/ctry=USBuckets:Data within partitions is divided into bucketsBuckets are calculated based on the hash of a column within thepartitionEach bucket is stored within a file in the partition directoryZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 6 / 37
  • 24. Column Data TypesPrimitive types: integers, floats, strings, dates, and booleansZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 7 / 37
  • 25. Column Data TypesPrimitive types: integers, floats, strings, dates, and booleansNestable collection types: arrays and mapsZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 7 / 37
  • 26. Column Data TypesPrimitive types: integers, floats, strings, dates, and booleansNestable collection types: arrays and mapsCustom types: user-definedZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 7 / 37
  • 27. HiveQLSupports select, project, join, aggregate, union all, and sub-queriesZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 8 / 37
  • 28. HiveQLSupports select, project, join, aggregate, union all, and sub-queriesTables are created using data definition statements with specificserialization formats, partitioning, and bucketingZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 8 / 37
  • 29. HiveQLSupports select, project, join, aggregate, union all, and sub-queriesTables are created using data definition statements with specificserialization formats, partitioning, and bucketingData is loaded from external sources and inserted into tablesZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 8 / 37
  • 30. HiveQLSupports select, project, join, aggregate, union all, and sub-queriesTables are created using data definition statements with specificserialization formats, partitioning, and bucketingData is loaded from external sources and inserted into tablesSupport for multi-table insert – multiple queries on the same input datausing a single HiveQL statementZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 8 / 37
  • 31. HiveQLSupports select, project, join, aggregate, union all, and sub-queriesTables are created using data definition statements with specificserialization formats, partitioning, and bucketingData is loaded from external sources and inserted into tablesSupport for multi-table insert – multiple queries on the same input datausing a single HiveQL statementUser-defined column transformation and aggregation functions in JavaZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 8 / 37
  • 32. HiveQLSupports select, project, join, aggregate, union all, and sub-queriesTables are created using data definition statements with specificserialization formats, partitioning, and bucketingData is loaded from external sources and inserted into tablesSupport for multi-table insert – multiple queries on the same input datausing a single HiveQL statementUser-defined column transformation and aggregation functions in JavaCustom map-reduce scripts written in any language can be embeddedZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 8 / 37
  • 33. Example: Facebook StatusStatus updates are stored on flat files in an NFS directory/logs/status_updatesZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 9 / 37
  • 34. Example: Facebook StatusStatus updates are stored on flat files in an NFS directory/logs/status_updatesThis data is loaded on a daily basis to a Hive table:status_updates(userid int,status string,dsstring)Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 9 / 37
  • 35. Example: Facebook StatusStatus updates are stored on flat files in an NFS directory/logs/status_updatesThis data is loaded on a daily basis to a Hive table:status_updates(userid int,status string,dsstring)Using:1 LOAD DATA LOCAL INPATH ’/logs/status_updates’2 INTO TABLE status_updates PARTITION (ds=’2013-05-26’)Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 9 / 37
  • 36. Example: Facebook StatusStatus updates are stored on flat files in an NFS directory/logs/status_updatesThis data is loaded on a daily basis to a Hive table:status_updates(userid int,status string,dsstring)Using:1 LOAD DATA LOCAL INPATH ’/logs/status_updates’2 INTO TABLE status_updates PARTITION (ds=’2013-05-26’)Detailed profile information, such as gender and academic institution ispresent in the table: profiles(userid int,schoolstring,gender int)Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 9 / 37
  • 37. Example: Facebook Status (2)Query to workout the frequency of status updates based on gender andacademic institutionZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 10 / 37
  • 38. Example: Facebook Status (2)Query to workout the frequency of status updates based on gender andacademic institution1 FROM (SELECT a.status, b.school, b.gender2 FROM status_updates a JOIN profiles b3 ON (a.userid = b.userid and4 a.ds=’2013-05-26’)5 ) subq16 INSERT OVERWRITE TABLE gender_summary7 PARTITION(ds=’2013-05-26’)8 SELECT subq1.gender, COUNT(1) GROUP BY subq1.gender9 INSERT OVERWRITE TABLE school_summary10 PARTITION(ds=’2013-05-26’)11 SELECT subq1.school, COUNT(1) GROUP BY subq1.schoolZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 10 / 37
  • 39. MetastoreSimilar to the metastore maintained by traditional warehousingsolutions such as Oracle and IBM DB2 (distinguishes Hive from Pig orCascading which have no such store)Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 11 / 37
  • 40. MetastoreSimilar to the metastore maintained by traditional warehousingsolutions such as Oracle and IBM DB2 (distinguishes Hive from Pig orCascading which have no such store)Stored in either a traditional DB such as MySQL or an FS such as NFSZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 11 / 37
  • 41. MetastoreSimilar to the metastore maintained by traditional warehousingsolutions such as Oracle and IBM DB2 (distinguishes Hive from Pig orCascading which have no such store)Stored in either a traditional DB such as MySQL or an FS such as NFSContains the following objects:Database: namespace for tablesZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 11 / 37
  • 42. MetastoreSimilar to the metastore maintained by traditional warehousingsolutions such as Oracle and IBM DB2 (distinguishes Hive from Pig orCascading which have no such store)Stored in either a traditional DB such as MySQL or an FS such as NFSContains the following objects:Database: namespace for tablesTable: metadata for a table including columns and their types, owner,storage, and serialization informationZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 11 / 37
  • 43. MetastoreSimilar to the metastore maintained by traditional warehousingsolutions such as Oracle and IBM DB2 (distinguishes Hive from Pig orCascading which have no such store)Stored in either a traditional DB such as MySQL or an FS such as NFSContains the following objects:Database: namespace for tablesTable: metadata for a table including columns and their types, owner,storage, and serialization informationPartition: metadata for a partition; similar to the information for a tableZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 11 / 37
  • 44. Outline1 Hive2 HadoopDB3 nCluster4 SummaryZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 12 / 37
  • 45. IntroductionTwo options for data analytics on shared nothing clusters:1 Parallel Databases, such as Teradata, Oracle etc. but,Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 13 / 37
  • 46. IntroductionTwo options for data analytics on shared nothing clusters:1 Parallel Databases, such as Teradata, Oracle etc. but,Assume that failures are a rare eventZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 13 / 37
  • 47. IntroductionTwo options for data analytics on shared nothing clusters:1 Parallel Databases, such as Teradata, Oracle etc. but,Assume that failures are a rare eventAssume that hardware is homogeneousZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 13 / 37
  • 48. IntroductionTwo options for data analytics on shared nothing clusters:1 Parallel Databases, such as Teradata, Oracle etc. but,Assume that failures are a rare eventAssume that hardware is homogeneousNever tested in deployments with more than a few dozen nodesZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 13 / 37
  • 49. IntroductionTwo options for data analytics on shared nothing clusters:1 Parallel Databases, such as Teradata, Oracle etc. but,Assume that failures are a rare eventAssume that hardware is homogeneousNever tested in deployments with more than a few dozen nodes2 MapReduce but,Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 13 / 37
  • 50. IntroductionTwo options for data analytics on shared nothing clusters:1 Parallel Databases, such as Teradata, Oracle etc. but,Assume that failures are a rare eventAssume that hardware is homogeneousNever tested in deployments with more than a few dozen nodes2 MapReduce but,All shortcomings pointed by DeWitt and Stonebraker, as discussedbeforeZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 13 / 37
  • 51. IntroductionTwo options for data analytics on shared nothing clusters:1 Parallel Databases, such as Teradata, Oracle etc. but,Assume that failures are a rare eventAssume that hardware is homogeneousNever tested in deployments with more than a few dozen nodes2 MapReduce but,All shortcomings pointed by DeWitt and Stonebraker, as discussedbeforeAt times an order of magnitude slower than parallel DBsZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 13 / 37
  • 52. HybridCombine scalability and non-existent monetary cost of MapReducewith performance of parallel DBs2http://hadapt.com/Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 14 / 37
  • 53. HybridCombine scalability and non-existent monetary cost of MapReducewith performance of parallel DBsHadoopDB is such a hybrid2http://hadapt.com/Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 14 / 37
  • 54. HybridCombine scalability and non-existent monetary cost of MapReducewith performance of parallel DBsHadoopDB is such a hybridUnlike Hive, Pig, Greenplum, Aster, etc. which are language andinterface level hybrids, Hadoop DB is a systems level hybrid2http://hadapt.com/Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 14 / 37
  • 55. HybridCombine scalability and non-existent monetary cost of MapReducewith performance of parallel DBsHadoopDB is such a hybridUnlike Hive, Pig, Greenplum, Aster, etc. which are language andinterface level hybrids, Hadoop DB is a systems level hybridUses MapReduce as the communication layer atop a cluster of nodesrunning single-node DBMS instances2http://hadapt.com/Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 14 / 37
  • 56. HybridCombine scalability and non-existent monetary cost of MapReducewith performance of parallel DBsHadoopDB is such a hybridUnlike Hive, Pig, Greenplum, Aster, etc. which are language andinterface level hybrids, Hadoop DB is a systems level hybridUses MapReduce as the communication layer atop a cluster of nodesrunning single-node DBMS instancesPostgreSQL as the database layer, Hadoop as the communicationlayer, and Hive as the translation layer2http://hadapt.com/Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 14 / 37
  • 57. HybridCombine scalability and non-existent monetary cost of MapReducewith performance of parallel DBsHadoopDB is such a hybridUnlike Hive, Pig, Greenplum, Aster, etc. which are language andinterface level hybrids, Hadoop DB is a systems level hybridUses MapReduce as the communication layer atop a cluster of nodesrunning single-node DBMS instancesPostgreSQL as the database layer, Hadoop as the communicationlayer, and Hive as the translation layerCommercialized through the start up, Hadapt22http://hadapt.com/Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 14 / 37
  • 58. HadoopDBConsists of four components:1 Database Connector: Interface between per-node database systemsand Hadoop TaskTrackersZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 15 / 37
  • 59. HadoopDBConsists of four components:1 Database Connector: Interface between per-node database systemsand Hadoop TaskTrackers2 Catalog: Meta-information about per-node databasesZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 15 / 37
  • 60. HadoopDBConsists of four components:1 Database Connector: Interface between per-node database systemsand Hadoop TaskTrackers2 Catalog: Meta-information about per-node databases3 Data Loader: Data partitioning across single-node databasesZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 15 / 37
  • 61. HadoopDBConsists of four components:1 Database Connector: Interface between per-node database systemsand Hadoop TaskTrackers2 Catalog: Meta-information about per-node databases3 Data Loader: Data partitioning across single-node databases4 SQL to MapReduce to SQL (SMS) Planner: Translation betweenSQL and MapReduceZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 15 / 37
  • 62. HadoopDB ArchitectureZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 16 / 37
  • 63. Database ConnectorUses the Java Database Connectivity (JDBC)-compliant HadoopInputFormatZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 17 / 37
  • 64. Database ConnectorUses the Java Database Connectivity (JDBC)-compliant HadoopInputFormatThe connector is served the SQL query and other information by theMapReduce jobZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 17 / 37
  • 65. Database ConnectorUses the Java Database Connectivity (JDBC)-compliant HadoopInputFormatThe connector is served the SQL query and other information by theMapReduce jobThe connector connects to the DB, executes the SQL query, andreturns results in the form of key/value pairsZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 17 / 37
  • 66. Database ConnectorUses the Java Database Connectivity (JDBC)-compliant HadoopInputFormatThe connector is served the SQL query and other information by theMapReduce jobThe connector connects to the DB, executes the SQL query, andreturns results in the form of key/value pairsHadoop in essence sees the DB as just another data sourceZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 17 / 37
  • 67. CatalogContains information, such as:1 Connection parameters, such as DB location, format, and anycredentialsZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 18 / 37
  • 68. CatalogContains information, such as:1 Connection parameters, such as DB location, format, and anycredentials2 Metadata about the datasets, replica locations, and partitioning schemeZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 18 / 37
  • 69. CatalogContains information, such as:1 Connection parameters, such as DB location, format, and anycredentials2 Metadata about the datasets, replica locations, and partitioning schemeStored as an XML file on the HDFSZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 18 / 37
  • 70. Data LoaderConsists of two key components:1 Global Hasher: Executes a custom Hadoop job to repartition raw datafiles from the HDFS into n parts, where n is the number of nodes in theclusterZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 19 / 37
  • 71. Data LoaderConsists of two key components:1 Global Hasher: Executes a custom Hadoop job to repartition raw datafiles from the HDFS into n parts, where n is the number of nodes in thecluster2 Local Hasher: Copies a partition from the HDFS to the node-local DBof each node and further partitions it into smaller size chunksZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 19 / 37
  • 72. SQL to MapReduce to SQL (SMS) PlannerExtends HiveQL in two key ways:1 Before query execution, the Hive Metastore is updated with referencesto HadoopDB tables, table schemas, formats, and serializationinformationZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 20 / 37
  • 73. SQL to MapReduce to SQL (SMS) PlannerExtends HiveQL in two key ways:1 Before query execution, the Hive Metastore is updated with referencesto HadoopDB tables, table schemas, formats, and serializationinformation2 All operators with partitioning keys similar to the node-local databaseare converted into SQL queries and pushed to the database layerZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 20 / 37
  • 74. Outline1 Hive2 HadoopDB3 nCluster4 SummaryZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 21 / 37
  • 75. IntroductionThe declarative nature of SQL is too limiting for describing most bigdata computationZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 22 / 37
  • 76. IntroductionThe declarative nature of SQL is too limiting for describing most bigdata computationThe underlying subsystems are also suboptimal as they do notconsider domain-specific optimizationsZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 22 / 37
  • 77. IntroductionThe declarative nature of SQL is too limiting for describing most bigdata computationThe underlying subsystems are also suboptimal as they do notconsider domain-specific optimizationsnCluster makes use of SQL/MR, a framework that inserts user-definedfunctions in any programming language into SQL queriesZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 22 / 37
  • 78. IntroductionThe declarative nature of SQL is too limiting for describing most bigdata computationThe underlying subsystems are also suboptimal as they do notconsider domain-specific optimizationsnCluster makes use of SQL/MR, a framework that inserts user-definedfunctions in any programming language into SQL queriesBy itself, nCluster is a shared-nothing parallel database gearedtowards analytic workloadsZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 22 / 37
  • 79. IntroductionThe declarative nature of SQL is too limiting for describing most bigdata computationThe underlying subsystems are also suboptimal as they do notconsider domain-specific optimizationsnCluster makes use of SQL/MR, a framework that inserts user-definedfunctions in any programming language into SQL queriesBy itself, nCluster is a shared-nothing parallel database gearedtowards analytic workloadsOriginally designed by Aster Data Systems and later acquired byTeradataZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 22 / 37
  • 80. IntroductionThe declarative nature of SQL is too limiting for describing most bigdata computationThe underlying subsystems are also suboptimal as they do notconsider domain-specific optimizationsnCluster makes use of SQL/MR, a framework that inserts user-definedfunctions in any programming language into SQL queriesBy itself, nCluster is a shared-nothing parallel database gearedtowards analytic workloadsOriginally designed by Aster Data Systems and later acquired byTeradataUsed by Barnes and Noble, LinkedIn, SAS, etc.Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 22 / 37
  • 81. SQL/MR FunctionsDynamically polymorphic: input and output schemes are decided atruntimeZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 23 / 37
  • 82. SQL/MR FunctionsDynamically polymorphic: input and output schemes are decided atruntimeParallelizable across cores and machinesZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 23 / 37
  • 83. SQL/MR FunctionsDynamically polymorphic: input and output schemes are decided atruntimeParallelizable across cores and machinesComposable because their input and output behaviour is identical toSQL subqueriesZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 23 / 37
  • 84. SQL/MR FunctionsDynamically polymorphic: input and output schemes are decided atruntimeParallelizable across cores and machinesComposable because their input and output behaviour is identical toSQL subqueriesAmenable to static and dynamic optimizations just like SQL subqueriesor a relationZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 23 / 37
  • 85. SQL/MR FunctionsDynamically polymorphic: input and output schemes are decided atruntimeParallelizable across cores and machinesComposable because their input and output behaviour is identical toSQL subqueriesAmenable to static and dynamic optimizations just like SQL subqueriesor a relationCan be implemented in a number of languages including Java, C#,C++, Python, etc. and can thus make use of third-party librariesZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 23 / 37
  • 86. SQL/MR FunctionsDynamically polymorphic: input and output schemes are decided atruntimeParallelizable across cores and machinesComposable because their input and output behaviour is identical toSQL subqueriesAmenable to static and dynamic optimizations just like SQL subqueriesor a relationCan be implemented in a number of languages including Java, C#,C++, Python, etc. and can thus make use of third-party librariesExecuted within processes to provide sandboxing and resourceallocationZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 23 / 37
  • 87. Syntax1 SELECT ...2 FROM functionname(3 ON table-or-query4 [PARTITION BY expr, ...]5 [ORDER BY expr, ...]6 [clausename(arg, ...) ...]7 )8 ...SQL/MR function appears in the FROM clauseZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 24 / 37
  • 88. Syntax1 SELECT ...2 FROM functionname(3 ON table-or-query4 [PARTITION BY expr, ...]5 [ORDER BY expr, ...]6 [clausename(arg, ...) ...]7 )8 ...SQL/MR function appears in the FROM clauseON is the only required clause which specifies the input to the functionZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 24 / 37
  • 89. Syntax1 SELECT ...2 FROM functionname(3 ON table-or-query4 [PARTITION BY expr, ...]5 [ORDER BY expr, ...]6 [clausename(arg, ...) ...]7 )8 ...SQL/MR function appears in the FROM clauseON is the only required clause which specifies the input to the functionPARTITION BY partitions the input to the function on one or moreattributes from the schemaZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 24 / 37
  • 90. Syntax (2)1 SELECT ...2 FROM functionname(3 ON table-or-query4 [PARTITION BY expr, ...]5 [ORDER BY expr, ...]6 [clausename(arg, ...) ...]7 )8 ...ORDER BY sorts the input to the function and can only be used after aPARTITION BY clauseZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 25 / 37
  • 91. Syntax (2)1 SELECT ...2 FROM functionname(3 ON table-or-query4 [PARTITION BY expr, ...]5 [ORDER BY expr, ...]6 [clausename(arg, ...) ...]7 )8 ...ORDER BY sorts the input to the function and can only be used after aPARTITION BY clauseAny number of custom clauses can also be defined whose names andarguments are passed as a key/value map to the functionZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 25 / 37
  • 92. Syntax (2)1 SELECT ...2 FROM functionname(3 ON table-or-query4 [PARTITION BY expr, ...]5 [ORDER BY expr, ...]6 [clausename(arg, ...) ...]7 )8 ...ORDER BY sorts the input to the function and can only be used after aPARTITION BY clauseAny number of custom clauses can also be defined whose names andarguments are passed as a key/value map to the functionImplemented as relations so easily nestableZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 25 / 37
  • 93. Execution ModelFunctions are equivalent to either map (row function) or reduce(partition function) functionsZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 26 / 37
  • 94. Execution ModelFunctions are equivalent to either map (row function) or reduce(partition function) functionsIdentical to MapReduce, these functions are executed across manynodes and machinesZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 26 / 37
  • 95. Execution ModelFunctions are equivalent to either map (row function) or reduce(partition function) functionsIdentical to MapReduce, these functions are executed across manynodes and machinesContracts identical to MapReduce functionsOnly one row function operates over a row from the input tableZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 26 / 37
  • 96. Execution ModelFunctions are equivalent to either map (row function) or reduce(partition function) functionsIdentical to MapReduce, these functions are executed across manynodes and machinesContracts identical to MapReduce functionsOnly one row function operates over a row from the input tableOnly one partition function operates over a group of rows defined by thePARTITION BY clause, in the order specified by the ORDER BYclauseZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 26 / 37
  • 97. Programming InterfaceA Runtime Contract is passed by the query planner to thefunction which contains the names and types of the input columns andthe names and values of the argument clausesZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 27 / 37
  • 98. Programming InterfaceA Runtime Contract is passed by the query planner to thefunction which contains the names and types of the input columns andthe names and values of the argument clausesThe function then completes this contract by filling in the outputschema and making a call to complete()Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 27 / 37
  • 99. Programming InterfaceA Runtime Contract is passed by the query planner to thefunction which contains the names and types of the input columns andthe names and values of the argument clausesThe function then completes this contract by filling in the outputschema and making a call to complete()Row and partition functions are implemented through theoperateOnSomeRows and operateOnPartition methods,respectivelyZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 27 / 37
  • 100. Programming InterfaceA Runtime Contract is passed by the query planner to thefunction which contains the names and types of the input columns andthe names and values of the argument clausesThe function then completes this contract by filling in the outputschema and making a call to complete()Row and partition functions are implemented through theoperateOnSomeRows and operateOnPartition methods,respectivelyThese methods are passed an iterator over their input rows and anemitter object for returning output rows to the databaseZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 27 / 37
  • 101. Programming InterfaceA Runtime Contract is passed by the query planner to thefunction which contains the names and types of the input columns andthe names and values of the argument clausesThe function then completes this contract by filling in the outputschema and making a call to complete()Row and partition functions are implemented through theoperateOnSomeRows and operateOnPartition methods,respectivelyThese methods are passed an iterator over their input rows and anemitter object for returning output rows to the databaseoperateOnPartition can also optionally implement the combinerinterfaceZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 27 / 37
  • 102. InstallationFunctions need to be installed first before they can be usedZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 28 / 37
  • 103. InstallationFunctions need to be installed first before they can be usedCan be supplied as a .zip along with third-party librariesZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 28 / 37
  • 104. InstallationFunctions need to be installed first before they can be usedCan be supplied as a .zip along with third-party librariesInstall-time examination also enables static analysis of properties, suchas row function or partition function, support for combining, etc.Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 28 / 37
  • 105. InstallationFunctions need to be installed first before they can be usedCan be supplied as a .zip along with third-party librariesInstall-time examination also enables static analysis of properties, suchas row function or partition function, support for combining, etc.Any arbitrary file can be installed which is replicated to all workers,such as configuration files, binaries, etc.Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 28 / 37
  • 106. InstallationFunctions need to be installed first before they can be usedCan be supplied as a .zip along with third-party librariesInstall-time examination also enables static analysis of properties, suchas row function or partition function, support for combining, etc.Any arbitrary file can be installed which is replicated to all workers,such as configuration files, binaries, etc.Each function is provided with a temporary directory which is garbagecollected after executionZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 28 / 37
  • 107. ArchitectureOne or more Queen nodes process queries and hash partition themacross Worker nodesZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 29 / 37
  • 108. ArchitectureOne or more Queen nodes process queries and hash partition themacross Worker nodesThe query planner honours the Runtime Contract with thefunction and invokes its initializer (Constructor in case of Java)Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 29 / 37
  • 109. ArchitectureOne or more Queen nodes process queries and hash partition themacross Worker nodesThe query planner honours the Runtime Contract with thefunction and invokes its initializer (Constructor in case of Java)Functions are executed within the Worker databases as separateprocesses for isolation, security, resource allocation, forcedtermination, etc.Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 29 / 37
  • 110. ArchitectureOne or more Queen nodes process queries and hash partition themacross Worker nodesThe query planner honours the Runtime Contract with thefunction and invokes its initializer (Constructor in case of Java)Functions are executed within the Worker databases as separateprocesses for isolation, security, resource allocation, forcedtermination, etc.The worker database implements a “bridge” which manages itscommunication with the SQL/MR functionZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 29 / 37
  • 111. ArchitectureOne or more Queen nodes process queries and hash partition themacross Worker nodesThe query planner honours the Runtime Contract with thefunction and invokes its initializer (Constructor in case of Java)Functions are executed within the Worker databases as separateprocesses for isolation, security, resource allocation, forcedtermination, etc.The worker database implements a “bridge” which manages itscommunication with the SQL/MR functionThe SQL/MR function process contains a “runner” which manages itscommunication with the worker databaseZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 29 / 37
  • 112. Architecture (2)Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 30 / 37
  • 113. Example: Wordcount1 SELECT token, COUNT(*)2 FROM tokenizer(3 ON input-table4 DELIMITER(’ ’)5 )6 GROUP BY token;Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 31 / 37
  • 114. Example: Clickstream SessionizationDivide a user’s clicks on a website into sessionsZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 32 / 37
  • 115. Example: Clickstream SessionizationDivide a user’s clicks on a website into sessionsA session includes the user’s clicks within a specified time periodZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 32 / 37
  • 116. Example: Clickstream SessionizationDivide a user’s clicks on a website into sessionsA session includes the user’s clicks within a specified time periodTimestamp User ID10:00:00 23890900:58:24 765610:00:24 23890902:30:33 765610:01:23 23890910:02:40 238909Timestamp User ID Session ID10:00:00 238909 010:00:24 238909 010:01:23 238909 010:02:40 238909 100:58:24 7656 002:30:33 7656 1Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 32 / 37
  • 117. Example: Clickstream Sessionization (2)1 SELECT ts, userid, session2 FROM sessionize (3 ON clicks4 PARTITION BY userid5 ORDER BY ts6 TIMECOLUMN (’ts’)7 TIMEOUT (60)8 );Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 33 / 37
  • 118. Example: Clickstream Sessionization (3)1 public class Sessionize implements PartitionFunction {23 private int timeColumnIndex;4 private int timeout;56 public Sessionize(RuntimeContract contract) {7 // Get time column and timeout from contract8 // Define output schema9 contract.complete();10 }1112 public void operationOnPartition(13 PartitionDefinition partition,14 RowIterator inputIterator,15 RowEmitter outputEmitter) {16 // Implement the partition function logic17 // Emit output rows18 }1920 }Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 34 / 37
  • 119. Outline1 Hive2 HadoopDB3 nCluster4 SummaryZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 35 / 37
  • 120. SummaryHive, HadoopDB, and nCluster explore three different points in the designspaceZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 36 / 37
  • 121. SummaryHive, HadoopDB, and nCluster explore three different points in the designspace1 Hive uses MapReduce to give DBMS-like functionalityZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 36 / 37
  • 122. SummaryHive, HadoopDB, and nCluster explore three different points in the designspace1 Hive uses MapReduce to give DBMS-like functionality2 HadoopDB uses MapReduce and DBMS side-by-sideZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 36 / 37
  • 123. SummaryHive, HadoopDB, and nCluster explore three different points in the designspace1 Hive uses MapReduce to give DBMS-like functionality2 HadoopDB uses MapReduce and DBMS side-by-side3 nCluster implements MapReduce within a DBMSZubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 36 / 37
  • 124. References1 Ashish Thusoo, Joydeep Sen Sarma, Namit Jain, Zheng Shao, PrasadChakka, Suresh Anthony, Hao Liu, Pete Wyckoff, and RaghothamMurthy. 2009. Hive: a warehousing solution over a map-reduceframework. Proc. VLDB Endow. 2, 2 (August 2009), 1626-1629.2 Azza Abouzeid, Kamil Bajda-Pawlikowski, Daniel Abadi, AviSilberschatz, and Alexander Rasin. 2009. HadoopDB: an architecturalhybrid of MapReduce and DBMS technologies for analytical workloads.Proc. VLDB Endow. 2, 1 (August 2009), 922-933.3 Eric Friedman, Peter Pawlowski, and John Cieslewicz. 2009.SQL/MapReduce: a practical approach to self-describing, polymorphic,and parallelizable user-defined functions. Proc. VLDB Endow. 2, 2(August 2009), 1402-1413.Zubair Nabi 12: MapReduce and DBMS Hybrids May 26, 2013 37 / 37