Alexey Grishchenko, profile picture
Uploaded byAlexey Grishchenko
85,884 views

Apache HAWQ Architecture

The document presents an overview of HAWQ, an analytical SQL-on-Hadoop engine, detailing its architecture, components, and the reasons for its necessity, such as providing a SQL interface for BI solutions compliant with ANSI SQL standards. It highlights the origins of HAWQ, its features including performance metrics, metadata structures, and resource management options, along with its evolution within the open-source community since its inception. Additionally, the document discusses storage formats, query execution, and overall design considerations for effective data processing.

Data & Analytics◦
Related topics:
Apache Spark•

Embed presentation

1 / 135
2 / 135
3 / 135
4 / 135
5 / 135
6 / 135
7 / 135
8 / 135
9 / 135
10 / 135
11 / 135
12 / 135
13 / 135
14 / 135
15 / 135
16 / 135
17 / 135
18 / 135
19 / 135
20 / 135
21 / 135
22 / 135
23 / 135
24 / 135
25 / 135
26 / 135
27 / 135
28 / 135
29 / 135
30 / 135
31 / 135
32 / 135
33 / 135
34 / 135
35 / 135
36 / 135
37 / 135
38 / 135
39 / 135
40 / 135
41 / 135
42 / 135
43 / 135
44 / 135
45 / 135
46 / 135
47 / 135
48 / 135
49 / 135
50 / 135
51 / 135
52 / 135
53 / 135
54 / 135
55 / 135
56 / 135
57 / 135
58 / 135
59 / 135
60 / 135
61 / 135
62 / 135
63 / 135
64 / 135
65 / 135
66 / 135
67 / 135
68 / 135
69 / 135
70 / 135
71 / 135
72 / 135
73 / 135
74 / 135
75 / 135
76 / 135
77 / 135
78 / 135
79 / 135
80 / 135
81 / 135
82 / 135
83 / 135
84 / 135
85 / 135
86 / 135
87 / 135
88 / 135
89 / 135
90 / 135
91 / 135
92 / 135
93 / 135
94 / 135
95 / 135
96 / 135
97 / 135
98 / 135
99 / 135
100 / 135
101 / 135
102 / 135
103 / 135
104 / 135
105 / 135
106 / 135
107 / 135
108 / 135
109 / 135
110 / 135
111 / 135
112 / 135
113 / 135
114 / 135
115 / 135
116 / 135
117 / 135
118 / 135
119 / 135
120 / 135
121 / 135
122 / 135
123 / 135
124 / 135
125 / 135
126 / 135
127 / 135
128 / 135
129 / 135
130 / 135
131 / 135
132 / 135
133 / 135
134 / 135
135 / 135
HAWQ Architecture Alexey Grishchenko
Who I am Enterprise Architect @ Pivotal • 7 years in data processing • 5 years of experience with MPP • 4 years with Hadoop • Using HAWQ since the first internal Beta • Responsible for designing most of the EMEA HAWQ and Greenplum implementations • Spark contributor • http://0x0fff.com
Agenda • What is HAWQ
Agenda • What is HAWQ • Why you need it
Agenda • What is HAWQ • Why you need it • HAWQ Components
Agenda • What is HAWQ • Why you need it • HAWQ Components • HAWQ Design
Agenda • What is HAWQ • Why you need it • HAWQ Components • HAWQ Design • Query execution example
Agenda • What is HAWQ • Why you need it • HAWQ Components • HAWQ Design • Query execution example • Competitive solutions
What is • Analytical SQL-on-Hadoop engine
What is • Analytical SQL-on-Hadoop engine • HAdoop With Queries
What is • Analytical SQL-on-Hadoop engine • HAdoop With Queries Postgres Greenplum HAWQ 2005 Fork Postgres 8.0.2
What is • Analytical SQL-on-Hadoop engine • HAdoop With Queries Postgres HAWQ 2005 Fork Postgres 8.0.2 2009 Rebase Postgres 8.2.15 Greenplum
What is • Analytical SQL-on-Hadoop engine • HAdoop With Queries Postgres HAWQ 2005 Fork Postgres 8.0.2 2009 Rebase Postgres 8.2.15 2011 Fork GPDB 4.2.0.0 Greenplum
What is • Analytical SQL-on-Hadoop engine • HAdoop With Queries Postgres HAWQ 2005 Fork Postgres 8.0.2 2009 Rebase Postgres 8.2.15 2011 Fork GPDB 4.2.0.0 2013 HAWQ 1.0.0.0 Greenplum
What is • Analytical SQL-on-Hadoop engine • HAdoop With Queries Postgres HAWQ 2005 Fork Postgres 8.0.2 2009 Rebase Postgres 8.2.15 2011 Fork GPDB 4.2.0.0 2013 HAWQ 1.0.0.0 HAWQ 2.0.0.0 Open Source 2015 Greenplum
HAWQ is … • 1’500’000 C and C++ lines of code
HAWQ is … • 1’500’000 C and C++ lines of code – 200’000 of them in headers only
HAWQ is … • 1’500’000 C and C++ lines of code – 200’000 of them in headers only • 180’000 Python LOC
HAWQ is … • 1’500’000 C and C++ lines of code – 200’000 of them in headers only • 180’000 Python LOC • 60’000 Java LOC
HAWQ is … • 1’500’000 C and C++ lines of code – 200’000 of them in headers only • 180’000 Python LOC • 60’000 Java LOC • 23’000 Makefile LOC
HAWQ is … • 1’500’000 C and C++ lines of code – 200’000 of them in headers only • 180’000 Python LOC • 60’000 Java LOC • 23’000 Makefile LOC • 7’000 Shell scripts LOC
HAWQ is … • 1’500’000 C and C++ lines of code – 200’000 of them in headers only • 180’000 Python LOC • 60’000 Java LOC • 23’000 Makefile LOC • 7’000 Shell scripts LOC • More than 50 enterprise customers
HAWQ is … • 1’500’000 C and C++ lines of code – 200’000 of them in headers only • 180’000 Python LOC • 60’000 Java LOC • 23’000 Makefile LOC • 7’000 Shell scripts LOC • More than 50 enterprise customers – More than 10 of them in EMEA
Apache HAWQ • Apache HAWQ (incubating) from 09’2015 – http://hawq.incubator.apache.org – https://github.com/apache/incubator-hawq • What’s in Open Source – Sources of HAWQ 2.0 alpha – HAWQ 2.0 beta is planned for 2015’Q4 – HAWQ 2.0 GA is planned for 2016’Q1 • Community is yet young – come and join!
Why do we need it?
Why do we need it? • SQL-interface for BI solutions to the Hadoop data complaint with ANSI SQL-92, -99, -2003
Why do we need it? • SQL-interface for BI solutions to the Hadoop data complaint with ANSI SQL-92, -99, -2003 – Example - 5000-line query with a number of window function generated by Cognos
Why do we need it? • SQL-interface for BI solutions to the Hadoop data complaint with ANSI SQL-92, -99, -2003 – Example - 5000-line query with a number of window function generated by Cognos • Universal tool for ad hoc analytics on top of Hadoop data
Why do we need it? • SQL-interface for BI solutions to the Hadoop data complaint with ANSI SQL-92, -99, -2003 – Example - 5000-line query with a number of window function generated by Cognos • Universal tool for ad hoc analytics on top of Hadoop data – Example - parse URL to extract protocol, host name, port, GET parameters
Why do we need it? • SQL-interface for BI solutions to the Hadoop data complaint with ANSI SQL-92, -99, -2003 – Example - 5000-line query with a number of window function generated by Cognos • Universal tool for ad hoc analytics on top of Hadoop data – Example - parse URL to extract protocol, host name, port, GET parameters • Good performance
Why do we need it? • SQL-interface for BI solutions to the Hadoop data complaint with ANSI SQL-92, -99, -2003 – Example - 5000-line query with a number of window function generated by Cognos • Universal tool for ad hoc analytics on top of Hadoop data – Example - parse URL to extract protocol, host name, port, GET parameters • Good performance – How many times the data would hit the HDD during a single Hive query?
HAWQ Cluster Server 1 SNameNode Server 4 ZK JM NameNode Server 3 ZK JM Server 2 ZK JM Server 6 Datanode Server N Datanode Server 5 Datanode interconnect …
HAWQ Cluster Server 1 SNameNode Server 4 ZK JM NameNode Server 3 ZK JM Server 2 ZK JM Server 6 Datanode Server N Datanode Server 5 Datanode YARN NM YARN NM YARN NM YARN RM YARN App Timeline interconnect …
HAWQ Cluster HAWQ Master Server 1 SNameNode Server 4 ZK JM NameNode Server 3 ZK JM HAWQ Standby Server 2 ZK JM HAWQ Segment Server 6 Datanode HAWQ Segment Server N Datanode HAWQ Segment Server 5 Datanode YARN NM YARN NM YARN NM YARN RM YARN App Timeline interconnect …
Master Servers Server 1 SNameNode Server 4 ZK JM NameNode Server 3 ZK JM Server 2 ZK JM HAWQ Segment Server 6 Datanode HAWQ Segment Server N Datanode HAWQ Segment Server 5 Datanode YARN NM YARN NM YARN NM YARN RM YARN App Timeline interconnect … HAWQ Master HAWQ Standby
Master Servers HAWQ Master Query Parser Query Optimizer Global Resource Manager Distributed Transactions Manager Query Dispatch Metadata Catalog HAWQ Standby Master Query Parser Query Optimizer Global Resource Manager Distributed Transactions Manager Query Dispatch Metadata Catalog WAL repl.
HAWQ Master HAWQ Standby Segments Server 1 SNameNode Server 4 ZK JM NameNode Server 3 ZK JM Server 2 ZK JM Server 6 Datanode Server N Datanode Server 5 Datanode YARN NM YARN NM YARN NM YARN RM YARN App Timeline interconnect HAWQ Segment HAWQ SegmentHAWQ Segment …
Segments HAWQ Segment Query Executor libhdfs3 PXF HDFS Datanode Local Filesystem Temporary Data Directory Logs YARN Node Manager
Metadata • HAWQ metadata structure is similar to Postgres catalog structure
Metadata • HAWQ metadata structure is similar to Postgres catalog structure • Statistics – Number of rows and pages in the table
Metadata • HAWQ metadata structure is similar to Postgres catalog structure • Statistics – Number of rows and pages in the table – Most common values for each field
Metadata • HAWQ metadata structure is similar to Postgres catalog structure • Statistics – Number of rows and pages in the table – Most common values for each field – Histogram of values distribution for each field
Metadata • HAWQ metadata structure is similar to Postgres catalog structure • Statistics – Number of rows and pages in the table – Most common values for each field – Histogram of values distribution for each field – Number of unique values in the field
Metadata • HAWQ metadata structure is similar to Postgres catalog structure • Statistics – Number of rows and pages in the table – Most common values for each field – Histogram of values distribution for each field – Number of unique values in the field – Number of null values in the field
Metadata • HAWQ metadata structure is similar to Postgres catalog structure • Statistics – Number of rows and pages in the table – Most common values for each field – Histogram of values distribution for each field – Number of unique values in the field – Number of null values in the field – Average width of the field in bytes
Statistics No Statistics How many rows would produce the join of two tables?
Statistics No Statistics How many rows would produce the join of two tables?  From 0 to infinity
Statistics No Statistics Row Count How many rows would produce the join of two tables?  From 0 to infinity How many rows would produce the join of two 1000- row tables?
Statistics No Statistics Row Count How many rows would produce the join of two tables?  From 0 to infinity How many rows would produce the join of two 1000- row tables?  From 0 to 1’000’000
Statistics No Statistics Row Count Histograms and MCV How many rows would produce the join of two tables?  From 0 to infinity How many rows would produce the join of two 1000- row tables?  From 0 to 1’000’000 How many rows would produce the join of two 1000- row tables, with known field cardinality, values distribution diagram, number of nulls, most common values?
Statistics No Statistics Row Count Histograms and MCV How many rows would produce the join of two tables?  From 0 to infinity How many rows would produce the join of two 1000- row tables?  From 0 to 1’000’000 How many rows would produce the join of two 1000- row tables, with known field cardinality, values distribution diagram, number of nulls, most common values?  ~ From 500 to 1’500
Metadata • Table structure information ID Name Num Price 1 Яблоко 10 50 2 Груша 20 80 3 Банан 40 40 4 Апельсин 25 50 5 Киви 5 120 6 Арбуз 20 30 7 Дыня 40 100 8 Ананас 35 90
Metadata • Table structure information – Distribution fields ID Name Num Price 1 Яблоко 10 50 2 Груша 20 80 3 Банан 40 40 4 Апельсин 25 50 5 Киви 5 120 6 Арбуз 20 30 7 Дыня 40 100 8 Ананас 35 90 hash(ID)
Metadata • Table structure information – Distribution fields – Number of hash buckets ID Name Num Price 1 Яблоко 10 50 2 Груша 20 80 3 Банан 40 40 4 Апельсин 25 50 5 Киви 5 120 6 Арбуз 20 30 7 Дыня 40 100 8 Ананас 35 90 hash(ID) ID Name Num Price 1 Яблоко 10 50 2 Груша 20 80 3 Банан 40 40 4 Апельсин 25 50 5 Киви 5 120 6 Арбуз 20 30 7 Дыня 40 100 8 Ананас 35 90
Metadata • Table structure information – Distribution fields – Number of hash buckets – Partitioning (hash, list, range) ID Name Num Price 1 Яблоко 10 50 2 Груша 20 80 3 Банан 40 40 4 Апельсин 25 50 5 Киви 5 120 6 Арбуз 20 30 7 Дыня 40 100 8 Ананас 35 90 hash(ID) ID Name Num Price 1 Яблоко 10 50 2 Груша 20 80 3 Банан 40 40 4 Апельсин 25 50 5 Киви 5 120 6 Арбуз 20 30 7 Дыня 40 100 8 Ананас 35 90
Metadata • Table structure information – Distribution fields – Number of hash buckets – Partitioning (hash, list, range) • General metadata – Users and groups
Metadata • Table structure information – Distribution fields – Number of hash buckets – Partitioning (hash, list, range) • General metadata – Users and groups – Access privileges
Metadata • Table structure information – Distribution fields – Number of hash buckets – Partitioning (hash, list, range) • General metadata – Users and groups – Access privileges • Stored procedures – PL/pgSQL, PL/Java, PL/Python, PL/Perl, PL/R
Query Optimizer • HAWQ uses cost-based query optimizers
Query Optimizer • HAWQ uses cost-based query optimizers • You have two options – Planner – evolved from the Postgres query optimizer – ORCA (Pivotal Query Optimizer) – developed specifically for HAWQ
Query Optimizer • HAWQ uses cost-based query optimizers • You have two options – Planner – evolved from the Postgres query optimizer – ORCA (Pivotal Query Optimizer) – developed specifically for HAWQ • Optimizer hints work just like in Postgres – Enable/disable specific operation – Change the cost estimations for basic actions
Storage Formats Which storage format is the most optimal?
Storage Formats Which storage format is the most optimal?  It depends on what you mean by “optimal”
Storage Formats Which storage format is the most optimal?  It depends on what you mean by “optimal” – Minimal CPU usage for reading and writing the data
Storage Formats Which storage format is the most optimal?  It depends on what you mean by “optimal” – Minimal CPU usage for reading and writing the data – Minimal disk space usage
Storage Formats Which storage format is the most optimal?  It depends on what you mean by “optimal” – Minimal CPU usage for reading and writing the data – Minimal disk space usage – Minimal time to retrieve record by key
Storage Formats Which storage format is the most optimal?  It depends on what you mean by “optimal” – Minimal CPU usage for reading and writing the data – Minimal disk space usage – Minimal time to retrieve record by key – Minimal time to retrieve subset of columns – etc.
Storage Formats • Row-based storage format – Similar to Postgres heap storage • No toast • No ctid, xmin, xmax, cmin, cmax
Storage Formats • Row-based storage format – Similar to Postgres heap storage • No toast • No ctid, xmin, xmax, cmin, cmax – Compression • No compression • Quicklz • Zlib levels 1 - 9
Storage Formats • Apache Parquet – Mixed row-columnar table store, the data is split into “row groups” stored in columnar format
Storage Formats • Apache Parquet – Mixed row-columnar table store, the data is split into “row groups” stored in columnar format – Compression • No compression • Snappy • Gzip levels 1 – 9
Storage Formats • Apache Parquet – Mixed row-columnar table store, the data is split into “row groups” stored in columnar format – Compression • No compression • Snappy • Gzip levels 1 – 9 – The size of “row group” and page size can be set for each table separately
Resource Management • Two main options – Static resource split – HAWQ and YARN does not know about each other
Resource Management • Two main options – Static resource split – HAWQ and YARN does not know about each other – YARN – HAWQ asks YARN Resource Manager for query execution resources
Resource Management • Two main options – Static resource split – HAWQ and YARN does not know about each other – YARN – HAWQ asks YARN Resource Manager for query execution resources • Flexible cluster utilization – Query might run on a subset of nodes if it is small
Resource Management • Two main options – Static resource split – HAWQ and YARN does not know about each other – YARN – HAWQ asks YARN Resource Manager for query execution resources • Flexible cluster utilization – Query might run on a subset of nodes if it is small – Query might have many executors on each cluster node to make it run faster
Resource Management • Two main options – Static resource split – HAWQ and YARN does not know about each other – YARN – HAWQ asks YARN Resource Manager for query execution resources • Flexible cluster utilization – Query might run on a subset of nodes if it is small – Query might have many executors on each cluster node to make it run faster – You can control the parallelism of each query
Resource Management • Resource Queue can be set with – Maximum number of parallel queries
Resource Management • Resource Queue can be set with – Maximum number of parallel queries – CPU usage priority
Resource Management • Resource Queue can be set with – Maximum number of parallel queries – CPU usage priority – Memory usage limits
Resource Management • Resource Queue can be set with – Maximum number of parallel queries – CPU usage priority – Memory usage limits – CPU cores usage limit
Resource Management • Resource Queue can be set with – Maximum number of parallel queries – CPU usage priority – Memory usage limits – CPU cores usage limit – MIN/MAX number of executors across the system
Resource Management • Resource Queue can be set with – Maximum number of parallel queries – CPU usage priority – Memory usage limits – CPU cores usage limit – MIN/MAX number of executors across the system – MIN/MAX number of executors on each node
Resource Management • Resource Queue can be set with – Maximum number of parallel queries – CPU usage priority – Memory usage limits – CPU cores usage limit – MIN/MAX number of executors across the system – MIN/MAX number of executors on each node • Can be set up for user or group
External Data • PXF – Framework for external data access – Easy to extend, many public plugins available – Official plugins: CSV, SequenceFile, Avro, Hive, HBase – Open Source plugins: JSON, Accumulo, Cassandra, JDBC, Redis, Pipe
External Data • PXF – Framework for external data access – Easy to extend, many public plugins available – Official plugins: CSV, SequenceFile, Avro, Hive, HBase – Open Source plugins: JSON, Accumulo, Cassandra, JDBC, Redis, Pipe • HCatalog – HAWQ can query tables from HCatalog the same way as HAWQ native tables
Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Resource Prepare Execute Result CleanupPlan
Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Resource Prepare Execute Result CleanupPlan QE
Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Resource Prepare Execute Result CleanupPlan QE
Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Resource Prepare Execute Result CleanupPlan QE
Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Resource Prepare Execute Result CleanupPlan QE
Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Resource Prepare Execute Result CleanupPlan QE ScanBars b HashJoinb.name =s.bar ScanSells s Filterb.city ='SanFrancisco' Projects.beer, s.price MotionGather MotionRedist(b.name)
Plan Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Prepare Execute Result Cleanup QE Resource
Plan Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Prepare Execute Result Cleanup QE Resource I need 5 containers Each with 1 CPU core and 256 MB RAM
Plan Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Prepare Execute Result Cleanup QE Resource I need 5 containers Each with 1 CPU core and 256 MB RAM Server 1: 2 containers Server 2: 1 container Server N: 2 containers
Plan Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Prepare Execute Result Cleanup QE Resource I need 5 containers Each with 1 CPU core and 256 MB RAM Server 1: 2 containers Server 2: 1 container Server N: 2 containers
Plan Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Prepare Execute Result Cleanup QE Resource I need 5 containers Each with 1 CPU core and 256 MB RAM Server 1: 2 containers Server 2: 1 container Server N: 2 containers
Plan Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Prepare Execute Result Cleanup QE Resource I need 5 containers Each with 1 CPU core and 256 MB RAM Server 1: 2 containers Server 2: 1 container Server N: 2 containers QE QE QE QE QE
ResourcePlan Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Execute Result Cleanup QE QE QE QE QE QE Prepare
ResourcePlan Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Execute Result Cleanup QE QE QE QE QE QE Prepare ScanBars b HashJoinb.name =s.bar ScanSells s Filterb.city ='SanFrancisco' Projects.beer, s.price MotionGather MotionRedist(b.name)
ResourcePlan Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Execute Result Cleanup QE QE QE QE QE QE Prepare ScanBars b HashJoinb.name =s.bar ScanSells s Filterb.city ='SanFrancisco' Projects.beer, s.price MotionGather MotionRedist(b.name)
ResourcePlan Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Result Cleanup QE QE QE QE QE QE Prepare Execute ScanBars b HashJoinb.name =s.bar ScanSells s Filterb.city ='SanFrancisco' Projects.beer, s.price MotionGather MotionRedist(b.name)
ResourcePlan Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Result Cleanup QE QE QE QE QE QE Prepare Execute ScanBars b HashJoinb.name =s.bar ScanSells s Filterb.city ='SanFrancisco' Projects.beer, s.price MotionGather MotionRedist(b.name)
ResourcePlan Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Result Cleanup QE QE QE QE QE QE Prepare Execute ScanBars b HashJoinb.name =s.bar ScanSells s Filterb.city ='SanFrancisco' Projects.beer, s.price MotionGather MotionRedist(b.name)
ResourcePlan Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Cleanup QE QE QE QE QE QE Prepare Execute Result
ResourcePlan Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Cleanup QE QE QE QE QE QE Prepare Execute Result
ResourcePlan Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Cleanup QE QE QE QE QE QE Prepare Execute Result
ResourcePlan Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster QE QE QE QE QE QE Prepare Execute Result Cleanup
ResourcePlan Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster QE QE QE QE QE QE Prepare Execute Result Cleanup Free query resources Server 1: 2 containers Server 2: 1 container Server N: 2 containers
ResourcePlan Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster QE QE QE QE QE QE Prepare Execute Result Cleanup Free query resources Server 1: 2 containers Server 2: 1 container Server N: 2 containers OK
ResourcePlan Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster QE QE QE QE QE QE Prepare Execute Result Cleanup Free query resources Server 1: 2 containers Server 2: 1 container Server N: 2 containers OK
ResourcePlan Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster QE QE QE QE QE QE Prepare Execute Result Cleanup Free query resources Server 1: 2 containers Server 2: 1 container Server N: 2 containers OK
ResourcePlan Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster QE QE QE QE QE QE Prepare Execute Result Cleanup Free query resources Server 1: 2 containers Server 2: 1 container Server N: 2 containers OK
ResourcePlan Query Example HAWQ Master Metadata Transaction Mgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Prepare Execute Result Cleanup
Query Performance • Data does not hit the disk unless this cannot be avoided
Query Performance • Data does not hit the disk unless this cannot be avoided • Data is not buffered on the segments unless this cannot be avoided
Query Performance • Data does not hit the disk unless this cannot be avoided • Data is not buffered on the segments unless this cannot be avoided • Data is transferred between the nodes by UDP
Query Performance • Data does not hit the disk unless this cannot be avoided • Data is not buffered on the segments unless this cannot be avoided • Data is transferred between the nodes by UDP • HAWQ has a good cost-based query optimizer
Query Performance • Data does not hit the disk unless this cannot be avoided • Data is not buffered on the segments unless this cannot be avoided • Data is transferred between the nodes by UDP • HAWQ has a good cost-based query optimizer • C/C++ implementation is more efficient than Java implementation of competitive solutions
Query Performance • Data does not hit the disk unless this cannot be avoided • Data is not buffered on the segments unless this cannot be avoided • Data is transferred between the nodes by UDP • HAWQ has a good cost-based query optimizer • C/C++ implementation is more efficient than Java implementation of competitive solutions • Query parallelism can be easily tuned
Competitive Solutions Hive SparkSQL Impala HAWQ Query Optimizer
Competitive Solutions Hive SparkSQL Impala HAWQ Query Optimizer ANSI SQL
Competitive Solutions Hive SparkSQL Impala HAWQ Query Optimizer ANSI SQL Built-in Languages
Competitive Solutions Hive SparkSQL Impala HAWQ Query Optimizer ANSI SQL Built-in Languages Disk IO
Competitive Solutions Hive SparkSQL Impala HAWQ Query Optimizer ANSI SQL Built-in Languages Disk IO Parallelism
Competitive Solutions Hive SparkSQL Impala HAWQ Query Optimizer ANSI SQL Built-in Languages Disk IO Parallelism Distributions
Competitive Solutions Hive SparkSQL Impala HAWQ Query Optimizer ANSI SQL Built-in Languages Disk IO Parallelism Distributions Stability
Competitive Solutions Hive SparkSQL Impala HAWQ Query Optimizer ANSI SQL Built-in Languages Disk IO Parallelism Distributions Stability Community
Roadmap • AWS and S3 integration
Roadmap • AWS and S3 integration • Mesos integration
Roadmap • AWS and S3 integration • Mesos integration • Better Ambari integration
Roadmap • AWS and S3 integration • Mesos integration • Better Ambari integration • Cloudera, MapR and IBM Hadoop distributions native support
Roadmap • AWS and S3 integration • Mesos integration • Better Ambari integration • Cloudera, MapR and IBM Hadoop distributions native support • Make the SQL-on-Hadoop engine ever!
Summary • Modern SQL-on-Hadoop engine • For structured data processing and analysis • Combines the best techniques of competitive solutions • Just released to the open source • Community is very young Join our community and contribute!
Questions Apache HAWQ http://hawq.incubator.apache.org dev@hawq.incubator.apache.org user@hawq.incubator.apache.org Reach me on http://0x0fff.com

More Related Content

PPTX
Green roof presentation
byAravind Murugesan
 
PPTX
Accelerating Innovation from Edge to Cloud
byRebekah Rodriguez
 
PDF
FINAL SHEET-2(podium lvl. pathway).pdf
bySOLAIsurya
 
PDF
Jamshed bhabha
byShrividhyah
 
PPTX
Introduction to Microsoft’s Hadoop solution (HDInsight)
byJames Serra
 
PDF
Preopening Presentation
byDushyant Singh
 
PPTX
Iiit delhi case study
byMOHAMMAD FARZAN
 
PPTX
Maruti Udyog Limited
byAijaz Aryan
 
Green roof presentation
byAravind Murugesan
 
Accelerating Innovation from Edge to Cloud
byRebekah Rodriguez
 
FINAL SHEET-2(podium lvl. pathway).pdf
bySOLAIsurya
 
Jamshed bhabha
byShrividhyah
 
Introduction to Microsoft’s Hadoop solution (HDInsight)
byJames Serra
 
Preopening Presentation
byDushyant Singh
 
Iiit delhi case study
byMOHAMMAD FARZAN
 
Maruti Udyog Limited
byAijaz Aryan
 

What's hot

PPTX
Digital technology 1
byDirector-Navnirman Bahu-Uddeshiya Mahila Sanstha,Suvarna Lele Architects.
 
PPTX
The edge.pptx
bydhanya510097
 
PDF
An architecture for federated data discovery and lineage over on-prem datasou...
byDataWorks Summit
 
PPTX
Green computing
byAbhay Nigam
 
PPTX
Mcg, australia
bysachin singla
 
PPTX
Latest trends in information technology
byEldos Kuriakose
 
PDF
Timber acoustic panels
byGARYFS1976
 
PDF
Green Buildings
byANSHITA AGGARWAL
 
PDF
mechatronics lecture notes.pdf
byLaggo Anelka
 
PPT
ATLANTIS - THE CLUB - AT JAYPEE INTEGRATED SPORETS COMPLEX GREATER NOIDA
byPARICHOWK DOT COM
 
PPTX
Green computing
byMariya Zafar
 
PDF
Shyam Thesis Report
byShyam Singh
 
PDF
AWS or Azure or Google Cloud | Best Cloud Platform | Cloud Platform Comparison
byMariya James
 
PPTX
Stadium
bysheikhtanzeel35
 
PPTX
Data Virtualization: An Introduction
byDenodo
 
PDF
Grand Hyatt Dubai Hotel
byGrand Hyatt Dubai
 
PPTX
Tree Plantation on Rooftop
byArif Faisal Emon
 
PDF
Snowflake free trial_lab_guide
byslidedown1
 
PPTX
Green building Manipal University
bySiddiq Salim
 
PPTX
Sustainable Architecture - Case Study.pptx
byKaleKale9
 
Digital technology 1
byDirector-Navnirman Bahu-Uddeshiya Mahila Sanstha,Suvarna Lele Architects.
 
The edge.pptx
bydhanya510097
 
An architecture for federated data discovery and lineage over on-prem datasou...
byDataWorks Summit
 
Green computing
byAbhay Nigam
 
Mcg, australia
bysachin singla
 
Latest trends in information technology
byEldos Kuriakose
 
Timber acoustic panels
byGARYFS1976
 
Green Buildings
byANSHITA AGGARWAL
 
mechatronics lecture notes.pdf
byLaggo Anelka
 
ATLANTIS - THE CLUB - AT JAYPEE INTEGRATED SPORETS COMPLEX GREATER NOIDA
byPARICHOWK DOT COM
 
Green computing
byMariya Zafar
 
Shyam Thesis Report
byShyam Singh
 
AWS or Azure or Google Cloud | Best Cloud Platform | Cloud Platform Comparison
byMariya James
 
Stadium
bysheikhtanzeel35
 
Data Virtualization: An Introduction
byDenodo
 
Grand Hyatt Dubai Hotel
byGrand Hyatt Dubai
 
Tree Plantation on Rooftop
byArif Faisal Emon
 
Snowflake free trial_lab_guide
byslidedown1
 
Green building Manipal University
bySiddiq Salim
 
Sustainable Architecture - Case Study.pptx
byKaleKale9
 

Similar to Apache HAWQ Architecture

PDF
SQL and Machine Learning on Hadoop
byMukund Babbar
 
PDF
HAWQ: a massively parallel processing SQL engine in hadoop
byBigData Research
 
PDF
Hawq wp 042313_final
byEMC
 
PDF
SQL on Hadoop
bynvvrajesh
 
PDF
Big Data Developers Moscow Meetup 1 - sql on hadoop
bybddmoscow
 
PPTX
Unit II Hadoop Ecosystem_Updated.pptx
byBhavanaHotchandani
 
PPT
Hive Evolution: ApacheCon NA 2010
byJohn Sichi
 
PPTX
SQL On Hadoop
byMuhammad Ali
 
PPTX
BDA: Introduction to HIVE, PIG and HBASE
bytripathineeharika
 
PPT
Hive Training -- Motivations and Real World Use Cases
bynzhang
 
PPTX
מיכאל
bysqlserver.co.il
 
PPTX
Hadoop and mysql by Chris Schneider
byDmitry Makarchuk
 
PDF
20080529dublinpt3
byJeff Hammerbacher
 
PDF
OPTIMIZATION OF MULTIPLE CORRELATED QUERIES BY DETECTING SIMILAR DATA SOURCE ...
byPuneet Kansal
 
PDF
Hadoop and Hive Development at Facebook
byS S
 
PDF
Hadoop and Hive Development at Facebook
byelliando dias
 
PPTX
Hive @ Bucharest Java User Group
byRemus Rusanu
 
PDF
Big SQL 3.0: Datawarehouse-grade Performance on Hadoop - At last!
byNicolas Morales
 
PPT
Hadoop Hive Talk At IIT-Delhi
byJoydeep Sen Sarma
 
PDF
Facebook hadoop-summit
byS S
 
SQL and Machine Learning on Hadoop
byMukund Babbar
 
HAWQ: a massively parallel processing SQL engine in hadoop
byBigData Research
 
Hawq wp 042313_final
byEMC
 
SQL on Hadoop
bynvvrajesh
 
Big Data Developers Moscow Meetup 1 - sql on hadoop
bybddmoscow
 
Unit II Hadoop Ecosystem_Updated.pptx
byBhavanaHotchandani
 
Hive Evolution: ApacheCon NA 2010
byJohn Sichi
 
SQL On Hadoop
byMuhammad Ali
 
BDA: Introduction to HIVE, PIG and HBASE
bytripathineeharika
 
Hive Training -- Motivations and Real World Use Cases
bynzhang
 
מיכאל
bysqlserver.co.il
 
Hadoop and mysql by Chris Schneider
byDmitry Makarchuk
 
20080529dublinpt3
byJeff Hammerbacher
 
OPTIMIZATION OF MULTIPLE CORRELATED QUERIES BY DETECTING SIMILAR DATA SOURCE ...
byPuneet Kansal
 
Hadoop and Hive Development at Facebook
byS S
 
Hadoop and Hive Development at Facebook
byelliando dias
 
Hive @ Bucharest Java User Group
byRemus Rusanu
 
Big SQL 3.0: Datawarehouse-grade Performance on Hadoop - At last!
byNicolas Morales
 
Hadoop Hive Talk At IIT-Delhi
byJoydeep Sen Sarma
 
Facebook hadoop-summit
byS S
 

More from Alexey Grishchenko

PDF
MPP vs Hadoop
byAlexey Grishchenko
 
PDF
Greenplum Architecture
byAlexey Grishchenko
 
PPTX
Apache Spark Architecture
byAlexey Grishchenko
 
PPTX
Modern Data Architecture
byAlexey Grishchenko
 
PPTX
Архитектура Apache HAWQ Highload++ 2015
byAlexey Grishchenko
 
PPTX
Pivotal hawq internals
byAlexey Grishchenko
 
MPP vs Hadoop
byAlexey Grishchenko
 
Greenplum Architecture
byAlexey Grishchenko
 
Apache Spark Architecture
byAlexey Grishchenko
 
Modern Data Architecture
byAlexey Grishchenko
 
Архитектура Apache HAWQ Highload++ 2015
byAlexey Grishchenko
 
Pivotal hawq internals
byAlexey Grishchenko
 

Recently uploaded

PDF
Community re-Invent re-Cap Slides AWS UserGroup Hamburg 01/2026
byAlexeyVidanov1
 
PDF
(UPDATE) JADWAL GEREJA OPS LILIN 2025.pdf
byMinlantasSumber
 
PDF
Operations and Data Support Portfolio | Delbert Ross Wanjala Investor Relatio...
byDelbertRossWanjala
 
PDF
Machine learning genrative AI and agents
bysakingul
 
PPTX
[DSC Europe 25] Harshvardhan Jain - From Pre-Trained to Purpose-Built: Fine-T...
byDataScienceConferenc1
 
PPTX
Full_Transformer_hjyhkkiProject_PPT.pptx
byhrishikeshmanu2008
 
PDF
The-Authoritarian-Gaze.pdfThe-Authoritarian-Gaze.pdf
by中 央社
 
PDF
[DSC Europe 25] Laila Kakar - Leveraging AI for Strategic Excellence: Enhanci...
byDataScienceConferenc1
 
PPTX
[DSC Europe 25] Marcos Heidemann - Beyond the Hype: Making AI Coding Assistan...
byDataScienceConferenc1
 
PDF
[DSC Europe 25] Tamas Srancsik - How To Teach Your AI Football? An Argument f...
byDataScienceConferenc1
 
PDF
[DSC Europe 25] Gordana Milutinovic Dumbelovic - From Insight to Oversight: A...
byDataScienceConferenc1
 
PPTX
[DSC Europe 25] Milos Belcevic - Product Professional's Journey to Full-Stack...
byDataScienceConferenc1
 
PDF
Best Web to Learn About Buying Verified Payeer Accounts (Los Angeles).pdf
byhttps://topsellerit.com/product/buy-verified-binance-accounts/
 
PPTX
2026 Retreat Presentations (GREBS and WRAPS).pptx
bytobacalebakanmu
 
PDF
SIH2025-IDEA-Presentation-Innogenix320250929144422.pdf
bydevichokkakulacse1
 
PPTX
[DSC Europe 25] Borko Kozomora - Optimizing business workflows with advances ...
byDataScienceConferenc1
 
PDF
Deep Learning Fundamentals with Neural Networks and AI Concepts
byaswinajay312004
 
PPTX
Orientation-of-DM2025-0104-Powerpoint.pptx
byNURSEKALOY
 
PDF
[DSC Europe 25] Bojan Djuricic - Predictive Design Process.pdf
byDataScienceConferenc1
 
PPTX
PPT sidang disertasi terbuka terbaru.pptx
byRifaiteguhSaputra
 
Community re-Invent re-Cap Slides AWS UserGroup Hamburg 01/2026
byAlexeyVidanov1
 
(UPDATE) JADWAL GEREJA OPS LILIN 2025.pdf
byMinlantasSumber
 
Operations and Data Support Portfolio | Delbert Ross Wanjala Investor Relatio...
byDelbertRossWanjala
 
Machine learning genrative AI and agents
bysakingul
 
[DSC Europe 25] Harshvardhan Jain - From Pre-Trained to Purpose-Built: Fine-T...
byDataScienceConferenc1
 
Full_Transformer_hjyhkkiProject_PPT.pptx
byhrishikeshmanu2008
 
The-Authoritarian-Gaze.pdfThe-Authoritarian-Gaze.pdf
by中 央社
 
[DSC Europe 25] Laila Kakar - Leveraging AI for Strategic Excellence: Enhanci...
byDataScienceConferenc1
 
[DSC Europe 25] Marcos Heidemann - Beyond the Hype: Making AI Coding Assistan...
byDataScienceConferenc1
 
[DSC Europe 25] Tamas Srancsik - How To Teach Your AI Football? An Argument f...
byDataScienceConferenc1
 
[DSC Europe 25] Gordana Milutinovic Dumbelovic - From Insight to Oversight: A...
byDataScienceConferenc1
 
[DSC Europe 25] Milos Belcevic - Product Professional's Journey to Full-Stack...
byDataScienceConferenc1
 
Best Web to Learn About Buying Verified Payeer Accounts (Los Angeles).pdf
byhttps://topsellerit.com/product/buy-verified-binance-accounts/
 
2026 Retreat Presentations (GREBS and WRAPS).pptx
bytobacalebakanmu
 
SIH2025-IDEA-Presentation-Innogenix320250929144422.pdf
bydevichokkakulacse1
 
[DSC Europe 25] Borko Kozomora - Optimizing business workflows with advances ...
byDataScienceConferenc1
 
Deep Learning Fundamentals with Neural Networks and AI Concepts
byaswinajay312004
 
Orientation-of-DM2025-0104-Powerpoint.pptx
byNURSEKALOY
 
[DSC Europe 25] Bojan Djuricic - Predictive Design Process.pdf
byDataScienceConferenc1
 
PPT sidang disertasi terbuka terbaru.pptx
byRifaiteguhSaputra
 

Apache HAWQ Architecture

  • 1.
  • 2.
    Who I am EnterpriseArchitect @ Pivotal • 7 years in data processing • 5 years of experience with MPP • 4 years with Hadoop • Using HAWQ since the first internal Beta • Responsible for designing most of the EMEA HAWQ and Greenplum implementations • Spark contributor • http://0x0fff.com
  • 3.
  • 4.
    Agenda • What isHAWQ • Why you need it
  • 5.
    Agenda • What isHAWQ • Why you need it • HAWQ Components
  • 6.
    Agenda • What isHAWQ • Why you need it • HAWQ Components • HAWQ Design
  • 7.
    Agenda • What isHAWQ • Why you need it • HAWQ Components • HAWQ Design • Query execution example
  • 8.
    Agenda • What isHAWQ • Why you need it • HAWQ Components • HAWQ Design • Query execution example • Competitive solutions
  • 9.
    What is • AnalyticalSQL-on-Hadoop engine
  • 10.
    What is • AnalyticalSQL-on-Hadoop engine • HAdoop With Queries
  • 11.
    What is • AnalyticalSQL-on-Hadoop engine • HAdoop With Queries Postgres Greenplum HAWQ 2005 Fork Postgres 8.0.2
  • 12.
    What is • AnalyticalSQL-on-Hadoop engine • HAdoop With Queries Postgres HAWQ 2005 Fork Postgres 8.0.2 2009 Rebase Postgres 8.2.15 Greenplum
  • 13.
    What is • AnalyticalSQL-on-Hadoop engine • HAdoop With Queries Postgres HAWQ 2005 Fork Postgres 8.0.2 2009 Rebase Postgres 8.2.15 2011 Fork GPDB 4.2.0.0 Greenplum
  • 14.
    What is • AnalyticalSQL-on-Hadoop engine • HAdoop With Queries Postgres HAWQ 2005 Fork Postgres 8.0.2 2009 Rebase Postgres 8.2.15 2011 Fork GPDB 4.2.0.0 2013 HAWQ 1.0.0.0 Greenplum
  • 15.
    What is • AnalyticalSQL-on-Hadoop engine • HAdoop With Queries Postgres HAWQ 2005 Fork Postgres 8.0.2 2009 Rebase Postgres 8.2.15 2011 Fork GPDB 4.2.0.0 2013 HAWQ 1.0.0.0 HAWQ 2.0.0.0 Open Source 2015 Greenplum
  • 16.
    HAWQ is … •1’500’000 C and C++ lines of code
  • 17.
    HAWQ is … •1’500’000 C and C++ lines of code – 200’000 of them in headers only
  • 18.
    HAWQ is … •1’500’000 C and C++ lines of code – 200’000 of them in headers only • 180’000 Python LOC
  • 19.
    HAWQ is … •1’500’000 C and C++ lines of code – 200’000 of them in headers only • 180’000 Python LOC • 60’000 Java LOC
  • 20.
    HAWQ is … •1’500’000 C and C++ lines of code – 200’000 of them in headers only • 180’000 Python LOC • 60’000 Java LOC • 23’000 Makefile LOC
  • 21.
    HAWQ is … •1’500’000 C and C++ lines of code – 200’000 of them in headers only • 180’000 Python LOC • 60’000 Java LOC • 23’000 Makefile LOC • 7’000 Shell scripts LOC
  • 22.
    HAWQ is … •1’500’000 C and C++ lines of code – 200’000 of them in headers only • 180’000 Python LOC • 60’000 Java LOC • 23’000 Makefile LOC • 7’000 Shell scripts LOC • More than 50 enterprise customers
  • 23.
    HAWQ is … •1’500’000 C and C++ lines of code – 200’000 of them in headers only • 180’000 Python LOC • 60’000 Java LOC • 23’000 Makefile LOC • 7’000 Shell scripts LOC • More than 50 enterprise customers – More than 10 of them in EMEA
  • 24.
    Apache HAWQ • ApacheHAWQ (incubating) from 09’2015 – http://hawq.incubator.apache.org – https://github.com/apache/incubator-hawq • What’s in Open Source – Sources of HAWQ 2.0 alpha – HAWQ 2.0 beta is planned for 2015’Q4 – HAWQ 2.0 GA is planned for 2016’Q1 • Community is yet young – come and join!
  • 25.
    Why do weneed it?
  • 26.
    Why do weneed it? • SQL-interface for BI solutions to the Hadoop data complaint with ANSI SQL-92, -99, -2003
  • 27.
    Why do weneed it? • SQL-interface for BI solutions to the Hadoop data complaint with ANSI SQL-92, -99, -2003 – Example - 5000-line query with a number of window function generated by Cognos
  • 28.
    Why do weneed it? • SQL-interface for BI solutions to the Hadoop data complaint with ANSI SQL-92, -99, -2003 – Example - 5000-line query with a number of window function generated by Cognos • Universal tool for ad hoc analytics on top of Hadoop data
  • 29.
    Why do weneed it? • SQL-interface for BI solutions to the Hadoop data complaint with ANSI SQL-92, -99, -2003 – Example - 5000-line query with a number of window function generated by Cognos • Universal tool for ad hoc analytics on top of Hadoop data – Example - parse URL to extract protocol, host name, port, GET parameters
  • 30.
    Why do weneed it? • SQL-interface for BI solutions to the Hadoop data complaint with ANSI SQL-92, -99, -2003 – Example - 5000-line query with a number of window function generated by Cognos • Universal tool for ad hoc analytics on top of Hadoop data – Example - parse URL to extract protocol, host name, port, GET parameters • Good performance
  • 31.
    Why do weneed it? • SQL-interface for BI solutions to the Hadoop data complaint with ANSI SQL-92, -99, -2003 – Example - 5000-line query with a number of window function generated by Cognos • Universal tool for ad hoc analytics on top of Hadoop data – Example - parse URL to extract protocol, host name, port, GET parameters • Good performance – How many times the data would hit the HDD during a single Hive query?
  • 32.
    HAWQ Cluster Server 1 SNameNode Server4 ZK JM NameNode Server 3 ZK JM Server 2 ZK JM Server 6 Datanode Server N Datanode Server 5 Datanode interconnect …
  • 33.
    HAWQ Cluster Server 1 SNameNode Server4 ZK JM NameNode Server 3 ZK JM Server 2 ZK JM Server 6 Datanode Server N Datanode Server 5 Datanode YARN NM YARN NM YARN NM YARN RM YARN App Timeline interconnect …
  • 34.
    HAWQ Cluster HAWQ Master Server1 SNameNode Server 4 ZK JM NameNode Server 3 ZK JM HAWQ Standby Server 2 ZK JM HAWQ Segment Server 6 Datanode HAWQ Segment Server N Datanode HAWQ Segment Server 5 Datanode YARN NM YARN NM YARN NM YARN RM YARN App Timeline interconnect …
  • 35.
    Master Servers Server 1 SNameNode Server4 ZK JM NameNode Server 3 ZK JM Server 2 ZK JM HAWQ Segment Server 6 Datanode HAWQ Segment Server N Datanode HAWQ Segment Server 5 Datanode YARN NM YARN NM YARN NM YARN RM YARN App Timeline interconnect … HAWQ Master HAWQ Standby
  • 36.
    Master Servers HAWQ Master QueryParser Query Optimizer Global Resource Manager Distributed Transactions Manager Query Dispatch Metadata Catalog HAWQ Standby Master Query Parser Query Optimizer Global Resource Manager Distributed Transactions Manager Query Dispatch Metadata Catalog WAL repl.
  • 37.
    HAWQ Master HAWQ Standby Segments Server 1 SNameNode Server4 ZK JM NameNode Server 3 ZK JM Server 2 ZK JM Server 6 Datanode Server N Datanode Server 5 Datanode YARN NM YARN NM YARN NM YARN RM YARN App Timeline interconnect HAWQ Segment HAWQ SegmentHAWQ Segment …
  • 38.
    Segments HAWQ Segment Query Executor libhdfs3 PXF HDFSDatanode Local Filesystem Temporary Data Directory Logs YARN Node Manager
  • 39.
    Metadata • HAWQ metadatastructure is similar to Postgres catalog structure
  • 40.
    Metadata • HAWQ metadatastructure is similar to Postgres catalog structure • Statistics – Number of rows and pages in the table
  • 41.
    Metadata • HAWQ metadatastructure is similar to Postgres catalog structure • Statistics – Number of rows and pages in the table – Most common values for each field
  • 42.
    Metadata • HAWQ metadatastructure is similar to Postgres catalog structure • Statistics – Number of rows and pages in the table – Most common values for each field – Histogram of values distribution for each field
  • 43.
    Metadata • HAWQ metadatastructure is similar to Postgres catalog structure • Statistics – Number of rows and pages in the table – Most common values for each field – Histogram of values distribution for each field – Number of unique values in the field
  • 44.
    Metadata • HAWQ metadatastructure is similar to Postgres catalog structure • Statistics – Number of rows and pages in the table – Most common values for each field – Histogram of values distribution for each field – Number of unique values in the field – Number of null values in the field
  • 45.
    Metadata • HAWQ metadatastructure is similar to Postgres catalog structure • Statistics – Number of rows and pages in the table – Most common values for each field – Histogram of values distribution for each field – Number of unique values in the field – Number of null values in the field – Average width of the field in bytes
  • 46.
    Statistics No Statistics How manyrows would produce the join of two tables?
  • 47.
    Statistics No Statistics How manyrows would produce the join of two tables?  From 0 to infinity
  • 48.
    Statistics No Statistics Row Count Howmany rows would produce the join of two tables?  From 0 to infinity How many rows would produce the join of two 1000- row tables?
  • 49.
    Statistics No Statistics Row Count Howmany rows would produce the join of two tables?  From 0 to infinity How many rows would produce the join of two 1000- row tables?  From 0 to 1’000’000
  • 50.
    Statistics No Statistics Row Count Histogramsand MCV How many rows would produce the join of two tables?  From 0 to infinity How many rows would produce the join of two 1000- row tables?  From 0 to 1’000’000 How many rows would produce the join of two 1000- row tables, with known field cardinality, values distribution diagram, number of nulls, most common values?
  • 51.
    Statistics No Statistics Row Count Histogramsand MCV How many rows would produce the join of two tables?  From 0 to infinity How many rows would produce the join of two 1000- row tables?  From 0 to 1’000’000 How many rows would produce the join of two 1000- row tables, with known field cardinality, values distribution diagram, number of nulls, most common values?  ~ From 500 to 1’500
  • 52.
    Metadata • Table structureinformation ID Name Num Price 1 Яблоко 10 50 2 Груша 20 80 3 Банан 40 40 4 Апельсин 25 50 5 Киви 5 120 6 Арбуз 20 30 7 Дыня 40 100 8 Ананас 35 90
  • 53.
    Metadata • Table structureinformation – Distribution fields ID Name Num Price 1 Яблоко 10 50 2 Груша 20 80 3 Банан 40 40 4 Апельсин 25 50 5 Киви 5 120 6 Арбуз 20 30 7 Дыня 40 100 8 Ананас 35 90 hash(ID)
  • 54.
    Metadata • Table structureinformation – Distribution fields – Number of hash buckets ID Name Num Price 1 Яблоко 10 50 2 Груша 20 80 3 Банан 40 40 4 Апельсин 25 50 5 Киви 5 120 6 Арбуз 20 30 7 Дыня 40 100 8 Ананас 35 90 hash(ID) ID Name Num Price 1 Яблоко 10 50 2 Груша 20 80 3 Банан 40 40 4 Апельсин 25 50 5 Киви 5 120 6 Арбуз 20 30 7 Дыня 40 100 8 Ананас 35 90
  • 55.
    Metadata • Table structureinformation – Distribution fields – Number of hash buckets – Partitioning (hash, list, range) ID Name Num Price 1 Яблоко 10 50 2 Груша 20 80 3 Банан 40 40 4 Апельсин 25 50 5 Киви 5 120 6 Арбуз 20 30 7 Дыня 40 100 8 Ананас 35 90 hash(ID) ID Name Num Price 1 Яблоко 10 50 2 Груша 20 80 3 Банан 40 40 4 Апельсин 25 50 5 Киви 5 120 6 Арбуз 20 30 7 Дыня 40 100 8 Ананас 35 90
  • 56.
    Metadata • Table structureinformation – Distribution fields – Number of hash buckets – Partitioning (hash, list, range) • General metadata – Users and groups
  • 57.
    Metadata • Table structureinformation – Distribution fields – Number of hash buckets – Partitioning (hash, list, range) • General metadata – Users and groups – Access privileges
  • 58.
    Metadata • Table structureinformation – Distribution fields – Number of hash buckets – Partitioning (hash, list, range) • General metadata – Users and groups – Access privileges • Stored procedures – PL/pgSQL, PL/Java, PL/Python, PL/Perl, PL/R
  • 59.
    Query Optimizer • HAWQuses cost-based query optimizers
  • 60.
    Query Optimizer • HAWQuses cost-based query optimizers • You have two options – Planner – evolved from the Postgres query optimizer – ORCA (Pivotal Query Optimizer) – developed specifically for HAWQ
  • 61.
    Query Optimizer • HAWQuses cost-based query optimizers • You have two options – Planner – evolved from the Postgres query optimizer – ORCA (Pivotal Query Optimizer) – developed specifically for HAWQ • Optimizer hints work just like in Postgres – Enable/disable specific operation – Change the cost estimations for basic actions
  • 62.
    Storage Formats Which storageformat is the most optimal?
  • 63.
    Storage Formats Which storageformat is the most optimal?  It depends on what you mean by “optimal”
  • 64.
    Storage Formats Which storageformat is the most optimal?  It depends on what you mean by “optimal” – Minimal CPU usage for reading and writing the data
  • 65.
    Storage Formats Which storageformat is the most optimal?  It depends on what you mean by “optimal” – Minimal CPU usage for reading and writing the data – Minimal disk space usage
  • 66.
    Storage Formats Which storageformat is the most optimal?  It depends on what you mean by “optimal” – Minimal CPU usage for reading and writing the data – Minimal disk space usage – Minimal time to retrieve record by key
  • 67.
    Storage Formats Which storageformat is the most optimal?  It depends on what you mean by “optimal” – Minimal CPU usage for reading and writing the data – Minimal disk space usage – Minimal time to retrieve record by key – Minimal time to retrieve subset of columns – etc.
  • 68.
    Storage Formats • Row-basedstorage format – Similar to Postgres heap storage • No toast • No ctid, xmin, xmax, cmin, cmax
  • 69.
    Storage Formats • Row-basedstorage format – Similar to Postgres heap storage • No toast • No ctid, xmin, xmax, cmin, cmax – Compression • No compression • Quicklz • Zlib levels 1 - 9
  • 70.
    Storage Formats • ApacheParquet – Mixed row-columnar table store, the data is split into “row groups” stored in columnar format
  • 71.
    Storage Formats • ApacheParquet – Mixed row-columnar table store, the data is split into “row groups” stored in columnar format – Compression • No compression • Snappy • Gzip levels 1 – 9
  • 72.
    Storage Formats • ApacheParquet – Mixed row-columnar table store, the data is split into “row groups” stored in columnar format – Compression • No compression • Snappy • Gzip levels 1 – 9 – The size of “row group” and page size can be set for each table separately
  • 73.
    Resource Management • Twomain options – Static resource split – HAWQ and YARN does not know about each other
  • 74.
    Resource Management • Twomain options – Static resource split – HAWQ and YARN does not know about each other – YARN – HAWQ asks YARN Resource Manager for query execution resources
  • 75.
    Resource Management • Twomain options – Static resource split – HAWQ and YARN does not know about each other – YARN – HAWQ asks YARN Resource Manager for query execution resources • Flexible cluster utilization – Query might run on a subset of nodes if it is small
  • 76.
    Resource Management • Twomain options – Static resource split – HAWQ and YARN does not know about each other – YARN – HAWQ asks YARN Resource Manager for query execution resources • Flexible cluster utilization – Query might run on a subset of nodes if it is small – Query might have many executors on each cluster node to make it run faster
  • 77.
    Resource Management • Twomain options – Static resource split – HAWQ and YARN does not know about each other – YARN – HAWQ asks YARN Resource Manager for query execution resources • Flexible cluster utilization – Query might run on a subset of nodes if it is small – Query might have many executors on each cluster node to make it run faster – You can control the parallelism of each query
  • 78.
    Resource Management • ResourceQueue can be set with – Maximum number of parallel queries
  • 79.
    Resource Management • ResourceQueue can be set with – Maximum number of parallel queries – CPU usage priority
  • 80.
    Resource Management • ResourceQueue can be set with – Maximum number of parallel queries – CPU usage priority – Memory usage limits
  • 81.
    Resource Management • ResourceQueue can be set with – Maximum number of parallel queries – CPU usage priority – Memory usage limits – CPU cores usage limit
  • 82.
    Resource Management • ResourceQueue can be set with – Maximum number of parallel queries – CPU usage priority – Memory usage limits – CPU cores usage limit – MIN/MAX number of executors across the system
  • 83.
    Resource Management • ResourceQueue can be set with – Maximum number of parallel queries – CPU usage priority – Memory usage limits – CPU cores usage limit – MIN/MAX number of executors across the system – MIN/MAX number of executors on each node
  • 84.
    Resource Management • ResourceQueue can be set with – Maximum number of parallel queries – CPU usage priority – Memory usage limits – CPU cores usage limit – MIN/MAX number of executors across the system – MIN/MAX number of executors on each node • Can be set up for user or group
  • 85.
    External Data • PXF –Framework for external data access – Easy to extend, many public plugins available – Official plugins: CSV, SequenceFile, Avro, Hive, HBase – Open Source plugins: JSON, Accumulo, Cassandra, JDBC, Redis, Pipe
  • 86.
    External Data • PXF –Framework for external data access – Easy to extend, many public plugins available – Official plugins: CSV, SequenceFile, Avro, Hive, HBase – Open Source plugins: JSON, Accumulo, Cassandra, JDBC, Redis, Pipe • HCatalog – HAWQ can query tables from HCatalog the same way as HAWQ native tables
  • 87.
    Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Resource Prepare Execute Result CleanupPlan
  • 88.
    Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Resource Prepare Execute Result CleanupPlan QE
  • 89.
    Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Resource Prepare Execute Result CleanupPlan QE
  • 90.
    Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Resource Prepare Execute Result CleanupPlan QE
  • 91.
    Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Resource Prepare Execute Result CleanupPlan QE
  • 92.
    Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Resource Prepare Execute Result CleanupPlan QE ScanBars b HashJoinb.name =s.bar ScanSells s Filterb.city ='SanFrancisco' Projects.beer, s.price MotionGather MotionRedist(b.name)
  • 93.
    Plan Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Prepare Execute Result Cleanup QE Resource
  • 94.
    Plan Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Prepare Execute Result Cleanup QE Resource I need 5 containers Each with 1 CPU core and 256 MB RAM
  • 95.
    Plan Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Prepare Execute Result Cleanup QE Resource I need 5 containers Each with 1 CPU core and 256 MB RAM Server 1: 2 containers Server 2: 1 container Server N: 2 containers
  • 96.
    Plan Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Prepare Execute Result Cleanup QE Resource I need 5 containers Each with 1 CPU core and 256 MB RAM Server 1: 2 containers Server 2: 1 container Server N: 2 containers
  • 97.
    Plan Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Prepare Execute Result Cleanup QE Resource I need 5 containers Each with 1 CPU core and 256 MB RAM Server 1: 2 containers Server 2: 1 container Server N: 2 containers
  • 98.
    Plan Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Prepare Execute Result Cleanup QE Resource I need 5 containers Each with 1 CPU core and 256 MB RAM Server 1: 2 containers Server 2: 1 container Server N: 2 containers QE QE QE QE QE
  • 99.
    ResourcePlan Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Execute Result Cleanup QE QE QE QE QE QE Prepare
  • 100.
    ResourcePlan Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Execute Result Cleanup QE QE QE QE QE QE Prepare ScanBars b HashJoinb.name =s.bar ScanSells s Filterb.city ='SanFrancisco' Projects.beer, s.price MotionGather MotionRedist(b.name)
  • 101.
    ResourcePlan Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Execute Result Cleanup QE QE QE QE QE QE Prepare ScanBars b HashJoinb.name =s.bar ScanSells s Filterb.city ='SanFrancisco' Projects.beer, s.price MotionGather MotionRedist(b.name)
  • 102.
    ResourcePlan Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Result Cleanup QE QE QE QE QE QE Prepare Execute ScanBars b HashJoinb.name =s.bar ScanSells s Filterb.city ='SanFrancisco' Projects.beer, s.price MotionGather MotionRedist(b.name)
  • 103.
    ResourcePlan Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Result Cleanup QE QE QE QE QE QE Prepare Execute ScanBars b HashJoinb.name =s.bar ScanSells s Filterb.city ='SanFrancisco' Projects.beer, s.price MotionGather MotionRedist(b.name)
  • 104.
    ResourcePlan Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Result Cleanup QE QE QE QE QE QE Prepare Execute ScanBars b HashJoinb.name =s.bar ScanSells s Filterb.city ='SanFrancisco' Projects.beer, s.price MotionGather MotionRedist(b.name)
  • 105.
    ResourcePlan Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Cleanup QE QE QE QE QE QE Prepare Execute Result
  • 106.
    ResourcePlan Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Cleanup QE QE QE QE QE QE Prepare Execute Result
  • 107.
    ResourcePlan Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Cleanup QE QE QE QE QE QE Prepare Execute Result
  • 108.
    ResourcePlan Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster QE QE QE QE QE QE Prepare Execute Result Cleanup
  • 109.
    ResourcePlan Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster QE QE QE QE QE QE Prepare Execute Result Cleanup Free query resources Server 1: 2 containers Server 2: 1 container Server N: 2 containers
  • 110.
    ResourcePlan Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster QE QE QE QE QE QE Prepare Execute Result Cleanup Free query resources Server 1: 2 containers Server 2: 1 container Server N: 2 containers OK
  • 111.
    ResourcePlan Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster QE QE QE QE QE QE Prepare Execute Result Cleanup Free query resources Server 1: 2 containers Server 2: 1 container Server N: 2 containers OK
  • 112.
    ResourcePlan Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster QE QE QE QE QE QE Prepare Execute Result Cleanup Free query resources Server 1: 2 containers Server 2: 1 container Server N: 2 containers OK
  • 113.
    ResourcePlan Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster QE QE QE QE QE QE Prepare Execute Result Cleanup Free query resources Server 1: 2 containers Server 2: 1 container Server N: 2 containers OK
  • 114.
    ResourcePlan Query Example HAWQ Master Metadata TransactionMgr. Query Parser Query Optimizer Query Dispatch Resource Mgr. NameNode Server 1 Local directory HAWQ Segment Postmaster HDFS Datanode Server 2 Local directory HAWQ Segment Postmaster HDFS Datanode Server N Local directory HAWQ Segment Postmaster HDFS Datanode YARN RMPostmaster Prepare Execute Result Cleanup
  • 115.
    Query Performance • Datadoes not hit the disk unless this cannot be avoided
  • 116.
    Query Performance • Datadoes not hit the disk unless this cannot be avoided • Data is not buffered on the segments unless this cannot be avoided
  • 117.
    Query Performance • Datadoes not hit the disk unless this cannot be avoided • Data is not buffered on the segments unless this cannot be avoided • Data is transferred between the nodes by UDP
  • 118.
    Query Performance • Datadoes not hit the disk unless this cannot be avoided • Data is not buffered on the segments unless this cannot be avoided • Data is transferred between the nodes by UDP • HAWQ has a good cost-based query optimizer
  • 119.
    Query Performance • Datadoes not hit the disk unless this cannot be avoided • Data is not buffered on the segments unless this cannot be avoided • Data is transferred between the nodes by UDP • HAWQ has a good cost-based query optimizer • C/C++ implementation is more efficient than Java implementation of competitive solutions
  • 120.
    Query Performance • Datadoes not hit the disk unless this cannot be avoided • Data is not buffered on the segments unless this cannot be avoided • Data is transferred between the nodes by UDP • HAWQ has a good cost-based query optimizer • C/C++ implementation is more efficient than Java implementation of competitive solutions • Query parallelism can be easily tuned
  • 121.
    Competitive Solutions Hive SparkSQLImpala HAWQ Query Optimizer
  • 122.
    Competitive Solutions Hive SparkSQLImpala HAWQ Query Optimizer ANSI SQL
  • 123.
    Competitive Solutions Hive SparkSQLImpala HAWQ Query Optimizer ANSI SQL Built-in Languages
  • 124.
    Competitive Solutions Hive SparkSQLImpala HAWQ Query Optimizer ANSI SQL Built-in Languages Disk IO
  • 125.
    Competitive Solutions Hive SparkSQLImpala HAWQ Query Optimizer ANSI SQL Built-in Languages Disk IO Parallelism
  • 126.
    Competitive Solutions Hive SparkSQLImpala HAWQ Query Optimizer ANSI SQL Built-in Languages Disk IO Parallelism Distributions
  • 127.
    Competitive Solutions Hive SparkSQLImpala HAWQ Query Optimizer ANSI SQL Built-in Languages Disk IO Parallelism Distributions Stability
  • 128.
    Competitive Solutions Hive SparkSQLImpala HAWQ Query Optimizer ANSI SQL Built-in Languages Disk IO Parallelism Distributions Stability Community
  • 129.
    Roadmap • AWS andS3 integration
  • 130.
    Roadmap • AWS andS3 integration • Mesos integration
  • 131.
    Roadmap • AWS andS3 integration • Mesos integration • Better Ambari integration
  • 132.
    Roadmap • AWS andS3 integration • Mesos integration • Better Ambari integration • Cloudera, MapR and IBM Hadoop distributions native support
  • 133.
    Roadmap • AWS andS3 integration • Mesos integration • Better Ambari integration • Cloudera, MapR and IBM Hadoop distributions native support • Make the SQL-on-Hadoop engine ever!
  • 134.
    Summary • Modern SQL-on-Hadoopengine • For structured data processing and analysis • Combines the best techniques of competitive solutions • Just released to the open source • Community is very young Join our community and contribute!
  • 135.