Download as PDF, PPTX
![void MaterializeTuple(char* tuple) {
for (int i = 0; i < num_slots_; ++i) {
char* slot = tuple + offsets_[i];
switch (types_[i]) {
case BOOLEAN:
*slot = ParseBoolean();
break;
case INT:
*slot = ParseInt();
case FLOAT: …
case STRING: …
// etc.
}
}
}
void MaterializeTuple(char* tuple) {
// i = 0
*(tuple + 0) = ParseInt();
// i = 1
*(tuple + 4) = ParseBoolean();
// i = 2
*(tuple + 5) = ParseInt();
}
Hot code path, called per row](https://image.slidesharecdn.com/impala-modernsqlengine-141113142045-conversion-gate02/85/Impala-A-Modern-Open-Source-SQL-Engine-for-Hadoop-29-320.jpg)
Uploaded byAll Things Open
Impala: A Modern, Open-Source SQL Engine for Hadoop
The document discusses a query execution framework involving components such as query planners, coordinators, and executors, which handle SQL requests and manage data across distributed systems like HDFS and HBase. It describes the process of transforming requests into executable plan fragments and handling intermediate results, with a focus on efficient data serialization. Additionally, it touches on various SQL functions and performance optimization techniques in data management.
More Related Content
Similar to Impala: A Modern, Open-Source SQL Engine for Hadoop
Impala: A Modern, Open-Source SQL Engine for Hadoop
- 17.
- 23. Query Planner QueryCoordinator Query Executor HDFS DN HBase SQL App ODBC HDFS NN Statestore & Catalog Query Planner Query Coordinator Query Executor HDFS DN HBase Query Planner Query Coordinator Query Executor HDFS DN HBase SQL request Hive Metastore
- 24. Query Planner QueryCoordinator Query Executor HDFS DN HBase SQL App ODBC Query Planner Query Coordinator Query Executor HDFS DN HBase HDFS NN Statestore & Catalog Query Planner Query Coordinator Query Executor HDFS DN HBase Planner turns request into collections of plan fragments Coordinator initiates execution on remotes nodes Hive Metastore
- 25. Intermediate results arestreamed between nodes Operation permitted, query results are streamed back to client Query Planner Query Coordinator Query Executor HDFS DN HBase SQL App ODBC Hive Metastore HDFS NN Statestore & Catalog Query Planner Query Coordinator Query Executor HDFS DN HBase Query Planner Query Coordinator Query Executor HDFS DN HBase query results
- 29. void MaterializeTuple(char* tuple){ for (int i = 0; i < num_slots_; ++i) { char* slot = tuple + offsets_[i]; switch (types_[i]) { case BOOLEAN: *slot = ParseBoolean(); break; case INT: *slot = ParseInt(); case FLOAT: … case STRING: … // etc. } } } void MaterializeTuple(char* tuple) { // i = 0 *(tuple + 0) = ParseInt(); // i = 1 *(tuple + 4) = ParseBoolean(); // i = 2 *(tuple + 5) = ParseInt(); } Hot code path, called per row
- 33. Query Fragment ImpalaDaemon Query Fragment Query Fragment IO Manager Disk Disk Disk Disk Disk Thread 0 Thread 1 Thread 2 Thread 3 Thread 4
- 35. container format forall popular serialization formats: Avro, Thrift, Protocol Buffers
- 36. From Twitter’s “DremelMade Simple” blog The most efficient IO, is one that never happens at all
- 48. OVER PARTITION, RANK,LEAD, LAG, NTILE, .. • VARCHAR, CHAR
- 49. ROLLUP, CUBE, GROUPINGSET SET MINUS INTERSECT
- 51. SELECT question FROMaudience WHERE has_question = true;