Apache Calcite Tutorial - BOSS 21

S
Stamatis ZampetakisSoftware Developer at Cloudera
••
Tutorial @BOSS’21 Copenhagen Stamatis Zampetakis, Julian Hyde • August 16, 2021
Apache Calcite Tutorial - BOSS 21
Tutorial @BOSS’21 Copenhagen Stamatis Zampetakis, Julian Hyde • August 16, 2021 # Follow these steps to set up your environment # (The first time, it may take ~3 minutes to download dependencies.) git clone --branch boss21 https://github.com/zabetak/calcite-tutorial.git java -version # need Java 8 or higher cd calcite-tutorial ./mvnw package -DskipTests
Setup Environment Requirements 1. Git 2. JDK version ≥ 1.8 Steps 1. Clone GitHub repository 2. Load to IDE (preferred IntelliJ) a. Click Open b. Navigate to calcite-tutorial c. Select pom.xml file d. Choose “Open as Project” 3. Compile the project git clone --branch boss21 https://github.com/zabetak/calcite-tutorial.git java -version cd calcite-tutorial ./mvnw package -DskipTests
Julian Hyde @julianhyde Senior Staff Engineer @ Google / Looker Creator of Apache Calcite PMC member of Apache Arrow, Drill, Eagle, Incubator and Kylin Stamatis Zampetakis @szampetak Senior Software Engineer @ Cloudera, Hive query optimizer team PMC member of Apache Calcite; Hive committer PhD in Data Management, INRIA & Paris-Sud University About us
Outline 1. Introduction 2. CSV Adapter Demo 3. Coding module I: Main components 4. Coding module I Exercises (Homework) 5. Hybrid planning 6. Coding module II: Custom operators/rules (Homework) 7. Volcano Planner internals 8. Dialects 9. Materialized views 10. Working with spatial data 11. Research using Apache Calcite optional optional optional optional
1. Calcite introduction
1. Retrieve books and authors 2. Display image, title, price of the book along with firstname & lastname of the author 3. Sort the books based on their id (price or something else) 4. Show results in groups of five Motivation: Data views
What, where, how data are stored? author 0..1 AUTHOR id int fname string lname string birth date BOOK id int title string price decimal year int
XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN FILESYSTEM What, where, how data are stored? author 0..1 AUTHOR id int fname string lname string birth date BOOK id int title string price decimal year int
BIN What, where, how data are stored? XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN FILESYSTEM author 0..1 AUTHOR id int fname string lname string birth date BOOK id int title string price decimal year int
What, where, how data are stored? 360+ DBMS XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN FILESYSTEM author 0..1 AUTHOR id int fname string lname string birth date BOOK id int title string price decimal year int
★ Open-source search engine ★ Java library ★ Powerful indexing & search features ★ Spell checking, hit highlighting ★ Advanced analysis/tokenization capabilities ★ ACID transactions ★ Ultra compact memory/disk format Apache Lucene
How to query the data? SELECT b.id, b.title, b.year, a.fname, a.lname FROM Book b LEFT OUTER JOIN Author a ON b.author=a.id ORDER BY b.id LIMIT 5 1. Retrieve books and authors 2. Display image, title, price of the book along with firstname & lastname of the author 3. Sort the books based on their id (price or something else) 4. Show results in groups of five
Query processor architecture Query Relational Algebra Query planner CatalogReader Schema Metadata (Cost, Statistics) Execution engine Results Parser Rules Relational Algebra
Apache Calcite SQL query SqlToRelConverter RelNode RelOptPlanner CatalogReader Schema RelMetadata Provider RelBuilder RelRunner Results API calls SqlParser SqlValidator RelNode SqlNode SqlNode RelRule
Apache Calcite SQL query SqlToRelConverter RelNode RelOptPlanner CatalogReader Schema RelRule RelBuilder RelRunner Results API calls SqlParser SqlValidator RelNode SqlNode SqlNode Dev & extensions RelMetadata Provider
2. CSV Adapter Demo
Adapter Implement SchemaFactory interface Connect to a data source using parameters Extract schema - return a list of tables Push down processing to the data source: ● A set of planner rules ● Calling convention (optional) ● Query model & query generator (optional) { "schemas": [ { "name": "BOOKSTORE", "type": "custom", "factory": "org.apache.calcite.adapter.file.FileSchemaFactory", "operand": { "directory": "bookstore" } } ] ]
3. Coding module I: Main components
Setup schema & type factory <<Interface>> Schema +getTable(name: Table): Table <<Interface>> Table +getRowType(typeFactory: RelDataTypeFactory): RelDataType <<Interface>> RelDataTypeFactory +createJavaType(clazz: Class): RelDataType +createSqlType(typeName: SqlTypeName): RelDataType * tables
Apache Calcite Tutorial - BOSS 21
SELECT b.id, b.title, b.year, a.fname, a.lname FROM Book AS b LEFT OUTER JOIN Author a ON b.author = a.id WHERE b.year > 1830 ORDER BY b.id LIMIT 5 Query to Abstract Syntax Tree (AST) SQL query SqlParser SqlNode Numeric Literal 5 Identifier b.id NodeList Select OrderBy Join BasicCall As Operator Identifier author Identifier a Literal LEFT Literal ON Identifier b.author Identifier a.id Binary Operator = As Operator Identifier book Identifier b BasicCall BasicCal l Identifier b.year Numeric Literal 1830 Binary Operator > BasicCall Identifier b.id NodeList Identifier b.title Identifier b.year Identifier a.fname Identifier a.lname
AST to logical plan SqlNode Join Select b.id, b.title, b.year, a.fname, a.lname BasicCall b.author = a.id BasicCall book AS b BasicCall author AS a Literal LEFT Literal ON LogicalJoin [$3==$4,type=left] LogicalTableScan [Book] LogicalTableScan [Author] LogicalFilter [$2>1830] LogicalProject [ID=$0,TITLE=$1,YEAR=$2, FNAME=$5,LNAME=$6] LogicalSort [sort0=$0,dir=ASC,fetch=5] SqlValidator SqlToRelConverter SqlNode RelNode OrderBy b.id FETCH 5 BasicCall b.year > 1830
Logical to physical plan RelRule EnumerableSortRule EnumerableProjectRule EnumerableFilterRule EnumerableJoinRule EnumerableTableScanRule RelOptPlanner RelNode RelNode LogicalJoin [$3==$4,type=left] LogicalTableScan [Book] LogicalTableScan [Author] LogicalFilter [$2>1830] LogicalProject [ID=$0,TITLE=$1,YEAR=$2, FNAME=$5,LNAME=$6] LogicalSort [sort0=$0,dir=ASC,fetch=5] EnumerableJoin [$3==$4,type=left] EnumerableTableScan [Book] EnumerableTableScan [Author] EnumerableFilter [$2>1830] EnumerableProject [ID=$0,TITLE=$1,YEAR=$2, FNAME=$5,LNAME=$6] EnumerableSort [sort0=$0,dir=ASC,fetch=5]
Physical to Executable plan EnumerableInterpretable Java code RelNode EnumerableJoin [$3==$4,type=left] EnumerableTableScan [Book] EnumerableTableScan [Author] EnumerableFilter [$2>1830] EnumerableProject [ID=$0,TITLE=$1,YEAR=$2, FNAME=$5,LNAME=$6] EnumerableSort [sort0=$0,dir=ASC,fetch=5]
4. Coding module I: Exercises (Homework)
Exercise I: Execute more SQL queries Include GROUP BY and other types of clauses: SELECT o.o_custkey, COUNT(*) FROM orders AS o GROUP BY o.o_custkey
Exercise I: Execute more SQL queries Include GROUP BY and other types of clauses: ● Missing rule to convert LogicalAggregate to EnumerableAggregate ● Add EnumerableRules.ENUMERABLE_AGGREGATE_RULE to the planner SELECT o.o_custkey, COUNT(*) FROM orders AS o GROUP BY o.o_custkey
Push filter below the join: Exercise II: Improve performance by applying more optimization rules SELECT c.c_name, o.o_orderkey, o.o_orderdate FROM customer AS c INNER JOIN orders AS o ON c.c_custkey = o.o_custkey WHERE c.c_custkey < 3 ORDER BY c.c_name, o.o_orderkey
Push filter below the join: 1. Add rule CoreRules.FILTER_INTO_JOIN to the planner 2. Compare plans before and after (or logical and physical) 3. Check cost estimates by using SqlExplainLevel.ALL_ATTRIBUTES Exercise II: Improve performance by applying more optimization rules SELECT c.c_name, o.o_orderkey, o.o_orderdate FROM customer AS c INNER JOIN orders AS o ON c.c_custkey = o.o_custkey WHERE c.c_custkey < 3 ORDER BY c.c_name, o.o_orderkey
Exercise II: Improve performance by applying more optimization rules LogicalJoin [$0=$9,type=inner] LogicalTableScan [CUSTOMER] LogicalTableScan [ORDERS] LogicalFilter [$0<3] LogicalProject [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] LogicalSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC] RelRule FilterIntoJoinRule LogicalJoin [$0=$9,type=inner] LogicalTableScan [CUSTOMER] LogicalTableScan [ORDERS] LogicalFilter [$0<3] LogicalProject [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] LogicalSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC]
Open LuceneBuilderProcessor.java and complete TODOs Exercise III: Use RelBuilder API to construct the logical plan SELECT o.o_custkey, COUNT(*) FROM orders AS o GROUP BY o.o_custkey SELECT o.o_custkey, COUNT(*) FROM orders AS o WHERE o.o_totalprice > 220388.06 GROUP BY o.o_custkey Q1: Q2:
Exercise III: Use RelBuilder API to construct the logical plan builder .scan("orders") .filter( builder.call( SqlStdOperatorTable.GREATER_THAN, builder.field("o_totalprice"), builder.literal(220388.06))) .aggregate( builder.groupKey("o_custkey"), builder.count());
5. Hybrid planning
Calling convention Scan Scan Join Filter Join Scan Initially all nodes belong to “logical” calling convention. Logical calling convention cannot be implemented, so has infinite cost
Calling convention Scan Scan Join Filter Join Scan Tables can’t be moved so there is only one choice of calling convention for each table. Examples: ● Enumerable ● Druid ● Drill ● HBase ● JDBC
Calling convention Scan Scan Join Filter Join Scan Rules fire to convert nodes to particular calling conventions. The calling convention propagates through the tree. Because this is Volcano, each node can have multiple conventions.
Calling convention Scan Scan Join Filter Join Scan Rules fire to convert nodes to particular calling conventions. The calling convention propagates through the tree. Because this is Volcano, each node can have multiple conventions.
Calling convention Scan Scan Join Filter Join Scan Rules fire to convert nodes to particular calling conventions. The calling convention propagates through the tree. Because this is Volcano, each node can have multiple conventions.
Converters Scan Scan Join Filter Join To keep things honest, we need to insert a converter at each point where the convention changes. (Recall: Volcano has an enforcer for each trait. Convention is a physical property, and converter is the enforcer.) Blue to Logical BlueFilterRule: LogicalFilter(BlueToLogical(Blue b)) → BlueToLogical(BlueFilter(b)) Green to Logical Scan
Converters Scan Scan Join Join Green to Logical To keep things honest, we need to insert a converter at each point where the convention changes. (Recall: Volcano has an enforcer for each trait. Convention is a physical property, and converter is the enforcer.) Blue to Logical Filter BlueFilterRule: LogicalFilter(BlueToLogical(Blue b)) → BlueToLogical(BlueFilter(b)) Scan
Generating programs to implement hybrid plans Hybrid plans are glued together using an engine - a convention that does not have a storage format. (Example engines: Drill, Spark, Presto.) To implement, we generate a program that calls out to query1 and query2. The "Blue-to-Orange" converter is typically a function in the Orange language that embeds a Blue query. Similarly "Green-to-Orange". Scan Scan Join Join Green to Orange Blue to Orange Filter Scan
6. Coding module II: Custom operators/rules (Homework)
What we want to achieve? EnumerableJoin [$0=$9,type=inner] EnumerableTableScan [CUSTOMER] EnumerableCalc [$0<3] EnumerableCalc [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] EnumerableSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC] EnumerableTableScan [ORDERS] EnumerableJoin [$0=$9,type=inner] LuceneTableScan [CUSTOMER] LuceneFilter [$0<3] EnumerableCalc [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] EnumerableSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC] LuceneTableScan [ORDERS] LuceneToEnumerableConverter LuceneToEnumerableConverter
What do we need? EnumerableJoin [$0=$9,type=inner] LuceneTableScan [CUSTOMER] LuceneFilter [$0<3] EnumerableCalc [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] EnumerableSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC] LuceneTableScan [ORDERS] LuceneToEnumerableConverter LuceneToEnumerableConverter Two calling conventions: 1. Enumerable 2. Lucene Three custom operators: 1. LuceneTableScan 2. LuceneToEnumerableConverter 3. LuceneFilter Three custom conversion rules: 1. LogicalTableScan → LuceneTableScan 2. LogicalFilter → LuceneFilter 3. LuceneANY → LuceneToEnumerableConverter
What do we need? EnumerableJoin [$0=$9,type=inner] LuceneTableScan [CUSTOMER] LuceneFilter [$0<3] EnumerableCalc [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] EnumerableSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC] LuceneTableScan [ORDERS] LuceneToEnumerableConverter LuceneToEnumerableConverter Two calling conventions: 1. Enumerable 2. Lucene Three custom operators: 1. LuceneTableScan 2. LuceneToEnumerableConverter 3. LuceneFilter Three custom conversion rules: 1. LogicalTableScan → LuceneTableScan 2. LogicalFilter → LuceneFilter 3. LuceneANY → LuceneToEnumerableConverter STEP 1 STEP 2 STEP 3 STEP 4 STEP 5 STEP 6
7. Volcano Planner Internals
Volcano planning algorithm Based on two papers by Goetz Graefe in the 1990s (Volcano, Cascades), now the industry standard for cost-based optimization. Dynamic programming: to optimize a relational expression R0 , convert it into equivalent expressions {R1 , R2 , …}, and pick the one with the lowest cost. Much of the cost of R is the cost of its input(s). So we apply dynamic programming to its inputs, too. R0 S0 T0
Volcano planning algorithm Based on two papers by Goetz Graefe in the 1990s (Volcano, Cascades), now the industry standard for cost-based optimization. Dynamic programming: to optimize a relational expression R0 , convert it into equivalent expressions {R1 , R2 , …}, and pick the one with the lowest cost. Much of the cost of R is the cost of its input(s). So we apply dynamic programming to its inputs, too. R0 S0 R1 T0
Volcano planning algorithm Based on two papers by Goetz Graefe in the 1990s (Volcano, Cascades), now the industry standard for cost-based optimization. Dynamic programming: to optimize a relational expression R0 , convert it into equivalent expressions {R1 , R2 , …}, and pick the one with the lowest cost. Much of the cost of R is the cost of its input(s). So we apply dynamic programming to its inputs, too. R0 S0 R1 T0 T1
Volcano planning algorithm Based on two papers by Goetz Graefe in the 1990s (Volcano, Cascades), now the industry standard for cost-based optimization. Dynamic programming: to optimize a relational expression R0 , convert it into equivalent expressions {R1 , R2 , …}, and pick the one with the lowest cost. Much of the cost of R is the cost of its input(s). So we apply dynamic programming to its inputs, too. R0 S0 R1 T0 T1 R2 Uo
Volcano planning algorithm We keep equivalence sets of expressions (class RelSet). Each input of a relational expression is an equivalence set + required physical properties (class RelSubset). R0 S0 R1 T0 T1 R2 Uo
Volcano planning algorithm Each relational expression has a memo (digest), so we will recognize it if we generate it again. R0 S0 R1 T0 T1 R2 Uo
Volcano planning algorithm If an expression transforms to an expression in another equivalence set, we can merge those equivalence sets. R0 S0 R1 T0 T1 R2 Uo
Matches and queues We register a new RelNode by adding it to a RelSet. Each rule instance declares a pattern of RelNode types (and other properties) that it will match. Suppose we have: ● Filter-on-Project ● Project-on-Project ● Project-on-Join On register, we detect rules that are newly matched. Project Filter Union Join Scan Project
Matches and queues We register a new RelNode by adding it to a RelSet. Each rule instance declares a pattern of RelNode types (and other properties) that it will match. Suppose we have: ● Filter-on-Project ● Project-on-Project ● Project-on-Join On register, we detect rules that are newly matched. Project Filter Union Project Join Scan Project
Matches and queues We register a new RelNode by adding it to a RelSet. Each rule instance declares a pattern of RelNode types (and other properties) that it will match. Suppose we have: ● Filter-on-Project ● Project-on-Project ● Project-on-Join On register, we detect rules that are newly matched. (4 matches.) Project Filter Union Project Join Scan Project
Matches and queues Should we fire these matched rules immediately? No! Because rule match #1 would generate new matches… which would generate new matches… and we'd never get to match #2. Instead, we put the matched rules on a queue. The queue allows us to: ● Search breadth-first (rather than depth-first) ● Prioritize (fire more "important" rules first) ● Potentially terminate when we have a "good enough" plan Equivalence sets containing registered RelNodes 0. Register each RelNode in the initial tree. Rule match queue 1. Each time a RelNode is registered, find rule matches, and put RuleMatch objects on the queue. 3. Pop the top rule match. Fire the rule. Register each RelNode generated by the rule, merging sets if equivalences are found. Goto 1. 2. If the queue is empty, or the cost is good enough, we're done.
Other planner engines, same great rules Three planner engines: ● Volcano ● Volcano top-down (Cascades style) ● Hep applies rules in a strict "program" The same rules are used by all engines. It takes a lot of time effort to write a high-quality rule. Rules can be reused, tested, improved, and they compose with other rules. Calcite's library of rules is valuable.
8. Dialects
Calcite architecture Apache Calcite JDBC adapter Pluggable rewrite rules Enumerable adapter MongoDB adapter File adapter (CSV, JSON, Http) Apache Kafka adapter Apache Spark adapter SQL SQL parser & validator Query planner Relational algebra At what points in the Calcite stack do ‘languages’ exist? ● Incoming SQL ● Validating SQL against built-in operators ● Type system (e.g. max size of INTEGER type) ● JDBC adapter generates SQL ● Other adapters generate other languages
Parsing & validating SQL - what knobs can I turn? SELECT deptno AS d, SUM(sal) AS [sumSal] FROM [HR].[Emp] WHERE ename NOT ILIKE "A%" GROUP BY d ORDER BY 1, 2 DESC SqlConformance.isGroupByAlias() = true SqlConformance.isSortByOrdinal() = true Lex.quoting = Quoting.BRACKET Lex.charLiteralStyle = CharLiteralStyle.BQ_DOUBLE SqlValidator.Config.defaultNullCollation = HIGH Lex.unquotedCasing = Casing.TO_UPPER Lex.quotedCasing = Casing.UNCHANGED PARSER_FACTORY = "org.apache.calcite.sql.parser.impl.SqlParserImpl.FACTORY" FUN = "postgres" (ILIKE is not standard SQL)
SQL dialect - APIs and properties Pluggable rewrite rules Pluggable parser, lexical, conformance, operators Pluggable SQL dialect SQL SQL SQL parser & validator Query planner Relational algebra JDBC adapter interface SqlParserImplFactory CalciteConnectionProperty.LEX enum Lex enum Quoting enum Casing enum CharLiteralStyle CalciteConnectionProperty.CONFORMANCE interface SqlConformance CalciteConnectionProperty.FUN interface SqlOperatorTable class SqlStdOperatorTable class SqlLibraryOperators class SqlOperator class SqlFunction extends SqlOperator class SqlAggFunction extends SqlFunction class RelRule class SqlDialect interface SqlDialectFactory
Contributing a dialect (or anything!) to Calcite For your first code contribution, pick a small bug or feature. Introduce yourself! Email dev@, saying what you plan to do. Create a JIRA case describing the problem. To understand the code, find similar features. Run their tests in a debugger. Write 1 or 2 tests for your feature. Submit a pull request (PR).
Other front-end languages Pig RelBuilder Adapter Physical operators Morel Storage Query planner Relational algebra Datalog SQL parser & validator SQL Calcite is an excellent platform for implementing your own data language Write a parser for your language, use RelBuilder to translate to relational algebra, and you can use any of Calcite's back-end implementations
9. Materialized views
Forwards planning Backwards planning Backwards planning Until now, we have seen forward planning. Forward planning transforms an expression (R0 ) to many equivalent forms and picks the one with lowest cost (Ropt ). Backwards planning transforms an expression to an equivalent form (RN ) that contains a target expression (T). R0 R0 T Ropt
Forwards planning Backwards planning Equivalence set Backwards planning Ropt R2 R1 R0 Equivalence set RN R2 R1 R0 T S Until now, we have seen forward planning. Forward planning transforms an expression (R0 ) to many equivalent forms and picks the one with lowest cost (Ropt ). Backwards planning transforms an expression to an equivalent form (RN ) that contains a target expression (T).
Applications of backwards planning Indexes (e.g. b-tree indexes). An index is a derived data structure whose contents can be described as a relational expression (generally project-sort). When we are planning a query, it already exists (i.e. the cost has already been paid). Summary tables. A summary table is a derived data structure (generally filter-project-join-aggregate). Replicas with different physical properties (e.g. copy the table from New York to Tokyo, or copy the table and partition by month(orderDate), sort by productId). Incremental view maintenance. Materialized view V is populated from base table T. Yesterday, we populated V with V0 = Q(T0 ). Today we want to make its contents equal to V1 = Q(T1 ). Can we find and apply a delta query, dQ = Q(T1 - T0 )?
Materialized views in Calcite { "schemas": { "name": "HR", "tables": [ { "name": "emp" } ], "materializations": [ { "table": "i_emp_job", "sql": "SELECT job, empno FROM emp ORDER BY job, empno" }, { "table": "add_emp_deptno", "sql": "SELECT deptno, SUM(sal) AS ss, COUNT(*) AS c FROM emp GROUP BY deptno" } ] } } /** Transforms a relational expression into a * semantically equivalent relational expression, * according to a given set of rules and a cost * model. */ public interface RelOptPlanner { /** Defines an equivalence between a table and * a query. */ void addMaterialization( RelOptMaterialization materialization); /** Finds the most efficient expression to * implement this query. */ RelNode findBestExp(); } /** Records that a particular query is materialized * by a particular table. */ public class RelOptMaterialization { public final RelNode tableRel; public final List<String> qualifiedTableName; public final RelNode queryRel; } You can define materializations in a JSON model, via the planner API, or via CREATE MATERIALIZED VIEW DDL (not shown).
More about materialized views ● There are several algorithms to rewrite queries to match materialized views ● A lattice is a data structure to model a star schema ● Calcite has algorithms to recommend an optimal set of summary tables for a lattice (given expected queries, and statistics about column cardinality) ● Data profiling algorithms estimate the cardinality of all combinations of columns
10. Working with spatial data
Spatial query Find all restaurants within 1.5 distance units of my current location: We cannot use a B-tree index (it can sort points by x or y coordinates, but not both) and specialized spatial indexes (such as R*-trees) are not generally available. restaurant x y Zachary’s pizza 3 1 King Yen 7 7 Filippo’s 7 4 Station burger 5 6 SELECT * FROM Restaurants AS r WHERE ST_Distance( ST_MakePoint(r.x, r.y), ST_MakePoint(6, 7)) < 1.5 • • • • Zachary’s pizza Filippo’s King Yen Station burger
Hilbert space-filling curve ● A space-filling curve invented by mathematician David Hilbert ● Every (x, y) point has a unique position on the curve ● Points near to each other typically have Hilbert indexes close together
• • • • Add restriction based on h, a restaurant’s distance along the Hilbert curve Must keep original restriction due to false positives Using Hilbert index restaurant x y h Zachary’s pizza 3 1 5 King Yen 7 7 41 Filippo’s 7 4 52 Station burger 5 6 36 Zachary’s pizza Filippo’s SELECT * FROM Restaurants AS r WHERE (r.h BETWEEN 35 AND 42 OR r.h BETWEEN 46 AND 46) AND ST_Distance( ST_MakePoint(r.x, r.y), ST_MakePoint(6, 7)) < 1.5 King Yen Station burger
Telling the optimizer 1. Declare h as a generated column 2. Sort table by h Planner can now convert spatial range queries into a range scan Does not require specialized spatial index such as R*-tree Very efficient on a sorted table such as HBase There are similar techniques for other spatial patterns (e.g. region-to-region join) CREATE TABLE Restaurants ( restaurant VARCHAR(20), x DOUBLE, y DOUBLE, h DOUBLE GENERATED ALWAYS AS ST_Hilbert(x, y) STORED) SORT KEY (h); restaurant x y h Zachary’s pizza 3 1 5 Station burger 5 6 36 King Yen 7 7 41 Filippo’s 7 4 52
11. Research using Apache Calcite
Apache Calcite Tutorial - BOSS 21
Yes, VLDB 2021! Go to the talk! 0900 Wednesday.
Apache Calcite Tutorial - BOSS 21
Thank you! @julianhyde @szampetak https://calcite.apache.org
Resources ● Calcite project https://calcite.apache.org ● Materialized view algorithms https://calcite.apache.org/docs/materialized_views.html ● JSON model https://calcite.apache.org/docs/model.html ● Lazy beats smart and fast (DataEng 2018) - MVs, spatial, profiling https://www.slideshare.net/julianhyde/lazy-beats-smart-and-fast ● Efficient spatial queries on vanilla databases (ApacheCon 2018) https://www.slideshare.net/julianhyde/spatial-query-on-vanilla-databases ● Graefe, McKenna. The Volcano Optimizer Generator, 1991 ● Graefe. The Cascades Framework for Query Optimization, 1995 ● Slideshare (past presentations by Julian Hyde, including several about Apache Calcite) https://www.slideshare.net/julianhyde
1 of 83

Apache Calcite Tutorial - BOSS 21

••
Download to read offline
Engineering

Apache Calcite is a dynamic data management framework. Think of it as a toolkit for building databases: it has an industry-standard SQL parser, validator, highly customizable optimizer (with pluggable transformation rules and cost functions, relational algebra, and an extensive library of rules), but it has no preferred storage primitives. In this tutorial, the attendees will use Apache Calcite to build a fully fledged query processor from scratch with very few lines of code. This processor is a full implementation of SQL over an Apache Lucene storage engine. (Lucene does not support SQL queries and lacks a declarative language for performing complex operations such as joins or aggregations.) Attendees will also learn how to use Calcite as an effective tool for research.

S
Stamatis ZampetakisSoftware Developer at Cloudera

Recommended

Apache Calcite (a tutorial given at BOSS '21) by
Apache Calcite (a tutorial given at BOSS '21)Apache Calcite (a tutorial given at BOSS '21)
Apache Calcite (a tutorial given at BOSS '21)Julian Hyde
2.6K views•88 slides
Apache Calcite: A Foundational Framework for Optimized Query Processing Over ... by
Apache Calcite: A Foundational Framework for Optimized Query Processing Over ...Apache Calcite: A Foundational Framework for Optimized Query Processing Over ...
Apache Calcite: A Foundational Framework for Optimized Query Processing Over ...Julian Hyde
2.3K views•23 slides
Introduction to Apache Calcite by
Introduction to Apache CalciteIntroduction to Apache Calcite
Introduction to Apache CalciteJordan Halterman
16.3K views•109 slides
Apache Calcite overview by
Apache Calcite overviewApache Calcite overview
Apache Calcite overviewJulian Hyde
19.5K views•9 slides
Apache Calcite: One planner fits all by
Apache Calcite: One planner fits allApache Calcite: One planner fits all
Apache Calcite: One planner fits allJulian Hyde
6.7K views•10 slides
The evolution of Apache Calcite and its Community by
The evolution of Apache Calcite and its CommunityThe evolution of Apache Calcite and its Community
The evolution of Apache Calcite and its CommunityJulian Hyde
330 views•26 slides

More Related Content

What's hot

Apache Calcite: One Frontend to Rule Them All by
Apache Calcite: One Frontend to Rule Them AllApache Calcite: One Frontend to Rule Them All
Apache Calcite: One Frontend to Rule Them AllMichael Mior
675 views•44 slides
Using Apache Calcite for Enabling SQL and JDBC Access to Apache Geode and Oth... by
Using Apache Calcite for Enabling SQL and JDBC Access to Apache Geode and Oth...Using Apache Calcite for Enabling SQL and JDBC Access to Apache Geode and Oth...
Using Apache Calcite for Enabling SQL and JDBC Access to Apache Geode and Oth...Christian Tzolov
3K views•40 slides
Data all over the place! How SQL and Apache Calcite bring sanity to streaming... by
Data all over the place! How SQL and Apache Calcite bring sanity to streaming...Data all over the place! How SQL and Apache Calcite bring sanity to streaming...
Data all over the place! How SQL and Apache Calcite bring sanity to streaming...Julian Hyde
4K views•56 slides
Parquet performance tuning: the missing guide by
Parquet performance tuning: the missing guideParquet performance tuning: the missing guide
Parquet performance tuning: the missing guideRyan Blue
40.5K views•44 slides
Cost-based Query Optimization in Apache Phoenix using Apache Calcite by
Cost-based Query Optimization in Apache Phoenix using Apache CalciteCost-based Query Optimization in Apache Phoenix using Apache Calcite
Cost-based Query Optimization in Apache Phoenix using Apache CalciteJulian Hyde
8K views•32 slides
SQL for NoSQL and how Apache Calcite can help by
SQL for NoSQL and how Apache Calcite can helpSQL for NoSQL and how Apache Calcite can help
SQL for NoSQL and how Apache Calcite can helpChristian Tzolov
1.2K views•34 slides

What's hot(20)

Apache Calcite: One Frontend to Rule Them All by Michael Mior
Apache Calcite: One Frontend to Rule Them AllApache Calcite: One Frontend to Rule Them All
Apache Calcite: One Frontend to Rule Them All
Michael Mior•675 views
Using Apache Calcite for Enabling SQL and JDBC Access to Apache Geode and Oth... by Christian Tzolov
Using Apache Calcite for Enabling SQL and JDBC Access to Apache Geode and Oth...Using Apache Calcite for Enabling SQL and JDBC Access to Apache Geode and Oth...
Using Apache Calcite for Enabling SQL and JDBC Access to Apache Geode and Oth...
Christian Tzolov•3K views
Data all over the place! How SQL and Apache Calcite bring sanity to streaming... by Julian Hyde
Data all over the place! How SQL and Apache Calcite bring sanity to streaming...Data all over the place! How SQL and Apache Calcite bring sanity to streaming...
Data all over the place! How SQL and Apache Calcite bring sanity to streaming...
Julian Hyde•4K views
Parquet performance tuning: the missing guide by Ryan Blue
Parquet performance tuning: the missing guideParquet performance tuning: the missing guide
Parquet performance tuning: the missing guide
Ryan Blue•40.5K views
Cost-based Query Optimization in Apache Phoenix using Apache Calcite by Julian Hyde
Cost-based Query Optimization in Apache Phoenix using Apache CalciteCost-based Query Optimization in Apache Phoenix using Apache Calcite
Cost-based Query Optimization in Apache Phoenix using Apache Calcite
Julian Hyde•8K views
SQL for NoSQL and how Apache Calcite can help by Christian Tzolov
SQL for NoSQL and how Apache Calcite can helpSQL for NoSQL and how Apache Calcite can help
SQL for NoSQL and how Apache Calcite can help
Christian Tzolov•1.2K views
Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in... by Julian Hyde
Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in...Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in...
Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in...
Julian Hyde•1.6K views
Tricks every ClickHouse designer should know, by Robert Hodges, Altinity CEO by Altinity Ltd
Tricks every ClickHouse designer should know, by Robert Hodges, Altinity CEOTricks every ClickHouse designer should know, by Robert Hodges, Altinity CEO
Tricks every ClickHouse designer should know, by Robert Hodges, Altinity CEO
Altinity Ltd•13.6K views
All About JSON and ClickHouse - Tips, Tricks and New Features-2022-07-26-FINA... by Altinity Ltd
All About JSON and ClickHouse - Tips, Tricks and New Features-2022-07-26-FINA...All About JSON and ClickHouse - Tips, Tricks and New Features-2022-07-26-FINA...
All About JSON and ClickHouse - Tips, Tricks and New Features-2022-07-26-FINA...
Altinity Ltd•1.2K views
Open Source SQL - beyond parsers: ZetaSQL and Apache Calcite by Julian Hyde
Open Source SQL - beyond parsers: ZetaSQL and Apache CalciteOpen Source SQL - beyond parsers: ZetaSQL and Apache Calcite
Open Source SQL - beyond parsers: ZetaSQL and Apache Calcite
Julian Hyde•1.1K views
The Parquet Format and Performance Optimization Opportunities by Databricks
The Parquet Format and Performance Optimization OpportunitiesThe Parquet Format and Performance Optimization Opportunities
The Parquet Format and Performance Optimization Opportunities
Databricks•8.2K views
Migration to ClickHouse. Practical guide, by Alexander Zaitsev by Altinity Ltd
Migration to ClickHouse. Practical guide, by Alexander ZaitsevMigration to ClickHouse. Practical guide, by Alexander Zaitsev
Migration to ClickHouse. Practical guide, by Alexander Zaitsev
Altinity Ltd•9.4K views
Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang by Databricks
Apache Spark Data Source V2 with Wenchen Fan and Gengliang WangApache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
Databricks•5.9K views
Parquet Strata/Hadoop World, New York 2013 by Julien Le Dem
Parquet Strata/Hadoop World, New York 2013Parquet Strata/Hadoop World, New York 2013
Parquet Strata/Hadoop World, New York 2013
Julien Le Dem•133K views
Advanced Postgres Monitoring by Denish Patel
Advanced Postgres MonitoringAdvanced Postgres Monitoring
Advanced Postgres Monitoring
Denish Patel•9.6K views
A Deep Dive into Query Execution Engine of Spark SQL by Databricks
A Deep Dive into Query Execution Engine of Spark SQLA Deep Dive into Query Execution Engine of Spark SQL
A Deep Dive into Query Execution Engine of Spark SQL
Databricks•6.6K views
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdf by Altinity Ltd
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdfDeep Dive on ClickHouse Sharding and Replication-2202-09-22.pdf
Deep Dive on ClickHouse Sharding and Replication-2202-09-22.pdf
Altinity Ltd•3.2K views
Adding measures to Calcite SQL by Julian Hyde
Adding measures to Calcite SQLAdding measures to Calcite SQL
Adding measures to Calcite SQL
Julian Hyde•394 views
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO by Altinity Ltd
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEOClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
Altinity Ltd•22.4K views
Materialize: a platform for changing data by Altinity Ltd
Materialize: a platform for changing dataMaterialize: a platform for changing data
Materialize: a platform for changing data
Altinity Ltd•281 views

Similar to Apache Calcite Tutorial - BOSS 21

Iac d.damyanov 4.pptx by
Iac d.damyanov 4.pptxIac d.damyanov 4.pptx
Iac d.damyanov 4.pptxDimitar Damyanov
445 views•33 slides
Incremental View Maintenance with Coral, DBT, and Iceberg by
Incremental View Maintenance with Coral, DBT, and IcebergIncremental View Maintenance with Coral, DBT, and Iceberg
Incremental View Maintenance with Coral, DBT, and IcebergWalaa Eldin Moustafa
599 views•57 slides
Apache DeltaSpike the CDI toolbox by
Apache DeltaSpike the CDI toolboxApache DeltaSpike the CDI toolbox
Apache DeltaSpike the CDI toolboxAntoine Sabot-Durand
7.5K views•36 slides
Apache DeltaSpike: The CDI Toolbox by
Apache DeltaSpike: The CDI ToolboxApache DeltaSpike: The CDI Toolbox
Apache DeltaSpike: The CDI ToolboxVirtual JBoss User Group
4.2K views•36 slides
Data Exploration with Apache Drill: Day 1 by
Data Exploration with Apache Drill: Day 1Data Exploration with Apache Drill: Day 1
Data Exploration with Apache Drill: Day 1Charles Givre
1.1K views•139 slides
Webinar: What's new in the .NET Driver by
Webinar: What's new in the .NET DriverWebinar: What's new in the .NET Driver
Webinar: What's new in the .NET DriverMongoDB
2.1K views•50 slides

Similar to Apache Calcite Tutorial - BOSS 21(20)

Iac d.damyanov 4.pptx by Dimitar Damyanov
Iac d.damyanov 4.pptxIac d.damyanov 4.pptx
Iac d.damyanov 4.pptx
Dimitar Damyanov•445 views
Incremental View Maintenance with Coral, DBT, and Iceberg by Walaa Eldin Moustafa
Incremental View Maintenance with Coral, DBT, and IcebergIncremental View Maintenance with Coral, DBT, and Iceberg
Incremental View Maintenance with Coral, DBT, and Iceberg
Walaa Eldin Moustafa•599 views
Apache DeltaSpike the CDI toolbox by Antoine Sabot-Durand
Apache DeltaSpike the CDI toolboxApache DeltaSpike the CDI toolbox
Apache DeltaSpike the CDI toolbox
Antoine Sabot-Durand•7.5K views
Apache DeltaSpike: The CDI Toolbox by Virtual JBoss User Group
Apache DeltaSpike: The CDI ToolboxApache DeltaSpike: The CDI Toolbox
Apache DeltaSpike: The CDI Toolbox
Virtual JBoss User Group•4.2K views
Data Exploration with Apache Drill: Day 1 by Charles Givre
Data Exploration with Apache Drill: Day 1Data Exploration with Apache Drill: Day 1
Data Exploration with Apache Drill: Day 1
Charles Givre•1.1K views
Webinar: What's new in the .NET Driver by MongoDB
Webinar: What's new in the .NET DriverWebinar: What's new in the .NET Driver
Webinar: What's new in the .NET Driver
MongoDB•2.1K views
6 tips for improving ruby performance by Engine Yard
6 tips for improving ruby performance6 tips for improving ruby performance
6 tips for improving ruby performance
Engine Yard•6.6K views
Bye bye $GLOBALS['TYPO3_DB'] by Jan Helke
Bye bye $GLOBALS['TYPO3_DB']Bye bye $GLOBALS['TYPO3_DB']
Bye bye $GLOBALS['TYPO3_DB']
Jan Helke•214 views
Scale By The Bay | 2020 | Gimel by Deepak Chandramouli
Scale By The Bay | 2020 | GimelScale By The Bay | 2020 | Gimel
Scale By The Bay | 2020 | Gimel
Deepak Chandramouli•183 views
LinuxCon 2013 Steven Dake on Using Heat for autoscaling OpenShift on Openstack by OpenShift Origin
LinuxCon 2013 Steven Dake on Using Heat for autoscaling OpenShift on OpenstackLinuxCon 2013 Steven Dake on Using Heat for autoscaling OpenShift on Openstack
LinuxCon 2013 Steven Dake on Using Heat for autoscaling OpenShift on Openstack
OpenShift Origin•1.4K views
22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl... by Athens Big Data
22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl...22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl...
22nd Athens Big Data Meetup - 1st Talk - MLOps Workshop: The Full ML Lifecycl...
Athens Big Data•29 views
The Ring programming language version 1.8 book - Part 95 of 202 by Mahmoud Samir Fayed
The Ring programming language version 1.8 book - Part 95 of 202The Ring programming language version 1.8 book - Part 95 of 202
The Ring programming language version 1.8 book - Part 95 of 202
Mahmoud Samir Fayed•16 views
Document validation in MongoDB 3.2 by Andrew Morgan
Document validation in MongoDB 3.2Document validation in MongoDB 3.2
Document validation in MongoDB 3.2
Andrew Morgan•20.7K views
Database Development Replication Security Maintenance Report by nyin27
Database Development Replication Security Maintenance ReportDatabase Development Replication Security Maintenance Report
Database Development Replication Security Maintenance Report
nyin27•420 views
What's New in MariaDB Server 10.2 and MariaDB MaxScale 2.1 by MariaDB plc
What's New in MariaDB Server 10.2 and MariaDB MaxScale 2.1What's New in MariaDB Server 10.2 and MariaDB MaxScale 2.1
What's New in MariaDB Server 10.2 and MariaDB MaxScale 2.1
MariaDB plc•160 views
What's New in MariaDB Server 10.2 and MariaDB MaxScale 2.1 by MariaDB plc
What's New in MariaDB Server 10.2 and MariaDB MaxScale 2.1What's New in MariaDB Server 10.2 and MariaDB MaxScale 2.1
What's New in MariaDB Server 10.2 and MariaDB MaxScale 2.1
MariaDB plc•73 views
Evaluating Your Learning to Rank Model: Dos and Don’ts in Offline/Online Eval... by Sease
Evaluating Your Learning to Rank Model: Dos and Don’ts in Offline/Online Eval...Evaluating Your Learning to Rank Model: Dos and Don’ts in Offline/Online Eval...
Evaluating Your Learning to Rank Model: Dos and Don’ts in Offline/Online Eval...
Sease•431 views
TYPO3 + CKEditor: Heaven for TYPO3 Developer & Editor by NITSAN Technologies
TYPO3 + CKEditor: Heaven for TYPO3 Developer & EditorTYPO3 + CKEditor: Heaven for TYPO3 Developer & Editor
TYPO3 + CKEditor: Heaven for TYPO3 Developer & Editor
NITSAN Technologies•444 views
Testing the frontend by Heiko Hardt
Testing the frontendTesting the frontend
Testing the frontend
Heiko Hardt•284 views
Analytics Metrics delivery and ML Feature visualization: Evolution of Data Pl... by Chester Chen
Analytics Metrics delivery and ML Feature visualization: Evolution of Data Pl...Analytics Metrics delivery and ML Feature visualization: Evolution of Data Pl...
Analytics Metrics delivery and ML Feature visualization: Evolution of Data Pl...
Chester Chen•339 views

Recently uploaded

Design of Structures and Foundations for Vibrating Machines, Arya-ONeill-Pinc... by
Design of Structures and Foundations for Vibrating Machines, Arya-ONeill-Pinc...Design of Structures and Foundations for Vibrating Machines, Arya-ONeill-Pinc...
Design of Structures and Foundations for Vibrating Machines, Arya-ONeill-Pinc...csegroupvn
5 views•210 slides
Effect of deep chemical mixing columns on properties of surrounding soft clay... by
Effect of deep chemical mixing columns on properties of surrounding soft clay...Effect of deep chemical mixing columns on properties of surrounding soft clay...
Effect of deep chemical mixing columns on properties of surrounding soft clay...AltinKaradagli
9 views•10 slides
Searching in Data Structure by
Searching in Data StructureSearching in Data Structure
Searching in Data Structureraghavbirla63
7 views•8 slides
MK__Cert.pdf by
MK__Cert.pdfMK__Cert.pdf
MK__Cert.pdfHassan Khan
11 views•1 slide
fakenews_DBDA_Mar23.pptx by
fakenews_DBDA_Mar23.pptxfakenews_DBDA_Mar23.pptx
fakenews_DBDA_Mar23.pptxdeepmitra8
15 views•34 slides
MSA Website Slideshow (16).pdf by
MSA Website Slideshow (16).pdfMSA Website Slideshow (16).pdf
MSA Website Slideshow (16).pdfmsaucla
76 views•8 slides

Recently uploaded(20)

Design of Structures and Foundations for Vibrating Machines, Arya-ONeill-Pinc... by csegroupvn
Design of Structures and Foundations for Vibrating Machines, Arya-ONeill-Pinc...Design of Structures and Foundations for Vibrating Machines, Arya-ONeill-Pinc...
Design of Structures and Foundations for Vibrating Machines, Arya-ONeill-Pinc...
csegroupvn•5 views
Effect of deep chemical mixing columns on properties of surrounding soft clay... by AltinKaradagli
Effect of deep chemical mixing columns on properties of surrounding soft clay...Effect of deep chemical mixing columns on properties of surrounding soft clay...
Effect of deep chemical mixing columns on properties of surrounding soft clay...
AltinKaradagli•9 views
Searching in Data Structure by raghavbirla63
Searching in Data StructureSearching in Data Structure
Searching in Data Structure
raghavbirla63•7 views
MK__Cert.pdf by Hassan Khan
MK__Cert.pdfMK__Cert.pdf
MK__Cert.pdf
Hassan Khan•11 views
fakenews_DBDA_Mar23.pptx by deepmitra8
fakenews_DBDA_Mar23.pptxfakenews_DBDA_Mar23.pptx
fakenews_DBDA_Mar23.pptx
deepmitra8•15 views
MSA Website Slideshow (16).pdf by msaucla
MSA Website Slideshow (16).pdfMSA Website Slideshow (16).pdf
MSA Website Slideshow (16).pdf
msaucla•76 views
Design of machine elements-UNIT 3.pptx by gopinathcreddy
Design of machine elements-UNIT 3.pptxDesign of machine elements-UNIT 3.pptx
Design of machine elements-UNIT 3.pptx
gopinathcreddy•32 views
Final Year Presentation by Comsat Universal Islamabad Wah Campus
Final Year PresentationFinal Year Presentation
Final Year Presentation
Comsat Universal Islamabad Wah Campus•9 views
SUMIT SQL PROJECT SUPERSTORE 1.pptx by Sumit Jadhav
SUMIT SQL PROJECT SUPERSTORE 1.pptxSUMIT SQL PROJECT SUPERSTORE 1.pptx
SUMIT SQL PROJECT SUPERSTORE 1.pptx
Sumit Jadhav •15 views
CHEMICAL KINETICS.pdf by AguedaGutirrez
CHEMICAL KINETICS.pdfCHEMICAL KINETICS.pdf
CHEMICAL KINETICS.pdf
AguedaGutirrez•13 views
K8S Roadmap.pdf by MaryamTavakkoli2
K8S Roadmap.pdfK8S Roadmap.pdf
K8S Roadmap.pdf
MaryamTavakkoli2•8 views
Investor Presentation by eser sevinç
Investor PresentationInvestor Presentation
Investor Presentation
eser sevinç•25 views
Pull down shoulder press final report docx (1).pdf by Comsat Universal Islamabad Wah Campus
Pull down shoulder press final report docx (1).pdfPull down shoulder press final report docx (1).pdf
Pull down shoulder press final report docx (1).pdf
Comsat Universal Islamabad Wah Campus•17 views
2023Dec ASU Wang NETR Group Research Focus and Facility Overview.pptx by lwang78
2023Dec ASU Wang NETR Group Research Focus and Facility Overview.pptx2023Dec ASU Wang NETR Group Research Focus and Facility Overview.pptx
2023Dec ASU Wang NETR Group Research Focus and Facility Overview.pptx
lwang78•83 views
START Newsletter 3 by Start Project
START Newsletter 3START Newsletter 3
START Newsletter 3
Start Project•5 views
What is Unit Testing by Sadaaki Emura
What is Unit TestingWhat is Unit Testing
What is Unit Testing
Sadaaki Emura•24 views
sam_software_eng_cv.pdf by sammyigbinovia
sam_software_eng_cv.pdfsam_software_eng_cv.pdf
sam_software_eng_cv.pdf
sammyigbinovia•5 views
Codes and Conventions.pptx by IsabellaGraceAnkers
Codes and Conventions.pptxCodes and Conventions.pptx
Codes and Conventions.pptx
IsabellaGraceAnkers•9 views
SPICE PARK DEC2023 (6,625 SPICE Models) by Tsuyoshi Horigome
SPICE PARK DEC2023 (6,625 SPICE Models) SPICE PARK DEC2023 (6,625 SPICE Models)
SPICE PARK DEC2023 (6,625 SPICE Models)
Tsuyoshi Horigome•28 views
Instrumentation & Control Lab Manual.pdf by NTU Faisalabad
Instrumentation & Control Lab Manual.pdfInstrumentation & Control Lab Manual.pdf
Instrumentation & Control Lab Manual.pdf
NTU Faisalabad •6 views

Apache Calcite Tutorial - BOSS 21

  • 1. Tutorial @BOSS’21 Copenhagen Stamatis Zampetakis, Julian Hyde • August 16, 2021
  • 3. Tutorial @BOSS’21 Copenhagen Stamatis Zampetakis, Julian Hyde • August 16, 2021 # Follow these steps to set up your environment # (The first time, it may take ~3 minutes to download dependencies.) git clone --branch boss21 https://github.com/zabetak/calcite-tutorial.git java -version # need Java 8 or higher cd calcite-tutorial ./mvnw package -DskipTests
  • 4. Setup Environment Requirements 1. Git 2. JDK version ≥ 1.8 Steps 1. Clone GitHub repository 2. Load to IDE (preferred IntelliJ) a. Click Open b. Navigate to calcite-tutorial c. Select pom.xml file d. Choose “Open as Project” 3. Compile the project git clone --branch boss21 https://github.com/zabetak/calcite-tutorial.git java -version cd calcite-tutorial ./mvnw package -DskipTests
  • 5. Julian Hyde @julianhyde Senior Staff Engineer @ Google / Looker Creator of Apache Calcite PMC member of Apache Arrow, Drill, Eagle, Incubator and Kylin Stamatis Zampetakis @szampetak Senior Software Engineer @ Cloudera, Hive query optimizer team PMC member of Apache Calcite; Hive committer PhD in Data Management, INRIA & Paris-Sud University About us
  • 6. Outline 1. Introduction 2. CSV Adapter Demo 3. Coding module I: Main components 4. Coding module I Exercises (Homework) 5. Hybrid planning 6. Coding module II: Custom operators/rules (Homework) 7. Volcano Planner internals 8. Dialects 9. Materialized views 10. Working with spatial data 11. Research using Apache Calcite optional optional optional optional
  • 8. 1. Retrieve books and authors 2. Display image, title, price of the book along with firstname & lastname of the author 3. Sort the books based on their id (price or something else) 4. Show results in groups of five Motivation: Data views
  • 9. What, where, how data are stored? author 0..1 AUTHOR id int fname string lname string birth date BOOK id int title string price decimal year int
  • 10. XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN FILESYSTEM What, where, how data are stored? author 0..1 AUTHOR id int fname string lname string birth date BOOK id int title string price decimal year int
  • 11. BIN What, where, how data are stored? XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN FILESYSTEM author 0..1 AUTHOR id int fname string lname string birth date BOOK id int title string price decimal year int
  • 12. What, where, how data are stored? 360+ DBMS XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN XML CSV JSON BIN FILESYSTEM author 0..1 AUTHOR id int fname string lname string birth date BOOK id int title string price decimal year int
  • 13. ★ Open-source search engine ★ Java library ★ Powerful indexing & search features ★ Spell checking, hit highlighting ★ Advanced analysis/tokenization capabilities ★ ACID transactions ★ Ultra compact memory/disk format Apache Lucene
  • 14. How to query the data? SELECT b.id, b.title, b.year, a.fname, a.lname FROM Book b LEFT OUTER JOIN Author a ON b.author=a.id ORDER BY b.id LIMIT 5 1. Retrieve books and authors 2. Display image, title, price of the book along with firstname & lastname of the author 3. Sort the books based on their id (price or something else) 4. Show results in groups of five
  • 15. Query processor architecture Query Relational Algebra Query planner CatalogReader Schema Metadata (Cost, Statistics) Execution engine Results Parser Rules Relational Algebra
  • 17. Apache Calcite SQL query SqlToRelConverter RelNode RelOptPlanner CatalogReader Schema RelRule RelBuilder RelRunner Results API calls SqlParser SqlValidator RelNode SqlNode SqlNode Dev & extensions RelMetadata Provider
  • 19. Adapter Implement SchemaFactory interface Connect to a data source using parameters Extract schema - return a list of tables Push down processing to the data source: ● A set of planner rules ● Calling convention (optional) ● Query model & query generator (optional) { "schemas": [ { "name": "BOOKSTORE", "type": "custom", "factory": "org.apache.calcite.adapter.file.FileSchemaFactory", "operand": { "directory": "bookstore" } } ] ]
  • 20. 3. Coding module I: Main components
  • 21. Setup schema & type factory <<Interface>> Schema +getTable(name: Table): Table <<Interface>> Table +getRowType(typeFactory: RelDataTypeFactory): RelDataType <<Interface>> RelDataTypeFactory +createJavaType(clazz: Class): RelDataType +createSqlType(typeName: SqlTypeName): RelDataType * tables
  • 23. SELECT b.id, b.title, b.year, a.fname, a.lname FROM Book AS b LEFT OUTER JOIN Author a ON b.author = a.id WHERE b.year > 1830 ORDER BY b.id LIMIT 5 Query to Abstract Syntax Tree (AST) SQL query SqlParser SqlNode Numeric Literal 5 Identifier b.id NodeList Select OrderBy Join BasicCall As Operator Identifier author Identifier a Literal LEFT Literal ON Identifier b.author Identifier a.id Binary Operator = As Operator Identifier book Identifier b BasicCall BasicCal l Identifier b.year Numeric Literal 1830 Binary Operator > BasicCall Identifier b.id NodeList Identifier b.title Identifier b.year Identifier a.fname Identifier a.lname
  • 24. AST to logical plan SqlNode Join Select b.id, b.title, b.year, a.fname, a.lname BasicCall b.author = a.id BasicCall book AS b BasicCall author AS a Literal LEFT Literal ON LogicalJoin [$3==$4,type=left] LogicalTableScan [Book] LogicalTableScan [Author] LogicalFilter [$2>1830] LogicalProject [ID=$0,TITLE=$1,YEAR=$2, FNAME=$5,LNAME=$6] LogicalSort [sort0=$0,dir=ASC,fetch=5] SqlValidator SqlToRelConverter SqlNode RelNode OrderBy b.id FETCH 5 BasicCall b.year > 1830
  • 25. Logical to physical plan RelRule EnumerableSortRule EnumerableProjectRule EnumerableFilterRule EnumerableJoinRule EnumerableTableScanRule RelOptPlanner RelNode RelNode LogicalJoin [$3==$4,type=left] LogicalTableScan [Book] LogicalTableScan [Author] LogicalFilter [$2>1830] LogicalProject [ID=$0,TITLE=$1,YEAR=$2, FNAME=$5,LNAME=$6] LogicalSort [sort0=$0,dir=ASC,fetch=5] EnumerableJoin [$3==$4,type=left] EnumerableTableScan [Book] EnumerableTableScan [Author] EnumerableFilter [$2>1830] EnumerableProject [ID=$0,TITLE=$1,YEAR=$2, FNAME=$5,LNAME=$6] EnumerableSort [sort0=$0,dir=ASC,fetch=5]
  • 26. Physical to Executable plan EnumerableInterpretable Java code RelNode EnumerableJoin [$3==$4,type=left] EnumerableTableScan [Book] EnumerableTableScan [Author] EnumerableFilter [$2>1830] EnumerableProject [ID=$0,TITLE=$1,YEAR=$2, FNAME=$5,LNAME=$6] EnumerableSort [sort0=$0,dir=ASC,fetch=5]
  • 27. 4. Coding module I: Exercises (Homework)
  • 28. Exercise I: Execute more SQL queries Include GROUP BY and other types of clauses: SELECT o.o_custkey, COUNT(*) FROM orders AS o GROUP BY o.o_custkey
  • 29. Exercise I: Execute more SQL queries Include GROUP BY and other types of clauses: ● Missing rule to convert LogicalAggregate to EnumerableAggregate ● Add EnumerableRules.ENUMERABLE_AGGREGATE_RULE to the planner SELECT o.o_custkey, COUNT(*) FROM orders AS o GROUP BY o.o_custkey
  • 30. Push filter below the join: Exercise II: Improve performance by applying more optimization rules SELECT c.c_name, o.o_orderkey, o.o_orderdate FROM customer AS c INNER JOIN orders AS o ON c.c_custkey = o.o_custkey WHERE c.c_custkey < 3 ORDER BY c.c_name, o.o_orderkey
  • 31. Push filter below the join: 1. Add rule CoreRules.FILTER_INTO_JOIN to the planner 2. Compare plans before and after (or logical and physical) 3. Check cost estimates by using SqlExplainLevel.ALL_ATTRIBUTES Exercise II: Improve performance by applying more optimization rules SELECT c.c_name, o.o_orderkey, o.o_orderdate FROM customer AS c INNER JOIN orders AS o ON c.c_custkey = o.o_custkey WHERE c.c_custkey < 3 ORDER BY c.c_name, o.o_orderkey
  • 32. Exercise II: Improve performance by applying more optimization rules LogicalJoin [$0=$9,type=inner] LogicalTableScan [CUSTOMER] LogicalTableScan [ORDERS] LogicalFilter [$0<3] LogicalProject [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] LogicalSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC] RelRule FilterIntoJoinRule LogicalJoin [$0=$9,type=inner] LogicalTableScan [CUSTOMER] LogicalTableScan [ORDERS] LogicalFilter [$0<3] LogicalProject [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] LogicalSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC]
  • 33. Open LuceneBuilderProcessor.java and complete TODOs Exercise III: Use RelBuilder API to construct the logical plan SELECT o.o_custkey, COUNT(*) FROM orders AS o GROUP BY o.o_custkey SELECT o.o_custkey, COUNT(*) FROM orders AS o WHERE o.o_totalprice > 220388.06 GROUP BY o.o_custkey Q1: Q2:
  • 34. Exercise III: Use RelBuilder API to construct the logical plan builder .scan("orders") .filter( builder.call( SqlStdOperatorTable.GREATER_THAN, builder.field("o_totalprice"), builder.literal(220388.06))) .aggregate( builder.groupKey("o_custkey"), builder.count());
  • 36. Calling convention Scan Scan Join Filter Join Scan Initially all nodes belong to “logical” calling convention. Logical calling convention cannot be implemented, so has infinite cost
  • 37. Calling convention Scan Scan Join Filter Join Scan Tables can’t be moved so there is only one choice of calling convention for each table. Examples: ● Enumerable ● Druid ● Drill ● HBase ● JDBC
  • 38. Calling convention Scan Scan Join Filter Join Scan Rules fire to convert nodes to particular calling conventions. The calling convention propagates through the tree. Because this is Volcano, each node can have multiple conventions.
  • 39. Calling convention Scan Scan Join Filter Join Scan Rules fire to convert nodes to particular calling conventions. The calling convention propagates through the tree. Because this is Volcano, each node can have multiple conventions.
  • 40. Calling convention Scan Scan Join Filter Join Scan Rules fire to convert nodes to particular calling conventions. The calling convention propagates through the tree. Because this is Volcano, each node can have multiple conventions.
  • 41. Converters Scan Scan Join Filter Join To keep things honest, we need to insert a converter at each point where the convention changes. (Recall: Volcano has an enforcer for each trait. Convention is a physical property, and converter is the enforcer.) Blue to Logical BlueFilterRule: LogicalFilter(BlueToLogical(Blue b)) → BlueToLogical(BlueFilter(b)) Green to Logical Scan
  • 42. Converters Scan Scan Join Join Green to Logical To keep things honest, we need to insert a converter at each point where the convention changes. (Recall: Volcano has an enforcer for each trait. Convention is a physical property, and converter is the enforcer.) Blue to Logical Filter BlueFilterRule: LogicalFilter(BlueToLogical(Blue b)) → BlueToLogical(BlueFilter(b)) Scan
  • 43. Generating programs to implement hybrid plans Hybrid plans are glued together using an engine - a convention that does not have a storage format. (Example engines: Drill, Spark, Presto.) To implement, we generate a program that calls out to query1 and query2. The "Blue-to-Orange" converter is typically a function in the Orange language that embeds a Blue query. Similarly "Green-to-Orange". Scan Scan Join Join Green to Orange Blue to Orange Filter Scan
  • 44. 6. Coding module II: Custom operators/rules (Homework)
  • 45. What we want to achieve? EnumerableJoin [$0=$9,type=inner] EnumerableTableScan [CUSTOMER] EnumerableCalc [$0<3] EnumerableCalc [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] EnumerableSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC] EnumerableTableScan [ORDERS] EnumerableJoin [$0=$9,type=inner] LuceneTableScan [CUSTOMER] LuceneFilter [$0<3] EnumerableCalc [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] EnumerableSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC] LuceneTableScan [ORDERS] LuceneToEnumerableConverter LuceneToEnumerableConverter
  • 46. What do we need? EnumerableJoin [$0=$9,type=inner] LuceneTableScan [CUSTOMER] LuceneFilter [$0<3] EnumerableCalc [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] EnumerableSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC] LuceneTableScan [ORDERS] LuceneToEnumerableConverter LuceneToEnumerableConverter Two calling conventions: 1. Enumerable 2. Lucene Three custom operators: 1. LuceneTableScan 2. LuceneToEnumerableConverter 3. LuceneFilter Three custom conversion rules: 1. LogicalTableScan → LuceneTableScan 2. LogicalFilter → LuceneFilter 3. LuceneANY → LuceneToEnumerableConverter
  • 47. What do we need? EnumerableJoin [$0=$9,type=inner] LuceneTableScan [CUSTOMER] LuceneFilter [$0<3] EnumerableCalc [C_NAME=$1,O_ORDERKEY=$8, O_ORDERDATE=$12] EnumerableSort [sort0=$0,dir0=ASC,sort1=$1,dir1=ASC] LuceneTableScan [ORDERS] LuceneToEnumerableConverter LuceneToEnumerableConverter Two calling conventions: 1. Enumerable 2. Lucene Three custom operators: 1. LuceneTableScan 2. LuceneToEnumerableConverter 3. LuceneFilter Three custom conversion rules: 1. LogicalTableScan → LuceneTableScan 2. LogicalFilter → LuceneFilter 3. LuceneANY → LuceneToEnumerableConverter STEP 1 STEP 2 STEP 3 STEP 4 STEP 5 STEP 6
  • 48. 7. Volcano Planner Internals
  • 49. Volcano planning algorithm Based on two papers by Goetz Graefe in the 1990s (Volcano, Cascades), now the industry standard for cost-based optimization. Dynamic programming: to optimize a relational expression R0 , convert it into equivalent expressions {R1 , R2 , …}, and pick the one with the lowest cost. Much of the cost of R is the cost of its input(s). So we apply dynamic programming to its inputs, too. R0 S0 T0
  • 50. Volcano planning algorithm Based on two papers by Goetz Graefe in the 1990s (Volcano, Cascades), now the industry standard for cost-based optimization. Dynamic programming: to optimize a relational expression R0 , convert it into equivalent expressions {R1 , R2 , …}, and pick the one with the lowest cost. Much of the cost of R is the cost of its input(s). So we apply dynamic programming to its inputs, too. R0 S0 R1 T0
  • 51. Volcano planning algorithm Based on two papers by Goetz Graefe in the 1990s (Volcano, Cascades), now the industry standard for cost-based optimization. Dynamic programming: to optimize a relational expression R0 , convert it into equivalent expressions {R1 , R2 , …}, and pick the one with the lowest cost. Much of the cost of R is the cost of its input(s). So we apply dynamic programming to its inputs, too. R0 S0 R1 T0 T1
  • 52. Volcano planning algorithm Based on two papers by Goetz Graefe in the 1990s (Volcano, Cascades), now the industry standard for cost-based optimization. Dynamic programming: to optimize a relational expression R0 , convert it into equivalent expressions {R1 , R2 , …}, and pick the one with the lowest cost. Much of the cost of R is the cost of its input(s). So we apply dynamic programming to its inputs, too. R0 S0 R1 T0 T1 R2 Uo
  • 53. Volcano planning algorithm We keep equivalence sets of expressions (class RelSet). Each input of a relational expression is an equivalence set + required physical properties (class RelSubset). R0 S0 R1 T0 T1 R2 Uo
  • 54. Volcano planning algorithm Each relational expression has a memo (digest), so we will recognize it if we generate it again. R0 S0 R1 T0 T1 R2 Uo
  • 55. Volcano planning algorithm If an expression transforms to an expression in another equivalence set, we can merge those equivalence sets. R0 S0 R1 T0 T1 R2 Uo
  • 56. Matches and queues We register a new RelNode by adding it to a RelSet. Each rule instance declares a pattern of RelNode types (and other properties) that it will match. Suppose we have: ● Filter-on-Project ● Project-on-Project ● Project-on-Join On register, we detect rules that are newly matched. Project Filter Union Join Scan Project
  • 57. Matches and queues We register a new RelNode by adding it to a RelSet. Each rule instance declares a pattern of RelNode types (and other properties) that it will match. Suppose we have: ● Filter-on-Project ● Project-on-Project ● Project-on-Join On register, we detect rules that are newly matched. Project Filter Union Project Join Scan Project
  • 58. Matches and queues We register a new RelNode by adding it to a RelSet. Each rule instance declares a pattern of RelNode types (and other properties) that it will match. Suppose we have: ● Filter-on-Project ● Project-on-Project ● Project-on-Join On register, we detect rules that are newly matched. (4 matches.) Project Filter Union Project Join Scan Project
  • 59. Matches and queues Should we fire these matched rules immediately? No! Because rule match #1 would generate new matches… which would generate new matches… and we'd never get to match #2. Instead, we put the matched rules on a queue. The queue allows us to: ● Search breadth-first (rather than depth-first) ● Prioritize (fire more "important" rules first) ● Potentially terminate when we have a "good enough" plan Equivalence sets containing registered RelNodes 0. Register each RelNode in the initial tree. Rule match queue 1. Each time a RelNode is registered, find rule matches, and put RuleMatch objects on the queue. 3. Pop the top rule match. Fire the rule. Register each RelNode generated by the rule, merging sets if equivalences are found. Goto 1. 2. If the queue is empty, or the cost is good enough, we're done.
  • 60. Other planner engines, same great rules Three planner engines: ● Volcano ● Volcano top-down (Cascades style) ● Hep applies rules in a strict "program" The same rules are used by all engines. It takes a lot of time effort to write a high-quality rule. Rules can be reused, tested, improved, and they compose with other rules. Calcite's library of rules is valuable.
  • 62. Calcite architecture Apache Calcite JDBC adapter Pluggable rewrite rules Enumerable adapter MongoDB adapter File adapter (CSV, JSON, Http) Apache Kafka adapter Apache Spark adapter SQL SQL parser & validator Query planner Relational algebra At what points in the Calcite stack do ‘languages’ exist? ● Incoming SQL ● Validating SQL against built-in operators ● Type system (e.g. max size of INTEGER type) ● JDBC adapter generates SQL ● Other adapters generate other languages
  • 63. Parsing & validating SQL - what knobs can I turn? SELECT deptno AS d, SUM(sal) AS [sumSal] FROM [HR].[Emp] WHERE ename NOT ILIKE "A%" GROUP BY d ORDER BY 1, 2 DESC SqlConformance.isGroupByAlias() = true SqlConformance.isSortByOrdinal() = true Lex.quoting = Quoting.BRACKET Lex.charLiteralStyle = CharLiteralStyle.BQ_DOUBLE SqlValidator.Config.defaultNullCollation = HIGH Lex.unquotedCasing = Casing.TO_UPPER Lex.quotedCasing = Casing.UNCHANGED PARSER_FACTORY = "org.apache.calcite.sql.parser.impl.SqlParserImpl.FACTORY" FUN = "postgres" (ILIKE is not standard SQL)
  • 64. SQL dialect - APIs and properties Pluggable rewrite rules Pluggable parser, lexical, conformance, operators Pluggable SQL dialect SQL SQL SQL parser & validator Query planner Relational algebra JDBC adapter interface SqlParserImplFactory CalciteConnectionProperty.LEX enum Lex enum Quoting enum Casing enum CharLiteralStyle CalciteConnectionProperty.CONFORMANCE interface SqlConformance CalciteConnectionProperty.FUN interface SqlOperatorTable class SqlStdOperatorTable class SqlLibraryOperators class SqlOperator class SqlFunction extends SqlOperator class SqlAggFunction extends SqlFunction class RelRule class SqlDialect interface SqlDialectFactory
  • 65. Contributing a dialect (or anything!) to Calcite For your first code contribution, pick a small bug or feature. Introduce yourself! Email dev@, saying what you plan to do. Create a JIRA case describing the problem. To understand the code, find similar features. Run their tests in a debugger. Write 1 or 2 tests for your feature. Submit a pull request (PR).
  • 66. Other front-end languages Pig RelBuilder Adapter Physical operators Morel Storage Query planner Relational algebra Datalog SQL parser & validator SQL Calcite is an excellent platform for implementing your own data language Write a parser for your language, use RelBuilder to translate to relational algebra, and you can use any of Calcite's back-end implementations
  • 68. Forwards planning Backwards planning Backwards planning Until now, we have seen forward planning. Forward planning transforms an expression (R0 ) to many equivalent forms and picks the one with lowest cost (Ropt ). Backwards planning transforms an expression to an equivalent form (RN ) that contains a target expression (T). R0 R0 T Ropt
  • 69. Forwards planning Backwards planning Equivalence set Backwards planning Ropt R2 R1 R0 Equivalence set RN R2 R1 R0 T S Until now, we have seen forward planning. Forward planning transforms an expression (R0 ) to many equivalent forms and picks the one with lowest cost (Ropt ). Backwards planning transforms an expression to an equivalent form (RN ) that contains a target expression (T).
  • 70. Applications of backwards planning Indexes (e.g. b-tree indexes). An index is a derived data structure whose contents can be described as a relational expression (generally project-sort). When we are planning a query, it already exists (i.e. the cost has already been paid). Summary tables. A summary table is a derived data structure (generally filter-project-join-aggregate). Replicas with different physical properties (e.g. copy the table from New York to Tokyo, or copy the table and partition by month(orderDate), sort by productId). Incremental view maintenance. Materialized view V is populated from base table T. Yesterday, we populated V with V0 = Q(T0 ). Today we want to make its contents equal to V1 = Q(T1 ). Can we find and apply a delta query, dQ = Q(T1 - T0 )?
  • 71. Materialized views in Calcite { "schemas": { "name": "HR", "tables": [ { "name": "emp" } ], "materializations": [ { "table": "i_emp_job", "sql": "SELECT job, empno FROM emp ORDER BY job, empno" }, { "table": "add_emp_deptno", "sql": "SELECT deptno, SUM(sal) AS ss, COUNT(*) AS c FROM emp GROUP BY deptno" } ] } } /** Transforms a relational expression into a * semantically equivalent relational expression, * according to a given set of rules and a cost * model. */ public interface RelOptPlanner { /** Defines an equivalence between a table and * a query. */ void addMaterialization( RelOptMaterialization materialization); /** Finds the most efficient expression to * implement this query. */ RelNode findBestExp(); } /** Records that a particular query is materialized * by a particular table. */ public class RelOptMaterialization { public final RelNode tableRel; public final List<String> qualifiedTableName; public final RelNode queryRel; } You can define materializations in a JSON model, via the planner API, or via CREATE MATERIALIZED VIEW DDL (not shown).
  • 72. More about materialized views ● There are several algorithms to rewrite queries to match materialized views ● A lattice is a data structure to model a star schema ● Calcite has algorithms to recommend an optimal set of summary tables for a lattice (given expected queries, and statistics about column cardinality) ● Data profiling algorithms estimate the cardinality of all combinations of columns
  • 73. 10. Working with spatial data
  • 74. Spatial query Find all restaurants within 1.5 distance units of my current location: We cannot use a B-tree index (it can sort points by x or y coordinates, but not both) and specialized spatial indexes (such as R*-trees) are not generally available. restaurant x y Zachary’s pizza 3 1 King Yen 7 7 Filippo’s 7 4 Station burger 5 6 SELECT * FROM Restaurants AS r WHERE ST_Distance( ST_MakePoint(r.x, r.y), ST_MakePoint(6, 7)) < 1.5 • • • • Zachary’s pizza Filippo’s King Yen Station burger
  • 75. Hilbert space-filling curve ● A space-filling curve invented by mathematician David Hilbert ● Every (x, y) point has a unique position on the curve ● Points near to each other typically have Hilbert indexes close together
  • 76. • • • • Add restriction based on h, a restaurant’s distance along the Hilbert curve Must keep original restriction due to false positives Using Hilbert index restaurant x y h Zachary’s pizza 3 1 5 King Yen 7 7 41 Filippo’s 7 4 52 Station burger 5 6 36 Zachary’s pizza Filippo’s SELECT * FROM Restaurants AS r WHERE (r.h BETWEEN 35 AND 42 OR r.h BETWEEN 46 AND 46) AND ST_Distance( ST_MakePoint(r.x, r.y), ST_MakePoint(6, 7)) < 1.5 King Yen Station burger
  • 77. Telling the optimizer 1. Declare h as a generated column 2. Sort table by h Planner can now convert spatial range queries into a range scan Does not require specialized spatial index such as R*-tree Very efficient on a sorted table such as HBase There are similar techniques for other spatial patterns (e.g. region-to-region join) CREATE TABLE Restaurants ( restaurant VARCHAR(20), x DOUBLE, y DOUBLE, h DOUBLE GENERATED ALWAYS AS ST_Hilbert(x, y) STORED) SORT KEY (h); restaurant x y h Zachary’s pizza 3 1 5 Station burger 5 6 36 King Yen 7 7 41 Filippo’s 7 4 52
  • 78. 11. Research using Apache Calcite
  • 80. Yes, VLDB 2021! Go to the talk! 0900 Wednesday.
  • 83. Resources ● Calcite project https://calcite.apache.org ● Materialized view algorithms https://calcite.apache.org/docs/materialized_views.html ● JSON model https://calcite.apache.org/docs/model.html ● Lazy beats smart and fast (DataEng 2018) - MVs, spatial, profiling https://www.slideshare.net/julianhyde/lazy-beats-smart-and-fast ● Efficient spatial queries on vanilla databases (ApacheCon 2018) https://www.slideshare.net/julianhyde/spatial-query-on-vanilla-databases ● Graefe, McKenna. The Volcano Optimizer Generator, 1991 ● Graefe. The Cascades Framework for Query Optimization, 1995 ● Slideshare (past presentations by Julian Hyde, including several about Apache Calcite) https://www.slideshare.net/julianhyde