The document discusses new features and improvements in the MySQL 8.0 optimizer. Key highlights include:
- New SQL syntax like SELECT...FOR UPDATE SKIP LOCKED and NOWAIT to handle row locking contention.
- Support for common table expressions to improve readability and allow referencing derived tables multiple times.
- Enhancements to the cost model to produce more accurate estimates based on factors like data location.
- Better support for data types like UUID and IPv6, including optimized storage formats and new functions.
The document is an introduction to the MySQL 8.0 optimizer guide. It includes a safe harbor statement noting that the guide outlines Oracle's general product direction but not commitments. The agenda lists 25 topics to be covered related to query optimization, diagnostic commands, examples from the "World Schema" sample database, and a companion website with more details.
SQL Now! How Optiq brings the best of SQL to NoSQL data.Julian Hyde
A talk given at the August, 2013 NoSQL Now! conference in San Jose, California.
2013 is the year of that SQL meets Hadoop & NoSQL. There is a surprisingly good fit between thirty-something SQL and the upstart millennial data technologies. SQL helps you deliver faster value, improve integration with visualization and analytics tools, and bring your data to a larger audience.
Julian Hyde, author of Mondrian and Optiq, and a contributor to Apache Drill and Cascading Lingual, shows how to quickly build a SQL interface to a NoSQL system using the Optiq framework.
As a case study, he shows how Optiq has allows the Mondrian in-memory analysis engine (part of Pentaho's business intelligence suite) to leverage the memory and compute power of a cluster and pull data from hybrid SQL and NoSQL sources.
In this talk, we’ll discuss technical designs of support of HBase as a “native” data source to Spark SQL to achieve both query and load performance and scalability: near-precise execution locality of query and loading, fine-tuned partition pruning, predicate pushdown, plan execution through coprocessor, and optimized and fully parallelized bulk loader. Point and range queries on dimensional attributes will benefit particularly well from the techniques. Preliminary test results vs. established SQL-on-HBase technologies will be provided. The speaker will also share the future plan and real-world use cases, particularly in the telecom industry.
Planning with Polyalgebra: Bringing Together Relational, Complex and Machine ...Julian Hyde
A talk from given by Julian Hyde and Tomer Shiran at Hadoop Summit, Dublin.
Data scientists and analysts want the best API, DSL or query language possible, not to be limited by what the processing engine can support. Polyalgebra is an extension to relational algebra that separates the user language from the engine, so you can choose the best language and engine for the job. It also allows the system to optimize queries and cache results. We demonstrate how Ibis uses Polyalgebra to execute the same Python-based machine learning queries on Impala, Drill and Spark. And we show how to build Polyalgebra expressions in Calcite and how to define optimization rules and storage handlers.
Enterprise data is moving into Hadoop, but some data has to stay in operational systems. Apache Calcite (the technology behind Hive’s new cost-based optimizer, formerly known as Optiq) is a query-optimization and data federation technology that allows you to combine data in Hadoop with data in NoSQL systems such as MongoDB and Splunk, and access it all via SQL.
Hyde shows how to quickly build a SQL interface to a NoSQL system using Calcite. He shows how to add rules and operators to Calcite to push down processing to the source system, and how to automatically build materialized data sets in memory for blazing-fast interactive analysis.
What's new in Mondrian 4? Slides for a talk given by Julian Hyde to the Pentaho Bay Area User Group meetup in San Francisco on April 3rd, 2014.
Topics covered include attributes and attribute hierarchies, measure groups and aggregate tables, physical schema, and how to download and start using Mondrian 4 beta with Pentaho CE.
OrientDB vs Neo4j - Comparison of query/speed/functionalityCurtis Mosters
This presentation gives an overview on OrientDB and Neo4j. It also compares some specific querys, their speed and the overall functionality of both databases.
The querys might not be optimized in both cases. At least they have the same outcome and are both written as querys. For sure in Neo4j you should do this in Java code. But that is way harder to write, so this presentation is more like a direkt comparision instead of really getting the best results.
Also it's done with real data and at the end round about 200 GB of data.
Don’t optimize my queries, optimize my data!Julian Hyde
The document discusses strategies for optimizing data through materialized views and how data systems can learn to optimize themselves. It proposes an algorithm that uses sketches and information theory to profile data cardinalities and recommend materialized views. The algorithm aims to defeat the combinatorial search space by only considering combinations with "surprising" cardinalities. This profiling provides the cost and benefit information needed to optimize data structures. The document also discusses using query logs and statistics to infer relationships between tables and design summary tables through lattices.
The document is an introduction to the MySQL 8.0 optimizer guide. It includes a safe harbor statement noting that the guide outlines Oracle's general product direction but not commitments. The agenda lists 25 topics to be covered related to query optimization, diagnostic commands, examples from the "World Schema" sample database, and a companion website with more details.
SQL Now! How Optiq brings the best of SQL to NoSQL data.Julian Hyde
A talk given at the August, 2013 NoSQL Now! conference in San Jose, California.
2013 is the year of that SQL meets Hadoop & NoSQL. There is a surprisingly good fit between thirty-something SQL and the upstart millennial data technologies. SQL helps you deliver faster value, improve integration with visualization and analytics tools, and bring your data to a larger audience.
Julian Hyde, author of Mondrian and Optiq, and a contributor to Apache Drill and Cascading Lingual, shows how to quickly build a SQL interface to a NoSQL system using the Optiq framework.
As a case study, he shows how Optiq has allows the Mondrian in-memory analysis engine (part of Pentaho's business intelligence suite) to leverage the memory and compute power of a cluster and pull data from hybrid SQL and NoSQL sources.
In this talk, we’ll discuss technical designs of support of HBase as a “native” data source to Spark SQL to achieve both query and load performance and scalability: near-precise execution locality of query and loading, fine-tuned partition pruning, predicate pushdown, plan execution through coprocessor, and optimized and fully parallelized bulk loader. Point and range queries on dimensional attributes will benefit particularly well from the techniques. Preliminary test results vs. established SQL-on-HBase technologies will be provided. The speaker will also share the future plan and real-world use cases, particularly in the telecom industry.
Planning with Polyalgebra: Bringing Together Relational, Complex and Machine ...Julian Hyde
A talk from given by Julian Hyde and Tomer Shiran at Hadoop Summit, Dublin.
Data scientists and analysts want the best API, DSL or query language possible, not to be limited by what the processing engine can support. Polyalgebra is an extension to relational algebra that separates the user language from the engine, so you can choose the best language and engine for the job. It also allows the system to optimize queries and cache results. We demonstrate how Ibis uses Polyalgebra to execute the same Python-based machine learning queries on Impala, Drill and Spark. And we show how to build Polyalgebra expressions in Calcite and how to define optimization rules and storage handlers.
Enterprise data is moving into Hadoop, but some data has to stay in operational systems. Apache Calcite (the technology behind Hive’s new cost-based optimizer, formerly known as Optiq) is a query-optimization and data federation technology that allows you to combine data in Hadoop with data in NoSQL systems such as MongoDB and Splunk, and access it all via SQL.
Hyde shows how to quickly build a SQL interface to a NoSQL system using Calcite. He shows how to add rules and operators to Calcite to push down processing to the source system, and how to automatically build materialized data sets in memory for blazing-fast interactive analysis.
What's new in Mondrian 4? Slides for a talk given by Julian Hyde to the Pentaho Bay Area User Group meetup in San Francisco on April 3rd, 2014.
Topics covered include attributes and attribute hierarchies, measure groups and aggregate tables, physical schema, and how to download and start using Mondrian 4 beta with Pentaho CE.
OrientDB vs Neo4j - Comparison of query/speed/functionalityCurtis Mosters
This presentation gives an overview on OrientDB and Neo4j. It also compares some specific querys, their speed and the overall functionality of both databases.
The querys might not be optimized in both cases. At least they have the same outcome and are both written as querys. For sure in Neo4j you should do this in Java code. But that is way harder to write, so this presentation is more like a direkt comparision instead of really getting the best results.
Also it's done with real data and at the end round about 200 GB of data.
Don’t optimize my queries, optimize my data!Julian Hyde
The document discusses strategies for optimizing data through materialized views and how data systems can learn to optimize themselves. It proposes an algorithm that uses sketches and information theory to profile data cardinalities and recommend materialized views. The algorithm aims to defeat the combinatorial search space by only considering combinations with "surprising" cardinalities. This profiling provides the cost and benefit information needed to optimize data structures. The document also discusses using query logs and statistics to infer relationships between tables and design summary tables through lattices.
Why is data independence (still) so important? Optiq and Apache Drill.Julian Hyde
Presentation to the Apache Drill Meetup in Sunnyvale, CA on 2012/9/13. Framing the debate about Drill's goals in terms of a "typical" modern DBMS architecture; and also introducing the Optiq extensible query optimizer.
Introduction to Apache Calcite. An overview of the core features that Apache Calcite provides to help build a database. The talk covers Calcite's SQL parser, cost-based optimizer, pluggable data sources, and JDBC-HTTP server "Avatica".
SparkSQL is a Spark component that allows SQL queries to be executed on Spark. It uses Catalyst, which provides an execution planning framework for relational operations like SQL parsing, logical optimization, and physical planning. Catalyst defines logical and physical operators, expressions, data types and provides rule-based optimizations of the logical query plan. The SQL core in SparkSQL converts logical plans to physical plans and enables reading/writing to data sources like Parquet files and in-memory columnar tables.
20190817 coscup-oracle my sql innodb cluster sharingIvan Ma
The document provides an agenda for a presentation on MySQL InnoDB Cluster. It discusses MySQL replication components, demonstrating MySQL InnoDB Cluster, network stability, operations, backup and recovery, GTID consistency, replication between clusters, and troubleshooting. It also covers MySQL innovations from version 5.7 to 8.0 and options for configuring an InnoDB Cluster, including consistency settings and member weights.
Apache Calcite: A Foundational Framework for Optimized Query Processing Over ...Julian Hyde
Apache Calcite is an open source framework for building data management systems that allows for optimized query processing over heterogeneous data sources. It uses a flexible relational algebra and extensible adapter-based architecture that allows it to incorporate diverse data sources. Calcite's rule-based optimizer transforms logical query plans into efficient physical execution plans tailored for different data sources. It has been adopted by many projects and companies and is also used in research.
Discardable In-Memory Materialized Queries With HadoopJulian Hyde
What to do with all that memory in a Hadoop cluster? Should we load all of our data into memory to process it?
The goal should be to put memory into its right place in the storage hierarchy, alongside disk and solid-state drives (SSD). Data should reside in the right place for how it is being used, and should be organized appropriately for where it resides. This proposed solution requires a new kind of data set called the Discardable, In-Memory, Materialized Query (DIMMQ).
In this session we will talk through how we can build on existing Hadoop facilities to deliver three key underlying concepts that enable this approach.
This document discusses how Apache Calcite makes it easier to write database management systems (DBMS) by decomposing them into modular components like a query parser, catalog, algorithms, and storage engines. It presents Calcite as a framework that allows these components to be mixed and matched, with a core relational algebra and rule-based optimization. Calcite powers systems like Apache Hive, Drill, Phoenix, and Kylin by translating SQL and other queries to relational algebra and optimizing queries using over 100 rules before executing them using configurable engines and data sources.
https://fosdem.org/2017/schedule/event/hpc_bigdata_calcite/
When working with BigData & IoT systems we often feel the need for a Common Query Language. The platform specific languages are often harder to integrate with and require longer adoption time.
To fill this gap many NoSql (Not-only-Sql) vendors are building SQL layers for their platforms. It is worth exploring the driving forces behind this trend, how it fits in your BigData stacks and how we can adopt it in our favorite tools. However building SQL engine from scratch is a daunting job and frameworks like Apache Calcite can help you with the heavy lifting. Calcite allow you to integrate SQL parser, cost-based optimizer, and JDBC with your big data system.
Calcite has been used to empower many Big-Data platforms such as Hive, Spark, Drill Phoenix to name some.
I will walk you through the process of building a SQL access layer for Apache Geode (In-Memory Data Grid). I will share my experience, pitfalls and technical consideration like balancing between the SQL/RDBMS semantics and the design choices and limitations of the data system.
Hopefully this will enable you to add SQL capabilities to your prefered NoSQL data system.
Why you care about relational algebra (even though you didn’t know it)Julian Hyde
A talk given by Julian Hyde at Enterprise Data World on in Washington, DC on April 2nd, 2015.
With data in different systems, in different formats, and accessed via different tools, we need a lingua franca for data. Not all tools speak SQL, and data cannot be moved into a single convenient location.
Relational algebra underpins SQL and many other DB languages. It is also perfect for optimizing, caching and mediating.
Apache Calcite (formerly Optiq) is a framework for building and optimizing expressions in relational algebra. We show how to write queries, optimize queries using rewrite rules, and write adapters for back-end systems. We also show to configure Calcite to materialize queries, so your interactive analytics are effectively running against a fast in-memory database.
A talk given by Julian Hyde at FlinkForward, Berlin, on 2016/09/12.
Streaming is necessary to handle data rates and latency, but SQL is unquestionably the lingua franca of data. Is it possible to combine SQL with streaming, and if so, what does the resulting language look like? Apache Calcite is extending SQL to include streaming, and Apache Flink is using Calcite to support both regular and streaming SQL. In this talk, Julian Hyde describes streaming SQL in detail and shows how you can use streaming SQL in your application. He also describes how Calcite’s planner optimizes queries for throughput and latency.
A smarter Pig: Building a SQL interface to Apache Pig using Apache CalciteJulian Hyde
What if Apache Pig had a SQL front-end and query optimizer? What if Apache Calcite was able to use Pig and MapReduce to run queries? In this project, we aimed to answer both questions by adding a Pig adapter for Calcite. In this talk, we describe Calcite's adapter framework, how we used it to write a Pig adapter, and how you can use this SQL interface to Pig for interactive and long-running queries.
A talk given by Eli Levine and Julian Hyde at Apache: Big Data, Miami, on May 17th, 2017.
Sasi, cassandra on full text search rideDuyhai Doan
This document discusses SASI (SSTable Attached Secondary Index), a new secondary index for Apache Cassandra that follows the SSTable lifecycle. It describes how SASI works, including its in-memory and on-disk structures. It also covers SASI's query planning optimizations and provides some benchmark results showing SASI's performance improvements over full scans. While SASI is not as full-featured as search engines, it can cover many search use cases within Cassandra.
Cost-based Query Optimization in Apache Phoenix using Apache CalciteJulian Hyde
This document summarizes a presentation on using Apache Calcite for cost-based query optimization in Apache Phoenix. Key points include:
- Phoenix is adding Calcite's query planning capabilities to improve performance and SQL compliance over its existing query optimizer.
- Calcite models queries as relational algebra expressions and uses rules, statistics, and a cost model to choose the most efficient execution plan.
- Examples show how Calcite rules like filter pushdown and exploiting sortedness can generate better plans than Phoenix's existing optimizer.
- Materialized views and interoperability with other Calcite data sources like Apache Drill are areas for future improvement beyond the initial Phoenix+Calcite integration.
Cost-based query optimization in Apache Hive 0.14Julian Hyde
Tez is making Hive faster, and now cost-based optimization (CBO) is making it smarter. A new initiative in Hive introduces cost-based optimization for the first time, based on the Optiq framework. Optiq's lead developer Julian Hyde shows the improvements that CBO is bringing in Apache Hive 0.14.
For those interested in Hive internals, he gives an overview of the Optiq framework and shows some of the improvements that are coming to future versions of Hive with the Stinger.next initiative.
Pivoting Data with SparkSQL by Andrew RaySpark Summit
This document discusses pivoting data with SparkSQL. It begins with an outline of topics to be covered, including what a pivot is, syntax, examples, tips, implementation details, and future work. It then provides examples of using pivots on retail sales and movie rating data to generate reports and features for modeling. It also offers tips on specifying pivot values, handling multiple aggregations, and pivoting multiple columns. The implementation details are discussed along with potential areas of future work, including adding pivot support to additional APIs and languages.
Apache Calcite is a dynamic data management framework. Think of it as a toolkit for building databases: it has an industry-standard SQL parser, validator, highly customizable optimizer (with pluggable transformation rules and cost functions, relational algebra, and an extensive library of rules), but it has no preferred storage primitives. In this tutorial, the attendees will use Apache Calcite to build a fully fledged query processor from scratch with very few lines of code. This processor is a full implementation of SQL over an Apache Lucene storage engine. (Lucene does not support SQL queries and lacks a declarative language for performing complex operations such as joins or aggregations.) Attendees will also learn how to use Calcite as an effective tool for research.
This document introduces Spark SQL and the Catalyst query optimizer. It discusses that Spark SQL allows executing SQL on Spark, builds SchemaRDDs, and optimizes query execution plans. It then provides details on how Catalyst works, including its use of logical expressions, operators, and rules to transform query trees and optimize queries. Finally, it outlines some interesting open issues and how to contribute to Spark SQL's development.
Apache Calcite: One Frontend to Rule Them AllMichael Mior
Apache Calcite is an open source framework that allows for a unified query interface over heterogeneous data sources. It provides an ANSI-compliant SQL parser, a logical query optimizer, and acts as a middleware layer that can integrate data from multiple sources. Calcite uses a relational algebra approach and has pluggable adapters that allow it to connect to different backends like MySQL, MongoDB, and streaming data sources. It supports features like SQL queries, views, optimization rules, and works across both batch and streaming data. The project aims to continue adding new capabilities like geospatial queries and improved cost modeling.
The document describes new features and enhancements in MySQL 8.0, including common table expressions, window functions, improved UTF-8 support, geospatial functions, new locking options for SELECT statements, JSON functions, index extensions, cost model improvements, query hints, and better support for IPv6 and UUID data types. The presentation agenda outlines each topic at a high level.
This document provides an overview of the MySQL query optimizer. It discusses the main phases of the optimizer including logical transformations, cost-based optimizations, analyzing access methods, join ordering, and plan refinements. Logical transformations prepare the query for cost-based optimization by simplifying conditions. Cost-based optimizations select the optimal join order and access methods to minimize resources used. Access methods analyzed include table scans, index scans, and ref access. The join optimizer searches for the best join order. Plan refinements include sort avoidance and index condition pushdown.
Why is data independence (still) so important? Optiq and Apache Drill.Julian Hyde
Presentation to the Apache Drill Meetup in Sunnyvale, CA on 2012/9/13. Framing the debate about Drill's goals in terms of a "typical" modern DBMS architecture; and also introducing the Optiq extensible query optimizer.
Introduction to Apache Calcite. An overview of the core features that Apache Calcite provides to help build a database. The talk covers Calcite's SQL parser, cost-based optimizer, pluggable data sources, and JDBC-HTTP server "Avatica".
SparkSQL is a Spark component that allows SQL queries to be executed on Spark. It uses Catalyst, which provides an execution planning framework for relational operations like SQL parsing, logical optimization, and physical planning. Catalyst defines logical and physical operators, expressions, data types and provides rule-based optimizations of the logical query plan. The SQL core in SparkSQL converts logical plans to physical plans and enables reading/writing to data sources like Parquet files and in-memory columnar tables.
20190817 coscup-oracle my sql innodb cluster sharingIvan Ma
The document provides an agenda for a presentation on MySQL InnoDB Cluster. It discusses MySQL replication components, demonstrating MySQL InnoDB Cluster, network stability, operations, backup and recovery, GTID consistency, replication between clusters, and troubleshooting. It also covers MySQL innovations from version 5.7 to 8.0 and options for configuring an InnoDB Cluster, including consistency settings and member weights.
Apache Calcite: A Foundational Framework for Optimized Query Processing Over ...Julian Hyde
Apache Calcite is an open source framework for building data management systems that allows for optimized query processing over heterogeneous data sources. It uses a flexible relational algebra and extensible adapter-based architecture that allows it to incorporate diverse data sources. Calcite's rule-based optimizer transforms logical query plans into efficient physical execution plans tailored for different data sources. It has been adopted by many projects and companies and is also used in research.
Discardable In-Memory Materialized Queries With HadoopJulian Hyde
What to do with all that memory in a Hadoop cluster? Should we load all of our data into memory to process it?
The goal should be to put memory into its right place in the storage hierarchy, alongside disk and solid-state drives (SSD). Data should reside in the right place for how it is being used, and should be organized appropriately for where it resides. This proposed solution requires a new kind of data set called the Discardable, In-Memory, Materialized Query (DIMMQ).
In this session we will talk through how we can build on existing Hadoop facilities to deliver three key underlying concepts that enable this approach.
This document discusses how Apache Calcite makes it easier to write database management systems (DBMS) by decomposing them into modular components like a query parser, catalog, algorithms, and storage engines. It presents Calcite as a framework that allows these components to be mixed and matched, with a core relational algebra and rule-based optimization. Calcite powers systems like Apache Hive, Drill, Phoenix, and Kylin by translating SQL and other queries to relational algebra and optimizing queries using over 100 rules before executing them using configurable engines and data sources.
https://fosdem.org/2017/schedule/event/hpc_bigdata_calcite/
When working with BigData & IoT systems we often feel the need for a Common Query Language. The platform specific languages are often harder to integrate with and require longer adoption time.
To fill this gap many NoSql (Not-only-Sql) vendors are building SQL layers for their platforms. It is worth exploring the driving forces behind this trend, how it fits in your BigData stacks and how we can adopt it in our favorite tools. However building SQL engine from scratch is a daunting job and frameworks like Apache Calcite can help you with the heavy lifting. Calcite allow you to integrate SQL parser, cost-based optimizer, and JDBC with your big data system.
Calcite has been used to empower many Big-Data platforms such as Hive, Spark, Drill Phoenix to name some.
I will walk you through the process of building a SQL access layer for Apache Geode (In-Memory Data Grid). I will share my experience, pitfalls and technical consideration like balancing between the SQL/RDBMS semantics and the design choices and limitations of the data system.
Hopefully this will enable you to add SQL capabilities to your prefered NoSQL data system.
Why you care about relational algebra (even though you didn’t know it)Julian Hyde
A talk given by Julian Hyde at Enterprise Data World on in Washington, DC on April 2nd, 2015.
With data in different systems, in different formats, and accessed via different tools, we need a lingua franca for data. Not all tools speak SQL, and data cannot be moved into a single convenient location.
Relational algebra underpins SQL and many other DB languages. It is also perfect for optimizing, caching and mediating.
Apache Calcite (formerly Optiq) is a framework for building and optimizing expressions in relational algebra. We show how to write queries, optimize queries using rewrite rules, and write adapters for back-end systems. We also show to configure Calcite to materialize queries, so your interactive analytics are effectively running against a fast in-memory database.
A talk given by Julian Hyde at FlinkForward, Berlin, on 2016/09/12.
Streaming is necessary to handle data rates and latency, but SQL is unquestionably the lingua franca of data. Is it possible to combine SQL with streaming, and if so, what does the resulting language look like? Apache Calcite is extending SQL to include streaming, and Apache Flink is using Calcite to support both regular and streaming SQL. In this talk, Julian Hyde describes streaming SQL in detail and shows how you can use streaming SQL in your application. He also describes how Calcite’s planner optimizes queries for throughput and latency.
A smarter Pig: Building a SQL interface to Apache Pig using Apache CalciteJulian Hyde
What if Apache Pig had a SQL front-end and query optimizer? What if Apache Calcite was able to use Pig and MapReduce to run queries? In this project, we aimed to answer both questions by adding a Pig adapter for Calcite. In this talk, we describe Calcite's adapter framework, how we used it to write a Pig adapter, and how you can use this SQL interface to Pig for interactive and long-running queries.
A talk given by Eli Levine and Julian Hyde at Apache: Big Data, Miami, on May 17th, 2017.
Sasi, cassandra on full text search rideDuyhai Doan
This document discusses SASI (SSTable Attached Secondary Index), a new secondary index for Apache Cassandra that follows the SSTable lifecycle. It describes how SASI works, including its in-memory and on-disk structures. It also covers SASI's query planning optimizations and provides some benchmark results showing SASI's performance improvements over full scans. While SASI is not as full-featured as search engines, it can cover many search use cases within Cassandra.
Cost-based Query Optimization in Apache Phoenix using Apache CalciteJulian Hyde
This document summarizes a presentation on using Apache Calcite for cost-based query optimization in Apache Phoenix. Key points include:
- Phoenix is adding Calcite's query planning capabilities to improve performance and SQL compliance over its existing query optimizer.
- Calcite models queries as relational algebra expressions and uses rules, statistics, and a cost model to choose the most efficient execution plan.
- Examples show how Calcite rules like filter pushdown and exploiting sortedness can generate better plans than Phoenix's existing optimizer.
- Materialized views and interoperability with other Calcite data sources like Apache Drill are areas for future improvement beyond the initial Phoenix+Calcite integration.
Cost-based query optimization in Apache Hive 0.14Julian Hyde
Tez is making Hive faster, and now cost-based optimization (CBO) is making it smarter. A new initiative in Hive introduces cost-based optimization for the first time, based on the Optiq framework. Optiq's lead developer Julian Hyde shows the improvements that CBO is bringing in Apache Hive 0.14.
For those interested in Hive internals, he gives an overview of the Optiq framework and shows some of the improvements that are coming to future versions of Hive with the Stinger.next initiative.
Pivoting Data with SparkSQL by Andrew RaySpark Summit
This document discusses pivoting data with SparkSQL. It begins with an outline of topics to be covered, including what a pivot is, syntax, examples, tips, implementation details, and future work. It then provides examples of using pivots on retail sales and movie rating data to generate reports and features for modeling. It also offers tips on specifying pivot values, handling multiple aggregations, and pivoting multiple columns. The implementation details are discussed along with potential areas of future work, including adding pivot support to additional APIs and languages.
Apache Calcite is a dynamic data management framework. Think of it as a toolkit for building databases: it has an industry-standard SQL parser, validator, highly customizable optimizer (with pluggable transformation rules and cost functions, relational algebra, and an extensive library of rules), but it has no preferred storage primitives. In this tutorial, the attendees will use Apache Calcite to build a fully fledged query processor from scratch with very few lines of code. This processor is a full implementation of SQL over an Apache Lucene storage engine. (Lucene does not support SQL queries and lacks a declarative language for performing complex operations such as joins or aggregations.) Attendees will also learn how to use Calcite as an effective tool for research.
This document introduces Spark SQL and the Catalyst query optimizer. It discusses that Spark SQL allows executing SQL on Spark, builds SchemaRDDs, and optimizes query execution plans. It then provides details on how Catalyst works, including its use of logical expressions, operators, and rules to transform query trees and optimize queries. Finally, it outlines some interesting open issues and how to contribute to Spark SQL's development.
Apache Calcite: One Frontend to Rule Them AllMichael Mior
Apache Calcite is an open source framework that allows for a unified query interface over heterogeneous data sources. It provides an ANSI-compliant SQL parser, a logical query optimizer, and acts as a middleware layer that can integrate data from multiple sources. Calcite uses a relational algebra approach and has pluggable adapters that allow it to connect to different backends like MySQL, MongoDB, and streaming data sources. It supports features like SQL queries, views, optimization rules, and works across both batch and streaming data. The project aims to continue adding new capabilities like geospatial queries and improved cost modeling.
The document describes new features and enhancements in MySQL 8.0, including common table expressions, window functions, improved UTF-8 support, geospatial functions, new locking options for SELECT statements, JSON functions, index extensions, cost model improvements, query hints, and better support for IPv6 and UUID data types. The presentation agenda outlines each topic at a high level.
This document provides an overview of the MySQL query optimizer. It discusses the main phases of the optimizer including logical transformations, cost-based optimizations, analyzing access methods, join ordering, and plan refinements. Logical transformations prepare the query for cost-based optimization by simplifying conditions. Cost-based optimizations select the optimal join order and access methods to minimize resources used. Access methods analyzed include table scans, index scans, and ref access. The join optimizer searches for the best join order. Plan refinements include sort avoidance and index condition pushdown.
Presentation of Common Table Expressions (CTE), recursive or not , a new feature in MySQL 8.0; slides written by Guilhem Bichot, developer of the feature, and presented by him at the Percona Live Conference in Dublin on 2017-09-26.
The document provides an overview of the MySQL query optimizer. It discusses how the optimizer performs logical transformations, cost-based optimizations, analyzes access methods, and optimizes join orders. The goal of the optimizer is to produce a query execution plan that uses the least resources. It considers factors like I/O and CPU costs to select optimal table access methods, join orders, and other optimizations to minimize the cost of executing the query.
This document discusses execution plans in Oracle Database. It begins by explaining what an execution plan is and how it shows the steps needed to execute a SQL statement. It then covers how to generate an execution plan using EXPLAIN PLAN or querying V$SQL_PLAN. The document discusses what the optimizer considers a "good" plan in terms of cost and performance. It also explores key elements of an execution plan like cardinality, access paths, join methods, and join order.
MySQL 8.0: What Is New in Optimizer and Executor?Norvald Ryeng
This document summarizes new features in MySQL 8.0 related to the optimizer and executor. It introduces common table expressions (CTEs) which provide better readability, allow references to the same CTE multiple times, and can refer to other CTEs. It also discusses window functions, UTF-8 support improvements, new geospatial features, the SKIP LOCKED and NOWAIT clauses for SELECT statements, expanded JSON functions and data type, index extensions, updates to the cost model, and query hints.
This document provides information about new features and improvements in MySQL 8.0. It discusses enhancements to JSON functionality including new functions and indexing support. It also summarizes added functionality for GIS, UUIDs, common table expressions, window functions, and other query optimizations. The document notes that MySQL 8.0 uses utf8mb4 as the default character set for improved Unicode support and performance.
Developers’ mDay u Banjoj Luci - Bogdan Kecman, Oracle – MySQL Server 8.0mCloud
This document provides information about new features and improvements in MySQL 8.0. It discusses enhancements to JSON functionality including new functions and indexing support. It also summarizes added functionality for GIS, Unicode character sets, UUIDs, window functions, common table expressions, and other query optimizations. The document outlines goals of improving performance, manageability, security and standards compliance for MySQL.
The document discusses new features and performance improvements in MySQL 5.7 InnoDB. Key features include native partitioning with reduced memory overhead, import/export of partitions, general tablespaces, larger page sizes up to 64K, spatial indexes, virtual columns and indexes, tablespace encryption, and transparent page compression. Performance improvements come from transaction pools, faster DDL operations, improved buffer pool management, and index optimizations. Benchmark results show increased performance for point selects, read-only operations, and read-write workloads compared to earlier versions.
The document discusses various techniques for optimizing database performance in Oracle, including:
- Using the cost-based optimizer (CBO) to choose the most efficient execution plan based on statistics and hints.
- Creating appropriate indexes on columns used in predicates and queries to reduce I/O and sorting.
- Applying constraints and coding practices like limiting returned rows to improve query performance.
- Tuning SQL statements through techniques like predicate selectivity, removing unnecessary objects, and leveraging indexes.
How to analyze and tune sql queries for better performance vts2016oysteing
This document provides an agenda and overview for a presentation on analyzing and tuning MySQL queries for better performance. The presentation covers topics like cost-based query optimization in MySQL, tools for monitoring queries, index selection, the join optimizer, and influencing the optimizer. It also provides examples of using tools like the performance schema, EXPLAIN, and optimizer traces to analyze query performance.
Common Table Expressions (CTE) & Window Functions in MySQL 8.0oysteing
This document discusses common table expressions (CTE) and window functions in MySQL 8.0. It provides examples of using CTEs to improve readability, allow referencing tables multiple times, and refer to other CTEs. It also discusses recursive CTEs, window functions, and examples calculating aggregates like sums over partitions of rows.
Databases are the heart of most PHP projects but roughly TWO PERCENT of PHP programmers have had any real training in Structured Query Language, SQL. Then they wonder why their queries perform poorly, why they get N+1 problems, and suddenly the database becomes the choke point of the project. This presentation will cover the basics of relational algebra (no algebra, math or calculus skills needed!!!!), how to think in sets with Venn Diagrams, and how to let the database do the heavy lifting for you. So if you want to write high performing database queries and be admired as a database deity by your co workers then you need to be in this session!
15 Ways to Kill Your Mysql Application Performanceguest9912e5
Jay is the North American Community Relations Manager at MySQL. Author of Pro MySQL, Jay has also written articles for Linux Magazine and regularly assists software developers in identifying how to make the most effective use of MySQL. He has given sessions on performance tuning at the MySQL Users Conference, RedHat Summit, NY PHP Conference, OSCON and Ohio LinuxFest, among others.In his abundant free time, when not being pestered by his two needy cats and two noisy dogs, he daydreams in PHP code and ponders the ramifications of __clone().
The document discusses new improvements to the parser and optimizer in MySQL 5.7. Key points include:
1) The parser and optimizer were refactored for improved maintainability and stability. Parsing was separated from optimization and execution.
2) The cost model was improved with better record estimation for joins, configurable cost constants, and additional explain output.
3) A new query rewrite plugin allows rewriting queries without changing application code.
DBA Commands and Concepts That Every Developer Should KnowAlex Zaballa
The document provides an overview of various DBA commands and concepts that every developer should know, including:
- Explain Plan and how it can sometimes provide inaccurate predictions of query performance
- Using DBMS_XPLAN.DISPLAY_CURSOR to view the actual execution plan for a SQL statement
- SQL Monitor and tools like SQLTXPLAIN and SQLD360 for monitoring SQL performance
- Factors that influence whether a Full Table Scan is more efficient than an index scan
- Techniques for analyzing tables, managing statistics, and using invisible indexes.
This document provides 20 tips and techniques for SAS programming. It begins with tips for creating pivot tables from SAS data in Excel and using Visual Basic scripts. It then provides tips on debugging complex macros, adding operators to the macro language, and using progress bars in SAS. The document continues sharing many other tips for optimizing SAS code, such as using views to improve efficiency, finding secret SAS options, using pipes to process large files, and formatting tables to visually indicate high and low values.
MySQL 8.0.18 latest updates: Hash join and EXPLAIN ANALYZENorvald Ryeng
This presentation focuses on two of the new features in MySQL 8.0.18: hash joins and EXPLAIN ANALYZE. It covers how these features work, both on the surface and on the inside, and how you can use them to improve your queries and make them go faster.
Both features are the result of major refactoring of how the MySQL executor works. In addition to explaining and demonstrating the features themselves, the presentation looks at how the investment in a new iterator based executor prepares MySQL for a future with faster queries, greater plan flexibility and even more SQL features.
MySQL Goes to 8! FOSDEM 2020 Database Track, January 2nd, 2020Geir Høydalsvik
Here are the basic steps to clone a MySQL instance using the new CLONE command directly from SQL:
1. Connect to the source instance you want to clone from.
2. Issue the CLONE statement to create a new instance from the source. For example:
CLONE INSTANCE FROM 'mysql://user:password@source_host:3306/' TO 'mysql://user:password@target_host:3306/';
3. The clone operation will copy over the data files, redo logs and configuration from the source to the target instance.
4. Once complete, the new cloned instance is ready for use as a read replica or independent instance as needed.
By automating the provisioning
The cost model is one of the core components of the MySQL optimizer. This presentation gives an overview over the MySQL Optimizer Cost Model, what is new in 5.7 and some ideas for further improvements.
Similar to MySQL Optimizer: What's New in 8.0 (20)
A Study of Variable-Role-based Feature Enrichment in Neural Models of CodeAftab Hussain
Understanding variable roles in code has been found to be helpful by students
in learning programming -- could variable roles help deep neural models in
performing coding tasks? We do an exploratory study.
- These are slides of the talk given at InteNSE'23: The 1st International Workshop on Interpretability and Robustness in Neural Software Engineering, co-located with the 45th International Conference on Software Engineering, ICSE 2023, Melbourne Australia
Hand Rolled Applicative User ValidationCode KataPhilip Schwarz
Could you use a simple piece of Scala validation code (granted, a very simplistic one too!) that you can rewrite, now and again, to refresh your basic understanding of Applicative operators <*>, <*, *>?
The goal is not to write perfect code showcasing validation, but rather, to provide a small, rough-and ready exercise to reinforce your muscle-memory.
Despite its grandiose-sounding title, this deck consists of just three slides showing the Scala 3 code to be rewritten whenever the details of the operators begin to fade away.
The code is my rough and ready translation of a Haskell user-validation program found in a book called Finding Success (and Failure) in Haskell - Fall in love with applicative functors.
DDS Security Version 1.2 was adopted in 2024. This revision strengthens support for long runnings systems adding new cryptographic algorithms, certificate revocation, and hardness against DoS attacks.
Do you want Software for your Business? Visit Deuglo
Deuglo has top Software Developers in India. They are experts in software development and help design and create custom Software solutions.
Deuglo follows seven steps methods for delivering their services to their customers. They called it the Software development life cycle process (SDLC).
Requirement — Collecting the Requirements is the first Phase in the SSLC process.
Feasibility Study — after completing the requirement process they move to the design phase.
Design — in this phase, they start designing the software.
Coding — when designing is completed, the developers start coding for the software.
Testing — in this phase when the coding of the software is done the testing team will start testing.
Installation — after completion of testing, the application opens to the live server and launches!
Maintenance — after completing the software development, customers start using the software.
Microservice Teams - How the cloud changes the way we workSven Peters
A lot of technical challenges and complexity come with building a cloud-native and distributed architecture. The way we develop backend software has fundamentally changed in the last ten years. Managing a microservices architecture demands a lot of us to ensure observability and operational resiliency. But did you also change the way you run your development teams?
Sven will talk about Atlassian’s journey from a monolith to a multi-tenanted architecture and how it affected the way the engineering teams work. You will learn how we shifted to service ownership, moved to more autonomous teams (and its challenges), and established platform and enablement teams.
Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics.
To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/
May Marketo Masterclass, London MUG May 22 2024.pdfAdele Miller
Can't make Adobe Summit in Vegas? No sweat because the EMEA Marketo Engage Champions are coming to London to share their Summit sessions, insights and more!
This is a MUG with a twist you don't want to miss.
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...Crescat
Crescat is industry-trusted event management software, built by event professionals for event professionals. Founded in 2017, we have three key products tailored for the live event industry.
Crescat Event for concert promoters and event agencies. Crescat Venue for music venues, conference centers, wedding venues, concert halls and more. And Crescat Festival for festivals, conferences and complex events.
With a wide range of popular features such as event scheduling, shift management, volunteer and crew coordination, artist booking and much more, Crescat is designed for customisation and ease-of-use.
Over 125,000 events have been planned in Crescat and with hundreds of customers of all shapes and sizes, from boutique event agencies through to international concert promoters, Crescat is rigged for success. What's more, we highly value feedback from our users and we are constantly improving our software with updates, new features and improvements.
If you plan events, run a venue or produce festivals and you're looking for ways to make your life easier, then we have a solution for you. Try our software for free or schedule a no-obligation demo with one of our product specialists today at crescat.io
What is Augmented Reality Image Trackingpavan998932
Augmented Reality (AR) Image Tracking is a technology that enables AR applications to recognize and track images in the real world, overlaying digital content onto them. This enhances the user's interaction with their environment by providing additional information and interactive elements directly tied to physical images.
E-commerce Development Services- Hornet DynamicsHornet Dynamics
For any business hoping to succeed in the digital age, having a strong online presence is crucial. We offer Ecommerce Development Services that are customized according to your business requirements and client preferences, enabling you to create a dynamic, safe, and user-friendly online store.
Neo4j - Product Vision and Knowledge Graphs - GraphSummit ParisNeo4j
Dr. Jesús Barrasa, Head of Solutions Architecture for EMEA, Neo4j
Découvrez les dernières innovations de Neo4j, et notamment les dernières intégrations cloud et les améliorations produits qui font de Neo4j un choix essentiel pour les développeurs qui créent des applications avec des données interconnectées et de l’IA générative.
Zoom is a comprehensive platform designed to connect individuals and teams efficiently. With its user-friendly interface and powerful features, Zoom has become a go-to solution for virtual communication and collaboration. It offers a range of tools, including virtual meetings, team chat, VoIP phone systems, online whiteboards, and AI companions, to streamline workflows and enhance productivity.
Artificia Intellicence and XPath Extension FunctionsOctavian Nadolu
The purpose of this presentation is to provide an overview of how you can use AI from XSLT, XQuery, Schematron, or XML Refactoring operations, the potential benefits of using AI, and some of the challenges we face.
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Takashi Kobayashi and Hironori Washizaki, "SWEBOK Guide and Future of SE Education," First International Symposium on the Future of Software Engineering (FUSE), June 3-6, 2024, Okinawa, Japan