Query Optimization, Importance, Query Processing, A Query-Evaluation Plan, Cost based Optimization (Physical), Heuristic Optimization (Logical), External sorting, External Merge sort, two-way merge sort

1. Prof. Neeraj Bhargava
Pooja Dixit
Department of Computer Science
School of Engineering & System Science
MDS, University Ajmer, Rajasthan, India
1

2.  Query optimization is a function of many relational database management systems. The
query optimizer attempts to determine the most efficient way to execute a given query
by considering the possible query plans.
 Generally, the query optimizer cannot be accessed directly by users: once queries are
submitted to the database server, and parsed by the parser, they are then passed to the
query optimizer where optimization occurs.
 Query optimization is a combination of:-
◦ Query: A query is a request for information from a database.
◦ Query Plans: A query plan (or query execution plan) is an ordered set of steps used to
access data in a SQL relational database management system.
◦ Query Optimization: A single query can be executed through different algorithms or
re-written in different forms and structures. Hence, the question of query
optimization comes into the picture – Which of these forms or pathways is the most
optimal? The query optimizer attempts to determine the most efficient way to execute
a given query by considering the possible query plans.
2

3.  Importance: The goal of query optimization is to reduce the system resources required to
fulfill a query, and ultimately provide the user with the correct result set faster.
◦ First, it provides the user with faster results, which makes the application seem faster to the user.
◦ Secondly, it allows the system to service more queries in the same amount of time, because each
request takes less time than unoptimized queries.
◦ Thirdly, query optimization ultimately reduces the amount of wear on the hardware (e.g. disk drives),
and allows the server to run more efficiently (e.g. lower power consumption, less memory usage).
 There are broadly two ways a query can be optimized:
◦ Analyze and transform equivalent relational expressions: Try to minimize the tuple and column counts
of the intermediate and final query processes (discussed here).
◦ Using different algorithms for each operation: These underlying algorithms determine how tuples are
accessed from the data structures they are stored in, indexing, hashing, data retrieval and hence
influence the number of disk and block accesses (discussed in query processing).
3

4.  Query processing refers to the range of activities involved in extracting data from a database.
The activities include translation of queries in high-level database languages into
expressions that can be used at the physical level of the file system, a variety of query-
optimizing transformations, and actual evaluation of queries.
 Overview
 The steps involved in processing a query appear in Figure. The basic steps are:
◦ Parsing and translation.
◦ Optimization.
◦ Evaluation.
4

5.  Before query processing can begin, the system must translate the query into a usable form. A
language such as SQL is suitable for human use, but is ill suited to be the system’s internal
representation of a query. A more useful internal representation is one based on the
extended relational algebra.
 Given a query, there are generally a variety of methods for computing the answer. For
example, we have seen that, in SQL, a query could be expressed in several different ways.
Each SQL query can itself be translated into a relationalalgebra expression in one of several
ways. Furthermore, the relational-algebra representation of a query specifies only partially
how to evaluate a query; there are usually several ways to evaluate relational-algebra
expressions. As an
select salary from instructor where salary < 75000;
 This query can be translated into either of the following relational-algebra expressions:
5

6.  Further, we can execute each relational-algebra operation by one of several different
algorithms. For example, to implement the preceding selection, we can search every tuple in
instructor to find tuples with salary less than 75000. If a B+-tree index is available on the
attribute salary, we can use the index instead to locate the tuples.
A Query-Evaluation Plan
6

7.  A sequence of primitive operations that can be used to evaluate a query is a query-
execution plan or query-evaluation plan.
 The query-execution engine takes a query-evaluation plan, executes that plan, and
returns the answers to the query.
 The query optimizer uses these two techniques to determine which process or
expression to consider for evaluating the query.
 There are two methods of query optimization.
1. Cost based Optimization (Physical)
2. Heuristic Optimization (Logical)
7

8.  Cost-Based Optimization also known as Cost-Based Query
Optimization or CBO Optimizer) is an optimization technique in
Spark SQL that uses table statistics to determine the most efficient
query execution plan of a structured query (given the logical query
plan).
 Cost-based optimization is disabled by default. Spark SQL uses
spark.sql.cbo.enabled configuration property to control whether
the CBO should be enabled and used for query optimization or not.
 Cost-Based Optimization uses logical optimization rules (e.g.
CostBasedJoinReorder) to optimize the logical plan of a structured
query based on statistics.
8

9. Heuristic Based Optimization
◦ Heuristic based optimization uses rule-based optimization approaches
for query optimization. These algorithms have polynomial time and
space complexity, which is lower than the exponential complexity of
exhaustive search-based algorithms. However, these algorithms do not
necessarily produce the best query plan.
◦ Some of the common heuristic rules are −
 Perform select and project operations before join operations. This is
done by moving the select and project operations down the query
tree. This reduces the number of tuples available for join.
 Perform the most restrictive select/project operations at first before
the other operations.
 Avoid cross-product operation since they result in very large-sized
intermediate tables.
9

10.  External sorting is a technique in which the data is stored on the secondary memory, in
which part by part data is loaded into the main memory and then sorting can be done over
there. Then this sorted data will be stored in the intermediate files. Finally, these files will be
merged to get a sorted data. Thus by using the external sorting technique, a huge amount of
data can be sorted easily. In case of external sorting, all the data cannot be accommodated
on the single memory, in this case, some amount of memory needs to be kept on a memory
such as hard disk, compact disk and so on.
 The requirement of external sorting is there, where the data we have to store in the main
memory does not fit into it. Basically, it consists of two phases that are:
 Sorting phase: This is a phase in which a large amount of data is sorted in an intermediate
file.
 Merge phase: In this phase, the sorted files are combined into a single larger file.
10

11.  One of the best examples of external sorting is external merge sort.
 External merge sort
 The external merge sort is a technique in which the data is stored in intermediate files and
then each intermediate files are sorted independently and then combined or merged to get a
sorted data.
 For example: Let us consider there are 10,000 records which have to be sorted. For this, we
need to apply the external merge sort method. Suppose the main memory has a capacity to
store 500 records in a block, with having each block size of 100 records.
11

12.  In this example, we can see 5 blocks will be sorted in intermediate files. This
process will be repeated 20 times to get all the records. Then by this, we start
merging a pair of intermediate files in the main memory to get a sorted output.
 Two-Way Merge Sort
 Two-way merge sort is a technique which works in two stages which are as follows
here:
◦ Stage 1: Firstly break the records into the blocks and then sort the individual record with
the help of two input tapes.
◦ Stage 2: In this merge the sorted blocks and then create a single sorted file with the help
of two output tapes.
 By this, it can be said that two-way merge sort uses the two input tapes and two
output tapes for sorting the data.
12

13.  Algorithm for Two-Way Merge Sort:
 Step 1) Divide the elements into the blocks of size M. Sort each block and then write
on disk.
 Step 2) Merge two runs
◦ Read first value on every two runs.
◦ Then compare it and sort it.
◦ Write the sorted record on the output tape.
 Step 3) Repeat the step 2 and get longer and longer runs on alternates tapes.
Finally, at last, we will get a single sorted list.
13

14.  Analysis
 This algorithm requires log(N/M) passes with initial run pass.
Therefore, at each pass the N records are processed and at last we
will get a time complexity as O(N log(N/M).
14