Well designed tables
and well written queries can improve your query speed and
reduce processing cost.
Optimization on Table side
Partitioning Hive Tables:
It is a kind of horizontal slicing of data. This slicing can be
on the range, single value or a set of values.
Imagine log files where each record includes a timestamp. If
we partitioned by date, then records for the same date would
be stored in the same partition.
Partition on date.
Partition on geography location.
Partition on number range.
Defining a table partition
Lets take a Apache log file example where we have log generated by web
server on visit of client.
These log contains data & time information about browser and location(IP).
So we can create table in hive and partition these log data using date & time
and we can create sub partition of location. Which looks like :
CREATE TABLE alogs (timstamp BIGINT, detail STRING) PARTITIONED BY (date STRING, loc STRING);
Bucketing Hive Tables:
Bucketing hive table result in more efficient queries.
Bucketing imposes extra structure on the table, which Hive
can take advantage of when performing certain queries.
It makes sampling more efficient.
The two tables are bucketed in the same way, a mapper
processing a bucket of the left table knows that the
matching rows in the right table are in its corresponding
bucket, so it need only retrieve that bucket.
Bucket may additionally be sorted by one or more columns.
This allows even more efficient map-side joins, since the
join of each bucket becomes an efficient merge-sort.
Parallel execution of queries
Hadoop can execute map reduce jobs in parallel and several queries executed on Hive make
automatically use of this parallelism.
The queries or sub queries which are not interdependent can be execute in parallel mode,like
some Join queries.
Following is the example how it is done:
SET hive.exce.parallel=true; #Can be used to set this mode on
Sub query 1
(1 & 2) Joined
Sub query 2
Query (1 & 2)
& 3 Joined
Sub query 3
So in the above flow, 1,2,4 can run in parallel as sub queries and
then joined finally to 3 and then to 5 and the final query result.
Since map join is faster than the common join, it's better to run the
map join whenever possible. Previously, Hive users needed to
give a hint in the query to specify the small table.
select /*+mapjoin(a)*/ * from src1 x join src2 y on x.key=y.key;
Newer hive automatically converts normal join to map join.
Which query is faster?
Select count(distinct(column)) from table.
Select count(*) from (select distinct(column) from table) ??
2nd one is faster
In first case :
Maps send each value to reducer
Single reducer counts them all(over head)
In Second Case:
Map splits the values to many reducer
Each reducer generated a list
Final job is to count the size of each list
Note : Singleton reducer is not always good.
Hive does not know whether query is bad.
So try to use “Explain” for queries which you doubt to be bad or
even don’t doubt.
Explain tells about following
Number of jobs
Number of map and reduce
What job is sorting by
What are the directories it will read.
So explain will help to see the difference between the two or
more queries for the same purpose.
Job configuration and history can be studied for the query