In today's data-driven world the challenge is to find meaningful nuggets of information quickly and efficiently. This requires an effective infrastructure. This talk presents few simple ideas to make big data infrastructure efficient and effective using simple strategies such as partitioning, bucketing, etc.

1. Simple Strategies For
Faster Knowledge
Discovery In Big Data
Dr. Ritesh Agrawal
Lead Data Scientist
Zheng Shao
Staff Software Engineer

2. * 6 continents, 70+ countries, 450 cities
* 1st billionth trip took 5 years. 2nd billionth trip took 1.5 years
* 5+ million trips every day

3. Data-Driven Culture

4. Infrastructure Challenge
*Help discover
*Quickly: having low query latency
*Efficiently: minimal cost

5. Data
eg: log presto queries
Analytics
eg: 20% compute
resources used by timeout
queries
How to Build An Effective Infrastructure
Optimization
eg: build query gate
Forecasting
eg: predict future
requirements

6. PRESTO HIVE
HDFS
KAFKA
*User
*Tables
*Join Clause
*Filter Clause
*Execution Related
Metrics
Data Analytics Optimization Forecasting

7. *Most queried tables
*Most expensive queries
*Top N users
*…
Overview Analytics
*Filter Clause
*Most Joined tables
*Hot & Cold Partitions…
Detailed Analytics
Data Analytics Optimization Forecasting

8. Data Analytics Optimization Forecasting
others
with
create
select
insert
PercentQueries
0
20
40
60
80
Number of Joins
0 1 2 3 4 5+
Pct of Total Queries Failure Pct in Bucket

9. 90% of compute resources
is consumed by 10% most
expensive queries

10. * 90% of queries using [table A] filter on [column x]
* 70% of queries using [table A] join to [table B] on
column X
* 90% of queries using [table A, Table B] end using
Column X, Y from table A and Column Z from table B
Data Analytics Optimization Forecasting

11. CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
[(col_name data_type [COMMENT col_comment], ... [constraint_specification])]
[ PARTITIONED BY (col_name data_type [COMMENT col_comment], ...)]
[
CLUSTERED BY (col_name, col_name, ...)
[SORTED BY (col_name [ASC|DESC], ...)] INTO num_buckets BUCKETS
]
[
SKEWED BY (col_name, col_name, ...)
ON ((col_value, col_value, ...), (col_value, col_value, ...), ...)
]
[
[ROW FORMAT row_format]
[STORED AS file_format]
]
Data Analytics Optimization Forecasting

12. CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
[(col_name data_type [COMMENT col_comment], ... [constraint_specification])]
[ PARTITIONED BY (col_name data_type [COMMENT col_comment], ...)]
[
CLUSTERED BY (col_name, col_name, ...)
[SORTED BY (col_name [ASC|DESC], ...)] INTO num_buckets BUCKETS
]
[
SKEWED BY (col_name, col_name, ...)
ON ((col_value, col_value, ...), (col_value, col_value, ...), ...)
]
[
[ROW FORMAT row_format]
[STORED AS file_format]
]
Data Analytics Optimization Forecasting

13. Optimization > Partitioned By
|—user
|—db
|— table
|—date=20170101
|—file_00000
|—file_00000
|—date=20170102
|—file_00000
|—file_00000
|—date=20170103
|—file_00000
|—file_00000
SELECT *
FROM [table]
WHERE
date = 20170102
AND event_type = ‘click’

14. Optimization > Partitioned By
|—user
|—db
|— table
|—date=20170101
|—file_00000
|—file_00001
|—date=20170102
|—file_00000
|—file_00001
|—date=20170103
|—file_00000
|—file_00001
SELECT *
FROM [table]
WHERE
date = 20170102
AND event_type = ‘click’

15. Optimization > Partitioned By
|—user
|—db
|— table
|—date=20170101
|—event_type=‘click’
|—file_00000
|—file_00001
|—date=20170102
|—event_type=‘click’
|—file_00000
|—file_00001
|—event_type=‘map’
|—file_00000
|—file_00001
|—….
SELECT *
FROM [table]
WHERE
date = 20170102
AND event_type = ‘click’

16. Optimization > Partitioned By
*A query that used to take 10 minutes now completes within 2 minutes. 80% reduction
in elapsed time
*95% reduction in resource consumption.

17. Partitioned By: Key Considerations
* Usage:
* Identify fields that are often used for filtering (FILTER CLAUSE)
* Cardinality:
* Low Cardinality — otherwise explodes meta-store
* Skewness:
* data should evenly distributed if all the keys are popular
* skewness is okie if filter value corresponds to smaller dataset

18. CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
[(col_name data_type [COMMENT col_comment], ... [constraint_specification])]
[ PARTITIONED BY (col_name data_type [COMMENT col_comment], ...)]
[
CLUSTERED BY (col_name, col_name, ...)
[SORTED BY (col_name [ASC|DESC], ...)] INTO num_buckets BUCKETS
]
[
SKEWED BY (col_name, col_name, ...)
ON ((col_value, col_value, ...), (col_value, col_value, ...), ...)
]
[
[ROW FORMAT row_format]
[STORED AS file_format]
]
Data Analytics Optimization Forecasting

19. Optimization > Clustered By/Bucketing

20. Optimization > Clustered By/Bucketing

21. * Identify tables that are often joined together.
* Cluster tables on join key.
Clustered By/Bucketing: Key Considerations

22. CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
[(col_name data_type [COMMENT col_comment], ... [constraint_specification])]
[ PARTITIONED BY (col_name data_type [COMMENT col_comment], ...)]
[
CLUSTERED BY (col_name, col_name, ...)
[SORTED BY (col_name [ASC|DESC], ...)] INTO num_buckets BUCKETS
]
[
SKEWED BY (col_name, col_name, ...)
ON ((col_value, col_value, ...), (col_value, col_value, ...), ...)
]
[
[ROW FORMAT row_format]
[STORED AS file_format]
]
Data Analytics Optimization Forecasting

23. StorageSKEWED BY
* Row vs Col.
* Significant improvement
by moving to ORC or
parquet format.
* splits data so that heavy
values are stored in
separate files
* Helps with query
optimization.
Data Analytics Optimization Forecasting
Other techniques
* execution engine:
MapReduce Vs Tez
* Vectorization
* Pre-joined tables

24. Key Takeaways
! Fast and efficient infrastructure is key to a business’s success.
! Infrastructure Optimization is a constantly ongoing process.
! Infrastructure Data Science is key to building an efficient infrastructure
! Start simple

25. First & Last Name
Ritesh Agrawal,
Lead Data Scientist, Uber Inc.
Thank you