This document discusses data warehousing over Hadoop and strategies for low-latency querying. It covers columnar formats like ORC and Parquet that provide optimizations for queries. It also discusses different query engines like Hive, Impala, Presto, and Spark SQL and their capabilities. The document notes that converting data to optimized formats during or after collection can help minimize latency conflicts between processing and query performance. Technologies like Sqoop, Hive ACID transactions, streaming ingest into Hive, and Flume can help convert or ingest data in optimized formats with lower latency.

1. DWH OVER HADOOPDWH OVER HADOOP

2. THETHE
BASICSBASICS

3. COLUMNAR FORMATS (ORC/PARQUET)COLUMNAR FORMATS (ORC/PARQUET)
Projection Push Down
Predicate Push Down
Excellent Compression Ratios
Column Indices
Max/Avg/Min values
Rows must be batched to benefit from these optimizations

4. PARQUETPARQUET
Strongly endorsed by Cloudera
One of the few formats Impala
supports (and the most optimal
for it)
Also supported by Hive, Spark,
Tajo, Drill & Presto.
Speaking from my
own personal experience a bit
more expensive to generate.
ORCORC
Endorsed by Hortonworks
Most optimal for Presto
Spark support was recently
introduced.

5. QUERYINGQUERYING
ENGINESENGINES

6. HIVEHIVE
Hive provides a SQL like interface of
accessing the data (files) called HiveQL.
The HQL is translated into
M/R code and executed immediately.
Batch Oriented
Fault tolerant and thus reliable
Not a DB!
Does not support updates & delete and has
no transaction (or does it ?)

7. LOWLOW
LATENCYLATENCY
SQLSQL
Map-Reduce can be compared to
a Tractor:
It's very strong and can plow a
field better than any other vehicle,
but it's also very slow.
As prices of memory dropped, a
demand emerged to better utilize
it for faster response times.

8. CLOUDERA IMPALACLOUDERA IMPALA
Writen in C++
Utilizes Hive's metadata
Very fast
Not fault tolerante
Doesn't support custom data
formats
Doesn't support complex data
types (maps/arrays/structs)
A bit complicated setup for non
CDH distributions

9. FACEBOOK PRESTOFACEBOOK PRESTO
Can connect to:
Cassandra
Hive
JMX Sources
Postgres & Mysql
Allows cross engine joins
Used in Facebook to serve online
dashboards
Easy to setup

10. SPARK SQLSPARK SQL
Not affiliated with any Hadoop
vendor
Support all of the optimized file
formats (ORC/Parquet/Avro)
Can auto discover schema
Aims to provide second/sub-
second latnecy
Still not very mature

11. THE USUAL DATA FLOWTHE USUAL DATA FLOW
Collect -> Store -> Convert -> Select
The Data Latency conflict - lots
of fragmented small files or big
optimized files with big latency
Processing efforts involved in
the conversion process should
be minimized
Example..

12. A BETTER DATA FLOWA BETTER DATA FLOW
Collec-tor-vert -> Select
Convert the data as it is being
collected where possible
Or convert the data as it is
being stored (streaming) but
without losing optimizations
How can this be achieved?

13. SQOOPSQOOP
Import data from RDBMS into
Hadoop
Create java classes and hive
tables on import
Export data back to RDBMS
Runs a "Map Only" job to
perform the task
Supports incremental imports
Now supports import right
away as Parquet

14. HIVE & ACIDHIVE & ACID
Recently a conceptual change has been
introduced into Hive: CRUD with ACID
Transactions.
It is not meant to replace your OLTP but
rather supply a better data modification
mechanism to a subset of the data.
Explanation on how it works
Demo simple insert
Still requires M/R :(

15. HIVE & STREAMING INGESTHIVE & STREAMING INGEST
With the new ACID capabilities it is now
possible to continously insert data into hive
Data apperas almost immediately
Data is optimized in a columnar format
Data is compacted by different triggers
Code snippet

16. FLUMEFLUME
Distributed
Durable
Scalable
Fault Tolerante
Serves for ingestion and basic
pre-processing of the data
Composed of
source -> channel -> Sink
(Draw Architecture)
Utilized Hive's ACID capabilities
to instantly stream data into hive
- demo