The document discusses various big data processing engines including Hive, HBase, Druid, and Spark, outlining their specific use cases and advantages. It emphasizes that each engine has a distinct niche, such as HBase for operational data, Druid for analytics and real-time queries, Hive for large-scale analytical queries, and Spark for complex ETL and machine learning tasks. The conclusion highlights the importance of selecting the right tool for the job, noting that no single engine can fulfill all processing needs.

1. © Hortonworks Inc. 2011- 2017. All rights reserved | 1
3.77 288.9 0.76

2. © Hortonworks Inc. 2011- 2017. All rights reserved | 2
Counting Rows

3. © Hortonworks Inc. 2011- 2017. All rights reserved | 3
3.77 288.9 0.76

4. © Hortonworks Inc. 2011- 2017. All rights reserved | 4
3.77 288.9 0.76
Hive HBase Druid

5. © Hortonworks Inc. 2011- 2017. All rights reserved | 5
Ad-hoc
analytics
Random
look-ups
Aggregations
drill-downs

6. © Hortonworks Inc. 2011 – 2017
Big Data Processing Engines –
Which one do I use?
Ashish Narasimham, Solutions Engineer @ Cloudera

7. © Hortonworks Inc. 2011- 2017. All rights reserved | 7
Processing Engines Overview

8. © Hortonworks Inc. 2011- 2017. All rights reserved | 8
Ad-hoc
analytics
Random
look-ups
Aggregations
drill-downs

9. What’s Hive good at?
⬢ Jack of all trades
⬢ Key component of the real-time
database
⬢ Familiar interface for analysts – unified
SQL
⬢ Can perform joins, filtering,
aggregations
⬢ Read structured (CSV) or semi-
structured (JSON) data
HiveInterface
HBase/Phoenix
Druid
JDBC
Files

10. © Hortonworks Inc. 2011- 2017. All rights reserved | 10
HDP3: EDW analyst pipeline
Tableau
BI systems
Materialized
view
Surrogate
key
Constraints
Query
Result
Cache
Workload
management
ACID v2
&
ACID on
default
• Results return
from HDFS/cache
directly
• Reduce load from
repetitive queries
• Allows more
queries to be run
in parallel
• Reduce resource
starvation in large
clusters
• Also:
Active/Passive HA
• More “tools” for
optimizer to use
• More ”tools” for
DBAs to
tune/optimize
• Invisible tuning of
DB from users’
perspective
• ACID v2 is as fast
as regular tables

11. © Hortonworks Inc. 2011- 2017. All rights reserved | 11
Ad-hoc,
analytics
Look-ups,
updates
Aggregations,
drill-downs

12. What Are Apache HBase and Phoenix?
Flexible Schema
Millisecond Latency
SQL and NoSQL Interfaces
Store and Process Petabytes of Data
Scale out on Commodity Servers
Integrated with YARN
100% Open Source
YARN : Data Operating System
HBase
RegionServer
1 ° ° ° ° ° ° ° ° ° °
° ° ° ° ° ° ° ° ° ° N
HDFS
(Permanent Data Storage)
HBase
RegionServer
HBase
RegionServer
Flexible Schema
Extreme Low Latency
Directly Integrated with Hadoop
SQL and NoSQL Interfaces
What Are Apache HBase and Phoenix?
Flexible Schema
Millisecond Latency
SQL and NoSQL Interfaces
Store and Process Petabytes of Data
Scale out on Commodity Servers
Integrated with YARN
100% Open Source
YARN : Data Operating System
HBase
RegionServer
1 ° ° ° ° ° ° ° ° ° °
° ° ° ° ° ° ° ° ° ° N
HDFS
(Permanent Data Storage)
HBase
RegionServer
HBase
RegionServer
Flexible Schema
Extreme Low Latency
Directly Integrated with Hadoop
SQL and NoSQL Interfaces

13. Kinds of Apps Built with HBase
Write Heavy Low-Latency
Search /
Indexing
Messaging
Audit /
Log Archive AdvertisingData Cubes
Time Series
Sensor /
Device

14. Key HBase Features
Page 14
High Availability
• Data is stored on multiple
nodes and HBase coordinates
failover.
• Data stays available if nodes
fail.
Strong Consistency
• HBase doesn’t sacrifice
consistency for scale.
• Improve quality by avoiding
difficult-to-detect bugs.
Deep Hadoop Integration
• Add deep insight to your apps
through seamless integration
with Hadoop tools like Hive
Multi Datacenter
• Replicate data between 2 or
more datacenters.
• Keeps data safe and available
through datacenter outages.

15. Data Storage – Relational vs. HBase
Column1 Column2 Column3 Column4
Row1 f - t5
a – t1
null null d – t4
Row2 null b – t1 null null
Row3 null null null e – t4
Row4 c – t3 null g – t5 null
Relational Data Base
f – t5
a – t1
C – t3
B – t1 g – t5 d – t4
e – t4
HBase Data is located by cell coordinates consisting of row key,
column family name, column qualifier and timestamp
Column1 Column2 Column3 Column4
HFile

16. © Hortonworks Inc. 2011- 2017. All rights reserved | 16
Ad-hoc
analytics
Random
look-ups
Aggregations
drill-downs

17. Druid is for real-time, providing aggregations and fast
access
 Streaming ingestion capability
 Data Freshness – analyze events as they occur
 Fast response time (ideally < 1sec query time)
 Arbitrary slicing and dicing
 Multi-tenancy – 1000s of concurrent users
 Scalability and Availability
 Rich real-time visualization with Superset
Superset
Druid is a distributed, real-time, column-oriented datastore
designed to quickly ingest and index large amounts of data and
make it available for real-time query.

18. Who is Using Druid
http://druid.io/druid-powered.html

19. Druid
cubing
Here’s how Druid usually fits into your architecture
Streaming
data source
(Kafka, etc.) Real-
time
ingest
Druid
Jobs, batch
processes,
scheduled
tasks
HDFS Hive
Superset
VisualizationQuery engineStorageData sources
Druid-backed
Hive tables,
predicate
pushdown
HDFS-backed
Hive tables
Tableau,
Qlik,
Excel
Query
Hive/Druid via
ODBC
Batch
ingest

20. © Hortonworks Inc. 2011- 2017. All rights reserved | 20
Ad-hoc
analytics
Random
look-ups
Aggregations
drill-downs

21. © Hortonworks Inc. 2011- 2017. All rights reserved | 21
And one more honorable mention

22. © Hortonworks Inc. 2011- 2017. All rights reserved | 22
Ad-hoc
analytics
Random
look-ups
Aggregations
drill-downs
Complex ETL, ML

23. What is Apache Spark?
Classification Regression
• Support vector
• logistic regression
Collaborative Filtering
Clustering
• K-means
Optimization
• Stochastic Gradient
Descent
ML lib (Machine Learning)
Scalable
• High-throughput, fault-
tolerant stream processing
of live data streams
(micro-batches)
Data Ingest Sources
• Kafka, Flume, Twitter,
ZeroMQ, Kinesis or TCP
sockets
Reuse Spark APIs
• Complex algorithms
expressed with high-level
functions like map, reduce,
join and window
Data Persistence
• Processed data can be
pushed out to file systems,
databases and live
dashboards
Spark Streaming
Structured Data Processing
• Programming abstraction
called DataFrames
• Distributed SQL query
engine
Infer Schema
• Automatically infer scheme
of a JSON dataset and
load it as a DataFrame.
Spark SQL
Resource Management
Storage
Applications
Spark Core Engine
Scala
Java
Python
libraries
MLlib
(Machine
learning)
Spark
SQL*
Spark
Streaming*

24. Benefits of Apache Spark
• Performance
– Deliver high performance large scale data processing and analysis by
leveraging in memory computing
• Ease of Use
– Easy to use APIs for operating on large datasets
– Operators for transforming data
– DataFrames provides support for manipulating structured and semi-
structured data
• Efficiency
– Enhanced developer productivity through prepackaged libraries that can be
combined in the same application
• SQL queries
• Streaming data
• Machine learning
• Graph processing
Resource Management
Storage
Applications
Spark Core Engine
Scala
Java
Python
libraries
MLlib
(Machine
learning)
Spark
SQL*
Spark
Streaming*
* Tech Preview

25. Driver
MetaStore
HiveServer+Tez
LLAP DaemonsExecutors
Spark
Meta
Hive
Meta
Executors LLAP Daemons
Isolate Spark and Hive Catalogs/Tables
Leverage connector for Spark <-> Hive
HWC
HWC

26. © Hortonworks Inc. 2011- 2017. All rights reserved | 26
Which one should I choose?

27. © Hortonworks Inc. 2011- 2017. All rights reserved | 27
Use Case Analysis - Each engine has its niche
HBase Hive Druid Spark
Ultra-low latency
Random access
(key-based
lookup)
ACID, real-time
database, EDW
Low-latency
OLAP, concurrent
queries
Complex ETL
Large-volume
OLTP
Unified SQL
interface, JDBC
Aggregations,
drilldowns
ML model
training
Updates Reporting, batch Time-series SparkSQL
Deletes Joins, large
aggregates, ad-
hoc
Real-time
ingestion
Spark
Streaming

28. © Hortonworks Inc. 2011- 2017. All rights reserved | 28
Which use cases make sense?
⬢ HBase – operational data store, lots of changing data
– Financial transaction data
– Frequent customer updates
– CDC
⬢ Druid – analytics across dataset, sums and other aggregations
– Analyzing number of cars being produced by region
– Number of flights departing from a certain airport
⬢ Hive – large queries across tons of data
– Fact-dimension join across billions of rows, e.g. joining loyalty data to a day’s retail
transactions for insights into spending
⬢ Spark – predictive modeling, complex ETL (ELT) jobs
– Building a predictive maintenance model for infrastructure that a transportation company
owns

29. © Hortonworks Inc. 2011- 2017. All rights reserved | 29
Ad-hoc
analytics
Random
look-ups
Aggregations
drill-downs
Complex ETL, ML

30. © Hortonworks Inc. 2011- 2017. All rights reserved | 30
Performance Analysis

31. © Hortonworks Inc. 2011- 2017. All rights reserved | 31
Performance Analysis - Setup
⬢ Caching disabled
⬢ Query types
– Simple count
– Select with a where
– Join
– Update
– An aggregation (e.g. Sum)
8 cores
16GB RAM
8 nodes
30GB,
200MM rows

32. 1.35
15.00 15.00 15.00
1.52
8.71
4.72
8.66
9.16
9.75
0.34 0.71
1.72
0.00 0.00
0.00
2.00
4.00
6.00
8.00
10.00
12.00
14.00
16.00
Select with filter Count(*) Aggregation with
filter
Select with join and
filter
Update with filter
Comparing Hive, HBase and Druid
HBase/Phoenix Hive Druid

33. © Hortonworks Inc. 2011- 2017. All rights reserved | 33
Data Load Times
Engine Load Time
Hive <1 hr
HBase 4+ hrs
Druid 2 hrs
⬢ Issues with HBase – sequential, serial

34. © Hortonworks Inc. 2011- 2017. All rights reserved | 34
Space Considerations
⬢ You may get better storage from HBase with different compression
Engine Size on Disk with Replication
Hive – ORC w/ Zlib 28.4GB
HBase – Snappy compression 89.5GB
Druid 31.5GB

35. © Hortonworks Inc. 2011- 2017. All rights reserved | 35
Ad-hoc
analytics
Random
look-ups
Aggregations
drill-downs

36. © Hortonworks Inc. 2011- 2017. All rights reserved | 36
Unified SQL

37. Hive as the Single Interface
HiveInterface
HBase/Phoenix
Druid
JDBC
Files

38. Hive Query Delegation by Calcite
filter time
group by
order by
Calcite rewrites to
Druid query fragment
Complex joins,
etc would be
computed here

39. BI on Hadoop : Different tools for different use cases
 File / RAW storage
 Unknown questions
 Latency is not a issue
 Non structured / Data Mining /
Data Science
 Structured Data
 Data cleansed / Enriched
 Questions are known but not answers
 Concepts and data regularly updated
 Streaming / low latency
 Pre-aggregation to answer specific
questions
 Known Questions and answers
 Operational dashboards
LLAP
Druid
Cold Warm
Hot

40. © Hortonworks Inc. 2011- 2017. All rights reserved | 40
Ad-hoc
analytics
Random
look-ups
Aggregations
drill-downs

41. © Hortonworks Inc. 2011- 2017. All rights reserved | 41
What conclusions can we draw?
⬢ The use case dictates the tool. This is seen in the numbers
– Druid is extremely fast for aggregations
– HBase is great with lookups and OLTP-style updates on fast-moving data
– Hive is used a lot for analytics on large quantities of data, where the query isn’t known
beforehand
– Spark has great libraries for ML and is customizable for complex ETL
⬢ Use case sprawl – watch for this (no one engine does it all)
⬢ Unified SQL – the tools complement each other in the larger enterprise
architecture

42. © Hortonworks Inc. 2011- 2017. All rights reserved | 42
Further Reading
⬢ Use case discussion on the engines
– https://hortonworks.com/blog/big-data-processing-engines-which-one-do-i-use-part-1/
⬢ Performance analysis
– https://community.hortonworks.com/articles/232317/big-data-processing-engines-the-
technical-series-p.html
– https://community.hortonworks.com/articles/233083/big-data-processing-engines-the-
technical-series-p-1.html