This document discusses how coordinating the many tools of big data has become more complex with the rise of cloud computing and large datasets. It argues that while having many tools provides flexibility, it also leads to inefficiencies as tools do not integrate well and developers end up duplicating work. The document proposes that Hadoop can help address these issues by providing shared services that tools can leverage, such as common table management, metadata access, and a new execution engine called Tez that allows for more efficient pipelining of jobs compared to the traditional MapReduce approach. Coordinating tools through shared services allows users to focus on selecting the right tool for a task while reducing redundant development work.

1. Coordinating the Many
Tools of Big Data
Strata 2013
Alan F. Gates
@alanfgates
Page 1

2. Big Data = Terabytes, Petabytes, …
Image Credit: Gizmodo
© Hortonworks 2013
Page 2

3. But It Is Also Complex Algorithms
• An example from a talk by Jimmy Lin at Hadoop Summit
2012 on calculations Twitter is doing via UDFs in Pig.
This equation uses stochastic gradient descent to do
machine learning with their data:
w(t+1) =w(t) −γ(t)∇(f(x;w(t)),y)
© Hortonworks 2013
Page 3

4. And New Tools
• Apache Hadoop brings with it a large selection of tools
and paradigms
– Apache HBase, Apache Cassandra – Distributed, high volume
reads and rights of individual data records
– Apache Hive - SQL
– Apache Pig, Cascading – Data flow programming for ETL, data
modeling, and exploration
– Apache Giraph – Graph processing
– MapReduce – Batch processing
– Storm, S4 – Stream processing
– Plus lots of commercial offerings
© Hortonworks 2013
Page 4

5. Pre-Cloud: One Tool per Machine
• Databases presented SQL or SQL-like paradigms for operating on data
• Other tools came in separate packages (e.g. R) or on separate platforms (e.g.
SAS).
Data
Mart
Statistical
Analysis
Data
Warehouse
Cube/M OLTP
OLAP
© Hortonworks 2013
Page 5

6. Cloud: Many Tools One Platform
• Users no longer want to be concerned with what platform their data is in – just
apply the tool to it
• SQL no longer the only or primary data access tool
Statistical
Data Analysis
Mart
Data
Warehouse
Cube/M OLT
OLAP P
© Hortonworks 2013
Page 6

7. Upside - Pick the Right Tool for the Job
© Hortonworks 2013
Page 7

8. Downside – Tools Don’t Play Well Together
• Hard for users to share data between tools
– Different storage formats
– Different data models
– Different user defined function interfaces
© Hortonworks 2013
Page 8

9. Downside – Wasted Developer Time
• Wastes developer time since each tool supplies the
redundant functionality
Hive
Pig Parser
Parser Metadata
Optimizer Optimizer
Physical Physical
Planner Planner
Executor Executor
© Hortonworks 2013
Page 9

10. Downside – Wasted Developer Time
• Wastes developer time since each tool supplies the
redundant functionality
Hive
Pig Parser
Parser Metadata
Optimizer Optimizer
Physical Physical
Overlap
Planner Planner
Executor Executor
© Hortonworks 2013
Page 10

11. Conclusion: We Need Services
• We need to find a way to share services where we can
• Gives users the same experience across tools
• Allows developers to share effort when it makes sense
© Hortonworks 2013
Page 11

12. Hadoop = Distributed Data Operating
System
Service Hadoop Component
Table Management Hive
Access To Metadata HCatalog
User authentication Knox
Resource management YARN
Notification HCatalog
REST/Connectors webhcat, webhdfs, Hive, HBase,
Oozie
Relational data processing Tez
Exists Pieces exist in this component New Project
© Hortonworks 2013
Page 12

13. Hadoop = Distributed Data Operating
System
Service Hadoop Component
Table Management Hive
Access To Metadata HCatalog
User authentication Knox
Resource management YARN
Notification HCatalog
REST/Connectors webhcat, webhdfs, Hive, HBase,
Oozie
Relational data processing Tez
Exists Pieces exist in this component New Project
© Hortonworks 2013
Page 13

14. HCatalog – Table Management
• Opens up Hive’s tables to other tools inside and outside
Hadoop
• Presents tools with a table paradigm that abstracts away
storage details
• Provides a shared data model
• Provides a shared code path for data and metadata access
© Hortonworks 2013
Page 14

15. HCatalog – Table Management
• Opens up Hive’s tables to other tools inside and outside
Hadoop
• Presents tools with a table paradigm that abstracts away
storage details
• Provides a shared data model
• Provides a shared code path for data and metadata access
Hive
Metastore
© Hortonworks 2013
Page 15

16. HCatalog – Table Management
• Opens up Hive’s tables to other tools inside and outside
Hadoop
• Presents tools with a table paradigm that abstracts away
storage details
• Provides a shared data model
• Provides a shared code path for data and metadata access
Hive Pig
HCat
Loader
Metastore MapReduce
HCatInput
Format
© Hortonworks 2013
Page 16

17. HCatalog – Table Management
• Opens up Hive’s tables to other tools inside and outside
Hadoop
• Presents tools with a table paradigm that abstracts away
storage details
• Provides a shared data model
• Provides a shared code path for data and metadata access
Hive Pig
External
Systems HCat
Loader
REST
WebHCat
Metastore MapReduce
HCatInput
Format
© Hortonworks 2013
Page 17

18. Tez – Moving Beyond MapReduce
• Low level data-processing execution engine
• Use it for the base of MapReduce, Hive, Pig, Cascading
etc.
• Enables pipelining of jobs
• Removes task and job launch times
• Hive and Pig jobs no longer need to move to the end of
the queue between steps in the pipeline
• Does not write intermediate output to HDFS
– Much lighter disk and network usage
• Built on YARN
© Hortonworks 2013
Page 18

19. Pig/Hive-MR versus Pig/Hive-Tez
SELECT a.state, COUNT(*), AVERAGE(c.price)
FROM a
JOIN b ON (a.id = b.id)
JOIN c ON (a.itemId = c.itemId)
GROUP BY a.state
Job 1
Job 2
I/O Synchronization
Barrier
I/O Synchronization
Barrier
Job 3
Pig/Hive - MR
© Hortonworks 2013
Page 19

20. Pig/Hive-MR versus Pig/Hive-Tez
SELECT a.state, COUNT(*), AVERAGE(c.price)
FROM a
JOIN b ON (a.id = b.id)
JOIN c ON (a.itemId = c.itemId)
GROUP BY a.state
Job 1
Job 2
I/O Synchronization
Barrier
I/O Synchronization
Barrier
Single Job
Job 3
Pig/Hive - MR Pig/Hive - Tez
© Hortonworks 2013
Page 20

21. FastQuery: Beyond Batch with YARN
Tez Generalizes Map-Reduce Always-On Tez Service
Simplified execution plans process Low latency processing for
data more efficiently all Hadoop data processing
© Hortonworks 2013
Page 21

22. Knox – Single Sign On
© Hortonworks 2013
Page 22

23. Today’s Access Options
• Direct Access
– Access Services via REST (WebHDFS, WebHCat)
– Need knowledge of and access to whole cluster
– Security handled by each component in the cluster
– Kerberos details exposed to users
User {REST} Hadoop Cluster
• Gateway / Portal Nodes
– Dedicated nodes behind firewall
– User SSH to node to access Hadoop services
SSH
GW
User Hadoop Cluster
Node
© Hortonworks 2013
Page 23

24. Knox Design Goals
• Operators can firewall cluster without end user access to
“gateway node”
• Users see one cluster end-point that aggregates
capabilities for data access, metadata and job control
• Provide perimeter security to make Hadoop security setup
easier
• Enable integration enterprise and cloud identity
management environments
© Hortonworks 2013
Page 24

25. Perimeter Verification & Authentication
Verification
- Verify identity token Authentication Hadoop Cluster
- SAML, propagation of identity
Authentication
User Store
- Establish identity at Gateway to
Authenticate with LDAP + AD KDC, AD, DN DN
LDAP
Web DN DN
HDFS
NN
{REST} Knox
Client Gateway
JT
Web
Hive
ID Provider HCat
KDC, AD,
LDAP HCat
Verification
© Hortonworks 2013
Page 25

26. Thank You
© Hortonworks 2012
Page 26