Hadoop - Looking to the Future By Arun Murthy (Founder of Hortonworks, Creator of YARN) The Apache Hadoop ecosystem began as just HDFS & MapReduce nearly 10 years ago in 2006. Very much like the Ship of Theseus (http://en.wikipedia.org/wiki/Ship_of_Theseus), Hadoop has undergone incredible amount of transformation from multi-purpose YARN to interactive SQL with Hive/Tez to machine learning with Spark. Much more lies ahead: whether you want sub-second SQL with Hive or use SSDs/Memory effectively in HDFS or manage Metadata-driven security policies in Ranger, the Hadoop ecosystem in the Apache Software Foundation continues to evolve to meet new challenges and use-cases. Arun C Murthy has been involved with Apache Hadoop since the beginning of the project - nearly 10 years now. In the beginning he led MapReduce, went on to create YARN and then drove Tez & the Stinger effort to get to interactive & sub-second Hive. Recently he has been very involved in the Metadata and Governance efforts. In between he founded Hortonworks, the first public Hadoop distribution company.

1. Hadoop – Looking to the Future
Arun C. Murthy
Hortonworks Co-Founder
@acmurthy

3. 1 ° ° ° ° °
° ° ° ° ° N
HDFS
(Hadoop Distributed File System)
MapReduce
Largely Batch Processing
2006
© Hortonworks Inc. 2011 – 2014. All Rights Reserved
Hadoop w/ MapReduce
Traditional Hadoop allowed early adopters to
deal with data at scale however…
• Single purpose clusters, specific data sets
• Primarily a batch system using MapReduce
• Difficult to natively integrate existing applications
• Limited enterprise capabilities:
Operations, Security & Governance
In the beginning…

4. 20092006
1 ° ° ° ° °
° ° ° ° ° N
HDFS
(Hadoop Distributed File System)
MapReduce
Largely Batch Processing
Hadoop w/ MapReduce
MAPREDUCE-279
Common data,
multiple applications
• Support multi-tenant cluster
• Batch, interactive & real-time
use cases can leverage the
most appropriate engine
Architectural Center
• Consistent security,
governance & operations
• Ecosystem applications
run natively in Hadoop
Apache Hadoop 2.0 & YARN
October 23, 2013
YARN : Data Operating System
1 ° ° ° ° ° ° ° ° °
° ° ° ° ° ° ° ° °
°
°N
HDFS
(Hadoop Distributed File System)
Batch Interactive Real-Time

5. Batch
MapReduce
Apache Tez: Flexible & More Efficient Execution Engine
YARN: Data Operating System
Batch & Interactive
Apache Tez
SQL
Apache
Hive
Data Flow
Apache
Pig
1
°
°
°
° ° ° ° ° ° °
° ° ° ° ° ° N
Java Apps
Cascading
Others Batch
MapReduce
1
°
°
°
° °
° °
HDFS
(Hadoop Distributed File System)
SQL
Apache
Hive
Data Flow
Apache Pig
° ° ° ° ° °
° ° ° ° ° N
Others
1
°
HDFS
(Hadoop Distributed File System)
Hadoop 1
Hadoop 2
Batch System w/
MapReduce as base
Apache Tez supports both
interactive & batch processing

6. YARN : Data Operating System
1 ° ° ° ° ° ° ° ° °
° ° ° ° ° ° ° ° °
°
°N
HDFS
(Hadoop Distributed File System)
Legacy
MapReduce
Interactive SQL
Apache Tez
Other Engines
& Workloads
Apache Hive
SQL
Business Analytics Custom Apps
Apache Hive and the Power of YARN
Stinger Initiative
Next generation SQL based
interactive query in Hadoop
Speed
Performance increased 100x for
interactive & batch use cases
Scale Queries from GBs,
to TBs to PBs
SQL Broadest range of SQL
semantics
Apache Hive Community
1,672Jira Tickets Closed
145Developers
44Companies
~390,000
Lines Of Code Added… (2x)
13Months
Hive
13
Hive
12
Hive
10
Dramatically
faster queries
speeds time
to insight
secondsthousands
of seconds

7. YARN : Data Operating System
1 ° ° ° ° ° ° ° ° °
° ° ° ° ° ° ° ° °
°
°N
HDFS
(Hadoop Distributed File System)
Legacy
MapReduce
Interactive SQL
Apache Tez
Other Engines
& Workloads
Apache Hive
SQL
Business Analytics Custom Apps
Apache Hive – Interactive SQL in Hadoop
Stinger
Next generation SQL based
interactive query in Hadoop
ORC
IO Improvements
Efficient processing via complex
pushdown
Tez Powerful primitives for
the SQL Planner
VQP Efficient CPU utilization in
Inner Loop

8. Sub-Second SQL with Hive LLAP
Stinger.Next
Sub-second SQL in Hadoop via
Hive/LLAP
CBO
The “right” plan executed
violently…
LLAP
Metastore Extensive stats &
scalability
YARN : Data Operating System
1 ° ° ° ° ° ° ° ° °
° ° ° ° ° ° ° ° °
°
°N
HDFS
(Hadoop Distributed File System)
LLAP Apache Tez
Other Engines
& Workloads
Apache Hive
Sub-second SQL
Business Analytics Custom Apps
Long-lived daemon for low-
latency startup, caching & CPU
efficiency via JIT

9. Apache Slider For “Always-on” Services
“Slide” apps on YARN
Democratize access to
storage (HDFS) and compute
(YARN)
Ease management (Ambari)
in addition to deployment
YARN: Data Operating System
Real-Time
Slider
NoSQL
Apache
HBase
NoSQL
Apache
Accumulo
1
°
°
°
° ° ° ° ° ° °
° ° ° ° ° ° N
HDFS
(Hadoop Distributed File System)
Stream
Apache
Storm
Others
ISV

10. © Hortonworks Inc. 2015. All Rights Reserved
Data Governance Initiative
Requirements
1. Hadoop must snap in to the
existing frameworks and
openly exchange metadata
2. Hadoop must address
governance within its own
stack of technologies
Engineers from a group of companies dedicated
to meeting these requirements in the open
New Apache
project proposal
Knowledge Store
Audit Store (Ranger)
ModelsType-System
Policy RulesTaxonomies
Tag Based
Policies
Data Lifecycle
Management
(Falcon)
Real-time Tag-based Access Control (Ranger)
REST API
Services
Search Lineage Exchange
Healthcare
HIPAA
HL7
Financial
SOX
Dodd-Frank
Energy
PPDM
Retail
PCI
PII
Other
CWM

11. Hadoop - Redefined
YARN
(Cluster Resource Management)
1 ° ° ° ° ° ° °
° ° ° ° ° ° ° °
° ° ° ° ° ° ° °
Script
Pig
SQL
Hive
TezTez
Others
Engines
Tez
Java
Cascading
Tez
° °
° °
° °
HBase
NoSQL
Storm
Stream
Slider Slider
Accumulo
NoSQL
Others
Engines
Slider Slider
° ° ° ° °
° ° ° ° °
° ° ° ° °
°
°
°
Spark
In-Memory
°
°
°
°
°
°
PaaS
KubernetesLASR
HPA
°
°
N
°
°
°
°
°
°
HDFS
(Storage Management)
Batch
MR
DGI
(Data Governance & Metadata Management)

12. © Hortonworks Inc. 2015. All Rights Reserved
HDFS - Futures
Sanjay Radia

13. HDFS – Tiered Storage

14. HDFS Ozone – Object Store

15. Thank You
@acmurthy