https://www.bigdataspain.org/2016/program/thu-enterprise-connected-data-trends-apache-hadoop-ecosystem.html https://www.youtube.com/watch?v=WArxOEmtv1s&index=5&list=PL6O3g23-p8Tr5eqnIIPdBD_8eE5JBDBik&t=52s

1. © Hortonworks Inc. 2011 – 2016. All Rights Reserved

2. The Enterprise and Connected Data,
Trends in the Apache Hadoop
Ecosystem
Alan Gates
Co-Founder
Hortonworks
@alanfgates

3. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Our Hadoop Journey Begins…
1 ° ° °
° ° ° N
HDFS
MapReduce
Batch apps
2006

4. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Today
Our Hadoop Journey: Ecosystem Innovation Accelerates
2006 2011

5. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
6 Years of Apache Hive and Beyond
• Apache Hive becomes a Top-Level Project
• HiveServer2 adds ODBC/JDBC
• SQL breadth expands with windowing
and more
• Apache Tez enters incubation
• Hive 0.13 marks delivery of the Stinger
Initiative with Tez, Vectorized Query
and ORCFile support
• Standard SQL authorization,
integration with Apache Ranger
• ACID transactions introduced
• Governance added with Apache
Atlas integration
• Hive 2 introduces LLAP and
intelligent in-memory caching
2010 2011 2012 2013 2014 2015 2016
A SQL data warehouse infrastructure that
delivers fast, scalable SQL processing on
Hadoop and in the Cloud
• Extensive SQL:2011 Support
• Compatible with every major BI Tool
• Proven at 300+ PB Scale

6. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hive 2 with LLAP: Architecture Overview
Deep
Storage
HDFS
S3 + Other HDFS
Compatible Filesystems
YARN Cluster
LLAP Daemon
Query
Executors
In-Memory
Cache
LLAP Daemon
Query
Executors
In-Memory
Cache
LLAP Daemon
Query
Executors
In-Memory
Cache
LLAP Daemon
Query
Executors
In-Memory
Cache
Query
Coordinators
Coord-
inator
Coord-
inator
Coord-
inator
HiveServer2
(Query
Endpoint)
ODBC /
JDBC SQL
Queries

7. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hive 2 with LLAP: 25+x Performance Boost
0
5
10
15
20
25
30
35
40
45
50
0
50
100
150
200
250
Speedup(xFactor)
QueryTime(s)(LowerisBetter)
Hive 2 with LLAP averages 26x faster than Hive 1
Hive 1 / Tez Time (s) Hive 2 / LLAP Time(s) Speedup (x Factor)

8. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
What’s new in Spark 2.0?
 API Improvements
– SparkSession – new entry point
– Unified DataFrame & DataSet API
– Structured Streaming/Continuous Application
 Performance Improvements
– Tungsten Phase 2 – Whole-stage code generation
 ML
– ML model persistence
– Distributed R algorithms (GLM, Naïve Bayes, K-Means, Survival Regression)
 SparkSQL
– SQL 2003 support (new ANSI SQL parser, subquery support)

9. © Hortonworks Inc. 2011 – 2016. All Rights Reserved

10. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
How to Secure and Govern Access to Your Data?
Classification
Prohibition
Time
Location
Streams
Pipelines
Feeds
Hive
Tables
HDFS
Files
HBase
Tables
Entities
in Data
Lake
Policies
?

11. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Secure and Govern Your Data with Tag-Based Access Policies
Classification
Prohibition
Time
Location
Policies
PDP
Resource
Cache
Ranger
Manage Access Policies
and Audit Logs
Track Metadata
and Lineage
Atlas Client
Subscribers
to Topic
Gets Metadata
Updates
Atlas
Metastore
Tags
Assets
Entitles
Streams
Pipelines
Feeds
Hive
Tables
HDFS
Files
HBase
Tables
Entities
in Data
Lake

12. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Data In Motion
 Constrained
 High-latency
 Localized context
 Hybrid – cloud/on-premises
 Low-latency
 Global context
SOURCES
REGIONAL
INFRASTRUCTURE
CORE
INFRASTRUCTURE

13. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Our Hadoop Journey: From the Data Center to the Cloud!
2006 Today

14. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Why Hadoop in the Cloud?
Unlimited
Elastic Scale
Ephemeral &
Long-Running
IT &
Business Agility
No Upfront
HW Costs

15. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Key Architectural Considerations for Hadoop in the Cloud
Shared Data
& Storage
On-Demand
Ephemeral Workloads
10101
10101010101
01010101010101
0101010101010101010
Elastic Resource
Management
Shared Metadata,
Security & Governance

16. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Shared Data and Storage
Understand and Leverage Unique Cloud Properties
 Shared data lake is cloud storage accessible
by all apps
 Cloud storage segregated from compute
 Built-in geo-distribution and DR
Focus Areas
 Address cloud storage consistency
and performance
 Enhance performance via memory
and local storage
Shared Data
& Storage
10101
10101010101
01010101010101
0101010101010101010

17. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enhance Performance via Caching
Tabular Data: LLAP Read + Write-thru Cache
 Shared across jobs / apps and across engines
 Cache only the needed columns
 Spills to SSD when memory is full (anti-caching)
 Read & Write-through cache
 Security: Column-level and row-level
HDFS Caching for Non-tabular Data
 Cache data from cloud storage as needed
 Write-through cache
Workloads
Cloud Storage
LLAP R/W TablesHDFS Files
Cache

18. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Prescriptive On-Demand Ephemeral Workloads
On-Demand
Ephemeral
Workloads
Data Science
R/W TablesCompute Fabric
ETL
R/W TablesCompute Fabric
Warehouse
R/W TablesCompute Fabric
Search
R/W TablesCompute Fabric

19. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Shared Data Requires Shared Metadata, Security, and Governance
Shared Metadata Across All Workloads
 Metadata considerations
– Tabular data metastore
– Lineage and provenance metadata
– Pipeline and job management metadata
– Add upon ingest
– Update as processing modifies data
 Access / tag-based policies and audit logs
 Centrally stored to facilitate use across clusters
– Ex. backed by Cloud RDS (or shared DB)
Classification
Prohibition
Time
Location
Streams
Pipelines
Feeds
Tables
Files Objects
Shared
Metadata
Policies

20. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Elastic Resource Management in Context of Workload
Workload Management vs. Cluster Management
 Understand resource needs of different
workload types
 Add / remove resources to meet workload SLAs
 Manage compute power and high-performance
data-access (ex., LLAP)
 Pricing-aware: instances (spot, reserved),
data, bandwidth
Elastic
Resource
Management

21. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Data in
Motion
Data at
Rest
Deep Historical
Analysis
DATA C E N TE R
Stream Analytics
Edge
Data
Data in
Motion
Machine
Learning
C LOU D Edge
Data
Edge
Analytics
Data at
Rest
Transformational Applications Require Connected Data

22. Thank You