This document discusses using Dell's Elastic Cloud Storage (ECS) to enable active/active Hadoop clusters across multiple sites with a shared global namespace and storage. Key benefits of the ECS solution include strong consistency with asynchronous replication, high storage efficiency compared to HDFS, and the ability to run Hive queries and transactions concurrently across sites. The architecture involves independent Hadoop installations at each site connected via a shared Hive metastore database and ECS file system. A demonstration shows Hive queries and DML statements functioning seamlessly across two sites. Storage efficiency is significantly better than HDFS, especially as the number of sites increases.

1. LET THE
TRANSFORMATION
BEGIN

2. The Active/Active
Data Lake with
Elastic Cloud Storage

3. © Copyright 2016 Dell Inc.
3
Multiple Hadoop Clusters - Challenges
MR/Hive/Pig
YARN
HDFS
MR/Hive/Pig
YARN
HDFS
MR/Hive/Pig
YARN
HDFS
ACTIVE/ACTIVE GLOBAL HADOOP STORAGE

4. © Copyright 2016 Dell Inc.
4
But we use DistCp…
• Active/Passive access
• Very high disk usage (3x replication on each site)
• Periodic transfers (once every few hours)
• Must carefully update Hive Metastore manually at each site
• Hard to build for 3 or more clusters
• Consumes YARN CPU/memory
• No Hive concurrency controls on target cluster

5. © Copyright 2016 Dell Inc.
5
Active/Active Hadoop with ECS
Global Namespace For Active/Active Hadoop
STORAGE EFFICIENCY
Denver Beijing Paris
GEO CACHING
STRONG CONSISTENCY ACTIVE/ACTIVE w/ FAILOVER

6. © Copyright 2016 Dell Inc.
6
Active/Active Hive with ECS: Solution Overview
• 2 to 8 sites
• A shared, common:
– Hadoop-Compatible File System – global namespace, readable and writable from all sites
– Hive Metastore DB
• Strong consistency
• Asynchronous replication (low latency updates)
• Fully recoverable from the failure of a single site and 4 drives in each site
• Very high storage efficiency (4.5 times better than HDFS with 3 sites)
• Hive concurrency and ACID transactions (insert, update, delete), even across sites

7. © Copyright 2016 Dell Inc.
7
Active/Active Hive with ECS: Architecture
ECS Node
ECS Node
ECS Node
ECS Node Node Manager
Metastore DB
Hadoop Master
Node Manager
Node Manager
Site 1
ECS Node
ECS Node
ECS Node
ECS Node Node Manager
Metastore DB
Hadoop Master
Node Manager
Node Manager
Site 2
ECS Node
ECS Node
ECS Node
ECS Node Node Manager
Metastore DB
Hadoop Master
Node Manager
Node Manager
Site 3
ECS Replication (async)
Metastore DB Repl (sync)
Hadoop installations at
different sites are
independent except for
the Hive Metastore DB and
the common file system
provided by ECS.

8. © Copyright 2016 Dell Inc.
8
Active/Active Hive Demonstration
hive> create table demotab1…
hive> insert into demotab1 partition (site=1)
values (11, ‘ant'), (12,'bear');
Site 1 Site 2
hive> select * from demotab1;
11 ant 1
12 bear 1
hive> insert into demotab1 partition (site=2)
values (21, 'cat'), (22,'dog');
hive> select * from demotab1;
11 ant 1
12 bear 1
21 cat 2
22 dog 2
hive> delete from demotab1 where site=2 and id=21;
hive> select * from demotab1;
11 ant 1
12 bear 1
22 dog 2
hive> create table demotab1
(
id int,
s string
)
partitioned by (site int)
clustered by (id) into 4 buckets
stored as orc tblproperties ('transactional'='true');

9. © Copyright 2016 Dell Inc.
9
Use Case: Enterprise Data Warehouse Offload
• Tables can be exported from an EDW to Hive on ECS with Apache Sqoop or similar tools
• Data will be efficiently distributed and protected across multiple sites
• If desired, delete exported records from the EDW
• Data can be queried in place using Hive SQL from any site
EDW SQL

10. © Copyright 2016 Dell Inc.
10
Storage Efficiency Comparison
38%
50%
56%
60% 63% 64% 66%
17%
11%
8% 7% 6% 5% 4%
0%
20%
40%
60%
80%
100%
2 3 4 5 6 7 8
Storage
Efficiency
Number of Sites
ECS Efficiency
HDFS Efficiency
Storage efficiency is the effective % of raw disk bytes that are usable by your data
4.5x Better Efficiency!