YARN and NTT's contribution @ Cloudera World Tokyo 2013

NTT meets Hadoop
- Our contribution to Hadoop and YARN -

@Cloudera World Tokyo 2013
2013/11/7
NTT
Tsuyoshi Ozawa

© 2012 NTT Software Innovation Center

About me
• Tsuyoshi Ozawa
• Researcher & Engineer @ NTT
Twitter: @oza_x86_64
• A Hadoop Contributor
• Author of “Hadoop 徹底入門 2nd Edition”
Chapter 22(YARN)


2

Agenda
• NTT and Hadoop
• Why Hadoop?
• Our Hadoop usage
• Our contribution to Apache Hadoop

• Technical hot topic at Hadoop Community
•
•
•
•

Hot topic of each Hadoop components
YARN
What’s YARN?
WIPs:
• ResourceManager HA
• Llama(long lived application master)
• Long running services

• Summary

3

Why Hadoop?
• Deep and wide experience introducing Open Source Software
technologies.
• For the data management, 11 years with PostgreSQL
including mission critical cases
• Leading Hadoop deployment in Japan for large scale and
high volume data processing, which natively fits “Big Data”
and "Enterprise Batch"
day

hour

Enterprise Batch
Processing

RDBMS

Latency

Online Batch
Processing

Hadoop

Query & Search
Processing

min

Online Processing

Low-Latency Serving Systems

sec
GB

Big Data
Processing

DWH, Search Engine, etc

TB

PB
Size


4

Why is RDBMS better for smaller data?
• Schema on Write
• Traditional Distributed DB’s approach

• Pros
• Minimal overhead at query time

• Cons
• Solid schema and workload
• High overhead at load time
E.g. Column store
Schema and workload aware
Data Load

Query
(Join Column1 and Column3
Based on id)

Column2
Column4

Column1
Column3

5

Why is Hadoop better for bigger data?
• Schema on Read
• Hadoop’s approach
• Relational mapping on Read time(processing time)

• Pros
• Flexible schema and workload
• Minimal overhead at load time

• Cons
• High overhead at query time

E.g. HDFS + MapReduce
Col1-1
Col2-1
Col3-1
Col4-1

Scalable loading

Query
(Reading all data and
Runtime filtering)

Col1-2
Col2-2
Col3-2
Col4-2

6

We’re Hadoop user! (Service)
• Mobile Spatial Statistics (NTT Docomo)
• http://www.nttdocomo.co.jp/english/corporate/technology/rd/tec
hnical_journal/bn/vol14_3/index.html

• Historical search for Twitter’s data (NTT DATA)

• Buzz Finder (NTT Communications)
• Twitter analytics
• Hadoop Summit 2011
• http://www.slideshare.net/cloudera/hadoop-world-2011-largescale-log-data-analysis-for-marketing-in-ntt-communications


7

We’re Hadoop user! (System Integrations)
• NTT DATA has over 6 years experience and over 30
production cases on Hadoop
• Help enterprise customer design, integrate, deploy
and run large clusters at the range of 20 ~ 1200+
nodes, up to 4PB
day

financial
media

media
Online Batch
Processing
public

hour

min

public
Enterprise Batch
Processing

Big Data telcom
telcom
Processing
telcom

Query & Search
Processing

Online
Processing

sec
GB

TB

PB

8

Why are we contributing to Apache Hadoop?
• Impedance mismatch
between Hadoop community and our needs
• Examples of our needs
•
•
•
•

More
More
More
More

easy operations(HA features)
metrics
documents
understandable logging

Important for both
Community and us
Important for us,
But its priority is not so high
for community

• Our answer:
*Writing code* is the fastest way to reflect our
needs!
• Bridging the gap
• Accelerating development for new features
• Getting know-how about new features

9

Example:
• More easy Operations
• Enhancing HA features
• Rethinking State Machine of HA components
from operators’ view

• More wider use cases
Optimization of MapReduce
• Node-level combiner for MapReduce
(MAPREDUCE-4502)
• Presentation at Pre Strata/Hadoop Meetup at NEWYORK

• Container reuse
(MAPREDUCE-3902)


10

Example: When startup RM-HA…
• Before: Returns IllegalArgumentException if
configuration is invalid
• Difficult to debug configuration!


11

Example: When startup RM-HA…
• After: Users can know why it fails


12

My current Activities
• Contributions
• ResourceManager HA
• Stabilization of MapReduce
• Wrote a YARN article for Hadoop book in Japanese

• Problem
• Time lag
• Silicon Valley is the core developing time of Hadoop
• PST and JST has 17 hours time lag

• Offline meetups
• Some offline meetups are held every week

• Solution
• Visiting Silicon Valley and develoing Hadoop there

13

NTT DATA members’ contributions
• NTT DATA members are contributing to Hadoop!
•
•
•
•

Akira Ajisaka / @ajis_ka
Kousuke Saruta
Masatake Iwasaki
Shinichi Yamashita

• They are contributing based on Hadoop integration
experiences for enterprise customer!
•
•
•
•

Enhance Metrics, Logging, Monitoring for robust design and rock -solid
operations
Adding/Improving documentations
Various bug fixes (HADOOP-9909, HIVE-5296, etc.)
Integration is our matter:
“Direct Connector for PostgreSQL” (SQOOP -390,999)


14

TECHNICAL PART


15

Hot topics of each Hadoop components
• MapReduce
• Shuffle Plugin(MAPREDUCE-4049)
• JobTracker(AppMaster) HA(MAPREDUCE-2708)
• MapReduce itself is at stable phase
• Optimization is being done in Apache Tez/Spark/Impala project

• HDFS
• Cache management(HDFS-4949)
• Snapshot(HDFS-2802)
• Symbolic links(HADOOP-6421 etc.)

• YARN(New!)
A new component for Hadoop 2.x


16

What’s YARN?
• Yet Another Resource Negotiator
• Proposed by Arun C Murthy in 2011
• Separate JobTracker’s role
• Resource Management/Isolation
• Task Scheduling

• MapReduce v2 is a name for MapReduce over
YARN
MRv1 architecture
MapReduce

YARN and frameworks architecture
MRv2

Impala

Spark

YARN


17

Why YARN?(Use case)
• Running various processing frameworks
on same cluster
• Batch processing with MapReduce
• Interactive query with Impala
• Interactive deep analytics(e.g. Machine Learning)
with Spark

Periodic long batch
query

MRv2

Interactive
Aggregation
query

Interactive
Machine Learning
query

Impala

Spark

YARN
HDFS

18

Why YARN?(Technical reason)
• More effective resource management for multiple
processing frameworks
• difficult to use entire resources without thrashing
• Cannot move *Real* big data from HDFS/S3
Job1

Job2

Each frameworks has own scheduler

Master for Impala

Master for MapReduce

Slave

Slave
map slot

Impala slave

Job1

Slave

Slave

reduce slot

MapReduce slave

HDFS slave
thrashing

19

MRv1 Architecture
• Resource is managed by JobTracker
• Task Scheduling
• Resource Management

Schedulers only now own resource usages
Master for MapReduce
Master for Impala

Slave

Slave

map slot

reduce slot

MapReduce slave

Slave

map slot

reduce slot

MapReduce slave


map slot

reduce slot

MapReduce slave

20

YARN Architecture
• Idea
• One global resource manager(ResourceManager)
• Common resource pool for all frameworks(NodeManager and
Container)
• Schedulers for each frameworks(AppMaster)
1. Submit jobs
ResourceManager
2. Launch Master

3. Launch Slaves

Slave
Master

Client

Slave

Slave
Slave

Slave

Slave

Master

Master

Slave

Slave

Slave

Container Container Container



NodeManager

NodeManager

NodeManager


21

YARN and Mesos
Policy/Philosophy is different
YARN
• AppMaster is launched
for each jobs
• More scalability
• Higher latency
• One container per req
• One Master per Job

ResourceManager

Master1

NM

• AppMaster is launched
for each
app(framework)
• Less scalability
• Lower latency
• Bundle of containers per
req
• One Master per
Framework
Master1
ResourceMaster
Master2

Master2

NM

Mesos

NM

Slave

Slave

Slave

22

Applications?
• From Hadoop World 2013
• YARN is becoming a real open kernel!
ht t ps://tw itter.co m/aj is_ka/ st a t us /39557 287540 060569 6/p hot o/1/ lar ge


23

Hot topics in YARN
• ResourceManager High Availability
(YARN-149, YARN-128)
• llama(Long-lived Application MAster)
• Long lived services in YARN(YARN-896)


24

ResourceManager High Availability
• What’s happen when ResourceManager fails?
• cannot submit new jobs
• NOTE:
• Launched Apps continues to run
• AppMaster recover is done in each frameworks
• MRv2

Submit jobs
ResourceManager

Client

Continue to run each jobs
Slave
Master

Slave

Slave
Slave

Slave

Slave

Master

Master

Slave

Slave

Slave




NodeManager

NodeManager

NodeManager


25

ResourceManager High Availability
• Approach
•
•
•
•

Storing RM information to ZooKeeper
Automatic Failover by ZKFC(ZooKeeper Failover Controller)
Manual Failover by RMHAUtils
NodeManagers uses local RMProxy to access them
2. failure

ResourceManager
Active
3. ZKFC
Detects
failure

4. Failover

ZKFC

ResourceManager
Standby

3. Standby
Node become
active

ZKFC

1. Active Node stores
all state into ZKStore

ZooKeeper
RMState

ZooKeeper
RMState

ZooKeeper
RMState


26

Llama (Long-lived Application MAster)
• YARN’s resource allocation is optimized for long
batch system(like Hadoop!)
• Impala/Spark is low-latency querying system
• Impedance mismatch between YARN and Impala/Spark
• http://cloudera.github.io/llama/

• Idea: Special AppMaster for low latency apps
• AppMaster pooling
• As a resource negotiators’
proxy

Impala
Server

ResourceManager

• Gang Scheduling
• Get multiple containers
at the same time

2. Submit
Job

3. Resource allocation
Llama
(Master)

http://cdn.oreillystatic.com/en/assets/1/event/100/From%20
NM
NM
Promise%20to%20a%20Platform_%20Next%20Steps%20i
n%20Bringing%20Workload%20Diversity%20to%20Hadoo
1. Launch llama(first time
p%20Presentation.pdf

NM
only)

27

Long lived services for YARN
• YARN as Multitenant platform
• Like CloudFoundry, Mesosphere
• Hoya: HBase on YARN
• http://hortonworks.com/blog/hoya-hbase-on-yarn-applicationarchitecture/


28

Summary
• We use Hadoop
• We contribute to Hadoop
• Described Hadoop’s hot topics based on our
experience in Apache Hadoop
• YARN is now real open kernel for BigData!
(not only for Hadoop)
• ResourceMamanger HA
• Llama
• Long running services


29

Links
• "Hadoop - Lessons Learned from Deploying Enterprise
Clusters" (Hadoop World NYC 2010)
http://www.slideshare.net/cloudera/hadoop-world-2010-nycv12recruitclean
• "Hadoop’s Life in Enterprise Systems" (Hadoop World NYC
2011) http://www.slideshare.net/cloudera/hadoop-world-2011hadoops-life-in-enterprise-systems-y-masatani-ntt-data
• NTT Docomo Technical Journal, Mobile Spatial Statistics
http://www.nttdocomo.co.jp/english/corporate/technology/rd/te
chnical_journal/bn/vol14_3/index.html
• Large Scale Log Data Analysis for Marketing in NTT
Communications(Hadoop World 2011)
http://www.slideshare.net/cloudera/hadoop -world-2011-large-scale-log-dataanalysis-for-marketing-in-ntt-communications

• "NTTデータのHadoopソリューション"
http://oss.nttdata.co.jp/hadoop/
• "SI事業の視点から見た Hadoop の適用領域と今後の展望" (Hadoop
Conference Japan 2009)
http://www.slideshare.net/hadoopxnttdata/20091113-hadoopconf-japan2009-v1a-clean

30

Links(YARN)
• The Next Generation of Apache Hadoop MapReduce
http://developer.yahoo.com/blogs/hadoop/next-generationapache-hadoop-mapreduce-3061.html
• Llama: Low Latency Application Master
http://cloudera.github.io/llama
• Resource Management with YARN and Impala
http://goo.gl/Rwq2aW
• Hoya: HBase on YARN
http://hortonworks.com/blog/hoya-hbase-on-yarnapplication-architecture/
• Mesos: A Platform for Fine-Grained Resource Sharing in the
Data Center, NSDI 2011
http://www.cs.berkeley.edu/~matei/papers/2011/nsdi_mesos.
pdf
• Apache Hadoop YARN: Yet Another Resource Negotiator,
SOCC 2013
http://goo.gl/Gnl9ZU
• Apache Tez
http://hortonworks.com/hadoop/tez/
http://incubator.apache.org/projects/tez.html

31

YARN and NTT's contribution @ Cloudera World Tokyo 2013

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (15)

More from Tsuyoshi OZAWA

More from Tsuyoshi OZAWA (13)

Recently uploaded

Recently uploaded (20)

YARN and NTT's contribution @ Cloudera World Tokyo 2013