SlideShare a Scribd company logo
1 of 43
Download to read offline
Slide 1 www.edureka.in/hadoop
Slide 2
Hello There!!
My name is Annie.
Let me test your Hadoop 1.x
knowledge?
Annie’s Introduction
Slide 3
Hello There!!
My name is Annie.
I love quizzes and
puzzles and I am here to
make you guys think and
answer my questions.
Can you store 1 billion files in a Hadoop 1.x cluster?
- Yes
- No
Annie’s Question
Slide 4
No. Even though you have hundreds of DataNodes in the cluster,
the NameNode keeps all its metadata in memory, so you are limited
to a maximum of only 50-100M files in the entire cluster because of
a Single NameNode in Hadoop 1.x.
Annie’s Answer
Slide 5
Hello There!!
My name is Annie.
I love quizzes and
puzzles and I am here to
make you guys think and
answer my questions.
A Hadoop 1.x cluster can have multiple HDFS Namespaces.
- True
- False
Annie’s Question
Slide 6
False. Not possible with Hadoop 1.x.
Annie’s Answer
Slide 7
Hello There!!
My name is Annie.
I love quizzes and
puzzles and I am here to
make you guys think and
answer my questions.
Which of the following is (are) a significant disadvantage in Hadoop
1.0?
- ‘Single Point Of Failure’ of NameNode
- Too much burden on Job Tracker
Annie’s Question
Slide 8
Single Point of Failure of NameNode and too much burden
on Job Tracker.
Annie’s Answer
Slide 9
Hello There!!
My name is Annie.
I love quizzes and
puzzles and I am here to
make you guys think and
answer my questions.
Can you use hundreds of Hadoop DataNode for any other processing
than MapReduce in Hadoop 1.x?
- Yes
- No
Annie’s Question
Slide 10
No. Hadoop 1.x dedicates all the DataNode resources to Map and
Reduce slots with no or little room for processing any other
workload.
Annie’s Answer
Slide 11
Hello There!!
My name is Annie.
I love quizzes and
puzzles and I am here to
make you guys think and
answer my questions.
Can you use Hadoop for Real-time processing?
- Yes
- No
Annie’s Question
Slide 12
No. Hadoop is designed and developer for massively parallel batch
processing.
Annie’s Answer
Limitations of Hadoop 1.x
 No horizontal scalability of NameNode
 Does not support NameNode High Availability
 Overburdened JobTracker
 Not possible to run Non-MapReduce Big Data Applications on HDFS
 Does not support Multi-tenancy
Slide 14 www.edureka.in/hadoop
Hadoop 1.x – In Summary
Client
HDFS Map Reduce
Secondary
NameNode
Data
BlocksDataNode
NameNode Job Tracker
Task Tracker
Map Reduce
DataNode Task Tracker
Map Reduce….
DataNode DataNodeTask Tracker
Map Reduce
Task Tracker
Map Reduce
Slide 15 www.edureka.in/hadoop
Problem Description
NameNode – No Horizontal
Scalability
Single NameNode and Single Namespace, limited by
NameNode RAM
NameNode – No High Availability (HA) NameNode is Single Point of Failure, Need manual recovery using
Secondary NameNode in case of failure
Job Tracker – Overburdened Spends significant portion of time and effort managing the life cycle of
Applications
MRv1 – Only Map and Reduce tasks Humongous Data stored in HDFS remains unutilized and cannot be used
for other workloads such as Graph processing etc.
Hadoop 1.x - Challenges
NameNode - No High Availability
NameNode - No Horizontal Scale
Data
Node
Data
Node
Data
Node
….
Client Get Block Locations
Block Management
Read Data
NameNode
NS
Slide 16 www.edureka.in/hadoop
NameNode – Scale and HA
Slide 17 www.edureka.in/hadoop
Name Node –Single Point of Failure
 Secondary NameNode:
 “Not a hot standby” for the NameNode
 Connects to NameNode every hour*
 Housekeeping, backup of NameNode metadata
 Saved metadata can build a failed NameNode
You give me
metadata every
hour, I will make
it secure
Single Point
Failure
Secondary
NameNode
NameNode
metadata
metadata
Slide 18 www.edureka.in/hadoop
Job Tracker – Overburdened
CPU
 Spends a very significant portion of time and effort managing
the life cycle of applications
Network
 Single Listener Thread to communicate with thousands of
Map and Reduce Jobs
Task Tracker Task Tracker Task Tracker….
Job
Tracker
Slide 19 www.edureka.in/hadoop
MRv1 – Unpredictability in Large Clusters
As the cluster size grow and reaches to 4000 Nodes
 Cascading Failures
 The DataNode failures results in a serious
deterioration of the overall cluster
performance because of attempts to replicate
data and overload live nodes, through network
flooding.
 Multi-tenancy
 As clusters increase in size, you may want to
employ these clusters for a variety of models.
MRv1 dedicates its nodes to Hadoop and
cannot be re-purposed for other applications
and workloads in an Organization. With the
growing popularity and adoption of cloud
computing among enterprises, this becomes
more important.
Unutilized Data in HDFS
 Terabytes and Petabytes of data in HDFS can only be used for MapReduce processing
Slide 11 www.edureka.in/hadoop
Introducing Hadoop 2.0
Features Hadoop 1.x Hadoop 2.0
HDFS Federation One NameNode and a Namespace Multiple NameNode and
Namespaces
NameNode High Availability Not present Highly Available
YARN - Processing Control and
Multi-tenancy
JobTracker, TaskTracker Resource Manager, Node
Manager, App Master, Capacity
Scheduler
Other important Hadoop 2.0 features
 HDFS Snapshots
 NFSv3 access to data in HDFS
 Support for running Hadoop on MS Windows
 Binary Compatibility for MapReduce applications built on Hadoop 1.0
 Substantial amount of Integration testing with rest of the projects (such as PIG, HIVE) in Hadoop ecosystem
Slide 12 www.edureka.in/hadoop
Namenode
Block Management
NS
Storage
Datanode Datanode…
NamespaceBlockStorage
Namespace
NS1 NSk NSn
NN-1 NN-k NN-n
Common Storage
Datanode 1
…
Datanode 2
…
Datanode m
…
BlockStorage
Pool 1 Pool k Pool n
Block Pools
… …
Hadoop 1.0 Hadoop 2.0
Slide 22 www.edureka.in/hadoop
http://hadoop.apache.org/docs/stable2/hadoop-project-dist/hadoop-hdfs/Federation.html
Hadoop 2.0 Cluster Architecture - Federation
Slide 23 www.edureka.in/hadoop
cluster.
Annie’s Question
How does HDFS Federation help HDFS Scale horizontally?
A) Reduces the load on any single NameNode by using the multiple,
independent NameNodes to manage individual parts of the file system
namespace.
B) Provides cross-data centre (non-local) support for HDFS, allowing a cluster
administrator to split the Block Storage outside the local cluster.
Slide 24 www.edureka.in/hadoop
Annie’s Answer
(A). In order to scale the name service horizontally, HDFS federation uses
multiple independent NameNodes. The NameNodes are federated, that is, the
NameNodes are independent and do not require coordination with each other.
Slide 25
Annie’s Question
You have configured two NameNodes to manage /marketing and /finance
namespaces respectively. What will happen if you try to ‘put’ a file in
/accounting directory?
www.edureka.in/hadoop
Slide 26
Annie’s Answer
The ‘put’ will fail. None of the namespaces will manage the file and you will get
an IOException with a “No such file or directory error”.
www.edureka.in/hadoop
Slide 27
Node Manager
HDFS
YARN
Resource
Manager
Shared
edit logs
All name space edits
logged to shared NFS
storage; single writer
(fencing)
Read edit logs and applies
to its own namespace
Data Node
Standby
NameNode
Active
NameNode
Container
App
Master
Node Manager
Data Node
Container
App
Master
Data Node
Client
Data Node
Container
App
Master
Node Manager
Data Node
Container
App
Master
Node Manager
Hadoop 2.0 Cluster Architecture - HA
NameNode High
Availability
Next Generation
MapReduce
HDFS HIGH AVAILABILITY
http://hadoop.apache.org/docs/stable2/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithNFS.html
Slide 28
Hadoop 2.0 Cluster Architecture - HA
www.edureka.in/hadoop
High Availability in
Hadoop 2.0
NameNode recovery in
Hadoop 1.0
Secondary
NameNode
Standby
NameNode
Active
NameNode
Secondary
NameNode
NameNode
Edit logs
Meta-Data
Automatic failover
to Standby
NameNode
Manually Recover
using Secondary
NameNode
FSImage
Slide 29
Annie’s Question
NameNode HA was developed to overcome the following disadvantage in
Hadoop 1.0?
a) Single Point Of Failure of NameNode
b) Too much burden on Job Tracker
www.edureka.in/hadoop
Slide 30
Annie’s Answer
Single Point of Failure of NameNode.
www.edureka.in/hadoop
Apache Oozie (Workflow)
HDFS
(Hadoop Distributed File System)
Pig Latin
Data Analysis
Hive
DW System
MapReduce Framework
HBase
Apache Oozie (Workflow)
HDFS
(Hadoop Distributed File System)
Pig Latin
Data Analysis
Hive
DW System
MapReduce Framework HBase
Other
YARN
Frameworks
(MPI, GIRAPH)
Slide 23 www.edureka.in/hadoop
YARN
Cluster Resource Management
YARN adds a more general interface to run non-MapReduce jobs (such as Graph Processing) within the Hadoop framework
YARN and Hadoop Ecosystem
BATCH
(MapReduce)
INTERACTIVE
(Text)
ONLINE
(HBase)
STREAMING
(Storm, S4, …)
GRAPH
(Giraph)
IN-MEMORY
(Spark)
HPC MPI
(OpenMPI)
OTHER
(Search)
(Weave..)
Slide 32 www.edureka.in/hadoop
http://hadoop.apache.org/docs/stable2/hadoop-yarn/hadoop-yarn-site/YARN.html
YARN – Moving beyond MapReduce
Slide 33 www.edureka.in/hadoop
 Organizes jobs into queues
 Queue shares as %’s of cluster
 FIFO scheduling within each
queue
 Data locality-aware Scheduling
 Hierarchical Queues
To manage the resource within an organization.
 Capacity Guarantees
A fraction to the total available capacity allocated to each Queue.
 Security
To safeguard applications from other users.
 Elasticity
Resources are available in a predictable and elastic manner to queues.
 Multi-tenancy
Set of limit to prevent over-utilization of resources by a single
application.
 Operability
Runtime configuration of Queues.
 Resource-based scheduling
If needed, Applications can request more resources than the default.
Multi-tenancy - Capacity Scheduler
Slide 34
Annie’s Question
YARN was developed to overcome the following disadvantage in Hadoop 1.0
MapReduce framework?
a) Single Point Of Failure Of NameNode
b) Too much burden on Job Tracker
www.edureka.in/hadoop
Slide 35
Annie’s Answer
Too much burden on Job Tracker.
www.edureka.in/hadoop
Slide 36
NameNode High
Availability
Next Generation
MapReduce
Hadoop 2.0 – In Summary
Client
HDFS YARN
Resource ManagerStandby
NameNode
Active
NameNode
Distributed Data Storage Distributed Data Processing
DataNode
Node Manager
Container
App
Master
…….
Masters
Slaves
Node Manager
DataNode
Container
App
Master
DataNode
Node Manager
Container
App
Master
Shared
edit logs
OR
Journal
Node
Scheduler
Applications
Manager
(AsM)
www.edureka.in/hadoop
Slide 37
Hello There!!
My name is Annie.
I love quizzes and
puzzles and I am here to
make you guys think and
answer my questions.
Can you use Hadoop 2.0 for Real-time processing?
- Yes
- No
Annie’s Question
Slide 38
No. Even though YARN in Hadoop 2.0 supports multiple frameworks
for different workloads other than batch, you need Storm or S4 for
real-time processing.
Annie’s Answer
Slide 39 www.edureka.in/hadoop
What about Real-time Processing?
Hadoop is good for Batch
but
How do I process Big Data in Real-
time?
Slide 40 www.edureka.in/hadoop
Storm is coming….
APACHE STORM
The Real-time Hadoop
• Continuous commutation system
Distributed, Reliable, Fault-tolerant,
Scalable and Robust
• Suitable for Big Data processing
• Guarantees no data loss
Programming Language agnostic
• JSON-based for Ruby, Python etc.
Use case
• Stream processing
• Distributed RPC
• Continuous Computation
Hadoop Vs. Storm
Hadoop Storm
Differences
Fundamentally as Batch
processing system
Real-time processing,
process unterminated
streams (e.g. twitter
feeds) of data, process
data as it arrives
MapReduce Jobs run to
completion
Topologies (Computation
Graph) run forever
Stateful Nodes
Stateless Nodes
Hadoop Storm
Similarities
Scalable Scalable
Guarantees no data loss Guarantees no data loss
Open Source Open Source
Storm Use Cases
 Data Normalization
• Groupon uses Storm to build real-time data integration
systems.
 Analytics
• Storm powers Twitter’s publisher analytics product,
processing every tweet and click that happens on Twitter to
provide analytics for Twitter's publisher partners.
• Flipboard use Storm across a wide range of services ranging
from Content Search to real-time analytics, to generating
custom magazine fields.
 Log processing
• Alibaba uses Storm to process the application log and data
change in databases to supply real-time data stats for data
apps.
• NaviSite uses Storm in its server log monitoring and auditing
system.
Thank You
See You in Next Class

More Related Content

What's hot

Hadoop File system (HDFS)
Hadoop File system (HDFS)Hadoop File system (HDFS)
Hadoop File system (HDFS)Prashant Gupta
 
Hadoop Tutorial For Beginners | Apache Hadoop Tutorial For Beginners | Hadoop...
Hadoop Tutorial For Beginners | Apache Hadoop Tutorial For Beginners | Hadoop...Hadoop Tutorial For Beginners | Apache Hadoop Tutorial For Beginners | Hadoop...
Hadoop Tutorial For Beginners | Apache Hadoop Tutorial For Beginners | Hadoop...Simplilearn
 
Hadoop MapReduce Framework
Hadoop MapReduce FrameworkHadoop MapReduce Framework
Hadoop MapReduce FrameworkEdureka!
 
I.ITERATIVE DEEPENING DEPTH FIRST SEARCH(ID-DFS) II.INFORMED SEARCH IN ARTIFI...
I.ITERATIVE DEEPENING DEPTH FIRST SEARCH(ID-DFS) II.INFORMED SEARCH IN ARTIFI...I.ITERATIVE DEEPENING DEPTH FIRST SEARCH(ID-DFS) II.INFORMED SEARCH IN ARTIFI...
I.ITERATIVE DEEPENING DEPTH FIRST SEARCH(ID-DFS) II.INFORMED SEARCH IN ARTIFI...vikas dhakane
 
Big Data and Hadoop
Big Data and HadoopBig Data and Hadoop
Big Data and HadoopFlavio Vit
 
Hadoop Overview & Architecture
Hadoop Overview & Architecture  Hadoop Overview & Architecture
Hadoop Overview & Architecture EMC
 
Introduction to HiveQL
Introduction to HiveQLIntroduction to HiveQL
Introduction to HiveQLkristinferrier
 
Presentation On RAID(Redundant Array Of Independent Disks) Basics
Presentation On RAID(Redundant Array Of Independent Disks) BasicsPresentation On RAID(Redundant Array Of Independent Disks) Basics
Presentation On RAID(Redundant Array Of Independent Disks) BasicsKuber Chandra
 
Heuristic Search in Artificial Intelligence | Heuristic Function in AI | Admi...
Heuristic Search in Artificial Intelligence | Heuristic Function in AI | Admi...Heuristic Search in Artificial Intelligence | Heuristic Function in AI | Admi...
Heuristic Search in Artificial Intelligence | Heuristic Function in AI | Admi...RahulSharma4566
 

What's hot (20)

Hadoop File system (HDFS)
Hadoop File system (HDFS)Hadoop File system (HDFS)
Hadoop File system (HDFS)
 
RAID LEVELS
RAID LEVELSRAID LEVELS
RAID LEVELS
 
Hadoop Tutorial For Beginners | Apache Hadoop Tutorial For Beginners | Hadoop...
Hadoop Tutorial For Beginners | Apache Hadoop Tutorial For Beginners | Hadoop...Hadoop Tutorial For Beginners | Apache Hadoop Tutorial For Beginners | Hadoop...
Hadoop Tutorial For Beginners | Apache Hadoop Tutorial For Beginners | Hadoop...
 
Data mining primitives
Data mining primitivesData mining primitives
Data mining primitives
 
Hadoop MapReduce Framework
Hadoop MapReduce FrameworkHadoop MapReduce Framework
Hadoop MapReduce Framework
 
Big Data Analytics Lab File
Big Data Analytics Lab FileBig Data Analytics Lab File
Big Data Analytics Lab File
 
I.ITERATIVE DEEPENING DEPTH FIRST SEARCH(ID-DFS) II.INFORMED SEARCH IN ARTIFI...
I.ITERATIVE DEEPENING DEPTH FIRST SEARCH(ID-DFS) II.INFORMED SEARCH IN ARTIFI...I.ITERATIVE DEEPENING DEPTH FIRST SEARCH(ID-DFS) II.INFORMED SEARCH IN ARTIFI...
I.ITERATIVE DEEPENING DEPTH FIRST SEARCH(ID-DFS) II.INFORMED SEARCH IN ARTIFI...
 
Hadoop technology
Hadoop technologyHadoop technology
Hadoop technology
 
Yarn
YarnYarn
Yarn
 
Big data and Hadoop
Big data and HadoopBig data and Hadoop
Big data and Hadoop
 
PPT on Hadoop
PPT on HadoopPPT on Hadoop
PPT on Hadoop
 
Hadoop Architecture
Hadoop ArchitectureHadoop Architecture
Hadoop Architecture
 
Big Data and Hadoop
Big Data and HadoopBig Data and Hadoop
Big Data and Hadoop
 
Hadoop ecosystem
Hadoop ecosystemHadoop ecosystem
Hadoop ecosystem
 
Hadoop Overview & Architecture
Hadoop Overview & Architecture  Hadoop Overview & Architecture
Hadoop Overview & Architecture
 
Introduction to HiveQL
Introduction to HiveQLIntroduction to HiveQL
Introduction to HiveQL
 
Presentation On RAID(Redundant Array Of Independent Disks) Basics
Presentation On RAID(Redundant Array Of Independent Disks) BasicsPresentation On RAID(Redundant Array Of Independent Disks) Basics
Presentation On RAID(Redundant Array Of Independent Disks) Basics
 
Map Reduce
Map ReduceMap Reduce
Map Reduce
 
Ddbms1
Ddbms1Ddbms1
Ddbms1
 
Heuristic Search in Artificial Intelligence | Heuristic Function in AI | Admi...
Heuristic Search in Artificial Intelligence | Heuristic Function in AI | Admi...Heuristic Search in Artificial Intelligence | Heuristic Function in AI | Admi...
Heuristic Search in Artificial Intelligence | Heuristic Function in AI | Admi...
 

Similar to Hadoop 2.0 Architecture | HDFS Federation | NameNode High Availability |

Hadoop Developer
Hadoop DeveloperHadoop Developer
Hadoop DeveloperEdureka!
 
Learn Hadoop
Learn HadoopLearn Hadoop
Learn HadoopEdureka!
 
Big Data & Hadoop Tutorial
Big Data & Hadoop TutorialBig Data & Hadoop Tutorial
Big Data & Hadoop TutorialEdureka!
 
Hadoop a Highly Available and Secure Enterprise Data Warehousing solution
Hadoop a Highly Available and Secure Enterprise Data Warehousing solutionHadoop a Highly Available and Secure Enterprise Data Warehousing solution
Hadoop a Highly Available and Secure Enterprise Data Warehousing solutionEdureka!
 
250hadoopinterviewquestions
250hadoopinterviewquestions250hadoopinterviewquestions
250hadoopinterviewquestionsRamana Swamy
 
Hadoop demo ppt
Hadoop demo pptHadoop demo ppt
Hadoop demo pptPhil Young
 
Hadoop tutorial-pdf.pdf
Hadoop tutorial-pdf.pdfHadoop tutorial-pdf.pdf
Hadoop tutorial-pdf.pdfSheetal Jain
 
Presentation sreenu dwh-services
Presentation sreenu dwh-servicesPresentation sreenu dwh-services
Presentation sreenu dwh-servicesSreenu Musham
 
Top Hadoop Big Data Interview Questions and Answers for Fresher
Top Hadoop Big Data Interview Questions and Answers for FresherTop Hadoop Big Data Interview Questions and Answers for Fresher
Top Hadoop Big Data Interview Questions and Answers for FresherJanBask Training
 
50 must read hadoop interview questions & answers - whizlabs
50 must read hadoop interview questions & answers - whizlabs50 must read hadoop interview questions & answers - whizlabs
50 must read hadoop interview questions & answers - whizlabsWhizlabs
 
Hadoop for Java Professionals
Hadoop for Java ProfessionalsHadoop for Java Professionals
Hadoop for Java ProfessionalsEdureka!
 
Best hadoop-online-training
Best hadoop-online-trainingBest hadoop-online-training
Best hadoop-online-trainingGeohedrick
 
Understanding hadoop
Understanding hadoopUnderstanding hadoop
Understanding hadoopRexRamos9
 
OPERATING SYSTEM .pptx
OPERATING SYSTEM .pptxOPERATING SYSTEM .pptx
OPERATING SYSTEM .pptxAltafKhadim
 
Introduction to Hadoop Administration
Introduction to Hadoop AdministrationIntroduction to Hadoop Administration
Introduction to Hadoop AdministrationEdureka!
 
Hadoop Adminstration with Latest Release (2.0)
Hadoop Adminstration with Latest Release (2.0)Hadoop Adminstration with Latest Release (2.0)
Hadoop Adminstration with Latest Release (2.0)Edureka!
 
Hadoop Administration pdf
Hadoop Administration pdfHadoop Administration pdf
Hadoop Administration pdfEdureka!
 

Similar to Hadoop 2.0 Architecture | HDFS Federation | NameNode High Availability | (20)

Hadoop Developer
Hadoop DeveloperHadoop Developer
Hadoop Developer
 
Learn Hadoop
Learn HadoopLearn Hadoop
Learn Hadoop
 
Big Data & Hadoop Tutorial
Big Data & Hadoop TutorialBig Data & Hadoop Tutorial
Big Data & Hadoop Tutorial
 
Hadoop a Highly Available and Secure Enterprise Data Warehousing solution
Hadoop a Highly Available and Secure Enterprise Data Warehousing solutionHadoop a Highly Available and Secure Enterprise Data Warehousing solution
Hadoop a Highly Available and Secure Enterprise Data Warehousing solution
 
250hadoopinterviewquestions
250hadoopinterviewquestions250hadoopinterviewquestions
250hadoopinterviewquestions
 
Hadoop demo ppt
Hadoop demo pptHadoop demo ppt
Hadoop demo ppt
 
Hadoop Tutorial for Beginners
Hadoop Tutorial for BeginnersHadoop Tutorial for Beginners
Hadoop Tutorial for Beginners
 
Hadoop tutorial-pdf.pdf
Hadoop tutorial-pdf.pdfHadoop tutorial-pdf.pdf
Hadoop tutorial-pdf.pdf
 
Presentation sreenu dwh-services
Presentation sreenu dwh-servicesPresentation sreenu dwh-services
Presentation sreenu dwh-services
 
Top Hadoop Big Data Interview Questions and Answers for Fresher
Top Hadoop Big Data Interview Questions and Answers for FresherTop Hadoop Big Data Interview Questions and Answers for Fresher
Top Hadoop Big Data Interview Questions and Answers for Fresher
 
50 must read hadoop interview questions & answers - whizlabs
50 must read hadoop interview questions & answers - whizlabs50 must read hadoop interview questions & answers - whizlabs
50 must read hadoop interview questions & answers - whizlabs
 
Hadoop for Java Professionals
Hadoop for Java ProfessionalsHadoop for Java Professionals
Hadoop for Java Professionals
 
Best hadoop-online-training
Best hadoop-online-trainingBest hadoop-online-training
Best hadoop-online-training
 
Hadoop seminar
Hadoop seminarHadoop seminar
Hadoop seminar
 
Understanding hadoop
Understanding hadoopUnderstanding hadoop
Understanding hadoop
 
HDFS
HDFSHDFS
HDFS
 
OPERATING SYSTEM .pptx
OPERATING SYSTEM .pptxOPERATING SYSTEM .pptx
OPERATING SYSTEM .pptx
 
Introduction to Hadoop Administration
Introduction to Hadoop AdministrationIntroduction to Hadoop Administration
Introduction to Hadoop Administration
 
Hadoop Adminstration with Latest Release (2.0)
Hadoop Adminstration with Latest Release (2.0)Hadoop Adminstration with Latest Release (2.0)
Hadoop Adminstration with Latest Release (2.0)
 
Hadoop Administration pdf
Hadoop Administration pdfHadoop Administration pdf
Hadoop Administration pdf
 

More from Edureka!

What to learn during the 21 days Lockdown | Edureka
What to learn during the 21 days Lockdown | EdurekaWhat to learn during the 21 days Lockdown | Edureka
What to learn during the 21 days Lockdown | EdurekaEdureka!
 
Top 10 Dying Programming Languages in 2020 | Edureka
Top 10 Dying Programming Languages in 2020 | EdurekaTop 10 Dying Programming Languages in 2020 | Edureka
Top 10 Dying Programming Languages in 2020 | EdurekaEdureka!
 
Top 5 Trending Business Intelligence Tools | Edureka
Top 5 Trending Business Intelligence Tools | EdurekaTop 5 Trending Business Intelligence Tools | Edureka
Top 5 Trending Business Intelligence Tools | EdurekaEdureka!
 
Tableau Tutorial for Data Science | Edureka
Tableau Tutorial for Data Science | EdurekaTableau Tutorial for Data Science | Edureka
Tableau Tutorial for Data Science | EdurekaEdureka!
 
Python Programming Tutorial | Edureka
Python Programming Tutorial | EdurekaPython Programming Tutorial | Edureka
Python Programming Tutorial | EdurekaEdureka!
 
Top 5 PMP Certifications | Edureka
Top 5 PMP Certifications | EdurekaTop 5 PMP Certifications | Edureka
Top 5 PMP Certifications | EdurekaEdureka!
 
Top Maven Interview Questions in 2020 | Edureka
Top Maven Interview Questions in 2020 | EdurekaTop Maven Interview Questions in 2020 | Edureka
Top Maven Interview Questions in 2020 | EdurekaEdureka!
 
Linux Mint Tutorial | Edureka
Linux Mint Tutorial | EdurekaLinux Mint Tutorial | Edureka
Linux Mint Tutorial | EdurekaEdureka!
 
How to Deploy Java Web App in AWS| Edureka
How to Deploy Java Web App in AWS| EdurekaHow to Deploy Java Web App in AWS| Edureka
How to Deploy Java Web App in AWS| EdurekaEdureka!
 
Importance of Digital Marketing | Edureka
Importance of Digital Marketing | EdurekaImportance of Digital Marketing | Edureka
Importance of Digital Marketing | EdurekaEdureka!
 
RPA in 2020 | Edureka
RPA in 2020 | EdurekaRPA in 2020 | Edureka
RPA in 2020 | EdurekaEdureka!
 
Email Notifications in Jenkins | Edureka
Email Notifications in Jenkins | EdurekaEmail Notifications in Jenkins | Edureka
Email Notifications in Jenkins | EdurekaEdureka!
 
EA Algorithm in Machine Learning | Edureka
EA Algorithm in Machine Learning | EdurekaEA Algorithm in Machine Learning | Edureka
EA Algorithm in Machine Learning | EdurekaEdureka!
 
Cognitive AI Tutorial | Edureka
Cognitive AI Tutorial | EdurekaCognitive AI Tutorial | Edureka
Cognitive AI Tutorial | EdurekaEdureka!
 
AWS Cloud Practitioner Tutorial | Edureka
AWS Cloud Practitioner Tutorial | EdurekaAWS Cloud Practitioner Tutorial | Edureka
AWS Cloud Practitioner Tutorial | EdurekaEdureka!
 
Blue Prism Top Interview Questions | Edureka
Blue Prism Top Interview Questions | EdurekaBlue Prism Top Interview Questions | Edureka
Blue Prism Top Interview Questions | EdurekaEdureka!
 
Big Data on AWS Tutorial | Edureka
Big Data on AWS Tutorial | Edureka Big Data on AWS Tutorial | Edureka
Big Data on AWS Tutorial | Edureka Edureka!
 
A star algorithm | A* Algorithm in Artificial Intelligence | Edureka
A star algorithm | A* Algorithm in Artificial Intelligence | EdurekaA star algorithm | A* Algorithm in Artificial Intelligence | Edureka
A star algorithm | A* Algorithm in Artificial Intelligence | EdurekaEdureka!
 
Kubernetes Installation on Ubuntu | Edureka
Kubernetes Installation on Ubuntu | EdurekaKubernetes Installation on Ubuntu | Edureka
Kubernetes Installation on Ubuntu | EdurekaEdureka!
 
Introduction to DevOps | Edureka
Introduction to DevOps | EdurekaIntroduction to DevOps | Edureka
Introduction to DevOps | EdurekaEdureka!
 

More from Edureka! (20)

What to learn during the 21 days Lockdown | Edureka
What to learn during the 21 days Lockdown | EdurekaWhat to learn during the 21 days Lockdown | Edureka
What to learn during the 21 days Lockdown | Edureka
 
Top 10 Dying Programming Languages in 2020 | Edureka
Top 10 Dying Programming Languages in 2020 | EdurekaTop 10 Dying Programming Languages in 2020 | Edureka
Top 10 Dying Programming Languages in 2020 | Edureka
 
Top 5 Trending Business Intelligence Tools | Edureka
Top 5 Trending Business Intelligence Tools | EdurekaTop 5 Trending Business Intelligence Tools | Edureka
Top 5 Trending Business Intelligence Tools | Edureka
 
Tableau Tutorial for Data Science | Edureka
Tableau Tutorial for Data Science | EdurekaTableau Tutorial for Data Science | Edureka
Tableau Tutorial for Data Science | Edureka
 
Python Programming Tutorial | Edureka
Python Programming Tutorial | EdurekaPython Programming Tutorial | Edureka
Python Programming Tutorial | Edureka
 
Top 5 PMP Certifications | Edureka
Top 5 PMP Certifications | EdurekaTop 5 PMP Certifications | Edureka
Top 5 PMP Certifications | Edureka
 
Top Maven Interview Questions in 2020 | Edureka
Top Maven Interview Questions in 2020 | EdurekaTop Maven Interview Questions in 2020 | Edureka
Top Maven Interview Questions in 2020 | Edureka
 
Linux Mint Tutorial | Edureka
Linux Mint Tutorial | EdurekaLinux Mint Tutorial | Edureka
Linux Mint Tutorial | Edureka
 
How to Deploy Java Web App in AWS| Edureka
How to Deploy Java Web App in AWS| EdurekaHow to Deploy Java Web App in AWS| Edureka
How to Deploy Java Web App in AWS| Edureka
 
Importance of Digital Marketing | Edureka
Importance of Digital Marketing | EdurekaImportance of Digital Marketing | Edureka
Importance of Digital Marketing | Edureka
 
RPA in 2020 | Edureka
RPA in 2020 | EdurekaRPA in 2020 | Edureka
RPA in 2020 | Edureka
 
Email Notifications in Jenkins | Edureka
Email Notifications in Jenkins | EdurekaEmail Notifications in Jenkins | Edureka
Email Notifications in Jenkins | Edureka
 
EA Algorithm in Machine Learning | Edureka
EA Algorithm in Machine Learning | EdurekaEA Algorithm in Machine Learning | Edureka
EA Algorithm in Machine Learning | Edureka
 
Cognitive AI Tutorial | Edureka
Cognitive AI Tutorial | EdurekaCognitive AI Tutorial | Edureka
Cognitive AI Tutorial | Edureka
 
AWS Cloud Practitioner Tutorial | Edureka
AWS Cloud Practitioner Tutorial | EdurekaAWS Cloud Practitioner Tutorial | Edureka
AWS Cloud Practitioner Tutorial | Edureka
 
Blue Prism Top Interview Questions | Edureka
Blue Prism Top Interview Questions | EdurekaBlue Prism Top Interview Questions | Edureka
Blue Prism Top Interview Questions | Edureka
 
Big Data on AWS Tutorial | Edureka
Big Data on AWS Tutorial | Edureka Big Data on AWS Tutorial | Edureka
Big Data on AWS Tutorial | Edureka
 
A star algorithm | A* Algorithm in Artificial Intelligence | Edureka
A star algorithm | A* Algorithm in Artificial Intelligence | EdurekaA star algorithm | A* Algorithm in Artificial Intelligence | Edureka
A star algorithm | A* Algorithm in Artificial Intelligence | Edureka
 
Kubernetes Installation on Ubuntu | Edureka
Kubernetes Installation on Ubuntu | EdurekaKubernetes Installation on Ubuntu | Edureka
Kubernetes Installation on Ubuntu | Edureka
 
Introduction to DevOps | Edureka
Introduction to DevOps | EdurekaIntroduction to DevOps | Edureka
Introduction to DevOps | Edureka
 

Recently uploaded

Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsDianaGray10
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3DianaGray10
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosErol GIRAUDY
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdfThe Good Food Institute
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfTejal81
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.IPLOOK Networks
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0DanBrown980551
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxNeo4j
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)codyslingerland1
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch TuesdayIvanti
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameKapil Thakar
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationKnoldus Inc.
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applicationsnooralam814309
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and businessFrancesco Corti
 
UiPath Studio Web workshop series - Day 1
UiPath Studio Web workshop series  - Day 1UiPath Studio Web workshop series  - Day 1
UiPath Studio Web workshop series - Day 1DianaGray10
 
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc
 
20140402 - Smart house demo kit
20140402 - Smart house demo kit20140402 - Smart house demo kit
20140402 - Smart house demo kitJamie (Taka) Wang
 
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTSIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTxtailishbaloch
 

Recently uploaded (20)

Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projects
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenarios
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch Tuesday
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First Frame
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its application
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applications
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile Brochure
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and business
 
UiPath Studio Web workshop series - Day 1
UiPath Studio Web workshop series  - Day 1UiPath Studio Web workshop series  - Day 1
UiPath Studio Web workshop series - Day 1
 
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
 
20140402 - Smart house demo kit
20140402 - Smart house demo kit20140402 - Smart house demo kit
20140402 - Smart house demo kit
 
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENTSIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
SIM INFORMATION SYSTEM: REVOLUTIONIZING DATA MANAGEMENT
 

Hadoop 2.0 Architecture | HDFS Federation | NameNode High Availability |

  • 2. Slide 2 Hello There!! My name is Annie. Let me test your Hadoop 1.x knowledge? Annie’s Introduction
  • 3. Slide 3 Hello There!! My name is Annie. I love quizzes and puzzles and I am here to make you guys think and answer my questions. Can you store 1 billion files in a Hadoop 1.x cluster? - Yes - No Annie’s Question
  • 4. Slide 4 No. Even though you have hundreds of DataNodes in the cluster, the NameNode keeps all its metadata in memory, so you are limited to a maximum of only 50-100M files in the entire cluster because of a Single NameNode in Hadoop 1.x. Annie’s Answer
  • 5. Slide 5 Hello There!! My name is Annie. I love quizzes and puzzles and I am here to make you guys think and answer my questions. A Hadoop 1.x cluster can have multiple HDFS Namespaces. - True - False Annie’s Question
  • 6. Slide 6 False. Not possible with Hadoop 1.x. Annie’s Answer
  • 7. Slide 7 Hello There!! My name is Annie. I love quizzes and puzzles and I am here to make you guys think and answer my questions. Which of the following is (are) a significant disadvantage in Hadoop 1.0? - ‘Single Point Of Failure’ of NameNode - Too much burden on Job Tracker Annie’s Question
  • 8. Slide 8 Single Point of Failure of NameNode and too much burden on Job Tracker. Annie’s Answer
  • 9. Slide 9 Hello There!! My name is Annie. I love quizzes and puzzles and I am here to make you guys think and answer my questions. Can you use hundreds of Hadoop DataNode for any other processing than MapReduce in Hadoop 1.x? - Yes - No Annie’s Question
  • 10. Slide 10 No. Hadoop 1.x dedicates all the DataNode resources to Map and Reduce slots with no or little room for processing any other workload. Annie’s Answer
  • 11. Slide 11 Hello There!! My name is Annie. I love quizzes and puzzles and I am here to make you guys think and answer my questions. Can you use Hadoop for Real-time processing? - Yes - No Annie’s Question
  • 12. Slide 12 No. Hadoop is designed and developer for massively parallel batch processing. Annie’s Answer
  • 13. Limitations of Hadoop 1.x  No horizontal scalability of NameNode  Does not support NameNode High Availability  Overburdened JobTracker  Not possible to run Non-MapReduce Big Data Applications on HDFS  Does not support Multi-tenancy
  • 14. Slide 14 www.edureka.in/hadoop Hadoop 1.x – In Summary Client HDFS Map Reduce Secondary NameNode Data BlocksDataNode NameNode Job Tracker Task Tracker Map Reduce DataNode Task Tracker Map Reduce…. DataNode DataNodeTask Tracker Map Reduce Task Tracker Map Reduce
  • 15. Slide 15 www.edureka.in/hadoop Problem Description NameNode – No Horizontal Scalability Single NameNode and Single Namespace, limited by NameNode RAM NameNode – No High Availability (HA) NameNode is Single Point of Failure, Need manual recovery using Secondary NameNode in case of failure Job Tracker – Overburdened Spends significant portion of time and effort managing the life cycle of Applications MRv1 – Only Map and Reduce tasks Humongous Data stored in HDFS remains unutilized and cannot be used for other workloads such as Graph processing etc. Hadoop 1.x - Challenges
  • 16. NameNode - No High Availability NameNode - No Horizontal Scale Data Node Data Node Data Node …. Client Get Block Locations Block Management Read Data NameNode NS Slide 16 www.edureka.in/hadoop NameNode – Scale and HA
  • 17. Slide 17 www.edureka.in/hadoop Name Node –Single Point of Failure  Secondary NameNode:  “Not a hot standby” for the NameNode  Connects to NameNode every hour*  Housekeeping, backup of NameNode metadata  Saved metadata can build a failed NameNode You give me metadata every hour, I will make it secure Single Point Failure Secondary NameNode NameNode metadata metadata
  • 18. Slide 18 www.edureka.in/hadoop Job Tracker – Overburdened CPU  Spends a very significant portion of time and effort managing the life cycle of applications Network  Single Listener Thread to communicate with thousands of Map and Reduce Jobs Task Tracker Task Tracker Task Tracker…. Job Tracker
  • 19. Slide 19 www.edureka.in/hadoop MRv1 – Unpredictability in Large Clusters As the cluster size grow and reaches to 4000 Nodes  Cascading Failures  The DataNode failures results in a serious deterioration of the overall cluster performance because of attempts to replicate data and overload live nodes, through network flooding.  Multi-tenancy  As clusters increase in size, you may want to employ these clusters for a variety of models. MRv1 dedicates its nodes to Hadoop and cannot be re-purposed for other applications and workloads in an Organization. With the growing popularity and adoption of cloud computing among enterprises, this becomes more important.
  • 20. Unutilized Data in HDFS  Terabytes and Petabytes of data in HDFS can only be used for MapReduce processing Slide 11 www.edureka.in/hadoop
  • 21. Introducing Hadoop 2.0 Features Hadoop 1.x Hadoop 2.0 HDFS Federation One NameNode and a Namespace Multiple NameNode and Namespaces NameNode High Availability Not present Highly Available YARN - Processing Control and Multi-tenancy JobTracker, TaskTracker Resource Manager, Node Manager, App Master, Capacity Scheduler Other important Hadoop 2.0 features  HDFS Snapshots  NFSv3 access to data in HDFS  Support for running Hadoop on MS Windows  Binary Compatibility for MapReduce applications built on Hadoop 1.0  Substantial amount of Integration testing with rest of the projects (such as PIG, HIVE) in Hadoop ecosystem Slide 12 www.edureka.in/hadoop
  • 22. Namenode Block Management NS Storage Datanode Datanode… NamespaceBlockStorage Namespace NS1 NSk NSn NN-1 NN-k NN-n Common Storage Datanode 1 … Datanode 2 … Datanode m … BlockStorage Pool 1 Pool k Pool n Block Pools … … Hadoop 1.0 Hadoop 2.0 Slide 22 www.edureka.in/hadoop http://hadoop.apache.org/docs/stable2/hadoop-project-dist/hadoop-hdfs/Federation.html Hadoop 2.0 Cluster Architecture - Federation
  • 23. Slide 23 www.edureka.in/hadoop cluster. Annie’s Question How does HDFS Federation help HDFS Scale horizontally? A) Reduces the load on any single NameNode by using the multiple, independent NameNodes to manage individual parts of the file system namespace. B) Provides cross-data centre (non-local) support for HDFS, allowing a cluster administrator to split the Block Storage outside the local cluster.
  • 24. Slide 24 www.edureka.in/hadoop Annie’s Answer (A). In order to scale the name service horizontally, HDFS federation uses multiple independent NameNodes. The NameNodes are federated, that is, the NameNodes are independent and do not require coordination with each other.
  • 25. Slide 25 Annie’s Question You have configured two NameNodes to manage /marketing and /finance namespaces respectively. What will happen if you try to ‘put’ a file in /accounting directory? www.edureka.in/hadoop
  • 26. Slide 26 Annie’s Answer The ‘put’ will fail. None of the namespaces will manage the file and you will get an IOException with a “No such file or directory error”. www.edureka.in/hadoop
  • 27. Slide 27 Node Manager HDFS YARN Resource Manager Shared edit logs All name space edits logged to shared NFS storage; single writer (fencing) Read edit logs and applies to its own namespace Data Node Standby NameNode Active NameNode Container App Master Node Manager Data Node Container App Master Data Node Client Data Node Container App Master Node Manager Data Node Container App Master Node Manager Hadoop 2.0 Cluster Architecture - HA NameNode High Availability Next Generation MapReduce HDFS HIGH AVAILABILITY http://hadoop.apache.org/docs/stable2/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithNFS.html
  • 28. Slide 28 Hadoop 2.0 Cluster Architecture - HA www.edureka.in/hadoop High Availability in Hadoop 2.0 NameNode recovery in Hadoop 1.0 Secondary NameNode Standby NameNode Active NameNode Secondary NameNode NameNode Edit logs Meta-Data Automatic failover to Standby NameNode Manually Recover using Secondary NameNode FSImage
  • 29. Slide 29 Annie’s Question NameNode HA was developed to overcome the following disadvantage in Hadoop 1.0? a) Single Point Of Failure of NameNode b) Too much burden on Job Tracker www.edureka.in/hadoop
  • 30. Slide 30 Annie’s Answer Single Point of Failure of NameNode. www.edureka.in/hadoop
  • 31. Apache Oozie (Workflow) HDFS (Hadoop Distributed File System) Pig Latin Data Analysis Hive DW System MapReduce Framework HBase Apache Oozie (Workflow) HDFS (Hadoop Distributed File System) Pig Latin Data Analysis Hive DW System MapReduce Framework HBase Other YARN Frameworks (MPI, GIRAPH) Slide 23 www.edureka.in/hadoop YARN Cluster Resource Management YARN adds a more general interface to run non-MapReduce jobs (such as Graph Processing) within the Hadoop framework YARN and Hadoop Ecosystem
  • 32. BATCH (MapReduce) INTERACTIVE (Text) ONLINE (HBase) STREAMING (Storm, S4, …) GRAPH (Giraph) IN-MEMORY (Spark) HPC MPI (OpenMPI) OTHER (Search) (Weave..) Slide 32 www.edureka.in/hadoop http://hadoop.apache.org/docs/stable2/hadoop-yarn/hadoop-yarn-site/YARN.html YARN – Moving beyond MapReduce
  • 33. Slide 33 www.edureka.in/hadoop  Organizes jobs into queues  Queue shares as %’s of cluster  FIFO scheduling within each queue  Data locality-aware Scheduling  Hierarchical Queues To manage the resource within an organization.  Capacity Guarantees A fraction to the total available capacity allocated to each Queue.  Security To safeguard applications from other users.  Elasticity Resources are available in a predictable and elastic manner to queues.  Multi-tenancy Set of limit to prevent over-utilization of resources by a single application.  Operability Runtime configuration of Queues.  Resource-based scheduling If needed, Applications can request more resources than the default. Multi-tenancy - Capacity Scheduler
  • 34. Slide 34 Annie’s Question YARN was developed to overcome the following disadvantage in Hadoop 1.0 MapReduce framework? a) Single Point Of Failure Of NameNode b) Too much burden on Job Tracker www.edureka.in/hadoop
  • 35. Slide 35 Annie’s Answer Too much burden on Job Tracker. www.edureka.in/hadoop
  • 36. Slide 36 NameNode High Availability Next Generation MapReduce Hadoop 2.0 – In Summary Client HDFS YARN Resource ManagerStandby NameNode Active NameNode Distributed Data Storage Distributed Data Processing DataNode Node Manager Container App Master ……. Masters Slaves Node Manager DataNode Container App Master DataNode Node Manager Container App Master Shared edit logs OR Journal Node Scheduler Applications Manager (AsM) www.edureka.in/hadoop
  • 37. Slide 37 Hello There!! My name is Annie. I love quizzes and puzzles and I am here to make you guys think and answer my questions. Can you use Hadoop 2.0 for Real-time processing? - Yes - No Annie’s Question
  • 38. Slide 38 No. Even though YARN in Hadoop 2.0 supports multiple frameworks for different workloads other than batch, you need Storm or S4 for real-time processing. Annie’s Answer
  • 39. Slide 39 www.edureka.in/hadoop What about Real-time Processing? Hadoop is good for Batch but How do I process Big Data in Real- time?
  • 40. Slide 40 www.edureka.in/hadoop Storm is coming…. APACHE STORM The Real-time Hadoop • Continuous commutation system Distributed, Reliable, Fault-tolerant, Scalable and Robust • Suitable for Big Data processing • Guarantees no data loss Programming Language agnostic • JSON-based for Ruby, Python etc. Use case • Stream processing • Distributed RPC • Continuous Computation
  • 41. Hadoop Vs. Storm Hadoop Storm Differences Fundamentally as Batch processing system Real-time processing, process unterminated streams (e.g. twitter feeds) of data, process data as it arrives MapReduce Jobs run to completion Topologies (Computation Graph) run forever Stateful Nodes Stateless Nodes Hadoop Storm Similarities Scalable Scalable Guarantees no data loss Guarantees no data loss Open Source Open Source
  • 42. Storm Use Cases  Data Normalization • Groupon uses Storm to build real-time data integration systems.  Analytics • Storm powers Twitter’s publisher analytics product, processing every tweet and click that happens on Twitter to provide analytics for Twitter's publisher partners. • Flipboard use Storm across a wide range of services ranging from Content Search to real-time analytics, to generating custom magazine fields.  Log processing • Alibaba uses Storm to process the application log and data change in databases to supply real-time data stats for data apps. • NaviSite uses Storm in its server log monitoring and auditing system.
  • 43. Thank You See You in Next Class