This presentation about Hadoop will help you learn the basics of Hadoop and its components. First, you will see what is Big Data and the significant challenges in it. Then, you will understand how Hadoop solved those challenges. You will have a glance at the History of Hadoop, what is Hadoop, the different companies using Hadoop, the applications of Hadoop in different companies, etc. Finally, you will learn the three essential components of Hadoop – HDFS, MapReduce, and YARN, along with their architecture. Now, let us get started with Introduction to Hadoop. Below topics are explained in this Hadoop presentation: 1. Big Data and its challenges 2. Hadoop as a solution 3. History of Hadoop 4. What is Hadoop 5. Applications of Hadoop 6. Components of Hadoop 7. Hadoop Distributed File System 8. Hadoop MapReduce 9. Hadoop YARN What is this Big Data Hadoop training course about? The Big Data Hadoop and Spark developer course have been designed to impart an in-depth knowledge of Big Data processing using Hadoop and Spark. The course is packed with real-life projects and case studies to be executed in the CloudLab. What are the course objectives? This course will enable you to: 1. Understand the different components of Hadoop ecosystem such as Hadoop 2.7, Yarn, MapReduce, Pig, Hive, Impala, HBase, Sqoop, Flume, and Apache Spark 2. Understand Hadoop Distributed File System (HDFS) and YARN as well as their architecture, and learn how to work with them for storage and resource management 3. Understand MapReduce and its characteristics, and assimilate some advanced MapReduce concepts 4. Get an overview of Sqoop and Flume and describe how to ingest data using them 5. Create database and tables in Hive and Impala, understand HBase, and use Hive and Impala for partitioning 6. Understand different types of file formats, Avro Schema, using Arvo with Hive, and Sqoop and Schema evolution 7. Understand Flume, Flume architecture, sources, flume sinks, channels, and flume configurations 8. Understand HBase, its architecture, data storage, and working with HBase. You will also understand the difference between HBase and RDBMS 9. Gain a working knowledge of Pig and its components 10. Do functional programming in Spark 11. Understand resilient distribution datasets (RDD) in detail 12. Implement and build Spark applications 13. Gain an in-depth understanding of parallel processing in Spark and Spark RDD optimization techniques 14. Understand the common use-cases of Spark and the various interactive algorithms 15. Learn Spark SQL, creating, transforming, and querying Data frames Learn more at https://www.simplilearn.com/big-data-and-analytics/introduction-to-big-data-and-hadoop-certification-training.

1. Hadoop Training

2. What’s in it for you?
1. Big data and its challenges
What’s in it for you?

3. What’s in it for you?
1. Big data and its challenges
2. Hadoop as a solution
What’s in it for you?

4. What’s in it for you?
1. Big data and its challenges
2. Hadoop as a solution
3. History of Hadoop
What’s in it for you?

5. What’s in it for you?
1. Big data and its challenges
2. Hadoop as a solution
3. History of Hadoop
4. What is Hadoop?
What’s in it for you?

6. What’s in it for you?
1. Big data and its challenges
2. Hadoop as a solution
3. History of Hadoop
4. What is Hadoop?
5. Components of Hadoop
What’s in it for you?

7. What’s in it for you?
1. Big data and its challenges
2. Hadoop as a solution
3. History of Hadoop
4. What is Hadoop?
5. Components of Hadoop
6. Hadoop Distributed File System
What’s in it for you?

8. What’s in it for you?
1. Big data and its challenges
2. Hadoop as a solution
3. History of Hadoop
4. What is Hadoop?
5. Components of Hadoop
6. Hadoop Distributed File System
7. Hadoop MapReduce
What’s in it for you?

9. What’s in it for you?
1. Big data and its challenges
2. Hadoop as a solution
3. History of Hadoop
4. What is Hadoop?
5. Components of Hadoop
6. Hadoop Distributed File System
7. Hadoop MapReduce
8. Hadoop YARN
What’s in it for you?

10. What’s in it for you?
Big data and its
challenges

11. Big Data and its challenges
Data in various forms Drawing insights from
data
Data has evolved significantly over the last 5 years and is being
generated in various forms whose volume is too large. Hence, it is
termed as Big Data
This data has to be handled efficiently to draw values from it and
take business decisions

12. Big Data and its challenges
Data in various forms Drawing insights from
data
Handling such vast volumes of data throws a lot
of challenges

13. Big Data and its challenges
Data in various forms Drawing insights from
data
Handling such vast volumes of data throws a lot
of challenges
Storing such large volumes of data

14. Big Data and its challenges
Data in various forms Drawing insights from
data
Handling such vast volumes of data throws a lot
of challenges
Storing such large volumes of data
Processing the data faster

15. Big Data and its challenges
Data in various forms Drawing insights from
data
Handling such vast volumes of data throws a lot
of challenges
Storing such large volumes of data
Processing the data faster
Maintaining scalability and data quality

16. Big Data and its challenges
Data in various forms Drawing insights from
data
Handling such vast volumes of data throws a lot
of challenges
Storing such large volumes of data
Processing the data faster
Keeping the data safe and secure
Maintaining scalability and data quality

17. What’s in it for you?
Hadoop as a solution

18. Hadoop as a solution
Parallel computing
Hadoop allows you to perform parallel
computing(processing) on large volumes of data
quickly and efficiently

19. Hadoop as a solution
Distributed storageParallel computing
Hadoop stores data on multiple systems instead
of a single system

20. Hadoop as a solution
Distributed storageParallel computing
Maintains data quality and
security
Provides end-to-end encryption that protects data
while it is at rest within the Hadoop cluster and in
motion across the network

21. What’s in it for you?
History of Hadoop

22. Doug Cutting and Mike Cafarella
created Apache Nutch
Google released paper on GFS to
describe how to store large datasets
Google released another paper on
MapReduce which described
processing large datasets
Doug Cutting started using GFS and
MapReduce in Nutch
Cutting joined Yahoo and took
Nutch with him
Cutting split out the distributed
computing parts from Nutch and
created Hadoop
Yahoo successfully tested Hadoop on
1000 node clusters
Yahoo released Hadoop as an open
source project to Apace Software
foundation (ASF)
Hadoop was successfully tested to
sort a PetaByte of data in less than 17
hours
Apache Software Foundation
released Apache Hadoop Version 1.0
Apache Hadoop Version 3.0 released
2002 2003 2004
200520062007
Jan
2008
July
2008
2009
20112017

23. What’s in it for you?
What is Hadoop?

24. What is Hadoop?
Hadoop is an open-source framework for storing data and running applications on clusters of
commodity hardware

25. What is Hadoop?
Hadoop is an open-source framework for storing data and running applications on clusters of
commodity hardware
Big Data
Datanode 1 Datanode 2
Datanode 3Datanode 4
Hadoop stores and processes data on multiple systems

26. What’s in it for you?
Advantages of Hadoop

27. Advantages of Hadoop
Store and
process huge
data
• Hadoop can store
huge volumes of data
of different types
such as text, social
media data, log files,
etc.

28. Advantages of Hadoop
Store and
process huge
data
Computing power
• Hadoop can store
huge volumes of data
of different types
such as text, social
media data, log files,
etc.
• Hadoop's distributed
computing model
processes big data fast
• The more computing
nodes you use, the
more processing power
you have

29. Advantages of Hadoop
Store and
process huge
data
Computing power Flexibility
• Hadoop can store
huge volumes of data
of different types
such as text, social
media data, log files,
etc.
• Hadoop's distributed
computing model
processes big data fast
• The more computing
nodes you use, the
more processing power
you have
• Unlike traditional
relational databases,
you don’t have to
preprocess data
before storing it
• You can store as
much data as you
want and decide how
to use it later

30. Advantages of Hadoop
Store and
process huge
data
Computing power Flexibility Scalability
• Hadoop can store
huge volumes of data
of different types
such as text, social
media data, log files,
etc.
• Hadoop's distributed
computing model
processes big data fast
• The more computing
nodes you use, the
more processing power
you have
• Unlike traditional
relational databases,
you don’t have to
preprocess data
before storing it
• You can store as
much data as you
want and decide how
to use it later
• Hadoop allows you to
quickly grow your
system to handle
more data just by
adding nodes
• Very little
administration is
required

31. Advantages of Hadoop
Store and
process huge
data
Computing power Flexibility Scalability Fault tolerance
• Hadoop can store
huge volumes of data
of different types
such as text, social
media data, log files,
etc.
• Hadoop's distributed
computing model
processes big data fast
• The more computing
nodes you use, the
more processing power
you have
• Unlike traditional
relational databases,
you don’t have to
preprocess data
before storing it
• You can store as
much data as you
want and decide how
to use it later
• Data is protected
against hardware
failure
• If a datanode goes
down, jobs are
automatically
redirected to other
nodes to make sure
the distributed
computing does not
fail
• Hadoop allows you to
quickly grow your
system to handle
more data just by
adding nodes
• Very little
administration is
required

32. What’s in it for you?
Companies using Hadoop

33. Companies using Hadoop

34. What’s in it for you?
Applications of Hadoop

35. Applications of Hadoop
• Royal Bank of Scotland
uses Hadoop to gain
intelligence from its online
customer chat
conversations
• It used machine learning
through Cloudera Hadoop
distribution to process
250,000 chat logs and
understand its customer
journeys

36. Applications of Hadoop
• Royal Bank of Scotland
uses Hadoop to gain
intelligence from its online
customer chat
conversations
• It used machine learning
through Cloudera Hadoop
distribution to process
250,000 chat logs and
understand its customer
journeys
• Marks and Spencer uses
Hadoop to analyze its data
from multiple sources to
understand its customer
behavior better
• It solves gaps in campaign
management, manage
customer loyalty data,
create more personalized
and targeted campaigns

37. Applications of Hadoop
• Royal Bank of Scotland
uses Hadoop to gain
intelligence from its online
customer chat
conversations
• It used machine learning
through Cloudera Hadoop
distribution to process
250,000 chat logs and
understand its customer
journeys
• Marks and Spencer uses
Hadoop to analyze its data
from multiple sources to
understand its customer
behavior better
• It solves gaps in campaign
management, manage
customer loyalty data,
create more personalized
and targeted campaigns
• Royal Mail is a British
postal service company
that uses Hadoop to
transform the way it
managed data across the
organization
• It can now identify
customers in particular
industries who are most at
risk of churn, allowing the
sales and marketing teams
to take preventative steps

38. Applications of Hadoop
• Royal Bank of Scotland
uses Hadoop to gain
intelligence from its online
customer chat
conversations
• It used machine learning
through Cloudera Hadoop
distribution to process
250,000 chat logs and
understand its customer
journeys
• Marks and Spencer uses
Hadoop to analyze its data
from multiple sources to
understand its customer
behavior better
• It solves gaps in campaign
management, manage
customer loyalty data,
create more personalized
and targeted campaigns
• Royal Mail is a British
postal service company
that uses Hadoop to
transform the way it
managed data across the
organization
• It can now identify
customers in particular
industries who are most at
risk of churn, allowing the
sales and marketing teams
to take preventative steps
• Spotify is an on-demand
music service that uses
Hadoop big data analytics
• It collects data from its
millions of users worldwide
and they analyzes the data
to give music
recommendations to
individual users

39. What’s in it for you?
Hadoop Components

40. Hadoop Components
Stores layer of Hadoop that stores and maintains rapidly increasingly data

41. Hadoop Components
Stores layer of Hadoop that stores and maintains rapidly increasingly data
Processing layer of Hadoop that perform parallel computing of data

42. Hadoop Components
Stores layer of Hadoop that stores and maintains rapidly increasingly data
Processing layer of Hadoop that perform parallel computing of data
Used for cluster resource management that assigns resources to applications

43. What’s in it for you?
Hadoop Distributed File
System

44. Hadoop Distributed File System
Hadoop Distributed File System (HDFS) is the storage unit of Hadoop that stores big
data in multiple server machines instead of a central server

45. Hadoop Distributed File System
Hadoop Distributed File System (HDFS) is the storage unit of Hadoop that stores big
data in multiple server machines instead of a central server
Input data
HDFS divides the input file into smaller chunks
and stores the data across the Hadoop cluster
Datanode
1
Datanode
2
Datanode
3
Datanode
4

46. Hadoop Distributed File System
A large file is split by Hadoop framework into blocks whose default block size is 128
MB
128 MB
128 MB
128 MB
128 MB
128 MB
128 MB
128 MB
128 MB
1 GB input file
The machines in the Hadoop cluster have disks that store
these blocks

47. HDFS Architecture
Namenode
MetaData (Name, replicas, ….):
/home/foo/data, 3, ….
DatanodesDatanodes
Metadata ops
Client
Rack 1 Rack 2
……………
…….
Master node
Slave nodes
Namenode stores
metadata information

48. HDFS Architecture
Namenode
MetaData (Name, replicas, ….):
/home/foo/data, 3, ….
DatanodesDatanodes
Metadata ops
Client
Rack 1 Rack 2
……………
…….
Read
A client can read data from the datanodes by
consulting the namenode

49. HDFS Architecture
Namenode
MetaData (Name, replicas, ….):
/home/foo/data, 3, ….
DatanodesDatanodes
Metadata ops
Client
Read
Rack 1 Rack 2Client
Write Write
Client can write data to the respective
datanodes
……………
…….

50. HDFS Architecture
Namenode
MetaData (Name, replicas, ….):
/home/foo/data, 3, ….
Block ops
DatanodesDatanodes
Metadata ops
Client
Read
Rack 1 Rack 2
Replication
Client
Write Write
HDFS allows you to perform block operation
and replication of data blocks to different
datanodes

51. What’s in it for you?
Hadoop MapReduce

52. Hadoop MapReduce
MapReduce is a programming framework that allows distributed computing and
performs parallel processing of large volumes of data

53. Hadoop MapReduce
Master node
Slave node
Slave node
Slave node
Slave node
Data is processed parallelly on the different slave
nodes
MapReduce is a programming framework that allows distributed computing and
performs parallel processing of large volumes of data

54. Hadoop MapReduce
Map Reduce

55. Hadoop MapReduce
Map Reduce
Input

56. Hadoop MapReduce
Map Reduce
Map tasks
Map()Input

57. Hadoop MapReduce
Map Reduce
Map tasks
Map()Input
Shuffle and
sort

58. Hadoop MapReduce
Map Reduce
Reduce tasks
Reduce()
Map tasks
Map()Input
Shuffle and
sort

59. Hadoop MapReduce
Map Reduce
Reduce tasks
Reduce()
Map tasks
Map()Input Output
Shuffle and
sort

60. What’s in it for you?
Hadoop YARN

61. Hadoop YARN
YARN was introduced in Hadoop 2.0 and it stands for
Yet Another Resource Negotiator

62. Hadoop YARN
YARN was introduced in Hadoop 2.0 and it stands for
Yet Another Resource Negotiator
• YARN is the cluster resource management layer of
Hadoop
• Schedules jobs and assigns resources to running
applications
Memory RAM
Network
bandwidth
Disk IO

63. YARN Architecture
Resource
ManagerClient
Job Submission
Submit job
request

64. YARN Architecture
Resource
ManagerClient
Node
Manager
container
App Master
App Master
container
Node
Manager
Node
Manager
container container
Job Submission
Submit job
request

65. YARN Architecture
Resource
ManagerClient
Node
Manager
container
container
Node
Manager
Node
Manager
container container
Job Submission
Node Status
Submit job
request
App Master
App Master

66. YARN Architecture
Resource
ManagerClient
Node
Manager
container
container
Node
Manager
Node
Manager
container container
Job Submission
Node Status
MapReduce Status
Submit job
request
App Master
App Master

67. YARN Architecture
Resource
ManagerClient
Node
Manager
container
container
Node
Manager
Node
Manager
container container
Job Submission
Node Status
MapReduce Status
Resource Request
Submit job
request
App Master
App Master

68. Key Takeaways