The document provides an overview of Hadoop, an open-source framework for storing and processing large datasets using distributed systems. It covers key components such as Hadoop Common, HDFS, and MapReduce, detailing their functionalities and architecture. The text emphasizes the efficiency, scalability, and cost-effectiveness of Hadoop in handling large-scale data processing tasks.

1. Hadoop Architecture
Presented by :
Yojana Nanaware
ME(CSE-I)

2. Agenda
• What is Hadoop?
• Why, When, Where?
• Hadoop : How?
• Hadoop Architecture
• Hadoop Common
• HDFS
• Hadoop Map/Reduce
• Process
• Hadoop Community
• Conclusion
• References

3. What is Hadoop?
• A SMART WAY TO STORE & ANALYAZE
DATA
• Douglas Reed Cutting, who is the creator
of Open-Source Technology & also
Hadoop. He originated Lucene and Nutch
• Open-source project administered by
Apache Software Foundation. Hadoop
consists of two key services:

4. What is Hadoop?
– Hadoop Distributed File System (HDFS).
– Map/Reduce .
• Hadoop is large-scale, high-performance
processing jobs — in spite of system
changes or failures

5. Why Hadoop?
• Need to process 100TB datasets
• On 1 node :
– Scanning @ 50MB/s=23 days
– MTBF = 3 years
• On 1000 node cluster :
– Scanning @ 50MB/s=33mins
– MTBF = 1 days
• Need efficient, Reliable & Usable
framework

6. Where & When?
• Where
– Batch Data
Processing, not
real-time/ user
facing
– Highly parallel data
intensive distributed
application
– Very large
production of
deployment
• When
– Process lots of
unstructured data
– When your processing
can easily be made
parallel
– Running batch jobs is
acceptable
– When you have to
access lots of cheap
hardware

7. Hadoop : How?
• Commodity hardware cluster
• Distributed File System
– Modeled on GFS
• Distributed Processing Framework
– Using Map/Reduce metaphor
• Open Source Java
– Apache Lucene Framework

8. Hadoop Architecture
Hadoop consists :
•Hadoop Common
– Support other Hadoop subprojects
•HDFS
– Provide high throughput access to application
data
•MapReduce
– Compute cluster of large data sets

9. Hadoop Common
• It is a set of utilities
• Includes File system, RPC, & Serialization
libraries

10. HDFS
• Primary storage system
• Creates multiple replicas of data blocks &
distributes them on compute nodes
throughout a cluster to enable reliable,
extremely rapid computations.
• Replication & locality

11. HDFS Architecture

12. Hadoop MapReduce
• The Map/Reduce programming language
– Framework
– Pluggable user code
• Common design pattern in design processing
cat * I grep I sort I unique –c I cat>file
input I map I shuffle I reduce I output
• Natural for
– log processing
– web search indexing
– Ad-hoc queries

13. Map/Reduce Implementation
1. input files split
2. Assign Masters &
Workers
3. Map tasks
4. Writing intermediate
data to disk
5. Intermediate data
read & sort
6. Reduce tasks
7. Return

14. Example of Map/Reduce word count
• Read text files & count how word often
occur.
– The input is text files
– The output is text file
• Each line : word, tab, count
• Map – Produce pair of (word, count)
• Reduce – For each word, sum up the
counts

15. Process
• Installation
– Requirements : Linux,
java1.6, sshd, rsync
– Configure SSH for
password free
authentication
– Unpack Hadoop
distribution
– Edit a few configuration
files
– Format the DFS on the
name node
– Start all the demon
process
• Execution
– Compile your job into a
jar files
– Copy input data into the
HDFS
– Execute bin/hadoop jar
with relevant arguments
– Monitor task via Web
interface (optional)
– Examine output when
job is complete

16. Hadoop Community
• Hadoop Users
– Adobe
– Alibaba
– Amazon
– AOL
– Facebook
– Google
– IBM
• Major Contributor
– Apache
– Cloudera
– Yahoo

17. Conclusion
• Designed to run on cheap commodity
power
• Handles data replication & node failure
• Cost saving & efficient & reliable data
processing

18. References
• http://www.newyorksys.com/hadoop-
online-training
• Hadoop on Wikipedia
(http://en.wikipedia.org/wiki/Hadoop )
• http://hadoop.apache.org/core/docs/curren
t/api/