Introduction to Hadoop Slide for lab Seminar Multimedia Information Management Lab Department of Multimedia Engineering, Mokpo National University

1. AN INTRODUCTION
TO
HADOOP
SudarshanPant
MOKPONATIONALUNIVERSITY
2017.06.01

2. HADOOP
An open-sourceimplementationof frameworks for reliable,scalable,
distributedcomputingand data storage.
A flexibleand highly-availablearchitecturefor large scalecomputation
and dataprocessing on a networkof commodity hardware.
Redundantand reliable.
Batch processing centric.

3. HADOOP
2003
2004
2006

4. 2005: Doug Cutting and Michael J. Cafarella developed Hadoop with funding from
Yahoo.
2006: Yahoo gave the project to Apache Software Foundation.
Google File system
Google MapReduce
BigTable
HADOOP

5. Hadoopis a system for large scale data processing.
It has two components
HDFS – Hadoop distributed file system(for storage)
 Distributed across nodes
 Natively redundant
Map Reduce(forprocessing)
 Splits a taskacross processors
 “near” the data and assembles results
 Clustered storage
Machine
HDFS
MapReduce
HADOOP

6.  The Mapreduce server on a typical machineis called TaskTracker
 The HDFSserver on a typical machineis called a DataNode
Machine
DataNode
TaskTracker
Machine
DataNode
TaskTracker
Machine
DataNode
TaskTracker
HADOOP

7. Machine
DataNode
TaskTracker
Machine
DataNode
TaskTracker
Machine
DataNode
TaskTracker
JobTracker
JOBTRACKER
 JobTracker keeps track of jobs being run.
 Keeps track of the nodes bydetectingthe nodefailureand reassigning
the taskto anothernode

8.  NameNodekeeps informationon data location
 Keeps noticesthenode failureand automaticallyre-replicate thedata
NAME NODE
Machine
DataNode
TaskTracker
Machine
DataNode
TaskTracker
Machine
DataNode
TaskTracker
NameNode

9. Pig
Hive
MapReduce
HDFS
HBase
Hadoop System
ZOOKEEPER
HADOOP ECOSYSTEM

10. Pig : High level languagethattranslatesdown into MapReduce Job.
Allows to write job description in highlevel languageinsteadof writing all
MapReduce job code in Java.
Hive: SQL likeinterface for computations. TakesSQL-like language and converts
it intoMapReduce Jobs
Hbase: provides simple interface to distribute datathat allowsthe realtime data
availabilityto the data.
ZooKeeper: provides scalable, highly availablecoordination service for servers
by keeping the metadata about different servers.
HADOOP ECOSYSTEM

11.  Mahout– for machinelearning
 Sqoop – helps tomove datato and from RelationationalSQL
databases.
 Oozie - workflow coordinationtool (definingscheduledtasks)
 Flume–for importingstreamingdatato Hadoop
 Protobuff,Avro, Thrift- for data serialization
OTHER USEFUL TOOLS

12. Client NameNode
 Allocationofblockstofile
 MonitoringdataNodeforfailure
 NewNodeAddition
 ReplicationManagement
 ManagingUser Requests(read/write)
DataNode1 DataNode2 DataNode3 DataNoden
Metadata operations
Read/Write
Block
NameNode
Commands
 Write/Readblocks to/from Local File
 Perform operation as directedbynameNode
 Register/Heartbeat itself with namenodeand provideblock report to name
node
FUNCTIONOFHDFSCONPONENTS

13. File A
70MB
File B
8MB
BlockA
Allocated64MB
Consumed64MB
SpaceLeft0MB
BlockB
Allocated64MB
Consumed8MB
SpaceLeft56MB
BlockC
Allocated64MB
Consumed8MB
SpaceLeft56MB
 Blocks arecreatedbasedonpredefined block size (default : 64MB)
 Blocks arethen replicated as perthe defined Replication Factor (default : 3)
 1 file –1 block rule :Oneblock does not contain the contents from two different files.
FILEHANDLING

14.  Parallel programming modelfor largeclusters
 Co-Designed, Co-developed andCo-deployed with HDFS
 Processing takes place where data is located.
 Conducted in two phases
 Map
 Reduce
Map: Processakey/valuepairtogenerateintermediatekey/valuepairs.
Reduce:Mergeall intermediatevaluesassociatedwiththesamekey.
Input
Data
Map
(Thread 1)
Map
(Thread P)
Sort
Reduce
(Thread 1)
Reduce
(Thread Q)
Output
DataSplit Split
Merge
MAPREDUCE

15. TheDriverconfigures how the program will run in Hadoop
TheMapperis fed input data from Hadoopand produces intermediate output
TheReduceris fed the intermediate output from the mappers
and produces the final output.
When data moves between the mappers andreducers,
it might cross the network so all the values mustbeserializable.
STRUCTUREOF
MAPREDUCEPROGRAM