Hadoop is an open source framework for distributed storage and processing of large datasets across clusters of commodity hardware. It uses Google's MapReduce programming model and Google File System for reliability. The Hadoop architecture includes a distributed file system (HDFS) that stores data across clusters and a job scheduling and resource management framework (YARN) that allows distributed processing of large datasets in parallel. Key components include the NameNode, DataNodes, ResourceManager and NodeManagers. Hadoop provides reliability through replication of data blocks and automatic recovery from failures.

1. Hadoop Fundamentals

2. Agenda
• What is Hadoop
• Revisiting Big Data
• Examples of Hadoop in Action
• Limitations of Hadoop
• Big Data and Cloud
• Hadoop Architecture

3. What is Hadoop?
• Open Source Project
• Written in Java
• Uses Google’s MapReduce and Google File system as its foundation
• Optimized to handle
• Massive amounts of data through parallelism
• A variety of data (structured, unstructured and semi structured)
• Using inexpensive commodity hardware
• Great Performance
• Reliability provided through replication
• Not suited for OLTP workloads, OLAP workloads
• Hadoop is used for Big Data.. Complements OLTP and OLAP

4. What is Big Data?
• With all the devices available to collect data viz. RFID readers, microphones,
cameras and sensors, there is an explosion of data worldwide
• Big data is large collections of data (also known as datasets) that may be
unstructured, and grow so large and quickly that it is difficult to manage
with regular database or statistical tools
Important statistics
• > 2 billion internet users & 7.3 billion active cellphones
• Twitter processes 7TB of data every day, Facebook processes 500TB
• approximately 80% of these data are unstructured
Need for fast, reliable, deep data insight. Hence relevance of Hadoop

5. Some of the Open Source Projects related to Hadoop
• Eclipse - popular IDE donated by IBM to the open source community
• Lucene – Text search engine developed in Java
• Hbase – Hadoop database
• Hive – provides data warehousing tools to extract, transform and load
data, and then, query this data stored
• Pig – High level language that generates MapReduce code to analyze
large datasets

6. Examples of Hadoop in Action
• 2011 – Watson, a Supercomputer developed by IBM participated in a
quiz show – Jeapordy
• Approximately 2 million pages of text were input using Hadoop to
distribute the workload for storing information into memory
• Use of advanced search and analysis
China Mobile
• Hadoop cluster to data mining on Call Data Records
• China Mobile was producing 5-8 TB of data
• Used Hadoop to process 10 times of data with 1/5th of the cost

7. Examples of Hadoop in Action
New York Times
• Hosted all public domain articles from 1851 to 1922
• 11 million files converted into 1.5 TB of PDF
• One employee who ran a job for 24 hours on a 100-instance Amazon
EC2 Hadoop
Yahoo
• Largest production user with an application running a Hadoop cluster
consisting of approximately 10,000 Linux machines
• Also largest contributor to Hadoop Opensource Project

8. Limitations of Hadoop
• Not good for processing transactions (random access)
• Not good when work cannot be parallalized
• Not good for low latency data access
• Not good for processing lots of small files
• Not good for intensive calculations with little data

9. Big Data Solutions and the Cloud
• Big Data solutions are more than just Hadoop
• Add business intelligence / analytics functionality
• Derive information from data in motion
• Big Data solutions and the cloud are a perfect fit.
• Cloud allows you to setup a cluster of systems in minutes and it’s relatively
inexpensive

10. Hadoop Architecture
Terminology Review
• Node – a computer typically non-enterprise, commodity hardware
• Rack – a collection of 30 – 40 nodes that are physically stored close to
each other and connected to same network switch
• Network bandwidth between any two nodes in rack is greater than
bandwidth between two nodes on different racks
• A Hadoop Cluster is a collection of racks

11. Pre Hadoop 2.2 Architecture
• Distributed File System
• Hadoop Distributed File System (HDFS)
• GPFS – FPO
• MapReduce Engine
• Framework for performing calculations on the data in the file system
• Has a built in resource manager and scheduler
Pre Hadoop 2.2 MapReduce is called MapReduce 1.0 has its own
resource management and scheduling

12. Hadoop Distributed File System
• A distributed file system that provides high-performance access to data
across Hadoop clusters.
• Key tool for managing pools of big data and supporting big data
analytics applications.

13. Hadoop Distributed File System
• HDFS runs on the existing file system
• Not POSIX compliant
• Designed to tolerate high component failure rate
• Reliability through replication
• Designed to handle very large files
• Large streaming data access patterns
• No random access
• Uses blocks to store a file or parts of a file

14. Basic Features of HDFS
• Large: A HDFS instance may consist of thousands of server machines,
each storing part of the file system’s data
• Replication – Each data block replicated many times (typically 3)
• Failure – failure is the norm rather than exception
• Highly fault-tolerant
• Name node consistently checking data nodes
• Detection of faults and quick automatic recovery from them is the goal
• High throughput
• Streaming access to file system data
• Can be built out of commodity hardware

15. Types of Nodes
Client Job Tracker
Data Node
Task Tracker
Data Node
Task Tracker
Data Node
Task Tracker
Data Node
Task Tracker
Name Node

16. Name Nodes & Data Nodes
• Name Node maintains metadata about the files
• Data Nodes
• Store actual data
• Files are divided into blocks
• Each block is replicated N times (default N=3)
 Master/slave architecture
 HDFS cluster consists of a single Namenode, a master server that manages the file
system namespace and regulates access to files by clients.
 There are a number of DataNodes usually one per node in a cluster.
 The DataNodes manage storage attached to the nodes that they run on.
 HDFS exposes a file system namespace and allows user data to be stored in files.
 A file is split into one or more blocks and set of blocks are stored in DataNodes.
 DataNodes: serves read, write requests, performs block creation, deletion, and
replication upon instruction from Namenode

17. Job Tracker and Task Tracker
• Job Tracker is the Master Node
• Receives the user’s job
• Decides on how many tasks will run (number of mappers)
• Decides on where to run each mapper (concept of locality)
• Task Tracker is the Slave Node
• Receives the task from Job Tracker
• Runs the task until completion (either map or reduce task)
• Always in communication with the Job Tracker reporting progress

18. Fault Tolerance
• Failure is the norm rather than exception
• A HDFS instance may consist of thousands of server machines, each
storing part of the file system’s data.
• Since we have huge number of components and that each component
has non-trivial probability of failure means that there is always some
component that is non-functional.
• Detection of faults and quick, automatic recovery from them is a core
architectural goal of HDFS

19. Data Characteristics
Streaming data access
Applications need streaming access to data
Batch processing rather than interactive user access.
Large data sets and files: gigabytes to terabytes size
High aggregate data bandwidth
Scale to hundreds of nodes in a cluster
Tens of millions of files in a single instance
Write-once-read-many: a file once created, written and closed need
not be changed – this assumption simplifies coherency
A map-reduce application or web-crawler application fits perfectly
with this model

20. Data Replication
HDFS is designed to store very large files across machines in a large
cluster.
Each file is a sequence of blocks.
All blocks in the file except the last are of the same size.
Blocks are replicated for fault tolerance.
Block size and replicas are configurable per file.
The Namenode receives a Heartbeat and a BlockReport from each
DataNode in the cluster.
BlockReport contains all the blocks on a Datanode

21. HDFS file blocks
• Not the same as operating system’s file blocks
• HDFS book made up of multiple operating system blocks
• Default for Hadoop is 64 MB
• Recommended is 128 MB (this is BigInsight’s block)
• Size of a block can be larger than the single disk in a cluster
• Blocks for single file are spread across multiple nodes in a cluster
• If a chunk of file is smaller than HDFS block size
• Only the needed space is used
• Blocks work well with replication
• Fault tolerant and availability with commodity hardware

22. HDFS Replication
• Blocks of data are replicated to multiple nodes
• Allows for node failure without data loss
• Replication can be done on many more nodes by
• Changing Hadoop configuration
• Setting replication factor for each file

23. MapReduce Framework
• Based on technology from Google
• Processes huge datasets with certain kind of distributable problems
using large number of nodes
• A MapReduce program consists of map and reduce functions
• Map-Phase – divides data into smaller subsets that are distributed over
different nodes
• Reduce – Phase – master node collects all the returned data and combines
into some sort of output that can be used again
• Allows for distributed processing of map and reduce operations
• Tasks run in parallel

24. Dataflow in MapReduce

25. Features of MapReduce
• Fine grained Map and Reduce tasks
• Improved load balancing
• Faster recovery from failed tasks
• Automatic re-execution on failure
• In a large cluster, some nodes are always slow or flaky
• Framework re-executes failed tasks
• Locality optimizations
• With large data, bandwidth to data is a problem
• Map-Reduce + HDFS is a very effective solution
• Map-Reduce queries HDFS for locations of input data
• Map tasks are scheduled close to the inputs when possible

26. Key Value Pairs
• Mappers and Reducers are users’ code (provided functions)
• Just need to obey the Key-Value pairs interface
• Mappers:
• Consume <key, value> pairs
• Produce <key, value> pairs
• Reducers:
• Consume <key, <list of values>>
• Produce <key, value>
• Shuffling and Sorting:
• Hidden phase between mappers and reducers
• Groups all similar keys from all mappers, sorts and passes them to a certain
reducer in the form of <key, <list of values>>

27. Word Count Example
• Mapper
• Input: value: lines of text of input
• Output: key: word, value: 1
• Reducer
• Input: key: word, value: set of counts
• Output: key: word, value: sum
• Launching program
• Defines this job
• Submits job to cluster

28. Example - Word Count Dataflow

29. Example – Color Count
29
Shuffle & Sorting
based on k
Reduce
Reduce
Reduce
Map
Map
Map
Map
Input blocks
on HDFS
Produces (k, v)
( , 1)
Parse-hash
Parse-hash
Parse-hash
Parse-hash
Consumes(k, [v])
( , [1,1,1,1,1,1..])
Produces(k’, v’)
( , 100)
Job: Count the number of each color in a data set
Part0003
Part0002
Part0001
That’s the output file, it has
3 parts on probably 3
different machines

30. How Many Maps and Reduces
• Maps
• Usually as many as the number of HDFS blocks being processed, this is the default
• Else the number of maps can be specified as a hint
• The number of maps can also be controlled by specifying the minimum split size
• The actual sizes of the map inputs are computed by:
• max(min(block_size,data/#maps), min_split_size
• Reduces
• Unless the amount of data being processed is small
• 0.95*num_nodes*mapred.tasktracker.tasks.maximum

31. Types of Nodes
• Named Node
• Only one Named Node in a cluster
• Stores metadata for the data node
• Manages the file system namespace and metadata
• Data does not go through the named node
• Data is not stored in the named node
• Single point of failure
• Good idea to mirror named node
• Do not use inexpensive commodity hardware
• Has large memory requirement
• File system metadata is maintained in RAM to server read requests

32. Types of Nodes
Data Node
• Many per Hadoop Cluster
• Manages the blocks with data and serves them to clients
• Blocks from different files can be stored on the same DataNode
• Periodically reports to the NameNode the list of blocks it stores
• Suitable for inexpensive commodity hardware – replication provided at
software level

33. Types of Nodes
Job Tracker
• Manages the MapReduce jobs in the cluster
• One per Hadoop Cluster
• Receives job requests submitted by the client
• Schedules and monitors MapReduce Jobs on TaskTrackers
• Attempts to direct a task to the TaskTracker where the data resides
• Monitors of any failing tasks that need to rescheduled

34. Types of Nodes
TaskTracker
• Many per Hadoop Cluster
• Executes the MapReduce operations
• Runs the MapReduce tasks in JVMs
• Have a set of slots used to run tasks
• Communicates with JobTracker
• Reads blocks from DataNodes

35. Hadoop 2.2 Architecture
Provides YARN
• Referred to as MapReduce V2
• Resource Manager and Scheduler external to anyframework
• DataNodes still exist
• JobTracker and TaskTrackers no longer exist
• Not Required to run YARN with Hadoop 2.2
• Still supports MapReduce V1

36. YARN
Two main ideas
• Provide generic scheduling and resource management
• Support more than just MapReduce
• Support more than just batch processing
• More efficient scheduling and workload management
• No more balancing between map slots and reduce slots

37. Hadoop High Availability
JournalNode JournalNode JournalNode
NameNode NameNode
DataNode DataNode DataNode DataNode DataNode