The document provides an overview of Hadoop and its role in managing big data, highlighting challenges like capture, storage, and analysis. It outlines Hadoop's architecture, including its data processing model (MapReduce) and filesystem (HDFS), which is designed for distributed computing. Key principles include scalability, handling failures, and the abstraction of complexity in concurrent applications.

1. Hadoop

2. Big Data
• Lots of Data
• The challenges include capture, storage, search, transfer,
analysis and visualization.
• Systems/Enterprise generates huge amount of data from
Terabyte to Petabytes of information.

3. Characteristics of Big
Data
The 3Vs are
• Volume
• Variety
• Velocity

4. What is Hadoop?
• Apache Hadoop is the framework that allows for
distributed processing of arrange datasets across cluster
of commodity computers using simple programming
model
• Its is Open source Data Management.

5. Hadoop System-
Principles
• Scale-Out rather then scale-up
• Bring code to data rather data to code
• Deal with failures – they are common
• Abstract complexity of distributed and concurrent
applications

6. HDFS
Filesystem cluster is managed by three types processes
• Name node
• Data node
• Secondary node

7. Files and Blocks
• Files are split into blocks(single unit of storage).
• Replicated across machine at load time.
• By default 3 replication.

8. Hadoop - MapReduce
• Model for processing large amount of data in parallel.
• Derived from functional programming.
• Can be implemented in multiple languages.

9. MapReduce Model
• Impose key-value input/output
• Defines map and reduce funtions
map : (k1,v1) -> list (k2,v2)
reduce : (k2,list(v2)) -> list (k3,v3)

10. MapReduce Framework
• Takes care of distributed processing and coordination
• Scheduling
• Task localization with Data
• Error Handling
• Data Synchronization

11. Yarn Daemons
- Node Manager
• Manages resources of single node
• There is one instance per node in the cluster
- Resource Manager
• Manages Resources for Cluster
• Instructs Node Manager to allocate resources