Hadoop is an open-source software platform for distributed storage and processing of large datasets across clusters of computers. It was designed to scale up from single servers to thousands of machines, with very high fault tolerance. The document outlines the history of Hadoop, why it was created, its core components HDFS for storage and MapReduce for processing, and provides an example word count problem. It also includes information on installing Hadoop and additional resources.

1. Rabindra Nath Nandi
Software Engineer(Big Data) , IPvision Canada Inc

2. Outlines
● A brief history of Hadoop
● Why Hadoop
● Hadoop Fundamental
● HDFS
● MapReduce
● A MapReduce Program - WordCount Problem
● Installation
● Resources

3. A brief history of Hadoop
The genesis of Hadoop came from the Google File System paper(2003)
This paper spawned another research paper from Google - MapReduce:
Simplified Data Processing on Large Clusters.(2004)
Hadoop Project Started from Project Apache Nutch(2006)
Douh Cutting a Yahoo Researcher initially handled the project
Yahoo the Main Contributor

4. Why Hadoop
Everyday millions on contents are uploaded generated in facebook, google
These data needs to be stored and processed on demand
Today's hardware facility are high , so data storage doesn’t matter
So faster data processing and data storing with low cost is needed

5. Why Hadoop
Ability to store and process huge amounts of any kind of data, quickly
Computing power
Fault tolerance
Flexibility
Low cost
Scalability

6. Hadoop Fundamental
● Hadoop Provides both data storage and data processing facility
● HDFS- Hadoop Distributed File System
● MapReduce - A Distributed Data Processing Engine

7. HDFS Fundamental
● File systems that manage the storage across a network of machines are
called distributed file systems
Two Types of Nodes
● NameNode(Master): Holds metadata and keeps tracks of block's location in
DataNodes
● DataNode: Slave Nodes that stores and retrieves data block.DNs Periodically
reports to NameNode about list of block that they are storing
● File splits into 128 mb blocks (default)
● Replicated to 3 datanodes(default)

8. HDFS Fundamental

9. HDFS Fundamental

10. MapReduce Fundamental
Map() Function:
Process a key/value pair to generate intermediate key/value pair
Reduce() Function
Merge all intermediate values associated with the same key

11. MapReduce Fundamental

12. A MapReduce Program - WordCount Problem
Mapper Example

13. A MapReduce Program - WordCount Problem
Reducer Example

14. A MapReduce Program - WordCount Problem
Driver Class

15. Installation
http://www.tutorialspoint.com/hadoop/hadoop_enviornment_setup.htm
https://www.digitalocean.com/community/tutorials/how-to-install-hadoop-on-
ubuntu-13-10

16. Resources
https://hadoop.apache.org
Hadoop: The Definitive Guide, 3rd Edition
Storage and Analysis at Internet Scale By Tom White