Hadoop is an open-source distributed processing framework created by Doug Cutting in 2005 at Yahoo. It allows for the distributed processing of large datasets across clusters of computers using simple programming models. Hadoop addresses problems of velocity, volume and variety of big data by distributing storage and processing across clusters of low-cost commodity hardware. It provides reliable storage and processing of large datasets through its Hadoop Distributed File System and MapReduce programming model.

1. APACHE HADOOP
All about it in a nutshell…………...

2. ★ Hadoop was created by Doug Cutting and Mike Cafarella in 2005.
★ Cutting, who was working at Yahoo! at the time, named it after his son's toy
elephant.
★ It was originally developed to support distribution for the Nutch search
engine project
HISTORY

3. ★ A batch processing framework of tools
★ These tools are to support running applications on Big Data
★ It is open source
★ Distributed under apache licence
★ Solves three big data problems; velocity, volume, variety
★ Traditional approach maps big data to a single high power machine --- single point of
failure/expensive
but Hadoop approach will harness the power of many low power computers into a powerful
one while creating redundancy in its distributed approach.
HADOOP DEFINED

4. ★ MapReduce
○ task and job trackers
★ Filesystem: HDFS
○ name and data nodes
★ Projects:
○ SLAVES -- Computers with a data node and a task tracker
○ MASTER -- A computer with a datanode, tasktracker, name node and job tracker
○ Project tools:
■ Hive, Hbase, Mahout, Pig, Oozie, Flume, Scoop
ARCHITECTURE

6. ★ Job tracker gets job :
○ distributes to task trackers on slaves
○ when job is done, it is assembled back to job tracker on master.
★ Name node indexes which data node has which data
○ for redundancy and fault tolerance, three copies of each data is
maintained on diff data nodes.
○ tables on name node are backed up and there is also a backup
master
HADOOP MAP-REDUCE ENGINE

7. ★ On which data node the file is located
★ What if a node fails?
★ How to share tasks among data nodes
★ Scalability-1 to 1000 clusters
★ Scalability cost is linear. i.e. The bigger the cluster, the higher the
processing power (x = number of PCs, y = processing speed)
WHAT HADOOP DOESN'T WANT US TO BE
WORRIED ABOUT

8. ★ Yahoo
★ Facebook
★ Amazon
★ Ebay
★ American airlines
★ The New York times
★ Federal reserve board
★ Chevron
★ IBM
★ Who’s next ?? DreamOval Products?? DreamOval business ?? could even be
you!
HADOOP BENEFICIARIES

9. ★ Adverts: mining of users behaviour to generate
recommendation
★ Searches: group related documents
★ Security: search for uncommon patterns, AML, fraud
etc...
HADOOP APPLICATIONS

10. ★ Admins: install, manage, maintain
★ Users: Designing applications, import and export data,
work with tools
….. EVERY DOer COULD BE A USER :)
HADOOP USERS

11. By 2015, 50% of enterprise data will be
processed by Hadoop...
YAHOO PREDICTING HADOOP'S FUTURE

12. 1. Standalone mode: all hadoop daemons on one PC and on one Java Virtual
Machine process
2. Pseudo distributed mode: all hadoop daemons on one PC and on different
Java Virtual Machine processes
3. Fully distributed mode: all hadoop daemons on diff PCs and on different
Java Virtual Machine processes
HADOOP INSTALLATION TYPES

13. ★ An ssh server for master slave communication
★ Java 6 or greater
★ Download and install hadoop
★ Add path to .bashrc
★ Configure hadoop environment
○ Edit env.sh-- Java_home TO RIGHT PATH, disable ipv6
○ Configure .xml (core-site- configure name nodes and ports; mapped-site- configure job
trackers and ports)
★ Launch Hadoop daemons
INSTALLATION OVERVIEW