Hadoop is an open-source software framework for distributed storage and processing of large datasets across clusters of commodity servers. It was designed to scale up from single servers to thousands of machines, with very high fault tolerance. Hadoop features two main components - the Hadoop Distributed File System (HDFS) for storage, and MapReduce for distributed processing of large datasets in a parallel and distributed manner. Hadoop saw widespread adoption for applications such as log analysis, data mining, and large-scale graph processing.

1. Introduction to Apache Hadoop Steve Watt - IBM Big Data Lead @wattsteve #datadayaustin http://stevewatt.blogspot.com

2. The Origins of Hadoop

5. So what exactly is Apache Hadoop ? It is a cluster technology with a single master and multiple slaves, designed for commodity hardware It consists of two runtimes, the Hadoop distributed file system ( HDFS ) and Map/Reduce As data is copied onto the HDFS, it ensures the data is blocked and replicated to other machines (node) to provide redundancy Self contained jobs are written in Map/Reduce and submitted to the cluster. The jobs run in parallel on each of the machines in the cluster, processing the data on the local machine ( data locality ). Hadoop may execute or re-execute a job on any node in the cluster. Node failures are automatically handled by the framework.

6. Hadoop – The Hadoop Cluster - Distributed File System - Map/Reduce

9. Hadoop - Map/Reduce on the Cluster

10. Hadoop - Map/Reduce Logical Flow

11. Hadoop – Map/Reduce – JobTracker Details

12. Hadoop – Map/Reduce – Job Details

14. The Hadoop Ecosystem ClusterChef / Apache Whirr Hadoop Pig / WuKong Cassandra / HBase Offline Systems (Analytics) Online Systems (OLTP @ Scale) BigSheets / DataMeer Hive Provisioning Nutch / SQOOP / Flume Scripting DBA Non-Programmer Load Tooling https://github.com/tomwhite/hadoop-ecosystem/raw/master/hadoop-ecosystem.dot.png