Intro to Apache Hadoop


Published on

A presentation I compiled for a weekly brown bag session held at EAI Technologies.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Intro to Apache Hadoop

  1. 1. Intro to Apache™ Hadoop® A Brown Bag Session at EAI Technologies by Sufi Nawaz
  2. 2. What is this Hadoop you speak of? "Apache Hadoop is an open- source software framework that supports data-intensive distributed applications, licensed under the Apache v2 license. It supports the running of applications on large clusters of commodity hardware." - Wikipedia Doug Cutting (Creator)
  3. 3. More about Hadoop ● It is a highly scalable, fault tolerant and distributed compute and storage platform. ● Based on Google GFS and MapReduce. ● Brings computation to data and not the other way around. ● Created by Doug Cutting and Mike Cafarella in 2005. ● Originally developed to support distribution for the Nutch search engine project.
  4. 4. Why use Hadoop? ● Process lots of data - in petabytes even ● Distributed processing ● Uses simple programming models ● Scalable - add new nodes simply ● Cost effective - uses commodity hardware ● Flexible - Hadoop is schema-less and can absorb any kind of data ● Fault tolerant - redistribution of failed jobs and data recovery by data replication
  5. 5. When to use Hadoop and not? Good for: ● Indexing Data ● Log Analysis ● Image Manipulation ● Sorting Large Scale Data ● Data Mining Bad for: ● For real time processing ● For processing intensive tasks with little data
  6. 6. Hadoop Modules - Hadoop Common - Hadoop Distributed File System (HDFS) - Hadoop YARN - Hadoop MapReduce
  7. 7. Hadoop Distributed File System (HDFS)
  8. 8. Hadoop Distributed File System The Apache HDFS is the primary distributed storage component used by applications under Apache Hadoop project. Apache HDFS can serve as a stand-alone distributed file system as well.
  9. 9. Hadoop Distributed File System A single Namenode maintains the directory tree and manages the namespace and access to files by clients. It holds Metadata for list of files, blocks, datanodes all in memory. Datanodes store and manage the data blocks as local files on servers throughout the rest of the cluster. Reports to Namenode with heartbeat.
  10. 10. Hadoop Distributed File System
  11. 11. Hadoop Distributed File System What is HDFS bad for? ● Low latency data access. It trades low latency to increase the throughput of the data. ● Lots of small files, since default block size is 64MB. Will increase memory requirements of namenode. ● Multiple writers and arbitrary modification.
  12. 12. Hadoop Distributed File System Anatomy of write ● DFSOutputStream splits data into packets. ● Writes into an internal queue. ● DataStreamer asks namenode to get list of datanodes and uses the internal data queue. ● Namenode gives a list of datanodes for the pipeline. ● Maintains internal queue of packets waiting to be acknowledged.
  13. 13. Hadoop Distributed File System Anatomy of read: ● Namenode returns locations of blocks. ● Datanode list is sorted according to their proximity to the client. ● FSDataInputStream wraps DFSInputStream, which manages datanode and namenode I/O. ● Read is called repeatedly on the datanode till end of the block is reached. ● Finds the next DataNode for next data block. ● All happens transparently to the client. ● Calls close after finishing reading the data.
  14. 14. Hadoop Distributed File System Accessibility ● DFS Shell ● DFS Admin ● Browser Interface ● Mountable HDFS
  15. 15. MapReduce
  16. 16. MapReduce
  17. 17. MapReduce Main Components ● JobClient ● JobTracker ● TaskTracker
  18. 18. MapReduce JobTracker (Master) ● Single Job Tracker per cluster ● Schedule Map and Reduce Tasks for TaskTrackers ● Monitors Tasks and keeps track of TaskTrackers status ● Re-execute tasks on failure TaskTracker (Slave) ● Single TaskTrackers per node (multiple in a cluster) ● Run Map and Reduce Tasks
  19. 19. Who uses Hadoop? ● Yahoo! ○ Support research for Ad Systems and Web Search ● Facebook ○ 2 major clusters (1100 + 300 machines w/ 8 cores) ○ Heavy users of both streaming and Java APIs. ○ Have developed a FUSE implementation on HDFS. ● EBay ○ 532 nodes cluster (8 * 532 cores, 5.3PB ● Hulu ○ 13 machine cluster (8 cores/machine, 4TB/machine) ○ Log storage and analysis ● Many more ○
  20. 20. Where can I find resources? ● Hadoop Docs ○ ● Mailing List: ○ ● White papers from Cloudera, Intel, Dell, etc. ● Hadoop in 20 Pages (http://blog.imaginea. com/hadoop-a-short-guide/) ● Yahoo! CDN Hadoop Tutorial ● Google Search Engine (!)
  21. 21. Some Additional Info ● Hadoop Streaming ○ Run MapReduce with any language supporting standard I/O e.g. ruby, python. ● Hadoop Distributed Cache ○ Puts contents of specified input path to memory in all datanodes across cluster. ● Hadoop Security ○ Secure Hadoop with Kerberos ● Hadoop Federation ○ Solution for NameNode High Availability (HA) and no Single Point of Failure of NameNode