Your SlideShare is downloading. ×
Hadoop
Hadoop
Hadoop
Hadoop
Hadoop
Hadoop
Hadoop
Hadoop
Hadoop
Hadoop
Hadoop
Hadoop
Hadoop
Hadoop
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Hadoop

424

Published on

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
424
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Ali Bahu10/17/2012APACHE HADOOP
  • 2. INTRODUCTION Apache Hadoop is an open-source softwareframework that supports data-intensivedistributed applications, licensed under theApache v2 license. It enables applications towork with thousands of computation-independent computers and petabytes ofdata. Hadoop was derived from GooglesMapReduce and Google File System (GFS)papers. Hadoop is implemented in Java.
  • 3. WHY HADOOP Need to process Multi Petabyte Datasets It is expensive to build reliability in each application Nodes failure is expected and Hadoop can help Number of nodes is not constant Efficient, reliable, Open Source Apache License Workloads are IO bound and not CPU bound
  • 4. WHO USES HADOOP Amazon/A9 Facebook Google IBM Joost Last.fm New York Times PowerSet Veoh Yahoo!
  • 5. COMMODITY HARDWARE Typically in 2 level architecture Nodes are commodity PCs 30-40 nodes/rack Uplink from rack is 3-4 gigabit Rack-internal is one gigabit
  • 6. HDFS ARCHITECTURE
  • 7. HDFS (HADOOP DISTRIBUTED FILESYSTEM) Very Large Distributed File System 10K nodes, 100 million files, 10 PB Assumes Commodity Hardware Files are replicated to handle hardware failure Detect failures and recovers from them Optimized for Batch Processing Data locations exposed so that computations canmove to where data resides Provides very high aggregate bandwidth User Space, runs on heterogeneous OS Single Namespace for entire cluster Data Coherency Write-once-read-many access model Client can only append to existing files Files are broken up into blocks Typically 128 MB block size Each block replicated on multiple Data Nodes Intelligent Client Client can find location of blocks Client accesses data directly from Data Node
  • 8. NAME NODE METADATA Meta-data in Memory The entire metadata is in main memory No demand paging of meta-data Types of Metadata List of files List of Blocks for each file List of Data Nodes for each block File attributes, e.g. creation time, replication factor, etc. Transaction Log Records file creations, file deletions, etc. is kept here
  • 9. DATA NODEBlock Server Stores data in the local file system (e.g. ext3) Stores meta-data of a block (e.g. CRC) Serves data and meta-data to ClientsBlock Report Periodically sends a report of all existing blocks tothe Name NodeFacilitates Pipelining of Data Forwards data to other specified Data Nodes
  • 10. BLOCK PLACEMENT Block Placement Strategy One replica on local node Second replica on a remote rack Third replica on same remote rack Additional replicas are randomly placed Clients read from nearest replica
  • 11. DATA CORRECTNESS Use Checksums to validate data Use CRC32, etc. File Creation Client computes checksum per 512 byte Data Node stores the checksum File access Client retrieves the data and checksum from Data Node If Validation fails, Client tries other replicas
  • 12. NAMENODE FAILURE A single point of failure Transaction Log stored in multiple directories A directory on the local file system A directory on a remote file system (NFS/CIFS)
  • 13. DATA PIPELINING Client retrieves a list of DataNodes on which to place replicas of ablock Client writes block to the first DataNode The first DataNode forwards the data to the next DataNode in thePipeline When all replicas are written, the Client moves on to write the nextblock in file Rebalancer is used to ensure that the % disk full on DataNodes aresimilar Usually run when new DataNodes are added Cluster is online when Rebalancer is active Rebalancer is throttled to avoid network congestion Command line tool
  • 14. HADOOP MAP/REDUCE The Map-Reduce programming model Framework for distributed processing of large data sets Pluggable user code runs in generic framework Common design pattern in data processing cat * | grep | sort | unique -c | cat > file input | map | shuffle | reduce | output Useful for: Log processing Web search indexing Ad-hoc queries, etc.

×