Hadoop: Distributed Data Processing

Outline
Scaling for Large Data Processing
What is Hadoop?
HDFS and MapReduce
Hadoop Ecosystem
Hadoop vsRDBMSes
Conclusion

Current Storage Systems Can’t Compute
Ad hoc Queries &
Data Mining
Interactive Apps
RDBMS (200GB/day)
ETL Grid
Non-Consumption
Filer heads are a bottleneck
Storage Farm for Unstructured Data (20TB/day)
Mostly Append
Collection
Instrumentation

The Solution: A Store-Compute Grid
Interactive Apps
“Batch” Apps
RDBMS
Ad hoc Queries
& Data Mining
ETL and Aggregations
Storage + Computation
Mostly Append
Collection
Instrumentation

What is Hadoop?
A scalable fault-tolerant grid operating system for data storage and processing
Its scalability comes from the marriage of:
HDFS: Self-Healing High-Bandwidth Clustered Storage
MapReduce: Fault-Tolerant Distributed Processing
Operates on unstructured and structured data
A large and active ecosystem (many developers and additions like HBase, Hive, Pig, …)
Open source under the friendly Apache License
http://wiki.apache.org/hadoop/

Hadoop History
2002-2004: Doug Cutting and Mike Cafarella started working on Nutch
2003-2004: Google publishes GFS and MapReduce papers
2004: Cutting adds DFS & MapReduce support to Nutch
2006: Yahoo! hires Cutting, Hadoop spins out of Nutch
2007: NY Times converts 4TB of archives over 100 EC2s
2008: Web-scale deployments at Y!, Facebook, Last.fm
April 2008: Yahoo does fastest sort of a TB, 3.5mins over 910 nodes
May 2009:
Yahoo does fastest sort of a TB, 62secs over 1460 nodes
Yahoo sorts a PB in 16.25hours over 3658 nodes
June 2009, Oct 2009: Hadoop Summit (750), Hadoop World (500)
September 2009: Doug Cutting joins Cloudera

Hadoop Design Axioms
System Shall Manage and Heal Itself
Performance Shall Scale Linearly
Compute Should Move to Data
Simple Core, Modular and Extensible

HDFS: Hadoop Distributed File System
Block Size = 64MB
Replication Factor = 3
Cost/GB is a few ¢/month vs $/month

MapReduce: Distributed Processing

MapReduce Example for Word Count
SELECT word, COUNT(1) FROM docs GROUP BY word;
cat *.txt | mapper.pl | sort | reducer.pl > out.txt
(docid, text)
(words, counts)
Map 1
(sorted words, counts)
Reduce 1
Output File 1
(sorted words, sum of counts)
Split 1
Be, 5
“To Be Or Not To Be?”
Be, 30
Be, 12
Reduce i
Output File i
(docid, text)
Map i
Split i
Be, 7
Be, 6
Shuffle
Reduce R
Output File R
(docid, text)
Map M
(sorted words, counts)
(words, counts)
Split N

Hadoop High-Level Architecture
Hadoop Client
Contacts Name Node for data
or Job Tracker to submit jobs
Name Node
Maintains mapping of file blocks
to data node slaves
Job Tracker
Schedules jobs across
task tracker slaves
Data Node
Stores and serves blocks of data
Task Tracker
Runs tasks (work units)
within a job
Share Physical Node

Apache Hadoop Ecosystem
BI Reporting
ETL Tools
RDBMS
Hive (SQL)
Sqoop
Pig (Data Flow)
MapReduce (Job Scheduling/Execution System)
(Streaming/Pipes APIs)
HBase(key-value store)
Avro (Serialization)
Zookeepr (Coordination)
HDFS(Hadoop Distributed File System)

Relational Databases:
Hadoop:
Use The Right Tool For The Right Job
When to use?
Affordable Storage/Compute
Structured or Not (Agility)
Resilient Auto Scalability
When to use?
Interactive Reporting (<1sec)
Multistep Transactions
Interoperability

Economics of Hadoop
Typical Hardware:
Two Quad Core Nehalems
24GB RAM
12 * 1TB SATA disks (JBOD mode, no need for RAID)
1 Gigabit Ethernet card
Cost/node: $5K/node
Effective HDFS Space:
¼ reserved for temp shuffle space, which leaves 9TB/node
3 way replication leads to 3TB effective HDFS space/node
But assuming 7x compression that becomes ~ 20TB/node
Effective Cost per user TB: $250/TB
Other solutions cost in the range of $5K to $100K per user TB

Sample Talks from Hadoop World ‘09
VISA: Large Scale Transaction Analysis
JP Morgan Chase: Data Processing for Financial Services
China Mobile: Data Mining Platform for Telecom Industry
Rackspace: Cross Data Center Log Processing
Booz Allen Hamilton: Protein Alignment using Hadoop
eHarmony: Matchmaking in the Hadoop Cloud
General Sentiment: Understanding Natural Language
Yahoo!: Social Graph Analysis
Visible Technologies: Real-Time Business Intelligence
Facebook: Rethinking the Data Warehouse with Hadoop and Hive
Slides and Videos at http://www.cloudera.com/hadoop-world-nyc

Cloudera Desktop

Conclusion
Hadoop is a data grid operating system which provides an economically scalable solution for storing and processing large amounts of unstructured or structured data over long periods of time.

Contact Information
AmrAwadallah
CTO, Cloudera Inc.
aaa@cloudera.com
http://twitter.com/awadallah
Online Training Videos and Info:
http://cloudera.com/hadoop-training
http://cloudera.com/blog
http://twitter.com/cloudera

http://www.slideshare.net	65
http://arundudee.blogspot.com	13
http://arundudee.blogspot.in	13

Hadoop: Distributed Data Processing

by Cloudera, Inc. on Dec 11, 2009

Accessibility

Categories

Upload Details

Usage Rights

3 Embeds 91

Statistics

Hadoop: Distributed Data Processing Presentation Transcript