Intro to Hadoop

Intro to Hadoop Eric Wendelin Jacob Rideout @eriwen @jrideout 11.9.2010

About Us • Hadoopers with @returnpath • Worked with Hadoop for about a year • Speaker/Facilitator for Boulder Big Data Meetup • Running 3 Hadoop clusters with 30 TBs of data in production (and growing)

Agenda • Hadoop Overview • Example use cases • How to interact with Hadoop • Hadoop project ecosystem

What is Hadoop? • Fault-tolerant storage for “Big Data” (100s of petabytes and larger installations) • Distributed system to analyze large quantities of data (via MapReduce)

What Hadoop is NOT • Able to query large datasets in real-time* • A system optimized for normalized data * HBase can be used for this, but complex queries are not possible

Hadoop is... HDFS + MapReduce

Anatomy of a MapReduce Job

Hadoop-able Problems • Lots of Complex Data • Multiple Data Sources • Batch Processing • Parallel Execution • Data spread across server cluster • Take the computation to the data

Types of applications • Basic aggregation and reporting • Baby photo processing • Index building • Pattern recognition • Predictive modeling • Collaborative recommendation

Simple Collaborative Recommendation

Modeling the problem • Users • Items (Products) • User preferences

User/Item Matrix Model Joe Bob Sam Ann Rock 3 2 Classical 4 Hip Hop 1 Folk 4 1 Blues 2 5 Indie 5 3

User/Item Matrix Model Joe Bob Sam Ann Rock 3 2 Classical ??? 4 Hip Hop 1 Folk 4 1 Blues 2 5 Indie 5 3 ???

High-level approach • To predict the outcome for an unknown user/item pair • Identify similar items (Items other users rated similarly to a given item) • Identify (and aggregate) user ratings for similar items to a given item

Item Similarity Users who like X also like Y and/or Users who hate A also hate B

Item Similarity Algorithm • Input: ratings, “like”s, or visits per user • Output: item/item similarity

Assumptions • Many users • Large number of items

Similarity MapReduce stage 1 Map Input: [User, Item, Rating] Output: {Key: User,Value: [Item,Rating]} Reduce Input: {Key: User, Value: Iterable<[Item,Rating]>} Output: {Key: User, Value: [[ItemA, RatingA],[ItemB, RatingB]]}

Similarity MapReduce stage 2 Map Input: [User, ItemA, RatingA, ItemB, RatingB] Output: {Key: [ItemA, ItemB], Value: [RatingA, RatingB]} Reduce Input: {Key: [ItemA, ItemB], Value: Iterable<[RatingA, RatingB]>} Output: {Key: [ItemA, ItemB], Value: [(Similarity of ItemA to ItemB)]}

Bulk Recommender MapReduce stage 1 Map Input: [ItemA, ItemB, Similarity] Output: {Key: ItemA,Value: [ItemB, Similarity]} Output: {Key: ItemB,Value: [ItemA, -Similarity]} Reduce Input: {Key: ItemA,Value:[ItemB, Similarity]} Input: {Key: ItemB,Value: [ItemA, -Similarity]} Output: {Key: ItemA, Value: [ItemB/SimilarityB, ItemC/SimilarityC]}

Bulk Recommender MapReduce stage 2 Map Input: [ItemA, ItemB/SimilarityB, ItemC/SimilarityC] Output: {Key: ItemA,Value: [ItemB/SimilarityB, ItemC/SimilarityC]} Input: [User, ItemA/RatingA, ItemB/RatingB] Output: {Key: ItemA,Value: [User/Rating]}

Bulk Recommender MapReduce stage 2 Reduce Input: {Key: ItemA,Value: [ItemB/SimilarityB, ItemC/SimilarityC]} Input: {Key: ItemA,Value: [User1/RatingA]} Input: {Key: ItemA,Value: [User2/RatingA]} Output: {Key: ItemA, Value: {Similarity: [ItemB/SimilarityB, ItemC/SimilarityC], Users: [[User1, RatingA], [User2, RatingA]]} }

Bulk Recommender MapReduce stage 3 Map Input: [ItemA, {Similarities: [ItemB/SimilarityB, ItemC/SimilarityC], Users: [[User1, RatingA], [User2, RatingA]]} }] Output: {Key: User1, Value: [ItemA, User1RatingA, SimilaritiesA]} Output: {Key: User2, Value: [ItemA, User2RatingA, SimilaritiesA]}

Bulk Recommender MapReduce stage 3 Reduce Input: {Key: User1, Value: [ItemA, User1RatingA, SimilaritiesA]} Input: {Key: User2, Value: [ItemA, User2RatingA, SimilaritiesA]} Output: {Key: User1, Value: PREDICTION}

Recommendations • Input into Reducer • Key: User • Values: ItemA/RatingA/SimilaritiesA ItemB/RatingB/SimilaritiesB For all items (i) where we didn’t receive a line: PREDICTION = (RatingA * Similarity_of_A_to_i + RatingB * Similarity_of_B_to_i) / (Sum of all similarities)

...or just use Mahout hadoop org.apache.mahout.cf.taste.hadoop.item.RecommenderJob -i /user/preferences.csv -o /output/dir --similarityClassname SIMILARITY_PEARSON_CORRELATION

Hadoop Distributed File System (HDFS)

HDFS Concepts • Namenodes (Manages ﬁlesystem tree) • Datanodes (Store and retrieve blocks) • Large blocks (64Mb default)

Writing to HDFS

Very large ﬁles Typically 100MB - TB in size

:( Bad: many small ﬁles Good: fewer large ﬁles :)

Built for commodity hardware RAID? Pfff. Let’s just replicate the ﬁle on a bunch of servers and then optimize on data- locality

Bad: My one $100k server died Less bad! Good: one of my 30 $3k servers died

Streaming data access Finding data on disk is slow. Optimize this away by reading large amounts of data at a time

Bad: random ﬁle access or modiﬁcation Good: read all from beginning, append to end

HDFS command-line

hadoop fs -mkdir hdfs:/// data

hadoop fs -mkdir /data

hadoop fs -put file:/// local/dir hdfs:///data/input

hadoop fs -ls /data

hadoop fs -rm /data/ badfile.gz

Interfaces to Hadoop

“Traditional” Java API hadoop jar /path/example.jar WordCount -files /path/words.txt /input/dir /output/dir

Demo github.com/emwendelin/ hadoop-examples

Hadoop Streaming hadoop jar ${HADOOP_HOME}/contrib/ streaming/hadoop-*-streaming.jar -input /logs/access.log -output /my/out/dir -mapper /path/to/mapper.groovy -reducer /path/to/reducer.groovy

Hadoop Pipes hadoop pipes -Dhadoop.pipes.java.recordreader=true -Dhadoop.pipes.java.recordwriter=true -input /path/to/input -output /my/output/dir -program bin/annihilate_spammers

Anatomy of Streaming/ Pipes MapReduce Jobs

Hadoop Subprojects

hive.apache.org • Data warehouse infrastructure that lives on HDFS • SQL-like query language translated into a series of MapReduce jobs

Hive Example CREATE TABLE weblogs (ip STRING, httpcode INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘t’ STORED AS TEXTFILE; SELECT log.ip, count(1) FROM weblogs log WHERE log.httpcode = 404;

pig.apache.org • Pig Latin statements turned into a series of MapReduce jobs under the covers • Shortens the development cycle and raises the level of abstraction for processing large datasets • Uses HDFS for storage

Pig Example records = LOAD ‘/logs/www/access.log’ AS (ip:chararray, httpcode:int); filtered = FILTER records BY httpcode == 404; grouped = GROUP filtered BY ip; bad_req_per_ip = FOREACH grouped GENERATE group, COUNT(filtered.ip); DUMP bad_req_per_ip;

ZooKeeper • Keeps your distributed system highly available • Allows loosely-coupled interactions between nodes that otherwise don’t need to care about each other’s existence

hbase.apache.org • Distributed, column-oriented datastore built on HDFS. • Allows real-time access to data spread across a Hadoop cluster • Dynamic columns, but not dynamic column families

Demo github.com/emwendelin/ hadoop-examples

avro.apache.org • Cross-platform data serialization • Enforces typed data efﬁciently • Allows schema to evolve without breaking!

Flume • Makes loading ﬁles to HDFS much simpler • Great for pulling logging into Hive, HBase, HDFS... • Interfaces with other Hadoop projects

Flume Example tail(“/usr/apache/log/access.log”) | collector(“hdfs:///logs”, “log-”)

Why use Hadoop? • Lower overall cost (commodity hardware) • Flexibility (OSS is a beautiful ecosystem) • Scalability (Scales ~linearly just by adding nodes to cluster)

Boulder/Denver Big Data Meetup http://bit.ly/9rgJGw

References Hadoop: The Deﬁnitive Guide - http://oreil.ly/aa5rLN Pig image - http://www.ﬂickr.com/photos/leedav/ 4033202828/

Thanks in advance for your feedback! spkr8.com/t/5018

Intro to Hadoop

by Eric Wendelin

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 3

Statistics

Intro to Hadoop Presentation Transcript