Tackling Big Data with Hadoop

TACKLING BIG DATA WITH HADOOP David HowellSunday, September 11, 11

WHAT IS BIG DATA?Sunday, September 11, 11

WHAT IS BIG DATA? Google web crawlSunday, September 11, 11

WHAT IS BIG DATA? stream of Twitter messagesSunday, September 11, 11

WHAT IS BIG DATA? Annoying Farmville requests on FacebookSunday, September 11, 11

WHAT IS BIG DATA? terabyte-scale data sets awkward to work with using traditional toolsSunday, September 11, 11

WHAT IS BIG DATA? requires distributed computingSunday, September 11, 11

MEDIUM DATA dozens to hundreds of gigabytes still awkward to work with using traditional toolsSunday, September 11, 11

MAP-REDUCE http://labs.google.com/papers/mapreduce.htmlSunday, September 11, 11

Sunday, September 11, 11

COUNTING AT SCALESunday, September 11, 11

function map_1(t, search_phrase) emit(search_phrase, 1) sort and shufﬂe function reduce_1(search_phrase, counts) total = 0 for count in counts total += count emit(search_phrase, total) function map_2(search_phrase, total) emit(total, search_phrase) sort and shufﬂe function reduce_2(total, search_phrases) for search_phrase in search_phrases emit(search_phrase, total)Sunday, September 11, 11

map shufﬂe reduce cat IN | sort | uniq -c > OUT map shufﬂe reduce awk ‘{print $2,$1}’ OUT | sort > FINALSunday, September 11, 11

WHY BOTHER?Sunday, September 11, 11

HADOOPSunday, September 11, 11

DISTRIBUTED COMPUTING PLATFORMSunday, September 11, 11

TOOLS IN THE PLATFORM Map-Reduce APIs Higher Level APIs •Java •Hive •C++ •Cascading •UNIX pipes •PigSunday, September 11, 11

THE ORIGIN STORYSunday, September 11, 11

WHO’S USING IT?Sunday, September 11, 11

HADOOP How does it work?Sunday, September 11, 11

Sunday, September 11, 11

DEMO!Sunday, September 11, 11

YOUR DATA PLATFORM ad hoc unstructured prototyping experiment data-driven curiosity playSunday, September 11, 11

LEARN MORE http://hadoop.apache.org/ http://www.cloudera.com/ Hadoop: The Deﬁnitive Guide @dehowell dave@poorlytrainedape.com http://github.com/dehowell/hadoop-crypto-demoSunday, September 11, 11

http://www.poorlytrainedape.com	248
http://poorlytrainedape.squarespace.com	32
http://paper.li	12

Tackling Big Data with Hadoop

by poorlytrainedape on Sep 11, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

3 Embeds 292

Statistics

Tackling Big Data with Hadoop — Presentation Transcript