This document presents a MapReduce-based programming model for self-maintaining aggregate views. It begins with motivation for incremental processing of aggregate views to avoid recomputing from scratch. It then provides background on related work in incremental MapReduce processing. The core functionality allows for distributed computation using MapReduce with incremental, overwrite, and full recomputation jobs. Evaluation shows incremental processing is faster than full recomputation when changes are small.

1. Ein MapReduce-basiertes
Programmiermodell für
selbstwartbare Aggregatsichten
2012-08-31
Johannes Schildgen
TU Kaiserslautern
schildgen@cs.uni-kl.de

2. Motivation

3. 11 26 14 23 37
39 41 26
19 8
25
19 22 15 18 10 16
27
8 9 12 14 15

4. Kinderriegel: 26
Balisto: 39
Hanuta: 14
Snickers: 19
Ritter Sport: 41
Pickup: 12
…

5. Kinderriegel: 26
Balisto: 39
Hanuta: 14
Snickers: 19
Ritter Sport: 41
Pickup: 12
…

6. Kinderriegel: 26
Balisto: 39
Hanuta: 14
Snickers: 19
Ritter Sport: 41
Δ
Balisto: -1
Pickup: 12
…

7. Kinderriegel: 26
Balisto: 39
Hanuta: 14
Snickers: 19
Ritter Sport: 41
Δ
Balisto: -8
Pickup: 12 Snickers: +24
… Ritter Sport: -7

8. Increment Installation
Kinderriegel: 26
Balisto: 31
Hanuta: 14
Snickers: 43
Ritter Sport: 34
Δ
Balisto: -8
Pickup: 12 Snickers: +24
… Ritter Sport: -7

9. Overwrite Installation
Kinderriegel: 26
Balisto: 39
Hanuta: 14
Snickers: 19
Δ
Balisto: -8
Ritter Sport: 41Kinderriegel: 26
Pickup: 12 Balisto: 31 Snickers: +24
Hanuta: 14 Ritter Sport: -7
… Snickers: 43
Ritter Sport: 34
Pickup: 12

12. Grundlagen &
Das Marimba-Framework Evaluation
Vorangegangene Arbeiten

16. public class WordCount extends Configured implements Tool {
public static class WordCountMapper extends
Mapper<LongWritable, Text, ImmutableBytesWritable,
LongWritable> {
private Text word = new Text();
@Override
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
this.word.set(tokenizer.nextToken());

17. A A E
B E F
C F B
D C D

18. > create 'person', 'default'
> put 'person', 'p27', 'default:vorname', 'Anton'
> put 'person', 'p27', 'default:nachname', 'Schmidt'
> get 'person', 'p27'
COLUMN CELL
default:nachname timestamp=1338991497408, value=Schmidt
default:vorname timestamp=1338991436688, value=Anton
2 row(s) in 0.0640 seconds

19. banane
die
iss
nimm
3
4
1
1
Δ
kaufe 1
schale 1
die 0
schäle 1
banane 1
schmeiß 1
schmeiß -1
weg 1
schale -1
weg -1

20. banane
die
iss
kaufe
nimm
4
4
1
1
1
Δ
kaufe 1
increment()
schale 0 die 0
schäle 1 banane 1
schmeiß 0 schmeiß -1
weg 0 schale -1
weg -1

21. banane
die
iss
nimm
3
4
1
1
Δ
kaufe 1
schale 1 overwrite() die 0
schäle 1
banane 1
schmeiß 1
schmeiß -1
weg 1
schale -1
weg -1

22. banane
die
iss
kaufe
nimm
4
4
1
1
1
Δ
kaufe 1
schale 0 overwrite() die 0
schäle 1 banane 1
schmeiß 0 schmeiß -1
weg 0 schale -1
weg -1

23. void map(key, value) {
if(value is inserted) {
for(word : value.split(" ")) {
write(word, 1);
}
else if(value is deleted) {
for(word : value.split(" ")) {
write(word, -1);
}
}
}

24. void map(key, value) {
if(value is inserted) {
for(word : value.split(" ")) {
write(word, 1);
}
else if(value is deleted) {
for(word : value.split(" ")) {
write(word, -1);
}
}
else { // old result
write(key, value);
}
}

25. Overwrite Installation
void reduce(key, values) {
sum = 0;
for(value : value) {
sum += value;
}
put = new Put(key);
put.add("fam", "col", sum);
context.write(key, put);
}

26. Increment Installation
void reduce(key, values) {
sum = 0;
for(value : value) {
sum += value;
}
inc = new Increment(key);
inc.add("fam", "col", sum);
context.write(key, inc);
}

27. Formalisierung

28. Formalisierung

29. Allgemeiner Mapper

30. Allgemeiner Reducer

31. Grundlagen &
Das Marimba-Framework Evaluation
Vorangegangene Arbeiten

32. Kernfunktionalität:
Verteiltes Rechnen mittels MapReduce
Ich kümmere mich um:
IncDec, Overwrite,
Lesen alter Ergebnisse,
Erzeugung von Inkrementen,…
Ich erkläre dir, wie man
die Eingabedaten liest,
Kernfunktionalität: aggregiert, invertiert und die
Ausgabe schreibt.
Inkrementelles Rechnen

33. public class WordTranslator extends
Translator<LongWritable, Text> {
public void translate(…) {
…
}
IncJob job = new IncJobOverwrite(conf);
job.setTranslatorClass(
WordTranslator.class);
job.setAbelianClass(WordAbelian.class);

34. public class WordAbelian implements
Abelian<WordAbelian> {
WordAbelian invert() { … }
WordAbelian aggregate(WordAbelian
other) { … }
WordAbelian neutral() { … }
boolean isNeutral() { … }
Writable extractKey() { … }
void write(…) { … }
void readFields(…) { … }
}

35. public class WordSerializer
implements Serializer<WordAbelian> {
Writable serialize(Writable key,
WordAbelian v) {
…
}
WordAbelian deserializeHBase(
byte[] rowId, byte[] colFamily,
byte[] qualifier, byte[] value) {
…
}
}

36. How To Write A Marimba-Job
1. Abelian-Klasse
2. Translator-Klasse
3. Serializer-Klasse
4. Hadoop-Job schreiben unter
Verwendung der Klasse IncJob

37. Implementierung
setInputTable(…)
IncJob setOutputTable(…)
IncJobFull
IncJobIncDec IncJobOverwrite
Recomputation
setResultInputTable(…)

38. NeutralOutputStrategy
(bei IncJobOverwrite)

39. public interface Abelian<T extends
Abelian<?>> extends
WritableComparable<BinaryComparable>{
T invert();
T aggregate(T other);
T neutral();
boolean isNeutral();
Writable extractKey();
void write(…);
void readFields(…);
}

40. public interface Serialzer<T extends
Abelian<?>> {
Writable serialize(T obj);
T deserializeHBase(
byte[] rowId, byte[] colFamily,
byte[] qualifier, byte[] value);
}

41. public abstract class Translator
<KEYIN, VALUEIN> {
public abstract void translate
(KEYIN key, VALUEIN value,
Context context);
this.mapContext.write(
abelianValue.extractKey(),
this.invertValue ?
abelianValue.invert() :
abelianValue);

42. GenericMapper
Eingabeformat liefert: Value
OverwriteResult InsertedValue DeletedValue PreservedValue
deserialisieren weitergeben setze invertValue=true; ignorieren
weitergeben

43. GenericReducer
1. Aggregieren
2. Serialisieren
3. Schreiben
IncDec: Overwrite:
PUT → schreiben
putToIncrement(…) IGNORE → nicht schreiben
DELETE → putToDelete(...)

44. GenericCombiner
„Write A Combiner“
-- 7 Tips for Improving MapReduce Performance, (Tipp 4)
1. Aggregieren

45. TextWindowInputFormat

46. Beispielanwendungen:
1. WordCount
void translate(key, value) {
WordAbelian invert() {
return new WordAbelian(
for(word : value.split(" ")) { this.word,
write( -1 * this.count);
new WordAbelian(word, 1)); }
}
} WordAbelian aggregate(
WordAbelian other) {
return new WordAbelian(
Writable serialize( this.word,
WordAbelian w) { this.count
Put p = new Put( + other.count);
w.getWord()); }
p.add(…);
return p; boolean neutral() {
} return new WordAbelian(
this.word, 0);
}
boolean isNeutral() {
Translator }
return (this.count == 0);
Serializer WordAbelian

47. Beispielanwendungen:
2. Friends Of Friends
FRIENDS
A
D
B FRIENDS OF FRIENDS
C
E

48. Beispielanwendungen:
2. Friends Of Friends
translate(person, friends):
aggregate(…):
Menge der Freundes-
freunde mischen

49. Beispielanwendungen:
3. Reverse WebLink-Graph
REVERSE WEB LINK GRAPH
(Row-ID -> Columns)
Google -> {eBay, Wikipedia}
aggregate(…): -> {Google, Wikipedia}
eBay
Menge der LinksMensa-KL -> {Google}
mischen
Facebook -> {Google, Mensa-
KL, Uni-KL}
Wikipedia -> {Google}
Uni-KL -> {Google, Wikipedia}

50. Beispielanwendungen:
4. Bigrams
Hi, kannst du mich ___?___ am
Bahnhof abholen? So in etwa
10 ___?___. Viele liebe ___?__.
P.S. Ich habe viel ___?___.

51. Idee:
Große Mengen von
Text mit MapReduce
analysieren.

54. Beispielanwendungen:
4. Bigrams
extractKey()
a invert():
count*=-1
b NGramAbelian
count
aggregate(… other):
write(…) count+=other.count
neutral():
isNeutral(): count=0
count==0
NGramStep2Abelian

55. Beispielanwendungen:
4. Bigrams
extractKey()
a invert():
count*=-1
b NGramAbelian
count
aggregate(… other):
write(…) count+=other.count
neutral():
isNeutral(): count=0
count==0
NGramStep2Abelian

56. „Woher Daten
nehmen?“

57. bitte
Hi, kannst du mich ___ ___ am
nicht
Bahnhof abholen? So in etwa
<num>Minuten
<num>
Jahre
10 ___ ___. Grüße
dich
Viele liebe ___ __.
zu
Spaß
P.S. Ich habe viel ___ ___.

58. Grundlagen &
Das Marimba-Framework Evaluation
Vorangegangene Arbeiten

59. WordCount
01:10
01:00
00:50
00:40
Zeit [hh:mm]
00:30 FULL
INCDEC
OVERWRITE
00:20
00:10
00:00
0% 10% 20% 30% 40% 50% 60% 70% 80%
Änderungen

60. Reverse Weblink-Graph
02:51
02:41
02:31
02:21
02:11
02:00
01:50
01:40
Zeit [hh:mm]
01:30
FULL
01:20
INCDEC
01:10
OVERWRITE
01:00
00:50
00:40
00:30
00:20
00:10
00:00
0% 10% 20% 30% 40% 50% 60% 70% 80%
Änderungen

61. Fazit
Vollständige Neuberechnung
IncDec / Overwrite

62. Verwendete Grafiken
Folie 5-9: Folie 37-44:
Flammen und Smilie: Microsoft Office 2010 Puzzle: http://www.flickr.com/photos/dps/136565237/
Folie 10: Folie 46 - 48:
Google: http://www.google.de Junge: Microsoft Office 2010
Folie 11: Folie 49:
Amazon: http://www.amazon.de Google: http://www.google.de
eBay: http://www.ebay.de
Folie 12: Mensa-KL: http://www.mensa-kl.de
Hadoop: http://hadoop.apache.org facebook: http://www.facebook.de
Casio Wristwatch: Wikipedia: http://de.wikipedia.org
http://www.flickr.com/photos/andresrueda/3448240252 TU Kaiserslautern: http://www.uni-kl.de
Folie 16: Folie 50-51:
Hadoop: http://hadoop.apache.org Handy: Microsoft Office 2010
Folie 17: Folie 56:
Hadoop: http://hadoop.apache.org Wikipedia: http://de.wikipedia.org
Notebook: Microsoft Office 2010 Twitter: http://www.twitter.com
Folie 18: Folie 57:
HBase: http://hbase.apache.org Handy: Microsoft Office 2010
Folie 31: Folie 58:
Hadoop: http://hadoop.apache.org Hadoop: http://hadoop.apache.org
Casio Wristwatch: Casio Wristwatch: http://www.flickr.com/photos/andresrueda/3448240252
http://www.flickr.com/photos/andresrueda/3448240252
Folie 32:
Gerüst: http://www.flickr.com/photos/michale/94538528/
Hadoop: http://hadoop.apache.org
Junge: Microsoft Office 2010

63. Literaturverzeichnis (1/2)
[0] Johannes Schildgen. Ein MapReduce-basiertes Programmiermodell für selbstwartbare Aggregatsichten.
Masterarbeit, TU Kaiserslautern, August 2012
[1] Apache Hadoop project. http://hadoop.apache.org/.
[2] Virga: Incremental Recomputations in MapReduce. http://wwwlgis.informatik.uni-kl.de/cms/?id=526.
[3] Philippe Adjiman. Hadoop Tutorial Series, Issue #4: To Use Or Not To Use A Combiner,2010.
http://www.philippeadjiman.com/blog/2010/01/14/hadoop-tutorial-series-issue-4-to-use-or-not-to-use-a-combiner/.
[4] Kai Biermann. Big Data: Twitter wird zum Fieberthermometer der Gesellschaft, April 2012.
http://www.zeit.de/digital/internet/2012-04/twitter-krankheiten-nowcast.
[5] Julie Bort. 8 Crazy Things IBM Scientists Have Learned Studying Twitter, January 2012.
http://www.businessinsider.com/8-crazy-things-ibm-scientists-have-learned-studying-twitter-2012-1.
[6] Jeffrey Dean and Sanjay Ghemawat. MapReduce: Simplified Data Processing on Large Clus-ters. OSDI, pages 137–150, 2004.
[7] Lars George. HBase: The Definitive Guide. O’Reilly Media, 1 edition, 2011.
[8] Brown University Data Management Group. A Comparison of Approaches to Large-Scale Data Analysis.
http://database.cs.brown.edu/projects/mapreduce-vs-dbms/.
[9] Ricky Ho. Map/Reduce to recommend people connection, August 2010.
http://horicky.blogspot.de/2010/08/mapreduce-to-recommend-people.html.
[10] Yong Hu. Efficiently Extracting Change Data from HBase. April 2012.
[11] Thomas Jörg, Roya Parvizi, Hu Yong, and Stefan Dessloch. Can mapreduce learnform materialized views?
In LADIS 2011, pages 1 – 5, 9 2011.
[12] Thomas Jörg, Roya Parvizi, Hu Yong, and Stefan Dessloch. Incremental recomputations in mapreduce. In CloudDB 2011, 10 2011.
[13] Steve Krenzel. MapReduce: Finding Friends, 2010. http://stevekrenzel.com/finding-friends-with-mapreduce.
[14] Todd Lipcon. 7 Tips for Improving MapReduce Performance, 2009.http://www.cloudera.com/blog/2009/12/7-tips-for-improving-
mapreduce-performance/.

64. Literaturverzeichnis (2/2)
[15] TouchType Ltd. SwiftKey X - Android Apps auf Google Play, February 2012.
http://play.google.com/store/apps/details?id=com.touchtype.swiftkey.
[16] Karl H. Marbaise. Hadoop - Think Large!, 2011. http://www.soebes.de/files/RuhrJUGEssenHadoop-20110217.pdf.
[17] Arnab Nandi, Cong Yu, Philip Bohannon, and Raghu Ramakrishnan. Distributed Cube Materia-lization on Holistic Measures.
ICDE, pages 183–194, 2011.
[18] Alexander Neumann. Studie: Hadoop wird ähnlich erfolgreich wie Linux, Mai 2012.
http://heise.de/-1569837.
[19] Owen O’Malley, Jack Hebert, Lohit Vijayarenu, and Amar Kamat. Partitioning your job into maps and reduces, September 2009.
http://wiki.apache.org/hadoop/HowManyMapsAndReduces?action=recall&#38;rev=7.
[20] Roya Parvizi. Inkrementelle Neuberechnungen mit MapReduce. Bachelorarbeit, TU Kaiserslautern,
Juni 2011.
[21] Arnd Poetzsch-Heffter. Konzepte objektorientierter Programmierung. eXamen.press.
Springer Berlin Heidelberg, Berlin, Heidelberg, 2009.
[22] Dave Rosenberg. Hadoop, the elephant in the enterprise, June 2012.
http://news.cnet.com/8301-1001 3-57452061-92/hadoop-the-elephant-in-the-enterprise/.
[23] Marc Schäfer. Inkrementelle Wartung von Data Cubes. Bachelorarbeit, TU Kaiserslautern, Januar 2012.
[24] Sanjay Sharma. Advanced Hadoop Tuning and Optimizations, 2009.
http://www.slideshare.net/ImpetusInfo/ppt-on-advanced-hadoop-tuning-n-optimisation.
[25] Jason Venner. Pro Hadoop. Apress, Berkeley, CA, 2009.
[26] DickWeisinger. Big Data: Think of NoSQL As Complementary to Traditional RDBMS, Juni 2012.
http://www.formtek.com/blog/?p=3032.
[27] Tom White. 10 MapReduce Tips, May 2009. http://www.cloudera.com/blog/2009/05/10-mapreduce-tips/.60