Mastering Hadoop Map Reduce was a presentation I gave to Orlando Data Science on April 23, 2015. The presentation provides a clear overview of how Hadoop Map Reduce works, and then dives into more advanced topics of how to optimize runtime performance and implement custom data types. The examples are written in Python and Java, and the presentation walks through how to create an n-gram count map reduce program using custom data types. You can get the full source code for the examples on my Github! http://www.github.com/scottcrespo/ngrams

1. Mastering Map Reduce
Scott Crespo

2. Path to Success
Map Reduce Refresher
Optimization Strategies
CustomType Example
Applications

3. What’s Hadoop?
A framework that facilitates data flow through a cluster of servers

4. What’s Map Reduce?
 A paradigm for analyzing distributed data sets
Raw Data ( K, [V1..Vn] )(K,V)

5. What About Hive And Pig?
Use them whenever possible!

6. Data States in Map Reduce (Letter Count)
HelloWorld
Hello
World
H,1
E,1
L,1
L,1
O,1
W,1
O,1
R,1
L,1
D,1
H,[1]
E,[1]
L,[1,1,1]
O,[1,1]
W,[1]
R,[1]
D,[1]
H,1
E,1
L,3
O,2
W,1
R,1
D,1
Split
Map Partition/Shuffle
Reduce

7. Basic Map Reduce Program Structure
MyMapReduceProgram {
MyMapperClass extends Mapper {
map() {
// map code
}
}
MyReducerClass extends Reducer {
reduce() {
//reduce code
}
}
main() {
//driver code
}
}

8. Advanced Optimizations
 Drivers
 CustomTypes
 Setup Methods
 Partitioning
 Combiners
 Chaining
 FaultTolerance

9. Generating N-Grams
 N-Gram: Set of all n sequential elements in a set.
Trigram: “The quick brown fox jumps over the lazy dog”
(the quick brown), (quick brown fox), (brown fox jumps),
(fox jumps over), (jumps over the), (the lazy dog)

10. Solution Design
NGramCounter {
NGramMapper {
map() {
//Tokenize and Sanitize Inputs
// Create NGram
// Output (NGram ngram, Int count)
}
}
NGramCombiner {
combine() {
// Sum local NGrams counts that are of the same key
// Output (NGram ngram, Int Count)
}
}
NGramReducer {
reduce() {
// Sum Ngrams counts of the same key
// Output (NGram ngram, Int Count)
}
}
}
CustomType!

11. Work Flow
 Prototype (Python)
 CustomType (Trigram)
 UnitTests
 Mapper
 Reducer

12. Prototype
Quick and Dirty Python

13. Prototype
def test_mapper():
lines = [“the quick brown fox jumped over the lazy dog", "the quick brown”]
for line in lines:
words = line.split()
length = len(words)
sys.stdout.write("nLength of %d n-------------------n" % length)
i = 0
while (i+2 < length):
first = words[i]
second = words [i+1]
third = words[i+2]
trigram = "%s %s %s n" % (first, second, third)
sys.stdout.write(trigram)
i += 1

14. Output
Length of 9
-------------------
the quick brown
quick brown fox
brown fox jumped
fox jumped over
jumped over the
over the lazy
the lazy dog
Length of 3
-------------------
the quick brown

15. Custom DataTypes

16. Custom KeyTypes
Must implement Hadoops WritableComparable interface
 Writable:The key can be serialized and transmitted across a
network
 Comparable:The key can be compared to other keys &
combined/sorted for the reduce phase
write() readFields() compareTo() hashCode()
toString() equals()

17. Trigram.java
public class Trigram implements WritableComparable<Trigram> {
…
public int compareTo(Trigram other) {
int compared = first.compareTo(other.first);
if (compared != 0) {
return compared;
}
compared = second.compareTo(other.second);
if (compared != 0) {
return compared;
}
return third.compareTo(other.third);
}
public int hashCode() {
return first.hashCode()*163 + second.hashCode() + third.hashCode();
}
}

18. Map Reduce Program

19. TrigramMapper
public static class TrigramMapper
extends Mapper<Object, Text, Trigram, IntWritable> {
…
public void map(Object key, Text value, Context context) {
String line = value.toString().toLowerCase(); // create string and lower case
line = line.replaceAll("[^a-zs]",""); // remove bad non-word chars
String[] words = line.split("s"); // split line into list of words
int len = words.length; // need the length for our loop condition
for(int i = 0; i+2 < len; i++) {
if(len <= 1) { continue; } // remove short lines
first.set(words[i]);
second.set(words[i+1]);
third.set(words[i+2]);
trigram.set(first, second, third);
context.write(trigram, one);

20. TrigramReducer
public static class TrigramReducer
extends Reducer<Trigram, IntWritable, Trigram, IntWritable> {
private IntWritable result = new IntWritable();
public void reduce(Trigram key, Iterable<IntWritable> values, Context context ) {
int sum = 0;
for(IntWritable value : values) {
sum += value.get();
}
result.set(sum);
context.write(key, result);
…

21. Driver
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "Trigram Count");
job.setJarByClass(TrigramCount.class);
job.setMapperClass(TrigramMapper.class);
job.setMapOutputKeyClass(Trigram.class);
job.setMapOutputValueClass(IntWritable.class);
job.setReducerClass(TrigramReducer.class);
job.setCombinerClass(TrigramReducer.class);
job.setOutputKeyClass(Trigram.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}

22. Applications

23. Speech Recognition
(Trigram1, 90)
(Trigram2, 76)
(Trigram3, 8)
(Trigram4, 1)

24. Other Applications
 Blog Posts
 Stocks
 GIS Coordinates
Any object with multiple attributes!

25. Stock
Attributes
Text timeStamp;
Text ticker;
Float price;

26. Conclusion
Custom DataTypes Can:
 Improve Runtime Performance
 Result in Reusable Code
 Provide a Consistent Interface

27. ThankYou!
Scott Crespo
scott@orlandods.com