The document discusses MapReduce frameworks and big data lakes. It provides examples of MapReduce operations that take key-value pairs as input, apply mapping and combining, and output key-value pairs. Code snippets show MapReduce jobs written in Python and Java that are run on Hadoop distributed file systems.

1. bigfish.
bigdata lake.
MapReduce Framework
<key, value>
(input) <k1, v1> -> map -> <k2, v2> -> combine-> <k2, v2> -> reduce-> <k3, v3> (output)
// Somedata
private int counter;
privatelong timestamp;
public void write(DataOutput out)
throws IOException {
out.writeInt(counter);
out.writeLong(timestamp);
{
{

2. _
_
Plugins
input of first map:
output of first map:
< Hello, 1>
< World, 1>
< Bye, 1>
< World, 1>
_
< Bye, 1>
< Hello, 1>
< World, 2>
_
conf configuration_file
Dproperty=value
fs host:port or local
_
_
files_
archives
lessthan
0.5%
of that data
will be analyzed
andused
murkywater
1.7 MB
of newinfo
will be created
everysecondfor
everyhumanbeing
onthe planet
by2020

3. flexingyour gills
ina bigdata lake
Collect
void collect(K key,
V value)
throws IOException
_ Parameters:
key - the key to collect.
value - to value to collect.
_
Throws:
IOException
_

4. #!/ usr/bin/python
import sys;
def generateLongCountToken(id):
return"LongValueSum:" + id+ "t" + "1"
def main(argv):
line= sys.stdin.readline();
try:
whileline:
line=line[:-1];
fields = line.split("t");
print generateLongCountToken(fields[0]);
line= sys.stdin.readline();
except "endof file":
returnNone
if __name__=="__main__":
main(sys.argv)
swimfast-to-wow

5. $ bin/ hadoop dfs -ls /usr/joe/wordcount/input/
/usr/joe/wordcount/input/file01
/usr/joe/wordcount/input/file02
$ bin/ hadoop dfs -cat / usr/joe/wordcount/input/
file01
HelloWorld ByeWorld
$ bin/ hadoop dfs -cat / usr/joe/wordcount/input/
file02
Hello Hadoop Goodbye Hadoop
$ mkdir wordcount_classes
$ javac -classpath ${HADOOP_HOME}/ hadoop-${HADOOP_VERSION}-core.jar -d wordcount_classes
WordCount.java
$ jar -cvf /usr/joe/wordcount.jar -Cwordcount_classes//
scale for digitization

6. get reel protective
bin/ hadoop pipes
[-input inputDir]
[-output outputDir]
[-jar applicationJarFile]
[-inputformat class]
[-mapclass]
[-partitioner class]
[-reduceclass]
[-writer class]
[-programprogram url]
[-conf configurationfile]
[-Dproperty=value]
[-fs local|namenode:port ]
[-jt local|jobtracker:port ]
[-files commaseparatedlist of files]
[-libjars commaseparatedlist of jars]
[-archives commaseparatedlist of archives]
incrCounter
void incrCounter(String group,
String counter ,
long amount)

7. incrCounter
void incrCounter(String group,
String counter ,
long amount)
off-the-hook
competitive advantage
a 10%
increase in
data accessibility
will result inmore than
$65 million
additional
net income
{
BufferedReader fis =new BufferedReader (new
FileReader(patternsFile.toString()));
String pattern=null;
while((pattern= fis.readLine()) !=null) {
patternsToSkip.add(pattern);
}
} catch(IOException ioe) {
System.err.println("Caught exceptionwhileparsing thecachedfile'" +
patternsFile + "' : " + StringUtils.stringifyException(ioe));
}
}
public voidmap(LongWritable key, Text value, OutputCollector
<Text, IntWritable> output, Reporter reporter) throws IOException {
String line=(caseSensitive) ?value.toString() : value.toString()
.toLowerCase(); for (String pattern: patternsToSkip) {
line=line.replaceAll(pattern, "");
}
StringTokenizer tokenizer =new StringTokenizer(line);
}