2. Can I choose how many maps to be executed?
•The number of maps is usually driven by the total size of the inputs, that is, the total number of blocks of the input files.
•The right level of parallelism for maps seems to be around 10-100 maps per-node, although it has been set up to 300 maps for very cpu-light map tasks.
•Task setup takes awhile, so it is best if the maps take at least a minute to execute.
•Thus, if you expect 10TB of input data and have a blocksizeof128MB, you'll end up with 82,000 maps, unlesssetNumMapTasks(int) (which only provides a hint to the framework) is used to set it even higher.
3. Reducer
•Reducerreduces a set of intermediate values which share a key to a smaller set of values.
•The number of reduces for the job is set by the user viaJobConf.setNumReduceTasks(int).
•Overall,Reducerimplementations are passed theJobConffor the job via theJobConfigurable.configure(JobConf)method and can override it to initialize themselves.
•The framework then callsreduce(WritableComparable, Iterator, OutputCollector, Reporter)method for each<key, (list of values)>pair in the grouped inputs.
•Applications can then override theCloseable.close()method to perform any required cleanup.
•Reducerhas 3 primary phases: shuffle, sort and reduce.
4. Reducer: Sort
•The framework groupsReducerinputs by keys (since different mappersmay have output the same key) in this stage.
•The shuffle and sort phases occur simultaneously; while map-outputs are being fetched they are merged.
•Secondary Sort: If equivalence rules for grouping the intermediate keys are required to be different from those for grouping keys before reduction, then one may specify aComparatorvia JobConf.setOutputValueGroupingComparator(Class). SinceJobConf.setOutputKeyComparatorClass(Class)can be used to control how intermediate keys are grouped, these can be used in conjunction to simulatesecondary sort on values.
5. Reducer: Reduce
•In this phase thereduce(WritableComparable, Iterator, OutputCollector, Reporter)method is called for each<key, (list of values)>pair in the grouped inputs.
•The output of the reduce task is typically written to theFileSystemviaOutputCollector.collect(WritableComparable, Writable).
•Applications can use theReporterto report progress, set application-level status messages and updateCounters, or just indicate that they are alive.
•The output of theReducerisnot sorted.
6. How many reducer?
•The right number of reduces seems to be betweeen0.95and1.75multiplied by (<no. of nodes> *mapred.tasktracker.reduce.tasks.maximum).
–With0.95all of the reduces can launch immediately and start transferring map outputs as the maps finish.
–With1.75the faster nodes will finish their first round of reduces and launch a second wave of reduces doing a much better job of load balancing.
•Increasing the number of reduces
–Increases the framework overhead,
–But
•Increases load balancing and
•lowers the cost of failures.
•The scaling factors above are slightly less than whole numbers to reserve a few reduce slots in the framework for speculative-tasks and failed tasks.
•It is legal to set the number of reduce-tasks tozeroif no reduction is desired.
•In this case the outputs of the map-tasks go directly to theFileSystem, into the output path set bysetOutputPath(Path).
•Again, the framework does not sort the map-outputs before writing them out to theFileSystem.
7. Partitioner
•Partitionercontrols the partitioning of the keys of the intermediate map- outputs.
•The key (or a subset of the key) is used to derive the partition, typically by ahash function.
•The total number of partitions is the same as the number of reduce tasks for the job.
•Hence this controls which of themreduce tasks the intermediate key (and hence the record) is sent to for reduction.
8. Reporter
•A facility for MapReduceapplications to report progress, set application- level status messages and updateCounters.
•In scenarios where the application takes a significant amount of time to process individual key/value pairs, this is crucial since the framework might assume that the task has timed-out and kill that task.
9. OutputCollector
•A generalization of the facility provided by the MapReduceframework to collect data output by theMapperor theReducer(either the intermediate outputs or the output of the job).
10. Job Configuration
•JobConfrepresents a MapReducejob configuration.
•JobConfis the primary interface for a user to describe a MapReducejob to the Hadoopframework for execution.
•The framework tries to faithfully execute the job as described byJobConf.
•JobConfis typically used to specify the
–Mapper,
–Combiner (if any),
–Partitioner,
–Reducer,
–InputFormat,
–OutputFormat
–OutputCommitter
–set of input files, and
–where the output files should be written
•Optionally,JobConfis used to specify other advanced facets of the job such as
–theComparatorto be used,
–files to be put in theDistributedCache,
–whether intermediate and/or job outputs are to be compressed (and how),
–debugging via user-provided scripts
–whether job tasks can be executed in aspeculativemanner
–maximum number of attempts per task
–percentage of tasks failure which can be tolerated by the job, and
–useset(String, String)/get(String, String)to set/get arbitrary parameters needed by applications
11. Job Input
•InputFormatdescribes the input-specification for a MapReducejob.
•The MapReduceframework relies on theInputFormatof the job to:
–Validate the input-specification of the job.
–Split-up the input file(s) into logicalInputSplitinstances, each of which is then assigned to an individualMapper.
–Provide theRecordReaderimplementation used to extract input records from the logicalInputSplitfor processing by theMapper.
•TextInputFormatis the defaultInputFormat.
•IfTextInputFormatis theInputFormatfor a given job, the framework detects input-files with the.gzextensions and automatically decompresses.
•InputSplitrepresents the data to be processed by an individualMapper. FileSplitis the defaultInputSplit.
•RecordReaderreads<key, value>pairs from anInputSplit.
12. Job Output
•OutputFormatdescribes the output-specification for a MapReducejob.
•The MapReduceframework relies on theOutputFormatof the job to:
–Validate the output-specification of the job; for example, check that the output directory doesn't already exist.
–Provide theRecordWriterimplementation used to write the output files of the job. Output files are stored in aFileSystem.
•TextOutputFormatis the defaultOutputFormat.
13. IsolationRunner
•IsolationRunneris a utility to help debug MapReduceprograms.
•To use theIsolationRunner, first setkeep.failed.task.filestotrue(also seekeep.task.files.pattern).
•Next, go to the node on which the failed task ran and go to theTaskTracker'slocal directory and run theIsolationRunner: $ cd<local path>/taskTracker/${taskid}/work$ bin/hadooporg.apache.hadoop.mapred.IsolationRunner../job.xml
•IsolationRunnerwill run the failed task in a single jvm, which can be in the debugger, over precisely the same input.
•IsolationRunnerwill only re-run map tasks.
14. Debugging
•MapReduceframework provides a facility to run user-provided scripts for debugging
•The script is given access to the task's stdoutand stderroutputs, syslogand jobconf.
•The output from the debug script's stdoutand stderris displayed on the console diagnostics and also as part of the job UI.
•The user needs to useDistributedCachetodistributeandsymlinkthe script file.
15. Counters
•There are often things you would like to know about the data you are analyzing but that are peripheral to the analysis you are performing.
•E.g. counting invalid records and discovered that the proportion of invalid records is very high.
•perhaps there is a bug in the part of the program that detects invalid records?
•Counters are a useful channel for gathering statistics about the job:
–for quality control
–or for application level-statistics
•It is often better to see whether you can use a counter instead to record that a particular condition occurred.
•Hadoop maintains some built-in counters for every job
•Which report various metrics for your job.
•For example, there are counters for the number of bytes and records processed, which allows you to confirm that the expected amount of input was consumed and the expected amount of output was produced.
•Example Use case: Controlling Number of Recursions.
16. User-Defined Java Counters
•MapReduce allows user code to define a set of counters, which are then incremented as desired in the mapperor reducer.
•Counters are defined by a Java enum, which serves to group related counters.
•A job may define an arbitrary number of enums, each with an arbitrary number of fields.
•The name of the enumis the group name, and the enum’sfields are the counter names.
•Counters are global: the MapReduce framework aggregates them across all maps and reduces to produce a grand total at the end of the job.
17. Example of user defined counters
public class MaxTemperatureWithCountersextends Configured implements Tool {
enumTemperature {
MISSING,
MALFORMED
}
static class MaxTemperatureMapperWithCountersextends MapReduceBase
implements Mapper<LongWritable, Text, Text, IntWritable> {
public void map(LongWritablekey, Text value,
OutputCollector<Text, IntWritable> output, Reporter reporter)
throws IOException{
parser.parse(value);
if (parser.isValidTemperature()) {
//do something
} else if (parser.isMalformedTemperature()) {
reporter.incrCounter(Temperature.MALFORMED, 1);
} else if (parser.isMissingTemperature()) {
reporter.incrCounter(Temperature.MISSING, 1);
}
// dynamic counter
reporter.incrCounter("TemperatureQuality", parser.getQuality(), 1);
...
Counters counters= job.getCounters();
18. Getting built-in counters
for (CounterGroupgroup : counters) {
System.out.println("* Counter Group: " + group.getDisplayName() + " (" + group.getName() + ")");
System.out.println(" number of counters in this group: " + group.size());
for (Counter counter: group) {
System.out.println(" -" + counter.getDisplayName() + ": " + counter.getName() + ": "+counter.getValue());
}
}
