Hadoop Slides 1.5 - Hadoop and MapReduce

MapReduce IBM Information ManagementCloud Computing Center of Competence IBM Toronto Lab 1

Agenda • Map operations • Reduce operations • Submitting a MapReduce job • Distributed Mergesort Engine • Two fundamental data types • Fault tolerance • Scheduling • Task execution2

What is a Map operation? • Doing something to every element in an array is a common operation: var a = [1,2,3]; for (i = 0; i < a.length; i++) a[i] = a[i] * 2;4

What is a Map operation? • Doing something to every element in an array is a common operation: var a = [1,2,3]; for (i = 0; i < a.length; i++) a[i] = a[i] * 2; • New value for variable a would be: var a = [2,4,6];5

What is a Map operation? • Doing something to every element in an array is a common operation: var a = [1,2,3]; for (i = 0; i < a.length; i++) This can a[i] = a[i] * 2; be written as a function • New value for variable a would be: var a = [2,4,6];6

What is a Map operation? • Doing something to every element in an array is a common operation: var a = [1,2,3]; Like this, for (i = 0; i < a.length; i++) where fn is a[i] * 2; a[i] = fn(a[i]); a function defined as: • New value for variable a would be: function fn(x) var a = [2,4,6]; {return x*2;}7

What is a Map operation? • Doing something to every element in an array is a common operation: var a = [1,2,3]; for (i = 0; i < a.length; i++) a[i] = fn(a[i]); Now, all of this can also be converted into a “map” function8

What is a Map operation? • …like this, where fn is a function passed as an argument: function map(fn, a) { for (i = 0; i < a.length; i++) a[i] = fn(a[i]); }9

What is a Map operation? • …like this, where fn is a function passed as an argument: function map(fn, a) { for (i = 0; i < a.length; i++) a[i] = fn(a[i]); } • You can invoke this map function like this: map(function(x){return x*2;}, a);10

What is a Map operation? • …like this, where fn is a function passed as an argument: function map(fn, a) { for (i = 0; i < a.length; i++) a[i] = fn(a[i]); } • You can invoke this map function like this: map(function(x){return x*2;}, a); This is function fn whose definition is included in the call11

What is a Map operation? • In summary, now you can rewrite: for (i = 0; i < a.length; i++) a[i] = a[i] * 2; } as a map operation: map(function(x){return x*2;}, a);12

What is a Reduce operation? • Another common operation on arrays is to combine all their values: function sum(a) { var s = 0; for (i = 0; i < a.length; i++) s += a[i]; return s; }14

What is a Reduce operation? • Another common operation on arrays is to combine all their values: function sum(a) { var s = 0; for (i = 0; i < a.length; i++) This can be written s += a[i]; as a function return s; }15

What is a Reduce operation? • Another common operation on arrays is to combine all their values: function sum(a) { var s = 0; Like this, where function fn is defined so for (i = 0; i < a.length; i++) it adds its arguments: function fn(a,b){ s = fn(s,a[i]); return a+b; } return s; }16

What is a Reduce operation? • Another common operation on arrays is to combine all their values: function sum(a) { var s = 0; for (i = 0; i < a.length; i++) s = fn(s, a[i]); return s; } The whole function sum can also be rewritten so that fn is passed as an argument17

What is a Reduce operation? • Another common operation on arrays is to combine all their values: function reduce(fn, a, init) { var s = init; for (i = 0; i < a.length; i++) s = fn(s, a[i]); return s; } Like this… The function name was changed to reduce, and now it takes three arguments, a function, an array, and an initial value18

What is a Reduce operation? • Another common operation on arrays is to combine all their values: function sum(a) { var s = 0; for (i = 0; i < a.length; i++) s += a[i]; return s; } as a reduce operation: reduce(function(a,b){return a+b;},a,0);19

Submitting a MapReduce job21

MapReduce – Distributed Mergesort Engine32

Two Fundamental data types • Key/value pairs • Lists Input Output map reduce44

Two Fundamental data types • Key/value pairs • Lists Input Output map <k1, v1> reduce45

Two Fundamental data types • Key/value pairs • Lists Input Output map <k1, v1> list(<k2, v2>) reduce46

Two Fundamental data types • Key/value pairs • Lists Input Output map <k1, v1> list(<k2, v2>) reduce <k2, list(v2)>47

Two Fundamental data types • Key/value pairs • Lists Input Output map <k1, v1> list(<k2, v2>) reduce <k2, list(v2)> list(<k3, v3>)48

Simple data flow example49

Data flow example with parallelism Map Reduce Node 1 Node 1 Map Reduce Input Node 2 Node 2 … … Map Reduce Node n Node n shuffle

Fault tolerance JobTracker node JobTracker 3 JobTracker fails heartbeat TaskTracker 2 TaskTracker fails child JVM Child 1 Task fails MapTask or ReduceTask TaskTracker node55

Fault tolerance • Task Failure • If a child task fails, the child JVM reports to the TaskTracker before it exits. Attempt is marked failed, freeing up slot for another task. • If the child task hangs, it is killed. JobTracker reschedules the task on another machine. • If task continues to fail, job is failed.56

Fault tolerance • TaskTracker Failure • JobTracker receives no heartbeat • Removes TaskTracker from pool of TaskTrackers to schedule tasks on.57

Fault tolerance • JobTracker Failure • Singe point of failure. Job fails58

Scheduling • FIFO scheduler (with priorities) • Fair scheduler • Capacity scheduler59

Scheduling – FIFO scheduler (with priorities) • Each job uses the whole cluster, so jobs wait their turn. • Can set priorities for the jobs in the queue (5 queues with priorities) • Preemption is not supportedVery Low Normal High VeryLow High J3 J6 J2J7 J5 J4 J160

Scheduling – Fair scheduler • Jobs are assigned to pools (1 pool per user by default) • A user job pool is a number of slots assigned for tasks for that user • Each pool gets the same number of task slots by default Mary submits many jobs … Slot for a task Job pool for user Mary61

Scheduling – Fair scheduler • Multiple users can run jobs on the cluster at the same time • Example: • Mary, John, Peter submit jobs that demand 80, 80, and 120 tasks respectively • Say the cluster has a limit to allocate 60 tasks at most • Default behavior: Distribute task fairly among 3 users (each get 20) Maximum amount of tasks that can be allocated in this cluster: 60 20 tasks 20 tasks 20 tasks Mary John Peter Demand: 80 Demand: 80 Demand: 12062

Scheduling – Fair scheduler • Minimum share can be set for a pool • In the previous example, say Mary has a minimum share of 40 • Mary would be allocated 40, then the rest is distributed evenly to other pools Maximum amount of tasks that can be allocated in this cluster: 60 40 tasks 10 tasks 10 tasks Mary John Peter Demand: 80 Demand: 80 Demand: 120 Minimum share: 4063

Scheduling – Capacity scheduler • Similar goal to Fair scheduler: Distribute jobs fairly among users • Works with queues instead of pools • A user thinks he has the cluster to himself with FIFO scheduling, but actually is sharing resources with other users Peter J2 J1 John J1 Mary J164

Task execution – Improving performance • Speculative Execution • Job execution is time sensitive to slow-running tasks. • Hadoop detects slow-running tasks and launches another, equivalent task as a backup. • The output from the first of these tasks to finish is used. • Task JVM Reuse • Tasks run in their own JVMs for isolation. • Starting up a JVM is relatively expensive if jobs are short • In these cases when jobs have many short tasks, improve performance by configuring JVM reuse65

Thank you!Thank you!

Hadoop Slides 1.5 - Hadoop and MapReduce

by Leons Petražickis on Nov 08, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

Hadoop Slides 1.5 - Hadoop and MapReduce — Presentation Transcript