Topic 8: Enhancements and Alternative Architectures

8: Enhancements and Alternative Architectures
Zubair Nabi
zubair.nabi@itu.edu.pk
April 19, 2013
Zubair Nabi 8: Enhancements and Alternative Architectures April 19, 2013 1 / 45

Outline
1 Major shortcomings
2 Pig Latin
3 Dryad
4 CIEL
5 Naiad

Outline
2 Pig Latin
3 Dryad
4 CIEL
5 Naiad

Focusing on some
Low-level programming interface

Focusing on some
Iterative and recursive applications

Focusing on some
Iterative and recursive applications
Incremental computations

Outline
2 Pig Latin
3 Dryad
4 CIEL
5 Naiad

Introduction
MapReduce is too low-level and rigid and leads to lots of custom user
code

Introduction
code
Pig Latin is a declarative language atop MapReduce designed by
Yahoo!

Introduction
code
Yahoo!
Finds the sweet spot between the declarative style of SQL and the
low-level interface of MapReduce

Introduction
code
Yahoo!
Finds the sweet spot between the declarative style of SQL and the
low-level interface of MapReduce
The Pig system compiles Pig Latin queries into physical plans that are
executed atop Hadoop

SQL query to ﬁnd average pagerank for each large category
of URLs
1 SELECT category , AVG(pagerank)
2 FROM urls WHERE pagerank > 0.2
3 GROUP BY category HAVING COUNT(∗) > 10^6

Equivalent Pig query
1 good_urls = FILTER urls BY pagerank > 0.2;
2 groups = GROUP good_urls BY category;
3 big_groups = FILTER groups BY COUNT(good_urls)>10^6;
4 output = FOREACH big_groups GENERATE
5 category , AVG(good_urls.pagerank);

Pig Interface
A Pig Latin program is a sequence of steps, reminiscent of traditional
programming languages

Pig Interface
In contrast, SQL consists of declarative constraints that collectively
deﬁne the result

Pig Interface
deﬁne the result
Each step carries out a single data transformation

Pig Interface
deﬁne the result
A Pig Latin program is similar to specifying a query execution or a
dataﬂow graph

Pig Interface
define the result
A Pig Latin program is similar to specifying a query execution or a
dataflow graph
Due to this dataflow model, it is easier for programmers to understand
and control how their data processing task is executed

Features
Support for a fully nested data model with complex data types

Features
Extensive support for user-deﬁned functions

Features
Ability to operate over plain, schema-less input ﬁles

Features
Ability to operate over plain, schema-less input ﬁles
Open-source Apache project

Interoperability
Queries can be performed atop raw data dumps directly

Interoperability
The user needs to provide a function to parse the content of the ﬁle into
tuples

Interoperability
tuples
Similarly, the user also needs to provide a function to convert tuples
into a byte sequence

Interoperability
tuples
Similarly, the user also needs to provide a function to convert tuples
into a byte sequence
Datasets can be laid across diverse data storage sources and
applications

UDFs as ﬁrst-class citizens
A signiﬁcant part of large-scale data analysis relies on custom
processing

processing
For instance, the user may be interested in ﬁguring out whether a
particular website is spam

processing
All aspects of processing in Pig Latin including grouping, ﬁltering,
joining, and per-tuple processing can be customized via UDFs

processing
UDFs take non-atomic parameters as input and produce non-atomic
values as output

processing
UDFs take non-atomic parameters as input and produce non-atomic
values as output
UDFs are deﬁned in Java
1 groups = GROUP urls BY category;
2 output = FOREACH groups GENERATE
3 category , top10(urls);

Data Model
Pig has four data types:
1 Atom: A single atomic value such as a string or an integer

Data Model
2 Tuple: A sequence of values, each with possibly a different data type

Data Model
3 Bag: A collection of tuples

Data Model
3 Bag: A collection of tuples
4 Map: A collection of data types, each with an associated key

Commands
LOAD: Load and deserialize an input ﬁle

Commands
FOREACH: Process each tuple of a dataset

Commands
FILTER: Filter a dataset based on some condition or UDF

Commands
COGROUP: Group together tuples which are related in some way from
one or more datasets

Commands
COGROUP: Group together tuples which are related in some way from
one or more datasets
STORE: Materialize the output of a Pig Latin expression to a ﬁle

Outline
2 Pig Latin
3 Dryad
4 CIEL
5 Naiad

Introduction
MapReduce is strictly two stage, single input set and single output set

Introduction
MapReduce is strictly two stage, single input set and single output set
Awkward architecture to perform multi-stage computation

MapReduce: Architecture

Dryad
Dryad allows computations that can form a Directed Acyclic Graph
(DAG)

Dryad
(DAG)
Each vertice within the graph is a computation while an edge depicts
communication channels

Dryad
(DAG)
Each computation can take in multiple ﬁles as input and produce
multiple outputs

Dryad
(DAG)
Each computation can take in multiple ﬁles as input and produce
multiple outputs
Developed by Microsoft Research

Dryad: Architecture

Dryad: Architecture (2)
Files, TCP, FIFO, Network
job schedule
data plane
control plane
NS PD PDPD
V V V
Job manager cluster

Dryad: Job
Job = Directed Acyclic Graph
Processing
vertices Channels
(file, pipe,
shared
memory)
Inputs
Outputs

Channel types and job inputs and outputs
Channel types: File, TCP pipe, Shared-memory FIFO

Encapsulation: Convert a graph into a single vertex, and run within
same process

Encapsulation: Convert a graph into a single vertex, and run within
same process
Job inputs and outputs: Can be logically concatenated

Vertices
Programming in C++/C#

Vertices
Runtime library sets up and executes vertices

Vertices
Map and Reduce classes

Vertices
Process wrapper: To support legacy executables

Vertices
Process wrapper: To support legacy executables
Supports event-based programming

Dryad: Example
D D
MM 4n
SS 4n
YY
H
n
n
X Xn
U UN N
U U
select distinct p.objID
from photoObjAll p
join neighbors n – call this join “X”
on p.objID = n.objID
and n.objID < n.neighborObjID
and p.mode = 1
join photoObjAll l – call this join “Y”
on l.objid = n.neighborObjID
and l.mode = 1
and abs((p.u-p.g)-(l.u-l.g))<0.05
and abs((p.g-p.r)-(l.g-l.r))<0.05
and abs((p.r-p.i)-(l.r-l.i))<0.05
and abs((p.i-p.z)-(l.i-l.z))<0.05

Operations
Create Vertices using C++ base class

Operations
Add edges

Operations
Add edges
Merge two graphs

Operations (2)

Job Execution
Vertex can specify “hard constraint” or “preference”

Job Execution
Job manager runs greedy scheduling algorithm: Only job running

Job Execution
Simple graph visualizer: State of each vertex and channel for small
jobs

Job Execution
jobs
Web-based interface: Regularly-updated statistics

Job Execution
jobs
Fault Tolerance: Vertices deterministic. Just re-schedule

Job Execution
jobs
Fault Tolerance: Vertices deterministic. Just re-schedule
Speculative execution within stages

Run-time Graph Reﬁnement
Aggregation tree: Distributed combiner

Run-time Graph Reﬁnement
Aggregation tree: Distributed combiner
Associative, and commutative computation

Outline
2 Pig Latin
3 Dryad
4 CIEL
5 Naiad

Introduction
MapReduce and Dryad are not amenable to iterative and recursive
applications

Introduction
applications
Most machine learning and data mining applications are iterative in
nature

Introduction
applications
Most machine learning and data mining applications are iterative in
nature
These applications require a data-dependent control ﬂow
The ability to spawn new tasks on the ﬂy based on previous
computations

CIEL
1 Data-centric execution engine from Cambridge: the goal of a CIEL job
is to produce one or more output objects

CIEL
2 A reference can be obtained to an object without materializing its full
contents, reminiscent of C pointers

CIEL
If objects do not have their full contents, their references are future
references; otherwise they are concrete references

CIEL
If objects do not have their full contents, their references are future
references; otherwise they are concrete references
3 A job makes progress by executing tasks

Tasks
1 Each task has dependencies on one of more objects via references
and it starts executing once all of its references become concrete

Tasks
2 The purpose of each task is to produce objects

Tasks
1 A task can publish one or more objects by creating a concrete
reference for them

Tasks
reference for them
2 A task can also spawn new tasks and delegate the creation of output to
them

Tasks
reference for them
2 A task can also spawn new tasks and delegate the creation of output to
them
3 The dynamic task graph stores the relation between tasks and
objects

CIEL: Architecture

CIEL: Dynamic Task Graph

Executors
1 CIEL maintains a decoupling between tasks and the underlying
framework through the concept of executors

Executors
2 Each programming language has a corresponding executor

Executors
2 Each programming language has a corresponding executor
3 As a result, a task can be written in any programming language, such
as Java, Python, shell-script, etc. as well as the indigenous Skywriting

Skywriting
1 Scripting language for expressing task-level parallelism atop CIEL

Skywriting
2 Contains data-dependent control ﬂow constructs such as while loops
and recursive functions

Skywriting
2 Contains data-dependent control ﬂow constructs such as while loops
and recursive functions
3 Ability to spawn new tasks in the middle of execution

Skywriting constructs
1 ref(url): Returns a reference to the object located (both local and
remote) at url

remote) at url
2 spawn(f, [arg, ...]): Spawns a task to evaluate f

remote) at url
3 exec(executor, args, n): Runs the given executor to
evaluate args

remote) at url
evaluate args
4 spawn_exec(executor, args, n): Spawns a new task to run
the given executor

remote) at url
evaluate args
4 spawn_exec(executor, args, n): Spawns a new task to run
the given executor
5 *-: De-references the given reference

Example: Skywriting
1 function process_chunk(chunk, prev_result) {
2 return spawn_exec(...);
3 }
4 function is_converged(curr_result , prev_result) {
5 return spawn_exec(...)[0];
6 }
7 input_data = [ref("ciel://host137/chunk0"),
8 ref("ciel://host223/chunk1"), ...];
9 curr = ...; // Initial guess at the result.
10 do {
11 prev = curr;
12 curr = [];
13 for (chunk in input_data) {
14 curr += process_chunk(chunk, prev);
15 }
16 } while (!∗is_converged(curr, prev));
17 return curr;

Outline
2 Pig Latin
3 Dryad
4 CIEL
5 Naiad

Introduction
A class of applications requires support for both iterative and
incremental computation

Introduction
For instance, to maintain in real-time the strongly connected
component structure in the graph induced by Twitter mentions

Introduction
For instance, to maintain in real-time the strongly connected
component structure in the graph induced by Twitter mentions
Currently, MapReduce itself has no support for either iterative or

Naiad
Data-intensive computing framework from Microsoft Research that
supports both incremental and iterative computation by leveraging
differential computation

Naiad
Differential computation adds two novelty factors to the framework:

Naiad
1 The state of the computation varies according to a partially ordered set
of versions rather than a total ordering

Naiad
2 The set of updates required to reconstruct the state at any version is
retained in an indexed data-structure

Naiad
2 The set of updates required to reconstruct the state at any version is
retained in an indexed data-structure
The state and updates to that state are associated with a
multi-dimensional logical timestamp (called a version)

Programming environment
Declarative query language based on the .NET Language Integrated
Query (LINQ)

Query (LINQ)
LINQ extends C# with declarative operators, such as Select,
Where, Join, and GroupBy, among others

Query (LINQ)
Naiad adds two more operators:
1 FixedPoint that takes a source collection and a function that
mutates the collection to another collection of the same type to achieve
ﬁxed-point convergence

Query (LINQ)
Naiad adds two more operators:
1 FixedPoint that takes a source collection and a function that
mutates the collection to another collection of the same type to achieve
ﬁxed-point convergence
2 PrioritizedFP additionally takes a priority function to apply to
every record in the source collection

Runtime
The Naiad runtime transforms declarative queries to a dataﬂow graph

Runtime
The user program can insert differences into the input collections and
register callbacks to be invoked when differences are received at the
output collection

Runtime
The user program can insert differences into the input collections and
register callbacks to be invoked when differences are received at the
output collection
The runtime transparently distributes the execution of the data ﬂow
graph (similar to Dryad) across several cores and nodes

References
1 Christopher Olston, Benjamin Reed, Utkarsh Srivastava, Ravi Kumar,
and Andrew Tomkins. 2008. Pig latin: a not-so-foreign language for
data processing. In Proceedings of the 2008 ACM SIGMOD
international conference on Management of data (SIGMOD ’08). ACM,
New York, NY, USA, 1099-1110.
2 Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, and Dennis
Fetterly. 2007. Dryad: distributed data-parallel programs from
sequential building blocks. In Proceedings of the 2nd ACM
SIGOPS/EuroSys European Conference on Computer Systems 2007
(EuroSys ’07). ACM, New York, NY, USA, 59-72.
3 Derek G. Murray, Malte Schwarzkopf, Christopher Smowton, Steven
Smith, Anil Madhavapeddy, and Steven Hand. 2011. CIEL: a universal
execution engine for distributed data-ﬂow computing. In Proceedings of
the 8th USENIX conference on Networked systems design and
implementation (NSDI’11). USENIX Association, Berkeley, CA, USA.

References (2)
4 Frank McSherry, Derek G. Murray, Rebecca Isaacs, and Michael Isard.
Differential dataﬂow. 2013. In Conference on Innovative Data Systems
Research (CIDR), 2013.

Topic 8: Enhancements and Alternative Architectures

More Related Content

What's hot

Similar to Topic 8: Enhancements and Alternative Architectures

More from Zubair Nabi

Recently uploaded

Topic 8: Enhancements and Alternative Architectures