This document presents m2r2, a framework for materializing and reusing results in high-level dataflow systems for big data. The framework operates at the logical plan level to be language-independent. It includes components for matching plans, rewriting queries to reuse past results, optimizing plans, caching results, and garbage collection. An evaluation using the TPC-H benchmark on Pig Latin showed the framework reduced query execution time by 65% on average by reusing past query results. Future work includes integrating it with more systems and minimizing materialization costs.

1. m2r2: A Framework for Results
Materialization and
Reuse in High-Level Dataflow Systems
for Big Data
2nd International Conference
on Big Data Science and Engineering (BDSE 2013)
Vasiliki Kalavri, Hui Shang, Vladimir Vlassov
{kalavri, hshang, vladv}@kth.se
4 December 2013, Sydney, Australia

2. Outline
➔ Motivation
➔ Materialized Views in Relational DBMSs
➔ High-Level Dataflow Systems for Big Data
◆ similarities in design and implementation
➔ m2r2 design
◆ design goals and system components
➔ Prototype Implementation Details
➔ Evaluation Results
➔ Conclusions and Future Work
2

3. Motivation
➔ Avoid computational redundancies
◆ filter out bad records, spam e-mail
◆ data representation transformations
➔ Microsoft has found a 30%-60% similarity
in queries submitted for execution
➔ A Berkeley MapReduce workload
characterization study shows a big need
for caching job results
3

4. Materialized Views in RDBMSs
➔ A derived relation, stored in the database
◆ Queries are computed using the views instead of
the base relations
➔ Challenges
◆ View Design: What to materialize?
◆ View Maintenance: How to update the views?
◆ View Exploitation: How to use the views for query
optimization?
● view matching and query rewriting
4

5. High-Level Dataflow Systems (1)
High-Level Dataflow Systems for Big Data
(Pig, Hive, Jaql, DryadLINQ, etc.) exhibit
wide similarities on multiple design levels:
➔ Language Layer
◆ Declarative, SQL-like language
◆ Statements define transformations on collections of datasets
➔ Data Operators
◆ Encapsulate the logic of the transformations to be performed
◆ Relational, Expressions, Control-flow
5

6. High-Level Dataflow Systems (2)
Pig Latin
HiveQL
Jaql
6

7. High-Level Dataflow Systems (3)
● The Logical Plan
○ Parser → AST → DAG of operators
● Compilation to an Execution Plan
7

8. m2r2: materialize - match - rewrite - reuse
➔ A language-independent, extensible
framework for
◆ storing
◆ managing and
◆ using
previous job and sub-job results
➔ Operates on the logical plan level, in
order to support different languages and
backend execution engines
8

9. m2r2 Components
➔ Plan Matcher and Rewriter
◆ How to be independent of the high-level
language and execution engine?
◆ Shark: Hive on Spark, PonIC: Pig on
Stratosphere, etc.? → Match at the Logical Plan
level!
➔ Plan Optimizer
➔ Results Cache
➔ Plan Repository
➔ Garbage Collector
9

10. Match and Rewrite Algorithm
10

11. m2r2 Implementation
➔ Built on top of
Pig/Hadoop
➔ HDFS as the Results Cache
➔ MySQL Cluster as the
Repository
◆ in-memory, highly-available
and fault-tolerant
➔ Garbage Collection as a
separate module
◆ policy on reuse frequency and
last access time
11

12. Evaluation Setup
12
➔ Cluster Setup
◆ Pig 0.11, Hadoop 1.0.4 and MySQL Cluster 7.2.12
deployed on top of OpenStack
◆ 20 Ubuntu 11.10 VMs
➔ Data and Queries
◆ TPC-H Benchmark for Pig
◆ 20 queries, out of which 6 with reuse
opportunity
◆ 107 GB of data using DBGEN tools of TPC-H

13. Speedup using Sub-Jobs
13

14. Speedup using Whole Jobs
14

15. Conclusions
15
➔ The logical plan is the proper layer to
build a language-independent reuse
framework
➔ When there exists reuse opportunity,
query execution time can be immensely
reduced
◆ 65% on average in our experiments
➔ The materialization overhead is quite
small and I/O dominant

16. Future Work
➔ Integrate with other high-level systems
➔ Explore the possibility of sharing results
among different frameworks
➔ Obtain execution traces and perform a
more realistic evaluation
➔ Minimize costs by overlapping
materialization with regular query
execution
16

17. m2r2: A Framework for Results
Materialization and
Reuse in High-Level Dataflow Systems
for Big Data
2nd International Conference
on Big Data Science and Engineering (BDSE 2013)
Vasiliki Kalavri, Hui Shang, Vladimir Vlassov
{kalavri, hshang, vladv}@kth.se
4 December 2013, Sydney, Australia