Download as PDF, PPTX
![Classification [Ammar and ¨Ozsu, 2015]
Graph Dynamism
Static
Graphs
Dynamic
Graphs
Streaming
Graphs
Evolving
Graphs
Algorithm Types
Offline Online
Streaming Incremental
Dynamic
Batch
Dynamic
Workload Types
Online
Queries
Analytics
Workloads
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-14-2048.jpg)
![Classification [Ammar and ¨Ozsu, 2015]
Graph Dynamism
Static
Graphs
Dynamic
Graphs
Streaming
Graphs
Evolving
Graphs
Algorithm Types
Offline Online
Streaming Incremental
Dynamic
Batch
Dynamic
Workload Types
Online
Queries
Analytics
Workloads
Focus here is on the
dynamism of the
graphs in whether or
not they change and
how they change.
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-15-2048.jpg)
![Classification [Ammar and ¨Ozsu, 2015]
Graph Dynamism
Static
Graphs
Dynamic
Graphs
Streaming
Graphs
Evolving
Graphs
Algorithm Types
Offline Online
Streaming Incremental
Dynamic
Batch
Dynamic
Workload Types
Online
Queries
Analytics
Workloads
Focus here is on the
dynamism of the
graphs in whether or
not they change and
how they change.
Focus here is on the
how algorithms behave
as their input changes.
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-16-2048.jpg)
![Classification [Ammar and ¨Ozsu, 2015]
Graph Dynamism
Static
Graphs
Dynamic
Graphs
Streaming
Graphs
Evolving
Graphs
Algorithm Types
Offline Online
Streaming Incremental
Dynamic
Batch
Dynamic
Workload Types
Online
Queries
Analytics
Workloads
Focus here is on the
dynamism of the
graphs in whether or
not they change and
how they change.
Focus here is on the
how algorithms behave
as their input changes.
The types of workloads
that the approaches are
designed to handle.
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-17-2048.jpg)
![Classification [Ammar and ¨Ozsu, 2015]
Graph Dynamism
Static
Graphs
Dynamic
Graphs
Streaming
Graphs
Evolving
Graphs
Algorithm Types
Offline Online
Streaming Incremental
Dynamic
Batch
Dynamic
Workload Types
Online
Queries
Analytics
Workloads
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-18-2048.jpg)
![Classification [Ammar and ¨Ozsu, 2015]
Graph Dynamism
Static
Graphs
Dynamic
Graphs
Streaming
Graphs
Evolving
Graphs
Algorithm Types
Offline Online
Streaming Incremental
Dynamic
Batch
Dynamic
Workload Types
Online
Queries
Analytics
Workloads
Graphs do not
change or we
are not inter-
ested in their
changes – only
a snapshot is
considered.
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-19-2048.jpg)
![Classification [Ammar and ¨Ozsu, 2015]
Graph Dynamism
Static
Graphs
Dynamic
Graphs
Streaming
Graphs
Evolving
Graphs
Algorithm Types
Offline Online
Streaming Incremental
Dynamic
Batch
Dynamic
Workload Types
Online
Queries
Analytics
Workloads
Graphs do not
change or we
are not inter-
ested in their
changes – only
a snapshot is
considered.
Graphs change
and we are
interested in
their changes.
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-20-2048.jpg)
![Classification [Ammar and ¨Ozsu, 2015]
Graph Dynamism
Static
Graphs
Dynamic
Graphs
Streaming
Graphs
Evolving
Graphs
Algorithm Types
Offline Online
Streaming Incremental
Dynamic
Batch
Dynamic
Workload Types
Online
Queries
Analytics
Workloads
Graphs do not
change or we
are not inter-
ested in their
changes – only
a snapshot is
considered.
Graphs change
and we are
interested in
their changes.
Dynamic
graphs with
high veloc-
ity changes –
not possible to
see the entire
graph at once.
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-21-2048.jpg)
![Classification [Ammar and ¨Ozsu, 2015]
Graph Dynamism
Static
Graphs
Dynamic
Graphs
Streaming
Graphs
Evolving
Graphs
Algorithm Types
Offline Online
Streaming Incremental
Dynamic
Batch
Dynamic
Workload Types
Online
Queries
Analytics
Workloads
Graphs do not
change or we
are not inter-
ested in their
changes – only
a snapshot is
considered.
Graphs change
and we are
interested in
their changes.
Dynamic
graphs with
high veloc-
ity changes –
not possible to
see the entire
graph at once.
Dynamic
graphs with un-
known changes
– requires re-
discovery of
the graph (e.g.,
LOD).
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-22-2048.jpg)
![Classification [Ammar and ¨Ozsu, 2015]
Graph Dynamism
Static
Graphs
Dynamic
Graphs
Streaming
Graphs
Evolving
Graphs
Algorithm Types
Offline Online
Streaming Incremental
Dynamic
Batch
Dynamic
Workload Types
Online
Queries
Analytics
Workloads
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-23-2048.jpg)
![Classification [Ammar and ¨Ozsu, 2015]
Graph Dynamism
Static
Graphs
Dynamic
Graphs
Streaming
Graphs
Evolving
Graphs
Algorithm Types
Offline Online
Streaming Incremental
Dynamic
Batch
Dynamic
Workload Types
Online
Queries
Analytics
Workloads
Computation accesses a
portion of the graph
and the results are
computed for a subset
of vertices; e.g., point-
to-point shortest path,
subgraph matching,
reachability, SPARQL.
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-24-2048.jpg)
![Classification [Ammar and ¨Ozsu, 2015]
Graph Dynamism
Static
Graphs
Dynamic
Graphs
Streaming
Graphs
Evolving
Graphs
Algorithm Types
Offline Online
Streaming Incremental
Dynamic
Batch
Dynamic
Workload Types
Online
Queries
Analytics
Workloads
Computation accesses a
portion of the graph
and the results are
computed for a subset
of vertices; e.g., point-
to-point shortest path,
subgraph matching,
reachability, SPARQL.
Computation accesses
the entire graph and
may require multiple
iterations; e.g., PageR-
ank, clustering, graph
colouring, all pairs
shortest path.
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-25-2048.jpg)
![Classification [Ammar and ¨Ozsu, 2015]
Graph Dynamism
Static
Graphs
Dynamic
Graphs
Streaming
Graphs
Evolving
Graphs
Algorithm Types
Offline Online
Streaming Incremental
Dynamic
Batch
Dynamic
Workload Types
Online
Queries
Analytics
Workloads
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-26-2048.jpg)
![Classification [Ammar and ¨Ozsu, 2015]
Graph Dynamism
Static
Graphs
Dynamic
Graphs
Streaming
Graphs
Evolving
Graphs
Algorithm Types
Offline Online
Streaming Incremental
Dynamic
Batch
Dynamic
Workload Types
Online
Queries
Analytics
Workloads
Sees the en-
tire input in
advance.
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-27-2048.jpg)
![Classification [Ammar and ¨Ozsu, 2015]
Graph Dynamism
Static
Graphs
Dynamic
Graphs
Streaming
Graphs
Evolving
Graphs
Algorithm Types
Offline Online
Streaming Incremental
Dynamic
Batch
Dynamic
Workload Types
Online
Queries
Analytics
Workloads
Sees the en-
tire input in
advance.
Sees the input
piece-meal as it
executes.
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-28-2048.jpg)
![Classification [Ammar and ¨Ozsu, 2015]
Graph Dynamism
Static
Graphs
Dynamic
Graphs
Streaming
Graphs
Evolving
Graphs
Algorithm Types
Offline Online
Streaming Incremental
Dynamic
Batch
Dynamic
Workload Types
Online
Queries
Analytics
Workloads
Sees the en-
tire input in
advance.
Sees the input
piece-meal as it
executes.
One-pass on-
line algorithm
with limited
memory.
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-29-2048.jpg)
![Classification [Ammar and ¨Ozsu, 2015]
Graph Dynamism
Static
Graphs
Dynamic
Graphs
Streaming
Graphs
Evolving
Graphs
Algorithm Types
Offline Online
Streaming Incremental
Dynamic
Batch
Dynamic
Workload Types
Online
Queries
Analytics
Workloads
Sees the en-
tire input in
advance.
Sees the input
piece-meal as it
executes.
One-pass on-
line algorithm
with limited
memory.
Online algo-
rithm with
some info
about forth-
coming input.
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-30-2048.jpg)
![Classification [Ammar and ¨Ozsu, 2015]
Graph Dynamism
Static
Graphs
Dynamic
Graphs
Streaming
Graphs
Evolving
Graphs
Algorithm Types
Offline Online
Streaming Incremental
Dynamic
Batch
Dynamic
Workload Types
Online
Queries
Analytics
Workloads
Sees the en-
tire input in
advance.
Sees the input
piece-meal as it
executes.
One-pass on-
line algorithm
with limited
memory.
Online algo-
rithm with
some info
about forth-
coming input.
Sees the en-
tire input
in advance,
which may
change; an-
swers computed
as change oc-
curs.
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-31-2048.jpg)
![Classification [Ammar and ¨Ozsu, 2015]
Graph Dynamism
Static
Graphs
Dynamic
Graphs
Streaming
Graphs
Evolving
Graphs
Algorithm Types
Offline Online
Streaming Incremental
Dynamic
Batch
Dynamic
Workload Types
Online
Queries
Analytics
Workloads
Sees the en-
tire input in
advance.
Sees the input
piece-meal as it
executes.
One-pass on-
line algorithm
with limited
memory.
Online algo-
rithm with
some info
about forth-
coming input.
Sees the en-
tire input
in advance,
which may
change; an-
swers computed
as change oc-
curs.
Similar to dy-
namic, but
computation
happens in
batches of
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-32-2048.jpg)
![Some Alternative Computational Models for Offline
Analytics
Vertex-centric (Scatter-Gather)
Specify (a) computation at each vertex, and (b) communication with
neighbour vertices
Synchronous – Pregel [Malewicz et al., 2010], Giraph
Asynchronous – GraphLab [Low et al., 2012]
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 18 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-50-2048.jpg)
![Some Alternative Computational Models for Offline
Analytics
Vertex-centric (Scatter-Gather)
Specify (a) computation at each vertex, and (b) communication with
neighbour vertices
Synchronous – Pregel [Malewicz et al., 2010], Giraph
Asynchronous – GraphLab [Low et al., 2012]
Block-centric
Similar to vertex-centric but on blocks for communication
Connected subgraph of the graph
Blogel [Yan et al., 2014]
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 18 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-51-2048.jpg)
![Some Alternative Computational Models for Offline
Analytics
Vertex-centric (Scatter-Gather)
Specify (a) computation at each vertex, and (b) communication with
neighbour vertices
Synchronous – Pregel [Malewicz et al., 2010], Giraph
Asynchronous – GraphLab [Low et al., 2012]
Block-centric
Similar to vertex-centric but on blocks for communication
Connected subgraph of the graph
Blogel [Yan et al., 2014]
MapReduce
Need to save in HDFS intermediate results of each iteration – both
good and bad
Hadoop, Haloop [Bu et al., 2012]
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 18 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-52-2048.jpg)
![Some Alternative Computational Models for Offline
Analytics
Vertex-centric (Scatter-Gather)
Specify (a) computation at each vertex, and (b) communication with
neighbour vertices
Synchronous – Pregel [Malewicz et al., 2010], Giraph
Asynchronous – GraphLab [Low et al., 2012]
Block-centric
Similar to vertex-centric but on blocks for communication
Connected subgraph of the graph
Blogel [Yan et al., 2014]
MapReduce
Need to save in HDFS intermediate results of each iteration – both
good and bad
Hadoop, Haloop [Bu et al., 2012]
Modified MapReduce
Based on Spark [Zaharia et al., 2010; Zaharia, 2016]
Keep intermediate states in memory
Provide fault-tolerance by keeping lineage
GraphX [Gonzalez et al., 2014]
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 18 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-53-2048.jpg)
![Summary of an Experiment [Han et al., 2014]
A large study comparing Giraph, GraphLab, GPS, Mizan.
1 Giraph scales better across graphs;
GraphLab scales better across more machines.
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 22 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-67-2048.jpg)
![Summary of an Experiment [Han et al., 2014]
A large study comparing Giraph, GraphLab, GPS, Mizan.
1 Giraph scales better across graphs;
GraphLab scales better across more machines.
64 machines TW UK
Giraph (byte array) 5.8GB 7.0GB
GraphLab (sync) 4.5GB 14GB
TW 16 machines 128 machines
Giraph (byte array) 8.5GB 5.8GB
GraphLab (sync) 11GB 3.3GB
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 22 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-68-2048.jpg)
![Summary of an Experiment [Han et al., 2014]
A large study comparing Giraph, GraphLab, GPS, Mizan.
1 Giraph scales better across graphs;
GraphLab scales better across more machines.
2 Distributed locking for asynchronous execution is not scalable –
Performance degrades as more machines are used due to lock
contention, termination scheme, lack of message batching
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 22 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-69-2048.jpg)
![Summary of an Experiment [Han et al., 2014]
A large study comparing Giraph, GraphLab, GPS, Mizan.
1 Giraph scales better across graphs;
GraphLab scales better across more machines.
2 Distributed locking for asynchronous execution is not scalable –
Performance degrades as more machines are used due to lock
contention, termination scheme, lack of message batching
3 Graph storage should be memory and mutation efficient.
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 22 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-70-2048.jpg)
![Summary of an Experiment [Han et al., 2014]
A large study comparing Giraph, GraphLab, GPS, Mizan.
1 Giraph scales better across graphs;
GraphLab scales better across more machines.
2 Distributed locking for asynchronous execution is not scalable –
Performance degrades as more machines are used due to lock
contention, termination scheme, lack of message batching
3 Graph storage should be memory and mutation efficient.
No Mutations
Time Memory
Byte array
Hash map
With Mutations (DMST)
Time Memory
Byte array
Hash map
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 22 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-71-2048.jpg)
![Summary of an Experiment [Han et al., 2014]
A large study comparing Giraph, GraphLab, GPS, Mizan.
1 Giraph scales better across graphs;
GraphLab scales better across more machines.
2 Distributed locking for asynchronous execution is not scalable –
Performance degrades as more machines are used due to lock
contention, termination scheme, lack of message batching
3 Graph storage should be memory and mutation efficient.
4 Message processing optimizations are very important.
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 22 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-72-2048.jpg)
![Summary of an Experiment [Han et al., 2014]
A large study comparing Giraph, GraphLab, GPS, Mizan.
1 Giraph scales better across graphs;
GraphLab scales better across more machines.
2 Distributed locking for asynchronous execution is not scalable –
Performance degrades as more machines are used due to lock
contention, termination scheme, lack of message batching
3 Graph storage should be memory and mutation efficient.
4 Message processing optimizations are very important.
5 Workloads have different resource demands
Algorithm CPU Memory Network
PageRank Medium Medium High
SSSP Low Low Low
WCC Low Medium Medium
DMST High High Medium
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 22 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-73-2048.jpg)
![Block-Centric Computation
Blogel [Yan et al., 2014]: “Think like a block”; also “think like a
graph” [Tian et al., 2013]
Vertex-centric assumes all vertices communicate over the network;
this is not efficient
Read-world graphs have skewed vertex degree distribution
Common in power-law graphs
Problem: imbalanced communication workloads
Real-world graphs have large diameters
Common in road networks, web graphs, terrain meshes
Problem: one superstep per hop ⇒ too many supersteps
Real-world graphs have high average vertex degree
Common in social networks, mobile communication networks
Problem: heavy average communication workloads
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 23 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-74-2048.jpg)
![MapReduce Basics [Li et al., 2014]
For data analysis of very large data sets
Highly dynamic, irregular, schemaless, etc.
SQL too heavy
“Embarrassingly parallel problems”
New, simple parallel programming model
Data structured as (key, value) pairs
E.g. (doc-id, content), (word, count), etc.
Functional programming style with two functions to be given:
Map(k1,v1) → list(k2,v2)
Reduce(k2, list (v2)) → list(v3)
Implemented on a distributed file system (e.g., Google File System)
on very large clusters
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 28 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-79-2048.jpg)
![Execution Flow with Architecture [Dean and Ghemawat, 2008]
MapReduce: Simplified Data Processing on Large Clusters
split 0
split 1
split 2
split 3
split 4
(1) fork
(3) read
(4) local write
(1) fork
(1) fork
(6) write
worker
worker
worker
Master
User
Program
output
file 0
output
file 1
worker
worker
(2)
assign
map
(2)
assign
reduce
(5) remote
(5) read
Input
files
Map
phasr
Intermediate files
(on local disks)
Reduce
phase
Output
files
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 31 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-82-2048.jpg)
![HaLoop [Bu et al., 2012]
Overcome MapReduce shortcomings for iterative jobs
Having to save data in HDFS in between each iteration
Checking the fixpoint requires a new job at each iteration
Scheduler change: assign to the same machine the map reduce
tasks that occur in different iterations but access the same data
Cache invariant data
Cache reduce output to easily check for fixpoint
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 33 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-84-2048.jpg)
![Spark Ecosystem [Michiardi, 2015]
Native
Spark
Apps
Spark
SQL
Spark
Streaming
MLlib
(machine
learning)
GraphX
(graph
processing)
Apache Spark
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 35 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-88-2048.jpg)
![Spark Programming Model [Zaharia et al., 2010, 2012]
HDFS
Create RDD
· · ·
RDD
Cache? Cache
Yes
Transform
RDD?
No
Process
No
Transform
Yes
HDFS
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 36 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-89-2048.jpg)
![Spark Programming Model [Zaharia et al., 2010, 2012]
HDFS
Create RDD
· · ·
RDD
Cache? Cache
Yes
Transform
RDD?
No
Process
No
Transform
Yes
HDFS
Each transform generates a
new RDD that may also be
cached or processed
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 36 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-90-2048.jpg)
![Spark Programming Model [Zaharia et al., 2010, 2012]
HDFS
Create RDD
· · ·
RDD
Cache? Cache
Yes
Transform
RDD?
No
Process
No
Transform
Yes
HDFS
Each transform generates a
new RDD that may also be
cached or processed
Created from HDFS or parallelized arrays;
Partitioned across worker machines;
May be made persistent lazily;
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 36 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-91-2048.jpg)
![Spark Programming Model [Zaharia et al., 2010, 2012]
HDFS
Create RDD
· · ·
RDD
Cache? Cache
Yes
Transform
RDD?
No
Process
No
Transform
Yes
HDFS
Each transform generates a
new RDD that may also be
cached or processed
Created from HDFS or parallelized arrays;
Partitioned across worker machines;
May be made persistent lazily;
Processing done on one of the RDDs;
Done in parallel across workers;
First processing on a RDD is from disk;
Subsequent processing of the same RDD from cache
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 36 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-92-2048.jpg)
![Example – Log Mining [Zaharia et al., 2010, 2012]
Load log messages from a file system, create a new file by filtering the
error messages, read this file into memory, then interactively search for
various patterns
Driver
WorkerWorkerWorker
Block 1 Block 2 Block 3
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 37 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-93-2048.jpg)
![Example – Log Mining [Zaharia et al., 2010, 2012]
Load log messages from a file system, create a new file by filtering the
error messages, read this file into memory, then interactively search for
various patterns
lines = spark.textFile(hdfs://...)
CreateRDD
Driver
WorkerWorkerWorker
Block 1 Block 2 Block 3
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 37 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-94-2048.jpg)
![Example – Log Mining [Zaharia et al., 2010, 2012]
Load log messages from a file system, create a new file by filtering the
error messages, read this file into memory, then interactively search for
various patterns
lines = spark.textFile(hdfs://...)
errors = lines.filter( .startsWith(ERROR))
Transform RDD
Driver
WorkerWorkerWorker
Block 1 Block 2 Block 3
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 37 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-95-2048.jpg)
![Example – Log Mining [Zaharia et al., 2010, 2012]
Load log messages from a file system, create a new file by filtering the
error messages, read this file into memory, then interactively search for
various patterns
lines = spark.textFile(hdfs://...)
errors = lines.filter( .startsWith(ERROR))
messages = errors.map( .split(‘t ’)(2))
Another transform
Driver
WorkerWorkerWorker
Block 1 Block 2 Block 3
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 37 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-96-2048.jpg)
![Example – Log Mining [Zaharia et al., 2010, 2012]
Load log messages from a file system, create a new file by filtering the
error messages, read this file into memory, then interactively search for
various patterns
lines = spark.textFile(hdfs://...)
errors = lines.filter( .startsWith(ERROR))
messages = errors.map( .split(‘t ’)(2))
cachedMsgs = messages.cache()
Cache results
Driver
WorkerWorkerWorker
Block 1 Block 2 Block 3
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 37 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-97-2048.jpg)
![Example – Log Mining [Zaharia et al., 2010, 2012]
Load log messages from a file system, create a new file by filtering the
error messages, read this file into memory, then interactively search for
various patterns
lines = spark.textFile(hdfs://...)
errors = lines.filter( .startsWith(ERROR))
messages = errors.map( .split(‘t ’)(2))
cachedMsgs = messages.cache()
cachedMsgs.filter( .contains(foo)).count
Action
Driver
WorkerWorkerWorker
Block 1 Block 2 Block 3
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 37 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-98-2048.jpg)
![Example – Log Mining [Zaharia et al., 2010, 2012]
Load log messages from a file system, create a new file by filtering the
error messages, read this file into memory, then interactively search for
various patterns
lines = spark.textFile(hdfs://...)
errors = lines.filter( .startsWith(ERROR))
messages = errors.map( .split(‘t ’)(2))
cachedMsgs = messages.cache()
cachedMsgs.filter( .contains(foo)).count
Driver
WorkerWorkerWorker
Block 1 Block 2 Block 3
Tasks
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 37 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-99-2048.jpg)
![Example – Log Mining [Zaharia et al., 2010, 2012]
Load log messages from a file system, create a new file by filtering the
error messages, read this file into memory, then interactively search for
various patterns
lines = spark.textFile(hdfs://...)
errors = lines.filter( .startsWith(ERROR))
messages = errors.map( .split(‘t ’)(2))
cachedMsgs = messages.cache()
cachedMsgs.filter( .contains(foo)).count
Driver
WorkerWorkerWorker
Block 1 Block 2 Block 3
TasksResults
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 37 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-100-2048.jpg)
![Example – Log Mining [Zaharia et al., 2010, 2012]
Load log messages from a file system, create a new file by filtering the
error messages, read this file into memory, then interactively search for
various patterns
lines = spark.textFile(hdfs://...)
errors = lines.filter( .startsWith(ERROR))
messages = errors.map( .split(‘t ’)(2))
cachedMsgs = messages.cache()
cachedMsgs.filter( .contains(foo)).count
Driver
WorkerWorkerWorker
Block 1 Block 2 Block 3
TasksResults
Cache Cache Cache
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 37 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-101-2048.jpg)
![Example – Log Mining [Zaharia et al., 2010, 2012]
Load log messages from a file system, create a new file by filtering the
error messages, read this file into memory, then interactively search for
various patterns
lines = spark.textFile(hdfs://...)
errors = lines.filter( .startsWith(ERROR))
messages = errors.map( .split(‘t ’)(2))
cachedMsgs = messages.cache()
cachedMsgs.filter( .contains(foo)).count
cachedMsgs.filter( .contains(bar)).count
Another Action
accesses cache
Driver
WorkerWorkerWorker
Block 1 Block 2 Block 3
TasksResults
Cache Cache Cache
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 37 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-102-2048.jpg)
![GraphX [Gonzalez et al., 2014]
Built on top of Spark
Objective is to combine data analytics with graph processing
Unify computation on tables and graphs
Carefully convert graph to tabular representation
Native GraphX API or can accommodate vertex-centric computation
Native
Spark
Apps
Spark
SQL
Spark
Streaming
MLlib
(machine
learning)
GraphX
(graph
processing)
Apache Spark
Vertex-
centric API
AppApp
App App
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 39 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-105-2048.jpg)
![Exhaustive Indexing
RDF-3X [Neumann and Weikum, 2008, 2009], Hexastore [Weiss
et al., 2008]
Strings are mapped to ids using a mapping table
Original triple table
Subject Property Object
mdb: film/2014 rdfs:label “The Shining”
mdb:film/2014 movie:initial release date “1980-05-23”
mdb:director/8476 movie:director name “Stanley Kubrick”
mdb:film/2685 movie:director mdb:director/8476
Encoded triple table
Subject Property Object
0 1 2
0 3 4
5 6 7
8 9 5
Mapping table
ID Value
0 mdb: film/2014
1 rdfs:label
2 “The Shining”
3 movie:initial release date
4 “1980-05-23”
5 mdb:director/8476
6 movie:director name
7 “Stanley Kubrick”
8 mdb:film/2685
9 movie:director
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 52 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-132-2048.jpg)
![Exhaustive Indexing
RDF-3X [Neumann and Weikum, 2008, 2009], Hexastore [Weiss
et al., 2008]
Strings are mapped to ids using a mapping table
Triples are indexed in a clustered B+ tree in lexicographic order
Subject Property Object
0 1 2
0 3 4
5 6 7
8 9 5
...
...
...
B+ tree
Easy querying
through mapping
table
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 52 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-133-2048.jpg)
![Exhaustive Indexing
RDF-3X [Neumann and Weikum, 2008, 2009], Hexastore [Weiss
et al., 2008]
Strings are mapped to ids using a mapping table
Triples are indexed in a clustered B+ tree in lexicographic order
Create indexes for permutations of the three columns: SPO, SOP,
PSO, POS, OPS, OSP
Subject Property Object
0 1 2
0 3 4
5 6 7
8 9 5
...
...
...
B+ tree
Easy querying
through mapping
table
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 52 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-134-2048.jpg)
![Property Tables
Grouping by entities; Jena [Wilkinson, 2006], DB2-RDF [Bornea
et al., 2013]
Clustered property table: group together the properties that tend to
occur in the same (or similar) subjects
Property-class table: cluster the subjects with the same type of
property into one property table
Subject Property Object
mdb:film/2014 rdfs:label “The Shining”
mdb:film/2014 movie:director mdb:director/8476
mdb:film/2685 movie:director mdb:director/8476
mdb:film/2685 rdfs:label “A Clockwork Orange”
mdb:actor/29704 movie:actor name “Jack Nicholson”
. . . . . . . . .
Subject refs:label movie:director
mob:film/2014 “The Shining” mob:director/8476
mob:film/2685 “The Clockwork Orange” mob:director/8476
Subject movie:actor name
mdb:actor “Jack Nicholson”
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 54 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-138-2048.jpg)
![Property Tables
Grouping by entities; Jena [Wilkinson, 2006], DB2-RDF [Bornea
et al., 2013]
Clustered property table: group together the properties that tend to
occur in the same (or similar) subjects
Property-class table: cluster the subjects with the same type of
property into one property table
Subject Property Object
mdb:film/2014 rdfs:label “The Shining”
mdb:film/2014 movie:director mdb:director/8476
mdb:film/2685 movie:director mdb:director/8476
mdb:film/2685 rdfs:label “A Clockwork Orange”
mdb:actor/29704 movie:actor name “Jack Nicholson”
. . . . . . . . .
Subject refs:label movie:director
mob:film/2014 “The Shining” mob:director/8476
mob:film/2685 “The Clockwork Orange” mob:director/8476
Subject movie:actor name
mdb:actor “Jack Nicholson”
Advantages
Fewer joins
If the data is structured, we have a relational system – similar to
normalized relations
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 54 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-139-2048.jpg)
![Property Tables
Grouping by entities; Jena [Wilkinson, 2006], DB2-RDF [Bornea
et al., 2013]
Clustered property table: group together the properties that tend to
occur in the same (or similar) subjects
Property-class table: cluster the subjects with the same type of
property into one property table
Subject Property Object
mdb:film/2014 rdfs:label “The Shining”
mdb:film/2014 movie:director mdb:director/8476
mdb:film/2685 movie:director mdb:director/8476
mdb:film/2685 rdfs:label “A Clockwork Orange”
mdb:actor/29704 movie:actor name “Jack Nicholson”
. . . . . . . . .
Subject refs:label movie:director
mob:film/2014 “The Shining” mob:director/8476
mob:film/2685 “The Clockwork Orange” mob:director/8476
Subject movie:actor name
mdb:actor “Jack Nicholson”
Advantages
Fewer joins
If the data is structured, we have a relational system – similar to
normalized relations
Disadvantages
Potentially a lot of NULLs
Clustering is not trivial
Multi-valued properties are complicated
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 54 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-140-2048.jpg)
![Binary Tables
Grouping by properties: For each property, build a two-column table,
containing both subject and object, ordered by subjects [Abadi et al.,
2007, 2009]
Also called vertical partitioned tables
n two column tables (n is the number of unique properties in the data)
Subject Property Object
mdb:film/2014 rdfs:label “The Shining”
mdb:film/2014 movie:director mdb:director/8476
mdb:film/2685 movie:director mdb:director/8476
mdb:film/2685 rdfs:label “A Clockwork Orange”
mdb:actor/29704 movie:actor name “Jack Nicholson”
. . . . . . . . .
Subject Object
mdb:film/2014 mdb:director/8476
mdb:film/2685 mdb:director/8476
movie:director
Subject Object
mob:film/2014 “The Shining”
mob:film/2685 “The Clockwork Orange”
refs:label
Subject Object
mdb:actor/29704 “Jack Nicholson”
movie:actor name
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 55 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-141-2048.jpg)
![Binary Tables
Grouping by properties: For each property, build a two-column table,
containing both subject and object, ordered by subjects [Abadi et al.,
2007, 2009]
Also called vertical partitioned tables
n two column tables (n is the number of unique properties in the data)
Subject Property Object
mdb:film/2014 rdfs:label “The Shining”
mdb:film/2014 movie:director mdb:director/8476
mdb:film/2685 movie:director mdb:director/8476
mdb:film/2685 rdfs:label “A Clockwork Orange”
mdb:actor/29704 movie:actor name “Jack Nicholson”
. . . . . . . . .
Subject Object
mdb:film/2014 mdb:director/8476
mdb:film/2685 mdb:director/8476
movie:director
Subject Object
mob:film/2014 “The Shining”
mob:film/2685 “The Clockwork Orange”
refs:label
Subject Object
mdb:actor/29704 “Jack Nicholson”
movie:actor name
Advantages
Supports multi-valued properties
No NULLs
No clustering
Read only needed attributes (i.e. less I/O)
Good performance for subject-subject joins
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 55 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-142-2048.jpg)
![Binary Tables
Grouping by properties: For each property, build a two-column table,
containing both subject and object, ordered by subjects [Abadi et al.,
2007, 2009]
Also called vertical partitioned tables
n two column tables (n is the number of unique properties in the data)
Subject Property Object
mdb:film/2014 rdfs:label “The Shining”
mdb:film/2014 movie:director mdb:director/8476
mdb:film/2685 movie:director mdb:director/8476
mdb:film/2685 rdfs:label “A Clockwork Orange”
mdb:actor/29704 movie:actor name “Jack Nicholson”
. . . . . . . . .
Subject Object
mdb:film/2014 mdb:director/8476
mdb:film/2685 mdb:director/8476
movie:director
Subject Object
mob:film/2014 “The Shining”
mob:film/2685 “The Clockwork Orange”
refs:label
Subject Object
mdb:actor/29704 “Jack Nicholson”
movie:actor name
Advantages
Supports multi-valued properties
No NULLs
No clustering
Read only needed attributes (i.e. less I/O)
Good performance for subject-subject joins
Disadvantages
Not useful for subject-object joins
Expensive inserts
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 55 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-143-2048.jpg)
![Graph-based Approach
Answering SPARQL query ≡ subgraph matching using homomorphism
gStore [Zou et al., 2011, 2014], chameleon-db [Alu¸c et al., 2013]
?m ?d
movie:director
?name
rdfs:label
?b
movie:relatedBook
“Stanley Kubrick”
movie:director name
?r
rev:rating
FILTER(?r 4.0)
mdb:film/2014
“1980-05-23”
movie:initial release date
“The Shining”
refs:label
bm:books/0743424425
4.7
rev:rating
bm:offers/0743424425amazonOffer
geo:2635167
“United Kingdom”
gn:name
62348447
gn:population
mdb:actor/29704
“Jack Nicholson”
movie:actor name
mdb:film/3418
“The Passenger”
refs:label
mdb:film/1267
“The Last Tycoon”
refs:label
mdb:director/8476
“Stanley Kubrick”
movie:director name
mdb:film/2685
“A Clockwork Orange”
refs:label
mdb:film/424
“Spartacus”
refs:label
mdb:actor/30013
movie:relatedBook
scam:hasOffer
foaf:based near
movie:actor
movie:director
movie:actor
movie:actor movie:actor
movie:director movie:director
Subgraph
M
atching
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 56 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-144-2048.jpg)
![Graph-based Approach
Answering SPARQL query ≡ subgraph matching using homomorphism
gStore [Zou et al., 2011, 2014], chameleon-db [Alu¸c et al., 2013]
?m ?d
movie:director
?name
rdfs:label
?b
movie:relatedBook
“Stanley Kubrick”
movie:director name
?r
rev:rating
FILTER(?r 4.0)
mdb:film/2014
“1980-05-23”
movie:initial release date
“The Shining”
refs:label
bm:books/0743424425
4.7
rev:rating
bm:offers/0743424425amazonOffer
geo:2635167
“United Kingdom”
gn:name
62348447
gn:population
mdb:actor/29704
“Jack Nicholson”
movie:actor name
mdb:film/3418
“The Passenger”
refs:label
mdb:film/1267
“The Last Tycoon”
refs:label
mdb:director/8476
“Stanley Kubrick”
movie:director name
mdb:film/2685
“A Clockwork Orange”
refs:label
mdb:film/424
“Spartacus”
refs:label
mdb:actor/30013
movie:relatedBook
scam:hasOffer
foaf:based near
movie:actor
movie:director
movie:actor
movie:actor movie:actor
movie:director movie:director
Subgraph
M
atching
Advantages
Maintains the graph structure
Full set of queries can be handled
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 56 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-145-2048.jpg)
![Graph-based Approach
Answering SPARQL query ≡ subgraph matching using homomorphism
gStore [Zou et al., 2011, 2014], chameleon-db [Alu¸c et al., 2013]
?m ?d
movie:director
?name
rdfs:label
?b
movie:relatedBook
“Stanley Kubrick”
movie:director name
?r
rev:rating
FILTER(?r 4.0)
mdb:film/2014
“1980-05-23”
movie:initial release date
“The Shining”
refs:label
bm:books/0743424425
4.7
rev:rating
bm:offers/0743424425amazonOffer
geo:2635167
“United Kingdom”
gn:name
62348447
gn:population
mdb:actor/29704
“Jack Nicholson”
movie:actor name
mdb:film/3418
“The Passenger”
refs:label
mdb:film/1267
“The Last Tycoon”
refs:label
mdb:director/8476
“Stanley Kubrick”
movie:director name
mdb:film/2685
“A Clockwork Orange”
refs:label
mdb:film/424
“Spartacus”
refs:label
mdb:actor/30013
movie:relatedBook
scam:hasOffer
foaf:based near
movie:actor
movie:director
movie:actor
movie:actor movie:actor
movie:director movie:director
Subgraph
M
atching
Advantages
Maintains the graph structure
Full set of queries can be handled
Disadvantages
Graph pattern matching is expensive
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 56 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-146-2048.jpg)
![Adaptivity to Workload
Applications that rely on RDF data are increasingly popular and are
more varied [Verborgh et al., 2014]
Data that are being handled are far more heterogeneous [Duan et al.,
2011]
SPARQL queries are becoming more diverse [Arias et al., 2011] and
dynamic [Kirchberg et al., 2011]
An experiment [Alu¸c et al., 2014a]
No single system is a sole winner across all queries
No single system is the sole loser across all queries, either
There can be 2–5 orders of magnitude difference in the performance
(i.e., query execution time) between the best and the worst system for
a given query
The winner in one query may timeout in another
Performance difference widens as dataset size increases
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 65 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-170-2048.jpg)
![Adaptivity to Workload
Applications that rely on RDF data are increasingly popular and are
more varied [Verborgh et al., 2014]
Data that are being handled are far more heterogeneous [Duan et al.,
2011]
SPARQL queries are becoming more diverse [Arias et al., 2011] and
dynamic [Kirchberg et al., 2011]
An experiment [Alu¸c et al., 2014a]
No single system is a sole winner across all queries
No single system is the sole loser across all queries, either
There can be 2–5 orders of magnitude difference in the performance
(i.e., query execution time) between the best and the worst system for
a given query
The winner in one query may timeout in another
Performance difference widens as dataset size increases
Can existing systems cope with these trends – workload diversity
dynamism
No! [Alu¸c et al., 2014b]
Fragmented data
Suboptimal pruning by indexes
Unnecessarily large sets of intermediate result tuples
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 65 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-171-2048.jpg)
![Adaptivity to Workload
Applications that rely on RDF data are increasingly popular and are
more varied [Verborgh et al., 2014]
Data that are being handled are far more heterogeneous [Duan et al.,
2011]
SPARQL queries are becoming more diverse [Arias et al., 2011] and
dynamic [Kirchberg et al., 2011]
An experiment [Alu¸c et al., 2014a]
No single system is a sole winner across all queries
No single system is the sole loser across all queries, either
There can be 2–5 orders of magnitude difference in the performance
(i.e., query execution time) between the best and the worst system for
a given query
The winner in one query may timeout in another
Performance difference widens as dataset size increases
Our proposal: Idea behind chameleon-db
When designing and implementing an RDF data management system,
assume nothing about the workload upfront
Organize data dynamically and purely based on the workload
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 65 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-172-2048.jpg)
![Challenges
Physical Data Layout: As the workloads change, the way data are
grouped together may no longer be suitable
Hierarchical Clustering Algorithm [Alu¸c et al., 2015]
Tunable-LSH [Alu¸c et al., 2015]
Indexing: Indexing upfront is not a choice
Query Evaluation: Can we execute queries efficiently even when the
physical layout is constantly changing? [Alu¸c et al., 2015]
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 67 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-181-2048.jpg)
![chameleon-db
Prototype system [Alu¸c et al., 2013]
35,000 lines of code in C++ under Linux (plus code for SPARQL 1.0
parser)
Structural Index
...
Vertex Index
Spill Index
ClusterIndexStorageSystem
StorageAdvisor
Query
Engine Plan Generation Evaluation
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 68 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-182-2048.jpg)
![Distributed RDF Processing [Kaoudi and Manolescu, 2015]
RDF data warehouse is partitioned and distributed
RDF data D = {D1, . . . , Dn}
Allocate each Di to a site
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 71 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-190-2048.jpg)
![Distributed RDF Processing [Kaoudi and Manolescu, 2015]
RDF data warehouse is partitioned and distributed
RDF data D = {D1, . . . , Dn}
Allocate each Di to a site
Partitioning alternatives
Table-based (e.g., [Husain et al., 2011])
Graph-based (e.g., [Huang et al., 2011; Zhang et al., 2013])
Unit-based (e.g., [Gurajada et al., 2014; Lee and Liu, 2013])
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 71 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-191-2048.jpg)
![Distributed RDF Processing [Kaoudi and Manolescu, 2015]
RDF data warehouse is partitioned and distributed
RDF data D = {D1, . . . , Dn}
Allocate each Di to a site
Partitioning alternatives
Table-based (e.g., [Husain et al., 2011])
Graph-based (e.g., [Huang et al., 2011; Zhang et al., 2013])
Unit-based (e.g., [Gurajada et al., 2014; Lee and Liu, 2013])
SPARQL query decomposed Q = {Q1, . . . , Qk}
Distributed execution of {Q1, . . . , Qk} over {D1, . . . , Dn}
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 71 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-192-2048.jpg)
![Distributed RDF Processing [Kaoudi and Manolescu, 2015]
RDF data warehouse is partitioned and distributed
RDF data D = {D1, . . . , Dn}
Allocate each Di to a site
Partitioning alternatives
Table-based (e.g., [Husain et al., 2011])
Graph-based (e.g., [Huang et al., 2011; Zhang et al., 2013])
Unit-based (e.g., [Gurajada et al., 2014; Lee and Liu, 2013])
SPARQL query decomposed Q = {Q1, . . . , Qk}
Distributed execution of {Q1, . . . , Qk} over {D1, . . . , Dn}
High performance
Great for parallelizing centralized RDF data
May not be possible to re-partition and re-allocate Web data (i.e.,
LOD)
Query decomposition may not be easy
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 71 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-193-2048.jpg)
![Distributed RDF Processing – 2
Data summary-based approaches
Build summaries (index) for the distributed RDF datasets (e.g., [Atre
et al., 2010; Prasser et al., 2012])
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 72 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-194-2048.jpg)
![Distributed RDF Processing – 2
Data summary-based approaches
Build summaries (index) for the distributed RDF datasets (e.g., [Atre
et al., 2010; Prasser et al., 2012])
SPARQL query Q = {Q1, . . . , Qk}
Distributed execution of {Q1, . . . , Qk} using the data summary
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 72 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-195-2048.jpg)
![Distributed RDF Processing – 2
Data summary-based approaches
Build summaries (index) for the distributed RDF datasets (e.g., [Atre
et al., 2010; Prasser et al., 2012])
SPARQL query Q = {Q1, . . . , Qk}
Distributed execution of {Q1, . . . , Qk} using the data summary
No data re-partitioning and re-allocation
Have to scan the data at each site
Index over distributed data with maintenance concerns
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 72 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-196-2048.jpg)
![SPARQL Endpoint Federation
Consider only the SPARQL endpoints for query execution
No data re-partitioning/re-distribution
Consider D = D1 ∪ D2 ∪ . . . ∪ Dn; Di : SPARQL endpoint
SPARQL query decomposed Q = {Q1, . . . , Qk}
Distributed execution of {Q1, . . . , Qk} over {D1, . . . , Dn}
Systems
DARQ, FedX [Schwarte et al., 2011], SPLENDID [G¨orlitz and Staab,
2011], ANAPSID [Acosta et al., 2011]
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 73 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-199-2048.jpg)
![SPARQL Endpoint Federation
Consider only the SPARQL endpoints for query execution
No data re-partitioning/re-distribution
Consider D = D1 ∪ D2 ∪ . . . ∪ Dn; Di : SPARQL endpoint
SPARQL query decomposed Q = {Q1, . . . , Qk}
Distributed execution of {Q1, . . . , Qk} over {D1, . . . , Dn}
Systems
DARQ, FedX [Schwarte et al., 2011], SPLENDID [G¨orlitz and Staab,
2011], ANAPSID [Acosta et al., 2011]
Data integration approach
May be the only way to proceed if data is distributed
Not all RDF data storage points are SPARQL endpoints
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 73 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-200-2048.jpg)
![Linked Data Model [Hartig, 2012]
Web of Linked Data
Given a finite or countably infinite set D of Linked Documents, a Web of
Linked Data is a tuple W = (D, adoc, data) where:
D ⊆ D,
adoc is a partial mapping from URIs to D, and
data is a total mapping from D to finite sets of RDF triples.
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 81 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-209-2048.jpg)
![Linked Data Model [Hartig, 2012]
Web of Linked Data
Given a finite or countably infinite set D of Linked Documents, a Web of
Linked Data is a tuple W = (D, adoc, data) where:
D ⊆ D,
adoc is a partial mapping from URIs to D, and
data is a total mapping from D to finite sets of RDF triples.
Data Links
A Web of Linked Data W = (D, adoc, data)
contains a data link from document d ∈ D to
document d ∈ D if there exists a URI u such
that:
u is mentioned in an RDF triple
t ∈ data(d), and
d = adoc(u).
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 81 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-210-2048.jpg)
![Traversal Approaches
Discover relevant URIs recursively by
traversing (specific) data links at query
execution runtime [Hartig, 2013;
Ladwig and Tran, 2011]
Implements reachability-based query
semantics
Start from a set of seed URIs
Recursively follow and discover new
URIs
Important issue is selection of seed URIs
Retrieved data serves to discover new
URIs and to construct result
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 83 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-213-2048.jpg)
![Traversal Approaches
Discover relevant URIs recursively by
traversing (specific) data links at query
execution runtime [Hartig, 2013;
Ladwig and Tran, 2011]
Implements reachability-based query
semantics
Start from a set of seed URIs
Recursively follow and discover new
URIs
Important issue is selection of seed URIs
Retrieved data serves to discover new
URIs and to construct result
Advantages
Easy to implement.
No data structure to maintain.
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 83 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-214-2048.jpg)
![Traversal Approaches
Discover relevant URIs recursively by
traversing (specific) data links at query
execution runtime [Hartig, 2013;
Ladwig and Tran, 2011]
Implements reachability-based query
semantics
Start from a set of seed URIs
Recursively follow and discover new
URIs
Important issue is selection of seed URIs
Retrieved data serves to discover new
URIs and to construct result
Advantages
Easy to implement.
No data structure to maintain.
Disadvantages
Possibilities for parallelized data retrieval are limited
Repeated data retrieval introduces significant query latency.
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 83 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-215-2048.jpg)
![Index Approaches
Use pre-populated index to determine relevant URIs (and to avoid as
many irrelevant ones as possible)
Different index keys possible; e.g., triple patterns [Umbrich et al.,
2011]
Index entries a set of URIs
Indexed URIs may appear multiple times (i.e., associated with multiple
index keys)
Each URI in such an entry may be paired with a cardinality (utilized for
source ranking)
Key: tp Entry: {uri1, uri2, , urin}
GET urii
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 84 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-216-2048.jpg)
![Index Approaches
Use pre-populated index to determine relevant URIs (and to avoid as
many irrelevant ones as possible)
Different index keys possible; e.g., triple patterns [Umbrich et al.,
2011]
Index entries a set of URIs
Indexed URIs may appear multiple times (i.e., associated with multiple
index keys)
Each URI in such an entry may be paired with a cardinality (utilized for
source ranking)
Key: tp Entry: {uri1, uri2, , urin}
GET urii
Advantages
Data retrieval can be fully parallelized
Reduces the impact of data retrieval on query execution time
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 84 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-217-2048.jpg)
![Index Approaches
Use pre-populated index to determine relevant URIs (and to avoid as
many irrelevant ones as possible)
Different index keys possible; e.g., triple patterns [Umbrich et al.,
2011]
Index entries a set of URIs
Indexed URIs may appear multiple times (i.e., associated with multiple
index keys)
Each URI in such an entry may be paired with a cardinality (utilized for
source ranking)
Key: tp Entry: {uri1, uri2, , urin}
GET urii
Advantages
Data retrieval can be fully parallelized
Reduces the impact of data retrieval on query execution time
Disadvantages
Querying can only start after index construction
Depends on what has been selected for the index
Freshness may be an issue
Index maintenance
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 84 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-218-2048.jpg)
![Hybrid Approach
Perform a traversal-based execution using a prioritized list of URIs to
look up [Ladwig and Tran, 2010]
Initial seed from the pre-populated index
Non-seed URIs are ranked by a function based on information in the
index
New discovered URIs that are not in the index are ranked according
to number of referring documents
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 85 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-219-2048.jpg)
![References V
Lee, K. and Liu, L. (2013). Scaling queries over big rdf graphs with semantic hash
partitioning. Proc. VLDB Endowment, 6(14):1894–1905.
Li, F., Ooi, B. C., ¨Ozsu, M. T., and Wu, S. (2014). Distributed data management using
MapReduce. ACM Comput. Surv., 46(3):Article No. 31.
Low, Y., Gonzalez, J., Kyrola, A., Bickson, D., Guestrin, C., and Hellerstein, J. M.
(2012). Distributed graphlab: A framework for machine learning in the cloud. Proc.
VLDB Endowment, 5(8):716–727.
Malewicz, G., Austern, M. H., Bik, A. J. C., Dehnert, J. C., Horn, I., Leiser, N., and
Czajkowski, G. (2010). Pregel: a system for large-scale graph processing. In Proc.
ACM SIGMOD Int. Conf. on Management of Data, pages 135–146.
Michiardi, P. (2015). Introduction to spark internals. Slideshare. Available from:
http://www.slideshare.net/michiard/introduction-to-spark-internals?
qid=511145e7-79d7-41d8-a133-9e705d4933c3v=qf1b=from_search=11 [Last
retrieved: 9 July 2015].
Neumann, T. and Weikum, G. (2008). RDF-3X: a RISC-style engine for RDF. Proc.
VLDB Endowment, 1(1):647–659.
© M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 93 / 96](https://image.slidesharecdn.com/graphlecturelong-160217142439/75/Graph-lecture-227-2048.jpg)
Uploaded byM. Tamer Özsu
Graph lecture
The document provides an overview of graph data management and analysis, discussing the prevalence of graph data in various domains such as social networks and biological networks. It outlines different types of graph processing, including property graph processing and RDF graph querying, while addressing algorithmic classifications based on graph dynamism. Key focuses include online querying, offline analytics, and data warehousing methodologies for effective graph data management.
![[DSC Europe 25] Marko Krstic - Understanding the AI Threat Landscape - Risks,...](https://cdn.slidesharecdn.com/ss_thumbnails/tiyim1ins5jvbrvzpzla-2-251209104645-c69d3553-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Milan Sekuloski - Data, Defence, and Development: Cybersecuri...](https://cdn.slidesharecdn.com/ss_thumbnails/dfrkwwx4qly6atqpbl4z-4-251209104645-c3d4b0ca-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Uros Pesic - The Reality of AI in Marketing.pdf](https://cdn.slidesharecdn.com/ss_thumbnails/rtkodnmtycovsllvzsyn-9-251215095918-b0c6bfe3-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Miodrag Pesovic & Vladislav Radonjic - Federated Data Archite...](https://cdn.slidesharecdn.com/ss_thumbnails/gsbe3y5it5uhndi4e08e-1-251212103249-f1008e0c-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Debmalya Biswas - Agentification: the art of transforming man...](https://cdn.slidesharecdn.com/ss_thumbnails/r5azlggvtqiaiiusrqdr-4-251212103249-5a12c89b-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Behzad Hosseini - AI Agents in the Wild: Deploying Models tha...](https://cdn.slidesharecdn.com/ss_thumbnails/3qtejajvsjqrzwfept2c-10-251212103250-7f2b1068-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Branko Urosevic -Rethinking Financial Talent: Integrating Cod...](https://cdn.slidesharecdn.com/ss_thumbnails/8jjrus8ttko6qj64f58f-3-251212103250-642c6374-thumbnail.jpg?width=640&height=640&fit=bounds)
Graph lecture
- 1. An Overview ofGraph Data Management and Analysis M. Tamer ¨Ozsu University of Waterloo David R. Cheriton School of Computer Science © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 1 / 96
- 2. Graph Data areVery Common Internet © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 2 / 96
- 3. Graph Data areVery Common Social networks © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 2 / 96
- 4. Graph Data areVery Common Trade volumes and connections © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 2 / 96
- 5. Graph Data areVery Common Biological networks © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 2 / 96
- 6. Graph Data areVery Common As of September 2011 Music Brainz (zitgist) P20 Turismo de Zaragoza yovisto Yahoo! Geo Planet YAGO World Fact- book El Viajero Tourism WordNet (W3C) WordNet (VUA) VIVO UF VIVO Indiana VIVO Cornell VIAF URI Burner Sussex Reading Lists Plymouth Reading Lists UniRef UniProt UMBEL UK Post- codes legislation data.gov.uk Uberblic UB Mann- heim TWC LOGD Twarql transport data.gov. uk Traffic Scotland theses. fr Thesau- rus W totl.net Tele- graphis TCM Gene DIT Taxon Concept Open Library (Talis) tags2con delicious t4gm info Swedish Open Cultural Heritage Surge Radio Sudoc STW RAMEAU SH statistics data.gov. uk St. Andrews Resource Lists ECS South- ampton EPrints SSW Thesaur us Smart Link Slideshare 2RDF semantic web.org Semantic Tweet Semantic XBRL SW Dog Food Source Code Ecosystem Linked Data US SEC (rdfabout) Sears Scotland Geo- graphy Scotland Pupils & Exams Scholaro- meter WordNet (RKB Explorer) Wiki UN/ LOCODE Ulm ECS (RKB Explorer) Roma RISKS RESEX RAE2001 Pisa OS OAI NSF New- castle LAAS KISTI JISC IRIT IEEE IBM Eurécom ERA ePrints dotAC DEPLOY DBLP (RKB Explorer) Crime Reports UK Course- ware CORDIS (RKB Explorer) CiteSeer Budapest ACM riese Revyu research data.gov. ukRen. Energy Genera- tors reference data.gov. uk Recht- spraak. nl RDF ohloh Last.FM (rdfize) RDF Book Mashup Rådata nå! PSH Product Types Ontology Product DB PBAC Poké- pédia patents data.go v.uk Ox Points Ord- nance Survey Openly Local Open Library Open Cyc Open Corpo- rates Open Calais OpenEI Open Election Data Project Open Data Thesau- rus Ontos News Portal OGOLOD Janus AMP Ocean Drilling Codices New York Times NVD ntnusc NTU Resource Lists Norwe- gian MeSH NDL subjects ndlna my Experi- ment Italian Museums medu- cator MARC Codes List Man- chester Reading Lists Lotico Weather Stations London Gazette LOIUS Linked Open Colors lobid Resources lobid Organi- sations LEM Linked MDB LinkedL CCN Linked GeoData LinkedCT Linked User Feedback LOV Linked Open Numbers LODE Eurostat (Ontology Central) Linked EDGAR (Ontology Central) Linked Crunch- base lingvoj Lichfield Spen- ding LIBRIS Lexvo LCSH DBLP (L3S) Linked Sensor Data (Kno.e.sis) Klapp- stuhl- club Good- win Family National Radio- activity JP Jamendo (DBtune) Italian public schools ISTAT Immi- gration iServe IdRef Sudoc NSZL Catalog Hellenic PD Hellenic FBD Piedmont Accomo- dations GovTrack GovWILD Google Art wrapper gnoss GESIS GeoWord Net Geo Species Geo Names Geo Linked Data GEMET GTAA STITCH SIDER Project Guten- berg Medi Care Euro- stat (FUB) EURES Drug Bank Disea- some DBLP (FU Berlin) Daily Med CORDIS (FUB) Freebase flickr wrappr Fishes of Texas Finnish Munici- palities ChEMBL FanHubz Event Media EUTC Produc- tions Eurostat Europeana EUNIS EU Insti- tutions ESD stan- dards EARTh Enipedia Popula- tion (En- AKTing) NHS (En- AKTing) Mortality (En- AKTing) Energy (En- AKTing) Crime (En- AKTing) CO2 Emission (En- AKTing) EEA SISVU educatio n.data.g ov.uk ECS South- ampton ECCO- TCP GND Didactal ia DDC Deutsche Bio- graphie data dcs Music Brainz (DBTune) Magna- tune John Peel (DBTune) Classical (DB Tune) Audio Scrobbler (DBTune) Last.FM artists (DBTune) DB Tropes Portu- guese DBpedia dbpedia lite Greek DBpedia DBpedia data- open- ac-uk SMC Journals Pokedex Airports NASA (Data Incu- bator) Music Brainz (Data Incubator) Moseley Folk Metoffice Weather Forecasts Discogs (Data Incubator) Climbing data.gov.uk intervals Data Gov.ie data bnf.fr Cornetto reegle Chronic- ling America Chem2 Bio2RDF Calames business data.gov. uk Bricklink Brazilian Poli- ticians BNB UniSTS UniPath way UniParc Taxono my UniProt (Bio2RDF) SGD Reactome PubMed Pub Chem PRO- SITE ProDom Pfam PDB OMIM MGI KEGG Reaction KEGG Pathway KEGG Glycan KEGG Enzyme KEGG Drug KEGG Com- pound InterPro Homolo Gene HGNC Gene Ontology GeneID Affy- metrix bible ontology BibBase FTS BBC Wildlife Finder BBC Program mes BBC Music Alpine Ski Austria LOCAH Amster- dam Museum AGROV OC AEMET US Census (rdfabout) Media Geographic Publications Government Cross-domain Life sciences User-generated content Linked data © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 2 / 96 Linking Open Data cloud diagram, by Richard Cyganiak and Anja Jentzsch. http://lod-cloud.net/
- 7. Outline 1 Introduction –Graph Types 2 Property Graph Processing Classification Online querying Offline analytics 3 RDF Graph Querying Data Warehousing Distributed SPARQL Execution Linked Object Data Querying © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 3 / 96
- 8. Outline 1 Introduction –Graph Types 2 Property Graph Processing Classification Online querying Offline analytics 3 RDF Graph Querying Data Warehousing Distributed SPARQL Execution Linked Object Data Querying © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 4 / 96
- 9. Graph Types Property graph film2014 (initial release date, “1980-05-23”) (label, “The Shining”) books 0743424425 (rating, 4.7) offers 0743424425amazonOffer geo 2635167 (name, “United Kingdom”) (population, 62348447) actor 29704 (actor name, “Jack Nicholson”) film 3418 (label, “The Passenger”) film 1267 (label, “The Last Tycoon”) director 8476 (director name, “Stanley Kubrick”) film 2685 (label, “A Clockwork Orange”) film 424 (label, “Spartacus”) actor 30013 (relatedBook) (hasOffer) (based near) (actor) (director) (actor) (actor) (actor) (director) (director) © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 5 / 96
- 10. Graph Types RDF graph mdb:film/2014 “1980-05-23” movie:initialrelease date “The Shining” refs:label bm:books/0743424425 4.7 rev:rating bm:offers/0743424425amazonOffer geo:2635167 “United Kingdom” gn:name 62348447 gn:population mdb:actor/29704 “Jack Nicholson” movie:actor name mdb:film/3418 “The Passenger” refs:label mdb:film/1267 “The Last Tycoon” refs:label mdb:director/8476 “Stanley Kubrick” movie:director name mdb:film/2685 “A Clockwork Orange” refs:label mdb:film/424 “Spartacus” refs:label mdb:actor/30013 movie:relatedBook scam:hasOffer foaf:based near movie:actor movie:director movie:actor movie:actor movie:actor movie:director movie:director © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 5 / 96
- 11. Graph Types Property graph film2014 (initial release date, “1980-05-23”) (label, “The Shining”) books 0743424425 (rating, 4.7) offers 0743424425amazonOffer geo 2635167 (name, “United Kingdom”) (population, 62348447) actor 29704 (actor name, “Jack Nicholson”) film 3418 (label, “The Passenger”) film 1267 (label, “The Last Tycoon”) director 8476 (director name, “Stanley Kubrick”) film 2685 (label, “A Clockwork Orange”) film 424 (label, “Spartacus”) actor 30013 (relatedBook) (hasOffer) (based near) (actor) (director) (actor) (actor) (actor) (director) (director) Workload: Online queries and analytic workloads Query execution: Varies RDF graph mdb:film/2014 “1980-05-23” movie:initial release date “The Shining” refs:label bm:books/0743424425 4.7 rev:rating bm:offers/0743424425amazonOffer geo:2635167 “United Kingdom” gn:name 62348447 gn:population mdb:actor/29704 “Jack Nicholson” movie:actor name mdb:film/3418 “The Passenger” refs:label mdb:film/1267 “The Last Tycoon” refs:label mdb:director/8476 “Stanley Kubrick” movie:director name mdb:film/2685 “A Clockwork Orange” refs:label mdb:film/424 “Spartacus” refs:label mdb:actor/30013 movie:relatedBook scam:hasOffer foaf:based near movie:actor movie:director movie:actor movie:actor movie:actor movie:director movie:director Workload: SPARQL queries Query execution: subgraph matching by homomorphism © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 5 / 96
- 12. Outline 1 Introduction –Graph Types 2 Property Graph Processing Classification Online querying Offline analytics 3 RDF Graph Querying Data Warehousing Distributed SPARQL Execution Linked Object Data Querying © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 6 / 96
- 13. Outline 1 Introduction –Graph Types 2 Property Graph Processing Classification Online querying Offline analytics 3 RDF Graph Querying Data Warehousing Distributed SPARQL Execution Linked Object Data Querying © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 7 / 96
- 14. Classification [Ammar and¨Ozsu, 2015] Graph Dynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96
- 15. Classification [Ammar and¨Ozsu, 2015] Graph Dynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads Focus here is on the dynamism of the graphs in whether or not they change and how they change. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96
- 16. Classification [Ammar and¨Ozsu, 2015] Graph Dynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads Focus here is on the dynamism of the graphs in whether or not they change and how they change. Focus here is on the how algorithms behave as their input changes. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96
- 17. Classification [Ammar and¨Ozsu, 2015] Graph Dynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads Focus here is on the dynamism of the graphs in whether or not they change and how they change. Focus here is on the how algorithms behave as their input changes. The types of workloads that the approaches are designed to handle. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96
- 18. Classification [Ammar and¨Ozsu, 2015] Graph Dynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96
- 19. Classification [Ammar and¨Ozsu, 2015] Graph Dynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads Graphs do not change or we are not inter- ested in their changes – only a snapshot is considered. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96
- 20. Classification [Ammar and¨Ozsu, 2015] Graph Dynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads Graphs do not change or we are not inter- ested in their changes – only a snapshot is considered. Graphs change and we are interested in their changes. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96
- 21. Classification [Ammar and¨Ozsu, 2015] Graph Dynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads Graphs do not change or we are not inter- ested in their changes – only a snapshot is considered. Graphs change and we are interested in their changes. Dynamic graphs with high veloc- ity changes – not possible to see the entire graph at once. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96
- 22. Classification [Ammar and¨Ozsu, 2015] Graph Dynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads Graphs do not change or we are not inter- ested in their changes – only a snapshot is considered. Graphs change and we are interested in their changes. Dynamic graphs with high veloc- ity changes – not possible to see the entire graph at once. Dynamic graphs with un- known changes – requires re- discovery of the graph (e.g., LOD). © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96
- 23. Classification [Ammar and¨Ozsu, 2015] Graph Dynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96
- 24. Classification [Ammar and¨Ozsu, 2015] Graph Dynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads Computation accesses a portion of the graph and the results are computed for a subset of vertices; e.g., point- to-point shortest path, subgraph matching, reachability, SPARQL. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96
- 25. Classification [Ammar and¨Ozsu, 2015] Graph Dynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads Computation accesses a portion of the graph and the results are computed for a subset of vertices; e.g., point- to-point shortest path, subgraph matching, reachability, SPARQL. Computation accesses the entire graph and may require multiple iterations; e.g., PageR- ank, clustering, graph colouring, all pairs shortest path. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96
- 26. Classification [Ammar and¨Ozsu, 2015] Graph Dynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96
- 27. Classification [Ammar and¨Ozsu, 2015] Graph Dynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads Sees the en- tire input in advance. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96
- 28. Classification [Ammar and¨Ozsu, 2015] Graph Dynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads Sees the en- tire input in advance. Sees the input piece-meal as it executes. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96
- 29. Classification [Ammar and¨Ozsu, 2015] Graph Dynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads Sees the en- tire input in advance. Sees the input piece-meal as it executes. One-pass on- line algorithm with limited memory. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96
- 30. Classification [Ammar and¨Ozsu, 2015] Graph Dynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads Sees the en- tire input in advance. Sees the input piece-meal as it executes. One-pass on- line algorithm with limited memory. Online algo- rithm with some info about forth- coming input. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96
- 31. Classification [Ammar and¨Ozsu, 2015] Graph Dynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads Sees the en- tire input in advance. Sees the input piece-meal as it executes. One-pass on- line algorithm with limited memory. Online algo- rithm with some info about forth- coming input. Sees the en- tire input in advance, which may change; an- swers computed as change oc- curs. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96
- 32. Classification [Ammar and¨Ozsu, 2015] Graph Dynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads Sees the en- tire input in advance. Sees the input piece-meal as it executes. One-pass on- line algorithm with limited memory. Online algo- rithm with some info about forth- coming input. Sees the en- tire input in advance, which may change; an- swers computed as change oc- curs. Similar to dy- namic, but computation happens in batches of © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 8 / 96
- 33. Example Design Points GraphDynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads Compute the query result/perform analytic computation over the graph as it exists. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 9 / 96
- 34. Example Design Points GraphDynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads Compute the query result/perform analytic computation over the graph as it is revealed. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 9 / 96
- 35. Example Design Points GraphDynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads Compute the query result/perform analytic computation on each snap- shot from scratch. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 9 / 96
- 36. Example Design Points GraphDynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads Continuously compute the query result/perform analytic computation as the input changes. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 9 / 96
- 37. Example Design Points GraphDynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads Compute the query result/perform analytic computation after a batch of input changes. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 9 / 96
- 38. Example Design Points– Not all alternatives make sense Graph Dynamism Static Graphs Dynamic Graphs Streaming Graphs Evolving Graphs Algorithm Types Offline Online Streaming Incremental Dynamic Batch Dynamic Workload Types Online Queries Analytics Workloads Dynamic (or batch-dynamic) algorithms do not make sense for static graphs. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 10 / 96
- 39. Graph Processing Systems SystemMemory/ Disk Architecture Computing paradigm Supported Workloads Hadoop Disk Parallel/Distributed MapReduce Analytical Haloop Disk Parallel/Distributed MapReduce Analytical Pegasus Disk Parallel/Distributed MapReduce Analytical GraphX Disk Parallel/Distributed MapReduce (Spark) Analytical Pregel/Giraph Memory Parallel/Distributed Vertex-Centric Analytical GraphLab Memory Parallel/Distributed Vertex-Centric Analytical GraphChi Disk Single machine Vertex-Centric Analytical Stream Disk Single machine Edge-Centric Analytical Trinity Memory Parallel/Distributed Flexible using K-V store on DSM Online & Analytical Titan Disk Parallel/Distributed K-V store (Cassandra) Online Neo4J Disk Single machine Procedural/ Linked-list Online © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 11 / 96
- 40. Graph Workloads Online graphquerying Reachability Single source shortest-path Subgraph matching SPARQL queries Offline graph analytics PageRank Clustering Strongly connected components Diameter finding Graph colouring All pairs shortest path Graph pattern mining Machine learning algorithms (Belief propagation, Gaussian non-negative matrix factorization) © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 12 / 96
- 41. Outline 1 Introduction –Graph Types 2 Property Graph Processing Classification Online querying Offline analytics 3 RDF Graph Querying Data Warehousing Distributed SPARQL Execution Linked Object Data Querying © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 13 / 96
- 42. Reachability Queries film 2014 (initialrelease date, “1980-05-23”) (label, “The Shining”) books 0743424425 (rating, 4.7) offers 0743424425amazonOffer geo 2635167 (name, “United Kingdom”) (population, 62348447) actor 29704 (actor name, “Jack Nicholson”) film 3418 (label, “The Passenger”) film 1267 (label, “The Last Tycoon”) director 8476 (director name, “Stanley Kubrick”) film 2685 (label, “A Clockwork Orange”) film 424 (label, “Spartacus”) actor 30013 (relatedBook) (hasOffer) (based near) (actor) (director) (actor) (actor) (actor) (director) (director) © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 14 / 96
- 43. Reachability Queries film 2014 (initialrelease date, “1980-05-23”) (label, “The Shining”) books 0743424425 (rating, 4.7) offers 0743424425amazonOffer geo 2635167 (name, “United Kingdom”) (population, 62348447) actor 29704 (actor name, “Jack Nicholson”) film 3418 (label, “The Passenger”) film 1267 (label, “The Last Tycoon”) director 8476 (director name, “Stanley Kubrick”) film 2685 (label, “A Clockwork Orange”) film 424 (label, “Spartacus”) actor 30013 (relatedBook) (hasOffer) (based near) (actor) (director) (actor) (actor) (actor) (director) (director) Can you reach film 1267 from film 2014? © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 14 / 96
- 44. Reachability Queries film 2014 (initialrelease date, “1980-05-23”) (label, “The Shining”) books 0743424425 (rating, 4.7) offers 0743424425amazonOffer geo 2635167 (name, “United Kingdom”) (population, 62348447) actor 29704 (actor name, “Jack Nicholson”) film 3418 (label, “The Passenger”) film 1267 (label, “The Last Tycoon”) director 8476 (director name, “Stanley Kubrick”) film 2685 (label, “A Clockwork Orange”) film 424 (label, “Spartacus”) actor 30013 (relatedBook) (hasOffer) (based near) (actor) (director) (actor) (actor) (actor) (director) (director) Is there a book whose rating is > 4.0 associated with a film that was directed by Stanley Kubrick? © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 14 / 96
- 45. Reachability Queries Think ofFacebook graph and finding friends of friends. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 14 / 96
- 46. Subgraph Matching ?m ?d movie:director ?name rdfs:label ?b movie:relatedBook “StanleyKubrick” movie:director name ?r rev:rating FILTER(?r > 4.0) mdb:film/2014 “1980-05-23” movie:initial release date “The Shining” refs:label bm:books/0743424425 4.7 rev:rating bm:offers/0743424425amazonOffer geo:2635167 “United Kingdom” gn:name 62348447 gn:population mdb:actor/29704 “Jack Nicholson” movie:actor name mdb:film/3418 “The Passenger” refs:label mdb:film/1267 “The Last Tycoon” refs:label mdb:director/8476 “Stanley Kubrick” movie:director name mdb:film/2685 “A Clockwork Orange” refs:label mdb:film/424 “Spartacus” refs:label mdb:actor/30013 movie:relatedBook scam:hasOffer foaf:based near movie:actor movie:director movie:actor movie:actor movie:actor movie:director movie:director Su bgraphMatching © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 15 / 96
- 47. Outline 1 Introduction –Graph Types 2 Property Graph Processing Classification Online querying Offline analytics 3 RDF Graph Querying Data Warehousing Distributed SPARQL Execution Linked Object Data Querying © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 16 / 96
- 48. PageRank Computation A webpage is important if it is pointed to by other important pages. P1 P2 P3 P5P6 P4 r(Pi ) = Pj ∈BPi r(Pj ) |FPj | r(P2) = r(P1) 2 + r(P3) 3 rk+1(Pi ) = Pj ∈BPi rk(Pj ) |FPj | BPi : in-neighbours of Pi FPi : out-neighbours of Pi © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 17 / 96
- 49. PageRank Computation A webpage is important if it is pointed to by other important pages. P1 P2 P3 P5P6 P4 rk+1(Pi ) = Pj ∈BPi rk(Pj ) |FPj | Iteration 0 Iteration 1 Iteration 2 Rank at Iter. 2 r0(P1) = 1/6 r1(P1) = 1/18 r2(P1) = 1/36 5 r0(P2) = 1/6 r1(P2) = 5/36 r2(P2) = 1/18 4 r0(P3) = 1/6 r1(P3) = 1/12 r2(P3) = 1/36 5 r0(P4) = 1/6 r1(P4) = 1/4 r2(P4) = 17/72 1 r0(P5) = 1/6 r1(P5) = 5/36 r2(P5) = 11/72 3 r0(P6) = 1/6 r1(P6) = 1/6 r2(P6) = 14/72 2 Iterative processing. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 17 / 96
- 50. Some Alternative ComputationalModels for Offline Analytics Vertex-centric (Scatter-Gather) Specify (a) computation at each vertex, and (b) communication with neighbour vertices Synchronous – Pregel [Malewicz et al., 2010], Giraph Asynchronous – GraphLab [Low et al., 2012] © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 18 / 96
- 51. Some Alternative ComputationalModels for Offline Analytics Vertex-centric (Scatter-Gather) Specify (a) computation at each vertex, and (b) communication with neighbour vertices Synchronous – Pregel [Malewicz et al., 2010], Giraph Asynchronous – GraphLab [Low et al., 2012] Block-centric Similar to vertex-centric but on blocks for communication Connected subgraph of the graph Blogel [Yan et al., 2014] © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 18 / 96
- 52. Some Alternative ComputationalModels for Offline Analytics Vertex-centric (Scatter-Gather) Specify (a) computation at each vertex, and (b) communication with neighbour vertices Synchronous – Pregel [Malewicz et al., 2010], Giraph Asynchronous – GraphLab [Low et al., 2012] Block-centric Similar to vertex-centric but on blocks for communication Connected subgraph of the graph Blogel [Yan et al., 2014] MapReduce Need to save in HDFS intermediate results of each iteration – both good and bad Hadoop, Haloop [Bu et al., 2012] © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 18 / 96
- 53. Some Alternative ComputationalModels for Offline Analytics Vertex-centric (Scatter-Gather) Specify (a) computation at each vertex, and (b) communication with neighbour vertices Synchronous – Pregel [Malewicz et al., 2010], Giraph Asynchronous – GraphLab [Low et al., 2012] Block-centric Similar to vertex-centric but on blocks for communication Connected subgraph of the graph Blogel [Yan et al., 2014] MapReduce Need to save in HDFS intermediate results of each iteration – both good and bad Hadoop, Haloop [Bu et al., 2012] Modified MapReduce Based on Spark [Zaharia et al., 2010; Zaharia, 2016] Keep intermediate states in memory Provide fault-tolerance by keeping lineage GraphX [Gonzalez et al., 2014] © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 18 / 96
- 54. Vertex-Centric Computation “Think likea vertex” vertex_scatter(vertex v) Push local computation to neighbours on the out-bound edges vertex_gather(vertex v) Gather local computation from neighbours on the in-bound edges Continue until all vertices are inactive ? © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 19 / 96
- 55. Vertex-Centric Computation “Think likea vertex” vertex_scatter(vertex v) Push local computation to neighbours on the out-bound edges vertex_gather(vertex v) Gather local computation from neighbours on the in-bound edges Continue until all vertices are inactive Vertex state machine ? Active Inactive Vote halt Message received © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 19 / 96
- 56. Synchronous Vertex-Centric Computation Computation ©M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 20 / 96
- 57. Synchronous Vertex-Centric Computation Superstep1 Superstep 2 Superstep 3 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 20 / 96
- 58. Synchronous Vertex-Centric Computation Machine1 Machine 2 Machine 3 Communication Barrier Each machine performs vertex-centric computation on its graph partition Superstep 1 Superstep 2 Superstep 3 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 20 / 96
- 59. Synchronous Vertex-Centric Computation Machine1 Machine 2 Machine 3 Machine 1 Machine 2 Machine 3 Communication Barrier Each machine performs vertex-centric computation on its graph partition Communication Barrier Superstep 1 Superstep 2 Superstep 3 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 20 / 96
- 60. Synchronous Vertex-Centric Computation Machine1 Machine 2 Machine 3 Machine 1 Machine 2 Machine 3 Machine 1 Machine 2 Machine 3 Communication Barrier Each machine performs vertex-centric computation on its graph partition Communication Barrier Superstep 1 Superstep 2 Superstep 3 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 20 / 96
- 61. Asynchronous Vertex-Centric Computation Nocommunication barriers. Uses the most recent vertex values. Machine 1 Machine 2 Machine 3 Machine 1 Machine 2 Machine 3 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 21 / 96
- 62. Asynchronous Vertex-Centric Computation Nocommunication barriers. Uses the most recent vertex values. Implemented via distributed locking Machine 1 Machine 2 Machine 3 Machine 1 Machine 2 Machine 3 v0 v1 v2 v3 v4 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 21 / 96
- 63. Asynchronous Vertex-Centric Computation Nocommunication barriers. Uses the most recent vertex values. Implemented via distributed locking Machine 1 Machine 2 Machine 3 Machine 1 Machine 2 Machine 3 v0 v1 v2 v3 v4 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 21 / 96
- 64. Asynchronous Vertex-Centric Computation Nocommunication barriers. Uses the most recent vertex values. Implemented via distributed locking Machine 1 Machine 2 Machine 3 Machine 1 Machine 2 Machine 3 v0 v1 v2 v3 v4 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 21 / 96
- 65. Asynchronous Vertex-Centric Computation Nocommunication barriers. Uses the most recent vertex values. Implemented via distributed locking Machine 1 Machine 2 Machine 3 Machine 1 Machine 2 Machine 3 v0 v1 v2 v3 v4 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 21 / 96
- 66. Asynchronous Vertex-Centric Computation Nocommunication barriers. Uses the most recent vertex values. Implemented via distributed locking Machine 1 Machine 2 Machine 3 Machine 1 Machine 2 Machine 3 v0 v1 v2 v3 v4 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 21 / 96
- 67. Summary of anExperiment [Han et al., 2014] A large study comparing Giraph, GraphLab, GPS, Mizan. 1 Giraph scales better across graphs; GraphLab scales better across more machines. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 22 / 96
- 68. Summary of anExperiment [Han et al., 2014] A large study comparing Giraph, GraphLab, GPS, Mizan. 1 Giraph scales better across graphs; GraphLab scales better across more machines. 64 machines TW UK Giraph (byte array) 5.8GB 7.0GB GraphLab (sync) 4.5GB 14GB TW 16 machines 128 machines Giraph (byte array) 8.5GB 5.8GB GraphLab (sync) 11GB 3.3GB © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 22 / 96
- 69. Summary of anExperiment [Han et al., 2014] A large study comparing Giraph, GraphLab, GPS, Mizan. 1 Giraph scales better across graphs; GraphLab scales better across more machines. 2 Distributed locking for asynchronous execution is not scalable – Performance degrades as more machines are used due to lock contention, termination scheme, lack of message batching © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 22 / 96
- 70. Summary of anExperiment [Han et al., 2014] A large study comparing Giraph, GraphLab, GPS, Mizan. 1 Giraph scales better across graphs; GraphLab scales better across more machines. 2 Distributed locking for asynchronous execution is not scalable – Performance degrades as more machines are used due to lock contention, termination scheme, lack of message batching 3 Graph storage should be memory and mutation efficient. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 22 / 96
- 71. Summary of anExperiment [Han et al., 2014] A large study comparing Giraph, GraphLab, GPS, Mizan. 1 Giraph scales better across graphs; GraphLab scales better across more machines. 2 Distributed locking for asynchronous execution is not scalable – Performance degrades as more machines are used due to lock contention, termination scheme, lack of message batching 3 Graph storage should be memory and mutation efficient. No Mutations Time Memory Byte array Hash map With Mutations (DMST) Time Memory Byte array Hash map © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 22 / 96
- 72. Summary of anExperiment [Han et al., 2014] A large study comparing Giraph, GraphLab, GPS, Mizan. 1 Giraph scales better across graphs; GraphLab scales better across more machines. 2 Distributed locking for asynchronous execution is not scalable – Performance degrades as more machines are used due to lock contention, termination scheme, lack of message batching 3 Graph storage should be memory and mutation efficient. 4 Message processing optimizations are very important. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 22 / 96
- 73. Summary of anExperiment [Han et al., 2014] A large study comparing Giraph, GraphLab, GPS, Mizan. 1 Giraph scales better across graphs; GraphLab scales better across more machines. 2 Distributed locking for asynchronous execution is not scalable – Performance degrades as more machines are used due to lock contention, termination scheme, lack of message batching 3 Graph storage should be memory and mutation efficient. 4 Message processing optimizations are very important. 5 Workloads have different resource demands Algorithm CPU Memory Network PageRank Medium Medium High SSSP Low Low Low WCC Low Medium Medium DMST High High Medium © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 22 / 96
- 74. Block-Centric Computation Blogel [Yanet al., 2014]: “Think like a block”; also “think like a graph” [Tian et al., 2013] Vertex-centric assumes all vertices communicate over the network; this is not efficient Read-world graphs have skewed vertex degree distribution Common in power-law graphs Problem: imbalanced communication workloads Real-world graphs have large diameters Common in road networks, web graphs, terrain meshes Problem: one superstep per hop ⇒ too many supersteps Real-world graphs have high average vertex degree Common in social networks, mobile communication networks Problem: heavy average communication workloads © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 23 / 96
- 75. Blogel Principles Exploit thepartitioning of the graph Message exchanges only among blocks Block: a connected subgraph of the graph Within a block, run a serial in-memory algorithm; no need to follow a BSP model © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 24 / 96
- 76. Benefits of Block-CentricComputation High-degree vertices inside a block send no messages Fewer number of supersteps Fewer number of blocks than vertices © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 25 / 96
- 77. Example: Weakly ConnectedComponent Algorithm exchanges vertex id’s with neighbours id(vi ) ← min{vi , vj , . . . , vk} where vj , . . . , vk are neighbours of vi Vertex-centric requires every vertex sends to its neighbours until every vertex is reached Block-centric needs two iterations: 1 All vertices in partition A exchange ids; X and Y send ids to neighbours in partition B 2 All vertices in partition B exchange ids A B 0 X Y © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 26 / 96
- 78. Block Construction The partitioningalgorithm needs to maximize number of vertices that have all their edges in the same partition Hash partitioning is not suitable because many vertices will probably have at least one cut-edge URL partitioner For web graphs: based on domain names of web page nodes 2D partitioner For spatial networks: based on coordinates of node Graph Voronoi diagram partitioner For general graphs © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 27 / 96
- 79. MapReduce Basics [Liet al., 2014] For data analysis of very large data sets Highly dynamic, irregular, schemaless, etc. SQL too heavy “Embarrassingly parallel problems” New, simple parallel programming model Data structured as (key, value) pairs E.g. (doc-id, content), (word, count), etc. Functional programming style with two functions to be given: Map(k1,v1) → list(k2,v2) Reduce(k2, list (v2)) → list(v3) Implemented on a distributed file system (e.g., Google File System) on very large clusters © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 28 / 96
- 80. MapReduce Processing ... Inputdataset Map Map Map Map (k1, v) (k2,v) (k2, v) (k2, v) (k1, v) (k1, v) (k2, v) Group by k Group by k (k1, (v, v, v)) (k1, (v, v, v, v)) Reduce Reduce Outputdataset © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 29 / 96
- 81. MapReduce Architecture Scheduler Master Input Module MapModule Combine Module Partition Module Map Process Worker Input Module Map Module Combine Module Partition Module Map Process Worker Input Module Map Module Combine Module Partition Module Map Process Worker Group Module Reduce Module Output Module Reduce Process Worker Group Module Reduce Module Output Module Reduce Process Worker © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 30 / 96
- 82. Execution Flow withArchitecture [Dean and Ghemawat, 2008] MapReduce: Simplified Data Processing on Large Clusters split 0 split 1 split 2 split 3 split 4 (1) fork (3) read (4) local write (1) fork (1) fork (6) write worker worker worker Master User Program output file 0 output file 1 worker worker (2) assign map (2) assign reduce (5) remote (5) read Input files Map phasr Intermediate files (on local disks) Reduce phase Output files © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 31 / 96
- 83. Hadoop Most popular MapReduceimplementation – developed by Yahoo! Two components Processing engine HDFS: Hadoop Distributed Storage System – others possible Can be deployed on the same machine or on different machines Processes Job tracker: hosted on the master node and implements the schedule Task tracker: hosted on the worker nodes and accepts tasks from job tracker and executes them HDFS Name node: stores how data are partitioned, monitors the status of data nodes, and data dictionary Data node: Stores and manages data chunks assigned to it Task Tracker Job Tracker Task Tracker Data Node Name Node Data Node Worker 1 Name Node Worker n MapReduce HDFS © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 32 / 96
- 84. HaLoop [Bu etal., 2012] Overcome MapReduce shortcomings for iterative jobs Having to save data in HDFS in between each iteration Checking the fixpoint requires a new job at each iteration Scheduler change: assign to the same machine the map reduce tasks that occur in different iterations but access the same data Cache invariant data Cache reduce output to easily check for fixpoint © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 33 / 96
- 85. Spark System MapReduce doesnot perform well in iterative computations Workflow model is acyclic Have to write to HDFS after each iteration and have to read from HDFS at the beginning of next iteration © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 34 / 96
- 86. Spark System MapReduce doesnot perform well in iterative computations Workflow model is acyclic Have to write to HDFS after each iteration and have to read from HDFS at the beginning of next iteration Spark objectives Better support for iterative programs Provide a complete ecosystem Similar abstraction (to MapReduce) for programming Maintain MapReduce fault-tolerance and scalability © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 34 / 96
- 87. Spark System MapReduce doesnot perform well in iterative computations Workflow model is acyclic Have to write to HDFS after each iteration and have to read from HDFS at the beginning of next iteration Spark objectives Better support for iterative programs Provide a complete ecosystem Similar abstraction (to MapReduce) for programming Maintain MapReduce fault-tolerance and scalability Fundamental concepts RDD: Reliable Distributed Datasets Caching of working set Maintaining lineage for fault-tolerance © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 34 / 96
- 88. Spark Ecosystem [Michiardi,2015] Native Spark Apps Spark SQL Spark Streaming MLlib (machine learning) GraphX (graph processing) Apache Spark © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 35 / 96
- 89. Spark Programming Model[Zaharia et al., 2010, 2012] HDFS Create RDD · · · RDD Cache? Cache Yes Transform RDD? No Process No Transform Yes HDFS © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 36 / 96
- 90. Spark Programming Model[Zaharia et al., 2010, 2012] HDFS Create RDD · · · RDD Cache? Cache Yes Transform RDD? No Process No Transform Yes HDFS Each transform generates a new RDD that may also be cached or processed © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 36 / 96
- 91. Spark Programming Model[Zaharia et al., 2010, 2012] HDFS Create RDD · · · RDD Cache? Cache Yes Transform RDD? No Process No Transform Yes HDFS Each transform generates a new RDD that may also be cached or processed Created from HDFS or parallelized arrays; Partitioned across worker machines; May be made persistent lazily; © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 36 / 96
- 92. Spark Programming Model[Zaharia et al., 2010, 2012] HDFS Create RDD · · · RDD Cache? Cache Yes Transform RDD? No Process No Transform Yes HDFS Each transform generates a new RDD that may also be cached or processed Created from HDFS or parallelized arrays; Partitioned across worker machines; May be made persistent lazily; Processing done on one of the RDDs; Done in parallel across workers; First processing on a RDD is from disk; Subsequent processing of the same RDD from cache © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 36 / 96
- 93. Example – LogMining [Zaharia et al., 2010, 2012] Load log messages from a file system, create a new file by filtering the error messages, read this file into memory, then interactively search for various patterns Driver WorkerWorkerWorker Block 1 Block 2 Block 3 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 37 / 96
- 94. Example – LogMining [Zaharia et al., 2010, 2012] Load log messages from a file system, create a new file by filtering the error messages, read this file into memory, then interactively search for various patterns lines = spark.textFile(hdfs://...) CreateRDD Driver WorkerWorkerWorker Block 1 Block 2 Block 3 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 37 / 96
- 95. Example – LogMining [Zaharia et al., 2010, 2012] Load log messages from a file system, create a new file by filtering the error messages, read this file into memory, then interactively search for various patterns lines = spark.textFile(hdfs://...) errors = lines.filter( .startsWith(ERROR)) Transform RDD Driver WorkerWorkerWorker Block 1 Block 2 Block 3 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 37 / 96
- 96. Example – LogMining [Zaharia et al., 2010, 2012] Load log messages from a file system, create a new file by filtering the error messages, read this file into memory, then interactively search for various patterns lines = spark.textFile(hdfs://...) errors = lines.filter( .startsWith(ERROR)) messages = errors.map( .split(‘t ’)(2)) Another transform Driver WorkerWorkerWorker Block 1 Block 2 Block 3 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 37 / 96
- 97. Example – LogMining [Zaharia et al., 2010, 2012] Load log messages from a file system, create a new file by filtering the error messages, read this file into memory, then interactively search for various patterns lines = spark.textFile(hdfs://...) errors = lines.filter( .startsWith(ERROR)) messages = errors.map( .split(‘t ’)(2)) cachedMsgs = messages.cache() Cache results Driver WorkerWorkerWorker Block 1 Block 2 Block 3 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 37 / 96
- 98. Example – LogMining [Zaharia et al., 2010, 2012] Load log messages from a file system, create a new file by filtering the error messages, read this file into memory, then interactively search for various patterns lines = spark.textFile(hdfs://...) errors = lines.filter( .startsWith(ERROR)) messages = errors.map( .split(‘t ’)(2)) cachedMsgs = messages.cache() cachedMsgs.filter( .contains(foo)).count Action Driver WorkerWorkerWorker Block 1 Block 2 Block 3 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 37 / 96
- 99. Example – LogMining [Zaharia et al., 2010, 2012] Load log messages from a file system, create a new file by filtering the error messages, read this file into memory, then interactively search for various patterns lines = spark.textFile(hdfs://...) errors = lines.filter( .startsWith(ERROR)) messages = errors.map( .split(‘t ’)(2)) cachedMsgs = messages.cache() cachedMsgs.filter( .contains(foo)).count Driver WorkerWorkerWorker Block 1 Block 2 Block 3 Tasks © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 37 / 96
- 100. Example – LogMining [Zaharia et al., 2010, 2012] Load log messages from a file system, create a new file by filtering the error messages, read this file into memory, then interactively search for various patterns lines = spark.textFile(hdfs://...) errors = lines.filter( .startsWith(ERROR)) messages = errors.map( .split(‘t ’)(2)) cachedMsgs = messages.cache() cachedMsgs.filter( .contains(foo)).count Driver WorkerWorkerWorker Block 1 Block 2 Block 3 TasksResults © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 37 / 96
- 101. Example – LogMining [Zaharia et al., 2010, 2012] Load log messages from a file system, create a new file by filtering the error messages, read this file into memory, then interactively search for various patterns lines = spark.textFile(hdfs://...) errors = lines.filter( .startsWith(ERROR)) messages = errors.map( .split(‘t ’)(2)) cachedMsgs = messages.cache() cachedMsgs.filter( .contains(foo)).count Driver WorkerWorkerWorker Block 1 Block 2 Block 3 TasksResults Cache Cache Cache © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 37 / 96
- 102. Example – LogMining [Zaharia et al., 2010, 2012] Load log messages from a file system, create a new file by filtering the error messages, read this file into memory, then interactively search for various patterns lines = spark.textFile(hdfs://...) errors = lines.filter( .startsWith(ERROR)) messages = errors.map( .split(‘t ’)(2)) cachedMsgs = messages.cache() cachedMsgs.filter( .contains(foo)).count cachedMsgs.filter( .contains(bar)).count Another Action accesses cache Driver WorkerWorkerWorker Block 1 Block 2 Block 3 TasksResults Cache Cache Cache © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 37 / 96
- 103. RDD and Processing HDFS lines= spark.textFile(hdfs://...) lines Error, msg1 Warn, msg2 Error, msg1 Info, msg8 Warn, msg2 Info, msg8 Error, msg3 Info, msg5 Info, msg5 Error, msg4 Warn, msg9 Error, msg1 errors errors = lines.filter( .startsWith(ERROR)) Error, msg1 Error, msg1 Error, msg3 Error, msg4 Error, msg1 messages messages = errors.map .split(‘t ’)(2) msg1 msg1 msg3 msg4 msg1 Thesearenotyetgenerated © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 38 / 96
- 104. RDD and Processing lines errors messages msg1 msg1 msg3msg4 msg1 lines messages.filter( .contains(foo)).count errors messages msg1 msg1 msg3 msg4 msg1 NowtheRDDsarematerialized; Commandnotyetexecuted Driver messages.filter( .contains(foo)).count © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 38 / 96
- 105. GraphX [Gonzalez etal., 2014] Built on top of Spark Objective is to combine data analytics with graph processing Unify computation on tables and graphs Carefully convert graph to tabular representation Native GraphX API or can accommodate vertex-centric computation Native Spark Apps Spark SQL Spark Streaming MLlib (machine learning) GraphX (graph processing) Apache Spark Vertex- centric API AppApp App App © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 39 / 96
- 106. GraphX: Representation ofGraphs as Tables A B C D E F G H I J © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 40 / 96
- 107. GraphX: Representation ofGraphs as Tables Partition 1 Partition 2 A B C D E F G H I J Edge-disjoint partitioning © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 40 / 96
- 108. GraphX: Representation ofGraphs as Tables Partition 1 Partition 2 Machine1Machine2 Vertex Table (RDD) v-prop:vertex prop. A B C D E F G H I J Edge-disjoint partitioning A v-prop B v-prop ... I v-prop D v-prop E v-prop F v-prop J v-prop © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 40 / 96
- 109. GraphX: Representation ofGraphs as Tables Partition 1 Partition 2 Machine1Machine2 Vertex Table (RDD) v-prop:vertex prop. Edge Table (RDD) e-prop:edge prop. A B C D E F G H I J Edge-disjoint partitioning A v-prop B v-prop ... I v-prop D v-prop E v-prop F v-prop J v-prop A e-prop B A e-prop C ... F e-prop G A e-prop D A e-prop E ... E e-prop F © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 40 / 96
- 110. GraphX: Representation ofGraphs as Tables Partition 1 Partition 2 Machine1Machine2 Vertex Table (RDD) v-prop:vertex prop. Edge Table (RDD) e-prop:edge prop. A B C D E F G H I J Edge-disjoint partitioning A v-prop B v-prop ... I v-prop D v-prop E v-prop F v-prop J v-prop A e-prop B A e-prop C ... F e-prop G A e-prop D A e-prop E ... E e-prop F Joining vertices and edges Move vertices to edges © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 40 / 96
- 111. GraphX: Representation ofGraphs as Tables Partition 1 Partition 2 Machine1Machine2 Vertex Table (RDD) v-prop:vertex prop. Edge Table (RDD) e-prop:edge prop. Routing Table (RDD) A B C D E F G H I J Edge-disjoint partitioning A v-prop B v-prop ... I v-prop D v-prop E v-prop F v-prop J v-prop A e-prop B A e-prop C ... F e-prop G A e-prop D A e-prop E ... E e-prop F A 1 2 B 1 ... I 1 F 1 2 D 2 E 2 J 2 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 40 / 96
- 112. GraphX: Computation Model Machine1Machine2 VertexTable Edge Table A v-prop B v-prop ... I v-prop D v-prop E v-prop F v-prop J v-prop A e-prop B A e-prop C ... F e-prop G A e-prop D A e-prop E ... E e-prop F © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 41 / 96
- 113. GraphX: Computation Model Machine1Machine2 VertexTable Edge Table A v-prop B v-prop ... I v-prop D v-prop E v-prop F v-prop J v-prop A e-prop B A e-prop C ... F e-prop G A e-prop D A e-prop E ... E e-prop F First Phase: Join Vertex table Edge table Triples View A v-prop e-prop B v-prop A v-prop e-prop C v-prop C v-prop e-prop G v-prop ... E v-prop e-prop G v-prop J v-prop e-prop G v-prop © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 41 / 96
- 114. GraphX: Computation Model Machine1Machine2 VertexTable Edge Table A v-prop B v-prop ... I v-prop D v-prop E v-prop F v-prop J v-prop A e-prop B A e-prop C ... F e-prop G A e-prop D A e-prop E ... E e-prop F Triples View A v-prop e-prop B v-prop A v-prop e-prop C v-prop C v-prop e-prop G v-prop ... E v-prop e-prop G v-prop J v-prop e-prop G v-prop Second Phase: Compute neighbourhood Group-by aggregate © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 41 / 96
- 115. GraphX: Operators Table transformoperators – inherited from Spark map(func) Return a new RDD formed by passing each element of the source through a function func filter(func) Return a new RDD formed by selecting those elements of the source on which func returns true flatMap(func) Similar to map, but each input item can be mapped to 0 or more output items mapPartitions(func) Similar to map, but runs separately on each partition (block) of the RDD, so func must be of type Iterator sample(repl, fraction, seed) Sample a fraction fraction of the data, with or without replacement (set repl accordingly), using a given random number generator seed union(otherDataset) intersection() Return a new RDD containing the union/intersection of the elements in the source RDD and the argument groupByKey() Operates on a RDD of (K, V) pairs, returns a RDD of (K, IterableV) pairs reduceByKey(func, . . .) Operates on a RDD of (K, V) pairs, returns a RDD of (K, V) pairs where the values for each key are aggregated using the given reduce function func © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 42 / 96
- 116. GraphX: Operators Table transformoperators – inherited from Spark Graph operators Graph(vertex coll, edge coll) Logically binds together a pair of vertex and edge property collections into a property graph; verifies that each vertex occurs only once and edges connect existing vertices triplets(vertex coll, vertex coll, edge coll) Returns the triplets view of the graph mrTriplets(map,reduce) MapReduce triplets - encodes the two-stage process of join to create triplets and group by © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 42 / 96
- 117. Outline 1 Introduction –Graph Types 2 Property Graph Processing Classification Online querying Offline analytics 3 RDF Graph Querying Data Warehousing Distributed SPARQL Execution Linked Object Data Querying © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 43 / 96
- 118. RDF Introduction Everything isan uniquely named resource http://data.linkedmdb.org/resource/actor/JN29704 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 44 / 96
- 119. RDF Introduction Everything isan uniquely named resource Prefixes can be used to shorten the names xmlns:y=http://data.linkedmdb.org/resource/actor/ y:JN29704 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 44 / 96
- 120. RDF Introduction Everything isan uniquely named resource Prefixes can be used to shorten the names Properties of resources can be defined xmlns:y=http://data.linkedmdb.org/resource/actor/ y:JN29704 y:JN29704:hasName “Jack Nicholson” y:JN29704:BornOnDate “1937-04-22” © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 44 / 96
- 121. RDF Introduction Everything isan uniquely named resource Prefixes can be used to shorten the names Properties of resources can be defined Relationships with other resources can be defined xmlns:y=http://data.linkedmdb.org/resource/actor/ y:JN29704 y:JN29704:hasName “Jack Nicholson” y:JN29704:BornOnDate “1937-04-22” y:TS2014:title “The Shining” y:TS2014:releaseDate “1980-05-23” y:TS2014 JN29704:movieActor © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 44 / 96
- 122. RDF Introduction Everything isan uniquely named resource Prefixes can be used to shorten the names Properties of resources can be defined Relationships with other resources can be defined Resource descriptions can be contributed by different people/groups and can be located anywhere in the web Integrated web “database” xmlns:y=http://data.linkedmdb.org/resource/actor/ y:JN29704 y:JN29704:hasName “Jack Nicholson” y:JN29704:BornOnDate “1937-04-22” y:TS2014:title “The Shining” y:TS2014:releaseDate “1980-05-23” y:TS2014 JN29704:movieActor © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 44 / 96
- 123. RDF Data Model Triple:Subject, Predicate (Property), Object (s, p, o) Subject: the entity that is described (URI or blank node) Predicate: a feature of the entity (URI) Object: value of the feature (URI, blank node or literal) (s, p, o) ∈ (U ∪ B) × U × (U ∪ B ∪ L) Set of RDF triples is called an RDF graph U Subject Object U B U B L U: set of URIs B: set of blank nodes L: set of literals Predicate Subject Predicate Object http://...imdb.../film/2014 rdfs:label “The Shining” http://...imdb.../film/2014 movie:releaseDate “1980-05-23” http://...imdb.../29704 movie:actor name “Jack Nicholson” . . . . . . . . . © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 45 / 96
- 124. RDF Example Instance Prefixes:mdb=http://data.linkedmdb.org/resource/; geo=http://sws.geonames.org/ bm=http://wifo5-03.informatik.uni-mannheim.de/bookmashup/ lexvo=http://lexvo.org/id/;wp=http://en.wikipedia.org/wiki/ Subject Predicate Object mdb: film/2014 rdfs:label “The Shining” mdb:film/2014 movie:initial release date “1980-05-23”’ mdb:film/2014 movie:director mdb:director/8476 mdb:film/2014 movie:actor mdb:actor/29704 mdb:film/2014 movie:actor mdb: actor/30013 mdb:film/2014 movie:music contributor mdb: music contributor/4110 mdb:film/2014 foaf:based near geo:2635167 mdb:film/2014 movie:relatedBook bm:0743424425 mdb:film/2014 movie:language lexvo:iso639-3/eng mdb:director/8476 movie:director name “Stanley Kubrick” mdb:film/2685 movie:director mdb:director/8476 mdb:film/2685 rdfs:label “A Clockwork Orange” mdb:film/424 movie:director mdb:director/8476 mdb:film/424 rdfs:label “Spartacus” mdb:actor/29704 movie:actor name “Jack Nicholson” mdb:film/1267 movie:actor mdb:actor/29704 mdb:film/1267 rdfs:label “The Last Tycoon” mdb:film/3418 movie:actor mdb:actor/29704 mdb:film/3418 rdfs:label “The Passenger” geo:2635167 gn:name “United Kingdom” geo:2635167 gn:population 62348447 geo:2635167 gn:wikipediaArticle wp:United Kingdom bm:books/0743424425 dc:creator bm:persons/Stephen+King bm:books/0743424425 rev:rating 4.7 bm:books/0743424425 scom:hasOffer bm:offers/0743424425amazonOffer lexvo:iso639-3/eng rdfs:label “English” lexvo:iso639-3/eng lvont:usedIn lexvo:iso3166/CA lexvo:iso639-3/eng lvont:usesScript lexvo:script/Latn URI Literal URI © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 46 / 96
- 125. RDF Graph mdb:film/2014 “1980-05-23” movie:initial releasedate “The Shining” refs:label bm:books/0743424425 4.7 rev:rating bm:offers/0743424425amazonOffer geo:2635167 “United Kingdom” gn:name 62348447 gn:population mdb:actor/29704 “Jack Nicholson” movie:actor name mdb:film/3418 “The Passenger” refs:label mdb:film/1267 “The Last Tycoon” refs:label mdb:director/8476 “Stanley Kubrick” movie:director name mdb:film/2685 “A Clockwork Orange” refs:label mdb:film/424 “Spartacus” refs:label mdb:actor/30013 movie:relatedBook scam:hasOffer foaf:based near movie:actor movie:director movie:actor movie:actor movie:actor movie:director movie:director © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 47 / 96
- 126. RDF Query Model– SPARQL Query Model - SPARQL Protocol and RDF Query Language Given U (set of URIs), L (set of literals), and V (set of variables), a SPARQL expression is defined recursively: an atomic triple pattern, which is an element of (U ∪ V ) × (U ∪ V ) × (U ∪ V ∪ L) ?x rdfs:label “The Shining” P FILTER R, where P is a graph pattern expression and R is a built-in SPARQL condition (i.e., analogous to a SQL predicate) ?x rev:rating ?p FILTER(?p 3.0) P1 AND/OPT/UNION P2, where P1 and P2 are graph pattern expressions Example: SELECT ?name WHERE { ?m r d f s : l a b e l ?name . ?m movie : d i r e c t o r ?d . ?d movie : director name ” Stanley Kubrick ” . ?m movie : relatedBook ?b . ?b rev : r a t i n g ? r . FILTER(? r 4.0) } © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 48 / 96
- 127. SPARQL Queries SELECT ?name WHERE{ ?m r d f s : l a b e l ?name . ?m movie : d i r e c t o r ?d . ?d movie : director name ” Stanley Kubrick ” . ?m movie : relatedBook ?b . ?b rev : r a t i n g ? r . FILTER(? r 4.0) } ?m ?d movie:director ?name rdfs:label ?b movie:relatedBook “Stanley Kubrick” movie:director name ?r rev:rating FILTER(?r 4.0) © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 49 / 96
- 128. Outline 1 Introduction –Graph Types 2 Property Graph Processing Classification Online querying Offline analytics 3 RDF Graph Querying Data Warehousing Distributed SPARQL Execution Linked Object Data Querying © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 50 / 96
- 129. Na¨ıve Triple StoreDesign SELECT ?name WHERE { ?m r d f s : l a b e l ?name . ?m movie : d i r e c t o r ?d . ?d movie : director name ” Stanley Kubrick ” . ?m movie : relatedBook ?b . ?b rev : r a t i n g ? r . FILTER(? r 4.0) } Subject Property Object mdb:film/2014 rdfs:label “The Shining” mdb:film/2014 movie:initial release date “1980-05-23” mdb:film/2014 movie:director mdb:director/8476 mdb:film/2014 movie:actor mdb:actor/29704 mdb:film/2014 movie:actor mdb: actor/30013 mdb:film/2014 movie:music contributor mdb: music contributor/4110 mdb:film/2014 foaf:based near geo:2635167 mdb:film/2014 movie:relatedBook bm:0743424425 mdb:film/2014 movie:language lexvo:iso639-3/eng mdb:director/8476 movie:director name “Stanley Kubrick” mdb:film/2685 movie:director mdb:director/8476 mdb:film/2685 rdfs:label “A Clockwork Orange” mdb:film/424 movie:director mdb:director/8476 mdb:film/424 rdfs:label “Spartacus” mdb:actor/29704 movie:actor name “Jack Nicholson” mdb:film/1267 movie:actor mdb:actor/29704 mdb:film/1267 rdfs:label “The Last Tycoon” mdb:film/3418 movie:actor mdb:actor/29704 mdb:film/3418 rdfs:label “The Passenger” geo:2635167 gn:name “United Kingdom” geo:2635167 gn:population 62348447 geo:2635167 gn:wikipediaArticle wp:United Kingdom bm:books/0743424425 dc:creator bm:persons/Stephen+King bm:books/0743424425 rev:rating 4.7 bm:books/0743424425 scom:hasOffer bm:offers/0743424425amazonOffer lexvo:iso639-3/eng rdfs:label “English” lexvo:iso639-3/eng lvont:usedIn lexvo:iso3166/CA lexvo:iso639-3/eng lvont:usesScript lexvo:script/Latn © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 51 / 96
- 130. Na¨ıve Triple StoreDesign SELECT ?name WHERE { ?m r d f s : l a b e l ?name . ?m movie : d i r e c t o r ?d . ?d movie : director name ” Stanley Kubrick ” . ?m movie : relatedBook ?b . ?b rev : r a t i n g ? r . FILTER(? r 4.0) } Subject Property Object mdb:film/2014 rdfs:label “The Shining” mdb:film/2014 movie:initial release date “1980-05-23” mdb:film/2014 movie:director mdb:director/8476 mdb:film/2014 movie:actor mdb:actor/29704 mdb:film/2014 movie:actor mdb: actor/30013 mdb:film/2014 movie:music contributor mdb: music contributor/4110 mdb:film/2014 foaf:based near geo:2635167 mdb:film/2014 movie:relatedBook bm:0743424425 mdb:film/2014 movie:language lexvo:iso639-3/eng mdb:director/8476 movie:director name “Stanley Kubrick” mdb:film/2685 movie:director mdb:director/8476 mdb:film/2685 rdfs:label “A Clockwork Orange” mdb:film/424 movie:director mdb:director/8476 mdb:film/424 rdfs:label “Spartacus” mdb:actor/29704 movie:actor name “Jack Nicholson” mdb:film/1267 movie:actor mdb:actor/29704 mdb:film/1267 rdfs:label “The Last Tycoon” mdb:film/3418 movie:actor mdb:actor/29704 mdb:film/3418 rdfs:label “The Passenger” geo:2635167 gn:name “United Kingdom” geo:2635167 gn:population 62348447 geo:2635167 gn:wikipediaArticle wp:United Kingdom bm:books/0743424425 dc:creator bm:persons/Stephen+King bm:books/0743424425 rev:rating 4.7 bm:books/0743424425 scom:hasOffer bm:offers/0743424425amazonOffer lexvo:iso639-3/eng rdfs:label “English” lexvo:iso639-3/eng lvont:usedIn lexvo:iso3166/CA lexvo:iso639-3/eng lvont:usesScript lexvo:script/Latn SELECT T1 . o b j e c t FROM T as T1 , T as T2 , T as T3 , T as T4 , T as T5 WHERE T1 . p=” r d f s : l a b e l ” AND T2 . p=” movie : relatedBook ” AND T3 . p=” movie : d i r e c t o r ” AND T4 . p=” rev : r a t i n g ” AND T5 . p=” movie : d i r e c t o r n a m e ” AND T1 . s=T2 . s AND T1 . s=T3 . s AND T2 . o=T4 . s AND T3 . o=T5 . s AND T4 . o 4.0 AND T5 . o=” S t a n l e y Kubrick ” © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 51 / 96
- 131. Na¨ıve Triple StoreDesign SELECT ?name WHERE { ?m r d f s : l a b e l ?name . ?m movie : d i r e c t o r ?d . ?d movie : director name ” Stanley Kubrick ” . ?m movie : relatedBook ?b . ?b rev : r a t i n g ? r . FILTER(? r 4.0) } Subject Property Object mdb:film/2014 rdfs:label “The Shining” mdb:film/2014 movie:initial release date “1980-05-23” mdb:film/2014 movie:director mdb:director/8476 mdb:film/2014 movie:actor mdb:actor/29704 mdb:film/2014 movie:actor mdb: actor/30013 mdb:film/2014 movie:music contributor mdb: music contributor/4110 mdb:film/2014 foaf:based near geo:2635167 mdb:film/2014 movie:relatedBook bm:0743424425 mdb:film/2014 movie:language lexvo:iso639-3/eng mdb:director/8476 movie:director name “Stanley Kubrick” mdb:film/2685 movie:director mdb:director/8476 mdb:film/2685 rdfs:label “A Clockwork Orange” mdb:film/424 movie:director mdb:director/8476 mdb:film/424 rdfs:label “Spartacus” mdb:actor/29704 movie:actor name “Jack Nicholson” mdb:film/1267 movie:actor mdb:actor/29704 mdb:film/1267 rdfs:label “The Last Tycoon” mdb:film/3418 movie:actor mdb:actor/29704 mdb:film/3418 rdfs:label “The Passenger” geo:2635167 gn:name “United Kingdom” geo:2635167 gn:population 62348447 geo:2635167 gn:wikipediaArticle wp:United Kingdom bm:books/0743424425 dc:creator bm:persons/Stephen+King bm:books/0743424425 rev:rating 4.7 bm:books/0743424425 scom:hasOffer bm:offers/0743424425amazonOffer lexvo:iso639-3/eng rdfs:label “English” lexvo:iso639-3/eng lvont:usedIn lexvo:iso3166/CA lexvo:iso639-3/eng lvont:usesScript lexvo:script/Latn SELECT T1 . o b j e c t FROM T as T1 , T as T2 , T as T3 , T as T4 , T as T5 WHERE T1 . p=” r d f s : l a b e l ” AND T2 . p=” movie : relatedBook ” AND T3 . p=” movie : d i r e c t o r ” AND T4 . p=” rev : r a t i n g ” AND T5 . p=” movie : d i r e c t o r n a m e ” AND T1 . s=T2 . s AND T1 . s=T3 . s AND T2 . o=T4 . s AND T3 . o=T5 . s AND T4 . o 4.0 AND T5 . o=” S t a n l e y Kubrick ” Easy to implement but too many self-joins! © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 51 / 96
- 132. Exhaustive Indexing RDF-3X [Neumannand Weikum, 2008, 2009], Hexastore [Weiss et al., 2008] Strings are mapped to ids using a mapping table Original triple table Subject Property Object mdb: film/2014 rdfs:label “The Shining” mdb:film/2014 movie:initial release date “1980-05-23” mdb:director/8476 movie:director name “Stanley Kubrick” mdb:film/2685 movie:director mdb:director/8476 Encoded triple table Subject Property Object 0 1 2 0 3 4 5 6 7 8 9 5 Mapping table ID Value 0 mdb: film/2014 1 rdfs:label 2 “The Shining” 3 movie:initial release date 4 “1980-05-23” 5 mdb:director/8476 6 movie:director name 7 “Stanley Kubrick” 8 mdb:film/2685 9 movie:director © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 52 / 96
- 133. Exhaustive Indexing RDF-3X [Neumannand Weikum, 2008, 2009], Hexastore [Weiss et al., 2008] Strings are mapped to ids using a mapping table Triples are indexed in a clustered B+ tree in lexicographic order Subject Property Object 0 1 2 0 3 4 5 6 7 8 9 5 ... ... ... B+ tree Easy querying through mapping table © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 52 / 96
- 134. Exhaustive Indexing RDF-3X [Neumannand Weikum, 2008, 2009], Hexastore [Weiss et al., 2008] Strings are mapped to ids using a mapping table Triples are indexed in a clustered B+ tree in lexicographic order Create indexes for permutations of the three columns: SPO, SOP, PSO, POS, OPS, OSP Subject Property Object 0 1 2 0 3 4 5 6 7 8 9 5 ... ... ... B+ tree Easy querying through mapping table © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 52 / 96
- 135. Exhaustive Indexing–Query Execution Eachtriple pattern can be answered by a range query Joins between triple patterns computed using merge join Join order is easy due to extensive indexing Subject Property Object 0 1 2 0 3 4 5 6 7 8 9 5 ... ... ... ID Value 0 mdb: film/2014 1 rdfs:label 2 “The Shining” 3 movie:initial release date 4 “1980-05-23” 5 mdb:director/8476 6 movie:director name 7 “Stanley Kubrick” 8 mdb:film/2685 9 movie:director © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 53 / 96
- 136. Exhaustive Indexing–Query Execution Eachtriple pattern can be answered by a range query Joins between triple patterns computed using merge join Join order is easy due to extensive indexing Subject Property Object 0 1 2 0 3 4 5 6 7 8 9 5 ... ... ... ID Value 0 mdb: film/2014 1 rdfs:label 2 “The Shining” 3 movie:initial release date 4 “1980-05-23” 5 mdb:director/8476 6 movie:director name 7 “Stanley Kubrick” 8 mdb:film/2685 9 movie:director Advantages Eliminates some of the joins – they become range queries Merge join is easy and fast © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 53 / 96
- 137. Exhaustive Indexing–Query Execution Eachtriple pattern can be answered by a range query Joins between triple patterns computed using merge join Join order is easy due to extensive indexing Subject Property Object 0 1 2 0 3 4 5 6 7 8 9 5 ... ... ... ID Value 0 mdb: film/2014 1 rdfs:label 2 “The Shining” 3 movie:initial release date 4 “1980-05-23” 5 mdb:director/8476 6 movie:director name 7 “Stanley Kubrick” 8 mdb:film/2685 9 movie:director Advantages Eliminates some of the joins – they become range queries Merge join is easy and fast Disadvantages Space usage © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 53 / 96
- 138. Property Tables Grouping byentities; Jena [Wilkinson, 2006], DB2-RDF [Bornea et al., 2013] Clustered property table: group together the properties that tend to occur in the same (or similar) subjects Property-class table: cluster the subjects with the same type of property into one property table Subject Property Object mdb:film/2014 rdfs:label “The Shining” mdb:film/2014 movie:director mdb:director/8476 mdb:film/2685 movie:director mdb:director/8476 mdb:film/2685 rdfs:label “A Clockwork Orange” mdb:actor/29704 movie:actor name “Jack Nicholson” . . . . . . . . . Subject refs:label movie:director mob:film/2014 “The Shining” mob:director/8476 mob:film/2685 “The Clockwork Orange” mob:director/8476 Subject movie:actor name mdb:actor “Jack Nicholson” © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 54 / 96
- 139. Property Tables Grouping byentities; Jena [Wilkinson, 2006], DB2-RDF [Bornea et al., 2013] Clustered property table: group together the properties that tend to occur in the same (or similar) subjects Property-class table: cluster the subjects with the same type of property into one property table Subject Property Object mdb:film/2014 rdfs:label “The Shining” mdb:film/2014 movie:director mdb:director/8476 mdb:film/2685 movie:director mdb:director/8476 mdb:film/2685 rdfs:label “A Clockwork Orange” mdb:actor/29704 movie:actor name “Jack Nicholson” . . . . . . . . . Subject refs:label movie:director mob:film/2014 “The Shining” mob:director/8476 mob:film/2685 “The Clockwork Orange” mob:director/8476 Subject movie:actor name mdb:actor “Jack Nicholson” Advantages Fewer joins If the data is structured, we have a relational system – similar to normalized relations © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 54 / 96
- 140. Property Tables Grouping byentities; Jena [Wilkinson, 2006], DB2-RDF [Bornea et al., 2013] Clustered property table: group together the properties that tend to occur in the same (or similar) subjects Property-class table: cluster the subjects with the same type of property into one property table Subject Property Object mdb:film/2014 rdfs:label “The Shining” mdb:film/2014 movie:director mdb:director/8476 mdb:film/2685 movie:director mdb:director/8476 mdb:film/2685 rdfs:label “A Clockwork Orange” mdb:actor/29704 movie:actor name “Jack Nicholson” . . . . . . . . . Subject refs:label movie:director mob:film/2014 “The Shining” mob:director/8476 mob:film/2685 “The Clockwork Orange” mob:director/8476 Subject movie:actor name mdb:actor “Jack Nicholson” Advantages Fewer joins If the data is structured, we have a relational system – similar to normalized relations Disadvantages Potentially a lot of NULLs Clustering is not trivial Multi-valued properties are complicated © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 54 / 96
- 141. Binary Tables Grouping byproperties: For each property, build a two-column table, containing both subject and object, ordered by subjects [Abadi et al., 2007, 2009] Also called vertical partitioned tables n two column tables (n is the number of unique properties in the data) Subject Property Object mdb:film/2014 rdfs:label “The Shining” mdb:film/2014 movie:director mdb:director/8476 mdb:film/2685 movie:director mdb:director/8476 mdb:film/2685 rdfs:label “A Clockwork Orange” mdb:actor/29704 movie:actor name “Jack Nicholson” . . . . . . . . . Subject Object mdb:film/2014 mdb:director/8476 mdb:film/2685 mdb:director/8476 movie:director Subject Object mob:film/2014 “The Shining” mob:film/2685 “The Clockwork Orange” refs:label Subject Object mdb:actor/29704 “Jack Nicholson” movie:actor name © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 55 / 96
- 142. Binary Tables Grouping byproperties: For each property, build a two-column table, containing both subject and object, ordered by subjects [Abadi et al., 2007, 2009] Also called vertical partitioned tables n two column tables (n is the number of unique properties in the data) Subject Property Object mdb:film/2014 rdfs:label “The Shining” mdb:film/2014 movie:director mdb:director/8476 mdb:film/2685 movie:director mdb:director/8476 mdb:film/2685 rdfs:label “A Clockwork Orange” mdb:actor/29704 movie:actor name “Jack Nicholson” . . . . . . . . . Subject Object mdb:film/2014 mdb:director/8476 mdb:film/2685 mdb:director/8476 movie:director Subject Object mob:film/2014 “The Shining” mob:film/2685 “The Clockwork Orange” refs:label Subject Object mdb:actor/29704 “Jack Nicholson” movie:actor name Advantages Supports multi-valued properties No NULLs No clustering Read only needed attributes (i.e. less I/O) Good performance for subject-subject joins © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 55 / 96
- 143. Binary Tables Grouping byproperties: For each property, build a two-column table, containing both subject and object, ordered by subjects [Abadi et al., 2007, 2009] Also called vertical partitioned tables n two column tables (n is the number of unique properties in the data) Subject Property Object mdb:film/2014 rdfs:label “The Shining” mdb:film/2014 movie:director mdb:director/8476 mdb:film/2685 movie:director mdb:director/8476 mdb:film/2685 rdfs:label “A Clockwork Orange” mdb:actor/29704 movie:actor name “Jack Nicholson” . . . . . . . . . Subject Object mdb:film/2014 mdb:director/8476 mdb:film/2685 mdb:director/8476 movie:director Subject Object mob:film/2014 “The Shining” mob:film/2685 “The Clockwork Orange” refs:label Subject Object mdb:actor/29704 “Jack Nicholson” movie:actor name Advantages Supports multi-valued properties No NULLs No clustering Read only needed attributes (i.e. less I/O) Good performance for subject-subject joins Disadvantages Not useful for subject-object joins Expensive inserts © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 55 / 96
- 144. Graph-based Approach Answering SPARQLquery ≡ subgraph matching using homomorphism gStore [Zou et al., 2011, 2014], chameleon-db [Alu¸c et al., 2013] ?m ?d movie:director ?name rdfs:label ?b movie:relatedBook “Stanley Kubrick” movie:director name ?r rev:rating FILTER(?r 4.0) mdb:film/2014 “1980-05-23” movie:initial release date “The Shining” refs:label bm:books/0743424425 4.7 rev:rating bm:offers/0743424425amazonOffer geo:2635167 “United Kingdom” gn:name 62348447 gn:population mdb:actor/29704 “Jack Nicholson” movie:actor name mdb:film/3418 “The Passenger” refs:label mdb:film/1267 “The Last Tycoon” refs:label mdb:director/8476 “Stanley Kubrick” movie:director name mdb:film/2685 “A Clockwork Orange” refs:label mdb:film/424 “Spartacus” refs:label mdb:actor/30013 movie:relatedBook scam:hasOffer foaf:based near movie:actor movie:director movie:actor movie:actor movie:actor movie:director movie:director Subgraph M atching © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 56 / 96
- 145. Graph-based Approach Answering SPARQLquery ≡ subgraph matching using homomorphism gStore [Zou et al., 2011, 2014], chameleon-db [Alu¸c et al., 2013] ?m ?d movie:director ?name rdfs:label ?b movie:relatedBook “Stanley Kubrick” movie:director name ?r rev:rating FILTER(?r 4.0) mdb:film/2014 “1980-05-23” movie:initial release date “The Shining” refs:label bm:books/0743424425 4.7 rev:rating bm:offers/0743424425amazonOffer geo:2635167 “United Kingdom” gn:name 62348447 gn:population mdb:actor/29704 “Jack Nicholson” movie:actor name mdb:film/3418 “The Passenger” refs:label mdb:film/1267 “The Last Tycoon” refs:label mdb:director/8476 “Stanley Kubrick” movie:director name mdb:film/2685 “A Clockwork Orange” refs:label mdb:film/424 “Spartacus” refs:label mdb:actor/30013 movie:relatedBook scam:hasOffer foaf:based near movie:actor movie:director movie:actor movie:actor movie:actor movie:director movie:director Subgraph M atching Advantages Maintains the graph structure Full set of queries can be handled © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 56 / 96
- 146. Graph-based Approach Answering SPARQLquery ≡ subgraph matching using homomorphism gStore [Zou et al., 2011, 2014], chameleon-db [Alu¸c et al., 2013] ?m ?d movie:director ?name rdfs:label ?b movie:relatedBook “Stanley Kubrick” movie:director name ?r rev:rating FILTER(?r 4.0) mdb:film/2014 “1980-05-23” movie:initial release date “The Shining” refs:label bm:books/0743424425 4.7 rev:rating bm:offers/0743424425amazonOffer geo:2635167 “United Kingdom” gn:name 62348447 gn:population mdb:actor/29704 “Jack Nicholson” movie:actor name mdb:film/3418 “The Passenger” refs:label mdb:film/1267 “The Last Tycoon” refs:label mdb:director/8476 “Stanley Kubrick” movie:director name mdb:film/2685 “A Clockwork Orange” refs:label mdb:film/424 “Spartacus” refs:label mdb:actor/30013 movie:relatedBook scam:hasOffer foaf:based near movie:actor movie:director movie:actor movie:actor movie:actor movie:director movie:director Subgraph M atching Advantages Maintains the graph structure Full set of queries can be handled Disadvantages Graph pattern matching is expensive © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 56 / 96
- 147. Two Systems gStoreSystem Architecture OfflineOnline Storage Input Input RDF Parser RDF Graph Builder Encoding Module VS*-tree builder RDF data RDF Triples RDF Graph Signature Graph Key-Value Store VS*-tree Store SPARQL Parser SPARQL Query Encoding Module VS*-tree Query Graph Filter Module Join Module Signature Graph Node Candidate Results Fig. 4. System Architecture itstrings, denoted as vS ig(u). We encode query Q with the ame encoding method. Consequently, the match between Q nd G can be verified by simply checking the match between orresponding encoded bitstrings. chameleon-db Structural Index ... Vertex Index Spill Index ClusterIndexStorageSystem StorageAdvisor Query Engine Plan Generation Evaluation © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 57 / 96
- 148. gStore General Approach: Work directlyon the RDF graph and the SPARQL query graph Use a signature-based encoding of each entity and class vertex to speed up matching Filter-and-evaluate Use a false positive algorithm to prune nodes and obtain a set of candidates; then do more detailed evaluation on those Use an index (VS∗-tree) over the data signature graph (has light maintenance load) for efficient pruning © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 58 / 96
- 149. 1. Encode Qand G to Get Signature Graphs Query signature graph Q∗ 0100 0000 1000 0000 00010 0000 0100 10000 Data signature graph G∗ 0010 1000 0100 0001 00001 1000 0001 00010 0000 0100 10000 0000 1000 10000 0000 0010 10000 0000 1001 00100 0001 0001 01000 0100 1000 01000 1001 1000 01000 0001 0100 01000 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 59 / 96
- 150. 2. Filter-and-Evaluate Query signaturegraph Q∗ 0100 0000 1000 0000 00010 0000 0100 10000 Data signature graph G∗ 0010 1000 0100 0001 00001 1000 0001 00010 0000 0100 10000 0000 1000 10000 0000 0010 10000 0000 1001 00100 0001 0001 01000 0100 1000 01000 1001 1000 01000 0001 0100 01000 Find matches of Q∗ over signature graph G∗ Verify each match in RDF graph G © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 60 / 96
- 151. How to GenerateCandidate List Two step process: 1 For each node of Q∗ get lists of nodes in G∗ that include that node. 2 Do a multi-way join to get the candidate list © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 61 / 96
- 152. How to GenerateCandidate List Two step process: 1 For each node of Q∗ get lists of nodes in G∗ that include that node. 2 Do a multi-way join to get the candidate list Alternatives: © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 61 / 96
- 153. How to GenerateCandidate List Two step process: 1 For each node of Q∗ get lists of nodes in G∗ that include that node. 2 Do a multi-way join to get the candidate list Alternatives: Sequential scan of G∗ Both steps are inefficient © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 61 / 96
- 154. How to GenerateCandidate List Two step process: 1 For each node of Q∗ get lists of nodes in G∗ that include that node. 2 Do a multi-way join to get the candidate list Alternatives: Sequential scan of G∗ Both steps are inefficient Use S-trees Height-balanced tree over signatures Run an inclusion query for each node of Q∗ and get lists of nodes in G∗ that include that node. • Given query signature q and a set of data signatures S, find all data signatures si ∈ S where qsi = q Does not support second step – expensive © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 61 / 96
- 155. How to GenerateCandidate List Two step process: 1 For each node of Q∗ get lists of nodes in G∗ that include that node. 2 Do a multi-way join to get the candidate list Alternatives: Sequential scan of G∗ Both steps are inefficient Use S-trees Height-balanced tree over signatures Run an inclusion query for each node of Q∗ and get lists of nodes in G∗ that include that node. • Given query signature q and a set of data signatures S, find all data signatures si ∈ S where qsi = q Does not support second step – expensive VS-tree (and VS∗ -tree) Multi-resolution summary graph based on S-tree Supports both steps efficiently Grouping by vertices © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 61 / 96
- 156. S-tree Solution 1111 1111 01101111 1101 1101 0000 1110 0110 1001 1100 1001 1001 1101 0000 1000 0000 0100 0000 0010 0010 1000 0100 0001 1000 0001 0000 1001 0100 1000 1001 1000 0001 0100 0001 0001 005 004 006 001 002 003 007 011 008 009 010 d1 1 d2 1 d2 2 d3 1 d3 2 d3 3 d3 4 G3 G2 G1 1000 00000100 0000 00010 0000 0100 10000 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 62 / 96
- 157. S-tree Solution 1111 1111 01101111 1101 1101 0000 1110 0110 1001 1100 1001 1001 1101 0000 1000 0000 0100 0000 0010 0010 1000 0100 0001 1000 0001 0000 1001 0100 1000 1001 1000 0001 0100 0001 0001 005 004 006 001 002 003 007 011 008 009 010 d1 1 d2 1 d2 2 d3 1 d3 2 d3 3 d3 4 G3 G2 G1 1000 00000100 0000 00010 0000 0100 10000 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 62 / 96
- 158. S-tree Solution 1111 1111 01101111 1101 1101 0000 1110 0110 1001 1100 1001 1001 1101 0000 1000 0000 0100 0000 0010 0010 1000 0100 0001 1000 0001 0000 1001 0100 1000 1001 1000 0001 0100 0001 0001 005 004 006 001 002 003 007 011 008 009 010 d1 1 d2 1 d2 2 d3 1 d3 2 d3 3 d3 4 G3 G2 G1 1000 00000100 0000 00010 0000 0100 10000 002 011 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 62 / 96
- 159. S-tree Solution 1111 1111 01101111 1101 1101 0000 1110 0110 1001 1100 1001 1001 1101 0000 1000 0000 0100 0000 0010 0010 1000 0100 0001 1000 0001 0000 1001 0100 1000 1001 1000 0001 0100 0001 0001 005 004 006 001 002 003 007 011 008 009 010 d1 1 d2 1 d2 2 d3 1 d3 2 d3 3 d3 4 G3 G2 G1 1000 00000100 0000 00010 0000 0100 10000 002 011 003 008 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 62 / 96
- 160. S-tree Solution 1111 1111 01101111 1101 1101 0000 1110 0110 1001 1100 1001 1001 1101 0000 1000 0000 0100 0000 0010 0010 1000 0100 0001 1000 0001 0000 1001 0100 1000 1001 1000 0001 0100 0001 0001 005 004 006 001 002 003 007 011 008 009 010 d1 1 d2 1 d2 2 d3 1 d3 2 d3 3 d3 4 G3 G2 G1 1000 00000100 0000 00010 0000 0100 10000 002 011 003 008 004 009 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 62 / 96
- 161. S-tree Solution 1111 1111 01101111 1101 1101 0000 1110 0110 1001 1100 1001 1001 1101 0000 1000 0000 0100 0000 0010 0010 1000 0100 0001 1000 0001 0000 1001 0100 1000 1001 1000 0001 0100 0001 0001 005 004 006 001 002 003 007 011 008 009 010 d1 1 d2 1 d2 2 d3 1 d3 2 d3 3 d3 4 G3 G2 G1 1000 00000100 0000 00010 0000 0100 10000 002 011 003 008 004 009 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 62 / 96
- 162. S-tree Solution 1111 1111 01101111 1101 1101 0000 1110 0110 1001 1100 1001 1001 1101 0000 1000 0000 0100 0000 0010 0010 1000 0100 0001 1000 0001 0000 1001 0100 1000 1001 1000 0001 0100 0001 0001 005 004 006 001 002 003 007 011 008 009 010 d1 1 d2 1 d2 2 d3 1 d3 2 d3 3 d3 4 G3 G2 G1 1000 00000100 0000 00010 0000 0100 10000 002 011 003 008 004 009 Possibly large join space! © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 62 / 96
- 163. VS-tree 1111 1111 0110 11111101 1101 0000 1110 0110 1001 1100 1001 1001 1101 0000 1000 0000 0100 0000 0010 0010 1000 0100 0001 1000 0001 0000 1001 0100 1000 1001 1000 0001 0100 0001 0001 005 004 006 001 002 003 007 011 008 009 010 d1 1 d2 1 d2 2 d3 1 d3 2 d3 3 d3 4 G3 G2 G1 11101 10010 10001 01100 10000 00001 01100 00010 10000 01000 01000 10000 10000 10000 10000 00010 00100 01000 01000 01000 01000 Super edge © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 63 / 96
- 164. Pruning with VS-Tree 11111111 0110 1111 1101 1101 0000 1110 0110 1001 1100 1001 1001 1101 0000 1000 0000 0100 0000 0010 0010 1000 0100 0001 1000 0001 0000 1001 0100 1000 1001 1000 0001 0100 0001 0001 005 004 006 001 002 003 007 011 008 009 010 d1 1 d2 1 d2 2 d3 1 d3 2 d3 3 d3 4 G3 G2 G1 11101 10010 10001 01100 10000 00001 01100 00010 10000 01000 01000 10000 10000 10000 10000 00010 00100 01000 01000 01000 01000 1000 00000100 0000 00010 0000 0100 10000 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 64 / 96
- 165. Pruning with VS-Tree 11111111 0110 1111 1101 1101 0000 1110 0110 1001 1100 1001 1001 1101 0000 1000 0000 0100 0000 0010 0010 1000 0100 0001 1000 0001 0000 1001 0100 1000 1001 1000 0001 0100 0001 0001 005 004 006 001 002 003 007 011 008 009 010 d1 1 d2 1 d2 2 d3 1 d3 2 d3 3 d3 4 G3 G2 G1 11101 10010 10001 01100 10000 00001 01100 00010 10000 01000 01000 10000 10000 10000 10000 00010 00100 01000 01000 01000 01000 1000 00000100 0000 00010 0000 0100 10000 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 64 / 96
- 166. Pruning with VS-Tree 11111111 0110 1111 1101 1101 0000 1110 0110 1001 1100 1001 1001 1101 0000 1000 0000 0100 0000 0010 0010 1000 0100 0001 1000 0001 0000 1001 0100 1000 1001 1000 0001 0100 0001 0001 005 004 006 001 002 003 007 011 008 009 010 d1 1 d2 1 d2 2 d3 1 d3 2 d3 3 d3 4 G3 G2 G1 11101 10010 10001 01100 10000 00001 01100 00010 10000 01000 01000 10000 10000 10000 10000 00010 00100 01000 01000 01000 01000 1000 00000100 0000 00010 0000 0100 10000 d3 2 d3 3 d3 3 d3 4 d3 1 d3 4 G3 00010 10000 01000 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 64 / 96
- 167. Pruning with VS-Tree 11111111 0110 1111 1101 1101 0000 1110 0110 1001 1100 1001 1001 1101 0000 1000 0000 0100 0000 0010 0010 1000 0100 0001 1000 0001 0000 1001 0100 1000 1001 1000 0001 0100 0001 0001 005 004 006 001 002 003 007 011 008 009 010 d1 1 d2 1 d2 2 d3 1 d3 2 d3 3 d3 4 G3 G2 G1 11101 10010 10001 01100 10000 00001 01100 00010 10000 01000 01000 10000 10000 10000 10000 00010 00100 01000 01000 01000 01000 1000 00000100 0000 00010 0000 0100 10000 d3 2 d3 3 d3 3 d3 4 d3 1 d3 4 G3 00010 10000 01000 003 008 002 011 004 009 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 64 / 96
- 168. Pruning with VS-Tree 11111111 0110 1111 1101 1101 0000 1110 0110 1001 1100 1001 1001 1101 0000 1000 0000 0100 0000 0010 0010 1000 0100 0001 1000 0001 0000 1001 0100 1000 1001 1000 0001 0100 0001 0001 005 004 006 001 002 003 007 011 008 009 010 d1 1 d2 1 d2 2 d3 1 d3 2 d3 3 d3 4 G3 G2 G1 11101 10010 10001 01100 10000 00001 01100 00010 10000 01000 01000 10000 10000 10000 10000 00010 00100 01000 01000 01000 01000 1000 00000100 0000 00010 0000 0100 10000 d3 2 d3 3 d3 3 d3 4 d3 1 d3 4 G3 00010 10000 01000 003 008 002 011 004 009 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 64 / 96
- 169. Pruning with VS-Tree 11111111 0110 1111 1101 1101 0000 1110 0110 1001 1100 1001 1001 1101 0000 1000 0000 0100 0000 0010 0010 1000 0100 0001 1000 0001 0000 1001 0100 1000 1001 1000 0001 0100 0001 0001 005 004 006 001 002 003 007 011 008 009 010 d1 1 d2 1 d2 2 d3 1 d3 2 d3 3 d3 4 G3 G2 G1 11101 10010 10001 01100 10000 00001 01100 00010 10000 01000 01000 10000 10000 10000 10000 00010 00100 01000 01000 01000 01000 1000 00000100 0000 00010 0000 0100 10000 d3 2 d3 3 d3 3 d3 4 d3 1 d3 4 G3 00010 10000 01000 003 008 002 011 004 009 Reduced join space! © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 64 / 96
- 170. Adaptivity to Workload Applicationsthat rely on RDF data are increasingly popular and are more varied [Verborgh et al., 2014] Data that are being handled are far more heterogeneous [Duan et al., 2011] SPARQL queries are becoming more diverse [Arias et al., 2011] and dynamic [Kirchberg et al., 2011] An experiment [Alu¸c et al., 2014a] No single system is a sole winner across all queries No single system is the sole loser across all queries, either There can be 2–5 orders of magnitude difference in the performance (i.e., query execution time) between the best and the worst system for a given query The winner in one query may timeout in another Performance difference widens as dataset size increases © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 65 / 96
- 171. Adaptivity to Workload Applicationsthat rely on RDF data are increasingly popular and are more varied [Verborgh et al., 2014] Data that are being handled are far more heterogeneous [Duan et al., 2011] SPARQL queries are becoming more diverse [Arias et al., 2011] and dynamic [Kirchberg et al., 2011] An experiment [Alu¸c et al., 2014a] No single system is a sole winner across all queries No single system is the sole loser across all queries, either There can be 2–5 orders of magnitude difference in the performance (i.e., query execution time) between the best and the worst system for a given query The winner in one query may timeout in another Performance difference widens as dataset size increases Can existing systems cope with these trends – workload diversity dynamism No! [Alu¸c et al., 2014b] Fragmented data Suboptimal pruning by indexes Unnecessarily large sets of intermediate result tuples © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 65 / 96
- 172. Adaptivity to Workload Applicationsthat rely on RDF data are increasingly popular and are more varied [Verborgh et al., 2014] Data that are being handled are far more heterogeneous [Duan et al., 2011] SPARQL queries are becoming more diverse [Arias et al., 2011] and dynamic [Kirchberg et al., 2011] An experiment [Alu¸c et al., 2014a] No single system is a sole winner across all queries No single system is the sole loser across all queries, either There can be 2–5 orders of magnitude difference in the performance (i.e., query execution time) between the best and the worst system for a given query The winner in one query may timeout in another Performance difference widens as dataset size increases Our proposal: Idea behind chameleon-db When designing and implementing an RDF data management system, assume nothing about the workload upfront Organize data dynamically and purely based on the workload © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 65 / 96
- 173. Group-by-Query Approach v1 v21 A v21 v20 B v1 v98 A v98 v30 C v1 v250 A v250 v40 D v0 v32 A v0 v52 C v0 v80 C v0 v66C v0 v47 B v6 v7 A v6 v8 B v6 v9 C C1C2 C3 C4 C5 Characteristics: Records are not necessarily of fixed length Records are not grouped into tables Records do not necessarily share the same set of RDF predicates Each record represents a very tiny part of the RDF graph © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 66 / 96
- 174. Group-by-Query Approach v1 v21 A v21 v20 B v1 v98 A v98 v30 C v1 v250 A v250 v40 D v0 v32 A v0 v52 C v0 v80 C v0 v66C v0 v47 B v6 v7 A v6 v8 B v6 v9 C C1C2 C3 C4 C5 ?x ?y A ?y ?z ?b Figure: Query 1 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 66 / 96
- 175. Group-by-Query Approach v1 v21 A v21 v20 B v1 v98 A v98 v30 C v1 v250 A v250 v40 D v0 v32 A v0 v52 C v0 v80 C v0 v66C v0 v47 B v6 v7 A v6 v8 B v6 v9 C C1C2 C3 C4 C5 ?x ?y A ?y ?z ?b Figure: Query 1 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 66 / 96
- 176. Group-by-Query Approach v1 v21 A v21 v20 B v1 v98 A v98 v30 C v1 v250 A v250 v40 D v0 v32 A v0 v52 C v0 v80 C v0 v66C v0 v47 B v6 v7 A v6 v8 B v6 v9 C C1C2 C3 C4 C5 ?x ?y A ?y ?z ?b Figure: Query 1 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 66 / 96
- 177. Group-by-Query Approach v1 v21 A v21 v20 B v1 v98 A v98 v30 C v1 v250 A v250 v40 D v0 v32 A v0 v52 C v0 v80 C v0 v66C v0 v47 B v6 v7 A v6 v8 B v6 v9 C C1C2 C3 C4 C5 ?a ?z C ?a ?x A ?a ?y B Figure: Query 2 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 66 / 96
- 178. Group-by-Query Approach v1 v21 A v21 v20 B v1 v98 A v98 v30 C v1 v250 A v250 v40 D v0 v32 A v0 v52 C v0 v80 C v0 v66C v0 v47 B v6 v7 A v6 v8 B v6 v9 C C1C2 C3 C4 C5 ?a ?z C ?a ?x A ?a ?y B Figure: Query 2 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 66 / 96
- 179. Group-by-Query Approach v1 v21 A v21 v20 B v1 v98 A v98 v30 C v1 v250 A v250 v40 D v0 v32 A v0 v52 C v0 v80 C v0 v66C v0 v47 B v6 v7 A v6 v8 B v6 v9 C C1C2 C3 C4 C5 ?a ?z C ?a ?x A ?a ?y B Figure: Query 2 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 66 / 96
- 180. Group-by-Query Approach v1 v21 A v21 v20 B v1 v98 A v98 v30 C v1 v250 A v250 v40 D v0 v32 A v0 v52 C v0 v80 C v0 v66C v0 v47 B v6 v7 A v6 v8 B v6 v9 C C1C2 C3 C4 C5 ?a ?z C ?a ?x A ?a ?y B Figure: Query 2 Advantages Data are physically clustered for the workload Better pruning by the indexes Fewer intermediate result tuples © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 66 / 96
- 181. Challenges Physical Data Layout:As the workloads change, the way data are grouped together may no longer be suitable Hierarchical Clustering Algorithm [Alu¸c et al., 2015] Tunable-LSH [Alu¸c et al., 2015] Indexing: Indexing upfront is not a choice Query Evaluation: Can we execute queries efficiently even when the physical layout is constantly changing? [Alu¸c et al., 2015] © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 67 / 96
- 182. chameleon-db Prototype system [Alu¸cet al., 2013] 35,000 lines of code in C++ under Linux (plus code for SPARQL 1.0 parser) Structural Index ... Vertex Index Spill Index ClusterIndexStorageSystem StorageAdvisor Query Engine Plan Generation Evaluation © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 68 / 96
- 183. Outline 1 Introduction –Graph Types 2 Property Graph Processing Classification Online querying Offline analytics 3 RDF Graph Querying Data Warehousing Distributed SPARQL Execution Linked Object Data Querying © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 69 / 96
- 184. Remember the Environment Distributedenvironment Some of the data sites can process SPARQL queries – SPARQL endpoints Not all data sites can process queries © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 70 / 96
- 185. Remember the Environment Distributedenvironment Some of the data sites can process SPARQL queries – SPARQL endpoints Not all data sites can process queries Alternatives © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 70 / 96
- 186. Remember the Environment Distributedenvironment Some of the data sites can process SPARQL queries – SPARQL endpoints Not all data sites can process queries Alternatives Data re-distribution + query decomposition © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 70 / 96
- 187. Remember the Environment Distributedenvironment Some of the data sites can process SPARQL queries – SPARQL endpoints Not all data sites can process queries Alternatives Data re-distribution + query decomposition Data re-distribution + partial evaluation © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 70 / 96
- 188. Remember the Environment Distributedenvironment Some of the data sites can process SPARQL queries – SPARQL endpoints Not all data sites can process queries Alternatives Data re-distribution + query decomposition Data re-distribution + partial evaluation SPARQL federation: just process at SPARQL endpoints © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 70 / 96
- 189. Remember the Environment Distributedenvironment Some of the data sites can process SPARQL queries – SPARQL endpoints Not all data sites can process queries Alternatives Data re-distribution + query decomposition Data re-distribution + partial evaluation SPARQL federation: just process at SPARQL endpoints Live querying (next section) © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 70 / 96
- 190. Distributed RDF Processing[Kaoudi and Manolescu, 2015] RDF data warehouse is partitioned and distributed RDF data D = {D1, . . . , Dn} Allocate each Di to a site © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 71 / 96
- 191. Distributed RDF Processing[Kaoudi and Manolescu, 2015] RDF data warehouse is partitioned and distributed RDF data D = {D1, . . . , Dn} Allocate each Di to a site Partitioning alternatives Table-based (e.g., [Husain et al., 2011]) Graph-based (e.g., [Huang et al., 2011; Zhang et al., 2013]) Unit-based (e.g., [Gurajada et al., 2014; Lee and Liu, 2013]) © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 71 / 96
- 192. Distributed RDF Processing[Kaoudi and Manolescu, 2015] RDF data warehouse is partitioned and distributed RDF data D = {D1, . . . , Dn} Allocate each Di to a site Partitioning alternatives Table-based (e.g., [Husain et al., 2011]) Graph-based (e.g., [Huang et al., 2011; Zhang et al., 2013]) Unit-based (e.g., [Gurajada et al., 2014; Lee and Liu, 2013]) SPARQL query decomposed Q = {Q1, . . . , Qk} Distributed execution of {Q1, . . . , Qk} over {D1, . . . , Dn} © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 71 / 96
- 193. Distributed RDF Processing[Kaoudi and Manolescu, 2015] RDF data warehouse is partitioned and distributed RDF data D = {D1, . . . , Dn} Allocate each Di to a site Partitioning alternatives Table-based (e.g., [Husain et al., 2011]) Graph-based (e.g., [Huang et al., 2011; Zhang et al., 2013]) Unit-based (e.g., [Gurajada et al., 2014; Lee and Liu, 2013]) SPARQL query decomposed Q = {Q1, . . . , Qk} Distributed execution of {Q1, . . . , Qk} over {D1, . . . , Dn} High performance Great for parallelizing centralized RDF data May not be possible to re-partition and re-allocate Web data (i.e., LOD) Query decomposition may not be easy © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 71 / 96
- 194. Distributed RDF Processing– 2 Data summary-based approaches Build summaries (index) for the distributed RDF datasets (e.g., [Atre et al., 2010; Prasser et al., 2012]) © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 72 / 96
- 195. Distributed RDF Processing– 2 Data summary-based approaches Build summaries (index) for the distributed RDF datasets (e.g., [Atre et al., 2010; Prasser et al., 2012]) SPARQL query Q = {Q1, . . . , Qk} Distributed execution of {Q1, . . . , Qk} using the data summary © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 72 / 96
- 196. Distributed RDF Processing– 2 Data summary-based approaches Build summaries (index) for the distributed RDF datasets (e.g., [Atre et al., 2010; Prasser et al., 2012]) SPARQL query Q = {Q1, . . . , Qk} Distributed execution of {Q1, . . . , Qk} using the data summary No data re-partitioning and re-allocation Have to scan the data at each site Index over distributed data with maintenance concerns © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 72 / 96
- 197. SPARQL Endpoint Federation Consideronly the SPARQL endpoints for query execution No data re-partitioning/re-distribution Consider D = D1 ∪ D2 ∪ . . . ∪ Dn; Di : SPARQL endpoint © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 73 / 96
- 198. SPARQL Endpoint Federation Consideronly the SPARQL endpoints for query execution No data re-partitioning/re-distribution Consider D = D1 ∪ D2 ∪ . . . ∪ Dn; Di : SPARQL endpoint SPARQL query decomposed Q = {Q1, . . . , Qk} Distributed execution of {Q1, . . . , Qk} over {D1, . . . , Dn} © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 73 / 96
- 199. SPARQL Endpoint Federation Consideronly the SPARQL endpoints for query execution No data re-partitioning/re-distribution Consider D = D1 ∪ D2 ∪ . . . ∪ Dn; Di : SPARQL endpoint SPARQL query decomposed Q = {Q1, . . . , Qk} Distributed execution of {Q1, . . . , Qk} over {D1, . . . , Dn} Systems DARQ, FedX [Schwarte et al., 2011], SPLENDID [G¨orlitz and Staab, 2011], ANAPSID [Acosta et al., 2011] © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 73 / 96
- 200. SPARQL Endpoint Federation Consideronly the SPARQL endpoints for query execution No data re-partitioning/re-distribution Consider D = D1 ∪ D2 ∪ . . . ∪ Dn; Di : SPARQL endpoint SPARQL query decomposed Q = {Q1, . . . , Qk} Distributed execution of {Q1, . . . , Qk} over {D1, . . . , Dn} Systems DARQ, FedX [Schwarte et al., 2011], SPLENDID [G¨orlitz and Staab, 2011], ANAPSID [Acosta et al., 2011] Data integration approach May be the only way to proceed if data is distributed Not all RDF data storage points are SPARQL endpoints © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 73 / 96
- 201. Distributed SPARQL UsingPartial Query Evaluation Two steps: 1 Evaluate a query at each site to find local matches Query is the function and each Di is the known input Inner match or local partial match D1 D2 D3 D4 © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 74 / 96
- 202. Distributed SPARQL UsingPartial Query Evaluation Two steps: 1 Evaluate a query at each site to find local matches Query is the function and each Di is the known input Inner match or local partial match 2 Assemble the partial matches to get final result Crossing match Centralized assembly Distributed assembly D1 D2 D3 D4 Crossing match © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 74 / 96
- 203. Outline 1 Introduction –Graph Types 2 Property Graph Processing Classification Online querying Offline analytics 3 RDF Graph Querying Data Warehousing Distributed SPARQL Execution Linked Object Data Querying © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 75 / 96
- 204. Linked Object Data– Closer Look © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 76 / 96
- 205. Globally Distributed Networkof Data © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 77 / 96
- 206. Traditional Hypertext-based WebAccess IMDb World Book Data exposed to the Web via HTML © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 78 / 96
- 207. Linked Data PublishingPrinciples IMDb World Book (http://...linkedmdb.../Shining,releaseDate, 23 May 1980) (http://...linkedmdb.../Shining, filmLocation, http://cia.../UK) (http://...linkedmdb.../29704,actedIn, http://...linkedmdb.../Shining) ... (http://cia.../UK, hasPopulation, 63230000) ... Shining UK Data model: RDF Global identifier: URI Access mechanism: HTTP Connection: data links © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 79 / 96
- 208. Live Query Processing Notall data resides at SPARQL endpoints Freshness of access to data important Potentially countably infinite data sources Live querying On-line execution Only rely on linked data principles Alternatives Traversal-based approaches Index-based approaches Hybrid approaches © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 80 / 96
- 209. Linked Data Model[Hartig, 2012] Web of Linked Data Given a finite or countably infinite set D of Linked Documents, a Web of Linked Data is a tuple W = (D, adoc, data) where: D ⊆ D, adoc is a partial mapping from URIs to D, and data is a total mapping from D to finite sets of RDF triples. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 81 / 96
- 210. Linked Data Model[Hartig, 2012] Web of Linked Data Given a finite or countably infinite set D of Linked Documents, a Web of Linked Data is a tuple W = (D, adoc, data) where: D ⊆ D, adoc is a partial mapping from URIs to D, and data is a total mapping from D to finite sets of RDF triples. Data Links A Web of Linked Data W = (D, adoc, data) contains a data link from document d ∈ D to document d ∈ D if there exists a URI u such that: u is mentioned in an RDF triple t ∈ data(d), and d = adoc(u). © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 81 / 96
- 211. SPARQL Query Semanticsin Live Querying Full-web semantics Scope of evaluating a SPARQL expression is all Linked Data Query result completeness cannot be guaranteed by any (terminating) execution © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 82 / 96
- 212. SPARQL Query Semanticsin Live Querying Full-web semantics Scope of evaluating a SPARQL expression is all Linked Data Query result completeness cannot be guaranteed by any (terminating) execution Reachability-based query semantics Query consists of a SPARQL expression, a set of seed URIs S, and a reachability condition c Scope: all data along paths of data links that satisfy the condition Computationally feasible © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 82 / 96
- 213. Traversal Approaches Discover relevantURIs recursively by traversing (specific) data links at query execution runtime [Hartig, 2013; Ladwig and Tran, 2011] Implements reachability-based query semantics Start from a set of seed URIs Recursively follow and discover new URIs Important issue is selection of seed URIs Retrieved data serves to discover new URIs and to construct result © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 83 / 96
- 214. Traversal Approaches Discover relevantURIs recursively by traversing (specific) data links at query execution runtime [Hartig, 2013; Ladwig and Tran, 2011] Implements reachability-based query semantics Start from a set of seed URIs Recursively follow and discover new URIs Important issue is selection of seed URIs Retrieved data serves to discover new URIs and to construct result Advantages Easy to implement. No data structure to maintain. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 83 / 96
- 215. Traversal Approaches Discover relevantURIs recursively by traversing (specific) data links at query execution runtime [Hartig, 2013; Ladwig and Tran, 2011] Implements reachability-based query semantics Start from a set of seed URIs Recursively follow and discover new URIs Important issue is selection of seed URIs Retrieved data serves to discover new URIs and to construct result Advantages Easy to implement. No data structure to maintain. Disadvantages Possibilities for parallelized data retrieval are limited Repeated data retrieval introduces significant query latency. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 83 / 96
- 216. Index Approaches Use pre-populatedindex to determine relevant URIs (and to avoid as many irrelevant ones as possible) Different index keys possible; e.g., triple patterns [Umbrich et al., 2011] Index entries a set of URIs Indexed URIs may appear multiple times (i.e., associated with multiple index keys) Each URI in such an entry may be paired with a cardinality (utilized for source ranking) Key: tp Entry: {uri1, uri2, , urin} GET urii © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 84 / 96
- 217. Index Approaches Use pre-populatedindex to determine relevant URIs (and to avoid as many irrelevant ones as possible) Different index keys possible; e.g., triple patterns [Umbrich et al., 2011] Index entries a set of URIs Indexed URIs may appear multiple times (i.e., associated with multiple index keys) Each URI in such an entry may be paired with a cardinality (utilized for source ranking) Key: tp Entry: {uri1, uri2, , urin} GET urii Advantages Data retrieval can be fully parallelized Reduces the impact of data retrieval on query execution time © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 84 / 96
- 218. Index Approaches Use pre-populatedindex to determine relevant URIs (and to avoid as many irrelevant ones as possible) Different index keys possible; e.g., triple patterns [Umbrich et al., 2011] Index entries a set of URIs Indexed URIs may appear multiple times (i.e., associated with multiple index keys) Each URI in such an entry may be paired with a cardinality (utilized for source ranking) Key: tp Entry: {uri1, uri2, , urin} GET urii Advantages Data retrieval can be fully parallelized Reduces the impact of data retrieval on query execution time Disadvantages Querying can only start after index construction Depends on what has been selected for the index Freshness may be an issue Index maintenance © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 84 / 96
- 219. Hybrid Approach Perform atraversal-based execution using a prioritized list of URIs to look up [Ladwig and Tran, 2010] Initial seed from the pre-populated index Non-seed URIs are ranked by a function based on information in the index New discovered URIs that are not in the index are ranked according to number of referring documents © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 85 / 96
- 220. Acknowledgements This presentation drawsupon collaborative research and discussions with the following colleagues (in alphabetical order) G¨une¸s Alu¸c, U. Waterloo Khaled Ammar, U. Waterloo Khuzaima Daudjee, U. Waterloo Young Han, U. Waterloo Olaf Hartig, U. Waterloo Lei Chen, Hong Kong UST Lei Zou, Peking Univ. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 86 / 96
- 221. Thank you! Research supportedby © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 87 / 96
- 222. © M. Tamer¨Ozsu Croucher ASI (2015/12/16-18) 88 / 96
- 223. References I Abadi, D.J., Marcus, A., Madden, S., and Hollenbach, K. (2009). SW-Store: a vertically partitioned DBMS for semantic web data management. VLDB J., 18(2):385–406. Abadi, D. J., Marcus, A., Madden, S. R., and Hollenbach, K. (2007). Scalable semantic web data management using vertical partitioning. In Proc. 33rd Int. Conf. on Very Large Data Bases, pages 411–422. Acosta, M., Vidal, M.-E., Lampo, T., Castillo, J., and Ruckhaus, E. (2011). ANAPSID: an adaptive query processing engine for SPARQL endpoints. In Proc. 10th Int. Semantic Web Conf., pages 18–34. Alu¸c, G., ¨Ozsu, M. T., and Daudjee, K. (2015). Clustering RDF databases using Tunable-LSH. CoRR, abs/1504.02523. Alu¸c, G., Hartig, O., ¨Ozsu, M. T., and Daudjee, K. (2014a). Diversified stress testing of RDF data management systems. In Proc. 13th Int. Semantic Web Conf., pages 197–212. Alu¸c, G., ¨Ozsu, M. T., and Daudjee, K. (2014b). Workload matters: Why RDF databases need a new design. Proc. VLDB Endowment, 7(10):837–840. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 89 / 96
- 224. References II Alu¸c, G.,¨Ozsu, M. T., Daudjee, K., and Hartig, O. (2013). chameleon-db: a workload-aware robust RDF data management system. Technical Report CS-2013-10, University of Waterloo. Available at https://cs.uwaterloo.ca/sites/ca. computer-science/files/uploads/files/CS-2013-10.pdf. Alu¸c, G., ¨Ozsu, M. T., Daudjee, K., and Hartig, O. (2015). Executing queries over schemaless RDF databases. In Proc. 31st Int. Conf. on Data Engineering, pages 807–818. Ammar, K. and ¨Ozsu, M. T. (2015). Approaches to graph processing – an overview. In preparation. Arias, M., Fern´andez, J. D., Mart´ınez-Prieto, M. A., and de la Fuente, P. (2011). An empirical study of real-world SPARQL queries. CoRR, abs/1103.5043. Atre, M., Chaoji, V., Zaki, M. J., and Hendler, J. A. (2010). Matrix “bit” loaded: A scalable lightweight join query processor for rdf data. In Proc. 19th Int. World Wide Web Conf., pages 41–50. Bornea, M. A., Dolby, J., Kementsietsidis, A., Srinivas, K., Dantressangle, P., Udrea, O., and Bhattacharjee, B. (2013). Building an efficient RDF store over a relational database. In Proc. ACM SIGMOD Int. Conf. on Management of Data, pages 121–132. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 90 / 96
- 225. References III Bu, Y.,Howe, B., Balazinska, M., and Ernst, M. D. (2012). The HaLoop approach to large-scale iterative data analysis. VLDB J., 21(2):169–190. Dean, J. and Ghemawat, S. (2008). Mapreduce: Simplified data processing on large clusters. Commun. ACM, 51(1):107–113. Duan, S., Kementsietsidis, A., Srinivas, K., and Udrea, O. (2011). Apples and oranges: a comparison of RDF benchmarks and real RDF datasets. In Proc. ACM SIGMOD Int. Conf. on Management of Data, pages 145–156. Gonzalez, J. E., Xin, R. S., Dave, A., Crankshaw, D., Franklin, M. J., and Stoica, I. (2014). GraphX: graph processing in a distributed dataflow framework. In Proc. 11th USENIX Symp. on Operating System Design and Implementation, pages 599–613. G¨orlitz, O. and Staab, S. (2011). SPLENDID: SPARQL endpoint federation exploiting VOID descriptions. In Proc. 2nd Int. Workshop on Consuming Linked Data. Gurajada, S., Seufert, S., Miliaraki, I., and Theobald, M. (2014). TriAD: A distributed shared-nothing RDF engine based on asynchronous message passing. In Proc. ACM SIGMOD Int. Conf. on Management of Data, pages 289–300. Han, M., Daudjee, K., Ammar, K., ¨Ozsu, M. T., Wang, X., and Jin, T. (2014). An experimental comparison of Pregel-like graph processing systems. Proc. VLDB Endowment, 7(12):1047–1058. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 91 / 96
- 226. References IV Hartig, O.(2012). SPARQL for a web of linked data: Semantics and computability. In Proc. 9th Extended Semantic Web Conf., pages 8–23. Hartig, O. (2013). SQUIN: a traversal based query execution system for the web of linked data. In Proc. ACM SIGMOD Int. Conf. on Management of Data, pages 1081–1084. Huang, J., Abadi, D. J., and Ren, K. (2011). Scalable SPARQL querying of large RDF graphs. Proc. VLDB Endowment, 4(11):1123–1134. Husain, M. F., McGlothlin, J., Masud, M. M., Khan, L. R., and Thuraisingham, B. (2011). Heuristics-based query processing for large RDF graphs using cloud computing. IEEE Trans. Knowl. and Data Eng., 23(9):1312–1327. Kaoudi, Z. and Manolescu, I. (2015). RDF in the clouds: A survey. VLDB J., 24:67–91. Kirchberg, M., Ko, R. K. L., and Lee, B.-S. (2011). From linked data to relevant data – time is the essence. CoRR, abs/1103.5046. Ladwig, G. and Tran, T. (2010). Linked data query processing strategies. In Proc. 9th Int. Semantic Web Conf., pages 453–469. Ladwig, G. and Tran, T. (2011). SIHJoin: Querying remote and local linked data. In Proc. 8th Extended Semantic Web Conf., pages 139–153. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 92 / 96
- 227. References V Lee, K.and Liu, L. (2013). Scaling queries over big rdf graphs with semantic hash partitioning. Proc. VLDB Endowment, 6(14):1894–1905. Li, F., Ooi, B. C., ¨Ozsu, M. T., and Wu, S. (2014). Distributed data management using MapReduce. ACM Comput. Surv., 46(3):Article No. 31. Low, Y., Gonzalez, J., Kyrola, A., Bickson, D., Guestrin, C., and Hellerstein, J. M. (2012). Distributed graphlab: A framework for machine learning in the cloud. Proc. VLDB Endowment, 5(8):716–727. Malewicz, G., Austern, M. H., Bik, A. J. C., Dehnert, J. C., Horn, I., Leiser, N., and Czajkowski, G. (2010). Pregel: a system for large-scale graph processing. In Proc. ACM SIGMOD Int. Conf. on Management of Data, pages 135–146. Michiardi, P. (2015). Introduction to spark internals. Slideshare. Available from: http://www.slideshare.net/michiard/introduction-to-spark-internals? qid=511145e7-79d7-41d8-a133-9e705d4933c3v=qf1b=from_search=11 [Last retrieved: 9 July 2015]. Neumann, T. and Weikum, G. (2008). RDF-3X: a RISC-style engine for RDF. Proc. VLDB Endowment, 1(1):647–659. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 93 / 96
- 228. References VI Neumann, T.and Weikum, G. (2009). The RDF-3X engine for scalable management of RDF data. VLDB J., 19(1):91–113. Prasser, F., Kemper, A., and Kuhn, K. A. (2012). Efficient distributed query processing for autonomous rdf databases. In Proc. 15th Int. Conf. on Extending Database Technology, pages 372–383. Schwarte, A., Haase, P., Hose, K., Schenkel, R., and Schmidt, M. (2011). Fedx: A federation layer for distributed query processing on linked open data. In Proc. 8th Extended Semantic Web Conf., pages 481–486. Tian, Y., Balmin, A., Corsten, S. A., Tatikonda, S., and McPherson, J. (2013). From “think like a vertex” to “think like a graph”. Proc. VLDB Endowment, 7(3):193–204. Umbrich, J., Hose, K., Karnstedt, M., Harth, A., and Polleres, A. (2011). Comparing data summaries for processing live queries over linked data. World Wide Web J., 14(5-6):495–544. Verborgh, R., Hartig, O., Meester, B. D., Haesendonck, G., Vocht, L. D., Sande, M. V., Cyganiak, R., Colpaert, P., Mannens, E., and de Walle, R. V. (2014). Querying datasets on the web with high availability. In Proc. 13th Int. Semantic Web Conf., pages 180–196. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 94 / 96
- 229. References VII Weiss, C.,Karras, P., and Bernstein, A. (2008). Hexastore: sextuple indexing for semantic web data management. Proc. VLDB Endowment, 1(1):1008–1019. Wilkinson, K. (2006). Jena property table implementation. Technical Report HPL-2006-140, HP Laboratories Palo Alto. Yan, D., Cheng, J., Lu, Y., and Ng, W. (2014). Blogel: A block-centric framework for distributed computation on real-world graphs. Proc. VLDB Endowment, 7(14):1981–1992. Zaharia, M. (2016). An Architecture for Fast and General Data Processing on Large Clusters. ACM Books. Forthcoming. Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauley, M., Franklin, M. J., Shenker, S., and Stoica, I. (2012). Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In Proc. 9th USENIX Symp. on Networked Systems Design Implementation, pages 2–2. Zaharia, M., Chowdhury, M., Franklin, M. J., Shenker, S., and Stoica, I. (2010). Spark: Cluster computing with working sets. In Proc. 2nd USENIX Workshop on Hot Topics in Cloud Computing, pages 10–10. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 95 / 96
- 230. References VIII Zhang, X.,Chen, L., Tong, Y., and Wang, M. (2013). EAGRE: Towards scalable I/O efficient SPARQL query evaluation on the cloud. In Proc. 29th Int. Conf. on Data Engineering, pages 565–576. Zou, L., Mo, J., Chen, L., ¨Ozsu, M. T., and Zhao, D. (2011). gStore: answering SPARQL queries via subgraph matching. Proc. VLDB Endowment, 4(8):482–493. Zou, L., ¨Ozsu, M. T., Chen, L., Shen, X., Huang, R., and Zhao, D. (2014). gStore: A graph-based SPARQL query engine. VLDB J., 23(4):565–590. © M. Tamer ¨Ozsu Croucher ASI (2015/12/16-18) 96 / 96