Efficient Distributed In-Memory Processing of RDF Datasets - PhD Viva

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Efﬁcient Distributed
In-Memory Processing of
RDF Datasets
Gezim Sejdiu
PhD Colloquium, Bonn 29.09.2020
Supervisor: Prof. D...

Introduction
Large-Scale RDF Dataset Statistics
Quality Assessment of RDF Datasets at Scale
Scalable RDF Querying
Use Case...

Introduction
Get me there!
3

No single deﬁnition
Extremely large data sets that may be analysed computationally to
reveal patterns, trends, and associa...

It’s relevance is increasing drastically and Big Data Analytics is an
emerging ﬁeld to explore
Why ‘BigData’ is so importa...

7
© Sorpresa meme on Memegen

Big Data Europe (BDE) Platform
8https://github.com/big-data-europe
Support Layer
Init Daemon
GUIs
Monitor
App Layer
Traffi...

Fast and generic-purpose cluster computing engine
Apache Spark
9
Spark Core Engine (RDD)
Deploy
SparkSQL&
DataFrames
CoreA...

Heterogeneity aka Variety
Key Observation From BDE
10
Banking
Finance
Our
Known
History
PurchaseEntertain
Gaming
Social
Me...

Modelling entities and their relationships
The RDF (Resource Description Framework) model
Knowledge Graphs
11
DPDHL Deutsc...

Modelling entities and their relationships
Analysis: ﬁnding underlying structure of the graph e.g. to predict
unknown rela...

Knowledge Graphs are everywhere
13
Entity Search and Summarization
Discovering Related Entities

Tasks that are hard to solve on single machines (>1 TB memory
consumption):
- Querying and processing LinkedGeoData
- Data...

Main Research Question
Is it possible to process large-scale RDF
datasets efﬁciently and effectively?
15

RQ1: How can we efﬁciently explore the structure of large-scale RDF
datasets?
RQ2: Can we scale RDF dataset quality assess...

RC1: A Scalable Distributed Approach for Computation of RDF Dataset
Statistics
RC2: A Scalable Framework for Quality Asses...

SANSA
Scalable Semantic Analytics Stack
18

SANSA [1] is a processing data flow engine that provides data
distribution, and fault tolerance for distributed computatio...

RQ1: How can we efﬁciently explore the structure of large-scale RDF
datasets?
RQ2: Can we scale RDF dataset quality assess...

Large-Scale RDF Dataset
Statistics
A Scalable Distributed Approach for
Computation of RDF Dataset Statistics [2]
21

Obtaining an overview over the Web of Data, it is important to gather
statistical information describing characteristics o...

A statistical criterion C is a triple C = (F, D, P), where:
- F is a SPARQL ﬁlter condition
- D is a derived dataset from ...

Architecture Overview
24

Experimental Setup
- Cluster conﬁguration
- 6 machines (1 master, 5 workers): Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
(3...

Distributed Processing on Large-Scale Datasets
* e) = d) / c) - 1
Evaluation
26
Runtime (in hours)
LODStats DistLODStats
a...

Performance evaluation of DistLODStats
Evaluation
27Node scalability (BSBM-50GB) Sizeup scalability

RQ1: How can we efﬁciently explore the structure of large-scale RDF
datasets?
RQ2: Can we scale RDF dataset quality assess...

Quality Assessment of RDF
Datasets at Scale
A Scalable Framework for Quality Assessment of
RDF Datasets [3]
29

Assessing data quality is of paramount importance to judge its ﬁtness
for particular use case
Existing solutions can not e...

Quality Assessment Pattern (QAP)
- A reusable template to implement and design scalable quality
metrics
Approach
31
Qualit...

Architecture Overview
32
Definition
● Define quality dimensions
● Define quality metrics, threshold and other configuratio...

Experimental Setup
- Cluster conﬁguration
- 7 machines (1 master, 6 workers): Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
(3...

Evaluation
34
Runtime (in minutes)
Luzzu DistQualityAssessment
-----> a) single b) joint c) local d) cluster
LinkedGeoData...

Performance evaluation of DistQualityAssessment
Evaluation
35Node scalability (BSBM-200GB) Sizeup scalability

RQ1: How can we efﬁciently explore the structure of large-scale RDF
datasets?
RQ2: Can we scale RDF dataset quality assess...

Scalable RDF Querying
Sparklify: A Scalable Software Component for
Efﬁcient evaluation of SPARQL queries over
distributed ...

Existing solutions are narrowed down to simple RDF constructs only
Hence they do not exploit the full potential of the kno...

Sparklify: Architecture Overview
39
Sparqlify
SANSA
SANSA Engine
RDF Layer
Data Ingestion
Partitioning
Query Layer
Sparkli...

Experimental Setup
- Cluster conﬁguration
- 7 nodes (1 master, 6 worker), each with Intel(R) Xeon(R) CPU E5-2620 v4 @
2.10...

Evaluation
41
Runtime (s) (mean)
SPARQLGX-SDE Sparklify
-----> a) total b) partitioning c) querying d) total
QC 103.24 134...

Evaluation
42
Runtime (s) (mean)
SPARQLGX-SDE Sparklify
-----> a) total b) partitioning c) querying d) total
Q1 1056.83 62...

Performance evaluation of Sparklify
Evaluation
43Node scalability (WatDiv 100M) Sizeup scalability

Sparklify vs SPARQLGX-SDE per query type performance on WatDiv
100M
Evaluation
44Query Types: (QS: Star pattern, QL: Linea...

Scalable RDF Querying
Towards A Scalable Semantic-based Distributed
Approach for SPARQL query evaluation [5]
45

Are existing solutions more effective i.e. using property tables which
leads to reducing the number of necessary joins and...

Semantic-Based: Architecture Overview
47
SANSA Engine
RDF Layer
Data Ingestion
Partitioning
Query Layer
Semantic
map map
D...

Experimental Setup
- Cluster conﬁguration
- 6 machines (1 master, 5 workers): Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
(3...

Evaluation
49
Runtime (s) (mean)
Queries SHARD
SPARQLGX-SD
E SANSA.Sparklify SANSA.Semantic
C3 n/a 38.79 72.94 90.48
F3 n/...

Evaluation
50
Runtime (s) (mean)
Queries SHARD
SPARQLGX-SD
E SANSA.Sparklify SANSA.Semantic
Q1 774.93 103.74 103.57 226.21...

Performance evaluation of Semantic-based approach
Evaluation
51Node scalability (LUBM-1K) Sizeup scalability

Powered By
Project and Organizations using our proposed
approaches
52

53
<https://aleth.io/>
Blockchain – Alethio
Use Case
Alethio is using SANSA in order to
perform large-scale batch
analytic...

The Hubs and Authorities Transaction
Network Analysis
54
Amazon S3
buckets
EthOn RDF
triples
Connected Components
SANSA En...

Analyze game performance and customer behaviors at scale
Profiting from Kitties on Ethereum
55

Pipe different clustering algorithms at once
Scalable Integration of Big POI Data
56
RDF POI
Data
Pre
processing
SPARQL
Fi...

Conclusion and Future
Directions
57

RQ1: How can we efﬁciently explore the structure of large-scale RDF
datasets?
- First algorithm for computing RDF dataset ...

RQ2: Can we scale RDF dataset quality assessment horizontally?
- A Quality Assessment Pattern QAP to characterize scalable...

RQ3: Can distributed RDF datasets be queried efﬁciently and
effectively?
- A novel approach for vertical partitioning incl...

Large-scale RDF Dataset Statistics
- Our approach is purely batch processing, in which the data chunks
are normally very l...

Assessment of RDF Datasets at Scale
- Intelligent partitioning strategies and perform dependency analysis
in order to eval...

Scalable RDF Querying
- Combine OBDA tools with dictionary encoding of RDF terms as
integers and evaluate the effects
- Ex...

Adaptive Distributed RDF Querying
- Optimize index structures and distribute data based on anticipated
query workloads of ...

With the increasing amount of the RDF data, processing large-scale RDF
datasets are constantly facing challenges
We have s...

66
@Gezim_Sejdiu
https://gezimsejdiu.github.io/
That’s all folks
>> SANSA: https://github.com/SANSA-Stack

[1]. Distributed Semantic Analytics using the SANSA Stack. Jens Lehmann; Gezim Sejdiu; Lorenz Bühmann; Patrick
Westphal; C...

Backup slides
68

SPARQL is a standard query language for retrieving and manipulating
RDF data
PREFIX dbr: <http://dbpedia.org/resource/>
PR...

Over the last years, the size of the Semantic Web has increased and
several large-scale datasets were published
> As of Ma...

Speedup Ratio and Efﬁciency of DistLODStats
Evaluation
71

Overall Breakdown of DistLODStats by Criterion Analysis (log scale)
Evaluation
72

STATisfy: A REST Interface for DistLODStats
73
CollaborativeAnalyticsServices
Marketplace
REST
Server
BigDataEurope
Local ...

QAP: consists of transformations and actions
- Transformation: Rule set or a union/intersection of transformations
- Rules...

Overall analysis of DistQualityAssessment by metric in the cluster mode
(log scale)
Evaluation
75

Overall analysis of queries on LUBM-1K dataset (cluster mode) using
Semantic-based approach
Evaluation
76

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1035460 views
Storytelling For The Web: Integrate... by Chiara Aliotta 947335 views
Artificial Intelligence, Data and C... by OECD Directorate ... 869118 views
How to Leverage AI to Boost Employe... by SocialHRCamp 364901 views
2024 State of Marketing Report – by... by Marius Sescu 207323 views
Everything You Need To Know About C... by Expeed Software 227648 views

View on Slideshare

1 of 76 Ad

View on Slideshare

1 of 76 Ad