The document discusses approaches for federating SPARQL queries over the web of data. It describes SPARQL endpoint federation, linked data federation, and distributed hash tables approaches. It also discusses techniques for optimizing query federation, including query rewriting, source selection, join order selection, and join implementations. Source selection algorithms discussed include index-free using SPARQL ASK queries, index-only using data summaries, and hybrid approaches.
FedX - Optimization Techniques for Federated Query Processing on Linked Dataaschwarte
The final slides of our talk about FedX at the 10th International Semantic Web Conference in Bonn. For details about FedX see http://www.fluidops.com/fedx/
HiBISCuS: Hypergraph-Based Source Selection for SPARQL Endpoint FederationMuhammad Saleem
Efficient federated query processing is of significant importance to tame the large amount of data available on the Web of Data. Previous works have focused on generating optimized query execution plans for fast result retrieval. However, devising source selection approaches beyond triple pattern-wise source selection has not received much attention. This work presents HiBISCuS, a novel hypergraph-based source selection approach to federated SPARQL querying. Our approach can be directly combined with existing SPARQL query federation engines to achieve the same recall while querying fewer data sources. We extend three well-known SPARQL query federation engines with HiBISCus and compare our extensions with the original approaches on FedBench. Our evaluation shows that HiBISCuS can efficiently reduce the total number of sources selected without losing recall. Moreover, our approach significantly reduces the execution time of the selected engines on most of the benchmark queries.
A single interface for accessing life sciences (LS) data is a natural consequence to master the data deluge in this domain. The data in the LS requires integration and current integrative solutions increasingly rely on the federation of queries for distributed resources. We introduce a federated query processing system name ``BioFed", customised for LS-LOD. BioFed federates SPARQL queries over more than 130 public SPARQL endpoints.
FedX - Optimization Techniques for Federated Query Processing on Linked Dataaschwarte
The final slides of our talk about FedX at the 10th International Semantic Web Conference in Bonn. For details about FedX see http://www.fluidops.com/fedx/
HiBISCuS: Hypergraph-Based Source Selection for SPARQL Endpoint FederationMuhammad Saleem
Efficient federated query processing is of significant importance to tame the large amount of data available on the Web of Data. Previous works have focused on generating optimized query execution plans for fast result retrieval. However, devising source selection approaches beyond triple pattern-wise source selection has not received much attention. This work presents HiBISCuS, a novel hypergraph-based source selection approach to federated SPARQL querying. Our approach can be directly combined with existing SPARQL query federation engines to achieve the same recall while querying fewer data sources. We extend three well-known SPARQL query federation engines with HiBISCus and compare our extensions with the original approaches on FedBench. Our evaluation shows that HiBISCuS can efficiently reduce the total number of sources selected without losing recall. Moreover, our approach significantly reduces the execution time of the selected engines on most of the benchmark queries.
A single interface for accessing life sciences (LS) data is a natural consequence to master the data deluge in this domain. The data in the LS requires integration and current integrative solutions increasingly rely on the federation of queries for distributed resources. We introduce a federated query processing system name ``BioFed", customised for LS-LOD. BioFed federates SPARQL queries over more than 130 public SPARQL endpoints.
This presentation was given at the International Workshop on Interacting with Linked Data (ILD 2012) co-located with the 9th Extended Semantic Web Conference 2012, Heraklion, and is related the publication of the same title.
Much research has been done to combine the fields of Data-bases and Natural Language Processing. While many works focus on the problem of deriving a structured query for a given natural language question, the problem of query verbalization -- translating a structured query into natural language -- is less explored. In this work we describe our approach to verbalizing SPARQL queries in order to create natural language expressions that are readable and understandable by the human day-to-day user. These expressions are helpful when having search engines that generate SPARQL queries for user-provided natural language questions or keywords. Displaying verbalizations of generated queries to a user enables the user to check whether the right question has been understood. While our approach enables verbalization of only a subset of SPARQL 1.1, this subset applies to 90% of the 209 queries in our training set. These observations are based on a corpus of SPARQL queries consisting of datasets from the QALD-1 challenge and the ILD2012 challenge.
The publication is available at http://www.aifb.kit.edu/images/b/b7/VerbalizingSparqlQueries.pdf
RDF is a general method to decompose knowledge into small pieces, with some rules about the semantics or meaning of those pieces. The point is to have a method so simple that it can express any fact, and yet so structured that computer applications can do useful things with knowledge expressed in RDF.
"SPARQL Cheat Sheet" is a short collection of slides intended to act as a guide to SPARQL developers. It includes the syntax and structure of SPARQL queries, common SPARQL prefixes and functions, and help with RDF datasets.
The "SPARQL Cheat Sheet" is intended to accompany the SPARQL By Example slides available at http://www.cambridgesemantics.com/2008/09/sparql-by-example/ .
This presentation looks in detail at SPARQL (SPARQL Protocol and RDF Query Language) and introduces approaches for querying and updating semantic data. It covers the SPARQL algebra, the SPARQL protocol, and provides examples for reasoning over Linked Data. We use examples from the music domain, which can be directly tried out and ran over the MusicBrainz dataset. This includes gaining some familiarity with the RDFS and OWL languages, which allow developers to formulate generic and conceptual knowledge that can be exploited by automatic reasoning services in order to enhance the power of querying.
Re-using Media on the Web: Media fragment re-mixing and playoutMediaMixerCommunity
A number of novel application ideas will be introduced based on the media fragment creation, specification and rights management technologies. Semantic search and retrieval allows us to organize sets of fragments by topical or conceptual relevance. These fragment sets can then be played out in a non-linear fashion to create a new media re-mix. We look at a server-client implementation supporting Media Fragments, before allowing the participants to take the sets of media they have selected and create their own re-mix.
This presentation was given at the International Workshop on Interacting with Linked Data (ILD 2012) co-located with the 9th Extended Semantic Web Conference 2012, Heraklion, and is related the publication of the same title.
Much research has been done to combine the fields of Data-bases and Natural Language Processing. While many works focus on the problem of deriving a structured query for a given natural language question, the problem of query verbalization -- translating a structured query into natural language -- is less explored. In this work we describe our approach to verbalizing SPARQL queries in order to create natural language expressions that are readable and understandable by the human day-to-day user. These expressions are helpful when having search engines that generate SPARQL queries for user-provided natural language questions or keywords. Displaying verbalizations of generated queries to a user enables the user to check whether the right question has been understood. While our approach enables verbalization of only a subset of SPARQL 1.1, this subset applies to 90% of the 209 queries in our training set. These observations are based on a corpus of SPARQL queries consisting of datasets from the QALD-1 challenge and the ILD2012 challenge.
The publication is available at http://www.aifb.kit.edu/images/b/b7/VerbalizingSparqlQueries.pdf
RDF is a general method to decompose knowledge into small pieces, with some rules about the semantics or meaning of those pieces. The point is to have a method so simple that it can express any fact, and yet so structured that computer applications can do useful things with knowledge expressed in RDF.
"SPARQL Cheat Sheet" is a short collection of slides intended to act as a guide to SPARQL developers. It includes the syntax and structure of SPARQL queries, common SPARQL prefixes and functions, and help with RDF datasets.
The "SPARQL Cheat Sheet" is intended to accompany the SPARQL By Example slides available at http://www.cambridgesemantics.com/2008/09/sparql-by-example/ .
This presentation looks in detail at SPARQL (SPARQL Protocol and RDF Query Language) and introduces approaches for querying and updating semantic data. It covers the SPARQL algebra, the SPARQL protocol, and provides examples for reasoning over Linked Data. We use examples from the music domain, which can be directly tried out and ran over the MusicBrainz dataset. This includes gaining some familiarity with the RDFS and OWL languages, which allow developers to formulate generic and conceptual knowledge that can be exploited by automatic reasoning services in order to enhance the power of querying.
Re-using Media on the Web: Media fragment re-mixing and playoutMediaMixerCommunity
A number of novel application ideas will be introduced based on the media fragment creation, specification and rights management technologies. Semantic search and retrieval allows us to organize sets of fragments by topical or conceptual relevance. These fragment sets can then be played out in a non-linear fashion to create a new media re-mix. We look at a server-client implementation supporting Media Fragments, before allowing the participants to take the sets of media they have selected and create their own re-mix.
Information access over linked data requires to determine
subgraph(s), in linked data's underlying graph, that correspond to the required information need. Usually, an information access framework is able to retrieve richer information by checking of a large number of possible subgraphs. However, on the ecking of a large number of possible subgraphs increases information access complexity. This makes information access frameworks less eective. A large number of contemporary linked data information access frameworks reduce the complexity by introducing dierent heuristics but they suer on retrieving richer information. Or, some frameworks do not care about the complexity. However, a practically usable framework should retrieve richer information with lower complexity. In linked data information access, we hypothesize that pre-processed data statistics of linked data can be used to eciently check a large number of possible subgraphs. This will help to retrieve comparatively richer information with lower data access complexity. Preliminary evaluation of our proposed hypothesis shows promising performance.
Invited talk at USEWOD2014 (http://people.cs.kuleuven.be/~bettina.berendt/USEWOD2014/)
A tremendous amount of machine-interpretable information is available in the Linked Open Data Cloud. Unfortunately, much of this data remains underused as machine clients struggle to use the Web. I believe this can be solved by giving machines interfaces similar to those we offer humans, instead of separate interfaces such as SPARQL endpoints. In this talk, I'll discuss the Linked Data Fragments vision on machine access to the Web of Data, and indicate how this impacts usage analysis of the LOD Cloud. We all can learn a lot from how humans access the Web, and those strategies can be applied to querying and analysis. In particular, we have to focus first on solving those use cases that humans can do easily, and only then consider tackling others.
Presentation done* at the 13th International Semantic Web Conference (ISWC) in which we approach a compressed format to represent RDF Data Streams. See the original article at: http://dataweb.infor.uva.es/wp-content/uploads/2014/07/iswc14.pdf
* Presented by Alejandro Llaves (http://www.slideshare.net/allaves)
MULDER: Querying the Linked Data Web by Bridging RDF Molecule TemplatesKemele M. Endris
The increasing number of RDF data sources that allow for
querying Linked Data via Web services form the basis for federated SPARQL query processing. Federated SPARQL query engines provide a unified view of a federation of RDF data sources, and rely on source descriptions for selecting the data sources over which unified queries will be executed. Albeit efficient, existing federated SPARQL query engines usually ignore the meaning of data accessible from a data source,
and describe sources only in terms of the vocabularies utilized in the data source. Lack of source description may conduce to the erroneous selection of data sources for a query, thus affecting the performance of query processing over the federation. We tackle the problem of federated SPARQL query processing and devise MULDER, a query engine for federations of RDF data sources. MULDER describes data sources in terms of RDF molecule templates, i.e., abstract descriptions of entities
belonging to the same RDF class. Moreover, MULDER utilizes RDF molecule templates for source selection, and query decomposition and optimization. We empirically study the performance of MULDER on existing benchmarks, and compare MULDER performance with state-of-the-art federated SPARQL query engines. Experimental results suggest that RDF molecule templates empower MULDER federated query processing, and allow for the selection of RDF data sources that not only reduce execution time, but also increase answer completeness.
Two graph data models : RDF and Property Graphsandyseaborne
Talk given at ApacheConEU Big Data 2015.
This talk describes the two common graph data approaches, RDF and Property Graphs. It concludes with observations about the different emphasis of each and where each is focused.
SPARQL Query Verbalization for Explaining Semantic Search Engine QueriesBasil Ell
This presentation was given at the 11th Extended Semantic Web Conference (ESWC '14), Anissaras/Heronissou, Crete, Greece, and is related the publication of the same title.
In this paper we introduce Spartiqulation, a system that translates SPARQL queries into English text. Our aim is to allow casual end users of semantic applications with limited to no expertise in the SPARQL query language to interact with these applications in a more intuitive way. The verbalization approach exploits domain-independent template-based natural language generation techniques, as well as linguistic cues in labels and URIs.
The life sciences domain has been one of the early adopters
of linked data and, a considerable portion of the Linked Open Data cloud is comprised of datasets from Life Sciences Linked Open Data (LSLOD). The deluge of biomedical data in the last few years, partially caused by the advent of high-throughput gene sequencing technologies, has been a primary motivation for these efforts. This success has lead to the growth in size of data sets and to the need for integrating multiples of these data-sets. This growth requires large scale distributed infrastructure and specific techniques for managing large linked data graphs. Especially in combination with Semantic Web and Linked Data technologies these promises to enable the processing of large as well as semantically heterogeneous data sources and the capturing of new knowledge from those. In this tutorial we present the state of the art in large data processing, as well as the amalgamation with Linked Data and Semantic Web technologies for better knowledge discovery and targeted applications. We aim to provide useful information for the Knowledge Acquisition research community as well as the working Data Scientist.
Linked lists represent a countable number of ordered values, and are among the most important abstract data types in computer science. With the advent of RDF as a highly expressive knowledge representation language for the Web, various implementations for RDF lists have been proposed. Yet, there is no benchmark so far dedicated to evaluate the performance of triple stores and SPARQL query engines on dealing with ordered linked data. Moreover, essential tasks for evaluating RDF lists, like generating datasets containing RDF lists of various sizes, or generating the same RDF list using different modelling choices, are cumbersome and unprincipled. In this paper, we propose List.MID, a systematic benchmark for evaluating systems serving RDF lists. List.MID consists of a dataset generator, which creates RDF list data in various models and of different sizes; and a set of SPARQL queries. The RDF list data is coherently generated from a large, community-curated base collection of Web MIDI files, rich in lists of musical events of arbitrary length. We describe the List.MID benchmark, and discuss its impact and adoption, reusability, design, and availability.
A presentation on Application Architecture for Semantic Web Applications based on chapter 4 of the book Semantic Web for the Working Ontologist by Dean Allemang and Jim Hendler. It focusses on RDF parsing and serialising and RDF stores.
How Representative Is a SPARQL Benchmark? An Analysis of RDF Triplestore Benc...Muhammad Saleem
Triplestores are data management systems for storing and querying RDF data. Over recent years, various benchmarks have been proposed to assess the performance of triplestores across different performance measures. However, choosing the most suitable benchmark for evaluating triplestores in practical settings is not a trivial task. This is because triplestores experience varying workloads when deployed in real applications. We address the problem of determining an appropriate benchmark for a given real-life workload by providing a fine-grained comparative analysis of existing triplestore benchmarks. In particular, we analyze the data and queries provided with the existing triplestore benchmarks in addition to several real-world datasets. Furthermore, we measure the correlation between the query execution time and various SPARQL query features and rank those features based on their significance levels. Our experiments reveal several interesting insights about the design of such benchmarks. With this fine-grained evaluation, we aim to support the design and implementation of more diverse benchmarks. Application developers can use our result to analyze their data and queries and choose a data management system.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
When stars align: studies in data quality, knowledge graphs, and machine lear...
Federated SPARQL query processing over the Web of Data
1. Federated SPARQL Query Processing
Over the Web of Data
Muhammad Saleem, Axel-Cyrille Ngonga
Ngomo
Agile Knowledge Engineering and Semantic Web (AKSW), University of Leipzig,
Germany, 25/11/2014
2. Agenda
• SPARQL Query Federation Approaches
• SPARQL Query Federation Optimization
– Query Rewriting
– Source Selection
– Data Integration Options
– Join Order Selection
– Join Order Optimization
– Join Implementations
• Performance Metrics and Discussion
4. SPARQL Endpoint Federation Approaches
• Most commonly used approaches
• Make use of SPARQL endpoints URLs
• Fast query execution
• RDF data needs to be exposed via SPARQL
endpoints
• E.g., HiBISCus, FedX, SPLENDID, ANAPSID, LHD etc.
5. Linked Data Federation Approaches
• Data needs not be exposed via SPARQL endpoints
• Uses URI lookups at runtime
• Data should follow Linked Data principles
• Slower as compared to previous approaches
• E.g., LDQPS, SIHJoin, WoDQA etc.
6. Query federation on top of Distributed Hash Tables
• Uses DHT indexing to federate SPARQL queries
• Space efficient
• Cannot deal with whole LOD
• E.g., ATLAS
7. Hybrid of SEF+LDF
• Federation over SPARQL endpoints and Linked
Data
• Can potentially deal with whole LOD
• E.g., ADERIS-Hybrid
17. Types of Source Selection
• Index-free
– Using SPARQL ASK queries
– No index maintenance required
– Potentially ensures result set completeness
– SPARQL ASK queries can be expensive
– Can make use of the cache to store recent SPARQL ASK queries results
– E.g., FedX
• Index-only
– Only make use of Index/data summaries
– Less efficient but fast source selection
– Result set completeness is not ensured
– E.g., DARQ, LHD
• Hybrid
– Make use of index+SPARQL ASK
– Most efficient
– Result set completeness is not ensured
– Can make use of the cache to store recent SPARQL ASK queries results
– E.g., HiBISCuS, ANAPSID, SPLENDID
18. Index-free Source Selection
Input: SPARQL query Q , set of all data sources D
Output: Triple pattern to relevant data sources map M
for each triple pattern ti in SPARQL query Q
Ri = {}; // set of relevant data sources for triple pattern ti
for each data source di in D
if SPARQL ASK(di , ti) = true
Ri = Ri U {di};
end if
end for
M = M U {Ri};
end for
return M What is the total number of SPARQL ASK requests used?
total number of triple patterns * total number of data sources
19. Index-free
Source Selection
FedBench (LD3): Return for all US presidents their party
membership and news pages about them.
SELECT ?president ?party ?page
WHERE {
?president rdf:type dbpedia:President .
?president dbpedia:nationality dbpedia:United_States .
?president dbpedia:party ?party .
?x nyt:topicPage ?page .
?x owl:sameAs ?president .
}
dbpedia
RDF
//TP3
//TP4
//TP5
Source Selection Algorithm
Triple pattern-wise source selection
TP1 = S1
KEGG
RDF
ChEBI
RDF
NYT
RDF
//TP1
SWDF
RDF
//TP2
LMDB
RDF
Jamendo
RDF
Geo
Names
RDF
DrugBank
RDF
S1 S2 S3 S4 S5 S6 S7 S8 S9
23. FedBench (LD3): Return for all US presidents their party
membership and news pages about them.
SELECT ?president ?party ?page
WHERE {
?president rdf:type dbpedia:President .
?president dbpedia:nationality dbpedia:United_States .
?president dbpedia:party ?party .
?x nyt:topicPage ?page .
?x owl:sameAs ?president .
}
dbpedia
RDF
//TP3
//TP4
//TP5
Source Selection Algorithm
Index-free
Source Selection
Triple pattern-wise source selection
TP1 = S1
KEGG
RDF
ChEBI
RDF
NYT
RDF
//TP1
SWDF
RDF
//TP2
LMDB
RDF
TP3 = S1 TP4 = S4
TP5 = S1 S2
S4-S9
Total number of SPARQL ASK requests used = 45
Total triple pattern-wise sources selected = 12
Jamendo
RDF
TP2 = S1
Geo
Names
RDF
DrugBank
RDF
S1 S2 S3 S4 S5 S6 S7 S8 S9
24. Index-only Source Selection (LHD)
Input: SPARQL query Q , set of all data sources D, data sources index I storing all distinct predicates for
all data sources in D
Output: Triple pattern to relevant data sources map M
for each triple pattern ti in SPARQL query Q
Ri = {}; // set of relevant data sources for triple pattern ti
p = Pred(ti) // predicate of ti
if (bound (p))
Ri = Lookup (I, p) // index lookup for predicate of ti
else
Ri = D ; // all data sources are relevant
end if
M = M U {Ri} ;
end for
return M Why it is the less efficient approach (i.e., greatly overestimate relevant data sources)?
• Source selection is only based on predicate of triple patterns
• Simply select all data sources for triple patterns having unbound predicates
25. Index-only
Source Selection
FedBench (LD3): Return for all US presidents their party
membership and news pages about them.
SELECT ?president ?party ?page
WHERE {
?president rdf:type dbpedia:President .
?president dbpedia:nationality dbpedia:United_States .
?president dbpedia:party ?party .
?x nyt:topicPage ?page .
?x owl:sameAs ?president .
}
dbpedia
RDF
//TP3
//TP4
//TP5
Source Selection Algorithm
Triple pattern-wise source selection
TP1 = S1-S9
KEGG
RDF
ChEBI
RDF
NYT
RDF
//TP1
SWDF
RDF
//TP2
LMDB
RDF
Jamendo
RDF
Geo
Names
RDF
DrugBank
RDF
S1 S2 S3 S4 S5 S6 S7 S8 S9
29. FedBench (LD3): Return for all US presidents their party
membership and news pages about them.
SELECT ?president ?party ?page
WHERE {
?president rdf:type dbpedia:President .
?president dbpedia:nationality dbpedia:United_States .
?president dbpedia:party ?party .
?x nyt:topicPage ?page .
?x owl:sameAs ?president .
}
dbpedia
RDF
//TP3
//TP4
//TP5
Source Selection Algorithm
Index-only
Source Selection
Triple pattern-wise source selection
TP1 =
KEGG
RDF
ChEBI
RDF
NYT
RDF
//TP1
SWDF
RDF
//TP2
LMDB
RDF
S1-S9
TP3 = S1 TP4 = S4
TP5 = S1 S2 S4-S9
Total number of SPARQL ASK requests used = 0
Total triple pattern-wise sources selected = 20
Jamendo
RDF
TP2 = S1
Geo
Names
RDF
DrugBank
RDF
S1 S2 S3 S4 S5 S6 S7 S8 S9
30. Hybrid Source Selection
Input: SPARQL query Q , set of all data sources D, data sources index I storing all distinct predicates for all data
sources in D
Output: Triple pattern to relevant data sources map M
for each triple pattern ti in SPARQL query Q
Ri = {}; // set of relevant data sources for triple pattern ti
s = Subj(ti) , p = Pred(ti) , o = Obj(ti) ; // subject, predicate, and object of ti
if (!bound (p) || bound (s) || bound (o) )
for each data source di in D
if SPARQL ASK(di , ti) = true
Ri = RiU {di};
end if
end for
else
Ri = Lookup (I, p) // index lookup for predicate of ti
end if
M = M U {Ri}
end for
return M
What is the total number of SPARQL ASK requests used?
total number of triple patterns with bound subject or bound object
or unbound predicate * total number of data sources
31. Hybrid Source
Selection
FedBench (LD3): Return for all US presidents their party
membership and news pages about them.
SELECT ?president ?party ?page
WHERE {
?president rdf:type dbpedia:President .
?president dbpedia:nationality dbpedia:United_States .
?president dbpedia:party ?party .
?x nyt:topicPage ?page .
?x owl:sameAs ?president .
}
dbpedia
RDF
//TP3
//TP4
//TP5
Source Selection Algorithm
Triple pattern-wise source selection
TP1 = S1
KEGG
RDF
ChEBI
RDF
NYT
RDF
//TP1
SWDF
RDF
//TP2
LMDB
RDF
Jamendo
RDF
Geo
Names
RDF
DrugBank
RDF
S1 S2 S3 S4 S5 S6 S7 S8 S9
35. FedBench (LD3): Return for all US presidents their party
membership and news pages about them.
SELECT ?president ?party ?page
WHERE {
?president rdf:type dbpedia:President .
?president dbpedia:nationality dbpedia:United_States .
?president dbpedia:party ?party .
?x nyt:topicPage ?page .
?x owl:sameAs ?president .
}
Anything still needs
to be improved?
dbpedia
RDF
//TP3
//TP4
//TP5
Source Selection Algorithm
Hybrid Source
Selection
Triple pattern-wise source selection
TP1 = S1
KEGG
RDF
ChEBI
RDF
NYT
RDF
//TP1
SWDF
RDF
//TP2
LMDB
RDF
TP3 = S1 TP4 = S4
TP5 = S1 S2
S4-S9
Total number of SPARQL ASK requests used = 18
Total triple pattern-wise sources selected = 12
Jamendo
RDF
TP2 = S1
Geo
Names
RDF
DrugBank
RDF
S1 S2 S3 S4 S5 S6 S7 S8 S9
36. Source Selection
• Triple pattern-wise source selection
– Ensures 100% recall
– Can over-estimate capable sources
– Can be expensive, e.g., total number of SPARQL ASK
requests used
– Performed by FedX, SPLENDID, LHD, DARQ, ADERIS etc.
• Join-aware triple-pattern wise source selection
– Ensures 100% recall
– May selects optimal/close to optimal capable sources
– Can be expensive, e.g., total number of SPARQL ASK
requests used
– Can significantly reduce the query execution time
– Performed by ANAPSID, HiBISCuS
37. HiBISCuS: Hypergraph-Based Source Selection for
SPARQL Endpoint Federation
• Hybrid source selection
• Join-aware triple-pattern wise source selection
• Makes use of the hypergraph representation of
SPARQL queries
• Makes use of the URI authorities
• Makes use of the cache to store recent SPARQL
ASK queries results
38. Motivation
FedBench (LD3): Return for all US presidents their party
membership and news pages about them.
SELECT ?president ?party ?page
WHERE {
?president rdf:type dbpedia:President .
?president dbpedia:nationality dbpedia:United_States .
?president dbpedia:party ?party .
?x nyt:topicPage ?page .
?x owl:sameAs ?president .
}
dbpedia
RDF
//TP3
//TP4
//TP5
Source Selection Algorithm
Triple pattern-wise source selection
TP1 = S1
KEGG
RDF
ChEBI
RDF
NYT
RDF
//TP1
SWDF
RDF
//TP2
LMDB
RDF
Jamendo
RDF
Geo
Names
RDF
DrugBank
RDF
S1 S2 S3 S4 S5 S6 S7 S8 S9
39. Motivation
FedBench (LD3): Return for all US presidents their party
membership and news pages about them.
SELECT ?president ?party ?page
WHERE {
?president rdf:type dbpedia:President .
?president dbpedia:nationality dbpedia:United_States .
?president dbpedia:party ?party .
?x nyt:topicPage ?page .
?x owl:sameAs ?president .
}
dbpedia
RDF
//TP3
//TP4
//TP5
Source Selection Algorithm
Triple pattern-wise source selection
TP1 = S1
KEGG
RDF
ChEBI
RDF
NYT
RDF
//TP1
SWDF
RDF
//TP2
LMDB
RDF
Jamendo
RDF
TP2 = S1
Geo
Names
RDF
DrugBank
RDF
S1 S2 S3 S4 S5 S6 S7 S8 S9
46. Problem Statement
• An overestimation of triple pattern-wise source selection can
be expensive
– Resources are wasted
– Query runtime is increased
– Extra traffic is generated
• How do we perform join-aware triple pattern wise source
selection in time efficient way?
47. HiBISCuS: Key Concept
• Makes use of the URI’s authorities
http://dbpedia.org/ontology/party
Scheme Authority Path
For URI details: http://tools.ietf.org/html/rfc3986
63. Complete Local Integration
• Triple patterns are individually and completely
evaluated against every endpoint
• Triple pattern results are locally integrated using
different join techniques, e.g., NLJ, Hash Join etc.
• Less efficient if query contains common
predicates such rdf:type and owl:sameAs
• Large amount of potentially irrelevant
intermediate results retrieval
64. Iterative Integration
• Evaluate query iteratively pattern by pattern
• Start with a single triple pattern
• Substitute mappings from previous triple pattern
in the subsequent evaluation
• Evaluate query in a NLJ fashion
• NLJ can cause many remote requests
• Block NLJ fashion minimize the remote requests
66. Join Order Selection
• Left-deep trees
– Joins take place in a left-to-right sequential order
– Result of the join is used as an outer input for the next join
– Used in FedX, DARQ
• Right-deep trees
– Joins take place in a right-to-left sequential order
– Result of the join is used as an inner input for the next join
• Bushy trees
– Joins take place in sub-tress both on left and right sides
– Used in ANAPSID
• Dynamic programming
– Used in SPLENDID
67. Join Order Selection Example
Compute Micronutrients using Drugbank and KEGG
SELECT ?drug ?title WHERE {
?drug drugbank:drugCategory drugbank-cat:micronutrient. // TP1
?drug drugbank:casRegistryNumber ?id . // TP2
?keggDrug rdf:type kegg:Drug . // TP3
?keggDrug bio2rdf:xRef ?id . // TP4
?keggDrug dc:title ?title . // TP5
}
67
휋 ? 푑푟푢푔, ? 푡푖푡푙푒
TP1 TP2
TP3
TP4
TP5
Left-deep tree
휋 ? 푑푟푢푔, ? 푡푖푡푙푒
TP1 TP2
TP3
TP4
TP5
Right-deep tree
Bushy tree
휋 ? 푑푟푢푔, ? 푡푖푡푙푒
TP1 TP2
TP3 TP5
TP4
Goal: Execute smallest cardinality joins first
69. Join Order Optimization
• Exclusive Groups
– Group triple patterns with the same relevant data source
– Evaluation in a single (remote) sub-query
– Push join to the data source, i.e., endpoint
• Variable count-heuristic
– Iteratively determine the join order based on free variables
count of triple patterns and groups
– Consider “resolved ” variable mappings from earlier iteration
• Using Selectivities
– Store distinct predicates, avg. subject selectivities , and avg.
object selectivities for each predicate in index
– Use the predicate count, avg. subject selectivities , and avg.
object selectivities to estimate the join cardinality
70. Exclusive Groups
SELECT ?President ?Party ?TopicPage WHERE {
?President rdf:type dbpedia-yago:PresidentsOfTheUnitedStates .
?President dbpedia:party ?Party .
?nytPresident owl:sameAs ?President .
?nytPresident nytimes:topicPage ?TopicPage .
}
Source Selection
@ DBpedia
@ DBpedia
@ DBpedia, NYTimes
@ NYTimes
Exclusive Group
Advantage:
Delegate joins to the endpoint by forming exclusive groups (i.e. executing the
respective patterns in a single subquery)
70
71. Exclusive Groups Join Order Optimization
2 Unoptimized Internal Representation
1 SPARQL Query
Compute Micronutrients using Drugbank and KEGG
SELECT ?drug ?title WHERE {
?drug drugbank:drugCategory drugbank-cat:micronutrient .
?drug drugbank:casRegistryNumber ?id .
?keggDrug rdf:type kegg:Drug .
?keggDrug bio2rdf:xRef ?id .
?keggDrug dc:title ?title .
}
3 Optimized Internal Representation
4x Local Join
=
4x NLJ
Exlusive Group
Remote Join
71
72. Selectivity Based Join Order Optimization
[] a sd:Service ;
sd:endpointUrl <http://localhost:8890/sparql> ;
sd:capability [
sd:predicate diseasome:name ;
sd:totalTriples 147 ; // Total number of triple patterns with predicate value sd:predicate
sd:avgSbjSel ``0.0068'' ; // 1/ distinct subjects with predicate value sd:predicate
sd:avgObjSel ``0.0069'' ; // 1/ distinct Objects with predicate value sd:predicate
] ;
sd:capability [
sd:predicate diseasome:chromosomalLocation ;
sd:totalTtriples 160 ;
sd:avgSbjSel ``0.0062'' ;
sd:avgObjSel ``0.0072'' ;
] ;
S1 P O1 .
S1 P O2 .
S2 P O1 .
S3 P O2 .
totalTriples = 4
avgSbjSel(p) = 1/3
avgObjSel(p) =1/2
73. Selectivity Based Join Order Optimization
• Triple pattern cardinality
• Join Cardinality
푝 = pred(tp) , 푇 = Total triple having predicate 푝
퐶(푡푝) =
푇 푖푓 푛푒푖푡ℎ푒푟 푠푢푏푗푒푐푡 푛표푟 표푏푗푒푐푡 푖푠 푏표푢푛푑
푇 × 푎푣푔푆푏푗푆푒푙 푝 푖푓 푠푢푏푗푒푐푡 푖푠 푏표푢푛푑
푇 × 푎푣푔푂푏푗푆푒푙 푝 푖푓표푏푗푒푐푡 푖푠 푏표푢푛푑
퐶(퐽 푡푝1, 푡푝2 ) =
퐶 푡푝1 × 퐶 푡푝2 × 푎푣푔푃푟푒푑퐽표푖푛푆푒푙 푡푝1 × 푎푣푔푃푟푒푑퐽표푖푛푆푒푙 푡푝2 푖푓 푝 − 푝 푗표푖푛
퐶 푡푝1 × 퐶 푡푝2 × 푎푣푔푆푏푗퐽표푖푛푆푒푙 푡푝1 × 푎푣푔푆푏푗퐽표푖푛푆푒푙 푡푝2 푖푓 푠 − 푠 푗표푖푛
퐶 푡푝1 × 퐶 푡푝2 × 푎푣푔푆푏푗퐽표푖푛푆푒푙 푡푝1 × 푎푣푔푂푏푗퐽표푖푛푆푒푙 푡푝2 푖푓 푠 − 표 푗표푖푛
How to calculate avgPredJoinSel, avgSbjJoinSel, and avgObjJoinSel?
DARQ selected 0.5 as the avgJoinSel value for all joins
75. Join Implementations
• Bound Joins
– Start with a single triple pattern (lowest cardinality)
– Substitute mappings from previous triple pattern in the
subsequent evaluation
– Bound Joins in NLJ fashion
• Execute bound joins in nested loop join fashion
• Too many remote requests
– Bound Joins in Block NLJ fashion
• Execute bound joins in block nested loop join fashion
• Make use of SPARQL UNION construct
• Remote requests are reduced by the block size
• Other Join techniques
– E.g, Hash Joins
76. Bound Joins in Block NLJ
SELECT ?President ?Party ?TopicPage WHERE {
?President rdf:type dbpedia:PresidentsOfTheUnitedStates .
?President dbpedia:party ?Party .
?nytPresident owl:sameAs ?President .
?nytPresident nytimes:topicPage ?TopicPage .
}
Assume that the following intermediate results have been computed as input for the last triple pattern
Block Input
“Barack Obama”
“George W. Bush”
…
Before (NLJ)
SELECT ?TopicPage WHERE { “Barack Obama” nytimes:topicPage ?TopicPage }
SELECT ?TopicPage WHERE { “George W. Bush” nytimes:topicPage ?TopicPage }
…
Now: Evaluation in a single remote request using a SPARQL UNION
construct + local post processing (SPARQL 1.0)
76
77. Parallelization and Pipelining
• Execute sub-queries concurrently on different data
sources
• Multithreaded worker pool to execute the joins
and UNION operators in parallel
• Pipelining approach for intermediate results
• See FedX and LHD implementations
79. Performance Metrics
• Efficient source selection in terms of
– Total triple pattern-wise sources selected
– Total number of SPARQL ASK requests used during source
selection
– Source selection time
• Query execution time
• Results completeness and correctness
• Number of remote requests during query execution
• Index compression ratio (1- index size/datadump size)
• See https://code.google.com/p/bigrdfbench/
80. Evaluation Setup
• Local dedicated network
• Local SPARQL endpoints (One per machine)
• Run each query 10 times and present the average results
• Statistically analyzed the results, e.g., Wilcoxon signed rank
test, student T-test
82. AKSW SPARQL Federation Publications
• HiBISCuS: Hypergraph-Based Source Selection for SPARQL Endpoint Federation by Muhammad
Saleem and Axel-Cyrille Ngonga Ngomo, in (ESWC, 2014)
• DAW: Duplicate-AWare Federated Query Processing over the Web of Data by Muhammad
Saleem Axel-Cyrille Ngonga Ngomo, Josiane Xavier Parreira , Helena Deus , and Manfred Hauswirth
, in (ISWC 2013).
• TopFed: TCGA Tailored Federated Query Processing and Linking to LOD by Muhammad Saleem,
Shanmukha Sampath , Axel-Cyrille Ngonga Ngomo , Aftab Iqbal, Jonas Almeida , and Helena F. Deus
, in (Journal of Biomedical Semantics, 2014).
• A Fine-Grained Evaluation of SPARQL Endpoint Federation Systems by Muhammad Saleem, Yasar
Khan, Ali Hasnain, Ivan Ermilov, and Axel-Cyrille Ngonga Ngomo , in (Semantic Web Journal, 2014)
• BigRDFBench: A Billion Triples Benchmark for SPARQL Query Federation by Muhammad Saleem,
Ali Hasnain, Axel-Cyrille Ngonga Ngomo , in (submitted WWW, 2015).
• SAFE: Policy-Aware SPARQL Query Federation Over RDF Data Cubes
By Yasar Khan, Muhammed Saleem , Aftab Iqbal, Muntazir Mehdi, Aidan Hogan, Panagiotis
Hasapis, Axel-Cyrille Ngonga Ngomo, Stefan Decker, and Ratnesh Sahay, in (SWAT4LS, 2014)
• QFed: Query Set For Federated SPARQL Query Benchmark by Nur Aini Rakhmawati, Sarasi lithsena
, Muhammad Saleem , Stefan Decker, in (iiWAS, 2014)
82