Publishing with
Triple Pattern Fragments
Ruben Taelman - @rubensworks
imec - Ghent University
1
Publishing with Triple Pattern Fragments
TPF server software
TPF client-side querying
Quad Pattern Fragments
Demo
2
Publishing with Triple Pattern Fragments
TPF server software
TPF client-side querying
Quad Pattern Fragments
Demo
3
http://linkeddatafragments.org/software/
JavaScript LDF server
Python
Perl
Ruby
PHP
Java
...
TPF server implementations in different languages
4
Requires Node.js ≥ 4.0
Installing and running the LDF server with Node
5
[sudo] npm install -g ldf-server
ldf-server config.json <port> <workers>
Documentation: https://github.com/LinkedDataFragments/Server.js
Requires Docker
Installing and running the LDF server with Docker
6
docker pull linkeddatafragments/server.js
docker run -p <port>:3000 -it --rm 
-v $(pwd)/config.json:/tmp/config.json 
ldf-server /tmp/config.json
Preconfigured LDF server with NGINX cache and web-client
Setting up a full stack with Docker (Compose)
7
docker-compose up
https://github.com/LinkedDataFragments/FullStackServer
{
"title": "My Linked Data Fragments server",
"datasources": {
"dbpedia": {
"title": "DBpedia 2014",
"type": "HdtDatasource",
"description": "DBpedia 2014 with an HDT back-end",
"settings": { "file": "data/dbpedia2014.hdt" }
},
"dbpedia-sparql": {
"title": "DBpedia 3.9 (Virtuoso)",
"type": "SparqlDatasource",
"description": "DBpedia 3.9 with a Virtuoso back-end",
"settings": { "endpoint": "http://dbpedia.restdesc.org/",
"defaultGraph": "http://dbpedia.org" }
}
}
}
LDF server is configured with config.json
8
{
"title": "My Linked Data Fragments server",
"datasources": {
"dbpedia": {
"title": "DBpedia 2014",
"type": "HdtDatasource",
"description": "DBpedia 2014 with an HDT back-end",
"settings": { "file": "data/dbpedia2014.hdt" }
},
"dbpedia-sparql": {
"title": "DBpedia 3.9 (Virtuoso)",
"type": "SparqlDatasource",
"description": "DBpedia 3.9 with a Virtuoso back-end",
"settings": { "endpoint": "http://dbpedia.restdesc.org/",
"defaultGraph": "http://dbpedia.org" }
}
}
}
Configure list of datasources
9
{
"title": "My Linked Data Fragments server",
"datasources": {
"dbpedia": {
"title": "DBpedia 2014",
"type": "HdtDatasource",
"description": "DBpedia 2014 with an HDT back-end",
"settings": { "file": "data/dbpedia2014.hdt" }
},
"dbpedia-sparql": {
"title": "DBpedia 3.9 (Virtuoso)",
"type": "SparqlDatasource",
"description": "DBpedia 3.9 with a Virtuoso back-end",
"settings": { "endpoint": "http://dbpedia.restdesc.org/",
"defaultGraph": "http://dbpedia.org" }
}
}
}
Load an HDT file
10
{
"title": "My Linked Data Fragments server",
"datasources": {
"dbpedia": {
"title": "DBpedia 2014",
"type": "HdtDatasource",
"description": "DBpedia 2014 with an HDT back-end",
"settings": { "file": "data/dbpedia2014.hdt" }
},
"dbpedia-sparql": {
"title": "DBpedia 3.9 (Virtuoso)",
"type": "SparqlDatasource",
"description": "DBpedia 3.9 with a Virtuoso back-end",
"settings": { "endpoint": "http://dbpedia.restdesc.org/",
"defaultGraph": "http://dbpedia.org" }
}
}
}
Act as a proxy to a SPARQL endpoint
11
HDT
N-Triples
Turtle
JSON-LD
SPARQL-endpoint
Different data sources are possible
12
by extending lib/datasources/Datasource.js
Return triple stream given triple pattern, offset and limit
...or write your own datasource implementation
13
Exposing multiple datasources as if it was one
Useful for fragmented datasets that can’t be merged
Compose multiple datasources
14
https://github.com/LinkedDataFragments/Server.js/blob/master/config/config-composite.json
Publishing with Triple Pattern Fragments
TPF server software
TPF client-side querying
Quad Pattern Fragments
Demo
15
http://linkeddatafragments.org/software/
JavaScript LDF client
Python
Perl
Java
...
Different client implementations exist as well
16
No installation required at all!
http://client.linkeddatafragments.org/
Use the web client
17
Requires Node.js ≥ 4.0
Run from command line or include in your source code
Installing and running the LDF client with Node
18
[sudo] npm install -g ldf-client
ldf-client <tpf-endpoint-url> <sparql-query-path>
Documentation: https://github.com/LinkedDataFragments/Client.js
Requires Docker
Installing and running the LDF client with Docker
19
docker pull linkeddatafragments/client.js
docker run -it --rm 
-v <sparql-query-path>:/tmp/query.sparql 
linkeddatafragments/client.js 
<tpf-endpoint-url> /tmp/query.sparql
Publishing with Triple Pattern Fragments
TPF server software
TPF client-side querying
Quad Pattern Fragments
20
Quads are triples extended with a fourth element, the graph
<s> <p> <o> <g>.
Triples sometimes need some context
21
Quad Pattern Fragments (QPF)
Adding the fourth element to the interface
22
SELECT *
FROM <http://example.org/graph0>
FROM NAMED <http://example.org/graph1>
WHERE {
GRAPH ?g {
?s ?p ?o.
}
}
QPF Clients are able to use quad features
23
QPF is backwards-compatible with TPF
24
TPF
QPF
TPF
QPF
Ignores graphs
Quad Pattern Fragments (WIP)
https://github.com/LinkedDataFragments/Server.js/tree/feature-qpf-latest
https://github.com/LinkedDataFragments/Client.js/tree/feature-qpf-latest
25
Publishing with Triple Pattern Fragments
TPF server software
TPF client-side querying
Quad Pattern Fragments
Demo
26
Let’s try overloading the DBpedia TPF endpoint!
27
http://fragments.dbpedia.org/
Uptime of 99.9967%
28
29
30
Some query types are very slow
~1 hour!
Server running TPF server software with any dataset
31
TPF server
software
RDF dataset
HDT dataset
SPARQL endpoint
...

EKAW - Publishing with Triple Pattern Fragments

  • 1.
    Publishing with Triple PatternFragments Ruben Taelman - @rubensworks imec - Ghent University 1
  • 2.
    Publishing with TriplePattern Fragments TPF server software TPF client-side querying Quad Pattern Fragments Demo 2
  • 3.
    Publishing with TriplePattern Fragments TPF server software TPF client-side querying Quad Pattern Fragments Demo 3
  • 4.
  • 5.
    Requires Node.js ≥4.0 Installing and running the LDF server with Node 5 [sudo] npm install -g ldf-server ldf-server config.json <port> <workers> Documentation: https://github.com/LinkedDataFragments/Server.js
  • 6.
    Requires Docker Installing andrunning the LDF server with Docker 6 docker pull linkeddatafragments/server.js docker run -p <port>:3000 -it --rm -v $(pwd)/config.json:/tmp/config.json ldf-server /tmp/config.json
  • 7.
    Preconfigured LDF serverwith NGINX cache and web-client Setting up a full stack with Docker (Compose) 7 docker-compose up https://github.com/LinkedDataFragments/FullStackServer
  • 8.
    { "title": "My LinkedData Fragments server", "datasources": { "dbpedia": { "title": "DBpedia 2014", "type": "HdtDatasource", "description": "DBpedia 2014 with an HDT back-end", "settings": { "file": "data/dbpedia2014.hdt" } }, "dbpedia-sparql": { "title": "DBpedia 3.9 (Virtuoso)", "type": "SparqlDatasource", "description": "DBpedia 3.9 with a Virtuoso back-end", "settings": { "endpoint": "http://dbpedia.restdesc.org/", "defaultGraph": "http://dbpedia.org" } } } } LDF server is configured with config.json 8
  • 9.
    { "title": "My LinkedData Fragments server", "datasources": { "dbpedia": { "title": "DBpedia 2014", "type": "HdtDatasource", "description": "DBpedia 2014 with an HDT back-end", "settings": { "file": "data/dbpedia2014.hdt" } }, "dbpedia-sparql": { "title": "DBpedia 3.9 (Virtuoso)", "type": "SparqlDatasource", "description": "DBpedia 3.9 with a Virtuoso back-end", "settings": { "endpoint": "http://dbpedia.restdesc.org/", "defaultGraph": "http://dbpedia.org" } } } } Configure list of datasources 9
  • 10.
    { "title": "My LinkedData Fragments server", "datasources": { "dbpedia": { "title": "DBpedia 2014", "type": "HdtDatasource", "description": "DBpedia 2014 with an HDT back-end", "settings": { "file": "data/dbpedia2014.hdt" } }, "dbpedia-sparql": { "title": "DBpedia 3.9 (Virtuoso)", "type": "SparqlDatasource", "description": "DBpedia 3.9 with a Virtuoso back-end", "settings": { "endpoint": "http://dbpedia.restdesc.org/", "defaultGraph": "http://dbpedia.org" } } } } Load an HDT file 10
  • 11.
    { "title": "My LinkedData Fragments server", "datasources": { "dbpedia": { "title": "DBpedia 2014", "type": "HdtDatasource", "description": "DBpedia 2014 with an HDT back-end", "settings": { "file": "data/dbpedia2014.hdt" } }, "dbpedia-sparql": { "title": "DBpedia 3.9 (Virtuoso)", "type": "SparqlDatasource", "description": "DBpedia 3.9 with a Virtuoso back-end", "settings": { "endpoint": "http://dbpedia.restdesc.org/", "defaultGraph": "http://dbpedia.org" } } } } Act as a proxy to a SPARQL endpoint 11
  • 12.
  • 13.
    by extending lib/datasources/Datasource.js Returntriple stream given triple pattern, offset and limit ...or write your own datasource implementation 13
  • 14.
    Exposing multiple datasourcesas if it was one Useful for fragmented datasets that can’t be merged Compose multiple datasources 14 https://github.com/LinkedDataFragments/Server.js/blob/master/config/config-composite.json
  • 15.
    Publishing with TriplePattern Fragments TPF server software TPF client-side querying Quad Pattern Fragments Demo 15
  • 16.
  • 17.
    No installation requiredat all! http://client.linkeddatafragments.org/ Use the web client 17
  • 18.
    Requires Node.js ≥4.0 Run from command line or include in your source code Installing and running the LDF client with Node 18 [sudo] npm install -g ldf-client ldf-client <tpf-endpoint-url> <sparql-query-path> Documentation: https://github.com/LinkedDataFragments/Client.js
  • 19.
    Requires Docker Installing andrunning the LDF client with Docker 19 docker pull linkeddatafragments/client.js docker run -it --rm -v <sparql-query-path>:/tmp/query.sparql linkeddatafragments/client.js <tpf-endpoint-url> /tmp/query.sparql
  • 20.
    Publishing with TriplePattern Fragments TPF server software TPF client-side querying Quad Pattern Fragments 20
  • 21.
    Quads are triplesextended with a fourth element, the graph <s> <p> <o> <g>. Triples sometimes need some context 21
  • 22.
    Quad Pattern Fragments(QPF) Adding the fourth element to the interface 22
  • 23.
    SELECT * FROM <http://example.org/graph0> FROMNAMED <http://example.org/graph1> WHERE { GRAPH ?g { ?s ?p ?o. } } QPF Clients are able to use quad features 23
  • 24.
    QPF is backwards-compatiblewith TPF 24 TPF QPF TPF QPF Ignores graphs
  • 25.
    Quad Pattern Fragments(WIP) https://github.com/LinkedDataFragments/Server.js/tree/feature-qpf-latest https://github.com/LinkedDataFragments/Client.js/tree/feature-qpf-latest 25
  • 26.
    Publishing with TriplePattern Fragments TPF server software TPF client-side querying Quad Pattern Fragments Demo 26
  • 27.
    Let’s try overloadingthe DBpedia TPF endpoint! 27 http://fragments.dbpedia.org/ Uptime of 99.9967%
  • 28.
  • 29.
  • 30.
    30 Some query typesare very slow ~1 hour!
  • 31.
    Server running TPFserver software with any dataset 31 TPF server software RDF dataset HDT dataset SPARQL endpoint ...