Injustice - Developers Among Us (SciFiDevCon 2024)
Linked open data - how to juggle with more than a billion triples
1. How to Juggle with more
than a Billion Triples?
Ansgar Scherp
Research Group on Data and
Web Science
Universität Mannheim
October 2012
Image source:
http://www.flickr.com/photos/pedromourapinheiro/2122754745/ 1
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide
2. My thanks go to …
• Marianna • Daniel Eißing
• Simon Schenk • Mathias Konrath
• Carsten Saathoff • Daniel Schmeiß
• Thomas Franz • Anton Baumesberger
• Thomas Gottron • Frederik Jochum
• Steffen Staab • Alexander Kleinen
• Arne Peters
• Bastian Krayer And many more …
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 2
3. Scenario
• Tim plans to travel
– from London
– to a customer in Cologne
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 3
4. Website of the German Railway
It works, why bother…?
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 4
5. Let„s Try Different Queries
Bottlenecks in public transportation?
Compare the connections with flights?
Visualize on a map?
…
All these queries cannot be answered,
because the data …
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 5
6. … locked in Silos!
– High Integration Effort
– Lack in Reuse of Data
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 6
B. Jagendorf, http://www.flickr.com/photos/bobjagendorf/, CC-BY
7. Linked Data
• Publishing and interlinking of data
• Different quality and purpose
• From different sources in the Web
World Wide Web Linked Data
Documents Data
Hyperlinks Typed Links
HTML RDF
Addresses (URIs) Addresses (URIs)
Example: http://www.uni-mannheim.de/
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 7
11. Example: Big Lynx
Matt Briggs
Scott Miller
?
Big Lynx
Company
Ansgar Scherp – ansgar@informatik.uni-mannheim.de
< 31 Milliarde Triple Source: http://lod-cloud.net
Slide 11
12. 1. Use URIs for Identification
Matt Briggs
Scott Miller
http://biglynx.co.uk/
people/matt-briggs
http://biglynx.co.uk/
people/scott-miller
Ansgar Scherp – ansgar@informatik.uni-mannheim.de
B. Gazen,http://www.flickr.com/photos/bayat/, CC-BY Slide 12
13. Example: Big Lynx
Matt Briggs
Scott Miller
Big Lynx
Company
How to model relationships like knows?
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 13
14. Resource DescriptionFramework (RDF)
• Description of Ressources with RDF triple
Matt Briggs is a Person
Subject Predicate Object
@prefix rdf:<http://w3.org/1999/02/22-rdf-
syntax-ns#> .
@prefix foaf:<http://xmlns.com/foaf/0.1/> .
<http://biglynx.co.uk/people/matt-briggs>
rdf:type foaf:Person .
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 14
15. 1. Use URIs also for Relations
http://biglynx.co.uk/
people/matt-briggs
http://biglynx.co.uk/
people/scott-miller
Ansgar Scherp – ansgar@informatik.uni-mannheim.de
B. Gazen,http://www.flickr.com/photos/bayat/, CC-BY Slide 15
16. Example: Big Lynx
Dave Smith
London
„lives here―
Matt Briggs
„same
Scott Miller
Big Lynx
… person―
Company
DBpedia Matt Briggs
Matts private
Webseite
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 16
18. Example: Big Lynx
Dave Smith
London
„lives here―
foaf:based_near
Matt Briggs
„same
owl:sameAs
Person― Big Lynx
Company
DBpedia Matt Briggs
Matts private
Webseite
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 18
19. 3. Dereferencing of URIs
• Looking up of web documents
• How can we ―look up‖ things of the real world?
http://biglynx.co.uk/
people/matt-briggs
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 19
20. Two Approaches
1. Hash URIs
– URI contains a part separated by #, e.g.,
http://biglynx.co.uk/vocab/sme#Team
2. Negotiation via „303 See Other― request
http://biglynx.co.uk/people/matt-briggs
Response: „Look here:―
http://biglynx.co.uk/people/matt-briggs.rdf
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 20
21. Example: Big Lynx
Dave Smith
London
foaf:based_near
Description of
Matt Briggs
Matt?
owl:sameAs
Big Lynx
Company
DBpedia Matt Briggs
Matts private
Webseite
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 21
23. Formalization of Description
Given a RDF graph G (V , P, E ) with
V R B L and E ( R B) P V
∩∞
SimpleCBD(n) = I j with
j=0
I 0 = { (s, p, o) | (s, p, o) E s=n}
I j+1 = { (o, p‗, o‗) E| (s, p, o) Ij : o B
∩j
(o, p‗, o‗) Ik}
k=0
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 23
26. Exploration of Linked Data
Word
Net
Swoogle
Geo
Names
Ansgar Scherp – ansgar@informatik.uni-mannheim.de
< 31 Billion Triples Source: http://lod-cloud.net
Slide 26
27. Naive Approach
• Download all data
• Store in really big
database RDFS
• Programming of WordNet Rules
queries Swoogle Geo
• Design of
user interface GeoNames
Inflexible Monolithic
Not
Ansgar Scherp – ansgar@informatik.uni-mannheim.de
scaleable
Slide 27
31. Searching for Linked Data Sources
?
Persons that are
- Politicians and
- Actors
?
<Ansgar Scherp – ansgar@informatik.uni-mannheim.de
31 Milliarde Triples Quelle: http://lod-cloud.net
Slide 31
32. Idea: Index of Data Sources
SELECT ?x
FROM …
WHERE {
?x rdf:type ex:Actor .
?x rdf:type ex:Politician .
}
Index
?
Query
“Politician and
Actor”
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 32
33. The Naive Approach
1. Download the entire LOD cloud
2. Put it into a (really) large triple store
3. Process the data and extract schema
4. Provide lookup
- Big machinery
- Late in processing the data
- High effort to scale with LOD cloud
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 33
34. Idea
Schema-level index
Define families of graph patterns
Assign instances to graph patterns
Map graph patterns to context (source URI)
Construction
Stream-based for scalability
Little loss of accuracy
Note
Index defined over instances
But stores the context
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 34
36. SchemEX Approach
• Stream-based schema extraction
• While crawling the data
FIFO
LOD-Crawler Instance-
RDF-Dump Cache RDF
Triple Store RDBMS
NxParser
Nquad- Schema- Schema-
Parser
Stream Extractor Level
Index
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 36
37. Building the Index from a Stream
Stream of n-quads (coming from a LD crawler)
… Q16, Q15, Q14, Q13, Q12, Q11, Q10, Q9, Q8, Q7, Q6, Q5, Q4, Q3, Q2, Q1
FiFo
1
C3 4
6
C2 3
4
2
C2 2
1 3
C1 5
• Linear runtime complexity wrt # of input triples
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 37
38. Building the Schema and Index
RDF
C1 C2 C3 … Ck
classes
consistsOf
Type
TC1 TC2 … TCm clusters
hasEQ
Class p1 p2
EQC1 EQC2 … EQCn Equivalence
classes
hasDataSource
… Data
DS1 DS2 DS3 DS4 DS5 DSx sources
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 38
39. Layer 1: RDF Classes
All instances of a C1
particular type
DS 1 DS 2 DS 3
SELECT ?x
FROM …
WHERE {
?x rdfs:type foaf:Person .
foaf:Person
}
http://dig.csail.mit.edu/2008/...
foaf:
timbl: Person
card#i http://www.w3.org/People/Berners-Lee/card
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 39
40. Layer 2: Type Clusters
All instances belonging C1 C2
to exactly the same set
TC1
of types
SELECT ?x DS 1 DS 2 DS 3
FROM …
WHERE {
foaf:Person pim:Male
?x rdfs:type foaf:Person .
?x rdfs:type pim:Male . tc4711
}
pim:
Male
http://www.w3.org/People/Berners-Lee/card
foaf:
timbl:
Person
card#i
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 40
41. Layer 3: Equivalence Classes
Two instances are C1 C2 C3
equivalent iff:
They are in the same TC TC1 TC2
They have the same p
properties
EQC1
The property targets are
in the same TC DS 1 DS 2 DS 3
Similar to 1-Bisimulation
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 41
43. Computing SchemEX: TimBL Data Set
• Analysis of a smaller data set
• 11 M triples, TimBL‘s FOAF profile
• LDspider with ~ 2k triples / sec
• Different cache sizes: 100, 1k, 10k, 50k, 100k
• Compared SchemEX with reference schema
• Index queries on all Types, TCs, EQCs
• Good precision/recall ratio at 50k+
• Commodity hardware (4GB RAM, single CPU)
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 43
44. Quality of Stream-based Index
Construction
+ Runtime increases hardly with window size
+ Memory consumption scales with window size
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 44
45. Computing SchemEX: Full BTC 2011 Data
Cache size: 50 k
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 45
50. Recommended Readings
• Maciej Janik, Ansgar Scherp, Steffen Staab: The Semantic Web:
Collective Intelligence on the Web. Informatik Spektrum 34(5): 469-483
(2011) URL: http://dx.doi.org/10.1007/s00287-011-0535-x
• Simon Schenk, Carsten Saathoff, Steffen Staab, Ansgar Scherp:
SemaPlorer - Interactive semantic exploration of data and media based on
a federated cloud infrastructure. J. Web Sem. 7(4): 298-304 (2009)
URL: http://dx.doi.org/10.1016/j.websem.2009.09.006
• Mathias Konrath, Thomas Gottron, Steffen Staab, Ansgar Scherp:
SchemEX — Efficient construction of a data catalogue by stream-based
indexing of linked data, J. of Web Semantics: Science, Services and
Agents on the World Wide Web, Available online 23 June 2012
URL: http://www.sciencedirect.com/science/article/pii/S1570826812000716
• Tom Heath, Christian Bizer: Linked Data: Evolving the Web into a Global
Data Space, Morgan & Claypool Publishers, 2011
URL: http://dx.doi.org/10.2200/S00334ED1V01Y201102WBE001
Ansgar Scherp – ansgar@informatik.uni-mannheim.de Slide 50