SlideShare a Scribd company logo
1 of 19
SchemEX
Creating the Yellow Pages of the LOD Cloud

Mathias Konrath, Thomas Gottron, Ansgar Scherp
Scenario
• People who are politicians and actors




• Who else?
• Where do they live?
• Whom do they know?
SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp   2 of 12
Problem
• Execute those queries on the LOD cloud
• No single federated query interface provided




       “politicians
       and actors”

SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp   3 of 12
Principle Solution
• Suitable index structure for looking up sources




       “politicians
       and actors”

SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp   4 of 12
The Naive Approach
1.     Download the entire LOD cloud
2.     Put it into a (really) large triple store
3.     Process the data and extract schema
4.     Provide lookup

- Big machinery
- Late in processing the data
- High effort to scale with LOD cloud



SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp   5 of 12
Yes, we can …



SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp   6 of 12
The SchemEX Approach
• Stream-based schema extraction
• While crawling the data


                                          FIFO
LOD-Crawler                                                Instance-
 RDF-Dump                                                    Cache
                                                                        RDF
 Triple Store                                                          RDBMS
                              NxParser

    Nquad-                                                 Schema-
                                Parser                                 Schema
    Stream                                                 Extractor

SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp   7 of 12
Efficient Instance Cache
• Observe a quadruple stream from LD spider




• Ring queue, backed up by a hash map
• Organizes triples with same subject URI
• Dismiss oldest, when cache full (FIFO)
→ Runtime complexity O(1)
SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp   8 of 12
Building the Schema and Index
                                                                              RDF
       c1               c2               c3                …         ck
                                                                             classes
                                         consistsOf
                                                                              Type
        TC1                     TC2                        …         TCm     clusters
hasEQ
Class                 p1                            p2
       EQC1                   EQC2                         … EQCn          Equivalence
                                                                             classes
                                           hasDataSource

                                                           …                 Data
  DS1 DS2 DS3 DS4 DS5                                            DSx        sources
SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp   9 of 12
Computing SchemEX: TimBL Data Set
• Analysis of a smaller data set
• 11 M triples, TimBL’s FOAF profile
• LDspider with ~ 2k triples / sec


•   Different cache sizes: 100, 1k, 10k, 50k, 100k
•   Compared SchemEX with reference schema
•   Index queries on all Types, TCs, EQCs
•   Good precision/recall ratio at 50k+

SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp   10 of 12
Computing SchemEX: Full BTC 2011 Data




Cache size: 50 k
SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp   11 of 12
Conclusions: SchemEX
• Stream-based approach to schema extraction
• Scalable to arbitrary amount of Linked Data
• Applicable on commodity hardware
  (4GB RAM, standard single CPU)




• Lookup-index to find relevant data sources
• Support federated queries on the LOD cloud
SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp   12 of 12
BACKUP




SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp   13 of 12
SchemEX Computation: Window Sizes
                                      Runtime increases hardly with
                                          greater window sizes




 Crawled TimBL dataset                                     Memory consumption scales
  (11M triples)                                                with window size


SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp   14 of 12
SchemEX Quality: Precision




SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp   15 of 12
SchemEX Quality: Recall




SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp   16 of 12
Example Data Graph




SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp   17 of 12
Output Vocabulary: voiD




SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp   18 of 12
SchemEX Extraction: Progress Plot

                  Type-cluster
                  Equivalence classes
 Count




                                 ##processed instances
                                        processed 12           instances
SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp 19 of

More Related Content

What's hot

Sebastian Schelter – Distributed Machine Learing with the Samsara DSL
Sebastian Schelter – Distributed Machine Learing with the Samsara DSLSebastian Schelter – Distributed Machine Learing with the Samsara DSL
Sebastian Schelter – Distributed Machine Learing with the Samsara DSL
Flink Forward
 
Magellan-Spark as a Geospatial Analytics Engine by Ram Sriharsha
Magellan-Spark as a Geospatial Analytics Engine by Ram SriharshaMagellan-Spark as a Geospatial Analytics Engine by Ram Sriharsha
Magellan-Spark as a Geospatial Analytics Engine by Ram Sriharsha
Spark Summit
 

What's hot (20)

Keeping Linked Open Data Caches Up-to-date by Predicting the Life-time of RDF...
Keeping Linked Open Data Caches Up-to-date by Predicting the Life-time of RDF...Keeping Linked Open Data Caches Up-to-date by Predicting the Life-time of RDF...
Keeping Linked Open Data Caches Up-to-date by Predicting the Life-time of RDF...
 
Parikshit Ram – Senior Machine Learning Scientist, Skytree at MLconf ATL
Parikshit Ram – Senior Machine Learning Scientist, Skytree at MLconf ATLParikshit Ram – Senior Machine Learning Scientist, Skytree at MLconf ATL
Parikshit Ram – Senior Machine Learning Scientist, Skytree at MLconf ATL
 
Mining Big Data Streams with APACHE SAMOA
Mining Big Data Streams with APACHE SAMOAMining Big Data Streams with APACHE SAMOA
Mining Big Data Streams with APACHE SAMOA
 
Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...
Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...
Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...
 
5.1 mining data streams
5.1 mining data streams5.1 mining data streams
5.1 mining data streams
 
MOA for the IoT at ACML 2016
MOA for the IoT at ACML 2016 MOA for the IoT at ACML 2016
MOA for the IoT at ACML 2016
 
Sebastian Schelter – Distributed Machine Learing with the Samsara DSL
Sebastian Schelter – Distributed Machine Learing with the Samsara DSLSebastian Schelter – Distributed Machine Learing with the Samsara DSL
Sebastian Schelter – Distributed Machine Learing with the Samsara DSL
 
R + Storm Moneyball - Realtime Advanced Statistics - Hadoop Summit - San Jose
R + Storm Moneyball - Realtime Advanced Statistics - Hadoop Summit - San JoseR + Storm Moneyball - Realtime Advanced Statistics - Hadoop Summit - San Jose
R + Storm Moneyball - Realtime Advanced Statistics - Hadoop Summit - San Jose
 
Artificial intelligence and data stream mining
Artificial intelligence and data stream miningArtificial intelligence and data stream mining
Artificial intelligence and data stream mining
 
Mining Big Data in Real Time
Mining Big Data in Real TimeMining Big Data in Real Time
Mining Big Data in Real Time
 
Accelerating the Experimental Feedback Loop: Data Streams and the Advanced Ph...
Accelerating the Experimental Feedback Loop: Data Streams and the Advanced Ph...Accelerating the Experimental Feedback Loop: Data Streams and the Advanced Ph...
Accelerating the Experimental Feedback Loop: Data Streams and the Advanced Ph...
 
Distributed GLM with H2O - Atlanta Meetup
Distributed GLM with H2O - Atlanta MeetupDistributed GLM with H2O - Atlanta Meetup
Distributed GLM with H2O - Atlanta Meetup
 
Mining high speed data streams: Hoeffding and VFDT
Mining high speed data streams: Hoeffding and VFDTMining high speed data streams: Hoeffding and VFDT
Mining high speed data streams: Hoeffding and VFDT
 
Introduction to Data streaming - 05/12/2014
Introduction to Data streaming - 05/12/2014Introduction to Data streaming - 05/12/2014
Introduction to Data streaming - 05/12/2014
 
Magellan-Spark as a Geospatial Analytics Engine by Ram Sriharsha
Magellan-Spark as a Geospatial Analytics Engine by Ram SriharshaMagellan-Spark as a Geospatial Analytics Engine by Ram Sriharsha
Magellan-Spark as a Geospatial Analytics Engine by Ram Sriharsha
 
Astronomical Data Processing on the LSST Scale with Apache Spark
Astronomical Data Processing on the LSST Scale with Apache SparkAstronomical Data Processing on the LSST Scale with Apache Spark
Astronomical Data Processing on the LSST Scale with Apache Spark
 
Ernest: Efficient Performance Prediction for Advanced Analytics on Apache Spa...
Ernest: Efficient Performance Prediction for Advanced Analytics on Apache Spa...Ernest: Efficient Performance Prediction for Advanced Analytics on Apache Spa...
Ernest: Efficient Performance Prediction for Advanced Analytics on Apache Spa...
 
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
 
Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies
Working with HDF and netCDF Data in ArcGIS: Tools and Case StudiesWorking with HDF and netCDF Data in ArcGIS: Tools and Case Studies
Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies
 
Graph databases: Tinkerpop and Titan DB
Graph databases: Tinkerpop and Titan DBGraph databases: Tinkerpop and Titan DB
Graph databases: Tinkerpop and Titan DB
 

Viewers also liked

Finding Good URLs: Aligning Entities in Knowledge Bases with Public Web Docum...
Finding Good URLs: Aligning Entities in Knowledge Bases with Public Web Docum...Finding Good URLs: Aligning Entities in Knowledge Bases with Public Web Docum...
Finding Good URLs: Aligning Entities in Knowledge Bases with Public Web Docum...
Thomas Gottron
 
ESWC 2013: A Systematic Investigation of Explicit and Implicit Schema Informa...
ESWC 2013: A Systematic Investigation of Explicit and Implicit Schema Informa...ESWC 2013: A Systematic Investigation of Explicit and Implicit Schema Informa...
ESWC 2013: A Systematic Investigation of Explicit and Implicit Schema Informa...
Thomas Gottron
 
Leveraging the Web of Data: Managing, Analysing and Making Use of Linked Open...
Leveraging the Web of Data: Managing, Analysing and Making Use of Linked Open...Leveraging the Web of Data: Managing, Analysing and Making Use of Linked Open...
Leveraging the Web of Data: Managing, Analysing and Making Use of Linked Open...
Thomas Gottron
 
A Framework for Iterative Signing of Graph Data on the Web
A Framework for Iterative Signing of Graph Data on the WebA Framework for Iterative Signing of Graph Data on the Web
A Framework for Iterative Signing of Graph Data on the Web
Ansgar Scherp
 
Establishing a Strategy for Data Quality
Establishing a Strategy for Data QualityEstablishing a Strategy for Data Quality
Establishing a Strategy for Data Quality
Database Answers Ltd.
 

Viewers also liked (20)

 Challenges in Managing Online Business Communities
 Challenges in Managing Online Business Communities Challenges in Managing Online Business Communities
 Challenges in Managing Online Business Communities
 
From Changes to Dynamics: Dynamics Analysis of Linked Open Data Sources
From Changes to Dynamics: Dynamics Analysis of Linked Open Data Sources From Changes to Dynamics: Dynamics Analysis of Linked Open Data Sources
From Changes to Dynamics: Dynamics Analysis of Linked Open Data Sources
 
SchemEX -- Building an Index for Linked Open Data
SchemEX -- Building an Index for Linked Open DataSchemEX -- Building an Index for Linked Open Data
SchemEX -- Building an Index for Linked Open Data
 
A Model of Events for Integrating Event-based Information in Complex Socio-te...
A Model of Events for Integrating Event-based Information in Complex Socio-te...A Model of Events for Integrating Event-based Information in Complex Socio-te...
A Model of Events for Integrating Event-based Information in Complex Socio-te...
 
Smart photo selection: interpret gaze as personal interest
Smart photo selection: interpret gaze as personal interestSmart photo selection: interpret gaze as personal interest
Smart photo selection: interpret gaze as personal interest
 
Linked open data - how to juggle with more than a billion triples
Linked open data - how to juggle with more than a billion triplesLinked open data - how to juggle with more than a billion triples
Linked open data - how to juggle with more than a billion triples
 
Finding Good URLs: Aligning Entities in Knowledge Bases with Public Web Docum...
Finding Good URLs: Aligning Entities in Knowledge Bases with Public Web Docum...Finding Good URLs: Aligning Entities in Knowledge Bases with Public Web Docum...
Finding Good URLs: Aligning Entities in Knowledge Bases with Public Web Docum...
 
Making Use of the Linked Data Cloud: The Role of Index Structures
Making Use of the Linked Data Cloud: The Role of Index StructuresMaking Use of the Linked Data Cloud: The Role of Index Structures
Making Use of the Linked Data Cloud: The Role of Index Structures
 
Challenging Retrieval Scenarios: Social Media and Linked Open Data
Challenging Retrieval Scenarios: Social Media and Linked Open DataChallenging Retrieval Scenarios: Social Media and Linked Open Data
Challenging Retrieval Scenarios: Social Media and Linked Open Data
 
Perplexity of Index Models over Evolving Linked Data
Perplexity of Index Models over Evolving Linked Data Perplexity of Index Models over Evolving Linked Data
Perplexity of Index Models over Evolving Linked Data
 
Of Sampling and Smoothing: Approximating Distributions over Linked Open Data
Of Sampling and Smoothing: Approximating Distributions over Linked Open DataOf Sampling and Smoothing: Approximating Distributions over Linked Open Data
Of Sampling and Smoothing: Approximating Distributions over Linked Open Data
 
Can you see it? Annotating Image Regions based on Users' Gaze Information
Can you see it? Annotating Image Regions based on Users' Gaze InformationCan you see it? Annotating Image Regions based on Users' Gaze Information
Can you see it? Annotating Image Regions based on Users' Gaze Information
 
Focused Exploration of Geospatial Context on Linked Open Data
Focused Exploration of Geospatial Context on Linked Open DataFocused Exploration of Geospatial Context on Linked Open Data
Focused Exploration of Geospatial Context on Linked Open Data
 
ESWC 2013: A Systematic Investigation of Explicit and Implicit Schema Informa...
ESWC 2013: A Systematic Investigation of Explicit and Implicit Schema Informa...ESWC 2013: A Systematic Investigation of Explicit and Implicit Schema Informa...
ESWC 2013: A Systematic Investigation of Explicit and Implicit Schema Informa...
 
Linked Open Data (Entwurfsprinzipien und Muster für vernetzte Daten)
Linked Open Data (Entwurfsprinzipien und Muster für vernetzte Daten)Linked Open Data (Entwurfsprinzipien und Muster für vernetzte Daten)
Linked Open Data (Entwurfsprinzipien und Muster für vernetzte Daten)
 
Leveraging the Web of Data: Managing, Analysing and Making Use of Linked Open...
Leveraging the Web of Data: Managing, Analysing and Making Use of Linked Open...Leveraging the Web of Data: Managing, Analysing and Making Use of Linked Open...
Leveraging the Web of Data: Managing, Analysing and Making Use of Linked Open...
 
Events in Multimedia - Theory, Model, Application
Events in Multimedia - Theory, Model, ApplicationEvents in Multimedia - Theory, Model, Application
Events in Multimedia - Theory, Model, Application
 
Identifying Objects in Images from Analyzing the User‘s Gaze Movements for Pr...
Identifying Objects in Images from Analyzing the User‘s Gaze Movements for Pr...Identifying Objects in Images from Analyzing the User‘s Gaze Movements for Pr...
Identifying Objects in Images from Analyzing the User‘s Gaze Movements for Pr...
 
A Framework for Iterative Signing of Graph Data on the Web
A Framework for Iterative Signing of Graph Data on the WebA Framework for Iterative Signing of Graph Data on the Web
A Framework for Iterative Signing of Graph Data on the Web
 
Establishing a Strategy for Data Quality
Establishing a Strategy for Data QualityEstablishing a Strategy for Data Quality
Establishing a Strategy for Data Quality
 

Similar to SchemEX - Creating the Yellow Pages for the Linked Open Data Cloud

Similar to SchemEX - Creating the Yellow Pages for the Linked Open Data Cloud (20)

Polyraptor
PolyraptorPolyraptor
Polyraptor
 
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
 
A Fast and Efficient Time Series Storage Based on Apache Solr
A Fast and Efficient Time Series Storage Based on Apache SolrA Fast and Efficient Time Series Storage Based on Apache Solr
A Fast and Efficient Time Series Storage Based on Apache Solr
 
Chronix: A fast and efficient time series storage based on Apache Solr
Chronix: A fast and efficient time series storage based on Apache SolrChronix: A fast and efficient time series storage based on Apache Solr
Chronix: A fast and efficient time series storage based on Apache Solr
 
FEC & File Multicast
FEC & File MulticastFEC & File Multicast
FEC & File Multicast
 
On the need for a W3C community group on RDF Stream Processing
On the need for a W3C community group on RDF Stream ProcessingOn the need for a W3C community group on RDF Stream Processing
On the need for a W3C community group on RDF Stream Processing
 
OrdRing 2013 keynote - On the need for a W3C community group on RDF Stream Pr...
OrdRing 2013 keynote - On the need for a W3C community group on RDF Stream Pr...OrdRing 2013 keynote - On the need for a W3C community group on RDF Stream Pr...
OrdRing 2013 keynote - On the need for a W3C community group on RDF Stream Pr...
 
Time Series Processing with Apache Spark
Time Series Processing with Apache SparkTime Series Processing with Apache Spark
Time Series Processing with Apache Spark
 
Time Series Processing with Apache Spark
Time Series Processing with Apache SparkTime Series Processing with Apache Spark
Time Series Processing with Apache Spark
 
Новая архитектура шардинга MongoDB, Leif Walsh (Tokutek)
Новая архитектура шардинга MongoDB, Leif Walsh (Tokutek)Новая архитектура шардинга MongoDB, Leif Walsh (Tokutek)
Новая архитектура шардинга MongoDB, Leif Walsh (Tokutek)
 
A New MongoDB Sharding Architecture for Higher Availability and Better Resour...
A New MongoDB Sharding Architecture for Higher Availability and Better Resour...A New MongoDB Sharding Architecture for Higher Availability and Better Resour...
A New MongoDB Sharding Architecture for Higher Availability and Better Resour...
 
The new time series kid on the block
The new time series kid on the blockThe new time series kid on the block
The new time series kid on the block
 
Xml::parent - Yet another way to store XML files
Xml::parent - Yet another way to store XML files Xml::parent - Yet another way to store XML files
Xml::parent - Yet another way to store XML files
 
Chronix Time Series Database - The New Time Series Kid on the Block
Chronix Time Series Database - The New Time Series Kid on the BlockChronix Time Series Database - The New Time Series Kid on the Block
Chronix Time Series Database - The New Time Series Kid on the Block
 
Apache cassandra
Apache cassandraApache cassandra
Apache cassandra
 
Low Level CPU Performance Profiling Examples
Low Level CPU Performance Profiling ExamplesLow Level CPU Performance Profiling Examples
Low Level CPU Performance Profiling Examples
 
Spark Summit EU talk by Qifan Pu
Spark Summit EU talk by Qifan PuSpark Summit EU talk by Qifan Pu
Spark Summit EU talk by Qifan Pu
 
Xml processing-by-asfak
Xml processing-by-asfakXml processing-by-asfak
Xml processing-by-asfak
 
7_mem_cache.ppt
7_mem_cache.ppt7_mem_cache.ppt
7_mem_cache.ppt
 
Porting a Streaming Pipeline from Scala to Rust
Porting a Streaming Pipeline from Scala to RustPorting a Streaming Pipeline from Scala to Rust
Porting a Streaming Pipeline from Scala to Rust
 

More from Ansgar Scherp

Text Localization in Scientific Figures using Fully Convolutional Neural Netw...
Text Localization in Scientific Figures using Fully Convolutional Neural Netw...Text Localization in Scientific Figures using Fully Convolutional Neural Netw...
Text Localization in Scientific Figures using Fully Convolutional Neural Netw...
Ansgar Scherp
 

More from Ansgar Scherp (7)

Analysis of GraphSum's Attention Weights to Improve the Explainability of Mul...
Analysis of GraphSum's Attention Weights to Improve the Explainability of Mul...Analysis of GraphSum's Attention Weights to Improve the Explainability of Mul...
Analysis of GraphSum's Attention Weights to Improve the Explainability of Mul...
 
STEREO: A Pipeline for Extracting Experiment Statistics, Conditions, and Topi...
STEREO: A Pipeline for Extracting Experiment Statistics, Conditions, and Topi...STEREO: A Pipeline for Extracting Experiment Statistics, Conditions, and Topi...
STEREO: A Pipeline for Extracting Experiment Statistics, Conditions, and Topi...
 
Text Localization in Scientific Figures using Fully Convolutional Neural Netw...
Text Localization in Scientific Figures using Fully Convolutional Neural Netw...Text Localization in Scientific Figures using Fully Convolutional Neural Netw...
Text Localization in Scientific Figures using Fully Convolutional Neural Netw...
 
A Comparison of Approaches for Automated Text Extraction from Scholarly Figures
A Comparison of Approaches for Automated Text Extraction from Scholarly FiguresA Comparison of Approaches for Automated Text Extraction from Scholarly Figures
A Comparison of Approaches for Automated Text Extraction from Scholarly Figures
 
About Multimedia Presentation Generation and Multimedia Metadata: From Synthe...
About Multimedia Presentation Generation and Multimedia Metadata: From Synthe...About Multimedia Presentation Generation and Multimedia Metadata: From Synthe...
About Multimedia Presentation Generation and Multimedia Metadata: From Synthe...
 
SchemEX -- Building an Index for Linked Open Data
SchemEX -- Building an Index for Linked Open DataSchemEX -- Building an Index for Linked Open Data
SchemEX -- Building an Index for Linked Open Data
 
strukt - A Pattern System for Integrating Individual and Organizational Knowl...
strukt - A Pattern System for Integrating Individual and Organizational Knowl...strukt - A Pattern System for Integrating Individual and Organizational Knowl...
strukt - A Pattern System for Integrating Individual and Organizational Knowl...
 

Recently uploaded

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 

Recently uploaded (20)

Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesICT role in 21st century education and its challenges
ICT role in 21st century education and its challenges
 
Ransomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdfRansomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdf
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot ModelNavi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 

SchemEX - Creating the Yellow Pages for the Linked Open Data Cloud

  • 1. SchemEX Creating the Yellow Pages of the LOD Cloud Mathias Konrath, Thomas Gottron, Ansgar Scherp
  • 2. Scenario • People who are politicians and actors • Who else? • Where do they live? • Whom do they know? SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp 2 of 12
  • 3. Problem • Execute those queries on the LOD cloud • No single federated query interface provided “politicians and actors” SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp 3 of 12
  • 4. Principle Solution • Suitable index structure for looking up sources “politicians and actors” SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp 4 of 12
  • 5. The Naive Approach 1. Download the entire LOD cloud 2. Put it into a (really) large triple store 3. Process the data and extract schema 4. Provide lookup - Big machinery - Late in processing the data - High effort to scale with LOD cloud SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp 5 of 12
  • 6. Yes, we can … SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp 6 of 12
  • 7. The SchemEX Approach • Stream-based schema extraction • While crawling the data FIFO LOD-Crawler Instance- RDF-Dump Cache RDF Triple Store RDBMS NxParser Nquad- Schema- Parser Schema Stream Extractor SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp 7 of 12
  • 8. Efficient Instance Cache • Observe a quadruple stream from LD spider • Ring queue, backed up by a hash map • Organizes triples with same subject URI • Dismiss oldest, when cache full (FIFO) → Runtime complexity O(1) SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp 8 of 12
  • 9. Building the Schema and Index RDF c1 c2 c3 … ck classes consistsOf Type TC1 TC2 … TCm clusters hasEQ Class p1 p2 EQC1 EQC2 … EQCn Equivalence classes hasDataSource … Data DS1 DS2 DS3 DS4 DS5 DSx sources SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp 9 of 12
  • 10. Computing SchemEX: TimBL Data Set • Analysis of a smaller data set • 11 M triples, TimBL’s FOAF profile • LDspider with ~ 2k triples / sec • Different cache sizes: 100, 1k, 10k, 50k, 100k • Compared SchemEX with reference schema • Index queries on all Types, TCs, EQCs • Good precision/recall ratio at 50k+ SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp 10 of 12
  • 11. Computing SchemEX: Full BTC 2011 Data Cache size: 50 k SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp 11 of 12
  • 12. Conclusions: SchemEX • Stream-based approach to schema extraction • Scalable to arbitrary amount of Linked Data • Applicable on commodity hardware (4GB RAM, standard single CPU) • Lookup-index to find relevant data sources • Support federated queries on the LOD cloud SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp 12 of 12
  • 13. BACKUP SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp 13 of 12
  • 14. SchemEX Computation: Window Sizes Runtime increases hardly with greater window sizes Crawled TimBL dataset Memory consumption scales (11M triples) with window size SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp 14 of 12
  • 15. SchemEX Quality: Precision SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp 15 of 12
  • 16. SchemEX Quality: Recall SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp 16 of 12
  • 17. Example Data Graph SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp 17 of 12
  • 18. Output Vocabulary: voiD SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp 18 of 12
  • 19. SchemEX Extraction: Progress Plot Type-cluster Equivalence classes Count ##processed instances processed 12 instances SchemEX – Mathias Konrath, Thomas Gottron, Ansgar Scherp 19 of