SlideShare a Scribd company logo
1 of 46
Detailed
Provenance Capture
of Data Processing
Ben De Meester, Anastasia Dimou,
Ruben Verborgh, and Erik Mannens
Ghent University – imec – IDLab, Belgium
*
Outline
Linked Data Generation
Problem
Solution
Results
Outline
Linked Data Generation
Problem
Solution
Results
Linked Data comes from
data
Unstructured data
Semi-structured data
Structured data
…
Linked Data comes from
processed data
Unstructured data Parse
Semi-structured data Extract
Structured data Add schema annotations
… …
Going from data to linked data
Data
Schema transformations
Data transformations
Linked Data
Linked Data generation =
schema + data transformations
dbr:
Barney_G
umble
Linked Data generation =
schema + data transformations
dbr:
Barney_G
umble
dbo:birthDate
dbp:voiceactor
dbp:gender
dbp:name
…
Linked Data generation =
schema + data transformations
dbr:
Hawaii
dbr:
Barney_G
umble
dbo:birthDate
dbp:voiceactor
dbp:gender
dbp:name
“1954-4-20"
dbr:
Dan_Caste
llaneta
“Male”
“Barney Gumble”@en
… …
Problem: there’s always a drunk Barney
Data
Schema processing
Data processing
Linked Data
Data
Data
Processing
Knowing where the data
comes from is as important as
the data itself
Oh Yeah?
Outline
Linked Data Generation
Problem
Solution
Results
dbr:
Hawaii
dbr:
Barney_G
umble
dbo:birthDate
dbp:voiceactor
dbp:gender
dbp:name
“1954-4-20"
dbr:
Dan_Caste
llaneta
“Male”
“Barney Gumble”@en
… …
Linked Data re-generation?
Provenance of those transformations
How it’s done for data processing
Provenance log:
Person A used Software B, on System C
Problem: how to reproduce?
Provenance log:
Person A used Software B, on System C
Software B offline?
System C not booting?
What do we need?
Fine-grained provenance
for schema transformations
for data transformations
Independent of the implementation
How can we tell where the
data comes from, without
depending on the system?
Outline
Linked Data Generation
Problem: Data Processing Provenance
Solution
Results
What do we want?
Term-level,
implementation-independent provenance
for schema transformations
for data transformations
What do we want?
Term-level,
implementation-independent provenance
for schema transformations
for data transformations
Generated automatically
What do we want?
Term-level,
implementation-independent provenance
for schema transformations
for data transformations
Generated automatically
Declarative generation process
Steps
Align schema and data transformations
in a declarative document
Generate provenance based on
declarative schema transformations
Generate provenance based on
declarative data transformations
Declarative generation process?
Align schema and data transformations in a declarative
document
Declarative generation process? Solved!
Align schema and data transformations in a declarative
document
RML + FnO
Declarative generation process? Solved!
Align schema and data transformations in a declarative
document
RML + FnO for DBpedia EF
Declarative data transformations for Linked Data generation: the case of DBpedia
De Meester, B., Maroy, W., Dimou, A., Verborgh, R., and Mannens, E.
Sustainable Linked Data Generation: The Case of DBpedia
Maroy, W., Dimou, A., Kontokostas, D., De Meester, B., Verborgh, R., Lehmann, J., Mannens, E. and Hellmann, S.
Schema transformations provenance?
Generate provenance based on declarative mapping
document
Schema transformations provenance?
Solved!
Generate provenance based on declarative mapping
document
RML + PROV
Automated metadata generation for Linked Data generation and publishing workflows
Dimou, A., De Nies, T., Verborgh, R., Mannens, E., and Van de Walle, R.
Data transformations provenance?
Generate provenance based on declarative data
transformations
Data transformations provenance?
Outline
Linked Data Generation
Problem: Data Processing Provenance
Solution
Declarative generation
FnO and PROV
Results
FnO: Function
expects
output
inputString
predicate
outputString
predicate
DBpedia_
date_parser
fno:Function
FnO: Execution
DBpedia_
date_parser
Function
“April 20th 1954”
parseExecution
fno:Execution
“1954-04-20”
outputString
executesinputString
FnO: General Execution
Function
Data Transformation Output
Input
Aligning FnO and PROV
Output
prov:Entity
Tool
prov:Agent
wasGeneratedBy
Data Transformation
prov:Activity
Function
prov:Entity
used
Input
prov:Entity
used wasAssociatedWith
wasAttributedTo
Outline
Linked Data Generation
Problem: Data Processing Provenance
Solution
Declarative generation
FnO and PROV
Results
Uncool thing:
It’s big
When including provenance generation,
for every processed term,
you add 10 triples
Cool thing #1:
System details complementary
Output
prov:Entity
Tool
prov:Agent
wasGeneratedBy
Data Transformation
prov:Activity
Function
prov:Entity
used
Input
prov:Entity
used wasAssociatedWith
wasAttributedTo
Cool thing #2:
Aligning with RML complementary
wasInformedBy
Output
prov:Entity
Tool
prov:Agent
wasGeneratedBy
Data Transformation
prov:Activity
Function
prov:Entity
used
Input
prov:Entity
Schema
Transformation
prov:Activity
used wasAssociatedWith
wasAttributedTo
Cool thing #3:
It actually works
RMLMapper
https://github.com/RMLio/RML-Mapper
FunctionProcessor
https://github.com/FnOio/function-processor-java
DBpedia Extraction Sample
https://fno.io/prov/dbpedia/
How can we find a drunk Barney?
Query for long-lasting processes
Query all outputs of a certain function/tool
Query all input-output pairs
What to do with a drunk Barney?
Performance evaluation
Qualitative comparison
Iterative improvement
(only changing what is needed!)
Outline
Linked Data Generation
Problem: Data Processing Provenance
Solution
Declarative generation
FnO and PROV
Results
Detailed
Provenance Capture
of Data Processing
Ben De Meester, Anastasia Dimou,
Ruben Verborgh, and Erik Mannens
Ghent University – imec – IDLab, Belgium

More Related Content

More from Ben De Meester

OrdRing2015 - Event-Driven Rule-based Reasoning using EYE
OrdRing2015 - Event-Driven Rule-based Reasoning using EYEOrdRing2015 - Event-Driven Rule-based Reasoning using EYE
OrdRing2015 - Event-Driven Rule-based Reasoning using EYEBen De Meester
 
LINKed2015 - SERIF - A Semantic ExeRcise Interchange Format
LINKed2015 - SERIF - A Semantic ExeRcise Interchange FormatLINKed2015 - SERIF - A Semantic ExeRcise Interchange Format
LINKed2015 - SERIF - A Semantic ExeRcise Interchange FormatBen De Meester
 
NLP&DBpedia2015 - Exposing Digital Content as Linked Data, and Linking them ...
NLP&DBpedia2015 - Exposing Digital Content as Linked Data, and Linking them ...NLP&DBpedia2015 - Exposing Digital Content as Linked Data, and Linking them ...
NLP&DBpedia2015 - Exposing Digital Content as Linked Data, and Linking them ...Ben De Meester
 
ISWC2015 P&D - StoryBlink
ISWC2015 P&D - StoryBlinkISWC2015 P&D - StoryBlink
ISWC2015 P&D - StoryBlinkBen De Meester
 
LocWeb2015 - Reconnecting Digital Publications to the Web Using their Spatial...
LocWeb2015 - Reconnecting Digital Publications to the Web Using their Spatial...LocWeb2015 - Reconnecting Digital Publications to the Web Using their Spatial...
LocWeb2015 - Reconnecting Digital Publications to the Web Using their Spatial...Ben De Meester
 
Creating discoverable learning content using a user-friendly authoring enviro...
Creating discoverable learning content using a user-friendly authoring enviro...Creating discoverable learning content using a user-friendly authoring enviro...
Creating discoverable learning content using a user-friendly authoring enviro...Ben De Meester
 

More from Ben De Meester (6)

OrdRing2015 - Event-Driven Rule-based Reasoning using EYE
OrdRing2015 - Event-Driven Rule-based Reasoning using EYEOrdRing2015 - Event-Driven Rule-based Reasoning using EYE
OrdRing2015 - Event-Driven Rule-based Reasoning using EYE
 
LINKed2015 - SERIF - A Semantic ExeRcise Interchange Format
LINKed2015 - SERIF - A Semantic ExeRcise Interchange FormatLINKed2015 - SERIF - A Semantic ExeRcise Interchange Format
LINKed2015 - SERIF - A Semantic ExeRcise Interchange Format
 
NLP&DBpedia2015 - Exposing Digital Content as Linked Data, and Linking them ...
NLP&DBpedia2015 - Exposing Digital Content as Linked Data, and Linking them ...NLP&DBpedia2015 - Exposing Digital Content as Linked Data, and Linking them ...
NLP&DBpedia2015 - Exposing Digital Content as Linked Data, and Linking them ...
 
ISWC2015 P&D - StoryBlink
ISWC2015 P&D - StoryBlinkISWC2015 P&D - StoryBlink
ISWC2015 P&D - StoryBlink
 
LocWeb2015 - Reconnecting Digital Publications to the Web Using their Spatial...
LocWeb2015 - Reconnecting Digital Publications to the Web Using their Spatial...LocWeb2015 - Reconnecting Digital Publications to the Web Using their Spatial...
LocWeb2015 - Reconnecting Digital Publications to the Web Using their Spatial...
 
Creating discoverable learning content using a user-friendly authoring enviro...
Creating discoverable learning content using a user-friendly authoring enviro...Creating discoverable learning content using a user-friendly authoring enviro...
Creating discoverable learning content using a user-friendly authoring enviro...
 

Recently uploaded

办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree
办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree
办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degreeyuu sss
 
modul pembelajaran robotic Workshop _ by Slidesgo.pptx
modul pembelajaran robotic Workshop _ by Slidesgo.pptxmodul pembelajaran robotic Workshop _ by Slidesgo.pptx
modul pembelajaran robotic Workshop _ by Slidesgo.pptxaleedritatuxx
 
Learn How Data Science Changes Our World
Learn How Data Science Changes Our WorldLearn How Data Science Changes Our World
Learn How Data Science Changes Our WorldEduminds Learning
 
Predictive Analysis for Loan Default Presentation : Data Analysis Project PPT
Predictive Analysis for Loan Default  Presentation : Data Analysis Project PPTPredictive Analysis for Loan Default  Presentation : Data Analysis Project PPT
Predictive Analysis for Loan Default Presentation : Data Analysis Project PPTBoston Institute of Analytics
 
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBoston Institute of Analytics
 
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...Boston Institute of Analytics
 
RABBIT: A CLI tool for identifying bots based on their GitHub events.
RABBIT: A CLI tool for identifying bots based on their GitHub events.RABBIT: A CLI tool for identifying bots based on their GitHub events.
RABBIT: A CLI tool for identifying bots based on their GitHub events.natarajan8993
 
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming PipelinesConf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming PipelinesTimothy Spann
 
Principles and Practices of Data Visualization
Principles and Practices of Data VisualizationPrinciples and Practices of Data Visualization
Principles and Practices of Data VisualizationKianJazayeri1
 
Easter Eggs From Star Wars and in cars 1 and 2
Easter Eggs From Star Wars and in cars 1 and 2Easter Eggs From Star Wars and in cars 1 and 2
Easter Eggs From Star Wars and in cars 1 and 217djon017
 
Decoding Patterns: Customer Churn Prediction Data Analysis Project
Decoding Patterns: Customer Churn Prediction Data Analysis ProjectDecoding Patterns: Customer Churn Prediction Data Analysis Project
Decoding Patterns: Customer Churn Prediction Data Analysis ProjectBoston Institute of Analytics
 
Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Cathrine Wilhelmsen
 
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Boston Institute of Analytics
 
FAIR, FAIRsharing, FAIR Cookbook and ELIXIR - Sansone SA - Boston 2024
FAIR, FAIRsharing, FAIR Cookbook and ELIXIR - Sansone SA - Boston 2024FAIR, FAIRsharing, FAIR Cookbook and ELIXIR - Sansone SA - Boston 2024
FAIR, FAIRsharing, FAIR Cookbook and ELIXIR - Sansone SA - Boston 2024Susanna-Assunta Sansone
 
Top 5 Best Data Analytics Courses In Queens
Top 5 Best Data Analytics Courses In QueensTop 5 Best Data Analytics Courses In Queens
Top 5 Best Data Analytics Courses In Queensdataanalyticsqueen03
 
The Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptx
The Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptxThe Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptx
The Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptxTasha Penwell
 
Digital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing worksDigital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing worksdeepakthakur548787
 
Student profile product demonstration on grades, ability, well-being and mind...
Student profile product demonstration on grades, ability, well-being and mind...Student profile product demonstration on grades, ability, well-being and mind...
Student profile product demonstration on grades, ability, well-being and mind...Seán Kennedy
 
Real-Time AI Streaming - AI Max Princeton
Real-Time AI  Streaming - AI Max PrincetonReal-Time AI  Streaming - AI Max Princeton
Real-Time AI Streaming - AI Max PrincetonTimothy Spann
 
Predicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdfPredicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdfBoston Institute of Analytics
 

Recently uploaded (20)

办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree
办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree
办美国阿肯色大学小石城分校毕业证成绩单pdf电子版制作修改#真实留信入库#永久存档#真实可查#diploma#degree
 
modul pembelajaran robotic Workshop _ by Slidesgo.pptx
modul pembelajaran robotic Workshop _ by Slidesgo.pptxmodul pembelajaran robotic Workshop _ by Slidesgo.pptx
modul pembelajaran robotic Workshop _ by Slidesgo.pptx
 
Learn How Data Science Changes Our World
Learn How Data Science Changes Our WorldLearn How Data Science Changes Our World
Learn How Data Science Changes Our World
 
Predictive Analysis for Loan Default Presentation : Data Analysis Project PPT
Predictive Analysis for Loan Default  Presentation : Data Analysis Project PPTPredictive Analysis for Loan Default  Presentation : Data Analysis Project PPT
Predictive Analysis for Loan Default Presentation : Data Analysis Project PPT
 
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBank Loan Approval Analysis: A Comprehensive Data Analysis Project
Bank Loan Approval Analysis: A Comprehensive Data Analysis Project
 
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
NLP Data Science Project Presentation:Predicting Heart Disease with NLP Data ...
 
RABBIT: A CLI tool for identifying bots based on their GitHub events.
RABBIT: A CLI tool for identifying bots based on their GitHub events.RABBIT: A CLI tool for identifying bots based on their GitHub events.
RABBIT: A CLI tool for identifying bots based on their GitHub events.
 
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming PipelinesConf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
 
Principles and Practices of Data Visualization
Principles and Practices of Data VisualizationPrinciples and Practices of Data Visualization
Principles and Practices of Data Visualization
 
Easter Eggs From Star Wars and in cars 1 and 2
Easter Eggs From Star Wars and in cars 1 and 2Easter Eggs From Star Wars and in cars 1 and 2
Easter Eggs From Star Wars and in cars 1 and 2
 
Decoding Patterns: Customer Churn Prediction Data Analysis Project
Decoding Patterns: Customer Churn Prediction Data Analysis ProjectDecoding Patterns: Customer Churn Prediction Data Analysis Project
Decoding Patterns: Customer Churn Prediction Data Analysis Project
 
Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)
 
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
 
FAIR, FAIRsharing, FAIR Cookbook and ELIXIR - Sansone SA - Boston 2024
FAIR, FAIRsharing, FAIR Cookbook and ELIXIR - Sansone SA - Boston 2024FAIR, FAIRsharing, FAIR Cookbook and ELIXIR - Sansone SA - Boston 2024
FAIR, FAIRsharing, FAIR Cookbook and ELIXIR - Sansone SA - Boston 2024
 
Top 5 Best Data Analytics Courses In Queens
Top 5 Best Data Analytics Courses In QueensTop 5 Best Data Analytics Courses In Queens
Top 5 Best Data Analytics Courses In Queens
 
The Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptx
The Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptxThe Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptx
The Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptx
 
Digital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing worksDigital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing works
 
Student profile product demonstration on grades, ability, well-being and mind...
Student profile product demonstration on grades, ability, well-being and mind...Student profile product demonstration on grades, ability, well-being and mind...
Student profile product demonstration on grades, ability, well-being and mind...
 
Real-Time AI Streaming - AI Max Princeton
Real-Time AI  Streaming - AI Max PrincetonReal-Time AI  Streaming - AI Max Princeton
Real-Time AI Streaming - AI Max Princeton
 
Predicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdfPredicting Salary Using Data Science: A Comprehensive Analysis.pdf
Predicting Salary Using Data Science: A Comprehensive Analysis.pdf
 

SemSci2017 - Detailed Provenance Capture of Data Processing

Editor's Notes

  1. Not just a person, could be buggy software as well
  2. Explain term-level (example)
  3. Because it’s declarative, it _can_ be generated automatically Declarative explains complete generation workflow without implementaiton
  4. Ideal because declarative and in RDF
  5. In summary, we propose a fully declarative generation process and applied this by aligning FnO to PROV. There’s a lot of cool things about this, but there’s one uncool thing…
  6. Any schema tf
  7. Provenance that provides more insight in the generation of a dataset, thus helps evaluation, comparison, and improvement of dataset generation