Using Web Data Provenance for Quality Assessment
Upcoming SlideShare
Loading in...5
×
 

Using Web Data Provenance for Quality Assessment

on

  • 2,838 views

With these slides I presented our paper at the provenance workshop (SWPM) at the International Semantic Web Conference (ISWC), Oct.2009

With these slides I presented our paper at the provenance workshop (SWPM) at the International Semantic Web Conference (ISWC), Oct.2009

Statistics

Views

Total Views
2,838
Views on SlideShare
2,835
Embed Views
3

Actions

Likes
1
Downloads
28
Comments
0

1 Embed 3

http://www.slideshare.net 3

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Using Web Data Provenance for Quality Assessment Using Web Data Provenance for Quality Assessment Presentation Transcript

  • Using Web Data Provenance for Quality Assessment Olaf Hartig* Jun Zhao˚ *Humboldt-Universität zu Berlin ˚University of Oxford
  • Information Quality (IQ) ● Common definition: fitness for use of information ● Multidimensional concept Category* Criteria / Dimensions Intrinsic Accuracy, Believability, Objectivity, ... Contextual Completeness, Relevance, Timeliness, ... Representational Conciseness, Understandability, ... Accessibility Availability, Security, ... *Classification by Wang and Strong, 1996 ● IQ criteria not independent of each other ● Relevancy of criteria determined by task and preferences Olaf Hartig - Using Web Data Provenance for Quality Assessment 2
  • IQ Assessment ● Assigning numerical values (IQ scores) to IQ criteria ● It is difficult! ● Precision vs. Practicality Manual methods Semi-automatic methods ● Questionnaires ● Rating-based ● Reputation-based Olaf Hartig - Using Web Data Provenance for Quality Assessment 3
  • Automated IQ Assessment ● Literature only outlines ideas for automatic methods ● Content analysis ● Comparison (e.g. outlier detection) ● Application of information retrieval methods ● Analysis of results from data cleansing ● Sampling techniques ● Context analysis ● Analysis of metadata ● Utilization of domain knowledge Olaf Hartig - Using Web Data Provenance for Quality Assessment 4
  • Our Goal: Methods to automatically assess IQ criteria of Web data Primary means: Provenance of assessed data Olaf Hartig - Using Web Data Provenance for Quality Assessment 5
  • Outline 1. Web Data Provenance 2. General Assessment Approach 3. Development of Assessment Methods Olaf Hartig - Using Web Data Provenance for Quality Assessment 6
  • Existing Provenance Research ● Main research areas: (scientific) workflows, DBMSs ● General focus: data creation Olaf Hartig - Using Web Data Provenance for Quality Assessment 7
  • Provenance of Web Data Olaf Hartig - Using Web Data Provenance for Quality Assessment 8
  • Provenance of Web Data Web data provenance comprises two dimensions: Data Creation • Data Access Olaf Hartig - Using Web Data Provenance for Quality Assessment 9
  • Model of Web Data Provenance ● Provenance graph describes provenance of a data item ● Nodes: provenance elements – pieces of provenance info ● Edges: relate provenance elements to each other ● Subgraphs for related data items possible Olaf Hartig - Using Web Data Provenance for Quality Assessment 10
  • Model of Web Data Provenance ● Provenance model defines: Actors ● Types of provenance elements Executions ● Relationships Artifacts Olaf Hartig - Using Web Data Provenance for Quality Assessment 11
  • Data Access Dimension Data Item Data Accessor (Non-Human) contains performs retrieved by Document Execution Time Data Access accessed Data Providing Service (Non-Human) controls uses Service Provider Data Publisher (Human) Relation to the provided Information Resource Olaf Hartig - Using Web Data Provenance for Quality Assessment 12
  • Data Access Dimension cont. (Verified) Artifact Integrity Verification Verification Result {incomplete} Signer Signature Verification Relation to the signed Data Signature Method Olaf Hartig - Using Web Data Provenance for Quality Assessment 13
  • Data Creation Dimension Provenance Information Source Data Execution Time Provenance Information Creation Guidelines Data Creator Data Creation (Human or Non-human) {complete,disjoint} Data Creating Device (e.g. Sensor) Data Item Data Creating Service (e.g. Software Agent) part of responsible for responsible for Provenance Data Creating Entity Information (e.g. Person, Group, Orga.) (Encompassing) Data Item Relation to the created Data Olaf Hartig - Using Web Data Provenance for Quality Assessment 14
  • Outline 1. Web Data Provenance 2. General Assessment Approach 3. Development of Assessment Methods Olaf Hartig - Using Web Data Provenance for Quality Assessment 15
  • A General Approach ● Blueprint for actual assessment methods that ● Address specific scenario ● Focus on specific IQ criterion ● Provenance elements have an influence on IQ ● Impact values represent these influences ● Assessment is affected by knowing about the influences ● Calculation of the IQ score with an assessment function that combines all impact values Olaf Hartig - Using Web Data Provenance for Quality Assessment 16
  • General Assessment Procedure Step 1 – Generate a provenance graph for the data item Step 2 – Annotate the provenance graph with impact values Step 3 – Execute the assessment function Olaf Hartig - Using Web Data Provenance for Quality Assessment 17
  • Outline 1. Web Data Provenance 2. General Assessment Approach 3. Development of Assessment Methods Olaf Hartig - Using Web Data Provenance for Quality Assessment 18
  • Designing Assessment Methods ● Developing the general approach into an actual method ● Fundamental design question: For which IQ criterion do we want to apply the method? Olaf Hartig - Using Web Data Provenance for Quality Assessment 19
  • Designing Assessment Methods ● Developing the general approach into an actual method ● Fundamental design question: For which IQ criterion do we want to apply the method? ● Timeliness: degree to which the data item is up-to-date with respect to the task at hand ● Representation* as an absolute measure in [0,1] ● 1 – meeting the most strict timeliness standards ● 0 – unacceptable *Following Ballou et al., 1998 Olaf Hartig - Using Web Data Provenance for Quality Assessment 20
  • 1 Generate the Provenance Graph What types of provenance elements are necessary? What level of detail (i.e. granularity) is necessary? Where and how do we get provenance information? ● Two complementary options: ● Recording ● Analyzing metadata Olaf Hartig - Using Web Data Provenance for Quality Assessment 21
  • 1 Generate the Provenance Graph Example: ● Sensors (e.g. sensor1) hourly take measurement (e.g. msr) ● All msr stored in a Web-accessible storage device (store) ● Our system (sys) accesses them for further processing ● sys assesses the timeliness of all msr Olaf Hartig - Using Web Data Provenance for Quality Assessment 22
  • 1 Generate the Provenance Graph Example: ● Sensors (e.g. sensor1) hourly take measurement (e.g. msr) ● All msr stored in a Web-accessible storage device (store) ● Our system (sys) accesses them for further processing ● sys assesses the timeliness of all msr msr created by performed by sensor1 type: Data Item cExc type: Data Creator type: Data Creation contained by Execution Time: 10:00 doc retrieved by store type: Document type: Data Providing Service aExc accessed type: Data Access sys performed by type: Data Accessor Execution Time: 10:13 Olaf Hartig - Using Web Data Provenance for Quality Assessment 23
  • 2 Annotation with Impact Values How might each provenance element influence the IQ criterion? ● Systematically analyze each type of provenance elements What kind of impact values are necessary? How do we represent the influences by impact values? ● Impact values not necessarily numerical ● Depends on the assessment function in step 3 How do we determine impact values? Olaf Hartig - Using Web Data Provenance for Quality Assessment 24
  • Determining Impact Values ● From the provenance information ● From user input ● Configuration options ● Rating-based, Reputation-based ● By content analysis ● Comparison (e.g. outlier detection) ● Adoption of information retrieval methods ● Adoption of data cleansing techniques ● By context analysis ● Further metadata ● Domain knowledge Olaf Hartig - Using Web Data Provenance for Quality Assessment 25
  • 2 Annotation with Impact Values How might each provenance element influence the IQ criterion? Data Creation Dimension: Prov. Element Type Impact Values Data Creation ● creation time ● weights Creation Guidelines - (Source) Data Item ● expiry time Data Creator - Olaf Hartig - Using Web Data Provenance for Quality Assessment 26
  • 2 Annotation with Impact Values msr created by performed by sensor1 type: Data Item cExc type: Data Creator type: Data Creation contained by Execution Time: 10:00 doc retrieved by store type: Document type: Data Providing Service aExc accessed type: Data Access sys performed by type: Data Accessor Execution Time: 10:13 Prov. Element Type Impact Values Data Creation ● creation time ● weights Creation Guidelines - (Source) Data Item ● expiry time Data Creator - Olaf Hartig - Using Web Data Provenance for Quality Assessment 27
  • 2 Annotation with Impact Values msr created by performed by sensor1 type: Data Item cExc type: Data Creator type: Data Creation creation time contained by 10:00 Execution Time: 10:00 doc retrieved by store type: Document type: Data Providing Service aExc accessed type: Data Access sys performed by type: Data Accessor Execution Time: 10:13 Prov. Element Type Impact Values Data Creation ● creation time ● weights Creation Guidelines - (Source) Data Item ● expiry time Data Creator - Olaf Hartig - Using Web Data Provenance for Quality Assessment 28
  • 2 Annotation with Impact Values msr created by performed by sensor1 type: Data Item cExc type: Data Creator expiry time type: Data Creation 11:00 creation time contained by 10:00 Execution Time: 10:00 doc retrieved by store type: Document type: Data Providing Service aExc accessed type: Data Access sys performed by type: Data Accessor Execution Time: 10:13 Prov. Element Type Impact Values Data Creation ● creation time ● weights Creation Guidelines - (Source) Data Item ● expiry time Data Creator - Olaf Hartig - Using Web Data Provenance for Quality Assessment 29
  • 3 Assessment Function How do we represent the IQ criterion by an IQ score? What does the assessment function look like? ● Develop the function together with the impact values ● Take incompleteness into consideration ● Provenance graphs could be fragmentary ● Annotations could be missing Olaf Hartig - Using Web Data Provenance for Quality Assessment 30
  • Step 3 – Assessment Function Olaf Hartig - Using Web Data Provenance for Quality Assessment 31
  • Step 3 – Assessment Function msr created by performed by sensor1 type: Data Item cExc type: Data Creator expiry time type: Data Creation 11:00 creation time contained by 10:00 Execution Time: 10:00 doc retrieved by store type: Document type: Data Providing Service aExc accessed type: Data Access sys performed by type: Data Accessor Execution Time: 10:13 Olaf Hartig - Using Web Data Provenance for Quality Assessment 32
  • Step 3 – Assessment Function msr created by performed by sensor1 type: Data Item cExc type: Data Creator expiry time type: Data Creation 11:00 creation time contained by 10:00 Execution Time: 10:00 doc retrieved by store type: Document type: Data Providing Service aExc accessed type: Data Access sys performed by type: Data Accessor Execution Time: 10:13 Olaf Hartig - Using Web Data Provenance for Quality Assessment 33
  • Step 3 – Assessment Function t(msr) = 1 – (10:15 – 10:00) / (11:00 – 10:00) =1– 0.25h / 1h = 0.75 msr created by performed by sensor1 type: Data Item cExc type: Data Creator expiry time type: Data Creation 11:00 creation time contained by 10:00 Execution Time: 10:00 doc retrieved by store type: Document type: Data Providing Service aExc accessed type: Data Access sys performed by type: Data Accessor Execution Time: 10:13 Olaf Hartig - Using Web Data Provenance for Quality Assessment 34
  • Conclusion ● Web Data Provenance (data creation + data access) ● General approach for provenance-based IQ assessment ● Impact values: influence of provenance elements on IQ ● Design decisions for actual assessment methods ● Application to timeliness (more in the paper) ● Future work: ● How do we deal with incompleteness? ● Application of the approach to other IQ criteria Olaf Hartig - Using Web Data Provenance for Quality Assessment 35
  • These slides have been created by Olaf Hartig http://olafhartig.de This work is licensed under a Creative Commons Attribution-Share Alike 3.0 License (http://creativecommons.org/licenses/by-sa/3.0/) Attribution: ● http://www.flickr.com/photos/rrrrred/3809362767/ ● http://www.hasslefreeclipart.com Olaf Hartig - Using Web Data Provenance for Quality Assessment 36