Invited talk @Aberdeen, '07: Modelling and computing the quality of information in e-science


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • From traditional DQ to the biologist’s problem of defining quality based on data semantics
  • Data produced for the first time Mention evolution of experimental techniques Its production not streamlined No agreement on how to define its quality
  • Searching for “nuggets of quality knowledge”
  • Here is the compilation model for mapping bound views to a sub-workflow
  • Embedding the sub-flow requires a deployment descriptor : Adapters between host flow and quality subflow Data and control links between host flow tasks and quality flow tasks
  • Activated during execution of the quality sub-flow – blocks the workflow for the duration of the interaction
  • Our quality view specification language allows users to define abstract quality processes. Evidence types are ontology classes. Evidence values are class individuals, which are represented by variables. These variables are bound to values at runtime; the values themselves are either fetched from a repository of persistent annotations, or they are computed on demand by annotation functions. In our use cases, we have found examples of both. This process steps abstracts out from the issue of annotation lifetime Assertions are computed by services, which are represented by ontology classes, too. The tagName is the single output of the service (one for each input data item) Finally, the action step contains the condition/action pairs – here conditions are expressed on the variables introduced earlier, which define the scope. The semantics of the action step is that the expression is evaluated for each data item, and the corresponding action is taken, eg the item is sent to a specific channel
  • Benefit of this model: Ability to share definitions within a community Consistency checking through reasoning -- cite previous papers? Flexibility
  • From right to left: Data / knowledge layer Framework services Quality views management Targeted compiler(s)
  • Invited talk @Aberdeen, '07: Modelling and computing the quality of information in e-science

    1. 1. Modelling and computing the quality of information in e-science Paolo Missier, Suzanne Embury, Mark Greenwood School of Computer Science University of Manchester, UK Alun Preece, Binling Jin Department of Computing Science University of Aberdeen, UK Aberdeen, 24/1/07
    2. 2. Quality of data <ul><li>Main driver, historically: data cleaning for </li></ul><ul><li>Integration : use of same IDs across data sources </li></ul><ul><li>Warehousing, analytics : </li></ul><ul><ul><li>restore completeness, </li></ul></ul><ul><ul><li>reconcile referential constraints </li></ul></ul><ul><ul><li>cross-validation of numeric data by aggregation </li></ul></ul><ul><li>Focus: </li></ul><ul><li>Record de-duplication, reconciliation, “linkage” </li></ul><ul><ul><li>Ample literature – see eg Nov 2006 issue of IEEE TKDE </li></ul></ul><ul><li>Consistency of data across sources </li></ul><ul><li>Managing uncertainty in databases (Trio - Stanford) </li></ul>The need for data quality control is rooted in the data management practice
    3. 3. Common quality issues <ul><li>Completeness : not missing any of the results </li></ul><ul><li>Correctness : each data should reflect the actual real-world entity that it is intended to model </li></ul><ul><ul><li>The actual address where you live, the correct balance in your bank account… </li></ul></ul><ul><li>Timeliness : delivered in time for use by a consumer process </li></ul><ul><ul><li>Eg stock information </li></ul></ul><ul><li>… </li></ul>
    4. 4. Taxonomy for data quality dimensions
    5. 5. Our motivation: quality in public e-science data <ul><li>Large volumes of data in many public repositories </li></ul><ul><li>Increasingly creative uses for this data </li></ul>Problem: using third party data of unknown quality may result in misleading scientific conclusions GenBank UniProt EnsEMBL Entrez dbSNP
    6. 6. Some quality issues in biology <ul><li>“ Quality” covers a broader spectrum of issues than traditional DQ </li></ul><ul><li>“ X% of database A may be wrong ( unreliable ) – but I have no easy way to test that” </li></ul><ul><li>“ This microarray data looks ok but is testing the wrong hypothesis ” </li></ul><ul><li>The output from this sequence matching algorithm produces false positives </li></ul><ul><li>… </li></ul>Each of these issues calls for a separate testing procedure Difficult to generalize
    7. 7. Correctness in biology - examples No false positives: Every protein in the output is actually present in the cell sample Generate peptides peak lists, match peak lists (eg Imprint) Qualitative proteomics: Protein identification No false positives, no false negatives Microarray data analysis Transcriptomics: Gene expression report (up/down-regulation) Functional annotation f for p correct if function f can reliably be attributed to p Manual curation Uniprot protein annotation Correctness Creation process Data type
    8. 8. Defining quality in e-science is challenging <ul><li>In-silico experiments express cutting-edge research </li></ul><ul><ul><li>Experimental data liable to change rapidly </li></ul></ul><ul><ul><li>Definitions of quality are themselves experimental </li></ul></ul><ul><li>Scientists’ quality requirements often just a hunch </li></ul><ul><ul><li>Quality tests missing or based on experimental heuristics </li></ul></ul><ul><ul><li>Definitions of quality criteria are personal and subjective </li></ul></ul><ul><li>Quality controls tightly coupled to data processing </li></ul><ul><ul><li>Often implicit and embedded in the experiment </li></ul></ul><ul><ul><li>Not reusable </li></ul></ul>
    9. 9. Research goals <ul><li>Make personal definitions of quality explicit and formal </li></ul><ul><ul><li>Identify a common denominator for quality concepts </li></ul></ul><ul><ul><li>Expressed as a conceptual model for Information Quality </li></ul></ul><ul><li>Make existing data processing quality-aware </li></ul><ul><ul><li>Define an architectural framework that accommodates personal definitions of quality </li></ul></ul><ul><ul><li>Compute quality levels and expose them to the user </li></ul></ul>Elicit “nuggets” of latent quality knowledge from the experts
    10. 10. Example: protein identification Data output Protein identification algorithm “ Wet lab” experiment Protein Hitlist Protein function prediction Correct entry  true positive This evidence is independent of the algorithm / SW package It is readily available and inexpensive to obtain Evidence : mass coverage (MC) measures the amount of protein sequence matched Hit ratio (HR) gives an indication of the signal to noise ratio in a mass spectrum ELDP reflects the completeness of the digestion that precedes the peptide mass fingerprinting
    11. 11. Correctness of protein identification Estimator function: (computes a score rather than a probability) PMF score = (HR x 100) + MC + (ELDP x 10) Prediction performance – comparing 3 models: ROC curve: True positives vs false positives
    12. 12. Quality process components Data output Protein identification algorithm “ Wet lab” experiment Protein Hitlist Protein function prediction Goal: to automatically add the additional filtering step in a principled way <ul><li>Evidence : </li></ul><ul><li>mass coverage (MC) </li></ul><ul><li>Hit ratio (HR) </li></ul><ul><li>ELDP </li></ul>PMF score = (HR x 100) + MC + (ELDP x 10) Quality filtering Quality assertion :
    13. 13. Quality Assertions <ul><li>QA(D): any function of evidence (metadata for D) that computes a partial order on D </li></ul><ul><li>Score model (total or partial order) </li></ul><ul><li>Classification model with class ordering: </li></ul>analyze Reject < analyze < accept      D         reject accept Actions associated to regions
    14. 14. Abstract quality views <ul><li>An operational definition for personal quality: </li></ul><ul><li>Formulate a quality assertion on the dataset: </li></ul><ul><ul><li>i.e. a ranking of proteins by PMF score </li></ul></ul><ul><li>Identify underlying evidence necessary to compute the assertion </li></ul><ul><ul><li>the variables used to compute the score (HR, MC, ELDP) </li></ul></ul><ul><li>Define annotation functions that compute evidence values </li></ul><ul><ul><li>Functions that compute HR, MC, ELDP </li></ul></ul><ul><li>Define quality regions on the ranked dataset </li></ul><ul><ul><li>In this case, intervals of acceptability </li></ul></ul><ul><li>Associate actions to each region </li></ul>
    15. 15. Computable quality views as commodities <ul><li>Cost-effective quality-awareness for data processing: </li></ul><ul><li>Reuse of high-level definitions of quality views </li></ul><ul><li>Compilation of abstract quality views into quality components </li></ul>Abstract quality views binding and compilation Executable Quality process <ul><li>runtime environment </li></ul><ul><li>data-specific quality services </li></ul>Qurator architectural framework:
    16. 16. Quality hypotheses discovery and testing abstract quality view Quality model Performance assessment Execution on test data Compilation Compilation Targeted Compilation Quality-enhanced User environment Quality-enhanced User environment Quality-enhanced User environment Target-specific Quality component Target-specific Quality component Target-specific Quality component Deployment Deployment Deployment <ul><li>Multiple target environments: </li></ul><ul><li>Workflow </li></ul><ul><li>query processor </li></ul>Quality model definition
    17. 17. Experimental quality <ul><li>Making data processing quality-aware using Quality Views </li></ul><ul><ul><li>Query, browsing, retrieval, data-intensive workflows </li></ul></ul> Discovery and validation of “Quality nuggets” Quality View Model testing Test datasets  Embedding quality views and flow-through testing +
    18. 18. Execution model for Quality views <ul><li>Binding  compilation  executable component </li></ul><ul><ul><li>Sub-flow of an existing workflow </li></ul></ul><ul><ul><li>Query processing interceptor </li></ul></ul>Host workflow Abstract Quality view Embedded quality workflow QV compiler D D’ Quality view on D’ Host workflow: D  D’ Qurator quality framework Services registry Services implementation
    19. 19. Example: original proteomics workflow Taverna workflow Quality flow embedding point
    20. 20. Example: embedded quality workflow
    21. 21. Interactive conditions / actions
    22. 22. Generic quality process pattern Collect evidence - Fetch persistent annotations - Compute on-the-fly annotations <variables <var variableName=&quot; Coverage “ evidence=&quot; q:Coverage &quot;/> <var variableName=&quot; PeptidesCount “ evidence=&quot; q:PeptidesCount &quot;/> </variables> Evaluate conditions Execute actions <action> <filter> <condition> ScoreClass in {``q:high'', ``q:mid''} and Coverage > 12 </condition> </filter> </action> Compute assertions Classifier Classifier Classifier <QualityAssertion serviceName=&quot; PIScoreClassifier &quot; serviceType=&quot; q:PIScoreClassifier &quot; tagSemType=&quot; q:PIScoreClassification &quot; tagName=&quot; ScoreClass &quot; Persistent evidence
    23. 23. A semantic model for quality concepts Quality “upper ontology” (OWL) Evidence annotations are class instances Quality evidence types Evidence Meta-data model (RDF)
    24. 24. Main taxonomies and properties assertion-based-on-evidence: QualityAssertion  QualityEvidence is-evidence-for: QualityEvidence  DataEntity Class restriction: MassCoverage   is-evidence-for . ImprintHitEntry Class restriction: PIScoreClassifier   assertion-based-on-evidence . HitScore PIScoreClassifier   assertion-based-on-evidence . Mass Coverage
    25. 25. The ontology-driven user interface Detecting inconsistencies: no annotators for this Evidence type Detecting inconsistencies: Unsatisfied input requirements for Quality Assertion
    26. 26. Qurator architecture
    27. 27. Quality-aware query processing
    28. 28. Research issues <ul><li>Quality modelling: </li></ul><ul><li>Provenance as evidence </li></ul><ul><ul><li>Can data/process provenance be turned into evidence? </li></ul></ul><ul><li>Experimental elicitation of new Quality Assertions </li></ul><ul><ul><li>Seeking new collaborations with biologists! </li></ul></ul><ul><li>Classification with uncertainty </li></ul><ul><ul><li>Data elements belong to a quality class with some probability </li></ul></ul><ul><li>Computing Quality Assertions with limited evidence </li></ul><ul><ul><li>Evidence may be expensive and sometimes unavailable </li></ul></ul><ul><ul><li>Robust classification / score models </li></ul></ul><ul><li>Architecture: </li></ul><ul><li>Metadata management model </li></ul><ul><ul><li>Quality Evidence is a type of metadata with known features… </li></ul></ul>
    29. 29. Summary <ul><li>For complex data types, often no single “correct” and agreed-upon definition of quality of data </li></ul><ul><li>Qurator provides an environment for fast prototyping of quality hypotheses </li></ul><ul><ul><li>Based on the notion of “evidence” supporting a quality hypothesis </li></ul></ul><ul><ul><li>With support for an incremental learning cycle </li></ul></ul><ul><li>Quality views offer an abstract model for making data processing environments quality-aware </li></ul><ul><ul><li>To be compiled into executable components and embedded </li></ul></ul><ul><ul><li>Qurator provides an invocation framework for Quality Views </li></ul></ul>Publications: Qurator is registered with OMII-UK