Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ROSeAnn: Reconciling Opinions of Semantic Annotators Poster


Published on

A growing number of resources are available for enriching documents with semantic annotations. While originally focused on a few standard classes of annotations, the ecosystem of annotators is now becoming increasingly diverse. Although annotators often have very different vocabularies, with both high-level and specialist
concepts, they also have many semantic interconnections. We will show that both the overlap and the diversity in annotator vocabularies motivate the need for semantic annotation integration: middleware that produces a unified annotation on top of diverse semantic annotators. On the one hand, the diversity of vocabulary allows applications
to benefit from the much richer vocabulary available in
an integrated vocabulary. On the other hand, we present evidence that the most widely-used annotators on the web suffer from serious accuracy deficiencies: the overlap in vocabularies from individual annotators allows an integrated annotator to boost accuracy by exploiting inter-annotator agreement and disagreement.

The integration of semantic annotations leads to new challenges, both compared to usual data integration scenarios and to standard aggregation of machine learning tools. We overview an approach to these challenges that performs ontology-aware aggregation. We
introduce an approach that requires no training data, making use of ideas from database repair. We experimentally compare this with a supervised approach, which adapts maximal entropy Markov models to the setting of ontology-based annotations. We further experimentally compare both these approaches with respect to ontology-unaware
supervised approaches, and to individual annotators.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

ROSeAnn: Reconciling Opinions of Semantic Annotators Poster

  1. 1. ROSeAnn:Reconciling Opinions of Semantic Annotations The Need for Integration Conflicting Opinions Supervised Aggregation (MEMM) Another brilliant goal by England midfielder David Beckham earned Manchester United a point from a 1-1 draw at Stamford Bridge on Saturday after Gianfranco Zola had put Chelsea ahead. United stayed top of the English league standings as Liverpool could only draw 0-0 at home to Blackburn despite dominating throughout. Newcastle moved into third as a Les Ferdinand goal was enough to win 1-0 at bottom club Middlesbrough. Ian Marshall scored a hat-trick in the first 27 minutes to help Leicester beat Derby 4-2 while Sheffield Wednesday came from two down to win 3-2 at Southampton. Leeds won 1-0 at Sunderland and Coventry against Everton and Nottingham Forest against Aston Villa ended goalless. Semantic annotators label text snippets as referring to certain entities, e.g., Barack Obama, London, or as instances of particular entity types, e.g., actors, governmental organisations, countries. Semantic Annotators A growing number of freely available online services can enrich documents with semantic annotations. Unfortunately, their opinions about an entity often disagree as they might be based on very diverse background knowledge such as training corpora, knowledge bases, contextual information, POS tags, and crowds. AlchemyAPI:Person DBPediaSpotlight:SoccerClub Lupedia:Settlement Wikimeta:Organisation StanfordNER:Organisation IllinoisNER:Organisation Extractiv:City AlchemyAPI:Facility AlchemyAPI:GeographicFeature DBPediaSpotlight:SportsTeam Zemanta:SportsTeam OpenCalais:GeographicFeature Luying Chen, Stefano Ortona, Giorgio Orsi, and Michael Benedikt University of Oxford - Department of Computer Science ( Unsupervised Weighted Repair (WR) Each annotator comes with a vocabulary of semantically-related entity types that often overlap on common-sense entities, such as places, persons, and companies. However, each annotator covers only a fraction of a much larger universe of concepts. By relating such vocabularies to each other via mappings we can achieve much better coverage. Museum Accuracy of individual annotators varies greatly, redundancy and logical relationships can be used to gain confidence about an entity. Each annotator contributes some original types. None of them can be dropped without losing recall. Empirical Evaluation 1 0.8 0.6 0.4 0.2 0 Precision Recall FScore Person Date Movie 1 0.8 0.6 0.4 0.2 0 Precision Recall Fscore Location Sport Movie Thing Organisation Facility Place Person Point of Interest Club Soccer Club Location Settlement Natural City Feature Geographic = Feature Organisation(X) Person(X) Organisation(X) Location(X) Settlement(X) GeographicFeature(X) Location(X) PointOfInterest(X) Place(X) Person(X) Person(X) Facility(X) Annotation vocabularies are semantically related to each other via existing knowledge bases (DBPedia, Freebase) or via common-sense. These relations can be used to map them to a common ontology to check for logical conflicts or compatibility. IS-A constraints enable type inference, thus increasing recall at the expense of precision. Disjointness constraints induce logical inconsistencies used to locate potentially erroneous annotations, thus increasing precision. WR computes an ontology-aware repair of the set of annotations that is logically consistent and “fair” to the annotators involved. WR is unsupervised and does not assume any prior knowledge about the annotators. If the global ontology only states IS-A and disjointness constraints, a solution can be computed efficiently (repair → 2-SAT). WR is designed for scenarios where training data is unavailable or sparse. Conflicts also occur at span-level, i.e., annotators agreeing on the entity type but not on the extension of the span. Notion of span adapted to support composite annotations consisting of tokens carrying logically incompatible types, e.g., “[[Subic] [Naval Base]]”. 0.85 0.8 0.75 0.7 0.65 0.6 Politician Politician Place Extractiv WeightedRepair MEMM μP μR μF1 MP MR MF1 Place ≠ Person 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 Politician Place Politician Zemanta WeightedRepair MEMM Score(Politician) -> +1 Score(Person) -> +1 Score(Place) -> -1 μP μR μF1 MP MR MF1 0.9 0.8 0.7 0.6 Fox WeightedRepair MEMM μP μR μF1 MP MR MF1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 NERD WeightedRepair MEMM μP μR μF1 MP MR MF1 WR and MEMM have been tested on ~670 documents from 4 corpora: MUC7, Reuters, NETagger, and Fox. Comparison have been carried out against individual annotators and competitor aggregators such as Fox and NERD. Semantic (micro/macro) precision and (micro/macro) recall metrics have been adopted for the comparison. When training data is available, supervision can be used to learn the most probable sequences of annotations given those available from gold standard annotated documents. MEMM can learn unorthodox relationships among annotations that do not necessarily follow standard inference rules, e.g., it can learn to predict a subclass C from (a set of) annotations mentioning a superclass C’ of C. Person Politician Place ¦ u c max Score(c) xc Politician Person Politician WR and MEMM perform in average better than all individual annotators and aggregators with the exception of OpenCalais. However, its vocabulary represents some 18% of all types. MEMM is more accurate than WR, but on sparse datasets WR shows better performance than MEMM. WR delivers higher recall than MEMM that, in turn, is more precise than WR. 512 256 128 64 32 16 WR Solution Computation Reuters MUC7 NETTagger FOX 2 5 8 11 msec # Annotators 10000 1000 msec 100 10 1 MEMM Prediction Reuters MUC7 NETTagger FOX 2 5 8 11 # Annotators Online aggregation of annotation is feasible in practice (~300ms for WR and ~1s for MEMM). The aggregation time is orders of magnitude less than the time required to invoke the online services and collect their answers. Apart from the entity type and the source annotator, the feature set for MEMM includes ontological features such as IS-A and disjointness. All features are token based. Online annotators are often black boxes characterised by a continuously evolving vocabulary, where entity types are added, merged, or removed. Region Person Country Scientist Planet Brand Product Planet Ocean Company Mansour WR receives as input an annotated span and produces as output a logically consistent set of annotations. An atomic score is computed for each opinion, based on (inferred) support / opposition by other opinions. Annotations are inserted or deleted from the initial solution to obtain a consistent set of annotation that maximizes the objective function. An initial solution consists of the possibly inconsistent union of all entity types. Only the most specific annotations are retained in the final solution. xc 1 xc 1