Formally Measuring Agreement and Disagreement in OntologiesMathieu d’AquinKMi, The Open University – m.daquin@open.ac.uk

Ontologies are knowledge artifacts……. and knowledge is subjectiveWhat do we mean?

What do we mean?Therefore, two different ontologies can express two different views (=disagree)

What do we mean?Therefore, two different ontologies can express two different view (=disagree)Or the same/similar view(s) (=agree)

What do we mean?Similarly, an ontology can agree or disagree with a single ontology statementSeafood subClassOf MeatNo, don’t think so…Yes, of course!?

And why is that interesting?Being able to measure these (dis)agreements could help in choosing the right ontology, in understanding what exist and in making sense of a collection of ontologies?

A naïve approach…To detect disagreements, one could “simply” merge ontolologies and check for incoherence/inconsitencySeaFooddisjointWithMeatSeaFoodsubClassOf MeatDISAGREEMENT

A naïve approach, but…… a bit limitedAnimal subClassOf Human?Human subClassOf AnimalLion subClassOf Species?Lion type Species?Car subClassOf VehicleEricCantona type FootballPlayer

RequirementsR1:Ontologies agree with themselvesKind of obviousR2: Covering different domains is not agreeingCar vs Footballer example.R3: There are different levels of agreements and disagreementsHuman subClassOf Animal vsHuman disjointWith AnimalHuman subClassOf Animal vsAnimal subClassOf HumanR4: (dis)agreement measures should be independent from matching techniquesMatching is necessary, but not part of the measureR5: It is possible to agree and disagree at the same timeLion type Species vsLion subClassOf Species

Basic frameworkThe clever bit: using 2 measures instead of one…Agreement(s, O)  [0..1]Disagreement(s, O)  [0..1]With s a statement and O an ontologyInterpretation:A (s, O) = 1, D(s, O) = 0, O fully agrees with sA (s, O) = 0, D(s, O) = 1,O fully disagrees with sA (s, O) = 0, D(s, O) = 0,O doesn’t care about sA (s, O) > 0, D(s, O) > 0,O agrees to a certain extent with s or disagrees to a certain extent with s, or both

But how to calculate that?Considering a statement <subject, relation, object>, an ontology might agree or disagree with the relation between entities corresponding to subject and object.Extracting information about the relation between matching entities in an ontology:O=AnimalHuman subClassOfAnimalMatchings=LivingBeingHumanBirdR-Module:Human subClassOf Animal, Animal subClassOf Human, Animal equivalentClass HumanMinimal RM:Animal equivalentClass Human

Simplified representation of MRMsWith subject’ and object’ the matching entities on O to the subject and object in s, the MRM of O regarding s can be represented as a list of relations:subject’ subClassOf object’ subClassOfobject’ subClassOf subject’  subClassOf-1etc.Assumptions:The MRM is non redundant (part of the definition){equivalenClass}  OK{equivalentClass, subClassOf, subClassOf-1}  not OKThe MRM should be coherent and consistent (guarantied if O is coherent and consistent, in accordance with our 1st requirement: an ontology agrees with itself){subClassOf}  OK {subClassOf, disjointWith}  not OKThe MRM should be homogeneous in terms of modeling, i.e., it should not imply that en entity is at the same time a class and a property for example.{fatherOf domain Person, fatherOf range Person}  OK {fatherOf domain Person, fatherOfsubClassOf Person}  not OK

Nice Property and Measure definitionsThe good news: There is a small finite set of possible MRM, whatever is are O and sWhich means?The measures of agreement and disagreement can be entirely defined by providing explicitly the values in two matrixesAgreementDisagreementRelation in s0 < A1 < A2 < 1MRM

So?A1/D1Animal subClassOf HumanHuman subClassOf AnimalLion subClassOf SpeciesLion type SpeciesA2/D20/0Car subClassOf VehicleEricCantona type FootballPlayer

Measuring agreement and disagreement between whole ontologies, to understand a set of ontologiesThe big formulas:What to do now…

Using 21 ontologies containing a concept SeaFoodCamp 1: seaFooddisjointWith MeatCamp 2: SeaFoodsubClassOf MeatDisagreementAgreement

Measuring consensus and controversy in a collection of ontologiesR, a repository of ontologies.Can be positive (high agreement, low disagreement) or negative (the contrary)High controversy means no clear cut between agreement and disagreementWhat else could we do?

Watson: Thousands of ontologies automatically crawled from the Web (http://watson.kmi.open.ac.uk)a: global agreement, d: global disagreement, cs: consensus, ct: controversyAssessing the statements related to SeaFood in WatsonExample

Using a set of 456 evaluated mappings between 2 large thesaurus in the agricultural domain (71.3% precision)Conclusion: There is less consensus on incorrect mappings. Controversy indicates mappings that need to be investigated more.Can we use it for assessing mappings?

We provided definitions of measures of agreement and disagreement in ontologies, including consensus and controversy in ontology repositories.We showed that when applied on real Web ontologies, this could help assessing statements and mappings, and getting an overview of a particular set of ontologies.We realized an implementation based on the Watson API. We intend to make it available through a Web service.Many applications to explore: visualization of ontology collections, ontology selection and reuse, propagation of trust based on agreement, …… and new directions: computing explanations for the (dis)agreement, different parameters and matching techniques for different applications, resolving disagreements (decide who’s right), etc.Also, complexity and performance are still difficult issues.Conclusion

Thank You!Mathieu d’Aquin @mdaquinm.daquin@open.ac.ukhttp://people.kmi.open.ac.uk/mathieu