Semantic Need: Guiding Metadata Annotations by Questions People #ask

Semantic Need
Guiding Metadata Annotations by Questions People #ask
Hans-Jörg Happel, FZI Karlsruhe, Germany
2010-11-09 @ 9th Int. Semantic Web Conference (ISWC 2010), Shanghai, China

Agenda
• Introduction
• Semantic Gap Heuristics
• Semantic MediaWiki Study
• Extension:Semantic Need
• Summary & Outlook
Semantic Need: Guiding Metadata Annotations by Questions People #ask - ISWC 2010; Shanghai, China 2

• SMW is a popular Semantic Web application that
allows to annotate Wiki pages semantically
• Semantic interpretation of the existing Wiki categories
• Syntax extension for [[Wiki links]]
– Relations to other pages: [[Capital::Abuja]]
– Literals: [[Inhabitants::182418]]
Semantic MediaWiki (SMW)

Structured Queries in SMW
• SMW also allows for structured queries
{{#ask:
[[Category:Country]] [[OnContinent::Africa]]
|?area
|?...
}}
 SMW resembles the Semantic Web in small

SMW Query Result
{{#ask:
|?area
|?...
}}
???
…?

What happend to „Nigeria“?
Info might
be missing
…not
annotated
properly
Different
property
name
Distributed
data source
not available

Semantic Gaps
• Observation:
– „Semantic gap between supply and demand on the Semantic
Web” [Mik09]
– Due to the evolutionary nature of the (Semantic) Web (OWA)
• What is missing? – i.e.:
– KB: Axioms that are known (e.g. statements about Nigeria)
– XKB: Axioms not yet known but people would like to know

Towards Semantic Need
• Research questions
– How to identify „Semantic Gaps“?
– Do „Semantic Gaps“ exist?
– If yes, how to close these gaps?
• Research approach
– Propose heuristics
– Explorative: Analyze Public Semantic Web
– Constructive: Design and evaluate tools
88Semantic Need: Guiding Metadata Annotations by Questions People #ask - ISWC 2010; Shanghai, China

Agenda
• Introduction

Idea: Guide Annotation by Information Needs
• Means for deriving information needs
– (Structured) queries
– Information access/browsing
– Context
– …?
• We chose to focus on queries
– Explicit; can be captured easily
– Express a „demand“ [Mik09]
– Recur across time and different people (at least in
IR! [Smy05, Tee06, Zha09])

Identifying „Semantic Gaps“
• Focus on
– Conjuctive queries
– Retrieving instances and their properties
• Core elements
{{#ask: [[Category:Country]]
[[OfContinent::Africa]]
|?hasArea
|?population
|?hasCapital
|?Currency
}}
Printout
Statement
Printout
Statement
ConditionsConditions

Semantic Gap Heuristic #1: Near Matches
• Instance I ∈ KB is considered a near match of a
query q if:
– I is not in the result set of q in KB
– There is at least one conjunctive query atom of q for
which I is part of the result set
– I would be in the result set of q in KB ∪ XKB
• Correspondingly, we consider q to have an
incomplete result set if it has „near matches“

Semantic Gap Heuristic #1: Example
{{#ask:
|?area
|?...
}}
Egypt 1.001.449 km2 83.082.869 Cairo Egyptian pound
Lacks annotation
[[OnContinent::Africa]]
„Near Match“
Lacks annotation
[[OnContinent::Africa]]
„Near Match“

Semantic Gap Heuristic #2: Missing Printout Values
• Instance I ∈ KB is considered to have missing
printout values for a query q if:
– I is part of the result set of q
– q contains a printout statement x for which no property
value of I exists in the KB
• Note: Technically, „missing printout values“ can be
considered equivalent to near matches
– SPARQL requires „OPTIONAL“ modifier to yield missing
printout values
– SMW-QL allows to set printout values required
• Correspondigly, we consider q to have an sparse
result set if it has at least one „missing printout
value“

Semantic Gap Heuristic #2: Example
„Missing Printout
Values“
„Missing Printout
Values“

Agenda
• Introduction

Public SMW Analysis: Design
• Goal
– Do „Semantic Gaps“ exist?
– Find out significance of missing result values and near
matches in real world queries
• Crawling public SMW installations
– Collected ~200 public SMW installations via overview lists and
search engines
– Selection of 8 SMW instances (filtered based on data and
technical reasons and random choice)
– Those have on average 1880 annotations and 35 inline
queries
• Checking for sparse & incomplete query results
– Analyzing 25 (out of 285) queries (only ASK-Queries, online
"Table"-output format, only queries with printout statements
resp. conjunctions)
– 17 of these queries were located on Template pages

Public SMW Analysis: Results
• Printout-Values
– In average, 16% of cells in a result set were empty due to
missing annotations (up to 63% for certain queries)
 Allows for identifying a total of 296 missing printout values
– Validation showed that 13 out of 15 manually investigated
empty cells could be considered missing information
• Near matches
– In average, 22% of all potential result pages of a query lack a
selective annotation (up to 94% for certain queries)
Allows for identifying a total of 147 potentially missing
annotations for “selective” properties
– Validation showed that 10 out of 15 manually investigated
near matches could be considered missing information
• Note: based on evaluation conditions, only around 9%
of the overall inline queries were analyzed

Agenda
• Introduction

Extension:Semantic Need: Idea
• Goal:
– How to close „Semantic Gaps“?
– Guide the creation of semantic annotations in SMW
• Design principles
– „Need-driven Knowledge Sharing“ [Hap09b]
– People are willing to contribute missing information, if
they recognize that there is concrete demand
– Derived from related work and supported by user studies
• Core features
– Capture and store needs (i.e. #ask-queries)
– Guide annotations by extending and modifying the SMW
user interface based on information need heuristics (i.e.
„near matches“ and „missing printout values“

Screenshot: In-Page Annotation
HintHint Sources of needSources of need

Semantic Need Online Survey: Design
• 34 questions on SMW and Semantic Need
• Target group: SMW experts (via mailinglist,
invitation)
• Data collected in June/July 2010
• 30 complete answers (out of 58)

Semantic Need Online Survey: Semantic Need can help
• Problem patterns do occur
– Sparse result set: 12/30 considered
problematic
– Incomplete result set: 23/30 considered
problematic
• Stressed in free text
• Core issue: „invisibility“ of the issue
• Usage of SMW differs
– „Structured“ settings  focus on quality
– „Open“ settings  focus on guidance
– Semantic Need generally considered helpful by
both groups

Maintenance practices: mostly ad hoc
• Methods & tools used to maintain semantic data
– (7: n.a.; due to given external data model)
– 12: none
– 5: „simple“
– 7: „advanced“ (e.g. scripts, documentation, team
decisions)
• How to find missing annotations for a given page
– 6: Compare similar pages („extensional“)
– 7: Check schema („intensional“)
– 4: Text analysis
– 10: Use query

Agenda
• Introduction

Insights
• „Semantic Gaps“ do exist
– Information needs are a valuable source to
find them
– „Missing printout values“ and „near
matches“ seem to be useful heuristics
– Especially „incomplete result sets“ are
considered problematic
• No systematic guidance & gardening of
SMW knowledge bases

Design Implications
• Semantic Annotation
– Issue: Costly, often driven by a pre-defined ontology
structure
– Idea: Consider “incentives for annotation” [Han05]
• Semantic Search
– Issue: Decoupling of provision & access
– Idea: Consider information needs
• Need specification/ontology
• Maintain semantic query logs
• Data Quality/Gardening/Maturing
– Issue: The Semantic Web evolves continuously
– Idea: Allow for better data quality modeling

Summary & Outlook
• Main contributions
– How to identify „Semantic Gaps“  Heuristics based on
queries
– Do „Semantic Gaps“ exist?  Yes
– If yes, how to close these gaps?  Semantic Need
• Next steps
– Large scale analysis of „Semantic Gaps“ (more public SMW
instances)
– Provide stable implementation und gather feedback from
field usage of Semantic Need
• Further research opportunities
– Use needs to guide the sharing of semantic annotations
– Use needs to create schema-level mappings or for
class/property evolution
– Many more (Semantic query logs, UI, Incentives, …)

References
– http://amazonas.fzi.de/semanticneed / (Demo Wiki)
– http://www.mediawiki.org/wiki/Extension:Semantic_Need
• Extension:Woogle4MediaWiki (for non SMW-Wikis)
– http://amazonas.fzi.de/wooglenative/ (Demo Wiki)
– http://www.mediawiki.org/wiki/Extension:Woogle4MediaWiki
• Literature
– [Han05] Handschuh, Siegfried: Creating ontology-based metadata by annotation for the semantic web,
Dissertation, 2005
– [Hap09b] Hans-Jörg Happel: Towards Need-driven Knowledge Sharing in Distributed Teams. In
Proceedings of the 9th International Conference on Knowledge Management (I-KNOW 2009)
– [Hap09c] Hans-Jörg Happel: Social Search and Need-driven Knowledge Sharing in Wikis with Woogle. In
Proceedings of the 5th international Symposium on Wikis and Open Collaboration (Orlando, Florida,
October 25 - 27, 2009). WikiSym '09. ACM, New York, NY, 1-10.
– [Mik09]: Mika, P., Meij, E., Zaragoza, H.: Investigating the semantic gap through query log analysis. In:
International Semantic Web Conference. Lecture Notes in Computer Science, vol. 5823, pp. 441–455.
Springer (2009)
– [Smy05] Smyth, Barry ; Balfe, Evelyn ; Freyne, Jill ; Briggs, Peter ; Coyle, Maurice ; Boydell, Oisin:
Exploiting Query Repetition and Regularity in an Adaptive Community-Based Web Search Engine. In:
User Modeling and User-Adapted Interaction 14 (2005), Nr. 5, S. 383–423.
– [Tee06] Teevan, Jaime ; Adar, Eytan ; Jones, Rosie ; Potts, Michael: History repeats itself: repeat
queries in Yahoo’s logs. In: SIGIR’06: Proceedings of the 29th annual international ACM SIGIR
conference on Research and development in information retrieval. New York, NY, USA : ACM, 2006, S.
703–704.
– [Zha09] Zhang, Dell ; Lu, Jinsong: What queries are likely to recur in web search? In: SIGIR ’09:
Proceedings of the 32nd international ACM SIGIR conference on Research and development in
information retrieval. New York, NY, USA : ACM, 2009, S. 827–828.

The Semantic Web: Problems
• Lack of resources: you might not annotate everything
– Metadata creation is costly
– Access to metadata might be restricted to different spheres of sharing
(private, friends, world…)
– “..probably the most important [open question] for the Semantic Web. How
to create incentives for annotation?” (Handschuh 2005, p198) [12]
• Lack of guidance: you might annotate the wrong things
– „ Semantic gap between supply and demand on the Semantic Web” [Mik09]
– The two processes of metadata creation and metadata use are decoupled
concerning time and actors
– Existing annotation approaches drive the annotation process by the pre-
defined ontology structure
 No unified theory, why metadata is created and how it is shared
– Semantic Web Vision does not address the creator side of metadata –
it spends a lot of effort to convince people using the Semantic Web but not
contributing to it

Semantic Need: Guiding Metadata Annotations by Questions People #ask

Recommended

Recommended

More Related Content

What's hot

What's hot (12)

Similar to Semantic Need: Guiding Metadata Annotations by Questions People #ask

Similar to Semantic Need: Guiding Metadata Annotations by Questions People #ask (20)

More from Hans-Joerg Happel

More from Hans-Joerg Happel (16)

Recently uploaded

Recently uploaded (20)

Semantic Need: Guiding Metadata Annotations by Questions People #ask