Linked Data offers an entity-based infrastructure to resolve indirect relations between resources, expressed as chains of links. If we could benchmark how effective retrieving chains of links from these sources is, we can motivate why they are a reliable addition for exploratory search interfaces. A vast number of applications could reap the benefits from encouraging insights in this field. Especially all kinds of knowledge discovery tasks related for instance to ad-hoc
decision support and digital assistance systems. In this paper, we explain a benchmark model for evaluating the effectiveness of associating chains of links with keyword-based queries. We illustrate the benchmark model with an example case using academic library and conference metadata where we measured precision involving targeted expert users and directed it towards search effectiveness. This kind of typical semantic search engine evaluation focusing on information
retrieval metrics such as precision is typically biased towards the final result only. However, in an exploratory search scenario, the dynamics of the intermediary links that could lead to potentially relevant discoveries are not to be neglected.
Exomoons & Exorings with the Habitable Worlds Observatory I: On the Detection...
Benchmarking the Effectiveness of Associating Chains of Links for Exploratory Semantic Search
1. Benchmarking the Effectiveness of Associating
Chains of Links for Exploratory Semantic Search
Laurens DeVocht
Selver Softic, RubenVerborgh, Erik Mannens, Martin Ebner, RikVan de Walle
15. [EXPLORATORY SEARCH: FROM FINDINGTO UNDERSTANDING, Machionini, 2006]
Lookup Learn Investigation
Exploratory Search
`Learning searches involve multiple iterations and return sets of
objects that require cognitive processing and interpretation’
`Searches that support investigation involve multiple iterations that take place
over perhaps very long periods of time and may return results that are critically
assessed before being integrated into personal and professional knowledge bases’
Definition
15
23. Effectiveness
The effectiveness E indicates the overall perception of the results by
the users taking into account expert-user feedback.
# user marked relevant objects
E =
# retrieved objects
Note:
E can be interpreted as precision in traditional IR.
Typical IR examine both precision and recall.
23
[TALKEXPLORER,Verbert et al., 2013]
31. Limitations
Only indicate comparisons to baseline within the same use case.
Not possible to use the benchmark as a leverage to compare different
approaches across use cases
Could better demonstrate in which aspects an exploratory approach
excels traditional systems.
31
32. Future Work
Put the results in perspective by indicating the nuances among different
expert user ratings.
Especially when there is expert disagreement or inconsistencies.
Facilitate generalization of the preliminary search context,
so results for engines can be reusable across datasets: avoiding that a
certain engine’s results differ strongly when changing the data and queries.
Make sure that the approach is generic and can be applied to other search
contexts with different data and use cases.
32
33. Benefits
Compare exploratory search engines to a baseline:
show use cases when the baseline can be outperformed;
for which queries the ‘engine under test’ is relatively more effective.
Sensitive to initial query keywords as inputted by the user:
when there are inconsistencies or vague terms,
even mismatches in the query context, or when expert users disagreed.
33
In this part of exploratory search - only precision
because we are interested in how a each search result is ‘effective’ in help the user reach its search goal, not giving complete results at this point.