Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Measuring the usefulness of
Knowledge Organization Systems
in
Information Retrieval applications
Philipp Mayr
Observatory ...
GESIS
• We are developing interactive information retrieval systems for
searching indexed literature and data sets
• We fo...
Intro
• Typical difficulties in searching digital libraries (DL)
– Vagueness between search and indexing terms
– How to su...
Case Study 1: Information retrieval
experiment
• Intra- and interdisciplinary cross-
concordances in the project
KoMoHe
– ...
Case Study 1
• How effective are the mappings in an actual search?
Does the application of term mappings (TT) improve
sear...
Case Study 2: Information retrieval
experiment
• Discipline-specific Search-Term-
Recommendation (STR) Services in IRM
pro...
Case Study 2
• Are recommendations from discipline specific STRs better suited for
query expansion (QE) than general ones?...
Case Study 3: Interactive IR
experiment
• Measuring the utility and
performance of Search Term
Recommendation (STR) Servic...
Case Study 3
• Usage of the STR significantly often implicates the
occurrence of positive signals during the following ses...
Conclusions
• Information retrieval and
interactice IR settings are able to
demonstrate the utility of KOS
usage (usefulne...
Outlook
• Integrate different recommender systems in
real retrieval tasks (search sessions)
• Use and evaluate recommender...
References
• Hienert, D. & Mutschke, P. (2016). A Usefulness-based Approach for
Measuring the Local and Global Effect of I...
Thank you
Contact:
Dr Philipp Mayr
GESIS - Leibniz Institute for the Social Sciences,
Germany
Email: philipp.mayr@gesis.or...
Upcoming SlideShare
Loading in …5
×

Measuring the usefulness of Knowledge Organization Systems in Information Retrieval applications

511 views

Published on

Paper presented at the Observatory for Knowledge Organisation Systems workshop in Malta, Valletta.

Published in: Science
  • Be the first to comment

Measuring the usefulness of Knowledge Organization Systems in Information Retrieval applications

  1. 1. Measuring the usefulness of Knowledge Organization Systems in Information Retrieval applications Philipp Mayr Observatory for Knowledge Organisation Systems KNOWeSCAPE workshop, Valletta, Malta February 01, 2017
  2. 2. GESIS • We are developing interactive information retrieval systems for searching indexed literature and data sets • We follow the principle „research-based service“; develop research prototypes, test and evaluate them and implement the features which are working for the end users 2
  3. 3. Intro • Typical difficulties in searching digital libraries (DL) – Vagueness between search and indexing terms – How to support searchers with controlled vocabulary? • Assumption: a user’s search (experience) should improve by using Knowledge Organization Systems (KOS): – Vague search tasks – Unfamiliar fields – Cross domain searches • Case studies to demonstrate the effectiveness of KOS in different search scenarios 3
  4. 4. Case Study 1: Information retrieval experiment • Intra- and interdisciplinary cross- concordances in the project KoMoHe – Social Sciences-SocSci; SocSci- Economics; SocSci-Psychology; Politics- Economics; Medicine-Psychology, … • Information retrieval evaluation of the mappings (effectiveness of intellectual mapping) 4Controlled terms
  5. 5. Case Study 1 • How effective are the mappings in an actual search? Does the application of term mappings (TT) improve search over a non-transformed subject (i.e. controlled vocabulary) search (CT)? • Real queries, only equivalence relations, 13 thesaurus mappings 5 Mayr/Petras 2008 • Overlap and more identical terms in intradisciplinary mappings • Interdisciplinary mappings made the strongest effect
  6. 6. Case Study 2: Information retrieval experiment • Discipline-specific Search-Term- Recommendation (STR) Services in IRM project • Are recommendations from discipline specific STRs better suited for query expansion than general ones? • Co-occurence of terms in title/description and assigned controlled terms • 17 STR services – 16 discipline-spec. – 1 global 6Lüke et al. 2012
  7. 7. Case Study 2 • Are recommendations from discipline specific STRs better suited for query expansion (QE) than general ones? • 100 topics from the GIRT corpus, top 4 recommendations to expand the original query • gSTR = global STR; tSTR = topical STR; bSTR = best-performing STR 7 Lüke et al. 2012 • QE with specific STRs leads to significantly better results than QE with a general STR • Selecting the best matching specific STR in an automatic way is a major
  8. 8. Case Study 3: Interactive IR experiment • Measuring the utility and performance of Search Term Recommendation (STR) Services in AMUR project • Logfile-based evaluation of STR usage and later search session success • We defined positive signals (export, save, email, full text …) in the system enter_search_term→select_term_from_reco mmender→search→view_record_1→view_r ecord_2→view_record_3→export_record • Analysis of one year of log data 8 Hienert/Mutschke 2016
  9. 9. Case Study 3 • Usage of the STR significantly often implicates the occurrence of positive signals during the following session steps 9 Hienert/Mutschke 2016
  10. 10. Conclusions • Information retrieval and interactice IR settings are able to demonstrate the utility of KOS usage (usefulness) – In experimental settings – In user evaluations • Each methodology has pros and cons – Effort and significance in small user studies – Too controlled, system-based, without real users • Terminology mapping projects 10 IR Interactive IR Availability of corpora high low Reproducibility high low Control high low Measures medium medium Effort low high Significance medium medium Generalisability medium medium Realistic Scenario no high
  11. 11. Outlook • Integrate different recommender systems in real retrieval tasks (search sessions) • Use and evaluate recommenders for query expansion and as dynamic features in IR, in the retrieval process (AMUR project) • Develop new measures of utility of recommender systems – E.g. measure task completion rates or goal satisfaction 11
  12. 12. References • Hienert, D. & Mutschke, P. (2016). A Usefulness-based Approach for Measuring the Local and Global Effect of IIR Services. In Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval (CHIIR '16). ACM, New York, NY, USA, 153-162. http://dx.doi.org/10.1145/2854946.2854962 • Lüke, T., Schaer, P., & Mayr, P. (2012). Improving Retrieval Results with discipline-specific Query Expansion. In International Conference on Theory and Practice of Digital Libraries (TPDL 2012) (pp. 408–413). Paphos, Cyprus: Springer Berlin Heidelberg. http://doi.org/10.1007/978-3-642- 33290-6_44 • Mayr, P., & Petras, V. (2008). Cross-concordances: terminology mapping and its effectiveness for information retrieval. In 74th IFLA World Library and Information Congress. Québec, Canada: IFLA. Retrieved from http://www.ifla.org/IV/ifla74/papers/129-Mayr_Petras-en.pdf 12
  13. 13. Thank you Contact: Dr Philipp Mayr GESIS - Leibniz Institute for the Social Sciences, Germany Email: philipp.mayr@gesis.org Twitter: @philipp_mayr 13

×