Doctoral session at ISI 2017 – 15th International Symposium on Information Science 14th March, Humboldt-Universität zu Berlin, Germany

1. RELEVANCE CLUES
Developing an experimental design to examine the criteria
behind relevance judgments
Christiane Behnert
Hamburg University of Applied Sciences, Germany
Department of Information
ISI 2017 – 15th International Symposium on Information Science
14th March, Humboldt-Universität zu Berlin, Germany

4. FACULTY OF DESIGN, MEDIA & INFORMATION
Department of Information
1. Research goal
2. Related research
1. Relevance criteria
2. Document representations
3. Experimental research design
1. Reasons
2. Requirements
4. Next steps
5. References
OUTLINE
4

5. FACULTY OF DESIGN, MEDIA & INFORMATION
Department of Information
What are the clues and criteria
by which users judge
a search result’s relevance?
1. RESEARCH GOAL
5

6. FACULTY OF DESIGN, MEDIA & INFORMATION
Department of Information
2.1. Relevance criteria [Mizzaro, 1997; Saracevic, 2016]
Topicality
Recency
Clarity
Availability
Novelty
…
2. RELATED RESEARCH
6

7. FACULTY OF DESIGN, MEDIA & INFORMATION
Department of Information
2.2. Document representations
Pre-access vs. post-access criteria [Watson, 2014]
– Surrogates vs. full-text documents
Popularity data in academia [Plassmeier et al., 2015]
– Number of citations (e.g., Google Scholar)
– Number of downloads (e.g., ACM Digital library)
– Circulation counts
– Number of copies in a library
– …
2. RELATED RESEARCH
7

8. FACULTY OF DESIGN, MEDIA & INFORMATION
Department of Information
3.1. Reasons
Knowledge about actual behaviour [Kelly, 2009; Kelly & Cresenzi, 2016]
Causal conclusions [Sedlmeier & Renkewitz, 2007]
Stimulus effect
(independent variable dependent variable)
Covariance
Temporal precedence
Exclusion of an alternative explanation
(confounding variables)
3. EXPERIMENTAL RESEARCH DESIGN
8

9. FACULTY OF DESIGN, MEDIA & INFORMATION
Department of Information
3.2. Requirements (1/2)
3. EXPERIMENTAL RESEARCH DESIGN
9

10. FACULTY OF DESIGN, MEDIA & INFORMATION
Department of Information
3.2. Requirements (2/2) [Sedlmeier & Renkewitz, 2007]
Manipulation (potential relevance clues)
 Changes in variable expressions
Control (possible confounding variables)
 Randomisation
Within-subjects design
 Reduces sample size
Multifactorial design
 Combination of variables
3. EXPERIMENTAL RESEARCH DESIGN
10

11. FACULTY OF DESIGN, MEDIA & INFORMATION
Department of Information
User model
Hypotheses
Experiments
Results
Adjust user model
4. NEXT RESEARCH STEPS
11

12. FACULTY OF DESIGN, MEDIA & INFORMATION
Department of Information
Kelly, D. (2009). Methods for evaluating interactive information retrieval systems with users. Foundations and
Trends® in Information Retrieval, 3(1—2).
Kelly, D., & Cresenzi, A. (2016). From design to analysis: Conducting controlled laboratory experiments with
users. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in
Information Retrieval - SIGIR ’16 (1207–1210). New York, New York, USA: ACM Press.
Mizzaro, S. (1997). Relevance: The whole history. Journal of the American Society for Information Science,
48(9), 810–832.
Plassmeier, K., Borst, T., Behnert, C., & Lewandowski, D. (2015). Evaluating popularity data for relevance
ranking in library information systems. In Proceedings of the 78th ASIS&T Annual Meeting (Vol. 51). Retrieved
from https://www.asist.org/files/meetings/am15/proceedings/submissions/posters/270poster.pdf
Saracevic, T. (2016). The notion of relevance in Information Science: Everybody knows what relevance is. But,
what is it really? (G. Marchionini, Ed.), Synthesis Lectures on Information Concepts, Retrieval, and Services; 50.
Morgan & Claypool.
Sedlmeier, P., & Renkewitz, F. (2007). Forschungsmethoden und Statistik in der Psychologie. München ;
Boston [u.a.]: Pearson Studium, 123-180.
Watson, C. (2014). An exploratory study of secondary students’ judgments of the relevance and reliability of
information. Journal of the Association for Information Science and Technology, 65(7), 1385–1408.
5. REFERENCES
12

13. Thank you very much!
I appreciate your feedback!
Christiane Behnert, M.A.
Hamburg University of Applied Sciences, Germany
christiane.behnert@haw-hamburg.de
http://searchstudies.org/christiane-behnert/