This document discusses evaluating search systems for digital cultural heritage collections. It argues that evaluation needs to consider more than just search results and look at the entire user experience and system from multiple perspectives. Effective evaluation requires combining both system-oriented and user-oriented methods. It also needs to account for how users differ and the contexts of their information needs and goals. The document outlines challenges in evaluating complex systems that go beyond basic search to include features like recommendations, visualizations and taxonomies. It provides an example evaluation framework used for the PATHS system that supported exploratory search and sense-making across various system components and user tests.

1. Evaluating Search in Digital Cultural
Heritage: Thinking Outside the
(Search) Box
Paul Clough
Information School, University of Sheffield, UK
EuropeanaTech Conference 2018, Rotterdam, 15‐16 May 2018 Page 1

2. Evaluating search
• Evaluation refers to assessing the worth or value of
something
– Essential activity in developing and maintaining search systems
• Most evaluation activity in search focuses on assessing
the quality of search results (system/algorithmic‐oriented
evaluation)
• But often we want to measure aspects of retrieval
performance beyond system effectiveness (user‐oriented
evaluation)
• In reality we need multiple approaches to evaluate
search systems that go beyond the search box and
evaluate the ‘system’ more widely and from multiple
stakeholder perspectives
EuropeanaTech Conference 2018, Rotterdam, 15‐16 May 2018 Page 2

3. Putting search and discovery in
context
User differences
• Age
• Gender
• Technical expertise
• Domain knowledge
• Cognitive styles
• …
Plus wider context
• Task and need
• Goal
• Device
• Location
• Time
• …
Will impact on
• Information seeking
behaviour
• User preferences
• Notions of success
• …
EuropeanaTech Conference 2018, Rotterdam, 15‐16 May 2018 Page 3

4. Kelly, D. (2009). Methods for evaluating interactive information retrieval systems with users.
Foundations and Trends in Information Retrieval, 3(1‐2), 1‐224. DOI: 10.1561/1500000012.
EuropeanaTech Conference 2018, Rotterdam, 15‐16 May 2018 Page 4

5. Criteria: what to measure
In the 1960’s Cyril
Cleverdon suggested the
following
•Coverage
•Time lag
•Presentation
•Effort
•Recall
•Precision
In 2016 Ryen White suggests
the following
– Search outcomes
• Relevance
(precision, effort,
etc.)
• Novelty / diversity
• Success
• Satisfaction
• Support for
creativity
• Adoption and
retention
– Search process
• Learning
• Efficiency
• Cognitive load
• Serendipity
• Enjoyment
• Frustration
• Engagement
EuropeanaTech Conference 2018, Rotterdam, 15‐16 May 2018 Page 5

6. Clough, P. (2015) Evaluation: Thinking Outside the (Search) Box, In Proceedings of the Forum for
Information Retrieval Evaluation (FIRE '14), Prasenjit Majumder, Mandar Mitra, Madhulika Agrawal, and
Parth Mehta (Eds.), ACM, New York, NY, USA, pp. 1‐9.
http://ir.shef.ac.uk/cloughie/papers/Clough_FIRE2014.pdf
EuropeanaTech Conference 2018, Rotterdam, 15‐16 May 2018 Page 6

7. Beyond IR evaluation
• Evaluation typically focuses on search box but search‐based
applications are typically rich in features to support
information searching and seeking
• Many search applications involve multiple components
– e.g. visualisations, recommendations, taxonomies, facets
• In practice evaluation will take place during system
development (formative and summative) using combinations
of system‐ and user‐oriented methods
• How do we evaluate components beyond the search box from
a system and user‐oriented perspective?
EuropeanaTech Conference 2018, Rotterdam, 15‐16 May 2018 Page 7

8. The PATHS system
• PATHS system aimed to support users’ information seeking and the
‘information journey’
– Recognising an information need
– Acquiring information
– Interpreting and validating this information
– Using the information
• PATHS also aimed to support exploration (exploratory search) and
help users to make sense of concepts and items in a digital library
collection (sense‐making)
– Providing functionalities to overview collections, aid
interpretation of information, use information to create paths
Clough, P. (2015) Supporting Exploration and Use of Digital Cultural Heritage Materials,
EuropeanaTech Insight, Issue 4. http://ir.shef.ac.uk/cloughie/publications.html
EuropeanaTech Conference 2018, Rotterdam, 15‐16 May 2018 Page 8

9. EuropeanaTech Conference 2018, Rotterdam, 15‐16 May 2018 Page 9

10. Evaluation activities
• System development followed classic user‐centred approach
of requirements gathering  prototyping  evaluate [repeat]
• Evaluation of components
– Search box
– Recommender systems
– Visualisations
– Related/similar items
– Subject hierarchies and facets
• System architecture/infrastructure testing
• Evaluation of user interface designs
• Evaluation of the integrated prototype
– Controlled lab‐based user testing
– Field trials
Evaluations carried out by
researchers to select best
algorithms; had to learn from
domains beyond search
Evaluations carried
out by software
developers
Evaluations carried
out by UI designers
Evaluations carried
out by ‘end users’
EuropeanaTech Conference 2018, Rotterdam, 15‐16 May 2018 Page 10

11. EuropeanaTech Conference 2018, Rotterdam, 15‐16 May 2018 Page 11

12. Europeana: evaluating search
EuropeanaTech Conference 2018, Rotterdam, 15‐16 May 2018 Page 12

13. Possible framework for project
evaluation activities as a whole
EuropeanaTech Conference 2018, Rotterdam, 15‐16 May 2018 Page 13

14. Issues and challenges
• Many issues and challenges facing evaluation when we think
outside the search box, including
– Combining user‐ and system‐oriented approaches (mixed methods)
– Understanding the relationship between evaluation criteria (and
associated measures)
– Sharing evaluation practices between domains and disciplines
– Thinking beyond ad hoc search tasks
– Combining the evaluation results (e.g., does the whole=sum of parts?)
– Evaluating whole‐page relevance
– ….
• What makes a search system successful?
• It depends ‐ on the stakeholder, the end user and their context, and
much more….
EuropeanaTech Conference 2018, Rotterdam, 15‐16 May 2018 Page 14

15. Thank you
Any questions?
EuropeanaTech Conference 2018, Rotterdam, 15‐16 May 2018 Page 15