Book search for information needs that go beyond standard bibliographic data is far from a solved problem. Such complex information needs often cover a combination of different aspects, such as specific genres or plot elements, engagement or novelty. By design, subject information in controlled vocabularies is not always adequate in covering such complex needs, and social tags have been proposed as an alternative. In this paper we present a large-scale empirical comparison and in-depth analysis of the value of controlled vocabularies and tags for book retrieval using a test collection of over 2 million book records and over 330 real-world book information needs. We find that while tags and controlled vocabulary terms provide complementary performance, tags perform better overall. However, this is not due to a popularity effect; instead, tags are better at matching the language of regular users. Finally, we perform a detailed failure analysis and show, using tags and controlled vocabulary terms, that some request types are inherently more diffcult to solve than others. (Paper presentation @ iConference 2017, Wuhan, China)

1. AN IN-DEPTH ANALYSIS OF TAGS AND CONTROLLED
METADATA FOR BOOK SEARCH
TOINE BOGERS
VIVIEN PETRAS
MARCH 23, 2017iCONFERENCE 2017

2. OUTLINE
▸ Introduction
▸ Methodology & Experimental Setup
▸ Analysis
– Tags vs. Controlled Vocabularies
– Book Search Requests
– Failure Analysis
▸ Conclusions & Future Work
2

3. INTRODUCTION

4. MOTIVATION
▸ Readers often struggle with existing systems (i.e., library
catalogs, Amazon, eBook sellers) to discover new books
– Information needs are contextual, personal & complex
– Book metadata does not contain the necessary information
4

5. EARLIER WORK
▸ iConference 2015
– Tags outperform controlled vocabularies for search, but
sometimes controlled vocabularies are better.
– Controlled vocabularies contains more unique terms, tags
more repetition of terms.
▸ Why?
– Terminology
– Popularity / frequency
– Type of request
5

6. STUDY OBJECTIVES
▸ Why are tags better than controlled vocabularies for book
search?
– Which types of book search requests are better addressed
using tags and which using CV?
– Which book search requests fail completely and what
characterizes such requests?
6

7. METHODOLOGY&
EXPERIMENTAL SETUP

8. EXPERIMENTAL SETUP
▸ Controlled Vocabulary content (CV)
– DDC class labels
– Subjects
– Geographic names
– Category labels
– LCSH terms
▸ Tags
– Each tag occurs as many times as it has been assigned by
the users
▸ Unique tags
– Each tag occurs only once
8

9. AMAZON/LIBRARYTHING COLLECTION 9
Tags
Tags
Controlled Vocabulary Content (CV)
DDC class labels
subjects
geographic names
category labels
LCSH terms
Unique Tags
Unique Tags per record

10. ANNOTATED LT TOPIC
10
Recommended
books
Topic title
Narrative

11. EXPERIMENTAL SETUP
▸ Amazon / LibraryThing collection of book records
– 2 million records
▸ LibraryThing forum topics for search requests
– 334 search requests for testing
▸ Relevance judgements
– Recommendations from LT members with graded relevance scoring
(highest relevance if book is added by searcher)
▸ Evaluation metric
– Normalized Discounted Cumulated Gain (NDCG@10)
▸ IR system
– Indri 5.4 toolkit
10

12. ANALYSIS

13. TAGS vs. CONTROLLED VOCABULARIES
▸ Question 1: Is there a difference in performance between
CV and Tags in retrieval?
▸ Answer
– Tags perform significantly
better than CV
– The combination of both
results in even better
performance than just for
tags, but not significantly so
– Losing tag frequency
information helps rather than
hurts performance (also not
significantly)
12

14. TAGS vs. CONTROLLED VOCABULARIES
▸ Question 2: Do tags outperform CV because of the so-
called popularity effect?
▸ Answer
– No, there does not seem to be a popularity effect
– Types = unique words in a record
– Tokens = all instances of words in a record
13

15. TAGS vs. CONTROLLED VOCABULARIES
▸ Question 3: Do Tags and
CV complement or cancel
each other out?
▸ Answer
– Tags and CV
complement each
other: they are
successful on different
sets of requests
– But most zero-difference
requests (74.0%)
actually fail completely!
When and why?
14

16. REQUESTS – RELEVANCE ASPECTS
▸ What makes a suggested book relevant to the user?
– Distinguish between eight relevance aspects (Reuter, 2007;
Koolen et al., 2015)
16

17. REQUESTS – RELEVANCE ASPECTS
Aspect Description
% of requests
(N = 87)
Accessibility Language, length, or level of difficulty of a book 9.2 %
Content Topic, plot, genre, style, or comprehensiveness 79.3 %
Engagement
Fit a certain mood or interest, are considered high
quality, or provide a certain reading experience
25.3 %
Familiarity
Similar to known books or related to a previous
experience
47.1 %
Known-item
The user is trying to identify a known book, but cannot
remember the metadata that would locate it
12.6 %
Metadata
With a certain title or by a certain author or publisher, in
a particular format, or certain year
23.0 %
Novelty Unusual or quirky, or containing novel content 3.4 %
Socio-cultural
Related to the user's socio-cultural background or
values; popular or obscure
13.8 %
16

18. REQUESTS – RELEVANCE ASPECTS
▸ Question 4: What types of book requests are best served
by the Unique tags and CV collections?
▸ Answer
– CV terms show a tendency to work best for requests that
touch upon aspects of engagement
– Other requests are best served by Unique tags
17

19. REQUESTS – RELEVANCE ASPECTS
0,00 0,10 0,20 0,30 0,40 0,50 0,60 0,70 0,80 0,90 1,00
Socio-cultural
(N = 10)
Novelty
(N = 2)
Metadata
(N = 17)
Known-item
(N = 11)
Familiarity
(N = 36)
Engagement
(N = 21)
Content
(N = 63)
Accessibility
(N = 7)
Unique tags
CV
0.0 0.20.1 0.40.3 0.60.5 0.80.7 1.00.9
Socio-cultural
(N = 10)
0.1127
0.0428
Novelty
(N = 2)
0.5304
0.0000
Metadata
(N = 17)
0.2454
0.1259
Known-item
(N = 11)
0.3593
0.1818
Familiarity
(N = 36)
0.1833
0.0701
Engagement
(N = 21)
0.1121
0.1425
Content
(N = 63)
0.1965
0.0821
Accessibility
(N = 7)
0.1235
0.0749
Performance grouped by relevance aspect
NDCG@10
18

20. REQUESTS – TYPE OF BOOK
▸ Question 5: What types of book requests (fiction or non-
fiction) are best served by Unique tags or CV?
▸ Answer
– Unique tags work significantly better for fiction
– CV work better for non-fiction (but not significantly so)
19

21. FAILURE ANALYSIS
▸ Question 6: Do failed book search requests fail because of
data sparsity, a lower recall base, or a lack of examples?
▸ Answer
– Neither sparsity nor the size of the recall base are the
reason for retrieval failure
– The number of examples provided by the requester has
significant positive influence on performance
(N = 247)
(N = 87)
(N = 334)
20

22. FAILURE ANALYSIS
▸ Question 7: Do book search requests fail because of their
relevance aspects?
▸ Answer
– No, relevance
aspects are
distributed equally
for successful &
failed requests
– Only Accessibility-
and Metadata-
related search
requests seem to
fail more often
21

23. FAILURE ANALYSIS
▸ Question 8: Does the type of book that is being requested
(fiction vs. non-fiction) have an influence on whether
requests succeed or fail?
▸ Answer
– Requests for works of fiction fail significantly more often
22

24. CONCLUSIONS &
FUTURE WORK

25. FINDINGS
▸ Tags outperform CV...
– ...probably because their terminology is closer to the user‘s
language (not because of the popularity effect)
▸ Sometimes CV are better, for example, for non-fiction books...
– ...whereas tags are better for fiction and for content-related,
familiarity or known-item searches
▸ We believe that tags are simply better able to match the user‘s
language when looking for books
– Although they are still not that great at it!
– Book search is still hard, especially for fiction books
25

26. OPEN QUESTIONS
▸ How can book metadata be adapted to be closer to the
vocabulary used in real-world book search requests?
▸ What other aspects (besides type of requested book or
relevance aspect of search request) contribute to request
difficulty?
▸ Our question to you:
– What other questions can we ask of this data?
26

27. QUESTIONS?
Paper URL: http://bit.ly/iconf2017