Presentation of joint work at the Interactive Social Book Search Lab at the CLEF 2015 conference (Toulouse, France)

1. Hugo Huurdeman, Jaap Kamps, !
Marijn Koolen and Sanna Kumpulainen!
!
University of Amsterdam
!
!
!
Social Book Search Workshop, CLEF conference 2015
The Value of Multistage
Search Systems for Book Search

3. 1. Introduction
• Interactive Social Book Search track!
!
• Amazon / LibraryThing book data !
• focused and open search tasks!
• baseline and multistage interfaces!
!
• social-book-search.humanities.uva.nl/#/interactive
!
• Joint study 2015 iteration: [Gaede-etAl15]:
!
• 192 participants!
• 97 in baseline interface, 95 in multistage interface

4. 1. Introduction: baseline interface
• ‘standard’ search
interface
!
• single screen
• query box
• results list
• left column: facet filters
• right column: ‘book bag’
for book selections
Baseline interface

5. • The multistage interface
contains three screens
!
• Aim: support different stages
in the search process:
browse, search & review
!
• Inspired by various models
of the information seeking
process [Vakkari01,Kuhlthau03]
Multistage interface: search
Multistage interface: browse
Multistage interface: review
1. Introduction: multistage

6. 2. Our Participation
• What is the value of a multistage interface
for book search?!
!
1. influence on task duration, book
selection and interaction patterns
2. influence on user engagement

7. 2.1 Task duration
• How much time was spent
in each task and interface?
!
• Participants spend more
time in the focused task
than the open task
!
• Participants spend
significantly more time in
the multistage interface,
regardless of the task
time spent (seconds)

8. 2.2 Bookbag: overview
• How many books were
selected per task and
interface?
!
• More books in focused task
than open task (significant)
!
• Slightly more books were
selected in the multistage
interface, as compared to the
baseline (not significant)
number of selected books

9. 2.2 Bookbag: overlap between the tasks
• Only 9 participants have
overlap in the book bags
for the two tasks
(7 out of 9 just one book..)
• User may interpret some sub-
tasks of the focused task in a
similar way as the open task
(e.g. hobbies/personal interests
or fun). What is the overlap in
collected books between the
tasks?
!
!
!
!

10. 2.2 Bookbag: overlap between participants
• What degree of selected
books is the same
across participants?
!
!
!
!
!
!

11. 2.2 Bookbag: overlap between participants
• What degree of selected
books is the same
across participants?
!
• More overlap in
baseline (BL)
• Evidence for more
varied selections in
the multistage
interface (MS)
0
0.06
0.12
0.18
0.24
Open task Focused task
14%
7%
23%
7%
MSBL MSBL

12. 2.3 User actions
• Paginate used more
in multistage
• deeper
exploration
!
!
!
!
!
!
!
!
!
!
!
mean frequencies of actions

13. 2.3 User actions
• Paginate used more
in multistage
• deeper
exploration
!
• Few filters and
queries in multistage
• more options to
explore and
review results
!
!
!
!
!
mean frequencies of actions

14. 2.3 User actions
• Paginate used more
in multistage
• deeper
exploration
!
• Few filters and
queries in multistage
• more options to
explore and
review results
!
• Less/more book
metadata views:
• triggered to view
more in open
task?
mean frequencies of actions

15. 2.4 Use of interface panels: transition prob.
Focused task Open task

16. 2.4 Use of interface panels: time spent
• Time spent!
• Focused task: search
screen used for longest
duration;
• open task: browse
!
!
!
!
!
!
Time spent in each panel (sec)

17. 2.4 Use of interface panels: switches
• Time spent!
• Focused task: search
screen used for longest
duration;
• open task: browse
!
• Switches!
• Training task: participants
went “in order” through
panels”
• Other tasks: more varied
• 5 subtasks
Switches between panels
Time spent in each panel (sec)

18. 2.5 Participants’ Perceptions
• Using multidimensional User Engagement Scale [O’Brien & Toms]

19. 2.5 Participants’ Perceptions
• Using multidimensional User Engagement Scale [O’Brien-Toms15]
!
• Significant (Mann-Whitney):
• Felt Involvement: “how much fun users were having during the
interaction and how drawn in”
• Endurability: “the assessment of users' perception of success with a
task, and their willingness to use an application in future or recommend
it to others”
p=0.006p=0.041
* engagement did not significantly differ between age groups (18-25, 26-35, 35+)

20. 3 Conclusion
• Longer task duration in multistage
interface
• Different use of search features
• less queries / filters in multistage
• Not necessarily more books, but more
varied sets of books
• More user engagement!
!
• Users seem to be willing to switch
interface panels and use them for different
activity types
!
• Positive evidence ‘multistage’ approach

21. Discussion
• Influence personal differences
participants
• motivation, location, language skills,
education, etc.
• users who frequently search for books in
(online) bookstores?
!
• Task properties
• open vs. focused task (5 subtasks),
• possible to differentiate more easily?
• task order
!
• From narrative to interactions

22. References
• [Gaede-etAl15] M. Gaede, M. Hall, H. Huurdeman, J. Kamps, M. Koolen, M. Skov, E.
Toms, D. Walsh. Overview of the SBS 2015 Interactive Track. CEUR-WS. 2015.
• [Hall-Toms15] M. Hall and E. Toms. Building a common framework for iir evaluation. In
P. Forner, H. Mueller, R. Paredes, P. Rosso, and B. Stein, editors, Information Access
Evaluation. Multilinguality, Multimodality, and Visualization, volume 8138 of Lecture
Notes in Computer Science, p. 17–28. Springer Berlin Heidelberg, 2013.
• [Huurdeman-etAl15] H. Huurdeman, J. Kamps, M. Koolen, S. Kumpulainen. The Value
of Multistage Search Systems for Book Search. CEUR-WS. 2015.
• [Koolen-etAl15] M. Koolen, T. Bogers, M. Gaede, M. Hall, H. Huurdeman, J. Kamps,
M. Skov, E. Toms, and D. Walsh. Overview of the CLEF 2015 Social Book Search Lab.
In CLEF'15: Experimental IR meets Multilinguality, Multimodality, and Interaction,
volume 9283 of LNCS. Springer (forthcoming).
• [Kuhlthau04] C. Kuhlthau. Seeking Meaning: A Process Approach to Library and
Information Services. Libraries Unlimited, Westport, Conn. 2004.
• [O’Brien-Toms15] H. O’Brien and E. Toms. The development and evaluation of a
survey to measure user engagement. JASIST, 61(1):50–69, 2010.
• [Vakkari01] P. Vakkari. A theory of the task-based information retrieval process: a
summary and generalisation of a longitudinal study. Journal of Documentation, 57:44–
60, 2001.

23. Hugo Huurdeman, Jaap Kamps, !
Marijn Koolen and Sanna Kumpulainen!
!
University of Amsterdam
!
!
!
Social Book Search Workshop, CLEF conference 2015
The Value of Multistage
Search Systems for Book Search