1. The Multi-Stage Experience
The Simulated Work Task Approach to
Studying Information Seeking Stages
Hugo Huurdeman Jaap Kamps Max L. Wilson
@timelessfuture @jkamps @gingdottwitt
Original CHIIR ’16 paper: bit.ly/ActivePassiveUtility
Experience paper: bit.ly/ChiirExperiencePaper
University of Amsterdam / University of Nottingham

2. Original CHIIR ’16 paper: bit.ly/ActivePassiveUtility

3. Original CHIIR ’16 paper: bit.ly/ActivePassiveUtility
Experience paper: bit.ly/ChiirExperiencePaper

4. image: Endless Time by davidmontse

5. image: https://www.artofplay.com/blogs/articles/fun-with-the-droste-effect

6. • Models of information seeking describe
fundamentally different macro-level stages
CHIIR ’16 paper: Premise

7. • Models of information seeking describe
fundamentally different macro-level stages
CHIIR ’16 paper: Premise
+uncertainty-
feelings
thoughts
actions
vague focused
seeking general
information (exploring)
seeking pertinent
information (documenting)
uncertainty optimism confusion clarity confidence (dis)satisfaction
doubt direction
FormulationInitiation Selection Exploration Collection Presentation
Prefocus Focus Postfocus

8. • Models of information seeking describe
fundamentally different macro-level stages
CHIIR ’16 paper: Premise
+uncertainty-
feelings
thoughts
actions
vague focused
seeking general
information (exploring)
seeking pertinent
information (documenting)
uncertainty optimism confusion clarity confidence (dis)satisfaction
doubt direction
FormulationInitiation Selection Exploration Collection Presentation
Prefocus Focus Postfocus
[Kuhlthau04,Vakkari01]

9. • Very valuable, but: difficult to translate to
concrete search system functionalitymoves & tactics
task progress
inform
pre-focus focus post-focus
Information seeking models
? ?
search engine functionality

11. a

12. a
Wealth of potential supportive features

13. a
Wealth of potential supportive features
but unknown role in process

14. a
Wealth of potential supportive features
but unknown role in process
(too many = overload)

15. • Valuable, but: difficult to translate to concrete
search systems functionality
Information seeking models
? ?
• We need a more accurate understanding
of what functionality is useful when
task progress
inform
pre-focus focus post-focus
search engine functionality

16. Introduction
•Our study @CHIIR16 investigated the utility of
various SUI features at different macro-level
stages
•Via an explicit multi-stage design

17. Multi-stage approach?

18. Multi-stage approach?
• Kuhlthau’s and Vakkari’s models had been
conceived based on longitudinal settings (Kuhlthau,
04;Vakkari,01)
• monitor process at various time points

19. Multi-stage approach?
• Kuhlthau’s and Vakkari’s models had been
conceived based on longitudinal settings (Kuhlthau,
04;Vakkari,01)
• monitor process at various time points
Versus

20. Multi-stage approach?
• Kuhlthau’s and Vakkari’s models had been
conceived based on longitudinal settings (Kuhlthau,
04;Vakkari,01)
• monitor process at various time points
Versus
• Studies looking at stages in singular sessions
(Diriye et al,10; Niu & Kelly, 14; Huurdeman & Kamps,14, etc.)
• “these studies can be better characterized as focusing
on phases of search or temporal search segments”
(Niu & Kelly, 14)

21. Finding a ‘middle point’
True
longitudinal
study
Stages as
temporal
phases of
singular
sessions
Potentially use many
data collection instruments
Limited number of
data collection instruments

22. Finding a ‘middle point’
True
longitudinal
study
Stages as
temporal
phases of
singular
sessions
Multiple
sessions
representing
stages
Potentially use many
data collection instruments
Limited number of
data collection instruments

23. Finding a ‘middle point’
• Multi-stage Simulated Work Task Approach
• Single simulated work task, multiple subtasks
• “Multi-stage experience”
True
longitudinal
study
Stages as
temporal
phases of
singular
sessions
Multiple
sessions
representing
stages
Potentially use many
data collection instruments
Limited number of
data collection instruments

24. Further outline
• 1. Terminology
• 2. Methodology
• 3. Re-use of previous resources
• 4. Potential for re-use of our approach
• 5. Further discussion points

25. 1. Terminology

26. Terminology: Information Seeking Models
• Information seeking modeled in a multitude of ways:
• as behavioral patterns (Ellis)
• as nonlinear activities (Foster)
• as temporal stages (Kuhlthau), ..
• Our main focus:
• temporally based IS models
• Kuhlthau [1991] (Vakkari [2001])
• cognitively complex (work) tasks
• involving learning & construction
• Using definitions Wilson (1999),
Ingwersen & Jarvelin (2005), Byström & Järvelin (1995)
information
search
information
seeking
information
behavior
[Wilson99]

28. SUI features categorized using Wilson’s
(2011) framework

30. Control

31. Control
Input

32. Control
Input
Informational

33. Control
Input
PersonalizableInformational

34. 2. Methodology

35. Methodology
• User study (26 participants + 2 pilot participants)
•24 analyzed
• Undergrads Univ. of Nottingham (6 F, 18 M / 18-25y)
• Experimental SUI resembling common Search Engine
• Within-participants
• Task stage independent variable
• Task design: explicit multistage approach

36. Methodology: Multistage Task Design
sim. work task: writing essay
subtask subtask subtask
prepare list of
3 topics
choose topic;
formulate specific
question
find and select
additional
pages to cite
~15 minutes ~15 minutes ~15 minutes
initiation
topic selection
exploration
focus formulation
collecting
presenting
pre-focus focus formulation post-focus

37. Methodology: Multistage Task Design
sim. work task: writing essay
subtask subtask subtask
prepare list of
3 topics
choose topic;
formulate specific
question
find and select
additional
pages to cite
~15 minutes ~15 minutes ~15 minutes
initiation
topic selection
exploration
focus formulation
collecting
presenting
Discussed with
information seeking
specialists —>
pre-focus focus formulation post-focus

38. Methodology: Multistage Task Design
sim. work task: writing essay
subtask subtask subtask
prepare list of
3 topics
choose topic;
formulate specific
question
find and select
additional
pages to cite
~15 minutes ~15 minutes ~15 minutes
initiation
topic selection
exploration
focus formulation
collecting
presenting
Tested with pilot
participants —>
Discussed with
information seeking
specialists —>
pre-focus focus formulation post-focus

39. Methodology: Multistage Task Design
sim. work task: writing essay
subtask subtask subtask
prepare list of
3 topics
choose topic;
formulate specific
question
find and select
additional
pages to cite
~15 minutes ~15 minutes ~15 minutes
initiation
topic selection
exploration
focus formulation
collecting
presenting
Tested with pilot
participants —>
Discussed with
information seeking
specialists —>
Based on Vakkari
/ Kuhlthau theory —>
pre-focus focus formulation post-focus

40. Methodology: Multistage Task Design
sim. work task: writing essay
subtask subtask subtask
prepare list of
3 topics
choose topic;
formulate specific
question
find and select
additional
pages to cite
~15 minutes ~15 minutes ~15 minutes
initiation
topic selection
exploration
focus formulation
collecting
presenting
Topics
• Autonomous Vehicles (AV)
• Virtual Reality (VR)
Tested with pilot
participants —>
Discussed with
information seeking
specialists —>
Based on Vakkari
/ Kuhlthau theory —>
pre-focus focus formulation post-focus

41. Methodology: Multistage Task Design
sim. work task: writing essay
subtask subtask subtask
prepare list of
3 topics
choose topic;
formulate specific
question
find and select
additional
pages to cite
~15 minutes ~15 minutes ~15 minutes
initiation
topic selection
exploration
focus formulation
collecting
presenting
Topics
• Autonomous Vehicles (AV)
• Virtual Reality (VR)
Discussions with
teaching staff —>
Tested with pilot
participants —>
Discussed with
information seeking
specialists —>
Based on Vakkari
/ Kuhlthau theory —>
pre-focus focus formulation post-focus

42. Methodology: Multistage Task Design
sim. work task: writing essay
subtask subtask subtask
prepare list of
3 topics
choose topic;
formulate specific
question
find and select
additional
pages to cite
~15 minutes ~15 minutes ~15 minutes
initiation
topic selection
exploration
focus formulation
collecting
presenting
Topics
• Autonomous Vehicles (AV)
• Virtual Reality (VR)
Discussions with
teaching staff —>
Tested with pilot
participants —>
Discussed with
information seeking
specialists —>
Based on Vakkari
/ Kuhlthau theory —>
pre-focus focus formulation post-focus
Standard amount
+ additional incentive —>

43. Methodology: Simplified Protocol
Training task
Pre-
Questionnaire
Topic
Assignment
Introduction
system
Task
Post-task
Questionnaire
3x
Post-experiment
questionnaire
Debriefing
interview

44. Methodology: Simplified Protocol
Training task
Pre-
Questionnaire
Topic
Assignment
Introduction
system
Task
Post-task
Questionnaire
3x
Post-experiment
questionnaire
Debriefing
interview
<— mitigate familiarity effects

45. Tutorial slides Pia Borlund (2012)

48. Task & task stage validation
• Validating the 2 topics (statistically)
• Various validation questions within the process
(post-task, 3x)
1. Randomized list of activities derived from Kuhlthau’s
model
2. Reported type of information sought (Kuhlthau, Todd)
3. Randomized word list which could represent users
feelings / state of mind (Kuhlthau, Todd)

49. Methodology: data collection instruments

50. Methodology: data collection instruments

51. Methodology: data collection instruments

52. Methodology: data collection instruments

53. Methodology: data collection instruments

54. Methodology: data collection instruments

55. 3. Re-use previous
resources

56. Task descriptions
Initial inspiration:
• Kuhlthau’s book, Vakkari’s articles
• Previous literature survey
(Huurdeman&Kamps,2014)
• Research process / IL literature
• RepAST repository

57. Task descriptions
Initial inspiration:
• Kuhlthau’s book, Vakkari’s articles
• Previous literature survey
(Huurdeman&Kamps,2014)
• Research process / IL literature
• RepAST repository
Kumar, 2010

58. Task descriptions
Some general inspiration: Liu & Belkin, 2015

59. Task descriptions
Kules & Shneiderman, 2009
Some general inspiration: Liu & Belkin, 2015

62. • Experimental system: SearchAssist
• Results, Query Corrections, Query
Suggestions: Bing Web API
• Category Filters, categories: DMOZ
• Tag cloud: Jquery plugin
• Task instructions: Google Doc
• Categorization and analysis:
• Max Wilson’s framework of SUI features
[Wilson11]

63. PyGaze,
PyTribe

64. PyGaze,
PyTribe

65. PyGaze,
PyTribe

66. PyGaze,
PyTribe

67. PyGaze,
PyTribe

68. PyGaze,
PyTribe

69. PyGaze,
PyTribe

70. 4. Potential for re-use

71. Re-use of research design: so far
• Two papers:
• Hoeber et al. (2017)
• Gaikwad and Hoeber (2019)

72. Re-use (1)
(Hoeber et al, 2017)

73. Re-use (1)
(Hoeber et al, 2017)

74. Re-use (1)
(Hoeber et al, 2017)

75. Re-use (1)
(Hoeber et al, 2017)

76. Re-use (2)
• Gaikwad and Hoeber (2019) used
Vakkari’s model
• as “a design guide, and as a
mechanism for controlling the
laboratory-based evaluation.”
• participants
• explored images (pre-focus)
• selected images (focus formulation)
• organized images (post-focus)
• Task descriptions: holiday plans,
food blogging and self-selected
tasks.

77. 5. Discussion points (1/3)
• Lack of space to document all aspects of our study
• Some in my PhD thesis (2018)
• (now: CHIIR extra pages)
• Typical lack of time within research process and
publication cycle
• at the time, publishing source code SearchAssist, but
not contextual documents or analysis scripts
• A restrictive consent form …

78. • Tension between flexibility in terms of research
questions, and the possibility to re-use
standardized systems and approaches
• Use of search API vs standard collection
• Remuneration of participants
5. Discussion points (2/3)

79. • Tension between flexibility in terms of research
questions, and the possibility to re-use
standardized systems and approaches
• Use of search API vs standard collection
• Remuneration of participants
5. Discussion points (2/3)

80. Discussion points (3/3)
• Maintenance: just four years after our study,
components of system have changed (e.g. Bing
API configurations)
• Persistence: various URLs of used resources
now only available in the Internet Archive

81. References
• For all citations see: bit.ly/ChiirExperiencePaper

82. The Multi-Stage Experience
The Simulated Work Task Approach to
Studying Information Seeking Stages
Hugo Huurdeman Jaap Kamps Max L. Wilson
@timelessfuture @jkamps @gingdottwitt
Original CHIIR ’16 paper: bit.ly/ActivePassiveUtility
Experience paper: bit.ly/ChiirExperiencePaper
University of Amsterdam / University of Nottingham