Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
The Multi-Stage Experience
The Simulated Work Task Approach to
Studying Information Seeking Stages
Hugo Huurdeman Jaap Kam...
Original CHIIR ’16 paper: bit.ly/ActivePassiveUtility
Original CHIIR ’16 paper: bit.ly/ActivePassiveUtility
Experience paper: bit.ly/ChiirExperiencePaper
image: Endless Time by davidmontse
image: https://www.artofplay.com/blogs/articles/fun-with-the-droste-effect
• Models of information seeking describe
fundamentally different macro-level stages
CHIIR ’16 paper: Premise
• Models of information seeking describe
fundamentally different macro-level stages
CHIIR ’16 paper: Premise
+uncertainty-...
• Models of information seeking describe
fundamentally different macro-level stages
CHIIR ’16 paper: Premise
+uncertainty-...
• Very valuable, but: difficult to translate to
concrete search system functionalitymoves & tactics
task progress
inform
pr...
a
a
Wealth of potential supportive features
a
Wealth of potential supportive features
but unknown role in process
a
Wealth of potential supportive features
but unknown role in process
(too many = overload)
• Valuable, but: difficult to translate to concrete
search systems functionality
Information seeking models
? ?
• We need a...
Introduction
•Our study @CHIIR16 investigated the utility of
various SUI features at different macro-level
stages
•Via an ...
Multi-stage approach?
Multi-stage approach?
• Kuhlthau’s and Vakkari’s models had been
conceived based on longitudinal settings (Kuhlthau,
04;Va...
Multi-stage approach?
• Kuhlthau’s and Vakkari’s models had been
conceived based on longitudinal settings (Kuhlthau,
04;Va...
Multi-stage approach?
• Kuhlthau’s and Vakkari’s models had been
conceived based on longitudinal settings (Kuhlthau,
04;Va...
Finding a ‘middle point’
True
longitudinal
study
Stages as
temporal
phases of
singular
sessions
Potentially use many 

dat...
Finding a ‘middle point’
True
longitudinal
study
Stages as
temporal
phases of
singular
sessions
Multiple
sessions
represen...
Finding a ‘middle point’
• Multi-stage Simulated Work Task Approach
• Single simulated work task, multiple subtasks
• “Mul...
Further outline
• 1. Terminology

• 2. Methodology

• 3. Re-use of previous resources

• 4. Potential for re-use of our ap...
1. Terminology
Terminology: Information Seeking Models
• Information seeking modeled in a multitude of ways:
• as behavioral patterns (El...
SUI features categorized using Wilson’s
(2011) framework
Control
Control
Input
Control
Input
Informational
Control
Input
PersonalizableInformational
2. Methodology
Methodology
• User study (26 participants + 2 pilot participants)
•24 analyzed
• Undergrads Univ. of Nottingham (6 F, 18 M...
Methodology: Multistage Task Design
sim. work task: writing essay
subtask subtask subtask
prepare list of 

3 topics
choos...
Methodology: Multistage Task Design
sim. work task: writing essay
subtask subtask subtask
prepare list of 

3 topics
choos...
Methodology: Multistage Task Design
sim. work task: writing essay
subtask subtask subtask
prepare list of 

3 topics
choos...
Methodology: Multistage Task Design
sim. work task: writing essay
subtask subtask subtask
prepare list of 

3 topics
choos...
Methodology: Multistage Task Design
sim. work task: writing essay
subtask subtask subtask
prepare list of 

3 topics
choos...
Methodology: Multistage Task Design
sim. work task: writing essay
subtask subtask subtask
prepare list of 

3 topics
choos...
Methodology: Multistage Task Design
sim. work task: writing essay
subtask subtask subtask
prepare list of 

3 topics
choos...
Methodology: Simplified Protocol
Training task
Pre-
Questionnaire
Topic
Assignment
Introduction
system
Task
Post-task
Quest...
Methodology: Simplified Protocol
Training task
Pre-
Questionnaire
Topic
Assignment
Introduction
system
Task
Post-task
Quest...
Tutorial slides Pia Borlund (2012)
Task & task stage validation
• Validating the 2 topics (statistically)
• Various validation questions within the process
(...
Methodology: data collection instruments
Methodology: data collection instruments
Methodology: data collection instruments
Methodology: data collection instruments
Methodology: data collection instruments
Methodology: data collection instruments
3. Re-use previous
resources
Task descriptions
Initial inspiration:
• Kuhlthau’s book, Vakkari’s articles

• Previous literature survey 

(Huurdeman&Ka...
Task descriptions
Initial inspiration:
• Kuhlthau’s book, Vakkari’s articles

• Previous literature survey 

(Huurdeman&Ka...
Task descriptions
Some general inspiration: Liu & Belkin, 2015
Task descriptions
Kules & Shneiderman, 2009
Some general inspiration: Liu & Belkin, 2015
• Experimental system: SearchAssist
• Results, Query Corrections, Query
Suggestions: Bing Web API
• Category Filters, cate...
PyGaze,
PyTribe
PyGaze,
PyTribe
PyGaze,
PyTribe
PyGaze,
PyTribe
PyGaze,
PyTribe
PyGaze,
PyTribe
PyGaze,
PyTribe
4. Potential for re-use
Re-use of research design: so far
• Two papers:
• Hoeber et al. (2017)
• Gaikwad and Hoeber (2019)
Re-use (1)
(Hoeber et al, 2017)
Re-use (1)
(Hoeber et al, 2017)
Re-use (1)
(Hoeber et al, 2017)
Re-use (1)
(Hoeber et al, 2017)
Re-use (2)
• Gaikwad and Hoeber (2019) used
Vakkari’s model
• as “a design guide, and as a
mechanism for controlling the
l...
5. Discussion points (1/3)
• Lack of space to document all aspects of our study
• Some in my PhD thesis (2018)
• (now: CHI...
• Tension between flexibility in terms of research
questions, and the possibility to re-use
standardized systems and approa...
• Tension between flexibility in terms of research
questions, and the possibility to re-use
standardized systems and approa...
Discussion points (3/3)
• Maintenance: just four years after our study,
components of system have changed (e.g. Bing
API c...
References
• For all citations see: bit.ly/ChiirExperiencePaper
The Multi-Stage Experience
The Simulated Work Task Approach to
Studying Information Seeking Stages
Hugo Huurdeman Jaap Kam...
The Multi-Stage Experience: the Simulated Work Task Approach to Studying Information Seeking Stages
The Multi-Stage Experience: the Simulated Work Task Approach to Studying Information Seeking Stages
The Multi-Stage Experience: the Simulated Work Task Approach to Studying Information Seeking Stages
The Multi-Stage Experience: the Simulated Work Task Approach to Studying Information Seeking Stages
The Multi-Stage Experience: the Simulated Work Task Approach to Studying Information Seeking Stages
The Multi-Stage Experience: the Simulated Work Task Approach to Studying Information Seeking Stages
The Multi-Stage Experience: the Simulated Work Task Approach to Studying Information Seeking Stages
Upcoming SlideShare
Loading in …5
×
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

1

Share

Download to read offline

The Multi-Stage Experience: the Simulated Work Task Approach to Studying Information Seeking Stages

Download to read offline

Presentation given at the CHIIR 2019 Workshop on Barriers to Interactive IR Resources Re-use (Glasgow, UK, March 14, 2019).

This experience paper shines more light on a simulated work task approach to studying information seeking stages. This explicit multi-stage approach was first utilized in a CHIIIR 2016 paper to investigate the utility of search user interface (SUI) features at different macro-level stages of complex tasks. We focus on the paper’s terminology, research design, methodology anduse of previous resources. Finally, based on our experience, we reflect on the potential for re-using our multistage approach and on general barriers to re-use in an Interactive Information Retrieval research context.
Proceedings available at: http://ceur-ws.org/Vol-2337/

Related Books

Free with a 30 day trial from Scribd

See all

The Multi-Stage Experience: the Simulated Work Task Approach to Studying Information Seeking Stages

  1. 1. The Multi-Stage Experience The Simulated Work Task Approach to Studying Information Seeking Stages Hugo Huurdeman Jaap Kamps Max L. Wilson @timelessfuture @jkamps @gingdottwitt Original CHIIR ’16 paper: bit.ly/ActivePassiveUtility Experience paper: bit.ly/ChiirExperiencePaper University of Amsterdam / University of Nottingham
  2. 2. Original CHIIR ’16 paper: bit.ly/ActivePassiveUtility
  3. 3. Original CHIIR ’16 paper: bit.ly/ActivePassiveUtility Experience paper: bit.ly/ChiirExperiencePaper
  4. 4. image: Endless Time by davidmontse
  5. 5. image: https://www.artofplay.com/blogs/articles/fun-with-the-droste-effect
  6. 6. • Models of information seeking describe fundamentally different macro-level stages CHIIR ’16 paper: Premise
  7. 7. • Models of information seeking describe fundamentally different macro-level stages CHIIR ’16 paper: Premise +uncertainty- feelings thoughts actions vague focused seeking general information (exploring) seeking pertinent information (documenting) uncertainty optimism confusion clarity confidence (dis)satisfaction doubt direction FormulationInitiation Selection Exploration Collection Presentation Prefocus Focus Postfocus
  8. 8. • Models of information seeking describe fundamentally different macro-level stages CHIIR ’16 paper: Premise +uncertainty- feelings thoughts actions vague focused seeking general information (exploring) seeking pertinent information (documenting) uncertainty optimism confusion clarity confidence (dis)satisfaction doubt direction FormulationInitiation Selection Exploration Collection Presentation Prefocus Focus Postfocus [Kuhlthau04,Vakkari01]
  9. 9. • Very valuable, but: difficult to translate to concrete search system functionalitymoves & tactics task progress inform pre-focus focus post-focus Information seeking models ? ? search engine functionality
  10. 10. a
  11. 11. a Wealth of potential supportive features
  12. 12. a Wealth of potential supportive features but unknown role in process
  13. 13. a Wealth of potential supportive features but unknown role in process (too many = overload)
  14. 14. • Valuable, but: difficult to translate to concrete search systems functionality Information seeking models ? ? • We need a more accurate understanding 
 of what functionality is useful when task progress inform pre-focus focus post-focus search engine functionality
  15. 15. Introduction •Our study @CHIIR16 investigated the utility of various SUI features at different macro-level stages •Via an explicit multi-stage design
  16. 16. Multi-stage approach?
  17. 17. Multi-stage approach? • Kuhlthau’s and Vakkari’s models had been conceived based on longitudinal settings (Kuhlthau, 04;Vakkari,01) • monitor process at various time points
  18. 18. Multi-stage approach? • Kuhlthau’s and Vakkari’s models had been conceived based on longitudinal settings (Kuhlthau, 04;Vakkari,01) • monitor process at various time points Versus
  19. 19. Multi-stage approach? • Kuhlthau’s and Vakkari’s models had been conceived based on longitudinal settings (Kuhlthau, 04;Vakkari,01) • monitor process at various time points Versus • Studies looking at stages in singular sessions 
 (Diriye et al,10; Niu & Kelly, 14; Huurdeman & Kamps,14, etc.) • “these studies can be better characterized as focusing on phases of search or temporal search segments” 
 (Niu & Kelly, 14)
  20. 20. Finding a ‘middle point’ True longitudinal study Stages as temporal phases of singular sessions Potentially use many data collection instruments Limited number of data collection instruments
  21. 21. Finding a ‘middle point’ True longitudinal study Stages as temporal phases of singular sessions Multiple sessions representing stages Potentially use many data collection instruments Limited number of data collection instruments
  22. 22. Finding a ‘middle point’ • Multi-stage Simulated Work Task Approach • Single simulated work task, multiple subtasks • “Multi-stage experience” True longitudinal study Stages as temporal phases of singular sessions Multiple sessions representing stages Potentially use many data collection instruments Limited number of data collection instruments
  23. 23. Further outline • 1. Terminology • 2. Methodology • 3. Re-use of previous resources • 4. Potential for re-use of our approach • 5. Further discussion points
  24. 24. 1. Terminology
  25. 25. Terminology: Information Seeking Models • Information seeking modeled in a multitude of ways: • as behavioral patterns (Ellis) • as nonlinear activities (Foster) • as temporal stages (Kuhlthau), .. • Our main focus: • temporally based IS models • Kuhlthau [1991] (Vakkari [2001]) • cognitively complex (work) tasks • involving learning & construction • Using definitions Wilson (1999), 
 Ingwersen & Jarvelin (2005), Byström & Järvelin (1995) information search information
 seeking information 
 behavior [Wilson99]
  26. 26. SUI features categorized using Wilson’s (2011) framework
  27. 27. Control
  28. 28. Control Input
  29. 29. Control Input Informational
  30. 30. Control Input PersonalizableInformational
  31. 31. 2. Methodology
  32. 32. Methodology • User study (26 participants + 2 pilot participants) •24 analyzed • Undergrads Univ. of Nottingham (6 F, 18 M / 18-25y) • Experimental SUI resembling common Search Engine • Within-participants • Task stage independent variable • Task design: explicit multistage approach
  33. 33. Methodology: Multistage Task Design sim. work task: writing essay subtask subtask subtask prepare list of 
 3 topics choose topic;
 formulate specific
 question find and select 
 additional
 pages to cite ~15 minutes ~15 minutes ~15 minutes initiation
 topic selection
 exploration focus formulation collecting presenting pre-focus focus formulation post-focus
  34. 34. Methodology: Multistage Task Design sim. work task: writing essay subtask subtask subtask prepare list of 
 3 topics choose topic;
 formulate specific
 question find and select 
 additional
 pages to cite ~15 minutes ~15 minutes ~15 minutes initiation
 topic selection
 exploration focus formulation collecting presenting Discussed with 
 information seeking specialists —> pre-focus focus formulation post-focus
  35. 35. Methodology: Multistage Task Design sim. work task: writing essay subtask subtask subtask prepare list of 
 3 topics choose topic;
 formulate specific
 question find and select 
 additional
 pages to cite ~15 minutes ~15 minutes ~15 minutes initiation
 topic selection
 exploration focus formulation collecting presenting Tested with pilot
 participants —> Discussed with 
 information seeking specialists —> pre-focus focus formulation post-focus
  36. 36. Methodology: Multistage Task Design sim. work task: writing essay subtask subtask subtask prepare list of 
 3 topics choose topic;
 formulate specific
 question find and select 
 additional
 pages to cite ~15 minutes ~15 minutes ~15 minutes initiation
 topic selection
 exploration focus formulation collecting presenting Tested with pilot
 participants —> Discussed with 
 information seeking specialists —> Based on Vakkari 
 / Kuhlthau theory —> pre-focus focus formulation post-focus
  37. 37. Methodology: Multistage Task Design sim. work task: writing essay subtask subtask subtask prepare list of 
 3 topics choose topic;
 formulate specific
 question find and select 
 additional
 pages to cite ~15 minutes ~15 minutes ~15 minutes initiation
 topic selection
 exploration focus formulation collecting presenting Topics • Autonomous Vehicles (AV) • Virtual Reality (VR) Tested with pilot
 participants —> Discussed with 
 information seeking specialists —> Based on Vakkari 
 / Kuhlthau theory —> pre-focus focus formulation post-focus
  38. 38. Methodology: Multistage Task Design sim. work task: writing essay subtask subtask subtask prepare list of 
 3 topics choose topic;
 formulate specific
 question find and select 
 additional
 pages to cite ~15 minutes ~15 minutes ~15 minutes initiation
 topic selection
 exploration focus formulation collecting presenting Topics • Autonomous Vehicles (AV) • Virtual Reality (VR) Discussions with teaching staff —> Tested with pilot
 participants —> Discussed with 
 information seeking specialists —> Based on Vakkari 
 / Kuhlthau theory —> pre-focus focus formulation post-focus
  39. 39. Methodology: Multistage Task Design sim. work task: writing essay subtask subtask subtask prepare list of 
 3 topics choose topic;
 formulate specific
 question find and select 
 additional
 pages to cite ~15 minutes ~15 minutes ~15 minutes initiation
 topic selection
 exploration focus formulation collecting presenting Topics • Autonomous Vehicles (AV) • Virtual Reality (VR) Discussions with teaching staff —> Tested with pilot
 participants —> Discussed with 
 information seeking specialists —> Based on Vakkari 
 / Kuhlthau theory —> pre-focus focus formulation post-focus Standard amount 
 + additional incentive —>
  40. 40. Methodology: Simplified Protocol Training task Pre- Questionnaire Topic Assignment Introduction system Task Post-task Questionnaire 3x Post-experiment questionnaire Debriefing interview
  41. 41. Methodology: Simplified Protocol Training task Pre- Questionnaire Topic Assignment Introduction system Task Post-task Questionnaire 3x Post-experiment questionnaire Debriefing interview <— mitigate familiarity effects
  42. 42. Tutorial slides Pia Borlund (2012)
  43. 43. Task & task stage validation • Validating the 2 topics (statistically) • Various validation questions within the process (post-task, 3x) 1. Randomized list of activities derived from Kuhlthau’s model 2. Reported type of information sought (Kuhlthau, Todd) 3. Randomized word list which could represent users feelings / state of mind (Kuhlthau, Todd)
  44. 44. Methodology: data collection instruments
  45. 45. Methodology: data collection instruments
  46. 46. Methodology: data collection instruments
  47. 47. Methodology: data collection instruments
  48. 48. Methodology: data collection instruments
  49. 49. Methodology: data collection instruments
  50. 50. 3. Re-use previous resources
  51. 51. Task descriptions Initial inspiration: • Kuhlthau’s book, Vakkari’s articles • Previous literature survey 
 (Huurdeman&Kamps,2014) • Research process / IL literature • RepAST repository
  52. 52. Task descriptions Initial inspiration: • Kuhlthau’s book, Vakkari’s articles • Previous literature survey 
 (Huurdeman&Kamps,2014) • Research process / IL literature • RepAST repository Kumar, 2010
  53. 53. Task descriptions Some general inspiration: Liu & Belkin, 2015
  54. 54. Task descriptions Kules & Shneiderman, 2009 Some general inspiration: Liu & Belkin, 2015
  55. 55. • Experimental system: SearchAssist • Results, Query Corrections, Query Suggestions: Bing Web API • Category Filters, categories: DMOZ • Tag cloud: Jquery plugin • Task instructions: Google Doc • Categorization and analysis: • Max Wilson’s framework of SUI features [Wilson11]
  56. 56. PyGaze, PyTribe
  57. 57. PyGaze, PyTribe
  58. 58. PyGaze, PyTribe
  59. 59. PyGaze, PyTribe
  60. 60. PyGaze, PyTribe
  61. 61. PyGaze, PyTribe
  62. 62. PyGaze, PyTribe
  63. 63. 4. Potential for re-use
  64. 64. Re-use of research design: so far • Two papers: • Hoeber et al. (2017) • Gaikwad and Hoeber (2019)
  65. 65. Re-use (1) (Hoeber et al, 2017)
  66. 66. Re-use (1) (Hoeber et al, 2017)
  67. 67. Re-use (1) (Hoeber et al, 2017)
  68. 68. Re-use (1) (Hoeber et al, 2017)
  69. 69. Re-use (2) • Gaikwad and Hoeber (2019) used Vakkari’s model • as “a design guide, and as a mechanism for controlling the laboratory-based evaluation.” • participants • explored images (pre-focus) • selected images (focus formulation) • organized images (post-focus) • Task descriptions: holiday plans, food blogging and self-selected tasks.
  70. 70. 5. Discussion points (1/3) • Lack of space to document all aspects of our study • Some in my PhD thesis (2018) • (now: CHIIR extra pages) • Typical lack of time within research process and publication cycle • at the time, publishing source code SearchAssist, but not contextual documents or analysis scripts • A restrictive consent form …
  71. 71. • Tension between flexibility in terms of research questions, and the possibility to re-use standardized systems and approaches • Use of search API vs standard collection • Remuneration of participants 5. Discussion points (2/3)
  72. 72. • Tension between flexibility in terms of research questions, and the possibility to re-use standardized systems and approaches • Use of search API vs standard collection • Remuneration of participants 5. Discussion points (2/3)
  73. 73. Discussion points (3/3) • Maintenance: just four years after our study, components of system have changed (e.g. Bing API configurations) • Persistence: various URLs of used resources now only available in the Internet Archive
  74. 74. References • For all citations see: bit.ly/ChiirExperiencePaper
  75. 75. The Multi-Stage Experience The Simulated Work Task Approach to Studying Information Seeking Stages Hugo Huurdeman Jaap Kamps Max L. Wilson @timelessfuture @jkamps @gingdottwitt Original CHIIR ’16 paper: bit.ly/ActivePassiveUtility Experience paper: bit.ly/ChiirExperiencePaper University of Amsterdam / University of Nottingham
  • NNMathabela

    Aug. 27, 2020

Presentation given at the CHIIR 2019 Workshop on Barriers to Interactive IR Resources Re-use (Glasgow, UK, March 14, 2019). This experience paper shines more light on a simulated work task approach to studying information seeking stages. This explicit multi-stage approach was first utilized in a CHIIIR 2016 paper to investigate the utility of search user interface (SUI) features at different macro-level stages of complex tasks. We focus on the paper’s terminology, research design, methodology anduse of previous resources. Finally, based on our experience, we reflect on the potential for re-using our multistage approach and on general barriers to re-use in an Interactive Information Retrieval research context. Proceedings available at: http://ceur-ws.org/Vol-2337/

Views

Total views

291

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

5

Shares

0

Comments

0

Likes

1

×