Recommended
Recommended
More Related Content
Similar to The Multi-Stage Experience: the Simulated Work Task Approach to Studying Information Seeking Stages
Similar to The Multi-Stage Experience: the Simulated Work Task Approach to Studying Information Seeking Stages (20)
More from TimelessFuture
More from TimelessFuture (20)
Recently uploaded
Recently uploaded (20)
The Multi-Stage Experience: the Simulated Work Task Approach to Studying Information Seeking Stages
- 1. The Multi-Stage Experience The Simulated Work Task Approach to Studying Information Seeking Stages Hugo Huurdeman Jaap Kamps Max L. Wilson @timelessfuture @jkamps @gingdottwitt Original CHIIR ’16 paper: bit.ly/ActivePassiveUtility Experience paper: bit.ly/ChiirExperiencePaper University of Amsterdam / University of Nottingham
- 2. Original CHIIR ’16 paper: bit.ly/ActivePassiveUtility
- 3. Original CHIIR ’16 paper: bit.ly/ActivePassiveUtility Experience paper: bit.ly/ChiirExperiencePaper
- 4. image: Endless Time by davidmontse
- 6. • Models of information seeking describe fundamentally different macro-level stages CHIIR ’16 paper: Premise
- 7. • Models of information seeking describe fundamentally different macro-level stages CHIIR ’16 paper: Premise +uncertainty- feelings thoughts actions vague focused seeking general information (exploring) seeking pertinent information (documenting) uncertainty optimism confusion clarity confidence (dis)satisfaction doubt direction FormulationInitiation Selection Exploration Collection Presentation Prefocus Focus Postfocus
- 8. • Models of information seeking describe fundamentally different macro-level stages CHIIR ’16 paper: Premise +uncertainty- feelings thoughts actions vague focused seeking general information (exploring) seeking pertinent information (documenting) uncertainty optimism confusion clarity confidence (dis)satisfaction doubt direction FormulationInitiation Selection Exploration Collection Presentation Prefocus Focus Postfocus [Kuhlthau04,Vakkari01]
- 9. • Very valuable, but: difficult to translate to concrete search system functionalitymoves & tactics task progress inform pre-focus focus post-focus Information seeking models ? ? search engine functionality
- 11. a
- 12. a Wealth of potential supportive features
- 13. a Wealth of potential supportive features but unknown role in process
- 14. a Wealth of potential supportive features but unknown role in process (too many = overload)
- 15. • Valuable, but: difficult to translate to concrete search systems functionality Information seeking models ? ? • We need a more accurate understanding of what functionality is useful when task progress inform pre-focus focus post-focus search engine functionality
- 16. Introduction •Our study @CHIIR16 investigated the utility of various SUI features at different macro-level stages •Via an explicit multi-stage design
- 18. Multi-stage approach? • Kuhlthau’s and Vakkari’s models had been conceived based on longitudinal settings (Kuhlthau, 04;Vakkari,01) • monitor process at various time points
- 19. Multi-stage approach? • Kuhlthau’s and Vakkari’s models had been conceived based on longitudinal settings (Kuhlthau, 04;Vakkari,01) • monitor process at various time points Versus
- 20. Multi-stage approach? • Kuhlthau’s and Vakkari’s models had been conceived based on longitudinal settings (Kuhlthau, 04;Vakkari,01) • monitor process at various time points Versus • Studies looking at stages in singular sessions (Diriye et al,10; Niu & Kelly, 14; Huurdeman & Kamps,14, etc.) • “these studies can be better characterized as focusing on phases of search or temporal search segments” (Niu & Kelly, 14)
- 21. Finding a ‘middle point’ True longitudinal study Stages as temporal phases of singular sessions Potentially use many data collection instruments Limited number of data collection instruments
- 22. Finding a ‘middle point’ True longitudinal study Stages as temporal phases of singular sessions Multiple sessions representing stages Potentially use many data collection instruments Limited number of data collection instruments
- 23. Finding a ‘middle point’ • Multi-stage Simulated Work Task Approach • Single simulated work task, multiple subtasks • “Multi-stage experience” True longitudinal study Stages as temporal phases of singular sessions Multiple sessions representing stages Potentially use many data collection instruments Limited number of data collection instruments
- 24. Further outline • 1. Terminology • 2. Methodology • 3. Re-use of previous resources • 4. Potential for re-use of our approach • 5. Further discussion points
- 25. 1. Terminology
- 26. Terminology: Information Seeking Models • Information seeking modeled in a multitude of ways: • as behavioral patterns (Ellis) • as nonlinear activities (Foster) • as temporal stages (Kuhlthau), .. • Our main focus: • temporally based IS models • Kuhlthau [1991] (Vakkari [2001]) • cognitively complex (work) tasks • involving learning & construction • Using definitions Wilson (1999), Ingwersen & Jarvelin (2005), Byström & Järvelin (1995) information search information seeking information behavior [Wilson99]
- 28. SUI features categorized using Wilson’s (2011) framework
- 30. Control
- 31. Control Input
- 34. 2. Methodology
- 35. Methodology • User study (26 participants + 2 pilot participants) •24 analyzed • Undergrads Univ. of Nottingham (6 F, 18 M / 18-25y) • Experimental SUI resembling common Search Engine • Within-participants • Task stage independent variable • Task design: explicit multistage approach
- 36. Methodology: Multistage Task Design sim. work task: writing essay subtask subtask subtask prepare list of 3 topics choose topic; formulate specific question find and select additional pages to cite ~15 minutes ~15 minutes ~15 minutes initiation topic selection exploration focus formulation collecting presenting pre-focus focus formulation post-focus
- 37. Methodology: Multistage Task Design sim. work task: writing essay subtask subtask subtask prepare list of 3 topics choose topic; formulate specific question find and select additional pages to cite ~15 minutes ~15 minutes ~15 minutes initiation topic selection exploration focus formulation collecting presenting Discussed with information seeking specialists —> pre-focus focus formulation post-focus
- 38. Methodology: Multistage Task Design sim. work task: writing essay subtask subtask subtask prepare list of 3 topics choose topic; formulate specific question find and select additional pages to cite ~15 minutes ~15 minutes ~15 minutes initiation topic selection exploration focus formulation collecting presenting Tested with pilot participants —> Discussed with information seeking specialists —> pre-focus focus formulation post-focus
- 39. Methodology: Multistage Task Design sim. work task: writing essay subtask subtask subtask prepare list of 3 topics choose topic; formulate specific question find and select additional pages to cite ~15 minutes ~15 minutes ~15 minutes initiation topic selection exploration focus formulation collecting presenting Tested with pilot participants —> Discussed with information seeking specialists —> Based on Vakkari / Kuhlthau theory —> pre-focus focus formulation post-focus
- 40. Methodology: Multistage Task Design sim. work task: writing essay subtask subtask subtask prepare list of 3 topics choose topic; formulate specific question find and select additional pages to cite ~15 minutes ~15 minutes ~15 minutes initiation topic selection exploration focus formulation collecting presenting Topics • Autonomous Vehicles (AV) • Virtual Reality (VR) Tested with pilot participants —> Discussed with information seeking specialists —> Based on Vakkari / Kuhlthau theory —> pre-focus focus formulation post-focus
- 41. Methodology: Multistage Task Design sim. work task: writing essay subtask subtask subtask prepare list of 3 topics choose topic; formulate specific question find and select additional pages to cite ~15 minutes ~15 minutes ~15 minutes initiation topic selection exploration focus formulation collecting presenting Topics • Autonomous Vehicles (AV) • Virtual Reality (VR) Discussions with teaching staff —> Tested with pilot participants —> Discussed with information seeking specialists —> Based on Vakkari / Kuhlthau theory —> pre-focus focus formulation post-focus
- 42. Methodology: Multistage Task Design sim. work task: writing essay subtask subtask subtask prepare list of 3 topics choose topic; formulate specific question find and select additional pages to cite ~15 minutes ~15 minutes ~15 minutes initiation topic selection exploration focus formulation collecting presenting Topics • Autonomous Vehicles (AV) • Virtual Reality (VR) Discussions with teaching staff —> Tested with pilot participants —> Discussed with information seeking specialists —> Based on Vakkari / Kuhlthau theory —> pre-focus focus formulation post-focus Standard amount + additional incentive —>
- 43. Methodology: Simplified Protocol Training task Pre- Questionnaire Topic Assignment Introduction system Task Post-task Questionnaire 3x Post-experiment questionnaire Debriefing interview
- 44. Methodology: Simplified Protocol Training task Pre- Questionnaire Topic Assignment Introduction system Task Post-task Questionnaire 3x Post-experiment questionnaire Debriefing interview <— mitigate familiarity effects
- 45. Tutorial slides Pia Borlund (2012)
- 48. Task & task stage validation • Validating the 2 topics (statistically) • Various validation questions within the process (post-task, 3x) 1. Randomized list of activities derived from Kuhlthau’s model 2. Reported type of information sought (Kuhlthau, Todd) 3. Randomized word list which could represent users feelings / state of mind (Kuhlthau, Todd)
- 49. Methodology: data collection instruments
- 50. Methodology: data collection instruments
- 51. Methodology: data collection instruments
- 52. Methodology: data collection instruments
- 53. Methodology: data collection instruments
- 54. Methodology: data collection instruments
- 56. Task descriptions Initial inspiration: • Kuhlthau’s book, Vakkari’s articles • Previous literature survey (Huurdeman&Kamps,2014) • Research process / IL literature • RepAST repository
- 57. Task descriptions Initial inspiration: • Kuhlthau’s book, Vakkari’s articles • Previous literature survey (Huurdeman&Kamps,2014) • Research process / IL literature • RepAST repository Kumar, 2010
- 58. Task descriptions Some general inspiration: Liu & Belkin, 2015
- 59. Task descriptions Kules & Shneiderman, 2009 Some general inspiration: Liu & Belkin, 2015
- 62. • Experimental system: SearchAssist • Results, Query Corrections, Query Suggestions: Bing Web API • Category Filters, categories: DMOZ • Tag cloud: Jquery plugin • Task instructions: Google Doc • Categorization and analysis: • Max Wilson’s framework of SUI features [Wilson11]
- 63. PyGaze, PyTribe
- 64. PyGaze, PyTribe
- 65. PyGaze, PyTribe
- 66. PyGaze, PyTribe
- 67. PyGaze, PyTribe
- 68. PyGaze, PyTribe
- 69. PyGaze, PyTribe
- 70. 4. Potential for re-use
- 71. Re-use of research design: so far • Two papers: • Hoeber et al. (2017) • Gaikwad and Hoeber (2019)
- 72. Re-use (1) (Hoeber et al, 2017)
- 73. Re-use (1) (Hoeber et al, 2017)
- 74. Re-use (1) (Hoeber et al, 2017)
- 75. Re-use (1) (Hoeber et al, 2017)
- 76. Re-use (2) • Gaikwad and Hoeber (2019) used Vakkari’s model • as “a design guide, and as a mechanism for controlling the laboratory-based evaluation.” • participants • explored images (pre-focus) • selected images (focus formulation) • organized images (post-focus) • Task descriptions: holiday plans, food blogging and self-selected tasks.
- 77. 5. Discussion points (1/3) • Lack of space to document all aspects of our study • Some in my PhD thesis (2018) • (now: CHIIR extra pages) • Typical lack of time within research process and publication cycle • at the time, publishing source code SearchAssist, but not contextual documents or analysis scripts • A restrictive consent form …
- 78. • Tension between flexibility in terms of research questions, and the possibility to re-use standardized systems and approaches • Use of search API vs standard collection • Remuneration of participants 5. Discussion points (2/3)
- 79. • Tension between flexibility in terms of research questions, and the possibility to re-use standardized systems and approaches • Use of search API vs standard collection • Remuneration of participants 5. Discussion points (2/3)
- 80. Discussion points (3/3) • Maintenance: just four years after our study, components of system have changed (e.g. Bing API configurations) • Persistence: various URLs of used resources now only available in the Internet Archive
- 81. References • For all citations see: bit.ly/ChiirExperiencePaper
- 82. The Multi-Stage Experience The Simulated Work Task Approach to Studying Information Seeking Stages Hugo Huurdeman Jaap Kamps Max L. Wilson @timelessfuture @jkamps @gingdottwitt Original CHIIR ’16 paper: bit.ly/ActivePassiveUtility Experience paper: bit.ly/ChiirExperiencePaper University of Amsterdam / University of Nottingham