Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Multi-Stage Experience: the Simulated Work Task Approach to Studying Information Seeking Stages

35 views

Published on

Presentation given at the CHIIR 2019 Workshop on Barriers to Interactive IR Resources Re-use (Glasgow, UK, March 14, 2019).

This experience paper shines more light on a simulated work task approach to studying information seeking stages. This explicit multi-stage approach was first utilized in a CHIIIR 2016 paper to investigate the utility of search user interface (SUI) features at different macro-level stages of complex tasks. We focus on the paper’s terminology, research design, methodology anduse of previous resources. Finally, based on our experience, we reflect on the potential for re-using our multistage approach and on general barriers to re-use in an Interactive Information Retrieval research context.
Proceedings available at: http://ceur-ws.org/Vol-2337/

Published in: Education
  • Be the first to comment

  • Be the first to like this

The Multi-Stage Experience: the Simulated Work Task Approach to Studying Information Seeking Stages

  1. 1. The Multi-Stage Experience The Simulated Work Task Approach to Studying Information Seeking Stages Hugo Huurdeman Jaap Kamps Max L. Wilson @timelessfuture @jkamps @gingdottwitt Original CHIIR ’16 paper: bit.ly/ActivePassiveUtility Experience paper: bit.ly/ChiirExperiencePaper University of Amsterdam / University of Nottingham
  2. 2. Original CHIIR ’16 paper: bit.ly/ActivePassiveUtility
  3. 3. Original CHIIR ’16 paper: bit.ly/ActivePassiveUtility Experience paper: bit.ly/ChiirExperiencePaper
  4. 4. image: Endless Time by davidmontse
  5. 5. image: https://www.artofplay.com/blogs/articles/fun-with-the-droste-effect
  6. 6. • Models of information seeking describe fundamentally different macro-level stages CHIIR ’16 paper: Premise
  7. 7. • Models of information seeking describe fundamentally different macro-level stages CHIIR ’16 paper: Premise +uncertainty- feelings thoughts actions vague focused seeking general information (exploring) seeking pertinent information (documenting) uncertainty optimism confusion clarity confidence (dis)satisfaction doubt direction FormulationInitiation Selection Exploration Collection Presentation Prefocus Focus Postfocus
  8. 8. • Models of information seeking describe fundamentally different macro-level stages CHIIR ’16 paper: Premise +uncertainty- feelings thoughts actions vague focused seeking general information (exploring) seeking pertinent information (documenting) uncertainty optimism confusion clarity confidence (dis)satisfaction doubt direction FormulationInitiation Selection Exploration Collection Presentation Prefocus Focus Postfocus [Kuhlthau04,Vakkari01]
  9. 9. • Very valuable, but: difficult to translate to concrete search system functionalitymoves & tactics task progress inform pre-focus focus post-focus Information seeking models ? ? search engine functionality
  10. 10. a
  11. 11. a Wealth of potential supportive features
  12. 12. a Wealth of potential supportive features but unknown role in process
  13. 13. a Wealth of potential supportive features but unknown role in process (too many = overload)
  14. 14. • Valuable, but: difficult to translate to concrete search systems functionality Information seeking models ? ? • We need a more accurate understanding 
 of what functionality is useful when task progress inform pre-focus focus post-focus search engine functionality
  15. 15. Introduction •Our study @CHIIR16 investigated the utility of various SUI features at different macro-level stages •Via an explicit multi-stage design
  16. 16. Multi-stage approach?
  17. 17. Multi-stage approach? • Kuhlthau’s and Vakkari’s models had been conceived based on longitudinal settings (Kuhlthau, 04;Vakkari,01) • monitor process at various time points
  18. 18. Multi-stage approach? • Kuhlthau’s and Vakkari’s models had been conceived based on longitudinal settings (Kuhlthau, 04;Vakkari,01) • monitor process at various time points Versus
  19. 19. Multi-stage approach? • Kuhlthau’s and Vakkari’s models had been conceived based on longitudinal settings (Kuhlthau, 04;Vakkari,01) • monitor process at various time points Versus • Studies looking at stages in singular sessions 
 (Diriye et al,10; Niu & Kelly, 14; Huurdeman & Kamps,14, etc.) • “these studies can be better characterized as focusing on phases of search or temporal search segments” 
 (Niu & Kelly, 14)
  20. 20. Finding a ‘middle point’ True longitudinal study Stages as temporal phases of singular sessions Potentially use many data collection instruments Limited number of data collection instruments
  21. 21. Finding a ‘middle point’ True longitudinal study Stages as temporal phases of singular sessions Multiple sessions representing stages Potentially use many data collection instruments Limited number of data collection instruments
  22. 22. Finding a ‘middle point’ • Multi-stage Simulated Work Task Approach • Single simulated work task, multiple subtasks • “Multi-stage experience” True longitudinal study Stages as temporal phases of singular sessions Multiple sessions representing stages Potentially use many data collection instruments Limited number of data collection instruments
  23. 23. Further outline • 1. Terminology • 2. Methodology • 3. Re-use of previous resources • 4. Potential for re-use of our approach • 5. Further discussion points
  24. 24. 1. Terminology
  25. 25. Terminology: Information Seeking Models • Information seeking modeled in a multitude of ways: • as behavioral patterns (Ellis) • as nonlinear activities (Foster) • as temporal stages (Kuhlthau), .. • Our main focus: • temporally based IS models • Kuhlthau [1991] (Vakkari [2001]) • cognitively complex (work) tasks • involving learning & construction • Using definitions Wilson (1999), 
 Ingwersen & Jarvelin (2005), Byström & Järvelin (1995) information search information
 seeking information 
 behavior [Wilson99]
  26. 26. SUI features categorized using Wilson’s (2011) framework
  27. 27. Control
  28. 28. Control Input
  29. 29. Control Input Informational
  30. 30. Control Input PersonalizableInformational
  31. 31. 2. Methodology
  32. 32. Methodology • User study (26 participants + 2 pilot participants) •24 analyzed • Undergrads Univ. of Nottingham (6 F, 18 M / 18-25y) • Experimental SUI resembling common Search Engine • Within-participants • Task stage independent variable • Task design: explicit multistage approach
  33. 33. Methodology: Multistage Task Design sim. work task: writing essay subtask subtask subtask prepare list of 
 3 topics choose topic;
 formulate specific
 question find and select 
 additional
 pages to cite ~15 minutes ~15 minutes ~15 minutes initiation
 topic selection
 exploration focus formulation collecting presenting pre-focus focus formulation post-focus
  34. 34. Methodology: Multistage Task Design sim. work task: writing essay subtask subtask subtask prepare list of 
 3 topics choose topic;
 formulate specific
 question find and select 
 additional
 pages to cite ~15 minutes ~15 minutes ~15 minutes initiation
 topic selection
 exploration focus formulation collecting presenting Discussed with 
 information seeking specialists —> pre-focus focus formulation post-focus
  35. 35. Methodology: Multistage Task Design sim. work task: writing essay subtask subtask subtask prepare list of 
 3 topics choose topic;
 formulate specific
 question find and select 
 additional
 pages to cite ~15 minutes ~15 minutes ~15 minutes initiation
 topic selection
 exploration focus formulation collecting presenting Tested with pilot
 participants —> Discussed with 
 information seeking specialists —> pre-focus focus formulation post-focus
  36. 36. Methodology: Multistage Task Design sim. work task: writing essay subtask subtask subtask prepare list of 
 3 topics choose topic;
 formulate specific
 question find and select 
 additional
 pages to cite ~15 minutes ~15 minutes ~15 minutes initiation
 topic selection
 exploration focus formulation collecting presenting Tested with pilot
 participants —> Discussed with 
 information seeking specialists —> Based on Vakkari 
 / Kuhlthau theory —> pre-focus focus formulation post-focus
  37. 37. Methodology: Multistage Task Design sim. work task: writing essay subtask subtask subtask prepare list of 
 3 topics choose topic;
 formulate specific
 question find and select 
 additional
 pages to cite ~15 minutes ~15 minutes ~15 minutes initiation
 topic selection
 exploration focus formulation collecting presenting Topics • Autonomous Vehicles (AV) • Virtual Reality (VR) Tested with pilot
 participants —> Discussed with 
 information seeking specialists —> Based on Vakkari 
 / Kuhlthau theory —> pre-focus focus formulation post-focus
  38. 38. Methodology: Multistage Task Design sim. work task: writing essay subtask subtask subtask prepare list of 
 3 topics choose topic;
 formulate specific
 question find and select 
 additional
 pages to cite ~15 minutes ~15 minutes ~15 minutes initiation
 topic selection
 exploration focus formulation collecting presenting Topics • Autonomous Vehicles (AV) • Virtual Reality (VR) Discussions with teaching staff —> Tested with pilot
 participants —> Discussed with 
 information seeking specialists —> Based on Vakkari 
 / Kuhlthau theory —> pre-focus focus formulation post-focus
  39. 39. Methodology: Multistage Task Design sim. work task: writing essay subtask subtask subtask prepare list of 
 3 topics choose topic;
 formulate specific
 question find and select 
 additional
 pages to cite ~15 minutes ~15 minutes ~15 minutes initiation
 topic selection
 exploration focus formulation collecting presenting Topics • Autonomous Vehicles (AV) • Virtual Reality (VR) Discussions with teaching staff —> Tested with pilot
 participants —> Discussed with 
 information seeking specialists —> Based on Vakkari 
 / Kuhlthau theory —> pre-focus focus formulation post-focus Standard amount 
 + additional incentive —>
  40. 40. Methodology: Simplified Protocol Training task Pre- Questionnaire Topic Assignment Introduction system Task Post-task Questionnaire 3x Post-experiment questionnaire Debriefing interview
  41. 41. Methodology: Simplified Protocol Training task Pre- Questionnaire Topic Assignment Introduction system Task Post-task Questionnaire 3x Post-experiment questionnaire Debriefing interview <— mitigate familiarity effects
  42. 42. Tutorial slides Pia Borlund (2012)
  43. 43. Task & task stage validation • Validating the 2 topics (statistically) • Various validation questions within the process (post-task, 3x) 1. Randomized list of activities derived from Kuhlthau’s model 2. Reported type of information sought (Kuhlthau, Todd) 3. Randomized word list which could represent users feelings / state of mind (Kuhlthau, Todd)
  44. 44. Methodology: data collection instruments
  45. 45. Methodology: data collection instruments
  46. 46. Methodology: data collection instruments
  47. 47. Methodology: data collection instruments
  48. 48. Methodology: data collection instruments
  49. 49. Methodology: data collection instruments
  50. 50. 3. Re-use previous resources
  51. 51. Task descriptions Initial inspiration: • Kuhlthau’s book, Vakkari’s articles • Previous literature survey 
 (Huurdeman&Kamps,2014) • Research process / IL literature • RepAST repository
  52. 52. Task descriptions Initial inspiration: • Kuhlthau’s book, Vakkari’s articles • Previous literature survey 
 (Huurdeman&Kamps,2014) • Research process / IL literature • RepAST repository Kumar, 2010
  53. 53. Task descriptions Some general inspiration: Liu & Belkin, 2015
  54. 54. Task descriptions Kules & Shneiderman, 2009 Some general inspiration: Liu & Belkin, 2015
  55. 55. • Experimental system: SearchAssist • Results, Query Corrections, Query Suggestions: Bing Web API • Category Filters, categories: DMOZ • Tag cloud: Jquery plugin • Task instructions: Google Doc • Categorization and analysis: • Max Wilson’s framework of SUI features [Wilson11]
  56. 56. PyGaze, PyTribe
  57. 57. PyGaze, PyTribe
  58. 58. PyGaze, PyTribe
  59. 59. PyGaze, PyTribe
  60. 60. PyGaze, PyTribe
  61. 61. PyGaze, PyTribe
  62. 62. PyGaze, PyTribe
  63. 63. 4. Potential for re-use
  64. 64. Re-use of research design: so far • Two papers: • Hoeber et al. (2017) • Gaikwad and Hoeber (2019)
  65. 65. Re-use (1) (Hoeber et al, 2017)
  66. 66. Re-use (1) (Hoeber et al, 2017)
  67. 67. Re-use (1) (Hoeber et al, 2017)
  68. 68. Re-use (1) (Hoeber et al, 2017)
  69. 69. Re-use (2) • Gaikwad and Hoeber (2019) used Vakkari’s model • as “a design guide, and as a mechanism for controlling the laboratory-based evaluation.” • participants • explored images (pre-focus) • selected images (focus formulation) • organized images (post-focus) • Task descriptions: holiday plans, food blogging and self-selected tasks.
  70. 70. 5. Discussion points (1/3) • Lack of space to document all aspects of our study • Some in my PhD thesis (2018) • (now: CHIIR extra pages) • Typical lack of time within research process and publication cycle • at the time, publishing source code SearchAssist, but not contextual documents or analysis scripts • A restrictive consent form …
  71. 71. • Tension between flexibility in terms of research questions, and the possibility to re-use standardized systems and approaches • Use of search API vs standard collection • Remuneration of participants 5. Discussion points (2/3)
  72. 72. • Tension between flexibility in terms of research questions, and the possibility to re-use standardized systems and approaches • Use of search API vs standard collection • Remuneration of participants 5. Discussion points (2/3)
  73. 73. Discussion points (3/3) • Maintenance: just four years after our study, components of system have changed (e.g. Bing API configurations) • Persistence: various URLs of used resources now only available in the Internet Archive
  74. 74. References • For all citations see: bit.ly/ChiirExperiencePaper
  75. 75. The Multi-Stage Experience The Simulated Work Task Approach to Studying Information Seeking Stages Hugo Huurdeman Jaap Kamps Max L. Wilson @timelessfuture @jkamps @gingdottwitt Original CHIIR ’16 paper: bit.ly/ActivePassiveUtility Experience paper: bit.ly/ChiirExperiencePaper University of Amsterdam / University of Nottingham

×