Collecting a dataset of information behaviour in context
•Download as PPTX, PDF•
1 like•1,166 views
We collected human-computer interaction data (keystrokes, active applications, typed text, etc.) from knowledge workers in the context of writing reports and preparing presentations. This has resulted in an interesting dataset that can be used for different types of information retrieval and information seeking research. The details of the dataset are presented in this paper.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (15)
Similar to Collecting a dataset of information behaviour in context
Similar to Collecting a dataset of information behaviour in context (20)
More from Leiden University
More from Leiden University (14)
Recently uploaded
Recently uploaded (20)
Collecting a dataset of information behaviour in context
- 1. http://www.swell-project.net Collecting a dataset of information behaviour in context Maya Sappelli, TNO & Radboud University Nijmegen Suzan Verberne, Radboud University Nijmegen Saskia Koldijk, TNO & Radboud University Nijmegen Wessel Kraaij, TNO & Radboud University Nijmegen Supported by the Dutch National Program:
- 2. http://www.swell-project.net Information behaviour in context Supported by the Dutch National Program: 2 / 15
- 3. http://www.swell-project.net Information behaviour in context Supported by the Dutch National Program: 3 / 15
- 4. http://www.swell-project.net But… • Controlled Search Experiment Lacks context for search Unnatural motive/behaviour • Uncontrolled Data Collection Privacy issues Noise Supported by the Dutch National Program: 4 / 15
- 5. http://www.swell-project.net Data Collection Supported by the Dutch National Program: 5 / 15
- 6. http://www.swell-project.net Data Collection Supported by the Dutch National Program: 6 / 15
- 7. http://www.swell-project.net Data Labeling: Event Stream to Event Blocks Supported by the Dutch National Program: 7 / 15 Event Block e yOutlook A nH Inbox
- 8. http://www.swell-project.net Data Labeling: presenting Event Blocks • Mechanical Turk • 9416 event blocks • Cohen’s kappa 0.78 Supported by the Dutch National Program: 8 / 15
- 9. http://www.swell-project.net Data Labeling: result Supported by the Dutch National Program: 9 / 15 Distribution of Labels Einstein Information Overload Stress Healthy Privacy Perth Roadtrip Napoleon Indeterminable No Label Total no. Event blocks 9416 Average no. Event blocks per participant 377
- 10. http://www.swell-project.net Examples of analyses with the data • Stress-related behavioural research • Information-related behavioural research 1. system-oriented 2. user-oriented Supported by the Dutch National Program: 10 / 15
- 11. http://www.swell-project.net System-oriented analysis Supported by the Dutch National Program: 11 / 15 Total # Queries: 980 Of which followed by a click on a URL: 732 Of which followed by a switch to Word/ Powerpoint: 125 Of which with Ctrl+C: 15 with a dwell- time of >=30 seconds: 44
- 12. http://www.swell-project.net User-oriented analysis Supported by the Dutch National Program: 12 / 15
- 13. http://www.swell-project.net Discussion: challenges • Combining data from multiple sources is not trivial • Incomplete queries logged due to Google’s query suggestions • Clicks without change in Window title (esp. Google Images) • Noise from browser logging Supported by the Dutch National Program: 13 / 15
- 14. http://www.swell-project.net Conclusions • Dataset of information behaviour of knowledge workers • Main contributions of the dataset: 1. Combination of data types 2. Natural information seeking behaviour 3. In-context recordings Supported by the Dutch National Program: 14 / 15
Editor's Notes
- Figure 4 shows an example of a behavioural analysis: a transition graph for the workers’ information behaviour. The size of the states represents their relative frequency. The state ‘query’ represents events where the active application is the web browser, in which a Google query and its results are shown. The state ‘Google’ represents events where a Google page is active without query. The state ‘OtherURL’ represents events where the active application is the web browser, with another URL than Google. The transition probability between states S 1 and S 2 was calculated as count(S 1 -> S 2 )/count(S 2 ). Only transitions with a probability > 0.1 are shown.The graph shows that when users are asked to write reports or prepare presentations on a relatively new topic, they spend more time on web pages than in the report they are writing, and they switch frequently between URLs and the report in order to gather the relevant information. The graph also shows the relatively frequent interruptions of e-mail, which is known to be very common for knowledge workers (S. Whittaker and C. Sidner, 1996)
- Combining the data from the keylogging software and the browser history software was not trivial, even with exactly matching timestamps. This was because the user could have multi ple tabs active in the browser, with not all tab titles being separately recorded by the keylogging software. Users clicking one of Google’s query suggestions sometimes led to incomplete queries and missing URLs. For example, we found that the query ‘napol’ lead to the URL http://nl.wikipedia.org/wiki/ Napoleon Dynamite. We sus pect that his happened because the user selected the sug gested query ‘napoleon dynamite’ after the offset ‘napol’, and then clicked the Wikipedia URL. In some cases the window title of the browser did not change when a user clicked on a result (especially when the click was a result from Google Images), which caused the clicked URL to be included in the same event block as the query, and dwell time was missing for this particular URL. Fourth, the browser logging resulted in a lot of noise. We had to filter out a large amount of on-page social media plugins, advertisements and icons. In addition, browsing the Google domain leads to many additional URLs. An extreme example was 25 occurrences of the Google Maps URL http://maps.google.nl/maps?hl=nl &q=usa&bav=on.2 in one event block.
- We collected and preprocessed a dataset of information behaviour of knowledge workers in the context of realistic work tasks. The data set is relatively small in terms of the number of participants, but large in terms of types of information collected. The contributions of this dataset are: it includes different types of data, including keylogging data, desktop video recordings and browser history; the information seeking behaviour is completely natural because it results from the recording of user behaviour during report writing, e-mail reading and presentation preparationthe search activities have been recorded together with the context of these other tasks, which allows for future research in context-aware information retrieval.