Today information items on user’s workstations are usually stored in separate collections depending on their format. This results in a disconnect between information systems and user needs leading to high lookup times during task related information retrieval. This paper presents an approach to reduce document based information fragmentation by semantically reconnecting electronic documents to each other without imposing additional training or tagging workload on the user. To this end the actions knowledge workers perform on their desktop are transparently monitored to analyze the user’s interaction with his computer system. These action metadata are further clustered by superordinate activities performed by the user. Finally documents attached to window instances within the identified activity clusters are semantically related to each other reducing the fragmentation of their contained information. This allows a subsequent associative information discovery navigating from one document instance to other related document instances. A prototypical implementation and evaluation in a small scale testing setup indicates the validity of the approach.

1. Semantically Reconnecting Fragmented Information through User Activity Monitoring Hinnerk Brügmann, 2011/02/17

2. Motivation Rapidly rising amount of unstructured information in personal and enterprise environments High effort to locate required Information 1 2 Potential redundancy 3 Orphaned documents 4 Outdated document versions

4. E-mails in a separate mailbox hierarchy

6. (Meta-)Information as a Source for semantic Relations maintenance operations concurrently open documents provenance contextual personal access and usage collaborative access and usage compliance status user classification static administrative access rights static file attributes inherent content Personal Domain Enterprise Domain

7. Existing Research Approaches Activity-based Relation Building maintenance operations concurrently open documents provenance contextual personal access and usage collaborative access and usage compliance status user classification static administrative access rights static file attributes Content-based Relation Building inherent content Personal Domain Enterprise Domain

8. Focus of the presented Approach Activity-based Relation Building maintenance operations concurrently open documents provenance contextual personal access and usage collaborative access and usage compliance status user classification static administrative access rights static file attributes Content-based Relation Building inherent content Personal Domain Enterprise Domain

9. Components of a User Task Task Motive Activity Goal Action Condition Operation Source: Kuutti (1996).

10. Action Context Action Scope Time User Operation Workplace Environment before during after

11. Action Context Action Scope Time User Operation Workplace Environment before during (UO1) copying text from price-list.doc into new document sales-offer.doc after

12. Action Context Action Scope Time User Operation Workplace Environment before (WE0) opened document price-list.doc during (WE1) open documents price-list.doc and sales-offer.doc (UO1) copying text from price-list.doc into new document sales-offer.doc after (WE2) opened document price-list.doc (UO2) saving document sales-offer.doc into folder customer-alpha on the local file system

13. Multitasking Begin primary task Alert for secondary task Begin secondary task End secondary task Resume primary task Interruption lag Resumption lag Rehearse primary task problem Clean up primary task Do primary task Do secondary task Recall primary task problem Do primary task

14. Snapshot Data groupedby Time Spans Sensor reading (08:00 – 16:00)monitor configurations 1024 x 768 and 1280 x 800 Time Span 1 (visible windows 8:00 – 8:01) Metadata window A Metadata window B . . . Time Span 2 (visible windows 08:01 – 08:03) Metadata window A Metadata window C . . . . . .

15. Snapshot Data groupedby Time Spans Sensor reading (08:00 – 16:00)monitor configurations 1024 x 768 and 1280 x 800 Time Span 1 (visible windows 8:00 – 8:01) X/Y/Z Position Height Width Window Handle Parent Window Handle Application ID Focus indicator Metadata window A Metadata window B . . . Time Span 2 (visible windows 08:01 – 08:03) Metadata window A Metadata window C . . . . . .

16. Snapshot Data groupedby Time Spans Sensor reading (08:00 – 16:00)monitor configurations 1024 x 768 and 1280 x 800 Time Span 1 (visible windows 8:00 – 8:01) X/Y/Z Position Height Width Window Handle Parent Window Handle Application ID Focus indicator Metadata window A Metadata window B . . . Time Span 2 (visible windows 08:01 – 08:03) Metadata window A Metadata window C File Path Document Title Textual Content . . . . . .

18. Parent window id

19. Textual similarity of title

21. Parent window id

22. Textual similarity of title

23. Textual similarity of content

25. Parent window id

26. Textual similarity of title

27. Textual similarity of content

29. Same application

30. Textual similarity of title

32. Parent window id

33. Textual similarity of title

34. Textual similarity of content

36. Same application

37. Textual similarity of title

39. ResultingActivity Custer

41. Evaluation Prototypically implemented sensor plugin was installed on the client desktops of 4 knowledge workers. 15 Working Items with durations ranging from 5 minutes to 5 hours Users denying generated relations  false positives Users stating relations not detected by the system false negatives Average combined error rate of ~4% Computed reliability significantly lower on erroneous relations

42. Thank you Semantically Reconnecting Fragmented Information through User Activity Monitoring Hinnerk Brügmann http://consense-project.com