Eye tracking can provide insights into how people search for information. Two experiments were described. Experiment 1 found that search tasks influenced reading vs scanning behavior and text acquisition from results. Experiment 2 found that an overview interface was faster than a list, task complexity impacted effort, and individual differences like working memory influenced interaction. Eye tracking measures reflected task and interface effects and hinted at how cognitive abilities relate to search behavior. The goal is to implicitly characterize searchers and help systems adapt.
Paper-Digital User Interfaces - Applications, Frameworks and Future ChallengesBeat Signer
Invited talk given at the User Interface Colloquium, Otto-von-Guericke University Magdeburg, Germany, November 2, 2009
While there have been dramatic increases in the use of digital technologies for information storage, processing and delivery over the last few decades, the affordances of paper have ensured its retention as a key information medium. Despite predictions of the paperless office, paper is ever more present in our daily work. However, there is a gap between the paper and digital worlds: information present in paper documents cannot be seamlessly transferred to digital media and digital services are not easily accessible from the paper world.
ABSTRACT: In this talk I will present an information-centric approach for integrating paper with digital as well as physical media based on a general cross-media information platform (iServer). Some details about the architecture and implementation of the iServer platform as well as the underlying resource-selector-link (RSL) metamodel for cross-media linking will be highlighted. A selection of interactive paper applications that have been developed based on this platform over the past nine years will be presented, including the EdFest interactive paper guide for the Edinburgh festivals, the PaperPoint presentation tool as well as the PaperProof proof-editing solution. Challenges and solutions for novel forms of interactive paper and cross-media publishing are discussed based on the presented applications. This includes specific extensions of the iServer platform and RSL model as well as the application of our solution in new domains such as digital libraries, cross-media annotation and retrieval or personal cross-media information management that goes beyond the hierarchical information management imposed by the desktop metaphor.
Paper-Digital User Interfaces - Applications, Frameworks and Future ChallengesBeat Signer
Invited talk given at the User Interface Colloquium, Otto-von-Guericke University Magdeburg, Germany, November 2, 2009
While there have been dramatic increases in the use of digital technologies for information storage, processing and delivery over the last few decades, the affordances of paper have ensured its retention as a key information medium. Despite predictions of the paperless office, paper is ever more present in our daily work. However, there is a gap between the paper and digital worlds: information present in paper documents cannot be seamlessly transferred to digital media and digital services are not easily accessible from the paper world.
ABSTRACT: In this talk I will present an information-centric approach for integrating paper with digital as well as physical media based on a general cross-media information platform (iServer). Some details about the architecture and implementation of the iServer platform as well as the underlying resource-selector-link (RSL) metamodel for cross-media linking will be highlighted. A selection of interactive paper applications that have been developed based on this platform over the past nine years will be presented, including the EdFest interactive paper guide for the Edinburgh festivals, the PaperPoint presentation tool as well as the PaperProof proof-editing solution. Challenges and solutions for novel forms of interactive paper and cross-media publishing are discussed based on the presented applications. This includes specific extensions of the iServer platform and RSL model as well as the application of our solution in new domains such as digital libraries, cross-media annotation and retrieval or personal cross-media information management that goes beyond the hierarchical information management imposed by the desktop metaphor.
Panel: Social Tagging and Folksonomies: Indexing, Retrieving... and Beyond? ...jacekg
Panel presentation from ASIST'2011 panel: Social Tagging and Folksonomies: Indexing, Retrieving…and Beyond?
Jacek Gwizdka's presentation on cognitive load during search and browsing via tag clouds. And on he role of tags in information search and navigation between documents.
Discovering Common Motifs in Cursor Movement DataYandex
Mouse cursor movements can provide valuable information on how users interact and engage with web documents. This interaction data is far richer than traditional click data, and can be used to improve evaluation and presentation of web information systems. Unfortunately, the diversity and complexity inherent in this interaction data make it more difficult to capture salient behavior characteristics through traditional feature engineering. To address this problem, we introduce a novel approach of automatically discovering frequent subsequences, or motifs, in mouse cursor movement data. In order to scale our approach to realistic datasets, we introduce novel optimizations for motif discovery, specifically designed for mining cursor movement data. We show that by encoding the motifs discovered from thousands of real web search sessions as features, enables significant improvements in important web search tasks. These results, complemented with visualization and qualitative analysis, demonstrate that our approach is able to automatically capture key characteristics of mouse cursor movement behavior, providing a valuable new tool for online user behavior analysis. In addition to the application of motifs to web mining, we demonstrate that similar technique can be successfully applied in medical domain for the task of predicting future decline of memory function and subsequent development of the Alzheimer Disease.
An interdisciplinary journey with the SAL spaceship – results and challenges ...Stefan Dietze
Keynote at HELMeTO2022 conference, Palermo, Italy on recent research in Search As Learning (SAL), at the intersection of machine learning and cognitive psychology.
This slides are for a presentation at the 2009 IEEE/WIC/ACM International Conference on Web Intelligence. The major emphasis to this paper is concentrating on how to provide more personalized search support for a specific user considering his/her historical interests or recent interests. Cognitive memory retention like models are proposed and implemented in this system. Other supporting functionalities, such as domain distribution support, etc. are briefly mentioned. The whole paper can be downloaded from http://www.iwici.org/~yizeng/papers/WI2009-camera-ready.pdf
When developers perform a maintenance task, they
follow an exploration strategy (ES) that is characterised by how
they navigate through the program entities. Studying ES can help
to assess how developers understand a program and perform
a change task. Various factors could influence how developers
explore a program and the way in which they explore a program
may affect their performance for a certain task. In this paper,
we investigate the ES followed by developers during maintenance
tasks and assess the impact of these ES on the duration and effort
spent by developers on the tasks. We want to know if developers
frequently revisit one (or a set) of program entities (referenced
exploration), or if they visit program entities with almost the
same frequency (unreferenced exploration) when performing a
maintenance task. We mine 1,705 Mylyn interaction histories
(IH) from four open-source projects (ECF, Mylyn, PDE, and
Eclipse Platform) and perform a user study to verify if both
referenced exploration (RE) and unreferenced exploration (UE)
were followed by some developers. Using the Gini inequality index
on the number of revisits of program entities, we automatically
classify interaction histories as RE and UE and perform an
empirical study to measure the effect of program exploration
on the task duration and effort. We report that, although a UE
may require more exploration effort than a RE, a UE is on
average 12.30% less time consuming than a RE.
Architecture of an ontology based domain-specific natural language question a...IJwest
Question answering (QA) system aims at retrieving precise information from a large collection of
documents against a query. This paper describes the architecture of a Natural Language Question
Answering (NLQA) system for a specifi
c domain based on the ontological information, a step towards
semantic web question answering. The proposed architecture defines four basic modules suitable for
enhancing current QA capabilities with the ability of processing complex questions. The first m
odule was
the question processing, which analyses and classifies the question and also reformulates the user query.
The second module allows the process of retrieving the relevant documents. The next module processes the
retrieved documents, and the last m
odule performs the extraction and generation of a response. Natural
language processing techniques are used for processing the question and documents and also for answer
extraction. Ontology and domain knowledge are used for reformulating queries and ident
ifying the
relations. The aim of the system is to generate short and specific answer to the question that is asked in the
natural language in a specific domain. We have achieved 94 % accuracy of natural language question
answering in our implementation
Classifying Reading Behaviours using Deep Learning Methods with Eye-Tracking Data.
This work is interesting in identifying four reading behaviors: detailed-reading, non-reading, skimming, and scanning, by implementing three deep learning models – deep neural network (DNN), convolutional neural network (CNN), and recurrent neural networks (RNN), with eye-tracking data.
Replying to the findings, this paper proposes an idea to categorize reading behaviors by applying deep learning algorithms on eye-tracking data. More specifically, four activities – detailed-reading, non-reading, skimming, and scanning are classified by several deep learning algorithms listing as DNN, CNN, and RNN. Consequently, the work answers two research questions about which models and data type provide the highest accuracy when classifying reading behaviors.
Predicting User Knowledge Gain in Informational Search SessionsRan Yu
Slides of our SIGIR 2018 paper "Predicting User Knowledge Gain in Informational Search Sessions", which is presented in Ann Arbor, MI, US on July 9th, 2018.
A presentation that I gave at the Query Log Analysis: From Research to Best Practice Workshop 27 - 28 May 20098 in London, UK http://ir.shef.ac.uk/cloughie/qlaw2009/index.html
presents the foundational aspects of web analytics and some specifics such as the hotel problem. Discusses trace data, behaviorism, and other cool web analytics stuff
Panel: Social Tagging and Folksonomies: Indexing, Retrieving... and Beyond? ...jacekg
Panel presentation from ASIST'2011 panel: Social Tagging and Folksonomies: Indexing, Retrieving…and Beyond?
Jacek Gwizdka's presentation on cognitive load during search and browsing via tag clouds. And on he role of tags in information search and navigation between documents.
Discovering Common Motifs in Cursor Movement DataYandex
Mouse cursor movements can provide valuable information on how users interact and engage with web documents. This interaction data is far richer than traditional click data, and can be used to improve evaluation and presentation of web information systems. Unfortunately, the diversity and complexity inherent in this interaction data make it more difficult to capture salient behavior characteristics through traditional feature engineering. To address this problem, we introduce a novel approach of automatically discovering frequent subsequences, or motifs, in mouse cursor movement data. In order to scale our approach to realistic datasets, we introduce novel optimizations for motif discovery, specifically designed for mining cursor movement data. We show that by encoding the motifs discovered from thousands of real web search sessions as features, enables significant improvements in important web search tasks. These results, complemented with visualization and qualitative analysis, demonstrate that our approach is able to automatically capture key characteristics of mouse cursor movement behavior, providing a valuable new tool for online user behavior analysis. In addition to the application of motifs to web mining, we demonstrate that similar technique can be successfully applied in medical domain for the task of predicting future decline of memory function and subsequent development of the Alzheimer Disease.
An interdisciplinary journey with the SAL spaceship – results and challenges ...Stefan Dietze
Keynote at HELMeTO2022 conference, Palermo, Italy on recent research in Search As Learning (SAL), at the intersection of machine learning and cognitive psychology.
This slides are for a presentation at the 2009 IEEE/WIC/ACM International Conference on Web Intelligence. The major emphasis to this paper is concentrating on how to provide more personalized search support for a specific user considering his/her historical interests or recent interests. Cognitive memory retention like models are proposed and implemented in this system. Other supporting functionalities, such as domain distribution support, etc. are briefly mentioned. The whole paper can be downloaded from http://www.iwici.org/~yizeng/papers/WI2009-camera-ready.pdf
When developers perform a maintenance task, they
follow an exploration strategy (ES) that is characterised by how
they navigate through the program entities. Studying ES can help
to assess how developers understand a program and perform
a change task. Various factors could influence how developers
explore a program and the way in which they explore a program
may affect their performance for a certain task. In this paper,
we investigate the ES followed by developers during maintenance
tasks and assess the impact of these ES on the duration and effort
spent by developers on the tasks. We want to know if developers
frequently revisit one (or a set) of program entities (referenced
exploration), or if they visit program entities with almost the
same frequency (unreferenced exploration) when performing a
maintenance task. We mine 1,705 Mylyn interaction histories
(IH) from four open-source projects (ECF, Mylyn, PDE, and
Eclipse Platform) and perform a user study to verify if both
referenced exploration (RE) and unreferenced exploration (UE)
were followed by some developers. Using the Gini inequality index
on the number of revisits of program entities, we automatically
classify interaction histories as RE and UE and perform an
empirical study to measure the effect of program exploration
on the task duration and effort. We report that, although a UE
may require more exploration effort than a RE, a UE is on
average 12.30% less time consuming than a RE.
Architecture of an ontology based domain-specific natural language question a...IJwest
Question answering (QA) system aims at retrieving precise information from a large collection of
documents against a query. This paper describes the architecture of a Natural Language Question
Answering (NLQA) system for a specifi
c domain based on the ontological information, a step towards
semantic web question answering. The proposed architecture defines four basic modules suitable for
enhancing current QA capabilities with the ability of processing complex questions. The first m
odule was
the question processing, which analyses and classifies the question and also reformulates the user query.
The second module allows the process of retrieving the relevant documents. The next module processes the
retrieved documents, and the last m
odule performs the extraction and generation of a response. Natural
language processing techniques are used for processing the question and documents and also for answer
extraction. Ontology and domain knowledge are used for reformulating queries and ident
ifying the
relations. The aim of the system is to generate short and specific answer to the question that is asked in the
natural language in a specific domain. We have achieved 94 % accuracy of natural language question
answering in our implementation
Classifying Reading Behaviours using Deep Learning Methods with Eye-Tracking Data.
This work is interesting in identifying four reading behaviors: detailed-reading, non-reading, skimming, and scanning, by implementing three deep learning models – deep neural network (DNN), convolutional neural network (CNN), and recurrent neural networks (RNN), with eye-tracking data.
Replying to the findings, this paper proposes an idea to categorize reading behaviors by applying deep learning algorithms on eye-tracking data. More specifically, four activities – detailed-reading, non-reading, skimming, and scanning are classified by several deep learning algorithms listing as DNN, CNN, and RNN. Consequently, the work answers two research questions about which models and data type provide the highest accuracy when classifying reading behaviors.
Predicting User Knowledge Gain in Informational Search SessionsRan Yu
Slides of our SIGIR 2018 paper "Predicting User Knowledge Gain in Informational Search Sessions", which is presented in Ann Arbor, MI, US on July 9th, 2018.
A presentation that I gave at the Query Log Analysis: From Research to Best Practice Workshop 27 - 28 May 20098 in London, UK http://ir.shef.ac.uk/cloughie/qlaw2009/index.html
presents the foundational aspects of web analytics and some specifics such as the hotel problem. Discusses trace data, behaviorism, and other cool web analytics stuff
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024Neo4j
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
Learning about Information Searchers from Eye-Tracking by Jacek Gwizdka
1. Jacek Gwizdka Department of Library and Information Science School of Communication and Information Rutgers University Monday, April 4, 2011 Learning about Information Searchers from Eye-Tracking CONTACT: www.jsg.tel
2. Outline Overall research goals Eye-tracking – fundamentals Eye-fixation patterns: reading models (Exp 1; Exp 3) Search results presentation and cognitive abilities (Exp 2) Summary and Challenges 2
3. Overall Research Goals Characterization and enhancement of human information interaction mediated by computing technology Characterization: cognitive and affective user states –traditionally: little access to the mental/emotional states of users while they are engaged in the search process Implicit data collection about searchers’ cognitive and affective states in relation to information search phases Enhancement: Personalization and Adaptation 3
4. Example: Implicit Characterization of Cognitive Load on Web Search 4 higher average cognitive load: Q & B 35% 27% higher peak cognitive load: C START Q formulate query Lview search results list B bookmark page Cview content page END 97% 58% 30% 42% 7% 95% (Gwizdka, JASIST, 2010)
5. Eye-Tracking? Early attempts late XIX c.; early 1950’s - using a movie camera and hand-coding (Fitts, Jones & Milton 1950) Now computerized and “easy to use” infrared light sources and cameras stationary and mobile 5 Current Tobii eye-trackers
6. Eye-tracking – fundamental assumptions Top-down vs. bottom-up control in between: language processing (higher-level) controls when eyes move, visual processing (lower-level) controls where eyes move(Reichle et al., 1998) Eye-mind link hypothesis: attention is where eyes are focused (Just & Carpenter, 1980; 1987) Overt and covert attention Attention can move with no eye movement BUT eyes cannot move without attention 6
7. Data from Eye-tracking Devices eye gaze points eye gaze points in screen coordinates + distance eye fixations in screen coordinates + validity pupil diameter [head position 3D, distance from monitor] 50/60Hz; 300Hz; 1000-2000Hz eye-trackers common: 60Hz: one data record every 16.67ms 7 Tobii T-60 eye-tracker
8. Eye-Tracking Can … Eye tracking can allow identification of the specific content acquired by the person from Web pages Eye tracking enables high resolution analysis of searcher’s activity during interactions with information systems And more… 8 Example: composing answer and from information on a Web page (video)
9. Related Work in Information Science Interaction with search results Interaction with SERPs (Granka et al., 2004; Lorigo et al., 2007; 2008) Effects results presentation (Cutrell et al., 2007; Kammerer al., 2010) Relevance detection (Buscher, et al. 2009) Implicit Feedback (Fu, X., 2009); Query expansion (Buscher, et al. 2009) Relevance detection Pupillometry (Oliveira, Aula, Russell, 2009) Detection of task differences from eye-gaze patterns Reading/reasoning/search/object manipulation (Iqbal & Bailey, 2004) Informational vs. transactional tasks (Terai , et al., 2008) Task detection is also one of our research interests 9
14. Experiment 1 – Research Questions Can we detect task type (differences in task facets) from implicit interaction data (e.g., eye-tracking) ? How do we aggregate information from eye-tracking data? 11
15.
16.
17. Scan Fixations vs. Reading Fixations Scanning fixations provide some semantic information, limited to foveal(1° visual acuity) visual field (Rayner & Fischer, 1996) Fixations in a reading sequence provide more information than isolated “scanning” fixations: information is gained from the larger parafoveal (5° beyond foveal focus) region (Rayner et al., 2003) (asymmetrical, in dir of reading) richer semantic structure available from text compositions (sentences, paragraphs, etc.) Some of the types of semantic information available only through reading sequences may be crucial to satisfy task requirements. 14
18. Reading Models We implemented the E-Z Reader reading model (Reichle et al., 2006) Inputs: (eye fixation location, duration) Fixation duration >113 ms – threshold for lexical processing (Reingold & Rayner, 2006) The algorithm distinguishes reading fixation sequences from isolated fixations, called 'scanning' fixations Each lexical fixation is classified to (S,R) (Scan, Reading) These sequences used to create a state model 15
19. Reading Model – States and Characteristics Two states: transition probabilities Number of lexical fixations and duration 16
22. For CPE to continue scanningSearchers are adopting different reading strategies for different task types (Cole, Gwizdka, Liu, Bierig, Belkin & Zhang, 2010)
23. Results: Search Task Facets and Text Acquisition For highly attended pages 19 Total Text Acquisition on SERPs and Content per page Total Text Acquired on SERPs and Content
24. Results: Search Task Facets and State Transitions For highly attended pages 20 Read Scan Scan Read Scan Read Read Scan State Transitions on Content pages per page State Transitions on SERPs per page
26. Scan<->Read Transition Probabilities in 2 Experiments Person’s tendency to readscan related to scanread? (i.e., is p related to q ?) p ~ 1-q Genomics tasks (N=40) Journalistic tasks (N=32) correlation (Spearman ρ): 0.914 and 0.830
27. Experiment 1: Conclusions Searchers’ reading / scanning behavior affected by task Tasks facets can be “detected” from eye-tracking data (from reading model properties) Reading models can be built on the fly (during search) real-time observations of eye movements can be used by adaptive search systems Challenge: Lack of baseline data about reading models of individuals 23
28. Experiment 2: Result List vs. Overview Tag-Cloud 37 participants Everyday information seeking tasks (travel, shopping…) - two levels of task complexity Two user interfaces 24 2. Overview UI (Tag Cloud) 1. List UI
29. Experiment 2: User Actions in Two Interfaces 25 1. List 2. Overview Tag Cloud
30. Experiment 2: Research Questions Does the search results overview benefit users? Task effects? Individual differences - cognitive ability effects? 26
31. General Results Search results overview (“tag cloud”) benefited users made them faster facilitated formulation of more effective queries More complex tasks were indeed more demanding – required more search effort 27 (Gwizdka, Information Research, 2009)
32. Task and UI and Reading Model differences Complex tasks required more reading effort Longer max reading fixation length and more reading fixation regressions Overview UI required less effort Scanning more likely (S-S higher; S-R lower; R-S higher) Total reading scan path length shorter but total scan path (including scanning) were longer Less and shorter mean fixations per page visited 28 List Overview
33. Task and UI Interaction and Reading model data For complex tasks UI effect Higher probability of short reading sequences in Overview UI For simple tasks UI effect Shorter length of reading scan paths per page and less fixations per page Task & UI interaction Speed of reading: for complex tasks faster reading in Overview than in List UI for simple tasks faster in List than in Overview UI 29
35. Individual Differences – Least Effort? Higher cognitive ability searchers were faster in Overview UI and on simple tasks (same number of queries) Higher ability searchers did more in more demanding situations higher search effort did not seem to improve task outcomes 31 For task complexity factor and working memory (WM) F(144,1)=4.2; p=.042 F(144,1)=3.1; p=.08
36. Task and Working Memory – Eye-tracking Data High WM less likely to keep scanning High WM higher reading speed (scan path/total fixation duration) Number and duration of reading sequences differs (borderline: 0.05<p<0.1) For high WM searchers: for complex more reading for simple tasks less reading For low WM no such difference! 32
37. Experiment 2: Conclusions Overview UI was faster – reflected in some eye-tracking measures Task complexity differences reflected in some eye-tracking measures Some effects of cognitive abilities on interaction e.g., task & high WM – more effort than needed opportunistic discovery of information? “violation” of the least effort principle not fully explained yet 33
40. Can we detect when searchers make information relevance decisions?Emotiv EPOC wireless EEG headset EEG Start with pupillometry info relevance (Oliveria, Russell, Aula, 2009) low-level decision timing (Einhäuser, et al. 2010) Also look at EEG, GSR Funded by Google Research Award pupil animation Eye tracking Tobii T-60 eye-tracker GSR
41. Summary & Conclusions Eye tracking enables high resolution analysis of searcher’s activity during interactions with information systems There is more beyond eye-gaze locations with timestamps Eye-tracking data: can support for identification of search task types reflects differences in searcher performance on user interfaces reflects individual differences between searchers High potential for implicit detection of a searcher’s states 36
42. Some Challenges High-resolution data (low-level) How do we create higher-level patterns? How do we detect them computationally? How do we deal with ind. diffs(baseline data)? 37 (Iqbal & Bailey, 2004) (Terai et al., 2008) (Lorigo et al., 2008)
43. High-resolution Eye-tracking is Coming Soon to You Eye tracking technology is declining in price and in 2-3 years could be part of standard displays. Already in luxury cars and semi-trucks (sleep detection) Computers with built in eye-tracking 38 Tobii / Lenovo proof of concept eye-tracking laptop - March 2011
44. Thank you! Questions? Jacek Gwizdka contact: http://jsg.tel PoODLEProject: Personalization of the Digital Library Experience IMLS grant LG-06-07-0105-07 http://comminfo.rutgers.edu/research/poodle or for short: http://bit.ly/poodle_project PoODLE PIs: Nicholas J. Belkin, Jacek Gwizdka, Xiangmin Zhang Post-Doc: Ralf Bierig, PhD Students: Michael Cole (Reading Models + E-Z Reader algorithm), Jingjing Liu, (now Asst Prof.), Chang Liu
Editor's Notes
Tasks varied in several dimensions: complexity defined as the number of necessary steps needed to achieve the task goal (e.g. identifying an expert and then finding their contact information), the task product (factual vs. intellectual, e.g. fact checking vs. production of a document), the information object (a complete document vs. a document segment), andthe nature of the task goal (specific vs. amorphous).
Tasks varied in several dimensions: complexity defined as the number of necessary steps needed to achieve the task goal (e.g. identifying an expert and then finding their contact information), the task product (factual vs. intellectual, e.g. fact checking vs. production of a document), the information object (a complete document vs. a document segment), andthe nature of the task goal (specific vs. amorphous).
Eye tracking work on reading behavior in information search have mostly analyzed eye gaze position aggregates ('hot spots').This does not address the fixation sub-sequences that are true reading behavior.
Reading models can be built on the fly They only requires analysis of the recent eye movement sequence to classify the observed fixations