The document discusses research into inferring cognitive and emotional states from eye tracking and other multimodal data during information searching tasks. It describes completed projects analyzing eye gaze patterns to infer task characteristics and user states/traits. Ongoing projects combine eye tracking with EEG and other physiological measures. The methodology section explains that eye movements reveal cognitive processing and are used to analyze reading behavior and infer effort. Results from user studies on journalistic and genomics search tasks show eye patterns can indicate task properties, cognitive effort and domain knowledge.
This is an update of a talk I originally gave in 2010. I had intended to make a wholesale update to all the slides, but noticed that one of them was worth keeping verbatim: a snapshot of the state of the art back then (see slide 38). Less than a decade has passed since then but there are some interesting and noticeable changes. For example, there was no word2vec, GloVe or fastText, or any of the neurally-inspired distributed representations and frameworks that are now so popular. Also no mention of sentiment analysis (maybe that was an oversight on my part, but I rather think that what we perceive as a commodity technology now was just not sufficiently mainstream back then).
Also if you compare with Jurafsky and Martin's current take on the state of the art (see slide 39), you could argue that POS tagging, NER, IE and MT have all made significant progress too (which I would agree with). I am not sure I share their view that summarisation is in the 'still really hard' category; but like many things, it depends on how & where you set the quality bar.
Ricardo Baeza-Yates, Luz Rello - Lexical Quality of Social Media - ICWSM - FO...Luz Rello
In this study we present an analysis of the lexical quality of social media in the Web, focusing on the Web 2.0, social networks, blogs and micro-blogs, multimedia and opinions. We find that blogs and social networks are the main players and also the main contributors to the bad lexical quality of the Web. We also compare our results with the rest of the Web finding that in general social media has worse lexical quality than the average Web and that their quality is one order of magnitude worse than high quality sites.
Ricardo Baeza-Yates, Luz Rello-On Measuring the Lexical Quality of the Web-WI...Luz Rello
In this paper we propose a measure for estimating the lexical quality of the Web, that is, the representational aspect of the textual web content. Our lexical quality measure is based in a small corpus of spelling errors and we apply it to English and Spanish. We first compute the correlation of our measure with web popularity measures to show that gives in- dependent information and then we apply it to different web segments, including social media. Our results shed a light on the lexical quality of the Web and show that authoritative websites have several orders of magnitude less misspellings than the overall Web. We also present an analysis of the geographical distribution of lexical quality throughout English and Spanish speaking countries as well as how this measure changes in about one year.
Theories in reading instruction
TOP-DOWN READING MODEL
Emphasizes what the reader brings to the text
Says reading is driven by meaning
Proceeds from whole to part
Views from some researchers
1. Frank Smith – Reading is not decoding written language to spoken language
2. reading is a matter of bringing meaning to print
FEATURES OF TOP-DOWN APPROACH
Readers can comprehend a selection even though they do not recognize each word.
Readers should use meaning and grammatical cues to identify unrecognized words.
Reading for meaning is the primary objective of reading, rather than mastery of letters, letters/sound relationships and words.
FEATURES OF TOP-DOWN APPROACH
Reading requires the use of meaning activities than the mastery of series of word- recognition skills.
The primary focus of instruction should be the reading of sentences, paragraphs, and whole selections
The most important aspect about reading is the amount and kind of information gained through reading.
BOTTOM UP
Emphasizes a single direction
Emphasizes the written or printed texts
Part to whole model
Reading is driven by a process that results in meaning
PROPONENTS OF THE BOTTOM UP
Flesch 1955
Gough 1985
FEATURES OF BOTTOM-UP
Believes the reader needs to:
Identify letter features
Link these features to recognize letters
Combine letter to recognize spelling patterns
Link spelling patterns to recognize words
Proceed to sentence, paragraph, and text- level processing
INTERACTIVE READING MODEL
It recognizes the interaction of bottom-up and top-down processes simultaneously throughout the reading process.
Reading as an active process that depends on reader characteristics, the text, and the reading situation (Rumelhart, 1985)
Attempts to combine the valid insights of bottom-up and top-down models.
PROPONENTS OF THE INTERACTIVE READING MODEL
Rumelhart, D. 1985
Barr, Sadow, and Blachowicz 1990
Ruddell and Speaker 1985
This is an update of a talk I originally gave in 2010. I had intended to make a wholesale update to all the slides, but noticed that one of them was worth keeping verbatim: a snapshot of the state of the art back then (see slide 38). Less than a decade has passed since then but there are some interesting and noticeable changes. For example, there was no word2vec, GloVe or fastText, or any of the neurally-inspired distributed representations and frameworks that are now so popular. Also no mention of sentiment analysis (maybe that was an oversight on my part, but I rather think that what we perceive as a commodity technology now was just not sufficiently mainstream back then).
Also if you compare with Jurafsky and Martin's current take on the state of the art (see slide 39), you could argue that POS tagging, NER, IE and MT have all made significant progress too (which I would agree with). I am not sure I share their view that summarisation is in the 'still really hard' category; but like many things, it depends on how & where you set the quality bar.
Ricardo Baeza-Yates, Luz Rello - Lexical Quality of Social Media - ICWSM - FO...Luz Rello
In this study we present an analysis of the lexical quality of social media in the Web, focusing on the Web 2.0, social networks, blogs and micro-blogs, multimedia and opinions. We find that blogs and social networks are the main players and also the main contributors to the bad lexical quality of the Web. We also compare our results with the rest of the Web finding that in general social media has worse lexical quality than the average Web and that their quality is one order of magnitude worse than high quality sites.
Ricardo Baeza-Yates, Luz Rello-On Measuring the Lexical Quality of the Web-WI...Luz Rello
In this paper we propose a measure for estimating the lexical quality of the Web, that is, the representational aspect of the textual web content. Our lexical quality measure is based in a small corpus of spelling errors and we apply it to English and Spanish. We first compute the correlation of our measure with web popularity measures to show that gives in- dependent information and then we apply it to different web segments, including social media. Our results shed a light on the lexical quality of the Web and show that authoritative websites have several orders of magnitude less misspellings than the overall Web. We also present an analysis of the geographical distribution of lexical quality throughout English and Spanish speaking countries as well as how this measure changes in about one year.
Theories in reading instruction
TOP-DOWN READING MODEL
Emphasizes what the reader brings to the text
Says reading is driven by meaning
Proceeds from whole to part
Views from some researchers
1. Frank Smith – Reading is not decoding written language to spoken language
2. reading is a matter of bringing meaning to print
FEATURES OF TOP-DOWN APPROACH
Readers can comprehend a selection even though they do not recognize each word.
Readers should use meaning and grammatical cues to identify unrecognized words.
Reading for meaning is the primary objective of reading, rather than mastery of letters, letters/sound relationships and words.
FEATURES OF TOP-DOWN APPROACH
Reading requires the use of meaning activities than the mastery of series of word- recognition skills.
The primary focus of instruction should be the reading of sentences, paragraphs, and whole selections
The most important aspect about reading is the amount and kind of information gained through reading.
BOTTOM UP
Emphasizes a single direction
Emphasizes the written or printed texts
Part to whole model
Reading is driven by a process that results in meaning
PROPONENTS OF THE BOTTOM UP
Flesch 1955
Gough 1985
FEATURES OF BOTTOM-UP
Believes the reader needs to:
Identify letter features
Link these features to recognize letters
Combine letter to recognize spelling patterns
Link spelling patterns to recognize words
Proceed to sentence, paragraph, and text- level processing
INTERACTIVE READING MODEL
It recognizes the interaction of bottom-up and top-down processes simultaneously throughout the reading process.
Reading as an active process that depends on reader characteristics, the text, and the reading situation (Rumelhart, 1985)
Attempts to combine the valid insights of bottom-up and top-down models.
PROPONENTS OF THE INTERACTIVE READING MODEL
Rumelhart, D. 1985
Barr, Sadow, and Blachowicz 1990
Ruddell and Speaker 1985
Enrique Vidal (Polytechnic University of Valencia, ES): Keyword Searching as a Trade-off between Recall and Precision. A new way to search large collections of digitised documents
co:op-READ-Convention Marburg
Technology meets Scholarship, or how Handwritten Text Recognition will Revolutionize Access to Archival Collections.
With a special focus on biographical data in archives
Hessian State Archives Marburg Friedrichsplatz 15, D - 35037 Marburg
19-21 January 2016
Our goal is to connect the knowledge base from cognitive development and neuroscience to practical knowledge about learning and teaching in educational environments. Grounding learning and teaching in research about learning, we have discovered a universal scale for learning – which greatly increases the power of assessments and makes possible the use of a common toolkit for learning sequences in any domain. In addition, we have been able to design on-line computer-based assessments that make assessment both less expensive and more convenient. The tests start with assessments that are connected to learning environments and can be used directly to promote and guide learning. Our goal is to move beyond using tests as sorting mechanisms and toward using them as powerful aids for education.
Graphic Organisers for L2 Reading
Presentation at JALT East Shikoku National Conference 2014
This presentation will report on a study that investigated the effects of direct instruction and guided practice in using graphic organisers as an expository text comprehension strategy in a college level EFL reading context. Participants in the study were two intact groups (n =21, n =31) of 1st year undergraduate engineers at Kochi University of Technology. The study utilised a mixed method research design, based on both quantitative and qualitative data collection and analysis techniques. The researcher examined if using graphic organisers (GOs) transferred into quantitative improvements in reading comprehension as measured through an 18-item multiple- choice test at the end of a 6-week study period, and if students perceived they had improved. Results showed that both groups had made quantitative improvements over the course of the study period; however the gains were significantly higher in the mapping group in non-parametric analysis (p < .05). Subjects in the treatment group also reported that the treatment had improved their overall reading skills and understanding of text organisation, and felt that using GOs was an effective strategy for L2 reading.
Deep Accessibility: Adapting Interfaces to Suit Our SensesSimon Harper
Citation:
@article{Harper2012uq,
Abstract = {Disabled people typically use methods of `sensory translation' to access a Web-page via assistive technology. These technologies conventionally render screen content under the direction of the user into a form that can be perceived by that user -- in effect the interface and content are adapted to suit their sensory requirements -- but simple sensory translation is not enough.
Why is this -- and how can things be better? In this talk we touch on accessibility, sensory transcoding, multi-talker systems, auditory perception, and Neuroscience to help us in our search for equivalent interactive experiences tailored to the sensory modality of the user.},
Author = {Simon Harper},
Date-Added = {2013-02-15 10:31:27 +0000},
Date-Modified = {2013-02-15 10:39:41 +0000},
Howpublished = {Slideshare},
Journal = {Invited Talk - Technical Superior Insitute, LaSIGE, Lisbon, Portugal},
Month = {September},
Title = {Deep Accessibility: Adapting Interfaces to Suit Our Senses - http://goo.gl/VT5BE},
Url = {\url{http://www.slideshare.net/simon-harper/adapting-sensory-interfaces}},
Year = {2012},
doi={10.6084/m9.figshare.678330},
Bdsk-Url-2 = {http://dx.doi.org/10.6084/m9.figshare.678330},
Bdsk-Url-1 = {http://www.slideshare.net/simon-harper/adapting-sensory-interfaces}}
Visual-Semantic Embeddings: some thoughts on LanguageRoelof Pieters
Language technology is rapidly evolving. A resurgence in the use of distributed semantic representations and word embeddings, combined with the rise of deep neural networks has led to new approaches and new state of the art results in many natural language processing tasks. One such exciting - and most recent - trend can be seen in multimodal approaches fusing techniques and models of natural language processing (NLP) with that of computer vision.
The talk is aimed at giving an overview of the NLP part of this trend. It will start with giving a short overview of the challenges in creating deep networks for language, as well as what makes for a “good” language models, and the specific requirements of semantic word spaces for multi-modal embeddings.
XIP Dashboard: Visual Analytics from Automated Rhetorical Parsing of Scient...Simon Buckingham Shum
XIP Dashboard: Visual Analytics from Automated Rhetorical Parsing of Scientific Metadiscourse
ABSTRACT
A key competency that we seek to build in learners is a critical mind, i.e. ability to engage with the ideas in the literature, and to identify when significant claims are being made in articles. The ability to decode such moves in texts is essential, as is the ability to make such moves in one’s own writing. Computational techniques for extracting them are becoming available, using Natural Language Processing (NLP) tuned to recognize the rhetorical signals that authors use when making a significant scholarly move. After reviewing related NLP work, we introduce the Xerox Incremental Parser (XIP), note previous work to render its output, and then motivate the design of the XIP Dashboard, a set of visual analytics modules built on XIP output, using the LAK/EDM open dataset as a test corpus. We report preliminary user reactions to a paper prototype of such a novel dashboard, describe the visualizations implemented to date, and present user scenarios for learners, educators and researchers. We conclude with a summary of ongoing design refinements, potential platform integrations, and questions that need to be investigated through end-user evaluations.
Development, distribution and use of open source software comprise a market of data (source code, bug reports, documentation, number of downloads, etc.) from projects, developers and users. This large amount of data makes it difficult for people involved to make sense of implicit links between software projects, e.g., dependencies, patterns, licenses. This context raises the question of what techniques and mechanisms can be used to help users and developers to link related pieces of information across software projects. In this paper, we propose a framework for a marketplace enhanced using linked open data (LOD) technology for linking software artifacts within projects as well as across software projects. The marketplace provides the infrastructure for collecting and aggregating software engineering data as well as developing services for mining, statistics, analytics and visualization of software data. Based on cross-linking software artifacts and projects, the marketplace enables developers and users to understand the individual value of components, their relationship to bigger software systems. Improved understanding creates new business opportunities for software companies: users will be better able to analyze and compare projects, developers can increase the visibility of their products, hosts may offer plug-ins and services over the data to paying customers.
More Related Content
Similar to Inferring Cognitive States from Multimodal Measures in Information Science
Enrique Vidal (Polytechnic University of Valencia, ES): Keyword Searching as a Trade-off between Recall and Precision. A new way to search large collections of digitised documents
co:op-READ-Convention Marburg
Technology meets Scholarship, or how Handwritten Text Recognition will Revolutionize Access to Archival Collections.
With a special focus on biographical data in archives
Hessian State Archives Marburg Friedrichsplatz 15, D - 35037 Marburg
19-21 January 2016
Our goal is to connect the knowledge base from cognitive development and neuroscience to practical knowledge about learning and teaching in educational environments. Grounding learning and teaching in research about learning, we have discovered a universal scale for learning – which greatly increases the power of assessments and makes possible the use of a common toolkit for learning sequences in any domain. In addition, we have been able to design on-line computer-based assessments that make assessment both less expensive and more convenient. The tests start with assessments that are connected to learning environments and can be used directly to promote and guide learning. Our goal is to move beyond using tests as sorting mechanisms and toward using them as powerful aids for education.
Graphic Organisers for L2 Reading
Presentation at JALT East Shikoku National Conference 2014
This presentation will report on a study that investigated the effects of direct instruction and guided practice in using graphic organisers as an expository text comprehension strategy in a college level EFL reading context. Participants in the study were two intact groups (n =21, n =31) of 1st year undergraduate engineers at Kochi University of Technology. The study utilised a mixed method research design, based on both quantitative and qualitative data collection and analysis techniques. The researcher examined if using graphic organisers (GOs) transferred into quantitative improvements in reading comprehension as measured through an 18-item multiple- choice test at the end of a 6-week study period, and if students perceived they had improved. Results showed that both groups had made quantitative improvements over the course of the study period; however the gains were significantly higher in the mapping group in non-parametric analysis (p < .05). Subjects in the treatment group also reported that the treatment had improved their overall reading skills and understanding of text organisation, and felt that using GOs was an effective strategy for L2 reading.
Deep Accessibility: Adapting Interfaces to Suit Our SensesSimon Harper
Citation:
@article{Harper2012uq,
Abstract = {Disabled people typically use methods of `sensory translation' to access a Web-page via assistive technology. These technologies conventionally render screen content under the direction of the user into a form that can be perceived by that user -- in effect the interface and content are adapted to suit their sensory requirements -- but simple sensory translation is not enough.
Why is this -- and how can things be better? In this talk we touch on accessibility, sensory transcoding, multi-talker systems, auditory perception, and Neuroscience to help us in our search for equivalent interactive experiences tailored to the sensory modality of the user.},
Author = {Simon Harper},
Date-Added = {2013-02-15 10:31:27 +0000},
Date-Modified = {2013-02-15 10:39:41 +0000},
Howpublished = {Slideshare},
Journal = {Invited Talk - Technical Superior Insitute, LaSIGE, Lisbon, Portugal},
Month = {September},
Title = {Deep Accessibility: Adapting Interfaces to Suit Our Senses - http://goo.gl/VT5BE},
Url = {\url{http://www.slideshare.net/simon-harper/adapting-sensory-interfaces}},
Year = {2012},
doi={10.6084/m9.figshare.678330},
Bdsk-Url-2 = {http://dx.doi.org/10.6084/m9.figshare.678330},
Bdsk-Url-1 = {http://www.slideshare.net/simon-harper/adapting-sensory-interfaces}}
Visual-Semantic Embeddings: some thoughts on LanguageRoelof Pieters
Language technology is rapidly evolving. A resurgence in the use of distributed semantic representations and word embeddings, combined with the rise of deep neural networks has led to new approaches and new state of the art results in many natural language processing tasks. One such exciting - and most recent - trend can be seen in multimodal approaches fusing techniques and models of natural language processing (NLP) with that of computer vision.
The talk is aimed at giving an overview of the NLP part of this trend. It will start with giving a short overview of the challenges in creating deep networks for language, as well as what makes for a “good” language models, and the specific requirements of semantic word spaces for multi-modal embeddings.
XIP Dashboard: Visual Analytics from Automated Rhetorical Parsing of Scient...Simon Buckingham Shum
XIP Dashboard: Visual Analytics from Automated Rhetorical Parsing of Scientific Metadiscourse
ABSTRACT
A key competency that we seek to build in learners is a critical mind, i.e. ability to engage with the ideas in the literature, and to identify when significant claims are being made in articles. The ability to decode such moves in texts is essential, as is the ability to make such moves in one’s own writing. Computational techniques for extracting them are becoming available, using Natural Language Processing (NLP) tuned to recognize the rhetorical signals that authors use when making a significant scholarly move. After reviewing related NLP work, we introduce the Xerox Incremental Parser (XIP), note previous work to render its output, and then motivate the design of the XIP Dashboard, a set of visual analytics modules built on XIP output, using the LAK/EDM open dataset as a test corpus. We report preliminary user reactions to a paper prototype of such a novel dashboard, describe the visualizations implemented to date, and present user scenarios for learners, educators and researchers. We conclude with a summary of ongoing design refinements, potential platform integrations, and questions that need to be investigated through end-user evaluations.
Development, distribution and use of open source software comprise a market of data (source code, bug reports, documentation, number of downloads, etc.) from projects, developers and users. This large amount of data makes it difficult for people involved to make sense of implicit links between software projects, e.g., dependencies, patterns, licenses. This context raises the question of what techniques and mechanisms can be used to help users and developers to link related pieces of information across software projects. In this paper, we propose a framework for a marketplace enhanced using linked open data (LOD) technology for linking software artifacts within projects as well as across software projects. The marketplace provides the infrastructure for collecting and aggregating software engineering data as well as developing services for mining, statistics, analytics and visualization of software data. Based on cross-linking software artifacts and projects, the marketplace enables developers and users to understand the individual value of components, their relationship to bigger software systems. Improved understanding creates new business opportunities for software companies: users will be better able to analyze and compare projects, developers can increase the visibility of their products, hosts may offer plug-ins and services over the data to paying customers.
Similar to Inferring Cognitive States from Multimodal Measures in Information Science (20)
Towards a Marketplace of Open Source Software Data
Inferring Cognitive States from Multimodal Measures in Information Science
1. Jacek Gwizdka & Michael J. Cole
Dept. of Library and Information Science, Rutgers University
New Brunswick, NJ, USA
Workshop on Inferring Cognitive and Emotional States from Multimodal Measures – MMCogEmS’2011
November 17, 2011
2. !! Overall research goal: infer and predict mental
states and context of a person engaged in
interactive information search (e.g., Web search)
!! Completed projects: measures derived from eye-
gaze patterns
!! eye-movement patterns and interaction logs to infer
"! task characteristics
"! dynamic user states (such as cognitive load/effort)
"! persistent user characteristics (such as domain knowledge)
!! On-going projects: multi-modal measures
!! eye-tracking + EEG + GSR
!! cognitive load + timing of relevance decisions
3. !! Methodology: Using eye-gaze patterns
!! Higher-order patterns: Reading Models
!! Measures of cognitive effort in reading
!! Results:
!! User study I: journalistic search tasks
"! task characteristics
"! cognitive effort
!! User study II: genomics search tasks
"! cognitive effort (& learning)
"! domain knowledge
!! On-going work
3
4.
5. !! Eye-tracking research have
frequently analyzed eye-
gaze position aggregates
('hot spots’)
!! spatiotemporal-intensity –
heat maps
!! also sequential – scan paths
! Higher-order patterns:
reading models & derived measures
5
6. !! We have developed a new methodology to analyze
eye-gaze patterns:
!! Model the reading process to represent (textual) information
acquisition in search
!! Measure the cognitive effort due to (textual) information
acquisition
!! Use both to correlate / infer higher-level constructs (task
characteristics, user knowledge, etc.)
7. Can be represented as units of reading experience:
((F F F) F (F F F) F F F F (F F F F F F) F)
F = fixation 7
8. 1.! Eye movements are cognitively controlled (Findlay &
Gilchrist, 2003)
2.! Eyes fixate until cognitive processing is completed
(Rayner, 1998)
Eye gaze pattern analysis is powerful:
!! Eye gaze is only way to acquire (textual)
information
!! 1. + 2. ! Direct causal connection between
observable (text) information search behavior and
user’s mental state
9. !! We implemented the E-Z Reader reading model
(Reichle et al., 2006)
!! Fixation duration >113 ms – threshold for lexical processing
(Reingold & Rayner, 2006)
!! The algorithm distinguishes reading fixation sequences from
isolated fixations, called 'scanning' fixations
!! Each lexical fixation is classified to (S,R) that is (Scan,
Reading)
!! Inputs: eye gaze location, duration
!! Add fixation to reading sequence if next saccade:
!! on the same line of text
!! and less than 120 pixels to the right
!! or is a regression on the same line of text
9
10.
11. !! Two states: reading and scanning
!! transition probabilities
!! each state characterized by the number of lexical fixations
and duration
q
p
Read Scan
1-p 1-q
11
12. Can be represented as units (Fixations) of reading experience:
(F F F) F (F F F) F F F F (F F F F F F) F
Using the reading model :
Reading state – R (green); Scanning state – S:
RSRSSSSRS
12
14. !! Eyes fixate until cognitive processing is completed
(Rayner 1998)
!! While reading, words already understood in the
parafoveal region are skipped (Reichle, et al., 2006)
!! Eye gaze patterns depend on cognitive processing
of information that is being acquired
!! Hypothesis: Analysis of reading fixation patterns
reveal some aspects of cognitive effort
14
15. text acquired
!! Reading Speed = -----------------
processing time
!! Perceptual Span - Average spacing of fixations
!! Lexical Fixation Duration Excess (LFDE):
!! Time needed to acquire meaning above the minimum for
lexical access
!! Fixation Regressions - Number of regression
fixations in the reading sequence
15
16. 1o (70px) foveal region
!! Reading speed will be slower for:
!! hard to read text (Rayner & Pollatsek, 1989),
!! unfamiliar words (Williams & Morris, 2004),
!! words used in less frequent senses (Sereno, O’Donnell, &
Rayner, 2006),
!! more complex concepts (Morris, 1994)
16
17. Perceptual span is the spacing of fixations
Perceptual span reflects a human limitation on the number
and difficulty of concepts that can be processed (e.g. Pollatsek
et al. 1986).
17
18. !! 10-15% of fixations are regressions
!! Reading goal affects reading regressions
!! More regressions when:
!! greater reader domain expertise,
!! conceptually complex & difficult text passages,
!! resolution of ambiguous (sense) words
18
19. !! Greater LFDE indicates less familiar words &
greater conceptual complexity
!! LFDE is also correlated with establishing word
meaning in context
example
19
21. !! 32 journalism students
!! 4 journalistic tasks (realistic, created by journalism faculty
and journalists)
!! Journalism tasks can be about any topic, but few
task types.
!! Tasks designed to vary in ways that affect search
behavior (Li, 2009)
!! Task difficulty was post-self-rated by participants (7-
point Likert scale: ’very easy’ to ’extremely difficult’)
21
22. !! Complexity - number of steps needed (ex: identify an expert, get
contact information)
!! Task Product (factual vs. intellectual, e.g., fact checking vs.
production of a document)
!! Named - is actual search target specified?
!! Level - the information object to process (a complete document vs.
a document segment)
!! Task Goal - the nature of the task goal (specific vs. amorphous)
Task Product Level Named Goal Complexity
Background BIC mixed Document No Specific High
Copy Editing CPE factual Segment Yes Specific Low
Interview Preparation INT mixed Document No Mixed A,S Low
Advance Obituary OBI factual Document No Amorphous High
!! Note: Copy Editing CPE & Advance Obituary OBI are most dissimilar
!! Copy Editing is expected to be easiest, Advance Obituary most difficult
23. !! User task characteristics
!! Can we detect task characteristics from eye-gaze patterns ?
!! Cognitive effort
!! Do the cognitive effort measures correlate with:
"! task properties expected to contribute to task difficulty?
"! the effort needed to complete the task?
"! user judgment of task difficulty?
23
25. !! Task effects on transition probabilities S!R & R!S
(all subjects & pages) OBI: advanced obituary
INT: interview preparation
CPE: copy editing
BIC: background information
•! For OBI, INT searchers
biased to continue
reading
•! For CPE to continue
scanning
Searchers are adopting
different reading
strategies for different
task types
(Cole, Gwizdka, Liu, Bierig, Belkin & Zhang, ECCE 2010; IwC 2011) 25
26. OBI: advanced obituary
INT: interview preparation
!! For highly attended pages CPE: copy editing
BIC: background information
Total Text Acquired on Total Text Acquired on
SERPs and Content SERPs and Content
per page 26
27. OBI: advanced obituary
INT: interview preparation
!! For highly attended pages CPE: copy editing
BIC: background information
Read ! Scan! Read
Read ! Scan! Read Scan
Scan
State Transitions State Transitions on
on SERPs per page Content pages per page
27
28. Task Product Level Named Goal Complexity
Background BIC mixed Document No Specific High
Copy Editing CPE factual Segment Yes Specific Low
Interview Preparation INT mixed Document No Mixed A,S Low
Advance Obituary OBI factual Document No Amorphous High
Measure Related Task Characteristics
level: document; goal other
bias to read Task level than specific (OBI & INT)
Number of For all
and task
state transitions level: segment and task pages
bias to scan goal
goal: specific (CPE)
Task complexity: More text acquired in
Total text acquired on SERPs
BIC and OBI
Text acquired and number of
Task level: segment and task product:
state transitions per page on
factual (CPE) For
content pages
highly
attended
pages
Cole, Gwizdka, Liu, Bierig, Belkin & Zhang. (2011). Task and User Effects on Reading Patterns in Information28
Search Interacting with Computers 23(4), 346 – 362.
31. !! Search
effort: task time, pages visited, queries
entered
!! Copy Editing (CPE) required the least effort of all tasks
!! Advance Obituary (OBI) required overall most effort (although not
the greatest effort of the tasks for every effort measure)
!! Forall tasks, for both greater perceived difficulty (self-
ratings) and search task effort:
!! higher median LFDE (Kruskal-Wallis chi-squared =125.02, p = 0.03)
!! slower reading speed (ANOVA F-value=5.5 p=0.02)
!! Strongest correlations obtained when considering only the
single longest reading sequence on a page
38. !! Words are indicative of concepts and concept
features
!! Reading involves:
!! knowledge used to understand words,
!! processing concepts expressed in the content, and
!! acquisition of information (and concepts) from the content
!! User knowledge controls interaction during search:
!! selects the words to read, and
!! imposes cognitive processing demands to understand the
concepts associated with the words
39. !! Does user’s knowledge influence information search
behavior?
!! Is cognitive effort related to domain knowledge?
39
40. !! 40 undergraduate and graduate students
!! Rated 409 genetics and genomics MeSH terms
!! 1: No knowledge, ... to 5: Can explain to others
!! Five tasks from 2004 TREC Genomics track
!! Tasks were hard!
!! We use the same methodology to create reading
models and calculate cognitive effort measures as in
study I
41. !! Participants’
domain knowledge (PDK) was
represented by sum of term ratings
!! participants rated MeSH terms
!! normalized by a hypothetical expert
•! ki is the term knowledge rating (1-5)
•! i ranges over all terms
•! ti is 1 if rated or 0 if not
•! m number of terms rated by a participant
•! The sum is normalized by a hypothetical expert
who rated all terms as 'can explain to others'
42.
43. These cognitive effort measures were individually
correlated with level of domain knowledge.
For all reading sequences:
!! higher domain knowledge ~ lower cognitive effort
!! perceptual span (Kruskal-Wallis "2 = 4734.254, p < 2.2e-16)
!! LFDE (Kruskal-Wallis "2 = 5570.103, p < 2.2e-16)
!! reading speed (Kruskal-Wallis "2 = 5570.103, p < 2.2e-16)
Similar correlations found for long reading sequences
Long reading sequences might better reflect concept use by participants
during information acquisition because of the attention allocated to acquiring
that text.
44. !! For long reading sequences
!! We used random forests to construct regression
models from the cognitive effort measures
!! Regression results were clustered
(agglomerate hierarchical clustering)
!! Random forest model gave us relative importance of
cognitive effort measures as contributing variables
in a predictive model
!! high importance: reading length (px), LDFE, total duration
of reading sequences (sum of lexical fix), perceptual span
!! less important: number of regressions
45. Random forest model classification errors
all participants PDKgroups! low! inter! high!
low! 8! 0! 0!
intermediate! 1! 23! 0!
high! 0! 0! 6!
only native English speakers PDKgroups! low! inter! high!
low! 3! 0! 0!
intermediate! 0! "#! 0!
high! 0! 0! $!
Random forest model cog effort !
domain knowledge correlation
with MeSH based domain knowledge
46. !! Ability to detect knowledge levels indicates a
possibility of real-time detection of learning of a
new material (new domain)
!! Task “phase” analysis: beginning, middle, end
!! same random forest model across the three phases
!! significant difference : LFDE drops from beg to mid to end
phase, while
!! numFix -- not significantly different between phases
!! and readingLength increased from middle to end (sig: Kruskal-
Wallis chi^2 = 885.2262, df = 817, p < 0.05)
!! Possible evidence for learning ?
47. !! Eye tracking enables high resolution analysis of
searcher’s activity during interactions with
information systems
!! There is more beyond eye-gaze locations with
timestamps
!! Eye-tracking data:
!! can be used to identification of task characteristics
!! … cognitive effort
!! … domain knowledge
!! High potential for implicit detection of a searcher’s
states
47
48. !! The reading model methodology and cognitive effort
measures are based on many years of empirical
research.
!! Eye movements have a direct causal connection to the
information acquisition process.
!! This connection is not mediated!
!! Domain independent
!! Document content is not involved
!! Culturally and individually independent
!! Method represents the user's experience of the information
acquisition process
!! Real-time modeling of user domain knowledge is possible
49. !! Processing requirements are low - just need fixation
location and duration.
!! Only recent eye movements are needed to calculate
cognitive effort.
!! Real-time assessment of cognitive effort
!! Early task session detection of user properties, e.g. domain
knowledge and perception of task difficulty
!! Soon enough for a system to make a difference in
providing user support
51. •! Implicit characterization of Information Search Process using
physiological devices
•! Can we detect when searchers make information relevance
decisions?
!! Start with eye- Eye
Emotiv EPOC
wireless EEG
tracking: tracking headset
pupillometry EEG
!! info relevance (Oliveria,
Russell, Aula, 2009)
!! low-level decision timing
(Einhäuser, et al. 2010)
Tobii T-60
!! Adds EEG, GSR eye-tracker
!! Funded by Google
Research Award
GSR
51
52. !! Jacek Gwizdka http://jsg.tel
!! Acknowledgements
!! Funding: IMLS Google
!! Collaborators:
"! Dr. Nicholas J. Belkin, Dr. Xiangmin Zhang
"! Post-Doc: Dr. Ralf Bierig
"! Collaborator & PhD student: Michael Cole
"! PhD students: Chang Liu, Jingjing Liu
"! Master students
52
53. !! Eye tracking technology is declining in price and in
2-3 years could be part of standard displays.
!! Already in luxury cars and semi-trucks (sleep detection)
!! Computers with built in eye-tracking
Tobii / Lenovo
proof of concept eye-tracking
laptop - March 2011
53