Jacek Gwizdka & Michael J. Cole      Dept. of Library and Information Science, Rutgers University                        N...
!!   Overall research goal: infer and predict mental     states and context of a person engaged in     interactive informa...
!!   Methodology: Using eye-gaze patterns     !! Higher-order patterns: Reading Models     !! Measures of cognitive effort...
!!   Eye-tracking research have     frequently analyzed eye-     gaze position aggregates     (hot spots’)     !! spatiote...
!!   We have developed a new methodology to analyze     eye-gaze patterns:     !! Model the reading process to represent (...
Can be represented as units of reading experience:       ((F F F) F (F F F) F F F F (F F F F F F) F)                      ...
1.! Eye movements are cognitively controlled     (Findlay &      Gilchrist, 2003)2.! Eyes fixate until cognitive processin...
!!   We implemented the E-Z Reader reading model     (Reichle et al., 2006)     !! Fixation duration >113 ms – threshold f...
!!   Two states: reading and scanning     !! transition probabilities     !! each state characterized by the number of lex...
Can be represented as units (Fixations) of reading experience:              (F F F) F (F F F) F F F F (F F F F F F) FUsing...
13
!!   Eyes fixate until cognitive processing is completed     (Rayner 1998)!!   While reading, words already understood in ...
text acquired!!   Reading Speed = -----------------                      processing time!!   Perceptual Span - Average spa...
1o (70px) foveal region!!   Reading speed will be slower for:      !! hard to read text (Rayner & Pollatsek, 1989),      !...
Perceptual span is the spacing of fixationsPerceptual span reflects a human limitation on the numberand difficulty of conc...
!!   10-15% of fixations are regressions!! Reading goal affects reading regressions!! More regressions when:     !! greate...
!!   Greater LFDE indicates less familiar words &     greater conceptual complexity!!   LFDE is also correlated with estab...
20
!!   32 journalism students!!   4 journalistic tasks         (realistic, created by journalism faculty                    ...
!!   Complexity - number of steps needed (ex: identify an expert, get     contact information)!!   Task Product (factual v...
!!   User task characteristics     !! Can we detect task characteristics from eye-gaze patterns ?!!   Cognitive effort    ...
24
!!   Task effects on transition probabilities S!R & R!S       (all subjects & pages)             OBI: advanced obituary   ...
OBI: advanced obituary                                        INT: interview preparation!!   For highly attended pages    ...
OBI: advanced obituary                                            INT: interview preparation!!   For highly attended pages...
Task                               Product        Level       Named         Goal       Complexity   Background BIC        ...
(self-rated after the task)                 BIC          CPE   INT   OBI
!! Search        effort: task time, pages visited, queries entered  !! Copy Editing (CPE) required the least effort of all...
32
!!   …         33
!!   …         34
!!   …         35
!!   Cognitive effort measures seem valid!!   Eye gaze pattern cognitive effort measures match     with subjective task di...
37  37
!!   Words are indicative of concepts and concept     features!!   Reading involves:     !! knowledge used to understand w...
!!   Does user’s knowledge influence information search     behavior?!!   Is cognitive effort related to domain knowledge?...
!!   40 undergraduate and graduate students!!   Rated 409 genetics and genomics MeSH terms     !! 1: No knowledge, ... to ...
!! Participants’             domain knowledge (PDK) was represented by sum of term ratings  !! participants rated MeSH ter...
These cognitive effort measures were individuallycorrelated with level of domain knowledge.For all reading sequences:!!   ...
!!   For long reading sequences!!   We used random forests to construct regression     models from the cognitive effort me...
Random forest model classification errors            all participants   PDKgroups!      low!   inter!   high!             ...
!!   Ability to detect knowledge levels indicates a     possibility of real-time detection of learning of a     new materi...
!!   Eye tracking enables high resolution analysis of     searcher’s activity during interactions with     information sys...
!!   The reading model methodology and cognitive effort     measures are based on many years of empirical     research.   ...
!!   Processing requirements are low - just need fixation     location and duration.     !! Only recent eye movements are ...
50
•! Implicit characterization of Information Search Process using   physiological devices•! Can we detect when searchers ma...
!!   Jacek Gwizdka http://jsg.tel!!   Acknowledgements     !! Funding: IMLS                        Google     !! Collabora...
!!   Eye tracking technology is declining in price and in     2-3 years could be part of standard displays.     !! Already...
Inferring Cognitive States from Multimodal Measures in Information Science
Inferring Cognitive States from Multimodal Measures in Information Science
Inferring Cognitive States from Multimodal Measures in Information Science
Inferring Cognitive States from Multimodal Measures in Information Science
Upcoming SlideShare
Loading in …5
×

Inferring Cognitive States from Multimodal Measures in Information Science

497 views

Published on

PapePresentation

  • Be the first to comment

  • Be the first to like this

Inferring Cognitive States from Multimodal Measures in Information Science

  1. 1. Jacek Gwizdka & Michael J. Cole Dept. of Library and Information Science, Rutgers University New Brunswick, NJ, USAWorkshop on Inferring Cognitive and Emotional States from Multimodal Measures – MMCogEmS’2011 November 17, 2011
  2. 2. !! Overall research goal: infer and predict mental states and context of a person engaged in interactive information search (e.g., Web search)!! Completed projects: measures derived from eye- gaze patterns !! eye-movement patterns and interaction logs to infer "! task characteristics "! dynamic user states (such as cognitive load/effort) "! persistent user characteristics (such as domain knowledge)!! On-going projects: multi-modal measures !! eye-tracking + EEG + GSR !! cognitive load + timing of relevance decisions
  3. 3. !! Methodology: Using eye-gaze patterns !! Higher-order patterns: Reading Models !! Measures of cognitive effort in reading!! Results: !! User study I: journalistic search tasks "! task characteristics "! cognitive effort !! User study II: genomics search tasks "! cognitive effort (& learning) "! domain knowledge!! On-going work 3
  4. 4. !! Eye-tracking research have frequently analyzed eye- gaze position aggregates (hot spots’) !! spatiotemporal-intensity – heat maps !! also sequential – scan paths! Higher-order patterns: reading models & derived measures 5
  5. 5. !! We have developed a new methodology to analyze eye-gaze patterns: !! Model the reading process to represent (textual) information acquisition in search !! Measure the cognitive effort due to (textual) information acquisition !! Use both to correlate / infer higher-level constructs (task characteristics, user knowledge, etc.)
  6. 6. Can be represented as units of reading experience: ((F F F) F (F F F) F F F F (F F F F F F) F) F = fixation 7
  7. 7. 1.! Eye movements are cognitively controlled (Findlay & Gilchrist, 2003)2.! Eyes fixate until cognitive processing is completed (Rayner, 1998)Eye gaze pattern analysis is powerful:!! Eye gaze is only way to acquire (textual) information!! 1. + 2. ! Direct causal connection between observable (text) information search behavior and user’s mental state
  8. 8. !! We implemented the E-Z Reader reading model (Reichle et al., 2006) !! Fixation duration >113 ms – threshold for lexical processing (Reingold & Rayner, 2006) !! The algorithm distinguishes reading fixation sequences from isolated fixations, called scanning fixations !! Each lexical fixation is classified to (S,R) that is (Scan, Reading)!! Inputs: eye gaze location, duration!! Add fixation to reading sequence if next saccade: !! on the same line of text !! and less than 120 pixels to the right !! or is a regression on the same line of text 9
  9. 9. !! Two states: reading and scanning !! transition probabilities !! each state characterized by the number of lexical fixations and duration q p Read Scan 1-p 1-q 11
  10. 10. Can be represented as units (Fixations) of reading experience: (F F F) F (F F F) F F F F (F F F F F F) FUsing the reading model : Reading state – R (green); Scanning state – S: RSRSSSSRS 12
  11. 11. 13
  12. 12. !! Eyes fixate until cognitive processing is completed (Rayner 1998)!! While reading, words already understood in the parafoveal region are skipped (Reichle, et al., 2006)!! Eye gaze patterns depend on cognitive processing of information that is being acquired!! Hypothesis: Analysis of reading fixation patterns reveal some aspects of cognitive effort 14
  13. 13. text acquired!! Reading Speed = ----------------- processing time!! Perceptual Span - Average spacing of fixations!! Lexical Fixation Duration Excess (LFDE): !! Time needed to acquire meaning above the minimum for lexical access!! Fixation Regressions - Number of regression fixations in the reading sequence 15
  14. 14. 1o (70px) foveal region!! Reading speed will be slower for: !! hard to read text (Rayner & Pollatsek, 1989), !! unfamiliar words (Williams & Morris, 2004), !! words used in less frequent senses (Sereno, O’Donnell, & Rayner, 2006), !! more complex concepts (Morris, 1994) 16
  15. 15. Perceptual span is the spacing of fixationsPerceptual span reflects a human limitation on the numberand difficulty of concepts that can be processed (e.g. Pollatseket al. 1986). 17
  16. 16. !! 10-15% of fixations are regressions!! Reading goal affects reading regressions!! More regressions when: !! greater reader domain expertise, !! conceptually complex & difficult text passages, !! resolution of ambiguous (sense) words 18
  17. 17. !! Greater LFDE indicates less familiar words & greater conceptual complexity!! LFDE is also correlated with establishing word meaning in context example 19
  18. 18. 20
  19. 19. !! 32 journalism students!! 4 journalistic tasks (realistic, created by journalism faculty and journalists)!! Journalism tasks can be about any topic, but few task types.!! Tasks designed to vary in ways that affect search behavior (Li, 2009)!! Task difficulty was post-self-rated by participants (7- point Likert scale: ’very easy’ to ’extremely difficult’) 21
  20. 20. !! Complexity - number of steps needed (ex: identify an expert, get contact information)!! Task Product (factual vs. intellectual, e.g., fact checking vs. production of a document)!! Named - is actual search target specified?!! Level - the information object to process (a complete document vs. a document segment)!! Task Goal - the nature of the task goal (specific vs. amorphous) Task Product Level Named Goal Complexity Background BIC mixed Document No Specific High Copy Editing CPE factual Segment Yes Specific Low Interview Preparation INT mixed Document No Mixed A,S Low Advance Obituary OBI factual Document No Amorphous High!! Note: Copy Editing CPE & Advance Obituary OBI are most dissimilar!! Copy Editing is expected to be easiest, Advance Obituary most difficult
  21. 21. !! User task characteristics !! Can we detect task characteristics from eye-gaze patterns ?!! Cognitive effort !! Do the cognitive effort measures correlate with: "! task properties expected to contribute to task difficulty? "! the effort needed to complete the task? "! user judgment of task difficulty? 23
  22. 22. 24
  23. 23. !! Task effects on transition probabilities S!R & R!S (all subjects & pages) OBI: advanced obituary INT: interview preparation CPE: copy editing BIC: background information •! For OBI, INT searchers biased to continue reading •! For CPE to continue scanning Searchers are adopting different reading strategies for different task types(Cole, Gwizdka, Liu, Bierig, Belkin & Zhang, ECCE 2010; IwC 2011) 25
  24. 24. OBI: advanced obituary INT: interview preparation!! For highly attended pages CPE: copy editing BIC: background information Total Text Acquired on Total Text Acquired on SERPs and Content SERPs and Content per page 26
  25. 25. OBI: advanced obituary INT: interview preparation!! For highly attended pages CPE: copy editing BIC: background information Read ! Scan! Read Read ! Scan! Read Scan Scan State Transitions State Transitions on on SERPs per page Content pages per page 27
  26. 26. Task Product Level Named Goal Complexity Background BIC mixed Document No Specific High Copy Editing CPE factual Segment Yes Specific Low Interview Preparation INT mixed Document No Mixed A,S Low Advance Obituary OBI factual Document No Amorphous High Measure Related Task Characteristics level: document; goal other bias to read Task level than specific (OBI & INT) Number of For all and task state transitions level: segment and task pages bias to scan goal goal: specific (CPE) Task complexity: More text acquired in Total text acquired on SERPs BIC and OBI Text acquired and number of Task level: segment and task product: state transitions per page on factual (CPE) For content pages highly attended pagesCole, Gwizdka, Liu, Bierig, Belkin & Zhang. (2011). Task and User Effects on Reading Patterns in Information28Search Interacting with Computers 23(4), 346 – 362.
  27. 27. (self-rated after the task) BIC CPE INT OBI
  28. 28. !! Search effort: task time, pages visited, queries entered !! Copy Editing (CPE) required the least effort of all tasks !! Advance Obituary (OBI) required overall most effort (although not the greatest effort of the tasks for every effort measure)!! Forall tasks, for both greater perceived difficulty (self- ratings) and search task effort: !! higher median LFDE (Kruskal-Wallis chi-squared =125.02, p = 0.03) !! slower reading speed (ANOVA F-value=5.5 p=0.02) !! Strongest correlations obtained when considering only the single longest reading sequence on a page
  29. 29. 32
  30. 30. !! … 33
  31. 31. !! … 34
  32. 32. !! … 35
  33. 33. !! Cognitive effort measures seem valid!! Eye gaze pattern cognitive effort measures match with subjective task difficulty!! Cognitive effort measure results correlate with task characteristics related to task effort !! e.g. Complex tasks, amorphous goals 36
  34. 34. 37 37
  35. 35. !! Words are indicative of concepts and concept features!! Reading involves: !! knowledge used to understand words, !! processing concepts expressed in the content, and !! acquisition of information (and concepts) from the content!! User knowledge controls interaction during search: !! selects the words to read, and !! imposes cognitive processing demands to understand the concepts associated with the words
  36. 36. !! Does user’s knowledge influence information search behavior?!! Is cognitive effort related to domain knowledge? 39
  37. 37. !! 40 undergraduate and graduate students!! Rated 409 genetics and genomics MeSH terms !! 1: No knowledge, ... to 5: Can explain to others!! Five tasks from 2004 TREC Genomics track!! Tasks were hard!!! We use the same methodology to create reading models and calculate cognitive effort measures as in study I
  38. 38. !! Participants’ domain knowledge (PDK) was represented by sum of term ratings !! participants rated MeSH terms !! normalized by a hypothetical expert •! ki is the term knowledge rating (1-5) •! i ranges over all terms •! ti is 1 if rated or 0 if not •! m number of terms rated by a participant •! The sum is normalized by a hypothetical expert who rated all terms as can explain to others
  39. 39. These cognitive effort measures were individuallycorrelated with level of domain knowledge.For all reading sequences:!! higher domain knowledge ~ lower cognitive effort!! perceptual span (Kruskal-Wallis "2 = 4734.254, p < 2.2e-16)!! LFDE (Kruskal-Wallis "2 = 5570.103, p < 2.2e-16)!! reading speed (Kruskal-Wallis "2 = 5570.103, p < 2.2e-16)Similar correlations found for long reading sequences Long reading sequences might better reflect concept use by participants during information acquisition because of the attention allocated to acquiring that text.
  40. 40. !! For long reading sequences!! We used random forests to construct regression models from the cognitive effort measures!! Regression results were clustered (agglomerate hierarchical clustering)!! Random forest model gave us relative importance of cognitive effort measures as contributing variables in a predictive model !! high importance: reading length (px), LDFE, total duration of reading sequences (sum of lexical fix), perceptual span !! less important: number of regressions
  41. 41. Random forest model classification errors all participants PDKgroups! low! inter! high! low! 8! 0! 0! intermediate! 1! 23! 0! high! 0! 0! 6!only native English speakers PDKgroups! low! inter! high! low! 3! 0! 0! intermediate! 0! "#! 0! high! 0! 0! $!Random forest model cog effort !domain knowledge correlationwith MeSH based domain knowledge
  42. 42. !! Ability to detect knowledge levels indicates a possibility of real-time detection of learning of a new material (new domain)!! Task “phase” analysis: beginning, middle, end !! same random forest model across the three phases !! significant difference : LFDE drops from beg to mid to end phase, while !! numFix -- not significantly different between phases !! and readingLength increased from middle to end (sig: Kruskal- Wallis chi^2 = 885.2262, df = 817, p < 0.05)!! Possible evidence for learning ?
  43. 43. !! Eye tracking enables high resolution analysis of searcher’s activity during interactions with information systems!! There is more beyond eye-gaze locations with timestamps!! Eye-tracking data: !! can be used to identification of task characteristics !! … cognitive effort !! … domain knowledge!! High potential for implicit detection of a searcher’s states 47
  44. 44. !! The reading model methodology and cognitive effort measures are based on many years of empirical research. !! Eye movements have a direct causal connection to the information acquisition process. !! This connection is not mediated!!! Domain independent !! Document content is not involved!! Culturally and individually independent !! Method represents the users experience of the information acquisition process!! Real-time modeling of user domain knowledge is possible
  45. 45. !! Processing requirements are low - just need fixation location and duration. !! Only recent eye movements are needed to calculate cognitive effort. !! Real-time assessment of cognitive effort !! Early task session detection of user properties, e.g. domain knowledge and perception of task difficulty!! Soon enough for a system to make a difference in providing user support
  46. 46. 50
  47. 47. •! Implicit characterization of Information Search Process using physiological devices•! Can we detect when searchers make information relevance decisions?!! Start with eye- Eye Emotiv EPOC wireless EEG tracking: tracking headset pupillometry EEG !! info relevance (Oliveria, Russell, Aula, 2009) !! low-level decision timing (Einhäuser, et al. 2010) Tobii T-60!! Adds EEG, GSR eye-tracker!! Funded by Google Research Award GSR 51
  48. 48. !! Jacek Gwizdka http://jsg.tel!! Acknowledgements !! Funding: IMLS Google !! Collaborators: "! Dr. Nicholas J. Belkin, Dr. Xiangmin Zhang "! Post-Doc: Dr. Ralf Bierig "! Collaborator & PhD student: Michael Cole "! PhD students: Chang Liu, Jingjing Liu "! Master students 52
  49. 49. !! Eye tracking technology is declining in price and in 2-3 years could be part of standard displays. !! Already in luxury cars and semi-trucks (sleep detection) !! Computers with built in eye-tracking Tobii / Lenovo proof of concept eye-tracking laptop - March 2011 53

×