Multimodal Cognitive Load Assessment


Published on

A paper presented at IACSS09

Published in: Sports, Technology, Education
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Multimodal Cognitive Load Assessment

  1. 1. Multimodal Cognitive Load Assessment for Athletes Natalie Ruiz *, Bo Yin*, Damian Farrow^, Lyndell Bruce^, Ronnie Taib*, and Fang Chen* * NICTA – National ICT Australia ^ AIS – Australian Institute of Sport IACSS2009, Canberra
  2. 2. Skill Acquisition and Cognitive Load <ul><li>Cognitive Load Theory [Sweller et. al. 1988] </li></ul><ul><ul><li>Degree of mental demand while completing a task </li></ul></ul><ul><ul><li>Intrinsic, extraneous and germane load </li></ul></ul><ul><ul><li>Must not exceed capacity of working memory </li></ul></ul><ul><li>High cognitive load not conducive for learning </li></ul><ul><li>Aim: Assessment of cognitive load </li></ul><ul><ul><li>Monitor when load is too high or too low </li></ul></ul><ul><ul><li>Either extreme is problematic for skill acquisition and optimal performance </li></ul></ul><ul><ul><li>Performance != Cognitive load </li></ul></ul>
  3. 3. Impact of high cognitive load on human responses <ul><li>Working memory is shared by cognitive processes e.g. perception and production of responses </li></ul><ul><li>Resulting disturbance may not be perceptible to humans, but may be machine measurable </li></ul>Long-term memory Short-term memory Shared space (7±2) Visual processing (Visuospatial sketchpad) Linguistic processing (Phonological loop) Central executive Perception Response Muscular action Excitation + vocal tract configuration Gesture… Multi-sensory perception . . . . . . Disturbance e.g. latency, pitch, jittering
  4. 4. Behavioural Indicators of High Load <ul><li>Possible behavioural data sources for cognitive load assessment: </li></ul><ul><ul><li>Speech, </li></ul></ul><ul><ul><li>Mouse speed and pressure, </li></ul></ul><ul><ul><li>Linguistic or dialogue patterns, </li></ul></ul><ul><ul><li>Pen input, </li></ul></ul><ul><ul><li>Eye-gaze, </li></ul></ul><ul><ul><li>Manual gesture, etc… </li></ul></ul><ul><li>Advantages </li></ul><ul><ul><li>Implicit data collection </li></ul></ul><ul><ul><li>(Relatively) non-intrusive </li></ul></ul><ul><ul><li>Does not interfere with task flow </li></ul></ul><ul><ul><li>Can be collected/analysed in real time </li></ul></ul>
  5. 5. <ul><li>Netball AIS Decision Making Study </li></ul><ul><ul><li>Opportunistic data collection </li></ul></ul><ul><ul><li>Participants: 20 underage-expert subjects </li></ul></ul><ul><ul><li>Task: </li></ul></ul><ul><ul><ul><li>32 netball clips of 5-10s projected onto a wall </li></ul></ul></ul><ul><ul><ul><li>Competitive scenarios of varying complexity </li></ul></ul></ul><ul><ul><ul><li>Decision-making: who to pass, what kind of pass, confidence </li></ul></ul></ul><ul><ul><li>Available Data: </li></ul></ul><ul><ul><ul><li>Speech responses </li></ul></ul></ul><ul><ul><ul><li>Video of subject (front) </li></ul></ul></ul><ul><ul><ul><li>Eye-Gaze (crosshairs) </li></ul></ul></ul><ul><ul><ul><li>Performance/ Accuracy </li></ul></ul></ul><ul><ul><ul><li>Subjective ratings </li></ul></ul></ul>Exploratory Data Collection: Netball Decision Making
  6. 6. Analysis - Speech <ul><li>Hypothesis: </li></ul><ul><ul><li>Speech signal features will vary as cognitive load changes </li></ul></ul><ul><li>600 mins speech data </li></ul><ul><ul><li>120 mins effective speech </li></ul></ul><ul><li>Existing speech classification </li></ul><ul><li>system developed at NICTA </li></ul><ul><ul><li>At least 3 level classification (L/M/H) </li></ul></ul><ul><ul><li>Use half of the data for training </li></ul></ul><ul><ul><li>Use the other half for testing </li></ul></ul><ul><ul><li>Applications </li></ul></ul><ul><ul><ul><li>Traffic control, crisis mgt, etc... </li></ul></ul></ul><ul><ul><li>Accuracy in the range of 75%-80% </li></ul></ul><ul><ul><li>Details in the paper + references </li></ul></ul>
  7. 7. Results - Speech <ul><li>Analysis 1: Intrinsic Complexity (post hoc) </li></ul><ul><ul><li>Expert labelling of tasks based on possible choices, difficulty of pass, and pressure </li></ul></ul><ul><ul><li>Trained and tested six levels of load/difficulty; two-thirds training, one-third testing </li></ul></ul><ul><ul><li>~19% accuracy </li></ul></ul><ul><li>Analysis 2: Grouping Levels </li></ul><ul><ul><li>Grouped Levels 1&2 (Low) and 5&6 (High) to double the amount of data per model </li></ul></ul><ul><ul><li>Trained and tested 2 levels (Highest vs Lowest); </li></ul></ul><ul><ul><li>~65% accuracy </li></ul></ul><ul><li>Subjective ratings and performance scores </li></ul><ul><ul><li>Subjective rating scores only within bandwidth of 2-3 points on 7pt scale </li></ul></ul><ul><ul><li>Performance scores not as expected for the six load levels. </li></ul></ul><ul><li>Analysis 3: Best and Worst Performances </li></ul><ul><ul><li>3 best performing tasks (100% score) and 3 worst performing tasks (0% score) </li></ul></ul><ul><ul><li>Trained and tested 2 levels (High and Low) </li></ul></ul><ul><ul><li>56% accuracy  </li></ul></ul><ul><ul><li>Due to low granularity of scoring? </li></ul></ul>
  8. 8. Discussion and Future Work <ul><li>So far, best result for speech: 65% accuracy on High vs Low </li></ul><ul><li>Exploratory study served to highlight issues </li></ul><ul><ul><li>Surprising that results not due to small amounts of data </li></ul></ul><ul><ul><li>Necessary to administer tasks of extreme load levels (high and low) </li></ul></ul><ul><ul><li>Higher granularity in performance scores </li></ul></ul><ul><ul><li>Look into intrinsic load structures to quantify “designed load” for Netball DM tasks </li></ul></ul><ul><ul><li>Theory building: does poor task performance mean higher or lower load? </li></ul></ul><ul><li>Analysis of other measures yet to be done: </li></ul><ul><ul><li>Eye gaze, fixation and trajectory patterns, search strategies </li></ul></ul><ul><ul><li>Head movement, proxy of video frame movement </li></ul></ul><ul><ul><li>Manual gesture (manual annotation) </li></ul></ul>
  9. 9. Application Contexts – Sports <ul><li>Multimodal assessment of cognitive load in sports contexts be applied to, for example: </li></ul><ul><ul><li>Objective supplementary coaching tool </li></ul></ul><ul><ul><ul><li>Individualised training progressions </li></ul></ul></ul><ul><ul><ul><li>Individualised athlete assessment (own baselines) </li></ul></ul></ul><ul><ul><ul><li>Highly customisable to the type of skill being learned </li></ul></ul></ul><ul><ul><li>Ongoing performance monitoring </li></ul></ul><ul><ul><ul><li>Possible to assess individual athletes in team sports </li></ul></ul></ul><ul><ul><ul><li>Performance != experienced load level </li></ul></ul></ul><ul><ul><ul><li>In training and in the field. </li></ul></ul></ul><ul><ul><li>Talent identification programs </li></ul></ul>