Your SlideShare is downloading. ×
0
Predicting and Explaining Individual Performance in Complex Tasks Marsha Lovett, Lynne Reder, Christian Lebiere, John Rehl...
Multi-Tasking <ul><li>A single person can perform multiple tasks. </li></ul><ul><li>A single model should be able to captu...
<ul><li>A way to keep the multiple-constraint advantage offered by unified theories of cognition while making their develo...
<ul><li>ZERO </li></ul><ul><li>PARAMETER </li></ul><ul><li>PREDICTIONS! </li></ul>
Basic Goals of Project <ul><li>Combine best features of cognitive modeling </li></ul><ul><ul><li>Study performance in a dy...
How to predict task performance <ul><li>Estimate each individual’s processing parameters </li></ul><ul><ul><li>Measure ind...
Example: Memory Task Performance <ul><li>Fit task A to estimate individuals’ parameters </li></ul>
Zero-Parameter Predictions <ul><li>Plug those parameters into model of task B </li></ul>(Lovett, Daily, & Reder, 2000)
Challenges of Complex Tasks <ul><li>Modeling the target task is harder </li></ul><ul><li>More than one individual differen...
What about knowledge differences? <ul><li>Develop tasks that reduce their relevance </li></ul><ul><li>Train participants o...
Individual Differences in ACT-R <ul><li>Most ACT-R models don’t account for impact of individual differences on performanc...
ACT-R & Individual Differences M1, M2, M3, … W1, W2, W3, … P1, P2, P3, …
Overview of Talk <ul><li>Review tasks we are studying </li></ul><ul><li>Illustrate methodology </li></ul><ul><li>Highlight...
Modified Digit Span (MODS)
Modified Digit Span (MODS)
P/M Tasks <ul><li>In our earlier studies, initial training phase of target task was used to collect data on individuals’ p...
How to predict task performance <ul><li>Estimate each individual’s processing parameters </li></ul><ul><ul><li>Measure ind...
W affects Performance <ul><li>W is the ACT-R parameter for  source activation,  which impacts the degree to which activati...
Estimating W <ul><li>Model of MODS task is fit to individual’s MODS performance by varying W </li></ul><ul><li>Best fittin...
Estimating PM <ul><li>For simplicity, we estimated a combined PM parameter directly from each individual’s perceptual/moto...
Joint Distribution of W and P/M W and P/M are tapping distinct characteristics
ACT-R & Individual Differences M1,  M2 , M3, … W1,  W2 , W3, … P1,  P2 , P3, …
Specifics of our Approach <ul><li>Estimate each individual’s processing parameters </li></ul><ul><ul><li>Measure individua...
AMBR: Air Traffic Control Task <ul><li>Complex and dynamic task </li></ul><ul><li>Spatial and verbal aspects </li></ul><ul...
AMBR Task AC=aircraft, ATC=air traffice controller <ul><li>As ATC, you communicate with AC and other ATC to handle all AC ...
Issuing an AMBR Command <ul><li>Text message or radar cues particular action </li></ul><ul><li>Click on  Command  Button <...
 
 
General Methods <ul><li>Empirical Methods </li></ul><ul><ul><li>Day 1: Collect MODS and P/M data and train on AMBR plus AM...
Experiments 1 & 2 <ul><li>AMBR Scenario Design </li></ul><ul><ul><li>Experiment 1: alternating 5 easy, 5 hard </li></ul></...
Off-the-shelf ACT-R Model of AMBR <ul><li>Scan for something to do: Radar, Left, Right, Bottom text windows </li></ul><ul>...
Model Captures Range of Performance
Model Predictions <ul><li>Prediction of whether a subject commits an error in a scenario, based on scenario details and in...
Ind’l Diffs’ Impact on  Hold Errors <ul><li>Hold errors only weakly dependent on W, more strongly on P/M and scenario diff...
Scenario Difficulty Scenario
Mean Errors by Scenario Scenario
Be Careful What (DM) you Model <ul><li>Error data too coarse to constrain model  </li></ul><ul><li>Even total RT/command d...
Observable Behaviors <ul><li>Subject </li></ul><ul><li>T 0.0 Cue: Accept T6? </li></ul><ul><li>T 3.6 ACCEPT button </li></...
The Details Are Inside <ul><li>Model I/O </li></ul><ul><li>T 0.0 Cue: Accept T6? </li></ul><ul><li>T 3.7 ACCEPT button </l...
Conclusion thus far… <ul><li>Visual search vs. memory strategies trade off in final performance => even when modeling a co...
Modifications for Experiment 3 <ul><li>Use more fine-grained measures: Action RT & Clicks </li></ul><ul><li>Modify the ATC...
Raw Characteristics of Data <ul><li>Experiment 3 </li></ul><ul><li>Action RT 12.1 sec, Holds 3.3 / subject </li></ul><ul><...
Model Modifications <ul><li>Search not only can give the answer sought (a specific AC’s location) but an additional rehear...
Model Predicts Hold Errors <ul><li>Predicts errors per subject, r = 0.81 </li></ul><ul><li>Hold errors depend more on W (c...
Model Predicts Number of Clicks
 
W, P/M affect RT click by click <ul><li>Set W-P/M parameters in model corresponding to participants (e.g., hi-hi & lo-lo) ...
W, P/M affect RT click by click <ul><li>Set W-P/M parameters in model corresponding to participants  </li></ul><ul><li>Run...
Conclusion thus far <ul><li>Modeling more fine-grained measures required task and model modifications, but this produced i...
Theoretical Interlude: Spatial vs. Verbal WM <ul><li>Our working assumption (parsimoniously) posits a single source activa...
Opportunity to Test in Current Work <ul><li>AMBR task has spatial and verbal aspects </li></ul><ul><li>Included verbal  an...
Opportunity to Test in Current Work <ul><li>Result </li></ul><ul><ul><li>Experiments 3 & 4: Spatial Span-based W predicts ...
Opportunity to Test in Current Work <ul><li>Result </li></ul><ul><ul><li>Experiments 3 & 4: Spatial Span-based W predicts ...
Spatial Span taps speed as well… <ul><li>Another study, spawned by this issue, shows relationship between individuals’ men...
Theoretical Interlude Conclusion <ul><li>Studying verbal vs. spatial memory resources in context of AMBR task moves theore...
Strategic Variation Emerges <ul><li>Experiment 4 also revealed several sources of strategic variation, explored further in...
Experiment 5 Details <ul><li>Scenarios designed to have low (6 ACs) vs. high memory load (total 12 ACs) </li></ul><ul><li>...
Modeling Specific AMBR Components Easy Scenarios Hard Scenarios Accuracy of first AC click Accuracy of first AC click
Modeling Specific AMBR Components Easy Scenarios Hard Scenarios RT to Correct AC click RT to Correct AC click
Model Predictions Match Data <ul><li>Main effects of scenario difficulty amplified for low W individuals </li></ul><ul><li...
Summary of Conclusions <ul><li>Complex tasks are not a modeling panacaea! Only by seeking extra constraint of modeling ind...
 
Features of Our Approach  <ul><li>Our approach aims to jointly provide </li></ul><ul><ul><li>Predictions that are accurate...
Joint Distribution of W and P/M W and P/M are tapping distinct characteristics
Upcoming SlideShare
Loading in...5
×

Predicting and Explaining Individual Performance in Complex Tasks

736

Published on

http://havatrafik.blogspot.com

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
736
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
21
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Transcript of "Predicting and Explaining Individual Performance in Complex Tasks"

    1. 1. Predicting and Explaining Individual Performance in Complex Tasks Marsha Lovett, Lynne Reder, Christian Lebiere, John Rehling, Baris Demiral This project is sponsored by the Department of the Navy, Office of Naval Research
    2. 2. Multi-Tasking <ul><li>A single person can perform multiple tasks. </li></ul><ul><li>A single model should be able to capture performance on those multiple tasks. </li></ul><ul><li>A single person brings to bear the same fundamental processing capacities to perform all those tasks. </li></ul><ul><li>A single model should be able to predict that person’s performance across tasks from his/her capacities. </li></ul>
    3. 3. <ul><li>A way to keep the multiple-constraint advantage offered by unified theories of cognition while making their development tractable is to do Individual Data Modeling. That is, to gather a large number of empirical/experimental observations on a single subject (or a few subjects analysed individually) using a variety of tasks that exercise multiple abilities (e.g., perception memory, problem solving), and then to use these data to develop a detailed computational model of the subject that is able to learn while performing the tasks. </li></ul>Gobet & Ritter, 2000
    4. 4. <ul><li>ZERO </li></ul><ul><li>PARAMETER </li></ul><ul><li>PREDICTIONS! </li></ul>
    5. 5. Basic Goals of Project <ul><li>Combine best features of cognitive modeling </li></ul><ul><ul><li>Study performance in a dynamic, multi-tasking situation (albeit less complex than real world) </li></ul></ul><ul><ul><li>Explain not only aggregate behavior but variation (using individual difference variables) </li></ul></ul><ul><ul><li>Predict (not fit/postdict) complex performance </li></ul></ul><ul><ul><ul><li>Use cognitive architecture and fixed parameters </li></ul></ul></ul><ul><ul><ul><li>Employ off-the-shelf models whenever possible </li></ul></ul></ul><ul><ul><ul><li>Plug in individual difference params for each person </li></ul></ul></ul>
    6. 6. How to predict task performance <ul><li>Estimate each individual’s processing parameters </li></ul><ul><ul><li>Measure individuals’ performance on “standard” tasks </li></ul></ul><ul><ul><li>Using models of these tasks, estimate participant’s corresponding architectural parameters (e.g., working memory capacity, perceptual/motor speed) </li></ul></ul><ul><li>Build/refine model of target task </li></ul><ul><li>Select global parameters for model of target task (e.g., from previously collected data) </li></ul><ul><li>Plug into model of target task each individual’s parameters to predict his/her target task performance </li></ul>
    7. 7. Example: Memory Task Performance <ul><li>Fit task A to estimate individuals’ parameters </li></ul>
    8. 8. Zero-Parameter Predictions <ul><li>Plug those parameters into model of task B </li></ul>(Lovett, Daily, & Reder, 2000)
    9. 9. Challenges of Complex Tasks <ul><li>Modeling the target task is harder </li></ul><ul><li>More than one individual difference variable likely impacting target task </li></ul><ul><li>Possibility of knowledge/strategy differences </li></ul>
    10. 10. What about knowledge differences? <ul><li>Develop tasks that reduce their relevance </li></ul><ul><li>Train participants on specific procedures </li></ul><ul><li>Measure skill/knowledge differences in another task and incorporate them in model </li></ul><ul><li>Use model to predict variation in relative use of strategies by way of estimates of individuals’ processing capacities </li></ul>
    11. 11. Individual Differences in ACT-R <ul><li>Most ACT-R models don’t account for impact of individual differences on performance, but the potential is there </li></ul><ul><li>There are many parameters with particular interpretations related to individual difference variables </li></ul><ul><li>Most ACT-R modelers set parameters to universal or global values, i.e., defaults or values that fit aggregate data </li></ul>
    12. 12. ACT-R & Individual Differences M1, M2, M3, … W1, W2, W3, … P1, P2, P3, …
    13. 13. Overview of Talk <ul><li>Review tasks we are studying </li></ul><ul><li>Illustrate methodology </li></ul><ul><li>Highlight key results </li></ul><ul><ul><li>Visual search vs. memory strategies trade off in final performance => complex task modeling offers best constraint with fine-grained analysis </li></ul></ul>
    14. 14. Modified Digit Span (MODS)
    15. 15. Modified Digit Span (MODS)
    16. 16. P/M Tasks <ul><li>In our earlier studies, initial training phase of target task was used to collect data on individuals’ perceptual/motor speed. </li></ul><ul><ul><li>e.g., Time to find object “A7” and click on it </li></ul></ul><ul><li>In later studies, separate task used to measure perceptual and motor speed. </li></ul>
    17. 17. How to predict task performance <ul><li>Estimate each individual’s processing parameters </li></ul><ul><ul><li>Measure individuals’ performance on MODS, PercMotor </li></ul></ul><ul><ul><li>Using models of these tasks, estimate participant’s corresponding architectural parameters (e.g., working memory capacity, perceptual/motor speed ) </li></ul></ul><ul><li>Build/refine model of target task </li></ul><ul><li>Select global parameters for model of target task (e.g., from previously collected data) </li></ul><ul><li>Plug into model of target task each individual’s parameters to predict his/her target task performance </li></ul>
    18. 18. W affects Performance <ul><li>W is the ACT-R parameter for source activation, which impacts the degree to which activation of goal-related facts rises above the sea of other facts’ activations </li></ul><ul><li>Higher W => goal-related facts relatively more activated => faster and more accurately retrieved => better MODS performance </li></ul>
    19. 19. Estimating W <ul><li>Model of MODS task is fit to individual’s MODS performance by varying W </li></ul><ul><li>Best fitting value of W is taken as estimate </li></ul>
    20. 20. Estimating PM <ul><li>For simplicity, we estimated a combined PM parameter directly from each individual’s perceptual/motor task performance. </li></ul><ul><li>This PM parameter was then used to scale the timing of the target task’s perceptual-motor productions. </li></ul>
    21. 21. Joint Distribution of W and P/M W and P/M are tapping distinct characteristics
    22. 22. ACT-R & Individual Differences M1, M2 , M3, … W1, W2 , W3, … P1, P2 , P3, …
    23. 23. Specifics of our Approach <ul><li>Estimate each individual’s processing parameters </li></ul><ul><ul><li>Measure individuals’ performance on modified digit span, spatial span, perceptual/motor speed </li></ul></ul><ul><ul><li>Using models of these tasks, estimate participant’s W, P, M </li></ul></ul><ul><li>Build/refine model of air traffic control task–AMBR </li></ul><ul><li>Select global parameters for AMBR model </li></ul><ul><li>Plug in individuals’ parameters to predict performance across different AMBR scenarios </li></ul>
    24. 24. AMBR: Air Traffic Control Task <ul><li>Complex and dynamic task </li></ul><ul><li>Spatial and verbal aspects </li></ul><ul><li>Multi-tasking </li></ul><ul><li>Testbed for cognitive modeling architectures </li></ul>
    25. 25. AMBR Task AC=aircraft, ATC=air traffice controller <ul><li>As ATC, you communicate with AC and other ATC to handle all AC in your airspace </li></ul><ul><li>Six commands with different triggers: </li></ul><ul><li>First ACCEPT, then WELCOME incoming AC (these two separated by short interval) </li></ul><ul><li>First TRANSFER, then order a CONTACT message from outgoing AC (these two separated by short interval) </li></ul><ul><li>Decide to OK or REJECT requests for speed increase </li></ul><ul><li>When a command is not handled before AC reaches zone boundary, this is a HOLD (error) </li></ul>
    26. 26. Issuing an AMBR Command <ul><li>Text message or radar cues particular action </li></ul><ul><li>Click on Command Button </li></ul><ul><li>Click on Aircraft (in radar screen) </li></ul><ul><li>Click on Air Traffic Controller (if nec’y) </li></ul><ul><li>Click on SEND Button </li></ul>
    27. 29. General Methods <ul><li>Empirical Methods </li></ul><ul><ul><li>Day 1: Collect MODS and P/M data and train on AMBR plus AMBR practice </li></ul></ul><ul><ul><li>Day 2: Review AMBR instructions, battery of AMBR scenarios </li></ul></ul><ul><li>Modeling Methods </li></ul><ul><ul><li>Use MODS & PM data to estimate W and PM for each subject </li></ul></ul><ul><ul><li>Plug individual W and PM values into AMBR model </li></ul></ul><ul><ul><li>Compare individuals’ AMBR performance with model predictions </li></ul></ul>
    28. 30. Experiments 1 & 2 <ul><li>AMBR Scenario Design </li></ul><ul><ul><li>Experiment 1: alternating 5 easy, 5 hard </li></ul></ul><ul><ul><li>Experiment 2: 9 scenarios of varying difficulty </li></ul></ul><ul><li>AMBR Dependent Measures </li></ul><ul><ul><li>Total time to handle each command </li></ul></ul><ul><ul><li>Number of hold errors </li></ul></ul>
    29. 31. Off-the-shelf ACT-R Model of AMBR <ul><li>Scan for something to do: Radar, Left, Right, Bottom text windows </li></ul><ul><li>When an action cue is noticed, determine if it has been handled or not: scan/remember </li></ul><ul><li>If the cue has not been handled, click command, AC, [ATC], SEND </li></ul><ul><li>Resume scanning </li></ul>
    30. 32. Model Captures Range of Performance
    31. 33. Model Predictions <ul><li>Prediction of whether a subject commits an error in a scenario, based on scenario details and individual’s W & P/M </li></ul>70 21 Model scenarios with no errors 4 205 Model scenarios with errors Subject scenarios with no errors Subject scenarios with errors
    32. 34. Ind’l Diffs’ Impact on Hold Errors <ul><li>Hold errors only weakly dependent on W, more strongly on P/M and scenario difficulty </li></ul># Hold Errors Parameter Value
    33. 35. Scenario Difficulty Scenario
    34. 36. Mean Errors by Scenario Scenario
    35. 37. Be Careful What (DM) you Model <ul><li>Error data too coarse to constrain model </li></ul><ul><li>Even total RT/command data insufficient </li></ul><ul><li>Model predicts that scanning strategy plays a large role in performance. </li></ul><ul><li>This is consistent with participant reports who may be doing any combination of visual search or memory retrieval </li></ul>
    36. 38. Observable Behaviors <ul><li>Subject </li></ul><ul><li>T 0.0 Cue: Accept T6? </li></ul><ul><li>T 3.6 ACCEPT button </li></ul><ul><li>T 5.9 AC “T6” </li></ul><ul><li>T 6.7 ATC “EAST” </li></ul><ul><li>T 7.7 SEND button </li></ul><ul><li>Model </li></ul><ul><li>T 0.0 Cue: Accept T6? </li></ul><ul><li>T 3.7 ACCEPT button </li></ul><ul><li>T 5.7 AC “T6” </li></ul><ul><li>T 7.0 ATC “EAST” </li></ul><ul><li>T 8.2 SEND button </li></ul>Stochastic variation on the single-action level is part of subject and model behavior
    37. 39. The Details Are Inside <ul><li>Model I/O </li></ul><ul><li>T 0.0 Cue: Accept T6? </li></ul><ul><li>T 3.7 ACCEPT button </li></ul><ul><li>T 5.7 AC “T6” </li></ul><ul><li>T 7.0 ATC “EAST” </li></ul><ul><li>T 8.2 SEND button </li></ul><ul><li>Model Trace </li></ul><ul><li>T 1.5 Notice cue </li></ul><ul><li>T 2.5 Subgoal task </li></ul><ul><li>T 3.7 Mouse click </li></ul><ul><li>T 3.8 Start AC search </li></ul><ul><li>T 4.9 Find AC </li></ul><ul><li>T 5.7 Mouse click </li></ul><ul><li>T 7.0 Mouse click </li></ul><ul><li>T 8.2 Mouse click </li></ul>
    38. 40. Conclusion thus far… <ul><li>Visual search vs. memory strategies trade off in final performance => even when modeling a complex task, coarse dependent measures (accuracy, total RT) hide important details </li></ul><ul><li>Previous AMBR model fit group data well </li></ul><ul><li>Only by seeking extra constraint of modeling individual participants were important gaps in model fidelity revealed </li></ul>
    39. 41. Modifications for Experiment 3 <ul><li>Use more fine-grained measures: Action RT & Clicks </li></ul><ul><li>Modify the ATC task to increase memory demand </li></ul><ul><ul><li>More interesting for our purposes </li></ul></ul><ul><ul><li>More realistic </li></ul></ul><ul><ul><li>Lengthen scenario length so same planes are in play </li></ul></ul><ul><ul><li>Hide AC names until click, then only after delay </li></ul></ul><ul><li>Use model to bracket appropriate difficulty level </li></ul>
    40. 42. Raw Characteristics of Data <ul><li>Experiment 3 </li></ul><ul><li>Action RT 12.1 sec, Holds 3.3 / subject </li></ul><ul><li>Action RT correlates with W (r = -0.314) and Pm (r = 0.485) </li></ul><ul><li>Holds correlates with W (r = -0.444) and Pm (r = 0.508) </li></ul>
    41. 43. Model Modifications <ul><li>Search not only can give the answer sought (a specific AC’s location) but an additional rehearsal of that information </li></ul><ul><li>In slack times, possible strategy of studying radar screen to rehearse AC names (called “exploratory clicks”) </li></ul>
    42. 44. Model Predicts Hold Errors <ul><li>Predicts errors per subject, r = 0.81 </li></ul><ul><li>Hold errors depend more on W (compared to previous version of task) but still mostly dependent on PM and scenario difficulty </li></ul><ul><li>Move to modeling more fine-grained aspects of data… </li></ul>
    43. 45. Model Predicts Number of Clicks
    44. 47. W, P/M affect RT click by click <ul><li>Set W-P/M parameters in model corresponding to participants (e.g., hi-hi & lo-lo) </li></ul><ul><li>Run model to produce RT predictions click by click (for 2 commands: Accept and Contact) </li></ul>Hi-Hi Model & Subject Lo-Lo Model & Subject
    45. 48. W, P/M affect RT click by click <ul><li>Set W-P/M parameters in model corresponding to participants </li></ul><ul><li>Run model to produce RT predictions click by click (for 2 commands: Accept and Contact) </li></ul>
    46. 49. Conclusion thus far <ul><li>Modeling more fine-grained measures required task and model modifications, but this produced individual participant predictions that were very promising. </li></ul><ul><li>Clicking on correct AC the first time ranges from 69% to 96% </li></ul><ul><ul><li>Akin to remember vs. scan strategies </li></ul></ul><ul><ul><li>Higher number -> more (accurate) remembering </li></ul></ul><ul><ul><li>This detailed aspect of performance relates to W </li></ul></ul>
    47. 50. Theoretical Interlude: Spatial vs. Verbal WM <ul><li>Our working assumption (parsimoniously) posits a single source activation parameter, W </li></ul><ul><li>W modulates the degree to which goal-relevant facts are activated above the sea of unrelated facts </li></ul><ul><li>… regardless of spatial/verbal representation </li></ul><ul><li>This perspective still allows for spatial/verbal distinctions in performance but explains them as a function of differences in spatial/verbal skills etc. </li></ul>
    48. 51. Opportunity to Test in Current Work <ul><li>AMBR task has spatial and verbal aspects </li></ul><ul><li>Included verbal and spatial working memory tasks in battery, starting with Experiment 3 </li></ul><ul><li>Which span task produces W estimates that best predict individuals’ AMBR performance? </li></ul><ul><li>Spatial Span task from Miyake and Shah (1996): </li></ul>R R R “ normal” “ normal” “ reversed”
    49. 52. Opportunity to Test in Current Work <ul><li>Result </li></ul><ul><ul><li>Experiments 3 & 4: Spatial Span-based W predicts AMBR performance better than MODS-based W </li></ul></ul><ul><li>Possible explanations: </li></ul><ul><ul><li>Spatial format more relevant for this task? </li></ul></ul><ul><ul><li>Spatial Span shows more variability -> more sensitive? </li></ul></ul><ul><ul><li>Spatial Span variability taps other sources of variation? </li></ul></ul><ul><ul><li>Are there separate W’s for verbal and spatial WM? </li></ul></ul>
    50. 53. Opportunity to Test in Current Work <ul><li>Result </li></ul><ul><ul><li>Experiments 3 & 4: Spatial Span-based W predicts AMBR performance better than MODS-based W </li></ul></ul><ul><li>Possible explanations: </li></ul><ul><ul><li>Spatial format more relevant for this task? </li></ul></ul><ul><ul><li>Spatial Span shows more variability -> more sensitive? </li></ul></ul><ul><ul><li>Spatial Span variability taps other sources of variation? </li></ul></ul><ul><ul><li>Are there separate W’s for verbal and spatial WM? </li></ul></ul>
    51. 54. Spatial Span taps speed as well… <ul><li>Another study, spawned by this issue, shows relationship between individuals’ mental rotation speed and Spatial Span </li></ul><ul><li>Pattern of correlations with PM: </li></ul><ul><ul><li>MODS: r=.25 Spatial Span: r=.65 </li></ul></ul><ul><li>Pattern of correlations with AMBR components: </li></ul>Mem+Mouse Mouse Mouse -.70 .-56 -.16 Welcome-Tot -.53 -.61 -.20 Welcome-AC -.39 -.55 -.62 SpeedReq-AC PM SS MODS
    52. 55. Theoretical Interlude Conclusion <ul><li>Studying verbal vs. spatial memory resources in context of AMBR task moves theoretical debate to more realistic arena </li></ul><ul><ul><li>This complements work with laboratory tasks and allows greater potential for generalization of results </li></ul></ul>
    53. 56. Strategic Variation Emerges <ul><li>Experiment 4 also revealed several sources of strategic variation, explored further in Experiment 5 </li></ul><ul><li>Waiting for AC name: ranges from 42% to 100% </li></ul><ul><ul><li>May reflect lack of confidence in memory, utility of checking one’s memory </li></ul></ul><ul><ul><li>Somewhat negatively correlated with W </li></ul></ul><ul><li>Initiating “welcome” and “contact” commands in anticipation of text cue (ranges from 0% to 100%) </li></ul><ul><li>Making exploratory clicks on ACs during slack time (ranges from never to > 5 per scenario) </li></ul>
    54. 57. Experiment 5 Details <ul><li>Scenarios designed to have low (6 ACs) vs. high memory load (total 12 ACs) </li></ul><ul><li>Speed requests most common command </li></ul><ul><ul><li>Most interesting for model predictions </li></ul></ul><ul><ul><li>Least susceptible to snowball effects </li></ul></ul><ul><li>Dependent measures include RTs for individual clicks and strategy use as a function of scenario difficulty and command </li></ul>
    55. 58. Modeling Specific AMBR Components Easy Scenarios Hard Scenarios Accuracy of first AC click Accuracy of first AC click
    56. 59. Modeling Specific AMBR Components Easy Scenarios Hard Scenarios RT to Correct AC click RT to Correct AC click
    57. 60. Model Predictions Match Data <ul><li>Main effects of scenario difficulty amplified for low W individuals </li></ul><ul><li>Main effects of command type (more/less memory-demanding) amplified for low W </li></ul><ul><li>Wait-for-AC-name strategy varied as a function of command type </li></ul><ul><li>Exploratory clicks strategy varied as a function of scenario difficulty </li></ul>
    58. 61. Summary of Conclusions <ul><li>Complex tasks are not a modeling panacaea! Only by seeking extra constraint of modeling individual participants were important gaps in model’s fidelity revealed. </li></ul><ul><li>Studying verbal vs. spatial memory resources in context of AMBR task moves theoretical debate to more realistic arena. </li></ul><ul><li>Variability in performance -- from different use of strategies and/or from differences in processing capacities -- is there for the looking. Studying performance on average offers incomplete understanding. </li></ul>
    59. 63. Features of Our Approach <ul><li>Our approach aims to jointly provide </li></ul><ul><ul><li>Predictions that are accurate and detailed </li></ul></ul><ul><ul><li>At the individual participant level </li></ul></ul><ul><ul><li>Generated in real time (or faster) </li></ul></ul><ul><ul><li>Based on an interpretable model with variation in meaningful individual difference parameters </li></ul></ul><ul><ul><li>That generalize to variants of the target task </li></ul></ul>
    60. 64. Joint Distribution of W and P/M W and P/M are tapping distinct characteristics
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×