Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Inverse Modeling for Cognitive Science "in the Wild"


Published on

Invited talk by Antti Oulasvirta / Aalto University, given for the Interacting Minds Center at Aarhus University. October 2017

Published in: Science
  • Be the first to comment

  • Be the first to like this

Inverse Modeling for Cognitive Science "in the Wild"

  1. 1. Inverse Modeling for Cognitive Science ”In the Wild” October 10, 2017 Antti Oulasvirta Aalto University work with Antti Kangasrääsiö, Andrew Howes, Jukka Corander, Samuel Kaski, Byungjoo Lee, Kashyap Todi, Jussi Jokinen Invited talk given for the Interacting Minds Center at Aarhus University, Denmark
  2. 2. It has become easy to collect (lots of) data about people. But it is hard to explain the what, how, and why
  3. 3. Sherlock Holmes
  4. 4. Sherlock’s problem: Inverse modeling How to estimate (theoretically plausible) model parameters without intervention and from limited, noisy, naturalistic observational data? • Complicated by the strategic flexibility and idiosyncratic properties of the human. Any behavior can be produced by numerous cognitions Absence of rigorous methodology hampers theory- formation 12.3.2018 4
  5. 5. Behavioral & cognitive sciences meet machine learning…
  6. 6. In this talk Theorizing in Cognitive Science and HCI: A Crisis? Inverse Modeling Examples ABC: Rigorous Inverse Modeling Methodology
  7. 7. Crisis?
  8. 8. Scientific theory-formation and modeling “A scientific model seeks to represent empirical objects, phenomena, and physical processes in a logical and objective way. All models are in simulacra, that is, simplified reflections of reality that, despite being approximations, can be extremely useful.”
  9. 9. A wealth of cognitive models Symbolic models E.g., ACT-R, EPIC, GOMS Neural models E.g., perceptron, HNNs Bounded rationality E.g., Information foraging theory Chronometric models E.g., drift diffusion models HVS models E.g., saliency models
  10. 10. “Psychology as the science of design” CogTool 12.3.2018 10
  11. 11. Ulric Neisser 1978: Ecological validity crisis “If X is an interesting or socially important aspect of memory, then psychologists have hardly ever studied X.”
  12. 12. John Carroll: Artefact as theory-nexus “Other theorists hold that pursuing the goal of developing cognitive science theories of HCI may impair progress toward usefully understanding HCI phenomena and effectively contributing to design. This approach stresses the distortion and oversimplification inherent in laboratory-bound psychology and in conventional views of theory-based design." 12.3.2018 12
  13. 13. Yvonne Rogers: “In the Wild Theories” “Likewise, it has proven difficult to say with any confidence the extent to which a system or particular interface function can be mapped back to a theory. Typically, theories end up as high-level design implications, guidelines, or principles in interaction design"
  14. 14. We need rethink how we construct models that are both theoretically plausible and fit data well 12.3.2018 15
  15. 15. Inverse modeling
  16. 16. General steps in modeling 1. Model design • How to formulate the model? 2. Study design • How to obtain data to evaluate models? 3. Parameter inference • Which parameter values are probable for the observations? 4. Model evaluation • How well is the model able to reproduce observations? 5. Model selection • Which model is the most probable explanation for the data?
  17. 17. Forward vs. inverse modeling From model to data (forward) -- from data to model (inverse) 12.3.2018 18
  18. 18. Almost nobody talks about inverse modeling….
  19. 19. Inverse modeling: the problem • Model fit is determined jointly by the adequacy of the model structure, and by the values of the model variables. • Some of the variable values have been decided prior to empirical observations, and have their place as theoretical postulates. • Others, called free parameters, are determined from new empirical data, and these parameters can take almost any value, ranging from theoretically justified case-specific values to values that cannot be justified theoretically. • How to determine these values?
  20. 20. Model parameters: Example Layout learning [Jokinen et al. CHI2017] 12.3.2018 21 evisual search model predictsvisual search times for new and changed layouts. For a noviceuser without any prior exposureto thelayout, edicts that of the three elements chosen for this comparison, the salient green element is the fastest to find. After learning the locations of the expert model finds all fairly quickly. At this point, one blue element and the green element change place. Search times for the moved arelonger than for thegreen element, becausethe model remembers thedistinctivefeaturesof thelatter. Figure 2. On the basis of expected utility, the controller requests atten- tion deployment to a new visual element from theeye-movement system. This directs attention to the most salient unattended visible object and results in its encoding. If locational or feature information is accessi- ble in the LTM, the controller, learning the utilities of its actions, can optionally also request these features to be considered in the attention deployment. Encoded objects are stored in VSTM, which inhibits revis- its. Location and visual features of the elements are stored in LTM for Encoding an object allows t the target or adistractor. Bef jects, it needs to attend one. holds a visual representatio controller’srequest it resolv to oneof theobjectsin it. Th the properties of thevisual o in the visual representation feature isvisually represente ae where eistheeccentricity o the size) and a and b are fre visual featurein question. Th a = 0.104 and b = 0.85 for and 0.142 and 0.96 for size On thebasis of therepresen given a total activation as a top-down activations. Botto an object, calculated as the d other objects of the environm of the linear distance d betw BAi = objects  j f eatu  k Two objects are dissimilar f shared exactly between them tion. Hence, bottom-up activ
  21. 21. Parameters 12.3.2018 22
  22. 22. Example: Text entry model [Jokinen et al. submitted] 12.3.2018 23
  23. 23. Inverse modeling 101 Finding best parameters for a model 12.3.2018 24
  24. 24. Some methods 1. Adopt values from literature 2. OLS (Ordinary least squares) for regression models 3. Manual tuning 4. Grid search 12.3.2018 25 Laborious Unsystematic Prone to error Prone to bias Prone to cheating Almost never done
  25. 25. We lack appropriate inverse modeling methods for generative models 12.3.2018 26
  26. 26. Long-term goal for inverse modeling in cognitive sciences and HCI Given behavioral data, infer: • capacities and traits like visual acuity, working memory capacity, personality types… • motivations, or goals, interests and preferences • beliefs: mental representations • behavioral strategies: individual and task-specific ways of acting • … 12.3.2018 27
  27. 27. Forward modeling problem “Define model design MA that explains interesting behaviors.” Data spaceModel space MA
  28. 28. Inverse modeling problem “Find parametrization θ for MA that maximizes model fit.” Data spaceModel space MA(●) BA θ
  29. 29. The essence of theorizing “Define a model for which parametrizations exist that yield valid predictions for a large set of interesting observations.” Data spaceModel space MA(●) BA ∃θ ⊂ Θ
  30. 30. In many areas of engineering and natural sciences, inverse modeling is integral to theory- formation 12.3.2018 31
  31. 31. Example: Climate models 12.3.2018 32
  32. 32. Example: Reservoir modeling 12.3.2018 33 Dc3Mzg4LWQ4ODctNDE3MS04YjY3LTgyZmMzZjJmMmVmNw.jpg
  33. 33. Examples: Inverse modeling in HCI
  34. 34. Inverse modeling in HCI From user data to a simulator model 12.3.2018 35
  35. 35. Inferring neuromuscular noise to optimize CD gain 12.3.2018 36 Kalman filter & submovement efficiency model Lee et al. arXiv 2017
  36. 36. Biomechanical simulation Motion data  Inverse kinematics  Inverse dynamics  Static optimization  Muscle activations  Fatigue 12.3.2018 37Bachynskyi et al. CHI 2106
  37. 37. “Remulation” Inverse modeling may be avoided if a person’s history is fully observable to a cognitive model that can “remulate” it 12.3.2018 38Todi et al. IUI 2018
  38. 38. “Familiarisation”1.Most-Encountered 2.Serial Position Curve 3.Visual Statistical Learning History earning 4.GenerativeModel of Positional Learning Original Todi et al. IUI 2018
  39. 39. In most cases we do not have access to full histories of people
  40. 40. Field data make inverse modeling even worse…
  41. 41. Inverse modeling Field data insist on a larger number of free parameters As the number of free parameters increases, model identifiability decreases Number of free parameters Parameter independence Unidentifiable Identifiable
  42. 42. For ”in the wild” cogsci, we need more powerful inverse modeling methods
  43. 43. Approximate Bayesian Computation (ABC)
  44. 44.  A handy review
  45. 45. ABC is a principled way to find optimal model parameters Figure 1. This paper studies methodology for inference of parameter values of cognitive models from observational data in HCI. At the bot- tom of the figure, we have behavioral data (orange histograms), such as task solution, only the objecti straints of thesituation, weca theoptimal behavior policy. H that isinferring theconstraints optimal, isexceedingly difficu quality and granularity of pre this inversereinforcement lear to beunreasonable when often data exists, such as isoften the Our application case is a rece [13]. The model studied here tation of search behavior, and completion times, in varioussi parametric assumptions about visual system (e.g., fixation dur
  46. 46. Approximate Bayesian Inference 1. Inverse modeling with black-box models 2. Works with likelihood-free models (simulators) 3. Noise-tolerant, sample-efficient 4. A global method (cf. local method) 5. Computes a posterior distribution for parameter space
  47. 47. How ABC works 1. Choose parameter values for the model 2. Simulate predictions 3. Evaluate discrepancy between predictions and observations 4. Use a probabilistic model to estimate discrepancy in different regions of parameter space 5. (Repeat until converged) 12.3.2018 49
  48. 48. How ABC works 12.3.2018 50 Approximate Bayesian Computation (ABC)
  49. 49. How ABC works 12.3.2018 51 Approximate Bayesian Computation (ABC)
  50. 50. How ABC works 12.3.2018 52 Approximate Bayesian Computation (ABC)
  51. 51. How ABC works 12.3.2018 53 Approximate Bayesian Computation (ABC)
  52. 52. How ABC works 12.3.2018 54 Approximate Bayesian Computation (ABC)
  53. 53. How ABC works 12.3.2018 55 Approximate Bayesian Computation (ABC) Indicates most likely value and uncertainty
  54. 54. Uses of ABC Optimal selection and calibration of model for data 1. Model selection (trying out different models) 2. Parameter inference (choosing best parameters) 3. Posterior inference (understanding the space of plausible explanations) 12.3.2018 56
  55. 55. Results in “Inverse Computational Rationality” Towards “Cognitive science in the wild”
  56. 56. 12.3.2018 58
  57. 57. Inverse modeling of human data is hard Multiple explanations to any observation • Different observations can be produced by same mechanism Stochasticity Sparse data Large individual and contextual variability 12.3.2018 59Kangasrääsiö et al. CHI 2107
  58. 58. “Bounded agents” Behavior manifests optimal adaptation to bounds
  59. 59. People as agents: Markov Decision Process 12.3.2018 61
  60. 60. Computational rationality: assumptions • Assume that users behave (approximately) to maximize utility given limits on their own capacity • People are “Bounded agents” • Optimality determined by (1) the environment; (2) goals; and (3) the user’s cognitive and perceptual capabilities • Optimal behavioral strategies can be estimated using reinforcement learning • No need for hard-wiring task procedures (cf. “old cognitive models”) 12.3.2018 62
  61. 61. Model of menu search [Chen et al. CHI’15] Finds optimal gaze pattern given menu design and parameters of the visual and cognitive system 12.3.2018 63
  62. 62. Case: Menu interaction 12.3.2018 64 Given click times only, predict parameters of HVS Kangasrääsiö et al. CHI 2107 Click times
  63. 63. ABC improves fit over manual tuning 12.3.2018 65Kangasrääsiö et al. CHI 2107 Mean TCT 0.92s Mean TCT 1.49 s Mean TCT 0.93 s
  64. 64. ABC allows comparison and exploration of model variants 12.3.2018 66Kangasrääsiö et al. CHI 2107
  65. 65. Posterior estimation ABC yields a posterior distribution for the parameters 12.3.2018 67
  66. 66. ABC increases model fit to individuals 12.3.2018 68Kangasrääsiö et al. CHI 2107
  67. 67. Explaining individual differences 12.3.2018 69
  68. 68. Conclusion Machine learning for theorizing in cognitive science
  69. 69. “Algorithmic Sherlock Holmes”
  70. 70. Lots of potential for cognitive and behavioral sciences to infer… 1. Cognitive capabilities like working memory capacity 2. Belief system (e.g., associative memory structures) 3. Interests, goals, preferences 4. Personality and other more stable traits 5. Cultural differences 6. Situational differences (tasks, context, …) 7. … 12.3.2018 72
  71. 71. Issues pointed out to me Computational costs; convergence; large number of parameters  Truly uncontrolled behavior remains out of reach? The psychologist’s fallacy Limits of mathematical description of human mind
  72. 72. ELFI package Needed 1. a model with tunable parameters 2. Prior knowledge of reasonable parameter ranges 3. Observation dataset 4. Discrepancy measure
  73. 73. From observed behavior to data-generating models October 10, 2017 Antti Oulasvirta Aalto University work with Antti Kangasrääsiö, Andrew Howes, Jukka Corander, Samuel Kaski, Byungjoo Lee, Kashyap Todi, Jussi Jokinen