Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Modeling decision making deficits in frontostriatal disorders                            Michael Frank             Laborato...
Computational Psychiatry and...                   Neurogenocomputomics• Many disorders broadly characterized by changes in...
Computational Psychiatry and...                   Neurogenocomputomics• Many disorders broadly characterized by changes in...
Computational Psychiatry and...                   Neurogenocomputomics• Many disorders broadly characterized by changes in...
Reinforcement learning and dopamine: prediction errors                           Positive PE:                    Negative ...
Reinforcement learning and dopamine: prediction errors                           Positive PE:                    Negative ...
D1 effects on striatal learning: Positive PE
D1 effects on striatal learning: Positive PE                    Three factor learning: presynaptic, postsynaptic and DA
D2 effects on striatal learning: Negative PE                                               Frank 2005
Neural model of basal ganglia and dopamineIntegrates a wide range of data into a single coherent frameworkSeparate Go and ...
Maximizing Reward via RT Adaptation:                   Temporal Utility Integration Task                        Reward Fre...
RL model: Fit to data across all subjectsRL model : adjust RTs as a function of reward prediction errors                  ...
Neurogenetic and pharmacological modulation of      reinforcement learning parameters                                     ...
Single subject Data...                  Single Subject CEV                                  Single Subject DEV          50...
Exploration vs Exploitation• By exploiting learned strategies, we know we can get a certain amount  of reward• But don’t k...
Exploration vs Exploitation• By exploiting learned strategies, we know we can get a certain amount  of reward• But don’t k...
Uncertainty-Based Exploration                                     Exploration                4000                         ...
PFC Gene-Dose Effect on Uncertainty-Based Exploration                        COMT gene-dose effects                       ...
Does the brain track relative uncertainty for exploration?
Does the brain track relative uncertainty for exploration?             ǫ > 0 (’explorers’)   explorers > non-explorers    ...
EEG reveals temporal dynamics
EEG reveals temporal dynamicsRelative uncertainty represented prior to choice, and more so in exploratory trials          ...
Negative symptoms in schizophrenia:                      Uncertainty-Based Exploration                                    ...
Obsessive Compulsive Disorder: Aversion to Uncertainty                                     Uncertainty-driven exploration ...
Summary• Dopamine modulates reinforcement learning and choice based on  positive and negative outcomes: patients, pharmaco...
Thanks To...Bradley DollChristina FigueroaJim CavanaghDavid BadreJeff CockburnAnne CollinsThomas WieckiJim GoldKent Hutchi...
Upcoming SlideShare
Loading in …5
×

Frank_NeuroInformatics11.pdf

736 views

Published on

Computational psychiatry symposium at NeuroInformatics 2011 meeting in Boston, MA, USA.

  • Be the first to comment

Frank_NeuroInformatics11.pdf

  1. 1. Modeling decision making deficits in frontostriatal disorders Michael Frank Laboratory for Neural Computation and Cognition Brown University
  2. 2. Computational Psychiatry and... Neurogenocomputomics• Many disorders broadly characterized by changes in motivation• Several fronto-striatal disorders have substantial genetic heritability• Individual differences in reinforcement learning?
  3. 3. Computational Psychiatry and... Neurogenocomputomics• Many disorders broadly characterized by changes in motivation• Several fronto-striatal disorders have substantial genetic heritability• Individual differences in reinforcement learning?• But... Candidate gene effects are generally small• Which genes? Which task? Which measure?
  4. 4. Computational Psychiatry and... Neurogenocomputomics• Many disorders broadly characterized by changes in motivation• Several fronto-striatal disorders have substantial genetic heritability• Individual differences in reinforcement learning?• But... Candidate gene effects are generally small• Which genes? Which task? Which measure?• Need theoretical model! (and converging pharmacology/imaging) Frank & Fossella, 2011; Maia & Frank, 2011; Huys et al, 2011
  5. 5. Reinforcement learning and dopamine: prediction errors Positive PE: Negative PE:dopamine: Montague, Dayan & Sejnowksi 96; Doya, 2002; O’Reilly, Frank, Hazy & Watz 06... ˆ ˆ δ(t) = r(t) + γ V (t + 1) − V (t)
  6. 6. Reinforcement learning and dopamine: prediction errors Positive PE: Negative PE:dopamine: Montague, Dayan & Sejnowksi 96; Doya, 2002; O’Reilly, Frank, Hazy & Watz 06... ˆ ˆ δ(t) = r(t) + γ V (t + 1) − V (t)
  7. 7. D1 effects on striatal learning: Positive PE
  8. 8. D1 effects on striatal learning: Positive PE Three factor learning: presynaptic, postsynaptic and DA
  9. 9. D2 effects on striatal learning: Negative PE Frank 2005
  10. 10. Neural model of basal ganglia and dopamineIntegrates a wide range of data into a single coherent frameworkSeparate Go and NoGo populations integrate statistics of reinforcement preSMAInput Striatum γ [Vm− Θ] cVm = gege[E Vm] y j ≈ γ [V − ] + 1 + m Θ+ e + g g [E V ] i i i m + g g [E Vm] β l l l net = ge ≈ <x i w ij > + N STN + ... w ij GPe xiGo NoGo Thalamus p p t t ∆wij ≈ (xi yj )−(xi yj ) SNc GPi/SNr Frank, 2005, 2006 J Cog Neurosci, Neural Networks
  11. 11. Maximizing Reward via RT Adaptation: Temporal Utility Integration Task Reward Frequency Reward Magnitude 1.0 350 0.9 CEV CEV DEV 300 DEV 0.8 IEV IEV 0.7 CEVR # Points Gained 250 CEVRProbability 0.6 200 0.5 0.4 150 0.3 100 0.2 50 0.1 0.0 0 0 1000 2000 3000 4000 5000 0 1000 2000 3000 4000 5000 Time (ms) Time (ms) Expected Value 60 Expected Value (freq*mag) 55 50 45 40 35 30 25 20 CEV 15 DEV 10 IEV 5 CEVR 0 0 1000 2000 3000 4000 5000 Time (ms)
  12. 12. RL model: Fit to data across all subjectsRL model : adjust RTs as a function of reward prediction errors Frank, Doll, Oas-Terpstra & Moreno (2009, Nature Neuroscience)
  13. 13. Neurogenetic and pharmacological modulation of reinforcement learning parameters Frank & Fossella, 2011
  14. 14. Single subject Data... Single Subject CEV Single Subject DEV 5000 5000 4500 4500 4000 4000 3500 3500RT (ms) RT (ms) 3000 3000 2500 2500 2000 2000 1500 1500 1000 1000 500 500 0 0 0 10 20 30 40 50 0 10 20 30 40 50 Trial Trial Single Subject IEV Single Subject CEVR 5000 5000 4500 4500 4000 4000 3500 3500RT (ms) RT (ms) 3000 3000 2500 2500 2000 2000 1500 1500 1000 1000 500 500 0 0 0 10 20 30 40 50 0 10 20 30 40 50 Trial Trial
  15. 15. Exploration vs Exploitation• By exploiting learned strategies, we know we can get a certain amount of reward• But don’t know how good it can get. ⇒ Need to Explore• Theory: Explore based on relative uncertainty about whether other actions might yield better outcomes than status quo (Dayan & Sejnowksi 96)
  16. 16. Exploration vs Exploitation• By exploiting learned strategies, we know we can get a certain amount of reward• But don’t know how good it can get. ⇒ Need to Explore• Theory: Explore based on relative uncertainty about whether other actions might yield better outcomes than status quo (Dayan & Sejnowksi 96)
  17. 17. Uncertainty-Based Exploration Exploration 4000 Model Exp term 3000 RT diff 2000RT Diff (ms) 1000 0 −1000 −2000 −3000 Single Subject, CEV −4000 5 10 15 20 25 30 35 40 45 50 Trial
  18. 18. PFC Gene-Dose Effect on Uncertainty-Based Exploration COMT gene-dose effects Uncertainty-exploration parameter 0.50 0.45 val/val 0.40 val/met (x 1e4) met/met 0.35 0.30 0.25 ε 0.20 0.15 0.10 0.05 0.00 Frank, Doll, Oas-Terpstra & Moreno (2009, Nature Neuroscience)
  19. 19. Does the brain track relative uncertainty for exploration?
  20. 20. Does the brain track relative uncertainty for exploration? ǫ > 0 (’explorers’) explorers > non-explorers Badre, Doll, Long & Frank, under review
  21. 21. EEG reveals temporal dynamics
  22. 22. EEG reveals temporal dynamicsRelative uncertainty represented prior to choice, and more so in exploratory trials Cavanagh, Cohen, Figueroa & Frank, under review
  23. 23. Negative symptoms in schizophrenia: Uncertainty-Based Exploration Anhedonia & Exploration Uncertainty-driven exploration 0.8 0.40 0.6 0.35 SZ CN 0.4 ε (x 1e4) 0.30 0.2 0 ε (x1e4) 0.25 0.20 -0.2 ** -0.4 0.15 -0.6 0.10 -0.8 r = -.44, p = .002 0.05 -1.0 0.00 -1.2 0 1 2 3 4 ε(uncert) Global Anhedonia• Anhedonia = behavioral component of reward seeking (e.g., initiating social/recreational activities) not capacity to experience pleasure• Anhedonia related to exploration and not learning from reward prediction errors Strauss et al, 2011, Biological Psychiatry
  24. 24. Obsessive Compulsive Disorder: Aversion to Uncertainty Uncertainty-driven exploration 0.6 CN 0.4 OCD ε (x 1e4) 0.2 0.0 -0.2 -0.4 gains lossespreliminary data, N=17 per group with Mascha van ’t Wout, Ben Greenberg, Steve Rasmussen
  25. 25. Summary• Dopamine modulates reinforcement learning and choice based on positive and negative outcomes: patients, pharmacology, genetics, imaging• Prefrontal cortex tracks outcome uncertainty so as to reduce it• Disruption of these mechanisms is associated with fronto-striatal disorders, Parkinson’s, schizophrenia, OCD• Models integrate between multiple levels of analysis: neural mechanism to abstract computation (see Thomas Wiecki demonstration tomorrow!).
  26. 26. Thanks To...Bradley DollChristina FigueroaJim CavanaghDavid BadreJeff CockburnAnne CollinsThomas WieckiJim GoldKent HutchisonMascha van ’t WoutNicole LongMike CohenAhmed MoustafaScott Sherman Lab for Neural Computation and CognitionThe patients

×