Procedural Skill Unlearning
•
2 likes•360 views
Slides for a talk I gave for a talk at the Cognition, Perception and Cognitive Neuroscience Colloquium, Psychology Department, University of California, Santa Barbara
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (12)
Similar to Procedural Skill Unlearning
Similar to Procedural Skill Unlearning (20)
Recently uploaded
Recently uploaded (20)
Procedural Skill Unlearning
- 1. Procedural Skill Unlearning Matthew J. Crossley Department of Psychological and Brain Sciences University of California, Santa Barbara, 93106
- 2. • Build and test a computational cognitive neuroscience (CCN) model of procedural skill learning and unlearning • CCN models predict neurobiology and behavior Talk Goals
- 3. • Procedural Skills • Model Architecture • Instrumental Conditioning Applications • Category Learning Applications • Closing Remarks Outline
- 4. • Procedural Skills • Model Architecture • Instrumental Conditioning Applications • Category Learning Applications • Closing Remarks Outline
- 5. • Learned incrementally from feedback • E.g., riding a bike or playing an instrument • E.g., radiology Procedural Skills
- 6. Procedural Skills Where are the tumors?
- 8. Procedural Skills in the Lab • Appetitive Instrumental Conditioning • II Category Learning
- 9. Appealing Choices • Much is known about the relevant neurobiology for each • Each has investigated unlearning
- 10. Procedural Skills Depend on the Basal Ganglia • Basal ganglia are a collection of subcortical nuclei • Interconnects with cortex in well defined circuits • Striatum is a major input structure
- 11. Cortex Drives the Striatum
- 12. Striatum Inhibits the GPi
- 13. GPi Inhibits the Thalamus High baseline firing rate
- 14. Striatum Disinhibits the Thalamus
- 17. Procedural Learning Depends on the Striatum • Single-cell recordings Carelli, Wolske, & West, 1997; Merchant, Zainos, Hernadez, Salinas, & Romo, 1997; Romo, Merchant, Ruiz, Crespo, & Zainos, 1995 • Lesion studies Eacott & Gaffan, 1991; Gaffan & Eacott, 1995; Gaffan & Harrison, 1987; McDonald & White, 1993, 1994; Packard, Hirsch, & White, 1989; Packard & McGaugh, 1992 • Neuropsychological patient studies Filoteo, Maddox, & Davis, 2001; Filoteo, Maddox, Salmon, & Song, 2005; Knowlton, Mangels, & Squire, 1996 • Neuroimaging Nomura et al., 2007; Seger & Cincotta, 2002; Waldschmidt & Ashby, 2011
- 18. Striatal Neurons Medium Spiny Projection Neurons (MSNs) 96% GABA Interneurons 2% TANs - Cholinergic Interneurons 2%
- 19. The TANs are of Particular Interest • Tonically active and pause to excitatory input • Presynaptically inhibit cortical input to MSNs • Get major input from CM-Pf • Learn to pause to stimuli that predict reward (requires dopamine)
- 20. • Procedural Skills • Model Architecture • Instrumental Conditioning Applications • Category Learning Applications • Closing Remarks Outline
- 21. Model Architecture Ashby and Crossley (2011)
- 22. Learning Occurs at the CTX-MSN Synapse and at Pf-TAN Synapses Pf-TAN Synapse CTX-MSN Synapse Ashby and Crossley (2011)
- 25. Network Dynamics - Early Trial
- 26. Network Dynamics - Early Trial
- 27. Network Dynamics - Early Trial SMA
- 28. Response and Feedback • Model responds if SMA crosses threshold • Model is given feedback after every trial
- 29. Learning Occurs at the CTX-MSN Synapse and at Pf-TAN Synapses Pf-TAN Synapse CTX-MSN Synapse Ashby and Crossley (2011)
- 30. CTX-MSN Synaptic Modification Requires a TANs Pause • Synaptic Strengthening: - Strong presynaptic activation - Strong postsynaptic activation - Elevated DA levels • Synaptic Weakening: - Strong presynaptic activation - Strong postsynaptic activation - Depressed DA levels Arbuthnott, Ingham, & Wickens (2000) Calabresi, Pisani, Mercuri, & Bernardi (1996) Reynolds & Wickens (2002)
- 31. Synaptic Plasticity in the Striatum Depends on Dopamine (DA) • Synaptic Strengthening: - Strong presynaptic activation - Strong postsynaptic activation - Elevated DA levels • Synaptic Weakening: - Strong presynaptic activation - Strong postsynaptic activation - Depressed DA levels Arbuthnott, Ingham, & Wickens (2000) Calabresi, Pisani, Mercuri, & Bernardi (1996) Reynolds & Wickens (2002)
- 32. DA Encodes Reward Prediciton Error (RPE) • Elevated after unexpected reward • Depressed after unexpected no-reward • Does nothing if anything expected happens Bayer & Glimcher (2005)
- 33. Computing RPE Obtained feedback on trial n: Predicted feedback on trial n: Rn = 1 if positive feedback 0 otherwise Pn = Pn 1 + (Rn 1 Pn 1) RPE on trial n: RPE(n) = Rn Pn
- 34. DA Released on Trial n DA(n) = ⌅⇤ ⌅⇥ 1 if RPE > 1 0.8RPE + 0.2 if 0.25 < RPE 1 0 if RPE < 0.25
- 35. Updating Synapses in the Model ! wK,J (n +1) = wK,J (n) + "wIK (n) SJ (n) #$NMDA[ ] + D(n) # Dbase[ ] + 1# wK,J (n)[ ] # %wIK (n) SJ (n) #$NMDA[ ] + Dbase # D(n)[ ] + wK,J (n) # &wIK (n) $NMDA # SJ (n)[ ] + ' SJ (n) #$AMPA[ ] + wK,J (n). Presynaptic Activity Presynaptic Activity Synaptic Strengthening Synaptic Weakening
- 36. Updating Synapses in the Model ! wK,J (n +1) = wK,J (n) + "wIK (n) SJ (n) #$NMDA[ ] + D(n) # Dbase[ ] + 1# wK,J (n)[ ] # %wIK (n) SJ (n) #$NMDA[ ] + Dbase # D(n)[ ] + wK,J (n) # &wIK (n) $NMDA # SJ (n)[ ] + ' SJ (n) #$AMPA[ ] + wK,J (n). Postsynaptic Activation Postsynaptic Activation Synaptic Strengthening Synaptic Weakening
- 37. Updating Synapses in the Model ! wK,J (n +1) = wK,J (n) + "wIK (n) SJ (n) #$NMDA[ ] + D(n) # Dbase[ ] + 1# wK,J (n)[ ] # %wIK (n) SJ (n) #$NMDA[ ] + Dbase # D(n)[ ] + wK,J (n) # &wIK (n) $NMDA # SJ (n)[ ] + ' SJ (n) #$AMPA[ ] + wK,J (n). Elevated DA Depressed DA Synaptic Strengthening Synaptic Weakening
- 40. Network Dynamics - Late Trial
- 41. Network Dynamics - Late Trial
- 42. Network Dynamics - Late Trial SMA
- 43. Model Accounts for Electrophysiological Recordings from TANs Ashby and Crossley (2011)
- 44. Model Accounts for Electrophysiological Recordings from MSNs Ashby and Crossley (2011)
- 45. Model Accounts for Basic Instrumental Conditioning Behavior Ashby and Crossley (2011) Fast reacquisition is evidence that extinction did not erase initial learning
- 46. • Procedural Skills • Model Architecture • Instrumental Conditioning Applications • Category Learning Applications • Closing Remarks Outline
- 47. Slowed Reacquisition Condition Phase Ext2 Ext8 Prf2 Prf8 Acquisition VI-30 sec VI-30 sec VI-30 sec VI-30 sec Extinction No Reinforcement No Reinforcement Lean Schedule Lean Schedule Reacquisition VI-2 min VI-8 min VI-2 min VI-8 min Woods and Bouton (2007)
- 48. Behavioral Results Crossley, Horvitz, Balsam, & Ashby (in prep)
- 49. Modeling Results Crossley, Horvitz, Balsam, & Ashby (in prep)
- 50. TANs Pause Less in Prf Conditions
- 51. Renewal - Basic Design Condition Phase ABA AAB ABC Acquisition Environment A Environment A Environment A Extinction Environment B Environment A Environment B Renewal (Extinction) Environment A Environment B Environment C Bouton et al. (2011)
- 52. Renewal
- 53. Model Architecture Crossley, Horvitz, Balsam, & Ashby (in prep)
- 54. Synaptic Plasticity at ALL Pf-TAN Synapses Crossley, Horvitz, Balsam, & Ashby (in prep)
- 55. Renewal Crossley, Horvitz, Balsam, & Ashby (in prep)
- 56. ABA Crossley, Horvitz, Balsam, & Ashby (in prep)
- 57. Instrumental Conditioning Summary • The TANs protect learning at CTX-MSN synapses. • Manipulations that keep the TANs paused during extinction leave learning at the CTX-MSN synapse subject to change.
- 58. • Procedural Skills • Model Architecture • Instrumental Conditioning Applications • Category Learning Applications • Closing Remarks Outline
- 59. The Basic Task A or B
- 60. Rule-Based Category Learning Spatial Frequency Orientation
- 62. Many Qualitative Differences Between RB and II RB II Unsupervised learning Yes No Observational learning Yes No Dual-task interference Yes No Time needed to process feedback Yes No Interference from button switch No Yes Interference from Feedback Delay No Yes II Category Learning is a Procedural Skill
- 63. The Basic Task Exemplars are randomly selected from the distribution and presented one at a time A
- 64. General Experiment Design Crossley, Maddox & Ashby (in prep) Condition Phase Active Condition Meta-Learning Condition Acquisition True Feedback Extinction Active Feedback Manipulation Reacquisition True Feedback
- 65. General Experiment Design Crossley, Maddox & Ashby (in prep) Condition Phase Active Condition Meta-Learning Condition Acquisition True Feedback True Feedback Extinction Active Feedback Manipulation Active Feedback Manipulation Reacquisition True Feedback True Feedback Rotated Categories
- 66. II Category-Unlearning Rotation of this kind massively interferes with category learning performance (Maddox, Glass, O’Brien, Filoteo & Ashby, 2010)
- 67. Experiment 1 Crossley, Maddox & Ashby (in prep) Condition Phase Random-Feedback Extinction Random-Feedback Meta-Learning Acquisition True Feedback True Feedback Extinction Random Feedback Random Feedback Reacquisition True Feedback True Feedback Rotated Categories
- 70. Experiment 1- Reacquisition Fast Reacquisition: Random feedback does not interfere with initial learning
- 71. Experiment 1- Overlay Active Meta-Learning
- 72. The results of experiment 1 are inconsistent with all existing theories of category learning Importance
- 73. Theoretical Account Network Architecture Crossley, Maddox & Ashby (in prep)
- 74. Problem with RPE DA Model • Random feedback prevents reward from becoming predicted • TANs don’t stop pausing during extinction • CTX-MSN synapses remain vulnerable to unlearning
- 75. • DA scaled by response-feedback contingency • Correlation between response confidence and feedback history A New Dopamine Model
- 76. Experiment 1 Model Results Reacquisition is steeper than Acquisition
- 77. DA model suggests the important factor to keep the TANs paused is response-feedback contingency Using the Model to Develop an Effective Unlearning Protocol
- 78. Experiment 2 Crossley, Maddox & Ashby (in prep) Condition Phase Partially-Contingent Extinction Partially-Contingent Meta-Learning Acquisition True Feedback True Feedback Extinction Partially-Contingent Partially-Contingent Reacquisition True Feedback True Feedback Rotated Categories
- 79. Experiment 2 - Acquisition
- 80. Experiment 2 - Extinction
- 81. Experiment 2 - Reacquisition Slow Reacquisition: Partially-Contingent Random feedback interferes with initial learning
- 82. Experiment 2- Overlay Active Meta-Learning
- 83. Comparing Experiment 1 and 2
- 84. Experiment 2 Model Results Reacquisition is shallower than Acquisition
- 85. Experiment 3 Crossley, Maddox & Ashby (in prep) Condition Phase Non-Contingent-40 Extinction Non-Contingent-40 Meta-Learning Acquisition True Feedback True Feedback Extinction Non-Contingent-40 Feedback Non-Contingent-40 Feedback Renewal (Extinction) True Feedback True Feedback Rotated Categories
- 86. Experiment 3 - Acquisition
- 87. Experiment 3 - Extinction
- 88. Experiment 3 - Acquisition Fast Reacquisition: Non-Contingent-40 feedback does not erase initial learning
- 89. Experiment 3- Overlay Active Meta-Learning
- 90. Experiment 3 Model Results Reacquisition is Steeper than Acquisition
- 91. Summary • TANs protect cortical-striatal synapses during periods when reward delivery is not contingent on behavior • RPE may not be sufficient to capture DA behavior under noisy feedback conditions • Key to unlearning may be to simulate a TAN pause (e.g., with drugs) or trick the TANs into pausing (e.g., with partial reliable feedback) during the unlearning process
- 92. Acknowledgments Collaborators: Greg Ashby Todd Maddox Jon Horvitz Peter Balsam ! Funding: NIMH Grant MH3760-2, Army Research Laboratory Todd Wilkinson