SlideShare a Scribd company logo
one critic for two actors
Nicolas Rougier, INRIA
Robotics & Neurosciences, Bordeaux, 2016
staRring CEREBRAL CORTEX, BASAL GANGLIA & THALAMUS
SPECIAL GUEST HEBBIAN LEARNING, REINFORCEMENT LEARNING & COVERT LEARNING
IN COOPERATION WITH THOMAS BORAUD, DAISUKE KASE, CAMILLE PIRON & MEROPI TOPALIDOU
http://www.labri.fr/perso/nrougier/
PLAYING november 17th
, 2016
(1) the reasonable sort, whereby we accede to
rational arguments (2) the sort that is triggered by
external circumstances, such as overhearing a
rumor; (3) the sort that is prompted by our
submission to something within ourselves, such
as a habit formed by past actions; (4) the sort that
results from a sudden change of mood such as
might be caused by a feeling of grief; and (5) the
rare sort that is a consequence of our own voluntary
choice, which will be identified as the “will to
believe.” 



William James, 1890
Decision making
(1) the reasonable sort, whereby we accede to
rational arguments (2) the sort that is triggered by
external circumstances, such as overhearing a
rumor; (3) the sort that is prompted by our
submission to something within ourselves, such
as a habit formed by past actions; (4) the sort that
results from a sudden change of mood such as
might be caused by a feeling of grief; and (5) the
rare sort that is a consequence of our own voluntary
choice, which will be identified as the “will to
believe.” 



William James, 1890
The transition from reflex action to volitional is not
abrupt and sharp. Familiar instances of individual
acquisition of motor coordination are furnished by
the cases in which short, simple movements,
whether reflex or not, are by practice under volition
combined into new sequences and become in
time habitual in the sense that though able to be
directed they no longer require concentration of
attention upon them for their execution.

Charles Sherrington, 1906
Decision making
(1) the reasonable sort, whereby we accede to
rational arguments (2) the sort that is triggered by
external circumstances, such as overhearing a
rumor; (3) the sort that is prompted by our
submission to something within ourselves, such
as a habit formed by past actions; (4) the sort that
results from a sudden change of mood such as
might be caused by a feeling of grief; and (5) the
rare sort that is a consequence of our own voluntary
choice, which will be identified as the “will to
believe.” 



William James, 1890
The transition from reflex action to volitional is not
abrupt and sharp. Familiar instances of individual
acquisition of motor coordination are furnished by
the cases in which short, simple movements,
whether reflex or not, are by practice under volition
combined into new sequences and become in
time habitual in the sense that though able to be
directed they no longer require concentration of
attention upon them for their execution. 

Charles Sherrington, 1906
CN
+
+
+
+
+
–
–
P
Brain stem structures
(e.g., superior colliculus, PPN)
pr
pc
VTA
––
+
Prefrontal
cortex
Premotor cortex
Motor
cortex
Parieto-
temporo-
occipital
cortex
Hippocampus
Thalamus
Globus
pallidus
Striatum
Cerebellum
Amygdala
Subthalamic
nucleus
Ventral
pallidum
Nucleus
accumbens
Substantia
nigra
Claustrum
Decision making
Brain is optional
• Decision-making without a brain: how an amoeboid organism solves the two-armed bandit (2016)

Chris R. Reid, Hannelore MacDonald, Richard P. Mann, James A. R. Marshall, Tanya Latty, Simon Garnier
• Habituation in non-neural organisms: Evidence from slime moulds (2016)

Romain P. Boisseau, David Vogel & Audrey Dussutour
• A two-neuron system for adaptive goal-directed decision-making in Lymnaea (2016)

Crossley M., Staras K., Kemenes G.
• Functional organization and adaptability of a decision-making network in Aplysia (2012)

Nargeot R., Simmers J.
• Neuronal microcircuits for decision making in C. Elegans (2012)

Faumont S., Lindsay T.H., Lockery, S. R.
• Decision-making in soccer game: a developmental perspective (2005)

Rulence-Pâquesa P., Frucharta E., Drub V., Mullet E.
Decision making
Rougier & Hutt, 2012
A dual particle system (degenerated neural field)
whose initial state governs final state.
dx/dt = α(1 − x) + (x − y)(1 − x), x > 0
dy/dt = α(1 − y) + (y − x)(1 − y), y > 0
Noise (i.e. initial position) induces symmetry
breaking and final decision.
The executive decision maker
Brazil, Terry Gilliam, 1985
Decision making
The habit factor
The habit factor
A tentative definition
Yin and Knowlton (2006), Graybiel (2008), Seger and Spiering (2011)
• Elicited by a particular context or stimulus

→ stimulus-response as opposed to action-outcome
• Acquired via experience

→ require extensive training or repetition
• Performed automatically

→ the mere presence of the stimulus induces the response
• Resistant to outcome devaluation

→ disengagement from the goal
• Performed unconsciously

→ without “thinking” about it
XKCD #242
A simple question
Action-outcome then stimulus-response?

Action-outcome comes first until being transformed into stimulus-response
Action-outcome versus stimulus-response?

Both processes are present and compete for expression
Action-outcome with stimulus-response?

Final decision is a mix of both processes
Action-outcome and stimulus-response?

Processes cooperate and influence each other, always
CN
+
+
+
+
+
–
–
P
Brain stem structures
(e.g., superior colliculus, PPN)
pr
pc
VTA
––
+
Prefrontal
cortex
Premotor cortex
Motor
cortex
Parieto-
temporo-
occipital
cortex
Hippocampus
Thalamus
Globus
pallidus
Striatum
Cerebellum
Amygdala
Subthalamic
nucleus
Ventral
pallidum
Nucleus
accumbens
Substantia
nigra
Claustrum
Cortex (CTX)
• Posterior

• Motor / Premotor

• Prefrontal

Thalamus (THL)

Amygdala (AMY)
Striatum (STR)
• Caudate

• Putamen

• Nucleus Accumbens

Subthalamic Nucleus (STN)

Globus Pallidus
• Internal (GPi)

• External (GPe)

Subtantia Nigra
• pars Compacta (SNc)

• pars Reticulata (SNr)
Main structures
Functional pathways
Cortex
Motor
Cognitive
Limbic
Cerebell
um
Brainstem
Spinal C
ord
Body
Thalamus
LimbicMotor
Striatum
Limbic
Cognitive
Motor
GPi/SNr
Limbic
M
oto
r
SNc
D
opamine
GPe
Motor
Lim
bic
STN
Motor
L
im
bic
Directpath
w
ay
D1
D2
Indirectpat
hw
ay
Hyperdirect
p
athway
Direct pathway (go pathway)
CTX → STR → GPi/SNr → THL → CTX

Indirect pathway (no go pathway)
CTX → STR → GPe → GPi/SNr → THL → CTX

→ STN → THL → CTX

Hyperdirect pathway (stop pathway)

CTX → STN → GPi/SNr → THL → CTX
Segregated loops
Dopamine as RPE
Sutton & Barto, 1998 Schulz et al., 1997
Stimulus
Strongest
Action
Goal

System
Habit

System
“A broad range of neural and behavioral data
suggests that the brain contains multiple systems
for behavioral choice, including one associated with
prefrontal cortex and another with dorsolateral
striatum. However, such a surfeit of control raises
an additional choice problem: how to arbitrate
between the systems when they disagree. Here, we
consider dual-action choice systems from a
normative perspective, using the computational
theory of reinforcement learning. We identify a key
trade-off pitting computational simplicity against the
flexible and statistically efficient use of experience.
The trade-off is realized in a competition between
the dorsolateral striatal and prefrontal systems…”
Daw et al. (2005)
Stimulus
Action
Goal

System
Habit

System
Dezfouli et al. (2013)
“… Model-based reinforcement learning (RL) has
been argued to underlie the goal-directed process;
however, the way in which it interacts with habits
and the structure of the habitual process has
remained unclear. According to a flat architecture,
the habitual process corresponds to model-free RL,
and its interaction with the goal-directed process is
coordinated by an external arbitration mechanism.
Alternatively, the interaction between these
systems has recently been argued to be
hierarchical, such that the formation of action
sequences underlies habit learning and a goal-
directed process selects between goal-directed
actions and habitual sequences of actions to reach
the goal…”
Action
(explicit)
Selection
Stimulus
Fastest
Action
Goal

System
Habit

System
Ashby et al. (2007)
“… The model assumes 2 neural pathways from
sensory association cortex to the premotor area
that mediates response selection. A longer and
slower path projects to the premotor area via the
striatum, globus pallidus, and thalamus. A faster,
purely cortical path projects directly to the
premotor area. The model assumes that the
subcortical path has greater neural plasticity
because of a dopamine-mediated learning signal
from the substantia nigra. In contrast, the cortical-
cortical path learns more slowly via (dopamine
independent) Hebbian learning…”
Stimulus
Habit

System
Goal

System
“… Said differently, we managed to explicitly
dissociate reinforcement learning from Hebbian
learning and demonstrated covert learning inside
the basal ganglia. These results suggest that a
behavioral decision results from both the
cooperation (acquisition) and competition
(expression) of two distinct but entangled memory
systems, the goal-directed system and the habit
system that may represent the two ends of the
same graded phenomenon.
Action
Cooperation
“Slow”

Competition
Piron et al. (2016)
→ Humans
→ Monkeys
→ Rodents
→ Birds (Krebs et al. 1978)

→ Fish (Thomas et al. 1985)
→ Bees (Keasar et al. 2002)
→ Slime mould (Reid et al. 2016)
→ Photon (Naruse et al. 2015)
Saline or muscimol injection
into the internal part of
the Globus Pallidus (GPi)
15 minutes before session
Cue
presentation
(1.0
- 1.5
second)
Trial Start
(0.5
- 1.5
second)
Decision
(1.0
- 1.5
second)
Go
Signal
Reward
U
p
D
ow
n
Left
R
ight
Reward (juice) delivered
according to the reward
probability associated
with the chosen stimulus
Control
P=0.75
P=0.25
Two-armed bandit
Two-armed bandit
Trial start Cue presentation Go signal Decision Reward Trial stop
0.5s - 1.5s 1.0s - 1.5s 1.0s - 1.5s
Time
Novel
Condition
… P = 0.75
… P = 0.25
Novel cues
Habitual
Condition
P = 0.75
P = 0.25
Habitual cues
Reverse
Condition
P = 0.00
P = 0.33
P = 0.66
P = 1.00
Reversed cues
Standard
Condition
P = 1.00
P = 0.66
P = 0.33
P = 0.00
Standard cues
Forced
Condition
P = PX
P = PY
Adjustable cues
Leblois et al.
(2006)
TOPALIDOU et al.
(2015)
(200 LINES of python)
Missing in Action

(few lines of C)
GUTHRIE et al.
(2013)
DEAD
(6000 Lines of Delphi)
PIRON et al.
(2016)
Topalidou et al.
(in prep)
…
We redo
Science !
escobar et al., 2016

NALLAPU eT AL., 2016
CARREIRE ET AL., 2015
A long series…
We redo
Science !
ReScienceReproducible science is good. Replicated science is better.
R E S C I E N C E . G I T H U B . I O
OPEN DATAOPEN SOURCE
OPEN PEER-REVIEW
OPEN (GREEN) ACCESS
NO “BUZZ” BARRIER
COMMUNITY SUPPORTED
0€ BUDGET
REGISTER AS REVIEWER
AND YOU CAN GAIN
FAME, FORTUNE, SUCCESS
OR …
A NICE STICKER !
(YES ! A STICKER !)
BENOÎT G.
RESCIENCE CHANGED MY LIFE
MEHDI K.
IT REALLY WORKS !
XAVIER H.
TOTALLY WORTH IT !
A unit is a set of arbitrary values that can vary along time under the influence of other units
and learning.
• Distributed

→ no supervisor
• Asynchronous

→ no central clock
• Numerical

→ no symbol
• Adaptative

→ to learn something




We want to make sure that emerging properties are those of the model and not those of
the software running the model.
Group Group
Unit
Link
Layer
Link
Value
Value
Value
DANA framework
Striatum
Cognitive 4 units
GPi
Cognitive 4 units
Thalamus
Cognitive 4 units
STN
Cognitive 4 units
Cognitive loop
Cortex
Cognitive 4 units
Striatum
Motor 4 units
GPi
Motor 4 units
Thalamus
Motor 4 units
STN
Motor 4 units
Motor loop
Cortex
Motor 4 units
Associative loop
Task
Environment
Cue
Positions
Cue
Identities
Substantia
nigra pars
compacta
Substantia
nigra pars
compacta
COMPETITION
RL
HL
dopamine dopamine
reward reward
Lesion
sites
RL
HL
Cortex
Associative
4x4 units
Striatum
Associative
4x4 units
EXT EXT EXT EXT
COMPETITION
Cortex

• Posterior
• Motor / Premotor
• Prefrontal
Thalamus

Striatum (STR)

• Caudate
• Putamen
• Nucleus Accumbens
Subthalamic Nucleus (STN)

Globus Pallidus

• Internal (GPi)
• External (GPe)
Subtantia Nigra

• pars Compacta (SNc)
• pars Reticulata (SNr)
Computational model
Striatum
Cognitive 4 units
GPi
Cognitive 4 units
Thalamus
Cognitive 4 units
STN
Cognitive 4 units
Cognitive loop
Cortex
Cognitive 4 units
Striatum
Motor 4 units
GPi
Motor 4 units
Thalamus
Motor 4 units
STN
Motor 4 units
Motor loop
Cortex
Motor 4 units
Associative loop
Task
Environment
Cue
Positions
Cue
Identities
Substantia
nigra pars
compacta
Substantia
nigra pars
compacta
COMPETITION
RL
HL
dopamine dopamine
reward reward
Lesion
sites
RL
HL
Cortex
Associative
4x4 units
Striatum
Associative
4x4 units
EXT EXT EXT EXT
COMPETITION
Computational model
Dopamine + Reinforcement learning

• Cognitive cortex to cognitive striatum
• Motor cortex to motor striatum
Lateral Competition + Hebbian learning

• Cognitive cortex to associative cortex
• Motor cortex to associative cortex
Lesion

• Motor GPi to motor thalamus
• Cognitive GPi to cognitive thalamus
Habit learning
BG Actor
BG Critic
Cortex Actor
Motor Decision
Reward
Sensory
Context
HebbianLearning
Reinforcementlearning
Habit learning
Intact model (GPi On)

Faster decision (before learning)
Lesioned model (GPi Off)

Slower decision (before learning)
Habit learning
Saline / GPi ON Muscimol / GPi OFF
Habit learning
Cortex
Motor
Cognitive
Limbic
Cerebell
um
Brainstem
Spinal C
ord
Body
Thalamus
LimbicMotor
Striatum
Limbic
Cognitive
Motor
GPi/SNr
Limbic
M
oto
r
SNc
D
opamine
GPe
Motor
Lim
bic
STN
Motor
L
im
bic
Directpath
w
ay
D1
D2
Indirectpat
hw
ay
Hyperdirect
p
athway
Saline / GPi ON Muscimol / GPi OFF
Covert learning
BG Actor
BG Critic
Cortex Actor
Motor Decision
Reward
Sensory
Context
HebbianLearning
Reinforcementlearning
Covert
learning
Covert learning
Cortex
Motor
Cognitive
Limbic
Cerebell
um
Brainstem
Spinal C
ord
Body
Thalamus
LimbicMotor
Striatum
Limbic
Cognitive
Motor
GPi/SNr
Limbic
M
oto
r
SNc
D
opamine
GPe
Motor
Lim
bic
STN
Motor
L
im
bic
Directpath
w
ay
D1
D2
Indirectpat
hw
ay
Hyperdirect
p
athway
Muscimol / GPi OFF Saline / GPI ON
Covert learning
Muscimol / GPi OFF Saline / GPI ON
Conclusion
Habit acquisition and habit expression
→ These are two different processes even though they’re entangled
→ Basal ganglia serves as an implicit supervisor
→ Habit can be expressed outside BG (at least in the primate)
The critic role of the BG
→ Basal ganglia serves as a generic critic, for any “actor”
→ No experimental evidence yet for the role of the cortex
→ Ongoing experiments to measure RL vs HL influence on behavior
Habits are a graded phenomenon
Goal directed HabitsConsolidation Time
XKCD #1066
Nicolas Rougier, INRIA
Robotics & Neurosciences, Bordeaux, 2016
http://www.labri.fr/perso/nrougier/
PLAYING november 17th
, 2016
T. Boraud C. PironM. Topalidou D. kase
Questions ?
THINKING, PLEASE WAIT…
Nicolas Rougier, INRIA
Robotics & Neurosciences, Bordeaux, 2016
http://www.labri.fr/perso/nrougier/
PLAYING november 17th
, 2016
T. Boraud C. PironM. Topalidou D. kase
Nicolas Rougier, INRIA
Robotics & Neurosciences, Bordeaux, 2016
http://www.labri.fr/perso/nrougier/
PLAYING november 17th
, 2016
T. Boraud C. PironM. Topalidou D. kase
Questions ?
NOOOOO!!!
Nicolas Rougier, INRIA
Robotics & Neurosciences, Bordeaux, 2016
http://www.labri.fr/perso/nrougier/
PLAYING november 17th
, 2016
T. Boraud C. PironM. Topalidou D. kase
Questions ?

More Related Content

What's hot

172
172172
Charting the human thalamus - basic contepts and recent developments
Charting the human thalamus - basic contepts and recent developmentsCharting the human thalamus - basic contepts and recent developments
Charting the human thalamus - basic contepts and recent developments
Dr. Jakab András
 
Fundamentals of visual communication unit iii
Fundamentals of visual communication unit iiiFundamentals of visual communication unit iii
Fundamentals of visual communication unit iii
RangarajanN6
 
A General Principle of Learning and its Application for Reconciling Einstein’...
A General Principle of Learning and its Application for Reconciling Einstein’...A General Principle of Learning and its Application for Reconciling Einstein’...
A General Principle of Learning and its Application for Reconciling Einstein’...
Jeffrey Huang
 
Nativism
NativismNativism
Nativism
csmair
 
Is there a difference in the pattern of eye movements during visual imagery w...
Is there a difference in the pattern of eye movements during visual imagery w...Is there a difference in the pattern of eye movements during visual imagery w...
Is there a difference in the pattern of eye movements during visual imagery w...
James Allen
 

What's hot (6)

172
172172
172
 
Charting the human thalamus - basic contepts and recent developments
Charting the human thalamus - basic contepts and recent developmentsCharting the human thalamus - basic contepts and recent developments
Charting the human thalamus - basic contepts and recent developments
 
Fundamentals of visual communication unit iii
Fundamentals of visual communication unit iiiFundamentals of visual communication unit iii
Fundamentals of visual communication unit iii
 
A General Principle of Learning and its Application for Reconciling Einstein’...
A General Principle of Learning and its Application for Reconciling Einstein’...A General Principle of Learning and its Application for Reconciling Einstein’...
A General Principle of Learning and its Application for Reconciling Einstein’...
 
Nativism
NativismNativism
Nativism
 
Is there a difference in the pattern of eye movements during visual imagery w...
Is there a difference in the pattern of eye movements during visual imagery w...Is there a difference in the pattern of eye movements during visual imagery w...
Is there a difference in the pattern of eye movements during visual imagery w...
 

Viewers also liked

Materiales y las energías en la informática
Materiales y las energías en la informáticaMateriales y las energías en la informática
Materiales y las energías en la informática
Mariana Gutierrez
 
БЮЛЛЕТЕНЬ НАУКИ И ПРАКТИКИ (BULLETIN OF SCIENCE AND PRACTICE) № 7
БЮЛЛЕТЕНЬ НАУКИ И ПРАКТИКИ (BULLETIN OF SCIENCE AND PRACTICE) № 7БЮЛЛЕТЕНЬ НАУКИ И ПРАКТИКИ (BULLETIN OF SCIENCE AND PRACTICE) № 7
БЮЛЛЕТЕНЬ НАУКИ И ПРАКТИКИ (BULLETIN OF SCIENCE AND PRACTICE) № 7
Елена Овечкина
 
Лекция № 6. Неорганические вяжущие строительные материалы
Лекция № 6. Неорганические вяжущие строительные материалыЛекция № 6. Неорганические вяжущие строительные материалы
Лекция № 6. Неорганические вяжущие строительные материалыПетрова Елена Александровна
 
Webschool 2016 - Ecrire pour le web : être vu, être lu !
Webschool 2016 - Ecrire pour le web : être vu, être lu !Webschool 2016 - Ecrire pour le web : être vu, être lu !
Webschool 2016 - Ecrire pour le web : être vu, être lu !
mariejura
 
Webschool du Jura - Financement participatif avec Bulb in Town
Webschool du Jura - Financement participatif avec Bulb in TownWebschool du Jura - Financement participatif avec Bulb in Town
Webschool du Jura - Financement participatif avec Bulb in Town
mariejura
 
Process mining explained by an example | Episode 6
Process mining explained by an example | Episode 6Process mining explained by an example | Episode 6
Process mining explained by an example | Episode 6
horsum
 
Get High-Octane Virtual Datacenter Performance
Get High-Octane Virtual Datacenter PerformanceGet High-Octane Virtual Datacenter Performance
Get High-Octane Virtual Datacenter Performance
SolarWinds
 
Survey: IT is Everywhere (End Users’ Perspective, Australia)
Survey: IT is Everywhere (End Users’ Perspective, Australia)Survey: IT is Everywhere (End Users’ Perspective, Australia)
Survey: IT is Everywhere (End Users’ Perspective, Australia)
SolarWinds
 
Digital Pedagogies: Technology and the Australian Curriculum
Digital Pedagogies: Technology and the Australian Curriculum Digital Pedagogies: Technology and the Australian Curriculum
Digital Pedagogies: Technology and the Australian Curriculum
Daniel M Groenewald
 
Systems Administrators AUS Survey Results
Systems Administrators AUS Survey ResultsSystems Administrators AUS Survey Results
Systems Administrators AUS Survey Results
SolarWinds
 
Coding and the curriculum
Coding and the curriculumCoding and the curriculum
Coding and the curriculum
Jemima Saunders
 
QFD Quality Function Deployment - "Despliegue de la Función de la Calidad"
QFD Quality Function Deployment - "Despliegue de la Función de la Calidad"QFD Quality Function Deployment - "Despliegue de la Función de la Calidad"
QFD Quality Function Deployment - "Despliegue de la Función de la Calidad"
Ronald Alexander Medina Pinto
 
Future of Education: GEMs Merredin
Future of Education: GEMs MerredinFuture of Education: GEMs Merredin
Future of Education: GEMs Merredin
Daniel M Groenewald
 
Industry 4.0 : Digital Reinvention in Manufacturing Industry
Industry 4.0 : Digital Reinvention in Manufacturing IndustryIndustry 4.0 : Digital Reinvention in Manufacturing Industry
Industry 4.0 : Digital Reinvention in Manufacturing Industry
Ethan Chee
 

Viewers also liked (14)

Materiales y las energías en la informática
Materiales y las energías en la informáticaMateriales y las energías en la informática
Materiales y las energías en la informática
 
БЮЛЛЕТЕНЬ НАУКИ И ПРАКТИКИ (BULLETIN OF SCIENCE AND PRACTICE) № 7
БЮЛЛЕТЕНЬ НАУКИ И ПРАКТИКИ (BULLETIN OF SCIENCE AND PRACTICE) № 7БЮЛЛЕТЕНЬ НАУКИ И ПРАКТИКИ (BULLETIN OF SCIENCE AND PRACTICE) № 7
БЮЛЛЕТЕНЬ НАУКИ И ПРАКТИКИ (BULLETIN OF SCIENCE AND PRACTICE) № 7
 
Лекция № 6. Неорганические вяжущие строительные материалы
Лекция № 6. Неорганические вяжущие строительные материалыЛекция № 6. Неорганические вяжущие строительные материалы
Лекция № 6. Неорганические вяжущие строительные материалы
 
Webschool 2016 - Ecrire pour le web : être vu, être lu !
Webschool 2016 - Ecrire pour le web : être vu, être lu !Webschool 2016 - Ecrire pour le web : être vu, être lu !
Webschool 2016 - Ecrire pour le web : être vu, être lu !
 
Webschool du Jura - Financement participatif avec Bulb in Town
Webschool du Jura - Financement participatif avec Bulb in TownWebschool du Jura - Financement participatif avec Bulb in Town
Webschool du Jura - Financement participatif avec Bulb in Town
 
Process mining explained by an example | Episode 6
Process mining explained by an example | Episode 6Process mining explained by an example | Episode 6
Process mining explained by an example | Episode 6
 
Get High-Octane Virtual Datacenter Performance
Get High-Octane Virtual Datacenter PerformanceGet High-Octane Virtual Datacenter Performance
Get High-Octane Virtual Datacenter Performance
 
Survey: IT is Everywhere (End Users’ Perspective, Australia)
Survey: IT is Everywhere (End Users’ Perspective, Australia)Survey: IT is Everywhere (End Users’ Perspective, Australia)
Survey: IT is Everywhere (End Users’ Perspective, Australia)
 
Digital Pedagogies: Technology and the Australian Curriculum
Digital Pedagogies: Technology and the Australian Curriculum Digital Pedagogies: Technology and the Australian Curriculum
Digital Pedagogies: Technology and the Australian Curriculum
 
Systems Administrators AUS Survey Results
Systems Administrators AUS Survey ResultsSystems Administrators AUS Survey Results
Systems Administrators AUS Survey Results
 
Coding and the curriculum
Coding and the curriculumCoding and the curriculum
Coding and the curriculum
 
QFD Quality Function Deployment - "Despliegue de la Función de la Calidad"
QFD Quality Function Deployment - "Despliegue de la Función de la Calidad"QFD Quality Function Deployment - "Despliegue de la Función de la Calidad"
QFD Quality Function Deployment - "Despliegue de la Función de la Calidad"
 
Future of Education: GEMs Merredin
Future of Education: GEMs MerredinFuture of Education: GEMs Merredin
Future of Education: GEMs Merredin
 
Industry 4.0 : Digital Reinvention in Manufacturing Industry
Industry 4.0 : Digital Reinvention in Manufacturing IndustryIndustry 4.0 : Digital Reinvention in Manufacturing Industry
Industry 4.0 : Digital Reinvention in Manufacturing Industry
 

Similar to One critic, two actors, one decision

BF_Introduction_2011.ppt
BF_Introduction_2011.pptBF_Introduction_2011.ppt
BF_Introduction_2011.ppt
AlexMazal
 
Models of embodied rationality.
Models of embodied rationality.Models of embodied rationality.
Models of embodied rationality.
Benoit Hardy-Vallée, Ph.D.
 
Conceitos de consciencia chalmers
Conceitos de consciencia chalmersConceitos de consciencia chalmers
Conceitos de consciencia chalmers
Anderson Balbinot
 
Wrangling Complex Systems
Wrangling Complex SystemsWrangling Complex Systems
Wrangling Complex Systems
FoCAS Initiative
 
Games in humans and non-human primates - the prospects for game theoretical a...
Games in humans and non-human primates - the prospects for game theoretical a...Games in humans and non-human primates - the prospects for game theoretical a...
Games in humans and non-human primates - the prospects for game theoretical a...
Kyongsik Yun
 
Research class (1)
Research class (1)Research class (1)
Research class (1)
Jan Muhammad Kalyar
 
Genetic Mutagenesis Screen
Genetic Mutagenesis ScreenGenetic Mutagenesis Screen
Genetic Mutagenesis Screen
Lisa Martinez
 
Mark2051 Notes
Mark2051 NotesMark2051 Notes
Mark2051 Notes
Lindsey Rivera
 
Behavioral economics
Behavioral economicsBehavioral economics
Behavioral economics
Ricardo Duarte Jr
 
Neuromarketing
NeuromarketingNeuromarketing
Neuromarketing
Melissa Moore
 
Autism Research Into Causes And Intervention
Autism  Research Into Causes And InterventionAutism  Research Into Causes And Intervention
Autism Research Into Causes And Intervention
Steven Wallach
 
Libertad y Neurociencia
Libertad y NeurocienciaLibertad y Neurociencia
Libertad y Neurociencia
cultura 3.0
 
La libertad desde la ciencia
La libertad desde la cienciaLa libertad desde la ciencia
La libertad desde la ciencia
cultura 3.0
 
συνείδηση 2016
συνείδηση 2016συνείδηση 2016
συνείδηση 2016
ssuser7c8f61
 
Myelination Essay
Myelination EssayMyelination Essay
Myelination Essay
Stephanie King
 
The psychology of decision making
The psychology of decision makingThe psychology of decision making
The psychology of decision making
Aram Jafari
 
Learning Theory
Learning TheoryLearning Theory
Learning Theory
Elizabeth Temburu
 
Biophotonic route to mind brain and world
Biophotonic route to mind brain and worldBiophotonic route to mind brain and world
Biophotonic route to mind brain and world
Rajendra Bajpai
 
Neuroscienze e Libero Arbitrio
Neuroscienze e Libero ArbitrioNeuroscienze e Libero Arbitrio
Neuroscienze e Libero Arbitrio
Fondazione Giannino Bassetti
 
Experiencing a jamesian approach
Experiencing a jamesian approachExperiencing a jamesian approach

Similar to One critic, two actors, one decision (20)

BF_Introduction_2011.ppt
BF_Introduction_2011.pptBF_Introduction_2011.ppt
BF_Introduction_2011.ppt
 
Models of embodied rationality.
Models of embodied rationality.Models of embodied rationality.
Models of embodied rationality.
 
Conceitos de consciencia chalmers
Conceitos de consciencia chalmersConceitos de consciencia chalmers
Conceitos de consciencia chalmers
 
Wrangling Complex Systems
Wrangling Complex SystemsWrangling Complex Systems
Wrangling Complex Systems
 
Games in humans and non-human primates - the prospects for game theoretical a...
Games in humans and non-human primates - the prospects for game theoretical a...Games in humans and non-human primates - the prospects for game theoretical a...
Games in humans and non-human primates - the prospects for game theoretical a...
 
Research class (1)
Research class (1)Research class (1)
Research class (1)
 
Genetic Mutagenesis Screen
Genetic Mutagenesis ScreenGenetic Mutagenesis Screen
Genetic Mutagenesis Screen
 
Mark2051 Notes
Mark2051 NotesMark2051 Notes
Mark2051 Notes
 
Behavioral economics
Behavioral economicsBehavioral economics
Behavioral economics
 
Neuromarketing
NeuromarketingNeuromarketing
Neuromarketing
 
Autism Research Into Causes And Intervention
Autism  Research Into Causes And InterventionAutism  Research Into Causes And Intervention
Autism Research Into Causes And Intervention
 
Libertad y Neurociencia
Libertad y NeurocienciaLibertad y Neurociencia
Libertad y Neurociencia
 
La libertad desde la ciencia
La libertad desde la cienciaLa libertad desde la ciencia
La libertad desde la ciencia
 
συνείδηση 2016
συνείδηση 2016συνείδηση 2016
συνείδηση 2016
 
Myelination Essay
Myelination EssayMyelination Essay
Myelination Essay
 
The psychology of decision making
The psychology of decision makingThe psychology of decision making
The psychology of decision making
 
Learning Theory
Learning TheoryLearning Theory
Learning Theory
 
Biophotonic route to mind brain and world
Biophotonic route to mind brain and worldBiophotonic route to mind brain and world
Biophotonic route to mind brain and world
 
Neuroscienze e Libero Arbitrio
Neuroscienze e Libero ArbitrioNeuroscienze e Libero Arbitrio
Neuroscienze e Libero Arbitrio
 
Experiencing a jamesian approach
Experiencing a jamesian approachExperiencing a jamesian approach
Experiencing a jamesian approach
 

More from Nicolas Rougier

The Art of Braincrafting
The Art of BraincraftingThe Art of Braincrafting
The Art of Braincrafting
Nicolas Rougier
 
SIGGRAPH 2018 - Digital typography
SIGGRAPH 2018 - Digital typographySIGGRAPH 2018 - Digital typography
SIGGRAPH 2018 - Digital typography
Nicolas Rougier
 
Machine learning
Machine learningMachine learning
Machine learning
Nicolas Rougier
 
Modern OpenGL scientific visualization
Modern OpenGL scientific visualizationModern OpenGL scientific visualization
Modern OpenGL scientific visualization
Nicolas Rougier
 
Open science
Open scienceOpen science
Open science
Nicolas Rougier
 
On ne voit que ce que l’on regarde (French)
On ne voit que ce  que l’on regarde (French)On ne voit que ce  que l’on regarde (French)
On ne voit que ce que l’on regarde (French)
Nicolas Rougier
 
Cybercriminalité (French)
Cybercriminalité (French)Cybercriminalité (French)
Cybercriminalité (French)
Nicolas Rougier
 
Computational neuroscience
Computational neuroscienceComputational neuroscience
Computational neuroscience
Nicolas Rougier
 
Scientific visualization
Scientific visualizationScientific visualization
Scientific visualization
Nicolas Rougier
 
ReScience
ReScienceReScience
ReScience
Nicolas Rougier
 

More from Nicolas Rougier (10)

The Art of Braincrafting
The Art of BraincraftingThe Art of Braincrafting
The Art of Braincrafting
 
SIGGRAPH 2018 - Digital typography
SIGGRAPH 2018 - Digital typographySIGGRAPH 2018 - Digital typography
SIGGRAPH 2018 - Digital typography
 
Machine learning
Machine learningMachine learning
Machine learning
 
Modern OpenGL scientific visualization
Modern OpenGL scientific visualizationModern OpenGL scientific visualization
Modern OpenGL scientific visualization
 
Open science
Open scienceOpen science
Open science
 
On ne voit que ce que l’on regarde (French)
On ne voit que ce  que l’on regarde (French)On ne voit que ce  que l’on regarde (French)
On ne voit que ce que l’on regarde (French)
 
Cybercriminalité (French)
Cybercriminalité (French)Cybercriminalité (French)
Cybercriminalité (French)
 
Computational neuroscience
Computational neuroscienceComputational neuroscience
Computational neuroscience
 
Scientific visualization
Scientific visualizationScientific visualization
Scientific visualization
 
ReScience
ReScienceReScience
ReScience
 

Recently uploaded

Direct Seeded Rice - Climate Smart Agriculture
Direct Seeded Rice - Climate Smart AgricultureDirect Seeded Rice - Climate Smart Agriculture
Direct Seeded Rice - Climate Smart Agriculture
International Food Policy Research Institute- South Asia Office
 
fermented food science of sauerkraut.pptx
fermented food science of sauerkraut.pptxfermented food science of sauerkraut.pptx
fermented food science of sauerkraut.pptx
ananya23nair
 
Gadgets for management of stored product pests_Dr.UPR.pdf
Gadgets for management of stored product pests_Dr.UPR.pdfGadgets for management of stored product pests_Dr.UPR.pdf
Gadgets for management of stored product pests_Dr.UPR.pdf
PirithiRaju
 
_Extraction of Ethylene oxide and 2-Chloroethanol from alternate matrices Li...
_Extraction of Ethylene oxide and 2-Chloroethanol from alternate matrices  Li..._Extraction of Ethylene oxide and 2-Chloroethanol from alternate matrices  Li...
_Extraction of Ethylene oxide and 2-Chloroethanol from alternate matrices Li...
LucyHearn1
 
在线办理(salfor毕业证书)索尔福德大学毕业证毕业完成信一模一样
在线办理(salfor毕业证书)索尔福德大学毕业证毕业完成信一模一样在线办理(salfor毕业证书)索尔福德大学毕业证毕业完成信一模一样
在线办理(salfor毕业证书)索尔福德大学毕业证毕业完成信一模一样
vluwdy49
 
Applied Science: Thermodynamics, Laws & Methodology.pdf
Applied Science: Thermodynamics, Laws & Methodology.pdfApplied Science: Thermodynamics, Laws & Methodology.pdf
Applied Science: Thermodynamics, Laws & Methodology.pdf
University of Hertfordshire
 
LEARNING TO LIVE WITH LAWS OF MOTION .pptx
LEARNING TO LIVE WITH LAWS OF MOTION .pptxLEARNING TO LIVE WITH LAWS OF MOTION .pptx
LEARNING TO LIVE WITH LAWS OF MOTION .pptx
yourprojectpartner05
 
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdfMending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
Selcen Ozturkcan
 
Summary Of transcription and Translation.pdf
Summary Of transcription and Translation.pdfSummary Of transcription and Translation.pdf
Summary Of transcription and Translation.pdf
vadgavevedant86
 
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
hozt8xgk
 
Methods of grain storage Structures in India.pdf
Methods of grain storage Structures in India.pdfMethods of grain storage Structures in India.pdf
Methods of grain storage Structures in India.pdf
PirithiRaju
 
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
Scintica Instrumentation
 
Compexometric titration/Chelatorphy titration/chelating titration
Compexometric titration/Chelatorphy titration/chelating titrationCompexometric titration/Chelatorphy titration/chelating titration
Compexometric titration/Chelatorphy titration/chelating titration
Vandana Devesh Sharma
 
The binding of cosmological structures by massless topological defects
The binding of cosmological structures by massless topological defectsThe binding of cosmological structures by massless topological defects
The binding of cosmological structures by massless topological defects
Sérgio Sacani
 
Microbiology of Central Nervous System INFECTIONS.pdf
Microbiology of Central Nervous System INFECTIONS.pdfMicrobiology of Central Nervous System INFECTIONS.pdf
Microbiology of Central Nervous System INFECTIONS.pdf
sammy700571
 
Pests of Storage_Identification_Dr.UPR.pdf
Pests of Storage_Identification_Dr.UPR.pdfPests of Storage_Identification_Dr.UPR.pdf
Pests of Storage_Identification_Dr.UPR.pdf
PirithiRaju
 
Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...
Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...
Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...
Sérgio Sacani
 
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
PsychoTech Services
 
Sciences of Europe journal No 142 (2024)
Sciences of Europe journal No 142 (2024)Sciences of Europe journal No 142 (2024)
Sciences of Europe journal No 142 (2024)
Sciences of Europe
 
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDSJAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
Sérgio Sacani
 

Recently uploaded (20)

Direct Seeded Rice - Climate Smart Agriculture
Direct Seeded Rice - Climate Smart AgricultureDirect Seeded Rice - Climate Smart Agriculture
Direct Seeded Rice - Climate Smart Agriculture
 
fermented food science of sauerkraut.pptx
fermented food science of sauerkraut.pptxfermented food science of sauerkraut.pptx
fermented food science of sauerkraut.pptx
 
Gadgets for management of stored product pests_Dr.UPR.pdf
Gadgets for management of stored product pests_Dr.UPR.pdfGadgets for management of stored product pests_Dr.UPR.pdf
Gadgets for management of stored product pests_Dr.UPR.pdf
 
_Extraction of Ethylene oxide and 2-Chloroethanol from alternate matrices Li...
_Extraction of Ethylene oxide and 2-Chloroethanol from alternate matrices  Li..._Extraction of Ethylene oxide and 2-Chloroethanol from alternate matrices  Li...
_Extraction of Ethylene oxide and 2-Chloroethanol from alternate matrices Li...
 
在线办理(salfor毕业证书)索尔福德大学毕业证毕业完成信一模一样
在线办理(salfor毕业证书)索尔福德大学毕业证毕业完成信一模一样在线办理(salfor毕业证书)索尔福德大学毕业证毕业完成信一模一样
在线办理(salfor毕业证书)索尔福德大学毕业证毕业完成信一模一样
 
Applied Science: Thermodynamics, Laws & Methodology.pdf
Applied Science: Thermodynamics, Laws & Methodology.pdfApplied Science: Thermodynamics, Laws & Methodology.pdf
Applied Science: Thermodynamics, Laws & Methodology.pdf
 
LEARNING TO LIVE WITH LAWS OF MOTION .pptx
LEARNING TO LIVE WITH LAWS OF MOTION .pptxLEARNING TO LIVE WITH LAWS OF MOTION .pptx
LEARNING TO LIVE WITH LAWS OF MOTION .pptx
 
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdfMending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
Mending Clothing to Support Sustainable Fashion_CIMaR 2024.pdf
 
Summary Of transcription and Translation.pdf
Summary Of transcription and Translation.pdfSummary Of transcription and Translation.pdf
Summary Of transcription and Translation.pdf
 
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
快速办理(UAM毕业证书)马德里自治大学毕业证学位证一模一样
 
Methods of grain storage Structures in India.pdf
Methods of grain storage Structures in India.pdfMethods of grain storage Structures in India.pdf
Methods of grain storage Structures in India.pdf
 
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
(June 12, 2024) Webinar: Development of PET theranostics targeting the molecu...
 
Compexometric titration/Chelatorphy titration/chelating titration
Compexometric titration/Chelatorphy titration/chelating titrationCompexometric titration/Chelatorphy titration/chelating titration
Compexometric titration/Chelatorphy titration/chelating titration
 
The binding of cosmological structures by massless topological defects
The binding of cosmological structures by massless topological defectsThe binding of cosmological structures by massless topological defects
The binding of cosmological structures by massless topological defects
 
Microbiology of Central Nervous System INFECTIONS.pdf
Microbiology of Central Nervous System INFECTIONS.pdfMicrobiology of Central Nervous System INFECTIONS.pdf
Microbiology of Central Nervous System INFECTIONS.pdf
 
Pests of Storage_Identification_Dr.UPR.pdf
Pests of Storage_Identification_Dr.UPR.pdfPests of Storage_Identification_Dr.UPR.pdf
Pests of Storage_Identification_Dr.UPR.pdf
 
Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...
Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...
Evidence of Jet Activity from the Secondary Black Hole in the OJ 287 Binary S...
 
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
Sexuality - Issues, Attitude and Behaviour - Applied Social Psychology - Psyc...
 
Sciences of Europe journal No 142 (2024)
Sciences of Europe journal No 142 (2024)Sciences of Europe journal No 142 (2024)
Sciences of Europe journal No 142 (2024)
 
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDSJAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
JAMES WEBB STUDY THE MASSIVE BLACK HOLE SEEDS
 

One critic, two actors, one decision

  • 1.
  • 2. one critic for two actors Nicolas Rougier, INRIA Robotics & Neurosciences, Bordeaux, 2016 staRring CEREBRAL CORTEX, BASAL GANGLIA & THALAMUS SPECIAL GUEST HEBBIAN LEARNING, REINFORCEMENT LEARNING & COVERT LEARNING IN COOPERATION WITH THOMAS BORAUD, DAISUKE KASE, CAMILLE PIRON & MEROPI TOPALIDOU http://www.labri.fr/perso/nrougier/ PLAYING november 17th , 2016
  • 3. (1) the reasonable sort, whereby we accede to rational arguments (2) the sort that is triggered by external circumstances, such as overhearing a rumor; (3) the sort that is prompted by our submission to something within ourselves, such as a habit formed by past actions; (4) the sort that results from a sudden change of mood such as might be caused by a feeling of grief; and (5) the rare sort that is a consequence of our own voluntary choice, which will be identified as the “will to believe.”  
 William James, 1890 Decision making
  • 4. (1) the reasonable sort, whereby we accede to rational arguments (2) the sort that is triggered by external circumstances, such as overhearing a rumor; (3) the sort that is prompted by our submission to something within ourselves, such as a habit formed by past actions; (4) the sort that results from a sudden change of mood such as might be caused by a feeling of grief; and (5) the rare sort that is a consequence of our own voluntary choice, which will be identified as the “will to believe.”  
 William James, 1890 The transition from reflex action to volitional is not abrupt and sharp. Familiar instances of individual acquisition of motor coordination are furnished by the cases in which short, simple movements, whether reflex or not, are by practice under volition combined into new sequences and become in time habitual in the sense that though able to be directed they no longer require concentration of attention upon them for their execution.
 Charles Sherrington, 1906 Decision making
  • 5. (1) the reasonable sort, whereby we accede to rational arguments (2) the sort that is triggered by external circumstances, such as overhearing a rumor; (3) the sort that is prompted by our submission to something within ourselves, such as a habit formed by past actions; (4) the sort that results from a sudden change of mood such as might be caused by a feeling of grief; and (5) the rare sort that is a consequence of our own voluntary choice, which will be identified as the “will to believe.”  
 William James, 1890 The transition from reflex action to volitional is not abrupt and sharp. Familiar instances of individual acquisition of motor coordination are furnished by the cases in which short, simple movements, whether reflex or not, are by practice under volition combined into new sequences and become in time habitual in the sense that though able to be directed they no longer require concentration of attention upon them for their execution. 
 Charles Sherrington, 1906 CN + + + + + – – P Brain stem structures (e.g., superior colliculus, PPN) pr pc VTA –– + Prefrontal cortex Premotor cortex Motor cortex Parieto- temporo- occipital cortex Hippocampus Thalamus Globus pallidus Striatum Cerebellum Amygdala Subthalamic nucleus Ventral pallidum Nucleus accumbens Substantia nigra Claustrum Decision making
  • 6. Brain is optional • Decision-making without a brain: how an amoeboid organism solves the two-armed bandit (2016)
 Chris R. Reid, Hannelore MacDonald, Richard P. Mann, James A. R. Marshall, Tanya Latty, Simon Garnier • Habituation in non-neural organisms: Evidence from slime moulds (2016)
 Romain P. Boisseau, David Vogel & Audrey Dussutour • A two-neuron system for adaptive goal-directed decision-making in Lymnaea (2016)
 Crossley M., Staras K., Kemenes G. • Functional organization and adaptability of a decision-making network in Aplysia (2012)
 Nargeot R., Simmers J. • Neuronal microcircuits for decision making in C. Elegans (2012)
 Faumont S., Lindsay T.H., Lockery, S. R. • Decision-making in soccer game: a developmental perspective (2005)
 Rulence-Pâquesa P., Frucharta E., Drub V., Mullet E. Decision making
  • 7. Rougier & Hutt, 2012 A dual particle system (degenerated neural field) whose initial state governs final state. dx/dt = α(1 − x) + (x − y)(1 − x), x > 0 dy/dt = α(1 − y) + (y − x)(1 − y), y > 0 Noise (i.e. initial position) induces symmetry breaking and final decision. The executive decision maker Brazil, Terry Gilliam, 1985 Decision making
  • 9. The habit factor A tentative definition Yin and Knowlton (2006), Graybiel (2008), Seger and Spiering (2011) • Elicited by a particular context or stimulus
 → stimulus-response as opposed to action-outcome • Acquired via experience
 → require extensive training or repetition • Performed automatically
 → the mere presence of the stimulus induces the response • Resistant to outcome devaluation
 → disengagement from the goal • Performed unconsciously
 → without “thinking” about it XKCD #242
  • 10. A simple question Action-outcome then stimulus-response? Action-outcome comes first until being transformed into stimulus-response Action-outcome versus stimulus-response? Both processes are present and compete for expression Action-outcome with stimulus-response? Final decision is a mix of both processes Action-outcome and stimulus-response? Processes cooperate and influence each other, always
  • 11. CN + + + + + – – P Brain stem structures (e.g., superior colliculus, PPN) pr pc VTA –– + Prefrontal cortex Premotor cortex Motor cortex Parieto- temporo- occipital cortex Hippocampus Thalamus Globus pallidus Striatum Cerebellum Amygdala Subthalamic nucleus Ventral pallidum Nucleus accumbens Substantia nigra Claustrum Cortex (CTX) • Posterior • Motor / Premotor • Prefrontal Thalamus (THL)
 Amygdala (AMY) Striatum (STR) • Caudate • Putamen • Nucleus Accumbens Subthalamic Nucleus (STN) Globus Pallidus • Internal (GPi) • External (GPe) Subtantia Nigra • pars Compacta (SNc) • pars Reticulata (SNr) Main structures
  • 12. Functional pathways Cortex Motor Cognitive Limbic Cerebell um Brainstem Spinal C ord Body Thalamus LimbicMotor Striatum Limbic Cognitive Motor GPi/SNr Limbic M oto r SNc D opamine GPe Motor Lim bic STN Motor L im bic Directpath w ay D1 D2 Indirectpat hw ay Hyperdirect p athway Direct pathway (go pathway) CTX → STR → GPi/SNr → THL → CTX Indirect pathway (no go pathway) CTX → STR → GPe → GPi/SNr → THL → CTX
 → STN → THL → CTX Hyperdirect pathway (stop pathway)
 CTX → STN → GPi/SNr → THL → CTX
  • 14. Dopamine as RPE Sutton & Barto, 1998 Schulz et al., 1997
  • 15. Stimulus Strongest Action Goal
 System Habit
 System “A broad range of neural and behavioral data suggests that the brain contains multiple systems for behavioral choice, including one associated with prefrontal cortex and another with dorsolateral striatum. However, such a surfeit of control raises an additional choice problem: how to arbitrate between the systems when they disagree. Here, we consider dual-action choice systems from a normative perspective, using the computational theory of reinforcement learning. We identify a key trade-off pitting computational simplicity against the flexible and statistically efficient use of experience. The trade-off is realized in a competition between the dorsolateral striatal and prefrontal systems…” Daw et al. (2005)
  • 16. Stimulus Action Goal
 System Habit
 System Dezfouli et al. (2013) “… Model-based reinforcement learning (RL) has been argued to underlie the goal-directed process; however, the way in which it interacts with habits and the structure of the habitual process has remained unclear. According to a flat architecture, the habitual process corresponds to model-free RL, and its interaction with the goal-directed process is coordinated by an external arbitration mechanism. Alternatively, the interaction between these systems has recently been argued to be hierarchical, such that the formation of action sequences underlies habit learning and a goal- directed process selects between goal-directed actions and habitual sequences of actions to reach the goal…” Action (explicit) Selection
  • 17. Stimulus Fastest Action Goal
 System Habit
 System Ashby et al. (2007) “… The model assumes 2 neural pathways from sensory association cortex to the premotor area that mediates response selection. A longer and slower path projects to the premotor area via the striatum, globus pallidus, and thalamus. A faster, purely cortical path projects directly to the premotor area. The model assumes that the subcortical path has greater neural plasticity because of a dopamine-mediated learning signal from the substantia nigra. In contrast, the cortical- cortical path learns more slowly via (dopamine independent) Hebbian learning…”
  • 18. Stimulus Habit
 System Goal
 System “… Said differently, we managed to explicitly dissociate reinforcement learning from Hebbian learning and demonstrated covert learning inside the basal ganglia. These results suggest that a behavioral decision results from both the cooperation (acquisition) and competition (expression) of two distinct but entangled memory systems, the goal-directed system and the habit system that may represent the two ends of the same graded phenomenon. Action Cooperation “Slow” Competition Piron et al. (2016)
  • 19. → Humans → Monkeys → Rodents → Birds (Krebs et al. 1978)
 → Fish (Thomas et al. 1985) → Bees (Keasar et al. 2002) → Slime mould (Reid et al. 2016) → Photon (Naruse et al. 2015) Saline or muscimol injection into the internal part of the Globus Pallidus (GPi) 15 minutes before session Cue presentation (1.0 - 1.5 second) Trial Start (0.5 - 1.5 second) Decision (1.0 - 1.5 second) Go Signal Reward U p D ow n Left R ight Reward (juice) delivered according to the reward probability associated with the chosen stimulus Control P=0.75 P=0.25 Two-armed bandit
  • 20. Two-armed bandit Trial start Cue presentation Go signal Decision Reward Trial stop 0.5s - 1.5s 1.0s - 1.5s 1.0s - 1.5s Time Novel Condition … P = 0.75 … P = 0.25 Novel cues Habitual Condition P = 0.75 P = 0.25 Habitual cues Reverse Condition P = 0.00 P = 0.33 P = 0.66 P = 1.00 Reversed cues Standard Condition P = 1.00 P = 0.66 P = 0.33 P = 0.00 Standard cues Forced Condition P = PX P = PY Adjustable cues
  • 21. Leblois et al. (2006) TOPALIDOU et al. (2015) (200 LINES of python) Missing in Action
 (few lines of C) GUTHRIE et al. (2013) DEAD (6000 Lines of Delphi) PIRON et al. (2016) Topalidou et al. (in prep) … We redo Science ! escobar et al., 2016
 NALLAPU eT AL., 2016 CARREIRE ET AL., 2015 A long series…
  • 22. We redo Science ! ReScienceReproducible science is good. Replicated science is better. R E S C I E N C E . G I T H U B . I O OPEN DATAOPEN SOURCE OPEN PEER-REVIEW OPEN (GREEN) ACCESS NO “BUZZ” BARRIER COMMUNITY SUPPORTED 0€ BUDGET REGISTER AS REVIEWER AND YOU CAN GAIN FAME, FORTUNE, SUCCESS OR … A NICE STICKER ! (YES ! A STICKER !) BENOÎT G. RESCIENCE CHANGED MY LIFE MEHDI K. IT REALLY WORKS ! XAVIER H. TOTALLY WORTH IT !
  • 23. A unit is a set of arbitrary values that can vary along time under the influence of other units and learning. • Distributed
 → no supervisor • Asynchronous
 → no central clock • Numerical
 → no symbol • Adaptative
 → to learn something 
 
 We want to make sure that emerging properties are those of the model and not those of the software running the model. Group Group Unit Link Layer Link Value Value Value DANA framework
  • 24. Striatum Cognitive 4 units GPi Cognitive 4 units Thalamus Cognitive 4 units STN Cognitive 4 units Cognitive loop Cortex Cognitive 4 units Striatum Motor 4 units GPi Motor 4 units Thalamus Motor 4 units STN Motor 4 units Motor loop Cortex Motor 4 units Associative loop Task Environment Cue Positions Cue Identities Substantia nigra pars compacta Substantia nigra pars compacta COMPETITION RL HL dopamine dopamine reward reward Lesion sites RL HL Cortex Associative 4x4 units Striatum Associative 4x4 units EXT EXT EXT EXT COMPETITION Cortex • Posterior • Motor / Premotor • Prefrontal Thalamus
 Striatum (STR) • Caudate • Putamen • Nucleus Accumbens Subthalamic Nucleus (STN) Globus Pallidus • Internal (GPi) • External (GPe) Subtantia Nigra • pars Compacta (SNc) • pars Reticulata (SNr) Computational model
  • 25. Striatum Cognitive 4 units GPi Cognitive 4 units Thalamus Cognitive 4 units STN Cognitive 4 units Cognitive loop Cortex Cognitive 4 units Striatum Motor 4 units GPi Motor 4 units Thalamus Motor 4 units STN Motor 4 units Motor loop Cortex Motor 4 units Associative loop Task Environment Cue Positions Cue Identities Substantia nigra pars compacta Substantia nigra pars compacta COMPETITION RL HL dopamine dopamine reward reward Lesion sites RL HL Cortex Associative 4x4 units Striatum Associative 4x4 units EXT EXT EXT EXT COMPETITION Computational model Dopamine + Reinforcement learning • Cognitive cortex to cognitive striatum • Motor cortex to motor striatum Lateral Competition + Hebbian learning • Cognitive cortex to associative cortex • Motor cortex to associative cortex Lesion • Motor GPi to motor thalamus • Cognitive GPi to cognitive thalamus
  • 26. Habit learning BG Actor BG Critic Cortex Actor Motor Decision Reward Sensory Context HebbianLearning Reinforcementlearning
  • 27. Habit learning Intact model (GPi On) Faster decision (before learning) Lesioned model (GPi Off) Slower decision (before learning)
  • 28. Habit learning Saline / GPi ON Muscimol / GPi OFF
  • 30. Covert learning BG Actor BG Critic Cortex Actor Motor Decision Reward Sensory Context HebbianLearning Reinforcementlearning Covert learning
  • 32. Covert learning Muscimol / GPi OFF Saline / GPI ON
  • 33. Conclusion Habit acquisition and habit expression → These are two different processes even though they’re entangled → Basal ganglia serves as an implicit supervisor → Habit can be expressed outside BG (at least in the primate) The critic role of the BG → Basal ganglia serves as a generic critic, for any “actor” → No experimental evidence yet for the role of the cortex → Ongoing experiments to measure RL vs HL influence on behavior Habits are a graded phenomenon Goal directed HabitsConsolidation Time XKCD #1066
  • 34. Nicolas Rougier, INRIA Robotics & Neurosciences, Bordeaux, 2016 http://www.labri.fr/perso/nrougier/ PLAYING november 17th , 2016 T. Boraud C. PironM. Topalidou D. kase Questions ?
  • 35. THINKING, PLEASE WAIT… Nicolas Rougier, INRIA Robotics & Neurosciences, Bordeaux, 2016 http://www.labri.fr/perso/nrougier/ PLAYING november 17th , 2016 T. Boraud C. PironM. Topalidou D. kase
  • 36. Nicolas Rougier, INRIA Robotics & Neurosciences, Bordeaux, 2016 http://www.labri.fr/perso/nrougier/ PLAYING november 17th , 2016 T. Boraud C. PironM. Topalidou D. kase Questions ?
  • 38. Nicolas Rougier, INRIA Robotics & Neurosciences, Bordeaux, 2016 http://www.labri.fr/perso/nrougier/ PLAYING november 17th , 2016 T. Boraud C. PironM. Topalidou D. kase Questions ?