Neural networks involving cycles may better account for intelligence.

1. Cyclic Neural Networks
2020-01 HASIDA Kôiti
The University of Tokyo & RIKEN
hasida.Koiti@i.u-tokyo.ac.jp

2. Only partial information is available to
cognitive agents, and the distribution of
available information varies across contexts.
Only some units in a neural network are fixed
(their values are given), and the distribution of the
fixed units varies across contexts.
examples
recommendation
 predict the user’s evaluation of a product
unknown to her when her evaluations of some
other products are given
sentence recognition
 infer the whole sentence when you see/hear only
parts of it
Partiality of Information
2

3. Intelligence is a combination of cycles
to keep creating value (suppressing entropy) by
hypothesis-test cycles (feedback loops) realizing
constraints (persisting generative models).
Cyclic neural networks could perform better
than feedforward ones (finite-iteration
approximation of cyclic ones), as programs
involving loops are much more powerful than
loop-free ones.
better accuracy, as demonstrated by dual learning
fewer layers & links
 faster learning & less overlearning
Cyclic networks can combine parallelly with
each other, whereas feedforward networks
can be cascaded only.
Ubiquity of Cycles
3

4. execution & learning
Some units are fixed (their
values are given).
 Sets of the fixed units
may differ for different
execution/learning
sessions.
The other units (their values)
are updated.
learning
Weights of links are updated.
convergence guaranteed?
probably yes, given
consistent data
Cyclic Network for Constraint Satisfaction
B
A
links from units in
A to units in B
layer (set of units)
4

5. autoencoder
One visible layer is fixed.
dual learning
One visible layer is fixed in
execution.
Two visible layers are fixed in
learning.
restricted Boltzmann machine
one visible and one hidden layer
Link weights are symmetric.
Different parts of the visible layer
are fixed in different sessions.
Examples of CNN
5
hidden
visible
visible
…
…
visible
visible

6.  Cyclic networks can share units to affect each other,
implementing compound constraints.
 applications
 Languages use phonetics, phonology, lexicon, syntax, semantics,
pragmatics, common sense, etc.
 Their integration could improve speech recognition in noisy
environment, etc.
 Robots use mechanics, vision, language, common sense, etc.
Parallel Combination of Cyclic Networks
6
some units are shared

7. Combinations (links) need learning.
low modularity
Parallel Combination of FNN Modules
7
…
…
…

8. meta-learning (evolution?)
cycles to improve learning
learning
cycles to improve execution
execution
cycles to improve real-time adaptation
Layers of Cycles/Meanings
8