Title: Exploring emergent properties of recurrent neural networks using novel energy function formalism Authors: Rakesh Sengupta (Center for Creative Cognition, SR University), Anindya Pattanayak (Department of Commerce, XIM University), Surampudi Bapiraju (Cognitive Science Lab, IIIT Hyderabad) Conference: 9th International Conference on machine Learning, Optimization and Data science – LOD 2023, Grassmere, UK

1. EXPLORING EMERGENT
PROPERTIES OF
RECURRENT NEURAL
NETWORKS USING A
NOVEL ENERGY FUNCTION
FORMALISM
Rakesh Sengupta
Anindya Pattanayak
Surampudi Bapiraju

2. INTRODUCTION
 Recurrent neural networks (RNNs) are neural
architectures with feedback loops between
nodes.
 Feedback can originate from the same or
different nodes at each time step, making
RNNs behave like nonlinear dynamical
systems.
 Applications of RNNs include memory
modeling, decision making, and visual sense
of numbers (Sengupta 2014, 2017).
SENGUPTA ET AL 9/25/2023 2

3. INTRODUCTION
 On-center off-surround recurrent networks
are widely used in various domains, including
short-term memory, decision making, and
pattern recognition.
 These networks are valued for their
versatility and self-organizing capabilities due
to their inherent nonlinear mathematical
properties.
 Traditionally, stability in such networks has
relied on Lyapunov functions or conditions
preventing network divergence.
SENGUPTA ET AL 9/25/2023 3

4. INTRODUCTION
9/25/2023 SENGUPTA ET AL 4
In this paper, we introduce a novel
method for constructing Lyapunov
functions applicable to recurrent
networks.
We demonstrate the effectiveness
of this approach in specific cases,
highlighting its potential for stability
analysis.
We compare stability criteria
obtained from the energy function
formalism with conventional
ordinary differential equation-based
approaches.
We illustrate how this framework
can be used to make predictions in
real-world biological systems,
drawing from previous research.

5. ENERGYFUNCTION
FORMALISM
 We begin by considering a single layer
of fully connected recurrent neural
nodes.
 To simplify the equation, we introduce
a variable transformation by letting yi =
xi + Ci
9/25/2023 SENGUPTA ET AL 5

6. ENERGY
FUNCTION
FORMALISM
9/25/2023 SENGUPTA ET AL 6
The general time evolution of all Cohen-Grossberg
systems can be described by the equation (1)
It is worth noting that this formulation is quite
general and applicable to various network models. It
encompasses additive and shunting model networks,
continuous-time McCulloch-Pitts models, Boltzmann
machines, mean field models, and more.

7. ENERGYFUNCTION
FORMALISM
 Drawing inspiration from the energy
function principle in classical mechanics,
we can express the relationships
between the derivatives of the
activations xi and the corresponding
energy function Hi as (Eqn 1,2)
 the energy function for a particular
node I can be expressed as (3).
9/25/2023 SENGUPTA ET AL 7

8. ENERGY
FUNCTION
FORMALISM
 as ሶ
𝑥𝑖 → 0, we have 𝑏𝑖 𝑥𝑖 → ∑𝑐𝑖𝑗 𝑑𝑗 (𝑥𝑗 )
 Thus ignoring the cubic term in the
steady state we can have Eq 3.
9/25/2023 Sengupta et al 8

9. FULLENERGY
FUNCTION
 Hence, the full energy
function for the system can be
written as (with some changes in
the dummy indices) as Eq 1
 Compare it with Cohen-
Grossberg Liapunov function (eq
2)
SENGUPTA ET AL 9/25/2023 9

10. APPLICATION:
ADDITIVE
RECURRENT
NETWORK
SENGUPTA ET AL 9/25/2023 10

11. REACTIONTIMES
(SENGUPTA ET. AL., 2017)
 The driving argument behind the formulation of
reaction time for enumeration was that the reaction
time should correlate with maximum allowed
fluctuation of energy for the network.
9/25/2023 Sengupta et al 11

12. APPLICATION:
ADDITIVE
RECURRENT
NETWORK
SENGUPTA ET AL 9/25/2023 12

13. WINNER-
TAKE-ALL
(WTA)
SENGUPTA ET AL 9/25/2023 13

14. WINNER-
TAKE-ALL
(WTA)
SENGUPTA ET AL 9/25/2023 14

15. KEYTAKEAWAYS
9/25/2023 Sengupta et al 15
An analytical energy function formalism that effectively derives
the Cohen-Grossberg Lyapunov function for recurrent neural
networks
We compared stability criteria from the energy function
formulation with network analysis and found a close
agreement
For additive networks, the energy-based stability criterion
predicts the onset ofWinner-take-all (WTA) behavior,
complementing the network analysis-based criterion.
We have previously used the energy function to predict
psychophysical attributes, such as reaction times, in human
biological recurrent networks related to the visual sense of
numbers, and these predictions were experimentally verified.

16. CONTACT:
RAKESH.SENGUPTA@SRU.EDU.IN
Rakesh Sengupta Anindya Pattanayak
Surampudi Bapiraju
9/25/2023 Sengupta et al 16