The document describes a temporal classifier system that uses spiking neural networks to handle tasks with continuous space and time. It uses Integrate-and-Fire neurons in the spiking networks to introduce temporal functionality. The system includes self-adaptive parameters that control mutation rates, neural constructivism for adding/removing neurons, and connection selection for pruning connections. This allows the system to autonomously control its learning and adapt the network topology based on the environment. The system is tested on continuous grid world and mountain car tasks, as well as a robotics simulation, and is able to learn optimal policies for the tasks by leveraging the temporal aspects of the spiking networks.

1. A temporal classifier system using spiking neural networks Gerard David Howard, Larry Bull & Pier-Luca Lanzi {david4.howard, larry.bull} @uwe.ac.uk pierluca.lanzi @polimi.it 1

2. Contents Intro & Motivation System architecture – Spiking XCSF Constructivism (nodes and connections) Working in continuous space Comparison to MLP / Q-learner Taking time into consideration Comparison to MLP Simulated robotics 2

3. Motivation Many real-world tasks incorporate continuous space and continuous time Autonomous robotics are an unanswered question: will require some degree of knowledge “self-shaping” or control over their internal knowledge representation We introduce an LCS containing spiking networks and demonstrate the usefulness of the representation Handles continuous space and continuous time Representation structure dependent on environment 3

4. XCSF Includes computed prediction, which is calculated from input state (augmented by constant x0) and a weight vector – each classifier has a weight vector Weights are updated linearly using modified delta rule Main differences from canonical: SNN replaces condition and calculates action Self-adaptive parameters give autonomous learning control Topology of networks altered in GA cycle Generalisation from computed prediction, computed actions and network topologies 4

5. Spiking networks Spiking networks have temporal functionality We use Integrate-and-Fire (IAF) neurons Each neuron has a membrane potential (m) that varies through time When m exceeds a threshold, the neuron sends a spike to every neuron in the network that it has a forward connection to, and resets m Membrane potential is a way of implementing memory 5

7. Self-adaptive parameters During a GA cycle, a parent’s µ value is copied to its offspring and altered The offspring then applies its own µ to itself (bounded [0-1]) before being inserted into the population. Similar to ES mutation alteration Mutate µ  µ * e N(0,1) Insert Copy µ [A] Mutate µ  µ * e N(0,1) Copy µ Insert 7

8. Constructivism Neural Constructivism - interaction with environment guides learning process by growing/pruning dendritic connectivity Constructivism can add or remove neurons from the hidden layer during a GA event Two new self-adaptive values control NC , ψ (probability of constructivism event occurring) and ω (probability of adding rather than removing a node). These are modified during a GA cycle as with µ 8 Randomly initialised weights

11. Continuous Grid World Two dimensional continuous grid environment Runs from 0 – 1 in both x and y axes Goal state is where (x+y>1.9) – darker regions of grid represent higher expected payoff. Reaching goal returns a reward of 1000, else 0 Agent starts randomly anywhere in the grid except the goal state, aims to reach goal (moving 0.05) in fewest possible steps (avg. opt. 18.6) 1.00 0.50 Agent 0.00 0.50 1.00 11

12. Discrete movement Agent can make a single discrete movement (N,E,S,W) N=(HIGH,HIGH), E=(HIGH,LOW) etc… Experimental parameters N=20000,γ =0.95, β=0.2, ε0=0.005, θGA=50, θDEL=50 XCSF parameters as normal. Initial prediction error in new classifiers=0.01, initial fitness=0.1 Additional trial from fixed location lets us perform t-tests. “Stability” shows first step that 50 consecutive trials reach goal state from this location. 12

14. Fewer macroclassifiers = greater generalisation

16. Continuous duration actions Reward usually calculated as Reward is now calculated as Two discount factors that favour overall effectiveness and efficient state transitions respectively =0.05, ρ=0.1 tt = total steps for entire trial ti = duration of a single action Timeout=20; new steps to goal is 1.5 15

20. Tabular Q-learner cannot learn – too many (s,a) combinations, long action chains!

21. Spiking non-TCS cannot learn – too many (s,a) combinations, long action chains!

22. MLP TCS cannot learn – lack of memory?

23. Spiking TCS canlearn to optimally solve this environment by extending an action set across multiple states and recalculating actions where necessary

26. Guide a car out of a valley, sometimes requiring non-obvious behaviour

27. State comprises position and velocity

28. Actions increase/decrease velocity: HIGH/HIGH = increase, LOW/LOW = decrease, anything else = no change.

29. Noise! (+/- 5% of both state elements)

30. TCS optimal steps to goal = 1

31. N=1000

34. Simulate a Khepera robot that uses 3 IR and 3 light sensors as input state

35. Two bump sensors detect a collision and reverse the robot/reform [M] if collision is detected

36. Task: similar to grid world, but with an obstacle to avoid and a light source to reach.

37. 3 actions possible

39. Robot start position constrained so that obstacle is always between it and the light source

40. Movement is much more granular than with grid(0.05) !

42. Robotics 26 Steps to goal Connected hidden layer nodes Percentage enabled connections Self-adaptive parameters, μ, ψ, τ all plotted on RHS axis

44. Initially seeding with 6 hidden layer nodes still let’s us use connection selection to generate behavioural variation in the networks

45. Temporal functionality of the networks is exploited so that a single action set canDrop unwanted classifiers to change it’s favoured action at specific points (e.g. just before a collision) Alter the action advocated action of a majority of classifiers in [A] for the same effect

46. Thanks for your time! 28