Dr Neil Walton from Manchester University shares his project on traffic analytics. ‘Deep Reinforcement Learning has seen a surge of interest in recent years. We investigate the applicability of Reinforcement Learning (RL) to Urban Traffic Control (UTC). We compare our results with commercially available UTC systems namely MOVA from the Transport Research Laboratory, BALANCE from PTV Group, and our own implementation of the SurTrac system.’

1. The Alan Turing Institute
AI for Urban Traffic Control
Neil Walton

2. The Alan Turing Institute
Summary

3. The Alan Turing Institute
o
L
• Develop SCOOT and MOVA
• 250 cities use SCOOT
• 3600 junctions in London

4. The Alan Turing Institute24/08/2018
Signal Control
The first traffic light by John Peake Knight.
Installed outside the Houses of Parliament,
1868.
The London’s first electric traffic at
St. James’s Street, 1926.
Vehicle Actuate timing from SCOOT, 1980.
Lots of solutions: TRANSYT, MOVA, SCATS,
SurTrac, PRODYM, UTOPIA, Rhodes, UTMS,
OPAC, InSync, VS-PLUS, Siemens RTO,
Flow AI,Vivacity.
Lots of methods: Queueing, Math Programming,
Genetic Algorithms, Cross Entropy,
Simulated Annealing, Lyapunov Control, MDP,
Forward DP, MPC, Reinforcement Learning.

5. The Alan Turing Institute
Video of green waves in
triple intersection (maybe
also trained vs. untrained)
Static plot of queue length vs time under different regimes
[Placeholder slide]
23/08/2018

6. The Alan Turing Institute
Reinforcement Learning
𝑶𝒃𝒋𝒆𝒄𝒕𝒊𝒗𝒆:
𝑸(𝒙, 𝒂) = 𝒎𝒂𝒙 𝝅 𝑬
𝒕
𝒓(𝒙 𝒕, 𝝅 𝒕)
𝑺𝒊𝒎𝒖𝒍𝒂𝒕𝒆: 𝒓𝒆𝒘𝒂𝒓𝒅 = 𝒓
𝑼𝒑𝒅𝒂𝒕𝒆: 𝑸 𝒙, 𝒂 𝑸 𝒙, 𝒂 + 𝜶 𝒓 + 𝑸 𝒂, 𝒙 − 𝒎𝒂𝒙 𝒂′ 𝑸(𝒙′, 𝒂′)
𝑨𝒑𝒑𝒓𝒐𝒙𝒊𝒎𝒂𝒕𝒆: 𝒘
𝜶
𝛁 𝐰 𝒓 + 𝑸 𝒘 𝒙, 𝒂 − 𝒎𝒂𝒙 𝒂′ 𝑸 𝒘(𝒙′, 𝒂′)
11/06/2018
SPB April 18
6
Agent Environment
Action
reward
state

7. The Alan Turing Institute
Software
11/06/2018
SPB April 18
7

8. The Alan Turing Institute
Deep RL Algorithms Implemented
• Deep Q-Network
• Double Deep Q-network
• Dueling Double Deep Q-network
• Dueling Double Deep Q-network with PER
• Actor-Critic
11/06/2018
SPB April 18
8

9. The Alan Turing Institute
SPB April 18

10. The Alan Turing Institute
Comparison with MOVA
• Very simple junctions about the same
23/09/2016
SPB Feb 18
10
0
10
20
30
40
50
60
70
80
90
200 400 600 800 1000 1200
VehiclesinQueue
Load
Single Cross Junction
DDQN AC MOVA

11. The Alan Turing Institute
Comparison with MOVA
• More complex junctions a big difference:
23/09/2016
SPB Feb 18
11

12. The Alan Turing Institute
Comparison with MOVA
• More complex junctions a big difference:
23/09/2016
SPB Feb 18
12

13. The Alan Turing Institute
What next: Expand, Extend, Engage
• Expand area of input. Wide area Junction-to-Junction
communication.
• Extend: different sensing input, LSTM loop detectors.
• Engage: Communicate with Vivacity and TRL on findings and
promising directs.
• Self-driving vehicles is lost for UK but what about traffic
lights…?
11/06/2018
SPB April 18
13

14. The Alan Turing Institute11/06/2018
SPB April 18
14
Thank you for listening!