This document discusses intelligent traffic light control using multi-agent reinforcement learning. It summarizes three research papers on the topic. The first paper proposes a distributed Q-learning approach that considers both motorized and non-motorized traffic to achieve near-global optimization. The second designs a two-stage negotiation system where traffic lights determine green times based on real-time traffic conditions. The third applies particle swarm optimization to find optimal light cycles for large vehicular networks under various scenarios.

1. Intelligent Traffic Light Control
Mohamed Khasawneh
CIISE, Concordia University
PhD Comprehensive Exam
1

2. Outline
1. Introduction
2. Context
3. Papers
1. Intelligent Traffic Light Control Using Distributed Multi-agent Q Learning
2. Intelligent Traffic Lights: Green Time Period Negotiation
3. Optimal Cycle Program of Traffic Lights with Particle Swarm Optimization
4. Conclusions
5. Timeline
6. Future works
2

3. 1. Introduction
• Nowadays, studying Intelligent Transport Systems (ITS) is considered to be one of the hottest
topics which can improve the quality of life in many ways. Designing a good ITS system can bring
us many benefits such as saving lives by reducing the percentage of accidents, increasing the
safety for pedestrians, reducing the waiting times for both vehicles and pedestrians, optimizing
fuel consumption, reducing the emissions of CO and CO2 gases, reducing the noise pollution, etc.
3

4. 2. Context
• Smart cities
• Traffic Congestion management problem
• Intelligent Traffic signaling problem
• Connected vehicles
4

5. Challenges
• There are many challenges in designing these systems:
The lack of global view of traffic conditions at the neighboring streets or intersections.
The unexpected change in traffic condition at any given time.
The high congestion status which makes it a hard and daunting task to accomplish
optimization between vehicles and pedestrian in a fair and satisfying manner.
5

6. Papers under study
1. Intelligent Traffic Light Control Using Distributed Multi-agent Q Learning.
2. Intelligent Traffic Lights: Green Time Period Negotiation.
3. Optimal Cycle Program of Traffic Lights with Particle Swarm Optimization.
6

7. Paper 1.
Intelligent Traffic Light Control Using Distributed
Multi-agent Q Learning
7

8. Problem Definition
• Designing Intelligent traffic light control, where two main aspects are not considered in the
previous studies:
Non motorized traffic.
Global optimization.
8

9. Literature Review
9
Figure 1.Classification Of Traffic Light Control Approach

10. Proposed System Design
• The proposed system design is based on the usage of Q learning which falls under the
reinforcement learning techniques.
• Q learning technique is composed of two main phases which are exploration and exploitation
phases.
• In exploration phase, the Q learning agent is randomly exploring the environment around, after
the environment is learned an exploitation phase is starting with an objective to maximize the
rewards in the whole system.
10

11. Proposed System Design
Figure 2. System Architecture for Distributed Q Learning Agent [2].
11

12. Methodology
• A Markov Decision Based model where states, actions, rewards are defined.
States :
𝑆𝑖,𝑑
𝑡
= {𝑞1𝑖,𝑑
𝑡
, 𝑞2𝑖,𝑑
𝑡
, … , 𝑞 𝑗𝑖,𝑑
𝑡
, 𝑚1𝑖,𝑑,𝐿
𝑡
, 𝑚1𝑖,𝑑,
𝑡
, … , 𝑚1𝑖,𝑑
𝑡
, 𝑚1𝑖,𝑑
𝑡
} (1)
Actions:
Fig. 3: Action sets (green lights) for “+” shape intersection
12

13. Methodology
Rewards:
𝑹𝒊,𝒅
𝒕
𝒂𝒊,𝒅
𝒕
, 𝒂𝒋,𝒅
𝒕
, 𝑺𝒊,𝒅
𝒕
, 𝑺𝒋,𝒅,
𝒕
𝑾𝒊,𝒅
𝒕
= −(
𝑾 𝟏,𝒅
𝒕
𝑵 𝒊
𝒋𝜺𝑵 𝒊
𝒒𝒋𝒊,𝒅
𝒕
+
𝑾 𝒛,𝒅
𝒕
𝑵 𝒊 𝑵 𝒋
𝒋𝜺𝑵 𝒊 𝒌𝜺𝑵 𝒋
𝒒 𝒌𝒋,𝒅
𝒕
+
𝑾 𝟑,𝒅
𝒕
𝟐 𝑵 𝒊
𝒋𝜺𝑵 𝒊
( 𝒎𝒋𝒊,𝒅,𝑳
𝒕
+ 𝒎𝒋𝒊,𝒅,𝑹
𝒕
)) (2)
Integrate real-world constraints and traffic rules.
13

14. Used parameters
Parameters Meaning
𝑺𝒊,𝒅
𝒕 The state at intersection i, at day d and time t.
𝒒 𝟏𝒊,𝒅
𝒕
, 𝒒 𝟐𝒊,𝒅
𝒕
, … , 𝒒𝒋𝒊,𝒅
𝒕 The queue length from intersection j to intersection i, at
day d and time t
𝒎 𝟏𝒊,𝒅,𝑳
𝒕
, 𝒎 𝟏𝒊,𝒅,𝑹
𝒕 The queue length for pedestrians at the left side from
intersection j to i, at day d and time t,
𝑹𝒊,𝒅
𝒕 The reward at intersection i, at day d and time
𝑾 𝟏,𝒅
𝒕 The weight to present the local vehicular queues at
intersection I
𝑾 𝟐,𝒅
𝒕 The weight to present the neighborhood vehicular
queues at the neighbors of intersection I
𝑾 𝟑,𝒅
𝒕 Weight to present the total pedestrian queues at
intersection I
𝒂𝒊,𝒅
𝒕 The action at intersection i, at day d and time t
14Table 1. Used Parameters

15. Results
Fig. 4: Total cumulated queue lengths for vehicles and
pedestrians in the case of high pedestrian rate
Fig. 5: Total waiting and traveling time for vehicles and
pedestrians in the case of high pedestrian rate
15

16. Results
Fig. 6: CO2 and CO emission of vehicles in the case
of high pedestrian rate
Fig. 7: Fuel consumption and noise pollution of vehicles in the
case of high pedestrian rate
Fig. 8: Total cumulated queue lengths for vehicles and
pedestrians in the case of medium pedestrian rate
Fig. 9: Total waiting and traveling time for vehicles and
pedestrians in the case of medium pedestrian rate
16

17. Conclusion
• Unlike other papers in the literature, one of the main contributions of this paper is handling both
motorized and non-motorized vehicles. Non-motorized traffic such as pedestrian and bicyclist are
also an important part of the whole system and should not be overlooked in order to achieve the
fairness among all users. Other contribution is the achievement of near global optimal solution by
gathering/exchanging all the information from/to local and neighboring intersections, this helps
in a better and more accurate decisions compared with other proposed algorithms in literature
which are either using fixed-time or dynamic control approach
17

18. Paper 2:
Intelligent Traffic Lights: Green Time Period
Negotiation
18

19. Problem Definition
• Designing Intelligent traffic light control, where two main aspects are not considered in the
previous studies:
Person-based.
The distribution of green time period between the traffic streams.
19

20. Literature Review
20
Author Approach
Dresner and Stone (2005) (2009) “reservation based” strategy
Vasirani and Ossowski(2009) (2011) “market based” strategy and extended
the work done by Dresner and Stone
Schepperle and Bohm(2008) this strategy will take in account
valuation of the driver
Table 2. Literature Review

21. Proposed System Design
• A multi-agent computerized system (MAS or "self-organized system") composed of multiple
interacting intelligent agents is proposed. Two stages are involved:
First stage : try to find a traffic plan through defining the phase and determining green time
periods
Second stage: include two decisions defining the time of terminating the current stage and
defining the next stage to be implemented.
21

22. First stage
𝑵 𝟎 =
𝑸𝑻 𝒇
𝟒
(𝑿𝒊−𝟏) + 𝒙𝒊 − 𝟏 𝟐 +
𝟏𝟐 𝒙 𝒊− .𝟔𝟕+
𝒔𝒈
𝟔𝟎𝟎
𝑸𝑻 𝒇
𝒊𝒇 𝑿𝒊 >. 𝟔𝟕 +
𝒔𝒈
𝟔𝟎𝟎
(3)
0 otherwise
𝑵 𝑴𝒂𝒙 =
𝒒 𝑪−𝒈
𝟏−𝒚
+ 𝑵 𝟎
𝒏𝒖𝒎𝒃𝒆𝒓 𝒐𝒇 𝒍𝒂𝒏𝒆𝒔
< 𝒍𝒂𝒏𝒆 𝒄𝒂𝒑𝒂𝒄𝒊𝒕𝒚 𝒒𝒖𝒆𝒖𝒆 (4)
𝑫 𝒕𝒐𝒕𝒂𝒍=𝒗𝒊𝒆𝒄𝒉𝒍𝒆𝒔+𝒑𝒆𝒅𝒆𝒔𝒕𝒓𝒂𝒊𝒏𝒔 = 𝒊=𝟏
𝑰 𝒒𝑪(𝟏−𝒈𝒊/𝑪) 𝟐
𝟐(𝟏−𝒙 𝒊 (𝒈𝒊/𝑪)
+ 𝑵 𝟎 𝑿𝒊 𝒗𝒆𝒉 𝒐𝒄𝒄 (5)
22

23. Second stage
23

24. Bid Mechanism
24

25. Case study
Variables values
Simulation Time 2 hours
The inter-green time 5s (3s yellow, 2s red).
The minimum green duration 7 second
The maximum planned cycle length 120 second
the maximum acceptable degree of saturation 80%
The saturation flows for straight-ahead
movements, right-turn movement and left-turn
movements
1800, 1600 and 1700 veh/h/lane, respectively.
average vehicle occupancy 1.2
Simulation Software AIMSUN
Table 3. Simulation Parameters
25

26. Result
Fig. 5 – Average travel time of each replication versus time: a) baseline; b) proposed
26

27. Conclusions
• This paper unveiled a person-oriented traffic signal control method for secluded intersection that
include signal timing optimization and concurrent signal plan designs with actuated-time data
about the dynamics of the network.
• The suggested control mechanism operates within two phases: first, it chooses traffic signals and
second, it undertakes negotiation of green time duration based on actual traffic situations. The
suggested control is assessed using a tiny traffic simulation tool.
27

28. Paper 3:
Optimal Cycle Program of Traffic Lights With Particle Swarm
Optimization
28

29. Problem definition
• Designing Intelligent traffic light control, where three main aspects are not considered in the
previous studies:
Large vehicular networks.
Many scenarios.
Comparison with other algorithms.
29

30. Literature Review
Mathematic models Fuzzy logic approaches Biologically inspired optimizers
McCrea and Moutari[5]( merged
knowledge-founded and continuous
calculus-founded models with the
intent of defining the road networks’
traffic flow)
 Tolba et al.( offered a Petri net-
based model to indicate traffic flow,
from both macroscopic and
macroscopic perceptions,
consideration of vehicles’ individual
trajectories and observation of
global variables, respectively
Lim et al. [4]( derived a fuzzy logic
controller during a single
intersection’s real-time local
optimization)
 Other experts [3] developed a traffic
simulator utilizing fuzzy logic agents
for traffic lights at isolated
intersections.
Cellular automata:
 Brockfeld et al. [6] utilized a CA
model whereby the city network
was instigated as a simple square
having some normal streets and
four junctions.
Neural networks (NN):
 Spall and Chin [1] illustrated an NN
considered in configuring traffic
lights’ control parameters.
Genetics Algorithm
30
Table 4.Literature Review(Paper 2)

31. Solution Approach
• PSO has been considered in [7] and [8] as a population-based meta-heuristic motivated by the
birds’ social behavior within a flock, whose initial purpose was to continually optimize problems.
• In this model, initial solutions are randomly generated .each solutions to the problem is
identified as particle locations while the particles’ population is known as swarm. For this
algorithm, xi, the position of each particle, is updated for each iteration g as follows:
where term 𝑣 𝑔+1
𝑖
is the velocity of the particle, given by:
31
𝑋 𝑔+1
𝑖
= 𝑋 𝑔
𝑖 + 𝑣 𝑔+1
𝑖
𝑣 𝑔+1
𝑖
= 𝑤. 𝑣𝑔
𝑖
+U[0,𝜑1]. (𝑃𝑔
𝑖
− 𝑥 𝑔
𝑖
)+U[0,𝜑2]. 𝑏 𝑔 − 𝑥 𝑔
𝑖
.

32. SUMO
32

33. PSO for Traffic Light Scheduling
The optimization solver proposed in this paper for optimal cycle programs for traffic light involves three
steps which are :
Solution Encoding: all traffic light logics with their phase durations are mapped into each solution and
those solutions are randomly generated by PSO
The fitness function :Each solution is evaluated, taking into account the information obtained from
events that occur during simulation by the following equation:
F(s)= 𝑣=0
𝑉 𝑗 𝑣 𝑠 + 𝑣=0
𝑉+𝐶 𝑤 𝑣 𝑠 +(𝐶 𝑠 .𝑆𝑡)
𝑉2 𝑠 +𝐶𝑟
The global optimization procedure: the optimization strategy is composed into two parts :
an optimization algorithm
a simulation procedure
33

34. Case study
34
Table 5. Simulation and PSO Parameters

35. Results
35

36. Papers comparison
Weakness Strengths
Paper 1 o Vehicle-based strategy
o does not react well to a sharp
changes in traffic patterns
o Global optimization
o Motorized and non-motorized traffic
Paper 2 o The proposed mechanism is
only admissible where the
demand of pedestrians is low.
o Person-based strategy
o the system is flexible
Paper 3 o priority for motorized traffic. o unlimited vehicular networks
o various scenario topologies
o Compared with four algorithms
36
Table 6. Papers comparison

37. Conclusions
• In this report, research papers tackling the efficient design of intelligent traffic light are discussed.
Considering different parameters to reach a different objective function.
• In [2], both motorized and non-motorized vehicles are taken into account to minimize the overall
congestion and waiting time. Whereas, in [9] an auction based intersection control strategy is
proposed with the objective function of minimizing the waiting time for each person by taking
into consideration the number of persons in each car.
• On the other hand, in another paper used an optimization method called Particle Swarm
Optimization (PSO) to find successful traffic light cycle programs with two main objectives which
are the number of vehicles arriving at the destination and the total travel time.
37

38. Timeline
38

39. Future works
• Designing a person based globally optimal intelligent traffic light which fairly considers many
parameters (motorized and non-motorized traffic).
• Designing a machine learning-based algorithm to predict traffic congestion and signal length
timings based on historical information
39

40. References
[1] Y. Feng, K. L. Head, S. Khoshmagham, and M. Zamanipour, ―A real-time adaptive signal control
in a connected vehicle environment,Transportation Research Part C: Emerging Technologies, vol. 55,
pp. 460–473, 2015.
[2] Ying Liu1, 2, Lei Liu1, Wei-Peng Chen1,Intelligent Traffic Light Control Using Distributed Multi-
Agent Q Learning, Fujitsu Laboratories of America, Inc., Sunnyvale, CA, USA
[3] Timóteo, I., Araújo, M., Rossetti, R. & Oliveira, E. 2010. TraSMAPI: An API oriented towards Multi-
Agent Systems real-time interaction with multiple Traffic Simulators. In: 13th Intern. IEEE
Conference on Intelligent Transportation Systems (ITSC), Funchal, pp. 1183-1188.
[4] Dion, F., Rakha, H. & Kang, Y.-S. 2004. Comparison of delay estimates at under-saturated and
over-saturated pre-timed signalized intersections. Transportation Research Part B: Methodological,
38(2):99-122.
[5] J. McCrea and S. Moutari, ―A hybrid macroscopic-based model for traffic flow in road
networks,Eur. J. Oper. Res., vol. 207, no. 1, pp. 676–684, 2010.
40

41. References
[6] R. Bretherton, N. Hounsell, and B. Radia. (1996). Public transport priority in SCOOT [Online].
Available: http://eprints.soton.ac.uk/75299/
[7] M. A. M. de Oca, T. St¨utzle, M. Birattari, and M. Dorigo, ―Frankenstein‘s PSO: A composite
particle swarm optimization algorithm,IEEE Trans. Evol. Comput., vol. 13, no. 5, pp. 1120–1132, Oct.
2009.
[8] J. Kennedy and R. C. Eberhart, Swarm Intelligence. San Francisco, CA, USA: Morgan Kaufmann,
2001.
[9] Cristina Vilarinho, José Pedro Tavares, Rosaldo J. F. Rossetti, Intelligent Traffic Lights: Green Time
Period Negotiato.
41

42. Thank you
42