Heterogeneous Multi-Agent Deep Reinforcement Learning for Traffic Lights Control

Abstract

Reinforcement Learning (RL) has been extensively used in Urban Traffic Control (UTC) optimization due its capability to learn the dynamics of complex problems from interactions with the environment. Recent advances in Deep Reinforcement Learning (DRL) have opened up the possibilities for extending this work to more complex situations due to it overcoming the curse of dimensionality resulting from the exponen- tial growth of the state and action spaces when incorporating fine-grained information. DRL has been shown to work very well for UTC on a single intersection, however, due to large training times, multi-junction imple- mentations have been limited to training a single agent and replicating behaviour to other junctions, assuming homogeneity of all agents. This study proposes the usage of Independent Deep Q-Network (IDQN) to train multiple heterogeneous agents, which is a more realistic scenario given heterogeneity of junction layouts in the city.We use Dueling Double Deep Q-Networks (DDDQNs) with prioritized experience replay to train each individual agent separately for its own conditions. We enrich this approach with fingerprinting to disambiguate the age of the data sampled from the replay memory to mitigate non-stationarity effects resulting from other agents affecting the environment. Our IDQN approach is evaluated on three connected heterogeneous junc- tions in low and high traffic conditions, implementing different combi- nations of standard and prioritized experience replay and fingerprinting. Results show that IDQN is suitable approach to optimization in hetero- geneous UTC with the best performance achieved by the combination of IDQN with prioritized experience replay but without fingerprinting.