Optimizing Traffic Flow and Vehicle Routing Using Deep Reinforcement Learning Models

Abstract

Reinforcement learning (RL) is a powerful method for enabling agents to learn in complex, uncertain, and indeterministic environments. Traditionally, RL has been applied to problems where the state-transition model is either known or, at least, learned over time. In some more sophisticated applications of RL, model-based RL has been used to iteratively improve a model of the world and then use that model to select actions. Thus, the decision policy is often based on the best estimate of the model, but the model is often inaccurate, especially in complex, changing domains. When there is no model or when all models are just ignorant guesses, model-free RL can be used. In environments that are highly stochastic and require pivots, this can work well, even without function approximation. However, if the reward function is very complex and has a global structure, traditional model-free RL methods have little chance of converging to an optimal policy. Model-based RL can be used if the model estimates are refined rapidly enough. However, this is often not the case in complex, changing domains. We present a novel architecture that applies deep reinforcement learning to the problem of traffic signal control, in which the signals adapt to approaching traffic conditions in real-time. Our models automatically search over a space of potential signal timings to optimize the flow through a bottleneck. The model-free technique involves learning a value function and a policy that selects actions based on these value estimates. These estimates function as a model but are given by complex functions that are estimated from data. Our deep network is composed of one dual-cell long short-term memory layer that can directly store past information, which can be used to track the number of vehicles without making assumptions about the inflow of the queue. We show that our deep RL model outperforms traditional models in a virtual environment and on a real traffic light. The results of our model and its deployment open opportunities to improve the efficiency of urban transportation networks.