Connected and Automated Vehicles

Benchmarks for Reinforcement Learning in Mixed-Autonomy Traffic

Vinitsky, Eugene
Kreidieh, Aboudy
Flem, Luc Le
Kheterpal, Nishant
Jang, Kathy
Bayen, Alexandre M.
2018

We release new benchmarks in the use of deep reinforcement learning (RL) to create controllers for mixed-autonomy traffic, where connected and autonomous vehicles (CAVs) interact with human drivers and infrastructure. Benchmarks, such as Mujoco or the Arcade Learning Environment, have spurred new research by enabling researchers to effectively compare their results so that they can focus on algorithmic improvements and control techniques rather than system design. To promote similar advances in traffic control via RL, we propose four benchmarks, based on three new traffic scenarios,...

Dissipating Stop-and-Go Waves in Closed and Open Networks via Deep Reinforcement Learning

Kreidieh, Abdul Rahman
Wu, Cathy
Bayen, Alexandre M.
2018

This article demonstrates the ability for model-free reinforcement learning (RL) techniques to generate traffic control strategies for connected and automated vehicles (CAVs) in various network geometries. This method is demonstrated to achieve near complete wave dissipation in a straight open road network with only 10% CAV penetration, while penetration rates as low as 2.5% are revealed to contribute greatly to reductions in the frequency and magnitude of formed waves. Moreover, a study of controllers generated in closed network scenarios exhibiting otherwise similar densities and...

Lagrangian Control through Deep-RL: Applications to Bottleneck Decongestion

Vinitsky, Eugene
Parvate, Kanaad
Kreidieh, Aboudy
Wu, Cathy
Bayen, Alexandre
2018

Using deep reinforcement learning, we derive novel control policies for autonomous vehicles to improve the throughput of a bottleneck modeled after the San Francisco-Oakland Bay Bridge. Using Flow, a new library for applying deep reinforcement learning to traffic micro-simulators, we consider the problem of improving the throughput of a traffic benchmark: a two-stage bottleneck where four lanes reduce to two and then reduce to one. We first characterize the inflow-outflow curve of this bottleneck without any control. We introduce an inflow of autonomous vehicles with the intent of...

Simulation to Scaled City: Zero-Shot Policy Transfer for Traffic Control via Autonomous Vehicles

Jang, Kathy
Vinitsky, Eugene
Chalaki, Behdad
Remer, Ben
Beaver, Logan E.
Malikopoulos, Andreas A.
Bayen, Alexandre
2019

Using deep reinforcement learning, we successfully train a set of two autonomous vehicles to lead a fleet of vehicles onto a round-about and then transfer this policy from simulation to a scaled city without fine-tuning. We use Flow, a library for deep reinforcement learning in microsimulators, to train two policies, (1) a policy with noise injected into the state and action space and (2) a policy without any injected noise. In simulation, the autonomous vehicles learn an emergent metering behavior for both policies which allows smooth merging. We then directly transfer this policy without...

Backstepping-Based Time-Gap Regulation for Platoons

Chou, Fang-Chieh
Tang, Shu-Xia
Lu, Xiao-Yun
Bayen, Alexandre
2019

The time-gap regulation problem for a cascaded system consisting of platooned automated vehicles following a leading non-automated vehicle is investigated in this article. Under the assumption of uniform boundedness of the acceleration of the leading vehicle, a control design scheme is proposed via an extension of integral backstepping control method, where additional terms that counter the impact due to the speed change of the non-automated vehicle are used. Each automated vehicle is actuated by one backstepping controller, demonstrated by a recursive control design procedure based on...

Zero-Shot Autonomous Vehicle Policy Transfer: From Simulation to Real-World via Adversarial Learning

Chalaki, Behdad
Beaver, Logan E.
Remer, Ben
Jang, Kathy
Vinitsky, Eugene
Bayen, Alexandre
Malikopoulos, Andreas A.
2020

In this article, we demonstrate a zero-shot transfer of an autonomous driving policy from simulation to University of Delaware's scaled smart city with adversarial multi-agent reinforcement learning, in which an adversary attempts to decrease the net reward by perturbing both the inputs and outputs of the autonomous vehicles during training. We train the autonomous vehicles to coordinate with each other while crossing a roundabout in the presence of an adversary in simulation. The adversarial policy successfully reproduces the simulated behavior and incidentally outperforms, in terms of...

Optimizing Mixed Autonomy Traffic Flow With Decentralized Autonomous Vehicles and Multi-Agent RL

Vinitsky, Eugene
Lichtle, Nathan
Parvate, Kanaad
Bayen, Alexandre
2020

We study the ability of autonomous vehicles to improve the throughput of a bottleneck using a fully decentralized control scheme in a mixed autonomy setting. We consider the problem of improving the throughput of a scaled model of the San Francisco-Oakland Bay Bridge: a two-stage bottleneck where four lanes reduce to two and then reduce to one. Although there is extensive work examining variants of bottleneck control in a centralized setting, there is less study of the challenging multi-agent setting where the large number of interacting AVs leads to significant optimization difficulties...

Continual Learning of Microscopic Traffic Models Using Neural Networks

Farid, Yashar Zeinali
Kreidieh, Abdul Rahman
Khalighi, Farnoush
Lobel, Hans
Bayen, Alexandre M.
2021

In a mixed-autonomy traffic scenario, where human drivers and autonomous vehicles share the streets, self-driving cars need to be able to predict in a robust manner the behaviour of human-driven vehicles, in order to guarantee a safe and smooth driving experience. Although traffic theory provides several models of human drivers, these models are often parameterized by few parameters which can limit their performance in modeling complex behaviors. The lack of sufficient model capacity and the behavioral shifts in human driving reduces the usefulness of these methods in real-life...

To Pool or Not to Pool? Understanding Opportunities, Challenges, and Equity Considerations to Expanding the Market for Pooling

Lazarus, Jessica
Caicedo, Juan
Bayen, Alexandre
Shaheen, Susan A
2021

On-demand mobility services such as bikesharing, scooter sharing, and transportation network companies (TNCs, also known as ridesourcing and ridehailing) are changing the way that people travel by providing dynamic, on-demand mobility that can supplement public transit and personal-vehicle use. Adoption of on-demand mobility has soared across the United States and abroad, driven by the flexibility and affordability that these services offer, particularly in urban areas where population density and land use patterns facilitate a reliable balance of supply and demand. The growth of app-based...

Fuel Consumption Reduction of Multi-Lane Road Networks using Decentralized Mixed-Autonomy Control

Lichtle, Nathan
Vinitsky, Eugene
Gunter, George
Velu, Akash
Bayen, Alexandre M.
2021

In this work, we demonstrate optimization of fuel economy in a large, calibrated model of a portion of the Ventura Freeway using a low penetration rate of controlled autonomous vehicles. We create waves in this network using a string-unstable car-following model and introduce a ghost cell to allow waves to propagate out of the network. Using multi-agent reinforcement learning, we then design a controller that manages to partially dampen the waves and thus increase the average energy efficiency of the system, yielding a 25% fuel consumption reduction at a 10% penetration rate. Finally, we...