Intelligent Transportation Systems

Robust Reinforcement Learning using Adversarial Populations

Vinitsky, Eugene
Du, Yuqing
Parvate, Kanaad
Jang, Kathy
Abbeel, Pieter
Bayen, Alexandre
2020

Reinforcement Learning (RL) is an effective tool for controller design but can struggle with issues of robustness, failing catastrophically when the underlying system dynamics are perturbed. The Robust RL formulation tackles this by adding worst-case adversarial noise to the dynamics and constructing the noise distribution as the solution to a zero-sum minimax game. However, existing work on learning solutions to the Robust RL formulation has primarily focused on training a single RL agent against a single adversary. In this work, we demonstrate that using a single adversary does not...

Zero-Shot Autonomous Vehicle Policy Transfer: From Simulation to Real-World via Adversarial Learning

Chalaki, Behdad
Beaver, Logan E.
Remer, Ben
Jang, Kathy
Vinitsky, Eugene
Bayen, Alexandre
Malikopoulos, Andreas A.
2020

In this article, we demonstrate a zero-shot transfer of an autonomous driving policy from simulation to University of Delaware's scaled smart city with adversarial multi-agent reinforcement learning, in which an adversary attempts to decrease the net reward by perturbing both the inputs and outputs of the autonomous vehicles during training. We train the autonomous vehicles to coordinate with each other while crossing a roundabout in the presence of an adversary in simulation. The adversarial policy successfully reproduces the simulated behavior and incidentally outperforms, in terms of...

Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms

Yu, Chao
Velu, Akash
Vinitsky, Eugene
Wang, Yu
Bayen, Alexandre
Wu, Yi
2020

We benchmark commonly used multi-agent deep reinforcement learning (MARL) algorithms on a variety of cooperative multi-agent games. While there has been significant innovation in MARL algorithms, algorithms tend to be tested and tuned on a single domain and their average performance across multiple domains is less characterized. Furthermore, since the hyperparameters of the algorithms are carefully tuned to the task of interest, it is unclear whether hyperparameters can easily be found that allow the algorithm to be repurposed for other cooperative tasks with different reward structure and...

Hamilton-Jacobi Formulation for State-Constrained Optimal Control and Zero-Sum Game Problems

Lee, Donggun
Keimer, Alexander
Bayen, Alexandre M.
Tomlin, Claire J.
2020

This paper presents a general Hamilton-Jacobi (HJ) framework for optimal control and two-player zero-sum game problems, both with state constraints. In the optimal control problem, a control signal and terminal time are determined to minimize the given cost and satisfy the state constraints. In the game problem, the two players interact via the system dynamics. Here, a strategy for each player, as well as a terminal time, are determined so that player A minimizes the cost and satisfies the state constraints while player B tries to prevent the success of player A. Dynamics, costs, and state...

ResiliNet: Failure-Resilient Inference in Distributed Neural Networks

Yousefpour, Ashkan
Nguyen, Brian Q.
Devic, Siddartha
Wang, Guanhua
Bayen, Alexandre
2020

Federated Learning aims to train distributed deep models without sharing the raw data with the centralized server. Similarly, in distributed inference of neural networks, by partitioning the network and distributing it across several physical nodes, activations and gradients are exchanged between physical nodes, rather than raw data. Nevertheless, when a neural network is partitioned and distributed among physical nodes, failure of physical nodes causes the failure of the neural units that are placed on those nodes, which results in a significant performance drop. Current approaches focus...

A Graph Convolutional Network with Signal Phasing Information for Arterial Traffic Prediction

Chan, Victor
Gan, Qijian
Bayen, Alexandre
2020

Accurate and reliable prediction of traffic measurements plays a crucial role in the development of modern intelligent transportation systems. Due to more complex road geometries and the presence of signal control, arterial traffic prediction is a level above freeway traffic prediction. Many existing studies on arterial traffic prediction only consider temporal measurements of flow and occupancy from loop sensors and neglect the rich spatial relationships between upstream and downstream detectors. As a result, they often suffer large prediction errors, especially for long horizons. We fill...

Deep Truck Cruise Control: Field Experiments and Validation of Heavy Duty Truck Cruise Control Using Deep Reinforcement Learning

Albeaik, Saleh
Wu, Trevor
Vurimi, Ganeshnikhil
Chou, Fang-Chieh
Lu, Xiao-Yun
Bayen, Alexandre M.
2022

Building control systems for heavy duty trucks have historically been dependent on availability of the details of the mechanical configuration of each target truck. This article investigates transfer and robustness of continuous control systems learned using model free deep-RL as an alternative; a configuration agnostic strategy for control system development. For this purpose, deep-RL cruise control policies are developed and validated in simulation and field experiments using two differently configured trucks; full-size Volvo and Freightliner trucks. Their performance are validated for...

Deep Truck Cruise Control: Field Experiments and Validation of Heavy Duty Truck Cruise Control Using Deep Reinforcement Learning

Albeaik, Saleh
Wu, Trevor
Vurimi, Ganeshnikhil
Chou, Fang-Chieh
Bayen, Alexandre M.
2022

Building control systems for heavy duty trucks have historically been dependent on availability of the details of the mechanical configuration of each target truck. This article investigates transfer and robustness of continuous control systems learned using model free deep-RL as an alternative; a configuration agnostic strategy for control system development. For this purpose, deep-RL cruise control policies are developed and validated in simulation and field experiments using two differently configured trucks; full-size Volvo and Freightliner trucks. Their performance are validated for...

Composing MPC with LQR and Neural Network for Amortized Efficiency and Stable Control

Wu, Fangyu
Wang, Guanhua
Zhuang, Siyuan
Wang, Kehan
Keimer, Alexander
Stoica, Ion
Bayen, Alexandre
2022

Model predictive control (MPC) is a powerful control method that handles dynamical systems with constraints. However, solving MPC iteratively in real time, i.e., implicit MPC, remains a computational challenge. To address this, common solutions include explicit MPC and function approximation. Both methods, whenever applicable, may improve the computational efficiency of the implicit MPC by several orders of magnitude. Nevertheless, explicit MPC often requires expensive pre-computation and does not easily apply to higher-dimensional problems. Meanwhile, function approximation, although...

Adaptive Coordination Offsets for Signalized Arterial Intersections using Deep Reinforcement Learning

Diaz, Keith Anshilo
Dailisan, Damian
Sharaf, Umang
Santos, Carissa
Bayen, Alexander M.
2022

Coordinating intersections in arterial networks is critical to the performance of urban transportation systems. Deep reinforcement learning (RL) has gained traction in traffic control research along with data-driven approaches for traffic control systems. To date, proposed deep RL-based traffic schemes control phase activation or duration. Yet, such approaches may bypass low volume links for several cycles in order to optimize the network-level traffic flow. Here, we propose a deep RL framework that dynamically adjusts offsets based on traffic states and preserves the planned phase timings...