Test Case Generation for Autonomous Vehicles

Scenes in CARLA simulation

There have been increased efforts recently to test and validate autonomous vehicles before being deployed to operate in the open world. However, testing the vehicle in the real world is difficult, time-consuming, and expensive. So, scenario description languages like Scenic and MSDL with samplers like random and grid search are available to generate synthetic data from simulators, replicating these real-world scenes. This setup is used to generate short scenes (test cases) by sampling across the different operating conditions of the vehicle (e.g., weather conditions, traffic density, pedestrian density). However, we point out three limitations in using these conventional samplers used for generating test cases. First, they are passive samplers, which do not use the feedback of previous results in the sampling process. Second, the variables to be sampled may have constraints that are often not included. Third, they do not balance the tradeoff between exploration and exploitation, which we hypothesize is necessary for better search space coverage. We present a testing workflow in carla simulator with a scene generation approach and two samplers called Random Neighborhood Search (RNS) and Guided Bayesian Optimization (GBO), which extend the conventional random search and Bayesian Optimization search to include the limitations. Also, to facilitate the samplers, we use a risk-based metric that evaluates how risky the scene was for the system. We demonstrate our approach using an Autonomous Vehicle example in CARLA simulation. These samplers sampled a higher number of high-risk scenes than the baselines of random search, grid search, and Halton sequence search. The figure shows different scenes generated by our samplers in the Carla simulation. GitHub

Shreyas Ramakrishna
Shreyas Ramakrishna
Senior Architect, System Safety Engineer