Machine learning-based test selection for simulation-based testing of self-driving cars software

Birchler, Christian; Khatiri, Sajad; Bosshard, Bill; Gambi, Alessio; Panichella, Sebastiano

doi:10.1007/s10664-023-10286-y

Please use this identifier to cite or link to this item: https://doi.org/10.21256/zhaw-27773

Publication type:	Article in scientific journal
Type of review:	Peer review (publication)
Title:	Machine learning-based test selection for simulation-based testing of self-driving cars software
Authors:	Birchler, Christian Khatiri, Sajad Bosshard, Bill Gambi, Alessio Panichella, Sebastiano
et. al:	No
DOI:	10.1007/s10664-023-10286-y 10.21256/zhaw-27773
Published in:	Empirical Software Engineering
Volume(Issue):	28
Issue:	71
Issue Date:	26-Apr-2023
Publisher / Ed. Institution:	Springer
ISSN:	1382-3256 1573-7616
Language:	English
Subjects:	Self-driving car; Software simulation; Regression testing; Test case selection; Industrial integration
Subject (DDC):	005: Computer programming, programs and data 006: Special computer methods
Abstract:	Simulation platforms facilitate the development of emerging Cyber-Physical Systems (CPS) like self-driving cars (SDC) because they are more efficient and less dangerous than field operational test cases. Despite this, thoroughly testing SDCs in simulated environments remains challenging because SDCs must be tested in a sheer amount of long-running test cases. Past results on software testing optimization have shown that not all the test cases contribute equally to establishing confidence in test subjects’ quality and reliability, and the execution of “safe and uninformative” test cases can be skipped to reduce testing effort. However, this problem is only partially addressed in the context of SDC simulation platforms. In this paper, we investigate test selection strategies to increase the cost-effectiveness of simulation-based testing in the context of SDCs. We propose an approach called SDC-Scissor (SDC coS t-effeC tI ve teS t S electOR) that leverages Machine Learning (ML) strategies to identify and skip test cases that are unlikely to detect faults in SDCs before executing them. Our evaluation shows that SDC-Scissor outperforms the baselines. With the Logistic model, we achieve an accuracy of 70%, a precision of 65%, and a recall of 80% in selecting tests leading to a fault and improved testing cost-effectiveness. Specifically, SDC-Scissor avoided the execution of 50% of unnecessary tests as well as outperformed two baseline strategies. Complementary to existing work, we also integrated SDC-Scissor into the context of an industrial organization in the automotive domain to demonstrate how it can be used in industrial settings.
Further description:	Erworben im Rahmen der Schweizer Nationallizenzen (http://www.nationallizenzen.ch)
URI:	https://digitalcollection.zhaw.ch/handle/11475/27773
Related research data:	https://doi.org/10.5281/zenodo.7011983
Fulltext version:	Published version
License (according to publishing contract):	CC BY 4.0: Attribution 4.0 International
Departement:	School of Engineering
Organisational Unit:	Institute of Computer Science (InIT)
Published as part of the ZHAW project:	COSMOS – DevOps for Complex Cyber-physical Systems of Systems
Appears in collections:	Publikationen School of Engineering

Files in This Item:

File	Description	Size	Format
2023_Birchler-etal_Machine-learning-based-test-selection-self-driving-car-software.pdf		5.06 MB	Adobe PDF	View/Open

Show full item record

Birchler, C., Khatiri, S., Bosshard, B., Gambi, A., & Panichella, S. (2023). Machine learning-based test selection for simulation-based testing of self-driving cars software. Empirical Software Engineering, 28(71). https://doi.org/10.1007/s10664-023-10286-y

Birchler, C. et al. (2023) ‘Machine learning-based test selection for simulation-based testing of self-driving cars software’, Empirical Software Engineering, 28(71). Available at: https://doi.org/10.1007/s10664-023-10286-y.

C. Birchler, S. Khatiri, B. Bosshard, A. Gambi, and S. Panichella, “Machine learning-based test selection for simulation-based testing of self-driving cars software,” Empirical Software Engineering, vol. 28, no. 71, Apr. 2023, doi: 10.1007/s10664-023-10286-y.

BIRCHLER, Christian, Sajad KHATIRI, Bill BOSSHARD, Alessio GAMBI und Sebastiano PANICHELLA, 2023. Machine learning-based test selection for simulation-based testing of self-driving cars software. Empirical Software Engineering. 26 April 2023. Bd. 28, Nr. 71. DOI 10.1007/s10664-023-10286-y

Birchler, Christian, Sajad Khatiri, Bill Bosshard, Alessio Gambi, and Sebastiano Panichella. 2023. “Machine Learning-Based Test Selection for Simulation-Based Testing of Self-Driving Cars Software.” Empirical Software Engineering 28 (71). https://doi.org/10.1007/s10664-023-10286-y.

Birchler, Christian, et al. “Machine Learning-Based Test Selection for Simulation-Based Testing of Self-Driving Cars Software.” Empirical Software Engineering, vol. 28, no. 71, Apr. 2023, https://doi.org/10.1007/s10664-023-10286-y.