Revisiting test smells in automatically generated tests : limitations, pitfalls, and opportunities

Panichella, Annibale; Panichella, Sebastiano; Fraser, Gordon; Sawant, Anand Ashok; Hellendoorn, Vincent J.

doi:10.1109/ICSME46990.2020.00056

Please use this identifier to cite or link to this item: https://doi.org/10.21256/zhaw-20869

Publication type:	Conference paper
Type of review:	Not specified
Title:	Revisiting test smells in automatically generated tests : limitations, pitfalls, and opportunities
Authors:	Panichella, Annibale Panichella, Sebastiano Fraser, Gordon Sawant, Anand Ashok Hellendoorn, Vincent J.
et. al:	No
DOI:	10.1109/ICSME46990.2020.00056 10.21256/zhaw-20869
Proceedings:	2020 IEEE International Conference on Software Maintenance and Evolution (ICSME)
Page(s):	523
Pages to:	533
Conference details:	International Conference on Software Maintenance (ICSM), Adelaide, Australia, 28 September - 2 October 2020
Issue Date:	2020
Publisher / Ed. Institution:	IEEE
ISBN:	978-1-7281-5619-4 978-1-7281-5620-0
ISSN:	2576-3148 1063-6773
Language:	English
Subject (DDC):	005: Computer programming, programs and data
Abstract:	Test smells attempt to capture design issues in test code that reduce their maintainability. Previous work found such smells to be highly common in automatically generated test-cases, but based this result on specific static detection rules; although these are based on the original definition of "test smells", a recent empirical study showed that developers perceive these as overly strict and non-representative of the maintainability and quality of test suites. This leads us to investigate how effective such test smell detection tools are on automatically generated test suites. In this paper, we build a dataset of 2,340 test cases automatically generated by EVOSUITE for 100 Java classes. We performed a multi-stage, cross-validated manual analysis to identify six types of test smells and label their instances. We benchmark the performance of two test smell detection tools: one widely used in prior work, and one recently introduced with the express goal to match developer perceptions of test smells. Our results show that these test smell detection strategies poorly characterized the issues in automatically generated test suites; the older tool’s detection strategies, especially, misclassified over 70% of test smells, both missing real instances (false negatives) and marking many smell-free tests as smelly (false positives). We identify common patterns in these tests that can be used to improve the tools, refine and update the definition of certain test smells, and highlight as of yet uncharacterized issues. Our findings suggest the need for (i) more appropriate metrics to match development practice; and (ii) more accurate detection strategies, to be evaluated primarily in industrial contexts.
Further description:	© 2020 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
URI:	https://digitalcollection.zhaw.ch/handle/11475/20869
Fulltext version:	Accepted version
License (according to publishing contract):	Licence according to publishing contract
Departement:	School of Engineering
Organisational Unit:	Institute of Computer Science (InIT)
Appears in collections:	Publikationen School of Engineering

Files in This Item:

File	Description	Size	Format
2020_Panichella-etal_Revisiting-test-smells-in-automatically-generated-tests_ICSME.pdf	Accepted Version	248.57 kB	Adobe PDF	View/Open

Show full item record

Panichella, A., Panichella, S., Fraser, G., Sawant, A. A., & Hellendoorn, V. J. (2020). Revisiting test smells in automatically generated tests : limitations, pitfalls, and opportunities [Conference paper]. 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), 523–533. https://doi.org/10.1109/ICSME46990.2020.00056

Panichella, A. et al. (2020) ‘Revisiting test smells in automatically generated tests : limitations, pitfalls, and opportunities’, in 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, pp. 523–533. Available at: https://doi.org/10.1109/ICSME46990.2020.00056.

A. Panichella, S. Panichella, G. Fraser, A. A. Sawant, and V. J. Hellendoorn, “Revisiting test smells in automatically generated tests : limitations, pitfalls, and opportunities,” in 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2020, pp. 523–533. doi: 10.1109/ICSME46990.2020.00056.

PANICHELLA, Annibale, Sebastiano PANICHELLA, Gordon FRASER, Anand Ashok SAWANT und Vincent J. HELLENDOORN, 2020. Revisiting test smells in automatically generated tests : limitations, pitfalls, and opportunities. In: 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME). Conference paper. IEEE. 2020. S. 523–533. ISBN 978-1-7281-5619-4

Panichella, Annibale, Sebastiano Panichella, Gordon Fraser, Anand Ashok Sawant, and Vincent J. Hellendoorn. 2020. “Revisiting Test Smells in Automatically Generated Tests : Limitations, Pitfalls, and Opportunities.” Conference paper. In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), 523–33. IEEE. https://doi.org/10.1109/ICSME46990.2020.00056.

Panichella, Annibale, et al. “Revisiting Test Smells in Automatically Generated Tests : Limitations, Pitfalls, and Opportunities.” 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), IEEE, 2020, pp. 523–33, https://doi.org/10.1109/ICSME46990.2020.00056.