A decade of code comment quality assessment : a systematic literature review

Rani, Pooja; Blasi, Arianna; Stulova, Nataliia; Panichella, Sebastiano; Gorla, Alessandra; Nierstrasz, Oscar

doi:10.1016/j.jss.2022.111515

Please use this identifier to cite or link to this item: https://doi.org/10.21256/zhaw-26598

Publication type:	Article in scientific journal
Type of review:	Peer review (publication)
Title:	A decade of code comment quality assessment : a systematic literature review
Authors:	Rani, Pooja Blasi, Arianna Stulova, Nataliia Panichella, Sebastiano Gorla, Alessandra Nierstrasz, Oscar
et. al:	No
DOI:	10.1016/j.jss.2022.111515 10.21256/zhaw-26598
Published in:	Journal of Systems and Software
Volume(Issue):	195
Issue:	111515
Issue Date:	2023
Publisher / Ed. Institution:	Elsevier
ISSN:	0164-1212 1873-1228
Language:	English
Subjects:	Code comment; Documentation quality; Systematic literature review
Subject (DDC):	005: Computer programming, programs and data
Abstract:	Code comments are important artifacts in software systems and play a paramount role in many software engineering (SE) tasks related to maintenance and program comprehension. However, while it is widely accepted that high quality matters in code comments just as it matters in source code, assessing comment quality in practice is still an open problem. First and foremost, there is no unique definition of quality when it comes to evaluating code comments. The few existing studies on this topic rather focus on specific attributes of quality that can be easily quantified and measured. Existing techniques and corresponding tools may also focus on comments bound to a specific programming language, and may only deal with comments with specific scopes and clear goals (e.g., Javadoc comments at the method level, or in-body comments describing TODOs to be addressed). In this paper, we present a Systematic Literature Review (SLR) of the last decade of research in SE to answer the following research questions: (i) What types of comments do researchers focus on when assessing comment quality? (ii) What quality attributes (QAs) do they consider? (iii) Which tools and techniques do they use to assess comment quality?, and (iv) How do they evaluate their studies on comment quality assessment in general? Our evaluation, based on the analysis of 2353 papers and the actual review of 47 relevant ones, shows that (i) most studies and techniques focus on comments in Java code, thus may not be generalizable to other languages, and (ii) the analyzed studies focus on four main QAs of a total of 21 QAs identified in the literature, with a clear predominance of checking consistency between comments and the code. We observe that researchers rely on manual assessment and specific heuristics rather than the automated assessment of the comment quality attributes, with evaluations often involving surveys of students and the authors of the original studies but rarely professional developers.
URI:	https://digitalcollection.zhaw.ch/handle/11475/26598
Fulltext version:	Published version
License (according to publishing contract):	CC BY-NC-ND 4.0: Attribution - Non commercial - No derivatives 4.0 International
Departement:	School of Engineering
Organisational Unit:	Institute of Computer Science (InIT)
Published as part of the ZHAW project:	COSMOS – DevOps for Complex Cyber-physical Systems of Systems
Appears in collections:	Publikationen School of Engineering

Files in This Item:

File	Description	Size	Format
2023_Rani-etal_Code-comment-quality-assessment-literature-review_JSS.pdf		1.87 MB	Adobe PDF	View/Open

Show full item record

Rani, P., Blasi, A., Stulova, N., Panichella, S., Gorla, A., & Nierstrasz, O. (2023). A decade of code comment quality assessment : a systematic literature review. Journal of Systems and Software, 195(111515). https://doi.org/10.1016/j.jss.2022.111515

Rani, P. et al. (2023) ‘A decade of code comment quality assessment : a systematic literature review’, Journal of Systems and Software, 195(111515). Available at: https://doi.org/10.1016/j.jss.2022.111515.

P. Rani, A. Blasi, N. Stulova, S. Panichella, A. Gorla, and O. Nierstrasz, “A decade of code comment quality assessment : a systematic literature review,” Journal of Systems and Software, vol. 195, no. 111515, 2023, doi: 10.1016/j.jss.2022.111515.

RANI, Pooja, Arianna BLASI, Nataliia STULOVA, Sebastiano PANICHELLA, Alessandra GORLA und Oscar NIERSTRASZ, 2023. A decade of code comment quality assessment : a systematic literature review. Journal of Systems and Software. 2023. Bd. 195, Nr. 111515. DOI 10.1016/j.jss.2022.111515

Rani, Pooja, Arianna Blasi, Nataliia Stulova, Sebastiano Panichella, Alessandra Gorla, and Oscar Nierstrasz. 2023. “A Decade of Code Comment Quality Assessment : A Systematic Literature Review.” Journal of Systems and Software 195 (111515). https://doi.org/10.1016/j.jss.2022.111515.

Rani, Pooja, et al. “A Decade of Code Comment Quality Assessment : A Systematic Literature Review.” Journal of Systems and Software, vol. 195, no. 111515, 2023, https://doi.org/10.1016/j.jss.2022.111515.