by Jackson A. Prado Lima, Willian D. F. Mendonça, Silvia R. Vergilio, Wesley K. G. Assunção
Abstract:
Highly-Configurable Software (HCSs) testing is usually costly, as a significant number of variants need to be tested. This becomes more problematic when Continuous Integration (CI) practices are adopted. CI leads the software to be integrated and tested multiple times a day, subject to time constraints (budgets). To address CI challenges, a learning-based test case prioritization approach named COLEMAN has been successfully applied. COLEMAN deals with test case volatility, in which some test cases can be included/removed over the CI cycles. Nevertheless, such an approach does not consider HCS particularities such as, by analogy, the volatility of variants. Given such a context, this work introduces two strategies for applying COLEMAN in the CI of HCS: the Variant Test Set Strategy (VTS) that relies on the test set specific for each variant; and the Whole Test Set Strategy (WST) that prioritizes the test set composed by the union of the test cases of all variants. Both strategies are applied to two real-world HCSs, considering three test budgets. Independently of the time budget, the proposed strategies using COLEMAN have the best performance in comparison with solutions generated randomly and by another learning approach from the literature. Moreover, COLEMAN produces, in more than 92% of the cases, reasonable solutions that are near to the optimal solutions obtained by a deterministic approach. Both strategies spend less than one second to execute. WTS provides better results in the less restrictive budgets, and VTS the opposite. WTS seems to better mitigate the problem of beginning without knowledge, and is more suitable when a new variant to be tested is added.
Reference:
Cost-effective learning-based strategies for test case prioritization in continuous integration of highly-configurable software (Jackson A. Prado Lima, Willian D. F. Mendonça, Silvia R. Vergilio, Wesley K. G. Assunção), In Empirical Software Engineering, Springer Science and Business Media LLC, volume 27, 2022.
Bibtex Entry:
@Article{PradoLima2022,
author = {Jackson A. Prado Lima and Willian D. F. Mendonça and Silvia R. Vergilio and Wesley K. G. Assunção},
journal = {Empirical Software Engineering},
title = {Cost-effective learning-based strategies for test case prioritization in continuous integration of highly-configurable software},
year = {2022},
issn = {1573-7616},
month = jul,
number = {6},
volume = {27},
abstract = {Highly-Configurable Software (HCSs) testing is usually costly, as a significant number of variants need to be tested. This becomes more problematic when Continuous Integration (CI) practices are adopted. CI leads the software to be integrated and tested multiple times a day, subject to time constraints (budgets). To address CI challenges, a learning-based test case prioritization approach named COLEMAN has been successfully applied. COLEMAN deals with test case volatility, in which some test cases can be included/removed over the CI cycles. Nevertheless, such an approach does not consider HCS particularities such as, by analogy, the volatility of variants. Given such a context, this work introduces two strategies for applying COLEMAN in the CI of HCS: the Variant Test Set Strategy (VTS) that relies on the test set specific for each variant; and the Whole Test Set Strategy (WST) that prioritizes the test set composed by the union of the test cases of all variants. Both strategies are applied to two real-world HCSs, considering three test budgets. Independently of the time budget, the proposed strategies using COLEMAN have the best performance in comparison with solutions generated randomly and by another learning approach from the literature. Moreover, COLEMAN produces, in more than 92% of the cases, reasonable solutions that are near to the optimal solutions obtained by a deterministic approach. Both strategies spend less than one second to execute. WTS provides better results in the less restrictive budgets, and VTS the opposite. WTS seems to better mitigate the problem of beginning without knowledge, and is more suitable when a new variant to be tested is added.},
doi = {10.1007/s10664-021-10093-3},
publisher = {Springer Science and Business Media LLC},
url = {https://link.springer.com/article/10.1007/s10664-021-10093-3},
}