Open Collections

A study of bugs in test code and a test model for analyzing tests

University of British Columbia

Testing has become a wide-spread practice among practitioners. Test cases are written to verify that production code functions as expected and are modified alongside the production code. Over time the quality of the test code can degrade. The test code might contain bugs, or it can accumulate redundant test cases or very similar ones with many redundant parts. The work presented in this dissertation has focused on addressing these issues by characterizing bugs in test code, and proposing a test model to analyze test cases and support test reorganization. To characterize the prevalence and root causes of bugs in the test code, we mine the bug repositories and version control systems of 448 Apache Software Foundation projects. Our results show that around half of all the projects had bugs in their test code; the majority of test bugs are false alarms, i.e., test fails while the production code is correct, while a minority of these bugs result in silent horrors, i.e., test passes while the production code is incorrect; missing and incorrect assertions are the dominant root cause of silent horror bugs; semantic, flaky, environment related bugs are the dominant root cause categories of false alarms. We present a test model for analyzing tests and performing test reorganization tasks in test code. Redundancies increase the maintenance overhead of the test suite and increase the test execution time without increasing the test suite coverage and effectiveness. We propose a technique that uses our test model to reorganize test cases in a way that reduces the redundancy in the test suite. We implement our approach in a tool and evaluate it on four open-source softwares. Our empirical evaluation shows that our approach can reduce the number of redundant test cases up to 85% and the test execution time by up to 2.5% while preserving the test suite’s behaviour.

Text

2016-10-19

Attribution-NonCommercial-NoDerivatives 4.0 International

http://hdl.handle.net/2429/59492

Electrical and Computer Engineering

University of British Columbia

UBCV

http://creativecommons.org/licenses/by-nc-nd/4.0/