Applied Science, Faculty of
Electrical and Computer Engineering, Department of
DSpace
UBCV
Dai, Ke
Master of Applied Science - MASc
University of British Columbia
Technical debt is a figurative expression to describe a phenomenon where software development organizations compromise some quality attributes of their products (e.g., maintainability and evolvability) in order to achieve some business benefits in the software development lifecycle. Technical debt may also be incurred unintentionally due to inconsiderate design decisions, poor programming practices or technological gaps. Different from bugs or defects, technical debt is mostly invisible to customers or end users as the software often works well from their perspective and even developers are often unconscious of the existence of technical debt. The invisibility of technical debt increases the risk of high maintenance and evolution cost in the future. Thus, it is crucial to manage technical debt effectively to assure the health of the software system. The first step of managing technical debt is the identification of technical debt. Many source code analysis tools have been developed to identify code-level technical debt; however, identifying non-code-level technical debt remains understudied and needs deep exploration. This thesis proposes an approach to identifying non-code-level technical debt through issue trackers using natural language processing and machine learning techniques and evaluates the performance of our approach using two issue-tracking data sets in English and Chinese respectively from open source and commercial software projects. We found that there are actually some common English and Chinese words that can be used as indicators of technical debt. We analyzed how these words contribute to indicating technical debt and compared the similarities between English and Chinese key phrases at the semantic level. Based on these key phrases, we developed different classifiers to detect technical debt issues automatically using machine learning techniques, evaluated and compared the performance of these classifiers in identifying technical debt issues recorded in English and Chinese respectively.
Identifying Technical Debt through Issue Trackers by Ke Dai B.Eng., Nanjing Institute of Technology, 2008 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF APPLIED SCIENCE in THE FACULTY OF GRADUATE AND POSTDOCTORAL STUDIES (Electrical and Computer Engineering) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) November 2018 © Ke Dai, 2018 ii The following individuals certify that they have read, and recommend to the Faculty of Graduate and Postdoctoral Studies for acceptance, a thesis/dissertation entitled: Identifying Technical Debt through Issue Trackers submitted by Ke Dai in partial fulfillment of the requirements for the degree of Master of Applied Science in Electrical and Computer Engineering Examining Committee: Dr. Philippe Kruchten, Electrical and Computer Engineering Supervisor Dr. Ali Mesbah, Electrical and Computer Engineering Supervisory Committee Member Dr. Julia Rubin, Electrical and Computer Engineering Supervisory Committee Member Additional Examiner Additional Supervisory Committee Members: Supervisory Committee Member Supervisory Committee Member iii Abstract Technical debt is a figurative expression to describe a phenomenon where software development organizations compromise some quality attributes of their products (e.g., maintainability and evolvability) in order to achieve some business benefits in the software development lifecycle. Technical debt may also be incurred unintentionally due to inconsiderate design decisions, poor programming practices or technological gaps. Different from bugs or defects, technical debt is mostly invisible to customers or end users as the software often works well from their perspective and even developers are often unconscious of the existence of technical debt. The invisibility of technical debt increases the risk of high maintenance and evolution cost in the future. Thus, it is crucial to manage technical debt effectively to assure the health of the software system. The first step of managing technical debt is the identification of technical debt. Many source code analysis tools have been developed to identify code-level technical debt; however, identifying non-code-level technical debt remains understudied and needs deep exploration. This thesis proposes an approach to identifying non-code-level technical debt through issue trackers using natural language processing and machine learning techniques and evaluates the performance of our approach using two issue-tracking data sets in English and Chinese respectively from open source and commercial software projects. We found that there are actually some common English and Chinese words that can be used as indicators of technical debt. We analyzed how these words contribute to indicating technical debt and compared the similarities between English and Chinese key phrases at the semantic level. Based on these key phrases, we developed different classifiers to detect technical debt issues automatically using machine learning techniques, evaluated and compared the performance of these classifiers in identifying technical debt issues recorded in English and Chinese respectively. iv Lay Summary Technical debt is a phrase used to describe a phenomenon where software development organizations achieve some business objectives at the expense of degraded quality attributes (e.g., maintainability and evolvability) of software products. Incurring an appropriate amount of technical debt accelerates the development of software; however, too much of it deteriorates the health of projects and even aborts the projects. Therefore, managing technical debt to assure software projects in a controllable state is critically essential in the process of software development. Different from bugs that are visible to users, technical debt may have a long latency before its negative effects on projects are exposed. Consequently, identifying technical debt items becomes an important step in the management of technical debt. The study presented in this thesis proposes an approach to identifying technical debt issues in English and Chinese through issue trackers to complement state-of-the-art technologies in the identification of technical debt. v Preface The work on manual data tagging presented in Section 4.2 was assisted by Di Zhang, a master student in the School of Computer Science from Wuhan University. The validation of results of data tagging was conducted by Zengyang Li, an associate professor in the School of Computer Science from Central China Normal University. I was responsible for collecting the data, analyzing the data, developing the tool, running the experiments, evaluating the performance of our tool, and writing the manuscript under the supervision of Dr. Philippe Kruchten. Part of work reported in this thesis was published as a full paper in the Proceedings of the 5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ) [1] at the 24th Asia-Pacific Software Engineering Conference (APSEC 2017). vi Table of Contents ABSTRACT .................................................................................................................................. iii LAY SUMMARY ......................................................................................................................... iv PREFACE ...................................................................................................................................... v TABLE OF CONTENTS ............................................................................................................ vi LIST OF TABLES ....................................................................................................................... ix LIST OF FIGURES ...................................................................................................................... x LIST OF ABBREVIATIONS ..................................................................................................... xi ACKNOWLEDGEMENTS ...................................................................................................... xiii DEDICATION............................................................................................................................ xiv CHAPTER 1 INTRODUCTION ................................................................................................ 1 1.1 PROBLEM STATEMENT ......................................................................................................... 2 1.2 RESEARCH OBJECTIVES ....................................................................................................... 2 1.3 CONTRIBUTIONS .................................................................................................................. 3 1.4 THESIS ORGANIZATION ........................................................................................................ 4 CHAPTER 2 BACKGROUND ................................................................................................... 5 2.1 TECHNICAL DEBT METAPHOR ............................................................................................. 5 2.2 CLASSIFICATION OF TECHNICAL DEBT................................................................................. 6 2.2.1 McConnell’s Technical Debt Taxonomy ..................................................................... 6 2.2.2 Fowler’s Technical Debt Quadrant ............................................................................ 8 2.2.3 Kruchten’s Technical Debt Landscape ....................................................................... 9 CHAPTER 3 RELATED WORK............................................................................................. 11 3.1 THE IDENTIFICATION OF TECHNICAL DEBT ........................................................................ 11 3.2 MINING ISSUE TRACKING DATABASES .............................................................................. 12 vii CHAPTER 4 METHODOLOGY ............................................................................................. 14 4.1 DATA COLLECTION ............................................................................................................ 14 4.2 DATA TAGGING ................................................................................................................. 15 4.3 DATA ANALYSIS ................................................................................................................ 18 4.3.1 Qualitative Analysis .................................................................................................. 18 4.3.2 Quantitative Analysis ................................................................................................ 18 CHAPTER 5 CLASSIFIER IMPLEMENTATION ............................................................... 20 5.1 TEXT PREPROCESSING........................................................................................................ 21 5.1.1 Data Cleaning ........................................................................................................... 21 5.1.2 Word Segmentation ................................................................................................... 22 5.1.3 Word Stemming ......................................................................................................... 22 5.1.4 Stop Word Removal................................................................................................... 22 5.2 TEXT REPRESENTATION MODEL ........................................................................................ 23 5.2.1 Boolean Model .......................................................................................................... 23 5.2.2 Vector Space Model .................................................................................................. 24 5.3 FEATURE SELECTION ......................................................................................................... 26 5.3.1 Chi-squared Statistic ................................................................................................. 27 5.3.2 L1 Regularization...................................................................................................... 27 5.4 CLASSIFICATION ALGORITHMS .......................................................................................... 28 5.4.1 Naïve Bayes ............................................................................................................... 28 5.4.2 Support Vector Machine ........................................................................................... 30 5.4.3 Random Forest .......................................................................................................... 31 5.5 PERFORMANCE EVALUATION ............................................................................................. 32 5.5.1 Evaluation Method .................................................................................................... 32 5.5.2 Evaluation Metrics .................................................................................................... 32 CHAPTER 6 FINDINGS .......................................................................................................... 34 6.1 DISTRIBUTION OF TECHNICAL DEBT .................................................................................. 34 6.2 CHARACTERISTICS OF TECHNICAL DEBT ........................................................................... 36 6.2.1 Open Time of Technical Debt ................................................................................... 36 viii 6.2.2 Priority of Technical Debt ........................................................................................ 38 6.3 TEXT PATTERNS OF TECHNICAL DEBT ............................................................................... 39 6.3.1 Indicators of Technical Debt .................................................................................... 40 6.3.2 Features Selected for Identifying Technical Debt .................................................... 48 6.3.3 The Semantic Dimensions of Technical Debt Issue Description .............................. 56 6.4 AUTOMATIC CLASSIFICATION RESULTS ............................................................................. 57 CHAPTER 7 DISCUSSION...................................................................................................... 61 7.1 ANSWERING RESEARCH QUESTIONS .................................................................................. 61 7.2 THREATS TO VALIDITY ...................................................................................................... 64 7.2.1 Internal Validity ........................................................................................................ 65 7.2.2 External Validity ....................................................................................................... 65 CHAPTER 8 CONCLUSION ................................................................................................... 67 BIBLIOGRAPHY ....................................................................................................................... 68 APPENDICES ............................................................................................................................. 74 ix List of Tables TABLE 4.1 THE SUMMARY OF CASES ............................................................................................. 15 TABLE 4.2 THE CRITERIA OF ISSUE CLASSIFICATION ..................................................................... 17 TABLE 5.1 FOUR ELEMENTS OF CLASSIFICATION RESULT ............................................................. 32 TABLE 6.1 THE NUMBER OF FEATURES OF EACH PART OF SPEECH................................................ 52 TABLE 6.2 THE FEATURES FOR ECLIPSE CHE ................................................................................. 53 TABLE 6.3 THE FEATURES FOR CORNERSTONE .............................................................................. 54 TABLE 6.4 EXAMPLE WORDS EXPRESSING SIMILAR SEMANTIC MEANING .................................... 55 TABLE 6.5 THE PERFORMANCE OF SIX CLASSIFIERS ...................................................................... 58 x List of Figures FIGURE 2.1 MCCONNELL’S TECHNICAL DEBT TAXONOMY ............................................................. 6 FIGURE 2.2 FOWLER’S TECHNICAL DEBT QUADRANT ..................................................................... 8 FIGURE 2.3 KRUCHTEN’S TECHNICAL DEBT LANDSCAPE ................................................................ 9 FIGURE 4.1 THE PROCEDURE OF DATA TAGGING........................................................................... 16 FIGURE 5.1 THE PROCEDURE OF CLASSIFIER IMPLEMENTATION .................................................... 20 FIGURE 5.2 THE EXAMPLE OF FORMAT CONTENT .......................................................................... 21 FIGURE 5.3 THE PROCESS OF BUILDING A BOOLEAN MODEL......................................................... 24 FIGURE 5.4 THE PROCESS OF BUILDING A VSM ............................................................................ 25 FIGURE 6.1 THE PROPORTION OF TECHNICAL DEBT IN TWO ISSUE DATA SETS ............................. 35 FIGURE 6.2 THE DISTRIBUTION OF TECHNICAL DEBT IN TWO ISSUE DATA SETS ........................... 36 FIGURE 6.3 OPEN TIME OF DIFFERENT TYPES OF TECHNICAL DEBT AND BUGS ............................. 37 FIGURE 6.4 THE STATISTICS OF OPEN TIME OF DIFFERENT TYPES OF TECHNICAL DEBT AND BUGS .................................................................................................................................... 38 FIGURE 6.5 THE DISTRIBUTION OF PRIORITIES OF DIFFERENT TYPES OF TECHNICAL DEBT AND DEFECTS ..................................................................................................................... 39 FIGURE 6.6 THE F1-SCORE OF THE SVM CLASSIFIER AGAINST THE NUMBER OF FEATURES USING CHI-SQUARED STATISTIC TECHNIQUE TO SELECT FEATURES ...................................... 49 FIGURE 6.7 THE F1-SCORE OF THE SVM CLASSIFIER AGAINST THE PENALTY PARAMETER C USING L1 REGULARIZATION TECHNIQUE TO SELECT FEATURES............................................ 50 FIGURE 6.8 THE F1-SCORE OF THE SVM CLASSIFIER AGAINST THE PENALTY PARAMETER C USING L1 REGULARIZATION TECHNIQUE TO SELECT FEATURES INCLUDING BIGRAMS AND TRIGRAMS ................................................................................................................... 51 FIGURE 6.9 THE ROC OF SIX CLASSIFIERS FOR ECLIPSE CHE ........................................................ 59 FIGURE 6.10 THE ROC OF SIX CLASSIFIERS FOR CORNERSTONE ................................................... 60 xi List of Abbreviations API Application Programming Interface APSEC Asia-Pacific Software Engineering Conference ATFD Access to Foreign Data AUC Area under the Curve BNB Bernoulli Naive Bayes BoW Bag-of-Words DF Document Frequency FN False Negatives FP False Positives FPR False Positive Rate ICPI Package Changing Impact IDE Integrated Development Environment IG Information Gain IPGF Index of Package Goal Focus IQR Interquartile Range IR Information Retrieval LSTM Long Short-Term Memory MI Mutual Information MLE Maximum Likelihood Estimation MNB Multinomial Naive Bayes NB Naïve Bayes NDA Non-disclosure Agreement NLP Natural Language Processing QuASoQ Quantitative Approaches to Software Quality RF Random Forest ROC Receiver Operating Characteristic SD Standard Deviation SVM Support Vector Machine xii TCC Tight Class Cohesion TFIDF Term Frequency-Inverse Document Frequency TN True Negatives TP True Positives TPR True Positive Rate UI User Interface VSM Vector Space Model WMC Weighted Method Count xiii Acknowledgements First and foremost, I truly express my heartfelt gratitude to my supervisor Dr. Philippe Kruchten for giving me this opportunity to work with him. His expertise, wisdom, and experience benefitted me considerably in both research work and thesis writing. I am grateful for his continuous support, guidance, and encouragement throughout this journey. I would also like to thank Di Zhang who assisted me with manual data tagging and Zengyang Li who spent his precious time in validating the results of data tagging and reviewing my thesis. Without their contribution, I might not have been able to complete this thesis. In addition, I appreciate that Mitacs provided me with the funding that supports me to finish this program and an opportunity to cooperate with a company to acquire valuable data. Special thanks are owed to my parents, who have always unconditionally supported me to pursue my dreams. xiv Dedication To my beloved parents, It is your unconditional love, support and giving that make me better and better. 1 Chapter 1 Introduction In software-intensive systems, technical debt is a collection of design or implementation constructs that are expedient in the short term, but set up a technical context that can make future changes more costly or impossible. Technical debt presents an actual or contingent liability whose impact is limited to internal system qualities, primarily maintainability and evolvability [2]. An appropriate amount of technical debt would accelerate the process of software development; however, too much of it would impede the progress and even abort the project [3]. Typically, some startup software companies tend to incur technical debt strategically to speed up the development at the early stage of development process for the purpose of entering the market earlier. But with the growth of the size and complexity of the software, it may become increasingly more difficult to maintain and evolve the product due to intertwined dependencies between modules or components without handling technical debt appropriately. Consequently, it is extremely significant to pay off technical debt regularly to keep the whole software system in a maintainable and evolvable state. In general, technical debt is differentiated from externally visible bugs or defects as Philippe Kruchten argued; technical debt deteriorates internal quality of the software and is invisible from users’ perspective [4]. The invisibility of technical debt underlies high likelihood of quick accumulation and long latency of technical debt and thus may lead to the risks of huge extra maintenance and evolution cost in the future. As a result, it is essentially critical for software development teams to be able to identify technical debt items existing in the current software system at any point in time so that they can conduct other management activities of technical debt such as measurement of technical debt and payment of technical debt later. Once technical debt can be identified systematically and timely, software development teams would be able to estimate future budget, prioritize future tasks, allocate limited resources and evaluate potential risks. They could also make informed decisions about when technical debt should be paid off to maximize their profits. 2 1.1 Problem Statement Technical debt may take many different forms and affect the system in very different ways when we observe technical debt at the software level [4]. Some forms of technical debt are fine granularity issues at the code level such as duplicated code, large class, bad naming, and misleading comments. These issues are also referred to as code smells. Many source code analysis techniques have been developed to detect this type of technical debt. There are also many other forms of technical debt that have larger granularity and cannot be detected directly through source code. For instance, design decisions at the architectural level (e.g., choice of framework, middleware, or technologies, etc.) made previously prove inconsiderate or thoughtless. Another possible example is that software development teams had to adopt the suboptimal but easily implemented approach to implement some functions due to time constraint. Herein we call these forms of technical debt non-code-level technical debt. The key difference between code-level and non-code-level technical debt is that code-level technical debt is more concrete, tangible and can be easily identified and manipulated by source code analyzers. However, not only is it impossible to detect non-code-level technical debt with existing tools, but also the cost of paying off the debt is much higher due to the deeply intertwined dependency network. The status quo of lack of mature and effective approaches to identifying non-code-level technical debt calls for more related studies to explore sophisticated techniques that contribute to tackling this problem. 1.2 Research Objectives We aim to explore a new approach to identifying technical debt as a complement to existing approaches. Specifically, we look into issue trackers to investigate whether we can identify technical debt through issue trackers since issue trackers are widely used to record, track, and prioritize various kinds of issues in software projects. 3 Furthermore, in view of the substantial effort of identifying technical debt through issue trackers manually, we attempt to automate the process using natural language processing (NLP) and machine learning techniques. To the best of our knowledge, our study is the first one that applies NLP and machine learning techniques to detect technical debt through issue trackers. The exploratory study we conducted is oriented towards answering the following questions: • (RQ1) How are different kinds of technical debt issues distributed? • (RQ2) Do technical debt issues share some distinguishing characteristics compared to bugs or defects? ▪ (RQ2.1) Do technical debt issues take longer time to resolve than defects? ▪ (RQ2.2) Do technical debt issues have lower priority than defects? • (RQ3) How do software practitioners communicate technical debt issues in issue trackers? ▪ (RQ3.1) Do software practitioners use technical debt as an issue type? ▪ (RQ3.2) Do software practitioners communicate technical debt explicitly or implicitly? ▪ (RQ3.3) Are there text patterns that may indicate technical debt items? • (RQ4) Can we really identify non-code-level technical debt through issue trackers? • (RQ5) Is it possible to apply NLP and machine learning techniques to identifying technical debt items through issue trackers? 1.3 Contributions Our study makes the following main contributions: • Conduct an empirical study on 2 issue-tracking data sets of software projects and propose a set of criteria for identifying technical debt through issue trackers manually. • Explore potential text patterns of technical debt issues in issue trackers and propose English and Chinese vocabulary that may indicate technical debt. • Develop an approach to identifying technical debt through issue trackers using NLP and machine learning techniques. 4 1.4 Thesis Organization The thesis is organized as follows: Chapter 2 provides background knowledge of technical debt Chapter 3 discusses the related work in this research area. Chapter 4 describes the methodology of this study in detail. Chapter 5 elaborates the procedure of developing our classifier step by step. Chapter 6 presents the results of data analysis and evaluates the performance of our classifier. Chapter 7 answers the research questions explicitly and discusses the threats to validity in our study. Chapter 8 concludes our research and presents future work. 5 Chapter 2 Background 2.1 Technical Debt Metaphor The term technical debt was coined by Ward Cunningham to explain to business stakeholders why refactoring happens during the process of software development [3]. Ward Cunningham made an analogy between writing immature code and going into financial debt to introduce this concept. As we know, financial debt provides leverage for a person to do the things which he cannot do within his current financial capacity. Typically, a person could establish a company with the loan provided by a bank or buy a big house now rather than waiting until she/he can afford it. Similarly, software development organizations could achieve some economic goals or benefits by compromising some dimensions of quality attributes (e.g., maintainability, extensibility, etc.) in the software development lifecycle. The compromised quality attributes are just like the debt a person owes the bank. For instance, some startup software companies shorten the time for their first release of products from half a year to 4 months by adopting shortcuts or expedient approaches to implement some features of their products in order to enter the market earlier and collect the feedback from users to verify their product design idea with as low cost as possible. However, risk accompanies gains. The borrower needs to handle the loan very cautiously otherwise she/he has to take the consequences of going bankrupt when the interest accumulates to the limit where she/he cannot afford to repay the loan. Likewise, technical debt can bring about high maintenance and evolution cost and even project abortion if not handled wisely although it enables software development organizations to gain some economic benefits and market opportunities. By analogy with financial debt, the principal for technical debt is the effort expended to eliminate the debt when it is incurred while the interest is the additional effort expended in the process of development if it remains in the system. The longer a technical debt item stays in the system, the more expensive it is to pay off the technical debt—in other words, it is more difficult to remove it from the system because many modules added later build on the previously developed artifact containing the technical debt item. As technical debt accrues, the maintainability and evolvability of the software decrease significantly and the development 6 productivity declines dramatically. Eventually, the software project goes into technical bankruptcy when the software development team finds it impossible to add a new feature as the code base has become too rigid to extend the system and they have to abandon the old code base and create a brand-new system from scratch if too much technical debt is taken on without being paid off regularly. 2.2 Classification of Technical Debt Since Ward Cunningham introduced the concept of technical debt, many scholars and software practitioners have shown great interest in this new terminology and developed the definition, classification, scope, and management strategy of technical debt. Among them, Steve McConnell and Martin Fowler divided technical debt into different types based on the motivation of incurring technical debt while Philippe Kruchten classified technical debt items according to their granularity. 2.2.1 McConnell’s Technical Debt Taxonomy From McConnell’s perspective, technical debt is a design or construction approach that’s expedient in the short term but that increases complexity and is costlier in the long term [5]. McConnell created a taxonomy to divide technical debt into different types shown in Figure 2.1: Figure 2.1 McConnell’s Technical Debt Taxonomy [5] Technical DebtUnintentional DebtIntentional DebtShort-Term DebtFocused Short-Term DebtUnfocused Short-Term DebtLong-Term Debt7 McConnell classified technical debt into two basic kinds: unintentional debt and intentional debt. Unintentional debt is the non-strategic debt incurred unintentionally for various kinds of reasons. For example, an inexperienced junior developer writes low quality code, or a software system integrates legacy code base which contains significant technical debt from another company. Intentional debt is the debt incurred intentionally to make a tradeoff between software quality and business objectives. Typically, a software development team implements some functions of the product using suboptimal approaches in order to deliver a release of the product on time. Intentional debt consists of two forms of technical debt: short-term and long-term debt. Short-term debt is the debt that is taken on tactically and reactively and will be paid off quickly, perhaps at the beginning of the next release cycle while long-term debt is the debt that is taken on strategically and proactively and can be carried for a few years or longer. For example, low code coverage of test suite belongs to short-term debt. However, provided that a software company delivers its product which is only compatible with Windows and has no intention of developing a Mac version in the near future, this type of technical debt is classified as long-term debt. Based on different granularity of short-term debt, McConnell further divided short-term debt into focused short-term debt and unfocused short-term debt. Focused short-term debt is a large granularity of debt which can be easily tracked and managed, such as an easily implemented but less efficient algorithm employed to implement a certain function. However, unfocused short-term debt is some small shortcuts which are difficult to track and manage. For example, some classes are not named in compliance with naming convention or not accompanied by appropriate comments. This kind of short-term debt should be avoided because it accumulates more quickly than what development teams may think and does not bring about any benefits. According to McConnell, focused short-term debt is like a car loan while unfocused short-term debt is comparable to credit card debt. McConnell also pointed out that some types of incomplete work such as backlogged, deferred or cut features are not technical debt because they do not require interest payments. 8 2.2.2 Fowler’s Technical Debt Quadrant Against the argument that messy code produced by people who are ignorant of good design practices should not be a debt, Fowler deemed that a mess is a reckless debt which is difficult to pay off [6]. However, with regard to the cases when people adopt a suboptimal design strategy whether it is a conscious behavior or not, Fowler call them prudent debt. For each of these two kinds of technical debt, Fowler divided them into two categories in another dimension based on whether it is incurred intentionally or not: deliberate and inadvertent debt. Thus, there are four types of technical debt as shown in Figure 2.2. Figure 2.2 Fowler’s Technical Debt Quadrant [6] Reckless deliberate debt is taken on when developers have the knowledge of good design practices and are capable of executing them but decide to go the “quick and dirty” way because Reckless"We don't have time for design"DeliberatePrudent"We must ship now and deal with consequences"Inadvertent"What's layering" "Now we know how we should have done it"9 they do not have time to write good and clean code. On the contrary, Reckless inadvertent debt arises when inexperienced junior developers do not know about good design practices and write low quality code unconsciously. Prudent deliberate debt is the outcome of tradeoff between software quality and business objectives. For example, a software development team has the idea what is the optimal solution clearly but adopts the suboptimal but easily implemented design strategy to save the time or cost. Prudent inadvertent debt occurs when a design strategy which was thought the optimal solution when it is adopted proves imperfect in retrospect after a period of time. 2.2.3 Kruchten’s Technical Debt Landscape Philippe Kruchten defined the scope of technical debt by drawing the boundary between technical debt and defect or unimplemented requirement based on whether it is visible to customers or users as Figure 2.3 shows [4]. Kruchten argued that technical debt should exclude low external quality and unimplemented functions, otherwise it makes technical debt too vast a category and thus the metaphor loses its original strength. He pointed out that technical debt is mostly invisible outside the software system’s development organization and can only be perceived by software developers when they find it difficult to maintain or evolve the system. Figure 2.3 Kruchten’s Technical Debt Landscape [4] Architecture Code Architectural Debt Low Internal Quality Structural Debt Code Complexity Code Smells Test Debt Coding Style Violations Documentation Debt Technological Gaps Visible Visible Mostly Invisible Evolution Issues: Evolvability Quality Issues: Maintainability New Features Additional Functionality Defects Low External Quality 10 Kruchten classified technical debt items according to their granularity. Source code is the final embodiment of software design and programming decisions and also where the dependencies between different modules of the system are manifested. Many technical debt items of fine granularity can be found in the source code such as the violation of modularity, unnecessary code complexity, badly named variables, and misleading or unclear comments. Incurring technical debt at the code level tends to degrade the maintainability of a system, rendering it difficult to make corrections to the system when needed. Fortunately, many source code analyzers can be used to identify this kind of technical debt instead of inspecting the code manually. Code-level technical debt is just the tip of iceberg; there are many other forms of technical debt that have a larger granularity. They relate to design decisions about software architecture including the choice of technologies, platforms, languages, design principles or patterns, etc. Kruchten named them architectural debt. When architectural debt is incurred in a system, the debt tends to deteriorate the evolvability of the system: it will be difficult to add new features to the system. Kruchten further elaborated that not all technical debt is associated with poor internal quality. Technical debt may not be incurred intentionally nor the result of having made a wrong decision originally, but arise from the evolution of context. A design decision may have been the optimal approach to implementing some features when the system was initially built; however, several years later, the decision was not quite right in retrospect. Technical debt in this case originates from external events: technological obsolescence, environmental change, the emergence of new technologies, etc. These reasons for incurring technical debt are summarized as technological gaps by Kruchten. Kruchten emphasized that both architectural debt and the debt incurred due to technological gaps are undetectable by source code analysis tools and appealed for paying more attention to these parts of technical debt as the principal and interest for them are often much higher than those for code-level technical debt. 11 Chapter 3 Related Work This chapter is organized in two sections. The first section discusses previous work on the identification of technical debt. The second section describes the work related to mining issue tracking databases. 3.1 The Identification of Technical Debt Many researches have been performed to identify code-level technical debt including code smells (e.g., God Class) [7] and bad programming practices (e.g., no comments) with the aid of static source code analysis techniques. Marinescu proposed an approach to detecting God Class using the combination of three metrics, Weighted Method Count (WMC), Tight Class Cohesion (TCC), and Access to Foreign Data (ATFD) that quantify class complexity, class cohesion and access of foreign data respectively [8]. Munro developed a prototype tool that can identify ten kinds of code smells in Java source code by introducing the justification for the selection of metrics on the basis of Marinescu’s work and validated the effectiveness of this approach on the identification of two types of code smells (Lazy Class and Temporary Field) in two case studies [9]. Olbrich et al. investigated how change frequency and change size of software components are affected by two kinds of code smells (God Class and Shotgun Surgery) during several years of development of two large scale open source software projects [10]. Wong et al. explored an approach to detecting modularity violations using the tool called CLIO they developed and evaluated this approach using the version histories of Hadoop Common and Eclipse JDT [11]. Maldonado et al. aimed to identify technical debt through source code comments using natural language processing techniques [12]. Other researches aimed at exploring approaches to identifying other types of technical debt such as architecture technical debt. Brondum et al. proposed a modelling approach to visualizing architecture technical debt based on the analysis of structural code [13]. Li et al. proposed to use two modularity metrics, Index of Package Changing Impact (ICPI) and Index of Package Goal Focus (IPGF), as indicators of architecture technical debt [14]. Further, they proposed an 12 architecture technical debt identification approach based on architecture decisions and change scenarios [15]. The work closest to ours is the work by Bellomo et al., where manual examination was conducted on 1,264 issues in four issue trackers from open source and government projects and 109 examples of technical debt were identified using a categorization method they developed [16]. The major difference is that we partially automated the process of identification while they identified technical debt items manually. To our knowledge, our study is the first one that applies NLP and machine learning techniques to detecting technical debt through issue trackers. 3.2 Mining Issue Tracking Databases Issue tracking systems are widely used in open source projects as well as in software industry to record, triage and track different kinds of issues occurred during the lifecycle of software: bugs finding, defects fixing, adding new features, future tasks, requirements updating, etc. They play an important role in facilitating software development teams to manage development and maintenance activities and thus promoting the success of software projects. Some researches have focused on mining issue tracking databases to retrieve valuable information for improved definition, development management, quality evaluation, predictive models, etc. Antoniol et al. applied NLP and machine learning techniques (alternating decision trees, naïve Bayes classifier, and logistic regression) to automate the process of distinguishing bugs from other kinds of issues, compared the performance of this approach with that of using regular expression matching and concluded machine learning techniques outperform regular expression matching in terms of predictive accuracy [17]. Runeson et al. developed a prototype tool which detects duplicate defect reports in issue tracking systems using NLP techniques, evaluated the identification capabilities of this approach in a case study and concluded that about 2/3 of the duplicates can possibly be found using this approach [18]. Wang et al., Jalbert and Weimer, Sun et al., and Sureka and Jalote performed similar researches to address the same problem [19, 20, 21, 22]. 13 Other work focused on concerned aspects of software quality attributes, say security. Cois et al. proposed an approach to detecting security-related text snippets in issue tracking systems using NLP and machine learning techniques [23]. 14 Chapter 4 Methodology This study is an empirical multiple-case study aiming to investigate how technical debt is communicated in issue trackers and also to explore the feasibility and effectiveness of identifying technical debt through issue trackers automatically, especially non-code-level technical debt. To achieve our research objectives, we collected the issue-tracking data sets through multiple channels, executed issue tagging manually and conducted both qualitative and quantitative analysis on these data. This chapter describes the details of data collection, tagging, and analysis in the following sections. 4.1 Data Collection Data were collected for this study through two approaches. One approach is searching the data on GitHub; the other approach is acquiring the data from an IT company collaborating with us. To ensure the quality and representativeness of data sets we examined, we made a deliberate selection based on the following criteria: • The software projects have been in development for at least two years so that the data sets can include as many technical debt issues of different types as possible. • Most of the issues in the data sets are recorded with detailed description that enables us to fulfill accurate issue tagging and meaningful data analysis. • The issue-tracking data sets have the fields like open time or issue priority we are interested in besides issue title, issue type, issue description. Two issue-tracking data sets were obtained after careful screening. One is from the well-known open source software project Eclipse Che; the other is from a commercial software project called Cornerstone developed by the collaborative IT company. Eclipse Che is a developer workspace server and cloud IDE built for teams and organizations. Cornerstone is a unified security management platform designed for industry control and network security products. Both projects had been in development for more than two years and their respective issue-tracking data sets contain more than 8000 samples. The issues of Eclipse Che were recorded in English while those 15 of Cornerstone were mainly recorded in Chinese. Table 4.1 shows the details of both issue-tracking data sets. Table 4.1 The Summary of Cases Project Name Project Type Project Domain # of Samples Language Eclipse Che Open Source IDE 8678 English Cornerstone Commercial Cyber Security 8149 Chinese For Eclipse Che (https://github.com/eclipse/che), we employed GitHub API for Java (http://github-api.kohsuke.org/) to export all closed issues. There are 8678 closed issues in total for this project at the time we performed this operation. For Cornerstone, we gained the permission to access and download the issues stored in YouTrack after we signed the NDA with the company. The screenshots of spreadsheets of both issue-tracking data sets are presented in Appendix A and Appendix B respectively. 4.2 Data Tagging Data tagging was conducted manually by reviewing the title and description of each issue to determine whether a specific issue is an instance of technical debt. The labels include only two types: technical debt and not technical debt. We drew up the initial criteria of issue classification shown in Table 4.2 based on the taxonomy and definition of technical debt types presented by Alves et al. [24] to help us make sound, and consistent decisions when tagging the issues. Besides, we also hid the predefined type field of the data sets in order to prevent developers’ classification of issues from affecting our judgments when we reviewed and tagged the issues. To minimize our subjective bias in terms of the understanding of issues, we had two researchers tag issues independently conforming to the classification criteria of issues. Each of them was asked not only to label an issue as technical debt or not technical debt, but also to specify which kind of technical debt it belongs to and highlight the sentences or phrases based on which she/he made the decision if the issue is an instance of technical debt. 16 We had a periodical discussion about the conflicting labels of issues we had tagged during the process of tagging. For the controversial issues, each of the two researchers was asked to explain why she/he tagged them as technical debt or not technical debt. By discussion, one researcher might change his initial opinion and agree with the other researcher. In this case, the issues were tagged with the labels that make more sense. If we still did not resolve the conflicting labels, we asked an expert in the field of technical debt to make a decision on the labels of the controversial issues. Meanwhile, we also refined our tagging rules based on the discussion and adopted the refined tagging rules to tag the remaining issues later. After we finished data tagging, we extracted a random sample of 200 issues respectively for both data sets and asked the expert to validate our classification results. The whole procedure is shown in Figure 4.1. Figure 4.1 The Procedure of Data Tagging 17 Table 4.2 The Criteria of Issue Classification Label Subtype Definition Not Technical Debt Requirement Change refers to the activities of requirements change management including receiving change requests, recording received change requests, implementing change requests, etc. New Features refers to the tasks of introducing new features or implementing new functions in the current version of software product. Critical Defects refers to defects that may cause fatal errors occurring when using the software or exerts a severely negative impact on user experience and therefore must be fixed immediately. Insufficient Description refers to the cases where the description of issues is insufficient for researchers to decide whether the issues are instances of technical debt. Technical Debt Design Debt refers to the violation of good object-oriented design principles (e.g., very large or tightly coupled classes) and inconsiderate design decisions. Code Debt refers to the problems found in the source code that can negatively affect the legibility of the code making it more difficult to maintain. Architecture Debt refers to the problems encountered in product architecture which can affect architectural requirements (e.g., extensibility, compatibility, stability, reliability, etc.). Defect Debt refers to known defects that should be fixed but have to be deferred to a later time due to competing priorities and limited resources. Requirement Debt refers to tradeoffs made with respect to what requirements the development team needs to implement or how to implement them. Usability Debt refers to inappropriate usability decisions that will need to be adjusted later. Test Debt refers to issues found in testing activities that can affect the quality of those activities. 18 4.3 Data Analysis The data collected for this study were analyzed both qualitatively and quantitatively to find sufficient and convincing evidence for research questions we were going to answer and finally to draw our conclusion. 4.3.1 Qualitative Analysis Data analysis started from qualitative analysis by reviewing the two issue-tracking data sets to gain a perceptual cognition about how technical debt is communicated in issue trackers since we did not have prior knowledge of it. By means of qualitative analysis, we could acquire the initial perception of expression mode of technical debt in issue trackers, such as whether technical debt is communicated explicitly and whether technical debt is used as an issue type. Meanwhile, the sentences or phrases providing the evidence that the issues are technical debt instances were extracted as the indicators of technical debt from the description of issues. These indicators of technical debt were further classified according to technical debt types to learn how each type of technical debt is communicated. Furthermore, qualitative analysis was also used to derive text patterns of technical debt issues from the indicators of technical debt. We observed the indicators from multiple perspectives including parts of speech of words, the meaning of words, and even the higher abstract level and also compared the phrases from the two issue-tracking data sets at different levels. 4.3.2 Quantitative Analysis Quantitative analysis was conducted to acquire deeper and more comprehensive knowledge about technical debt in these two issue-tracking data sets including its proportion, distribution, and characteristics. The corresponding statistics for each issue-tracking data set were computed and visualized with various charts to answer the research questions with quantified and intuitive evidence. 19 Besides, quantitative analysis was also applied to the automatic identification of technical debt described in the next chapter. Different feature selection techniques are compared in terms of F1-score achieved training the classifier on the selected features and the most informative features indicating technical debt are obtained based on the adopted feature selection method. The performance of our classifiers is evaluated with three widely-used metrics in binary classification (i.e., precision, recall, and f1-score) as well as the ROC plot. 20 Chapter 5 Classifier Implementation After we finished tagging issues, we could develop a classifier to identify technical debt automatically using NLP and supervised learning methods. The procedure of classifier implementation is shown in Figure 5.1. The details of each step in the procedure are presented in the following sections. Figure 5.1 The Procedure of Classifier Implementation 21 5.1 Text Preprocessing Before we classified the issues using machine learning methods, we had to conduct text preprocessing including data cleaning, word segmentation, word stemming, stop word removal, etc. 5.1.1 Data Cleaning Data cleaning, as the first step of text preprocessing, aims to remove the unused part of the texts such as format content, email address, and URL, which may degrade the performance of feature selection and issue classification. We found that much format content is extensively distributed throughout the description of issues in Eclipse Che data set when viewing the issues in the phase of manual tagging. These parts of texts are not only useless for issue classification, but also may be noise for automatic classification. Therefore, they must be removed from the raw data set before automatic classification. Figure 5.2 demonstrates what the format content is like. The texts in bold like “### What does this PR do?” are what we call format content. We removed the format content by matching it with regular expressions. Figure 5.2 The Example of Format Content 22 5.1.2 Word Segmentation Word segmentation is the process of dividing a string of written language into its component words [25]. It is well known that Chinese, as an ideographic language, does not delimit words with space like English. As there is no explicit word boundary between words, word segmentation is the very first step for Chinese text processing, and its accuracy has a significant and direct bearing on the performance of subsequent steps including keyword extraction, feature selection, text classification, etc. Since the issues for Cornerstone are recorded in Chinese, word segmentation is an indispensable step in the process of text preprocessing for Cornerstone data set. In this study, we did not build a word segmenter from scratch, but employed a popular open source Chinese text segmentation tool called Jieba (https://github.com/fxsjy/jieba) [26] to implement word segmentation. 5.1.3 Word Stemming Word stemming is the process of reducing inflected (or sometimes derived) words to their word stem, base or root form—generally a written word form [27]. English texts use different forms of a word for grammatical reasons like the verb “fix” appears in the form of “fixes”, “fixed”, or “fixing”. Obviously, the various forms of a word should be converted into the original form rather than being taken as different words in text processing, such as keyword extraction, feature selection and text classification, where word occurrences or frequency counting is needed. This is what word stemming deals with. In this study, word stemming was conducted only for Eclipse Che data set as it is not applicable for Chinese texts. 5.1.4 Stop Word Removal Stop words refer to the most common words in a language that are useless in NLP tasks like information retrieval (IR) and text classification [28]. These words not only increase the 23 dimensionality of feature space, and thus cause the explosion of computational complexity, but also may degrade the IR or classification accuracy. For example, articles (i.e., a, an, and the), pronouns (e.g., I, he, and she), prepositions (e.g., at, in, on, and with) are typical stop words in English. Therefore, we need to remove stop words from the texts before conducting text classification. The list of stop words can be created either by linguists based on their subjective selection of words or automatically based on probabilistic statistics of words. There are many versions of English stop word lists proposed by different scholars such as Van Rijsbergen [29] and Christopher Fox [30]. In this study, we used the English stop word list adopted by Scikit-learn (an open source machine learning library for python) [31] and the Chinese stop word list adopted by Baidu (a well-known Chinese web search engine). 5.2 Text Representation Model To apply machine learning algorithms in text classification, we must extract the features representing text content from the texts and convert the texts to structural data. In this study, we adopted the well-known natural language model called Bag-of-Words (BoW) model [32] to extract text features. In this model, a text can be taken as a bag of words, disregarding grammar and even word order. Thus, the feature space of the corpus is the vocabulary in the corpus and the dimensionality of the feature space is the size of the vocabulary. According to the possible values of each feature, BoW model can be classified into Boolean Model and Vector Space Model which are elaborated in the following two subsections. 5.2.1 Boolean Model Boolean model is a classical IR model based on Boolean logic and set theory. In a Boolean model, each feature has only two possible values: 1 and 0. If a term occurs in a text, the value of the feature represented by the term in this text is 1; otherwise the value of the feature is 0. Therefore, each document in a corpus can be converted to a Boolean vector. Suppose a corpus 𝔻 includes 𝑚 documents, then the corpus can be represented as: 𝔻 = {𝐷1, 𝐷2, 𝐷3, ⋯ , 𝐷𝑚}, (5.1) 24 where 𝐷𝑖 , 𝑖 ∈ {1,2,3, ⋯ , 𝑚} is a document in the corpus 𝔻 and the vocabulary in the corpus can be represented as: 𝕋 = {𝑇1, 𝑇2, 𝑇3, ⋯ , 𝑇𝑛}, (5.2) where 𝑇𝑗 , 𝑗 ∈ {1,2,3, ⋯ , 𝑛} represents a term in the vocabulary. A term either appears or is absent in a document; that is, the value of the corresponding feature in the document is either 1 or 0. Figure 5.3 shows the process of building a Boolean model. Figure 5.3 The Process of Building a Boolean Model 5.2.2 Vector Space Model Vector Space Model (VSM) is a widely-used NLP model proposed by Gerard Salton, Andrew Wong and Chungshu Yang in 1975 [33]. Similar to Boolean Model, VSM converts a text to a feature vector. Different from Boolean Model, the value of a feature in VSM is the weight of a term in a document, rather than representing the presence or absence of the term. The term weight is different for each document as the distribution of terms varies from document to document. For a document 𝐷𝑖 in the corpus 𝔻 mentioned before, its feature vector can be represented as: 𝕎𝐷𝑖 = {𝑊𝑖1, 𝑊𝑖2, 𝑊𝑖3, ⋯ , 𝑊𝑖𝑛}, (5.3) where 𝑊𝑖𝑗 is the weight of the term 𝑇𝑗 in the document 𝐷𝑖. Basically, Boolean model is a special form of VSM. Figure 5.4 shows the process of building a VSM. Corpus Feature 𝑻𝟏 Feature 𝑻𝟐 Feature 𝑻𝟑 Feature 𝑻𝒏 … 𝑻𝟏 𝑻𝟐 𝑻𝟑 … 𝑻𝒏 𝑫𝟏 𝟎 𝟏 𝟎 … 𝟎 𝑫𝟐 𝟎 𝟎 𝟏 … 𝟏 𝑫𝟑 𝟏 𝟎 𝟏 … 𝟏 … … 𝑫𝒎 𝟏 𝟏 𝟎 … 𝟎 25 Figure 5.4 The Process of Building a VSM There are various forms of term weight such as term frequency and term count. Term Frequency-Inverse Document Frequency (TFIDF) is most widely used as term weight. TFIDF is a numeric statistic intended for evaluating how important a word is to a document in a collection or corpus [34]. The importance of a word grows proportionally with the number of its occurrences in a document, but is inversely proportional to its frequency in the corpus. Intuitively speaking, a word or phrase is more representative to a document if it appears more frequently in this document, but more rarely in other documents. This is the basic idea behind TFIDF. TFIDF is defined as the product of Term Frequency (TF) and Inverse Document Frequency (IDF) mathematically. TF refers to the number of times 𝐶𝑜𝑢𝑛𝑡𝑖𝑗 a given term 𝑖 appears in a document 𝑗, normalized by dividing it by the total number of words 𝑊𝑗 in the document 𝑗 to remove the effect of length of the document on the TF value, as shown in Equation (5.4). 𝑇𝐹𝑖𝑗 =𝐶𝑜𝑢𝑛𝑡𝑖𝑗𝑊𝑗 (5.4) IDF for the given term 𝑖 can be computed as: Corpus Feature 𝑻𝟏 Feature 𝑻𝟐 Feature 𝑻𝟑 Feature 𝑻𝒏 … 𝑻𝟏 𝑻𝟐 𝑻𝟑 … 𝑻𝒏 𝑫𝟏 𝑾𝟏𝟏 𝑾𝟏𝟐 𝑾𝟏𝟑 … 𝑾𝟏𝒏 𝑫𝟐 𝑾𝟐𝟏 𝑾𝟐𝟐 𝑾𝟐𝟑 … 𝑾𝟐𝒏 𝑫𝟑 𝑾𝟑𝟏 𝑾𝟑𝟐 𝑾𝟑𝟑 … 𝑾𝟑𝒏 … … 𝑫𝒎 𝑾𝒎𝟏 𝑾𝒎𝟐 𝑾𝒎𝟑 … 𝑾𝒎𝒏 26 𝐼𝐷𝐹𝑖 = log2𝑁1 + 𝑛𝑖, (5.5) where 𝑁 is the total number of documents in the Corpus and 𝑛𝑖 is the number of documents where the term 𝑖 occurs. Here the denominator uses 1 + 𝑛𝑖 instead of 𝑛𝑖 in case the term 𝑖 is not in the corpus, leading to the division-by-zero. Then the TFIDF value for the term 𝑖 in the document 𝑗 can be computed as: 𝑇𝐹𝐼𝐷𝐹𝑖𝑗 = 𝑇𝐹𝑖𝑗 ∙ 𝐼𝐷𝐹𝑖 (5.6) The terms with the highest TFIDF values tend to be the terms that best represent the topic of the document. 5.3 Feature Selection Feature selection is a process of selecting a subset of original features for use in model construction [35]. It is a key step in text classification as the relevance of features directly affects classification performance. We used the BoW model to represent a document as discussed in last section; in practice, the size of vocabulary in a corpus is quite large, attaining several thousands and even more, which causes superhigh dimensionality of feature space. Most terms in the vocabulary are absent in a single document, resulting in high sparsity of feature space. High dimensionality and sparsity not only increase computational complexity and time, but also cause many classification algorithms are not applicable. Moreover, high dimensional and sparse feature space may introduce the noise, incur overfitting, and thus degrade classification accuracy. By selecting the most relevant and informative features, we can reduce computational complexity, filter out the noise, mitigate overfitting, and thus improve classification performance. Based on different feature evaluation functions, many varieties of feature selection techniques were developed, including Document Frequency (DF), Chi-squared Statistics, Information Gain (IG), Mutual Information (MI), etc. There are also other feature selection approaches that select the important features based on some metrics after data fitting, such as L1 regularization. For this study, we adopted Chi-squared Statistics and L1 regularization for feature selection due to their recognized good performance. 27 5.3.1 Chi-squared Statistic Chi-squared test, also known as 𝜒2 test, is a hypothesis test that evaluates how likely the null hypothesis is true based on the extent to which the observed values deviate from the expected values [36]. The larger the deviation is, the less likely the null hypothesis is true. To measure the deviation of observed values from expected values, Karl Pearson introduced the Chi-squared statistic (𝜒2) [37], which can be simply represented as: 𝜒2 = ∑(𝑥𝑖−𝐸𝑖)2𝐸𝑖𝑛𝑖=1 , (5.7) where 𝑥𝑖 and 𝐸𝑖 are an observed value and the corresponding expected value respectively. A typical application of Chi-squared test is testing whether two variables, say 𝑋1 and 𝑋2, are independent. As far as text classification is concerned, Chi-squared test can be used to evaluate how relevant a certain term 𝑇𝑖 is with a document category 𝐶𝑗. The essence of the idea is that there should be no big difference in the probability that the term 𝑇𝑖 appears in a document whether the category of the document is 𝐶𝑗 if there is no relevance between the term 𝑇𝑖 and the category 𝐶𝑗. Based on the idea, 𝜒2 can be computed as: 𝜒2(𝑇𝑖, 𝐶𝑗) =𝑁(𝑃(𝑇𝑖,𝐶𝑗)𝑃(𝑇𝑖,𝐶𝑗)−𝑃(𝑇𝑖,𝐶𝑗)𝑃(𝑇𝑖,𝐶𝑗))2𝑃(𝑇𝑖)𝑃(𝑇𝑖)𝑃(𝐶𝑗)𝑃(𝐶𝑗), (5.8) where 𝑇𝑖 represents that the term 𝑇𝑖 is in a document and 𝐶𝑗 represents that the category of a document is 𝐶𝑗 while 𝑇𝑖 and 𝐶𝑗 represent the opposite events, and N is the number of documents. Suppose the term 𝑇𝑖 is irrelevant with the category 𝐶𝑗 , then the value of 𝜒2(𝑇𝑖 , 𝐶𝑗) should be small. Conversely, the larger the value of 𝜒2(𝑇𝑖 , 𝐶𝑗) is, the more relevant the term 𝑇𝑖 is with the category 𝐶𝑗 . Thus, the process of feature selection is computing the Chi-squared statistic 𝜒2(𝑇𝑖, 𝐶𝑗) for each feature, sorting the features in descending order by the value, and then selecting the best k features, where you specify the value of k according to the needs. 5.3.2 L1 Regularization Regularization is an approach to preventing overfitting and thus to enhance the generalizability of the trained model in machine learning [38]. Mathematically, regularization avoids training too 28 complex a model by adding to the cost function a regularization term that penalizes the complexity of the fitting function. L1 regularization is one of the most common regularization techniques, which adds the L1 penalty equal to the sum of absolute values of coefficients. It can be described as: 𝐶𝑜𝑠𝑡 𝐹𝑢𝑛𝑐𝑡𝑖𝑜𝑛 = 𝐿𝑜𝑠𝑠 + 𝜆 ∑ |𝑤𝑖|𝑘𝑖=1 , (5.9) where 𝜆 is a parameter controlling the tradeoff between the loss function and the penalty term and 𝑤𝑖 is one of the coefficients to be trained. By this way, L1 regularization can limit the value of the coefficients and even shrink some coefficients to zero, yielding a sparse coefficient vector which means the corresponding features do not contribute to the output and thus can be removed from the model. This property of L1 regularization allows it to be used for feature selection in addition to overfitting prevention. 5.4 Classification Algorithms For this study, we adopted three supervised learning algorithms (i.e., Naïve Bayes, Support Vector Machine, and Random Forest) to implement our classifiers due to their acknowledged good performance. 5.4.1 Naïve Bayes Naïve Bayes (NB) is a supervised learning algorithm based on applying Bayes’ theorem with the naïve assumption of conditional independence between every pair of features given the category [39]. Given a data set {(𝕏1, 𝕪1), (𝕏2, 𝕪2), (𝕏3, 𝕪3), ⋯ , (𝕏𝑚 , 𝕪𝑚)}, where m is the number of samples, each sample has n features, i.e., 𝕏𝑖 = (𝑥𝑖1 , 𝑥𝑖1 , ⋯ , 𝑥𝑖𝑛), and the class label 𝕪𝑖 ∈{𝑦1, 𝑦2, ⋯ , 𝑦𝑙}, 1 ≤ 𝑖 ≤ 𝑚, 𝑝(𝕪 = 𝑦𝑗| 𝕏 = x), 1 ≤ j ≤ l is defined as the probability that the label of a given sample x is 𝑦𝑗. Given a new sample x, to decide which category this sample belongs to, 𝑝(𝕪 = 𝑦1| x),𝑝(𝕪 = 𝑦2| x), 𝑝(𝕪 = 𝑦3| x), ⋯ , 𝑝(𝕪 = 𝑦𝑙| x) can be computed respectively and then its 29 category label is the one which has the biggest probability value. According to Bayes theorem, we have 𝑝(𝕪 = 𝑦𝑗|x) = 𝑝(𝑦𝑗)𝑝(x|𝑦𝑗)𝑝(x) (5.10) In general, Naïve Bayes algorithm supposes that each feature is independent of one another. Then Equation (5.10) can be decomposed as: 𝑝(𝕪 = 𝑦𝑗|x) = 𝑝(𝑦𝑗) ∏ 𝑝(𝑥𝑘|𝑦𝑗)𝑛𝑘=1∏ 𝑝(𝑥𝑘)𝑛𝑘=1 (5.11) Thus the problem is to solve 𝕪 = 𝑎𝑟𝑔 max𝑦𝑗𝑝(𝑦𝑗) ∏ 𝑝(𝑥𝑘|𝑦𝑗)𝑛𝑘=1∏ 𝑝(𝑥𝑘)𝑛𝑘=1 (5.12) As ∏ 𝑝(𝑥𝑘)𝑛𝑘=1 is the same for a given corpus, Equation (5.12) can be simplified as: 𝕪 = 𝑎𝑟𝑔 max𝑦𝑗𝑝(𝑦𝑗) ∏ 𝑝(𝑥𝑘|𝑦𝑗)𝑛𝑗=1 (5.13) ➢ Maximum Likelihood Estimation To estimate the value of 𝑝(𝑦𝑗) and 𝑝(𝑥𝑘|𝑦𝑗) , the general method is Maximum Likelihood Estimation (MLE), which estimates the probability according to the frequency on the limited samples. Specifically, for 𝑝(𝑦𝑗), the value estimation is 𝑝(𝑦𝑖) = 𝑁𝑦𝑗𝑁, (5.14) where 𝑁𝑦𝑗 is the number of samples in class 𝑦𝑗 and 𝑁 is the total number of samples. Supposing the value set of the k𝑡ℎ feature of all the samples is {𝑎𝑘1, 𝑎𝑘2, … , 𝑎𝑘𝑠𝑘} , the conditional probability 𝑝(𝑥𝑘 = 𝑎𝑘𝑝|𝑦𝑗), 𝑎𝑘𝑝 ∈ {𝑎𝑘1, 𝑎𝑘2, … , 𝑎𝑘𝑠𝑘} is estimated as the relative frequency of feature 𝑥𝑘 = 𝑎𝑘𝑝 in the samples belonging to class 𝑦𝑗: 𝑝(𝑥𝑘 = 𝑎𝑘𝑝|𝑦𝑗) = 𝑁𝑥𝑘=𝑎𝑘𝑝∩𝑦𝑗𝑁𝑦𝑗, (5.15) where 𝑁𝑥𝑘=𝑎𝑘𝑝∩𝑦𝑗 is the number of samples whose k𝑡ℎ feature 𝑥𝑘 = 𝑎𝑘𝑝 in class 𝑦𝑗. 30 ➢ Multinomial Naive Bayes The problem with MLE is that the estimated probability is zero when the case that a sample belongs to class 𝑦𝑗 and the value of its k𝑡ℎ feature is 𝑎𝑘𝑝 never occurred in the training data set. This will affect the computation of the posterior probability, resulting in wrong classification. To solve this problem, Multinomial Naive Bayes (MNB) model can be adopted to smooth the prior probability and conditional probability. Specifically, the prior probability 𝑝(𝑦𝑗) is estimated as: 𝑝(𝑦𝑗) = 𝑁𝑦𝑗+ 𝛼𝑁+𝑙𝛼 , (5.16) and the conditional probability 𝑝(𝑥𝑘 = 𝑎𝑘𝑝|𝑦𝑗) is estimated as: 𝑝(𝑥𝑘 = 𝑎𝑘𝑝|𝑦𝑗) = 𝑁𝑥𝑘=𝑎𝑘𝑝∩𝑦𝑗+ 𝛼𝑁𝑦𝑗+ 𝑠𝑘𝛼, (5.17) where l is the size of the class set 𝑦, 𝑠𝑘 is the size of the value set of the k𝑡ℎ feature, and 𝛼 is the smoothing value. When α = 1, it is called Laplace smoothing. ➢ Bernoulli Naive Bayes Different from MNB model, each sample is a binary vector with each dimension being either 0 or 1 in Bernoulli Naive Bayes (BNB) model, i.e., 𝑥𝑘 ∈ {0, 1}. Accordingly, the conditional probability 𝑝(𝑥𝑘|𝑦𝑗) can be described as: 𝑝(𝑥𝑘|𝑦𝑗) = 𝑝(𝑥𝑘 = 𝑏|𝑦𝑗)𝑏(1 − 𝑝(𝑥𝑘 = 𝑏|𝑦𝑗))(1−𝑏), 𝑏 ∈ {0, 1} (5.18) Then the conditional probability 𝑝(𝑥𝑘 = 1|𝑦𝑗) is estimated as: 𝑝(𝑥𝑘 = 1|𝑦𝑗) = 𝑁𝑥𝑘=1∩𝑦𝑗+ 𝛼𝑁𝑦𝑗+ 2𝛼, (5.19) 5.4.2 Support Vector Machine Support vector machine (SVM) is a supervised learning model proposed by Corinna Cortes and Vladimir Vapnik in 1995 [40], which has been generally used in pattern recognition, classification, and regression due to its high generalizability nowadays. The basic idea behind SVM is mapping the linearly inseparable data to the high-dimensional space where the data can be separated by a separating hyperplane and then finding the hyperplane that can best separate the data. Here the best separating hyperplane refers to the 31 hyperplane that has the largest distance from the closest point or vector in each class. The closest vectors to the separating hyperplane are called support vectors. Finally, the problem is to find the optimal solution to the objective function that maximizes the distance from the separating hyperplane to the support vectors. It is generally recognized that SVM manifests good performance on the non-linear, small data sets in terms of classification accuracy and is insusceptible to overfitting. It is also considered by some people to be the best stock classification model; in other words, it can work well on a data set in its basic form. 5.4.3 Random Forest Random forest is an ensemble learning method based on decision tree, which performs by constructing multiple decision trees in randomly selected subspaces of the feature space, training the decision trees individually and outputting the result that is the mode of classes (for classification tasks) or the mean of values (for regression tasks) predicted by all trees [41]. Random forest employs the bagging technique to train the individual decision trees. Specifically, it samples a random subset of the data with replacement for each tree, selects a random subset of the features at each node of a tree, and conducts a binary split on the node using the feature that best splits the data subset according to some objective function. Each data sampling leaves one third of the data out of the sample for making an unbiased estimate of the classification error. The data out of the sample is known as out-of-bag data. It is also used to estimate the importance of features. It is proved that random forest generally outperforms decision tree as the former is insensitive to the noise or outliers in the data and also robust on the unbalanced data even with missing values. 32 5.5 Performance Evaluation 5.5.1 Evaluation Method In this study, we adopted 10-fold cross validation to evaluate our model. Cross validation is a generally-used model evaluation technique in machine learning for assessing how the trained model can generalize to the unseen data [42]. More specifically, k-fold cross validation randomly splits the data set into k subsets of equal size {𝑆1, 𝑆2, … , 𝑆𝑖 , 𝑆𝑖+1, … , 𝑆𝑘} and repeats the training and testing k times. In the ith validation, the subset 𝑆𝑖 is selected as the test data set and the remaining subsets are used as the training data set. The k results are averaged as the final evaluation result after k iterations. 5.5.2 Evaluation Metrics Considering our data sets are extremely unbalanced as technical debt issues only account for a very small proportion in the whole data set, we use the precision, recall and F1-score for the positive class (i.e., technical debt issues) rather than predictive accuracy to evaluate our model. In a binary classification task, each item classified falls into one of the four elements according to the classification result and label: true positives (TP), false positives (FP), true negatives (TN), false negatives (FN) [43] shown in Table 5.1. TP is the items that are correctly classified as the positive class while FP is the items that are incorrectly classified as the positive class. Similarly, TN is the items that are correctly classified as the negative class while FN is the items that are incorrectly classified as the negative class. Table 5.1 Four Elements of Classification Result Label Classification Positive Negative Positive TP FP Negative FN TN 33 Precision, recall, and F1-score are computed by the number of the four classification elements [43]. Precision is the number of TP divided by the number of items that are classified as the positive class whether the classification is correct or not, which can be calculated as: 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =𝑇𝑃𝑇𝑃+𝐹𝑃 . (5.20) Recall is the number of TP divided by the number of items that truly belong to the positive class whether they are classified as positive or negative, which can be calculated as: 𝑅𝑒𝑐𝑎𝑙𝑙 =𝑇𝑃𝑇𝑃+𝐹𝑁 . (5.21) F1-score is the harmonic mean of precision and recall, evaluating the classification performance by combining precision with recall as shown in Equation (5.22): 𝐹1 − 𝑠𝑐𝑜𝑟𝑒 = 2 ∙𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 ∙𝑅𝑒𝑐𝑎𝑙𝑙𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛+𝑅𝑒𝑐𝑎𝑙𝑙 . (5.22) We also use the receiver operating characteristic (ROC) curve to evaluate the performance of our model. A ROC curve is a graphical plot that illustrates the diagnostic ability of a binary classifier system as its discrimination threshold is varied [44]. In a ROC curve, the horizontal axis represents false positive rate (FPR) while the vertical axis represents true positive rate (TPR). Actually, TPR is another name of recall, which measures the proportion of TP out of the total number of actually positive items as shown in Equation (5.23). 𝑇𝑃𝑅 =𝑇𝑃𝑇𝑃+𝐹𝑁 . (5.23) FPR measures the fraction of FP out of the total number of actually negative items as shown in Equation (5.24): 𝐹𝑃𝑅 =𝐹𝑃𝐹𝑃+𝑇𝑁 . (5.24) An ideal binary classifier should achieve a high TPR in the expense of a low FPR, which is reflected by a large area under the curve (AUC) in the ROC curve. 34 Chapter 6 Findings In this chapter, we present our findings of both qualitative and quantitative analysis corresponding to our research questions proposed in Chapter 1. The findings are organized in terms of distribution, characteristics, and text patterns of technical debt as well as the performance of our classification model. 6.1 Distribution of Technical Debt After manual analysis, 921 and 331 instances of technical debt are identified for Eclipse Che and Cornerstone data set, accounting for 10.61% and 4.06% of their respective data sets shown in Figure 6.1. In Eclipse Che issue data set, design debt is the most common technical debt type, accounting for 39% of all technical debt instances. Usability debt, test debt, code debt, and architecture debt are also frequent technical debt types, occupying 15%, 15%, 14%, and 9% respectively. The remaining technical debt items (e.g., requirement debt, build debt, and infrastructure debt) constitute 8% in total. Likewise, design debt is also the most frequent technical debt type in Cornerstone issue data set, making up 43% of all technical debt instances. Requirement debt, usability debt ranks the second and third in terms of the number of instances, consuming 32% and 11% respectively. Code debt, defect debt, and other types of technical debt compose the remaining 14% of technical debt instances. Figure 6.2 demonstrates how different types of technical debt are distributed in the two issue data sets. 35 Figure 6.1 The Proportion of Technical Debt in Two Issue Data Sets The distribution of technical debt varies between Eclipse Che and Cornerstone issue data set although design debt and usability debt are common technical debt types in both issue data sets. The two biggest differences come from requirement debt and test debt as you may have found. Requirement debt is very rare in Eclipse issue data set but has highly extensive instances in Cornerstone issue data set. On the contrary, test debt is frequent in Eclipse Che issue data set, but uncommon in Cornerstone issue data set. 36 Figure 6.2 The Distribution of Technical Debt in Two Issue Data Sets 6.2 Characteristics of Technical Debt 6.2.1 Open Time of Technical Debt We hypothesized that technical debt related issues may take longer to resolve than bugs. That is, technical debt related issues have longer open time than bugs. Since many issues in Cornerstone data set were recorded without creating time, closing time or open time, we only took a deep 37 probe into open time in Eclipse Che issue data set. Figure 6.3 compares the distribution of open time (hours) of different types of technical debt with that of bugs. Figure 6.3 Open Time of Different Types of Technical Debt and Bugs Usability debt and architecture debt shows longer open time on the whole than bugs; however, there is no significant difference between other types of technical debt (i.e., design debt, code debt, and test debt) and bugs. Meanwhile, the figure also demonstrates that architecture and usability debt have a wider distribution in open time. The statistics of open time for these different types of issues are further shown in Figure 6.4. 38 Figure 6.4 The Statistics of Open Time of Different Types of Technical Debt and Bugs It is evident that although architecture and usability debt have larger mean and median values in open time than bugs, the other types of technical debt show inconsistent trend in terms of open time. Moreover, the standard deviation (SD) and interquartile range (IQR) of open time for architecture and usability debt are much bigger than other types of issues, indicating that open time is significantly different even for technical debt of the same type. Therefore, the previous hypothesis is not true. Open time is not a distinguishing characteristic of technical debt at least for this issue data set; however, different results may be yielded when analyzing larger and more data sets. 6.2.2 Priority of Technical Debt We hypothesized that defects have higher priority than technical debt. For this hypothesis, we only analyzed Cornerstone issue data set since Eclipse Che issue data set does not have the priority variable. Figure 6.5 shows the priority of different types of technical debt and defects. 39 Figure 6.5 The Distribution of Priorities of Different Types of Technical Debt and Defects There are four types of priority in Cornerstone issue data set: critical, major, normal, and show-stopper. Normal priority is the most common in all types of technical debt issues and defect issues. Major priority is widely distributed in all issue types except code debt. However, Critical priority is mainly frequent in defect, requirement debt, and usability debt, and show-stopper only appears in defect issues. On the other hand, all types of technical debt but code debt include at least two types of priorities while only defect issues cover all types of priority. Based on the figure, we have no evidence to conclude that defects have higher priority than technical debt or technical debt has some consistent patterns in terms of priority. 6.3 Text Patterns of Technical Debt In this section, text patterns in the description of technical debt issues we found are expounded in detail, including the indicators of different types of technical debt, the selected features for 40 identifying technical debt, the similarities in the expression of technical debt for two issue tracking data sets, and the semantic dimensions of technical debt issue description. 6.3.1 Indicators of Technical Debt This subsection presents the example phrases or sentences which provide the evidence that we labeled the issues as technical debt items explicitly or implicitly by technical debt types. 6.3.1.1 Explicit Indicators None of these two data sets used technical debt as an issue type. To investigate whether software developers communicate technical debt explicitly in issue trackers using the phrase like “technical debt” or “debt”, we searched for the term “technical debt” and “debt” in both data sets, and the corresponding Chinese term “技术债务” and “债务” in Cornerstone. Only Eclipse Che returned positive results. There are 8 issues whose description contains explicit expression of technical debt in total for Eclipse Che. The examples of snippets of issue description are shown below with explicit indicators underscored. • [Eclipse Che #3963] We don't use Machine API anymore. We should remove client in IDE since it is additional technical debt for maintainers. • [Eclipse Che #3621] SSH machine is not widely used anymore and prevents refactoring in docker machines implementations. So it would reduce technical debt if we move this implementation with all needed classes to separate plugin aka ssh machine plugin. Also we can cleanup docker plugin after that move. In both examples above, the term “technical debt” is used in the description of issues to indicate that they are instances of technical debt. For issue #3963 of Eclipse Che, the description indicates that the obsolete functions should be removed otherwise increasing additional technical debt. Issue #3621 suggests separating a plugin with another to reduce technical debt. • [Eclipse Che #3621] ### Technical debts and bugs 41 [x] Cannot register a user after it has been deleted in Keycloak | https://github.com/eclipse/che/issues/8081 8sp [ ] Unable to override idle timeout for a particular user | https://github.com/eclipse/che/issues/8951 8sp [x] Provide some meneful error message during user manipulation operation in case if Che is configured to work with Keycloak | https://github.com/eclipse/che/issues/9015 8sp [x] Document how to configure JDBC datasource of workspace master | https://github.com/eclipse/che/issues/9037 5sp The above snippet is another example of issues with explicit indicators of technical debt. In this issue, the term “technical debts” appears in the format text instead of descriptive text. The underscore texts indicate this issue contains an instance of usability debt. 6.3.1.2 Implicit Indicators Although the term “technical debt” or “technical debts” is used in Eclipse Che data set, most issues for Eclipse Che and all issues for Cornerstone are expressed implicitly. The examples of each type of technical debt we focused on for both data sets are shown in order. To allow you to understand the examples for Cornerstone, the translation of issue description is also appended below each example. ➢ Design Debt • [Eclipse Che #6142] Temporary workaround limit search result to 50 file for preventing freeze/crash browser Common solution will be provided in the https://github.com/eclipse/che/issues/6143 • [Eclipse Che #6085] Note that it is supposed to be a temporary solution until we realize the proper way to fix it. • [Eclipse Che #5871] DockerMachineStarter class is old and big and it doesn't follow single responsibility principle. Because of this, it is hard to write tests to it as part of [this issue](https://github.com/eclipse/che/issues/5575). We need to refactor it and cover its functionality with tests. 42 I suggest moving all docker system settings from this class to InfrastructureProvider class impl. I also suggest that it can be split into several components for work with networks, images, containers. • [Eclipse Che #5775] In previous version client itself was responsible for ID generation and providing to server-side, that approach had its obvious shortcomings. The new approach rearrange responsibilities in a such way that server-side now have a centralized way of generating a unique identifier, while client can get that identifier and use it for its identification further. • [Eclipse Che #5639] As for the time being it is already implemented and requires improvements only, I just would like to decouple Docker issues from SPI issue (which contains big amount of sub-tasks already) itself to make it clear. • [Eclipse Che #5581] So we need to rework our workspace links injector in accordance with client`s needs and changes in spi branch. We also want to simplify Links model. It is supposed to be done as a part of this issue. The current model is complicated and, lately, we used to use it in the simplest use cases only. So we decided to simplify it significantly. • [Eclipse Che #2242] In current state our dto generator builds the objects using the string builder, but it is not effective so we should replace the current implementation on string templates for server and client dtos. • [Eclipse Che #1864] But now code of environment start is tightly coupled with workspace runtime management code. It makes addition of SPI hard and very complex task. Instead of that this issue should introduce refactoring of that code to simplify SPI addition. The above snippets of issue description are typical examples of design debt instances in Eclipse Che issue data set. The key phrases like temporary solution and workaround are commonly used to communicate that the current solution is expedient and should be replaced by a better solution in the future, which implies design debt is taken on. A set of verbs including move, split, decouple, simplify, replace, and so forth indicate code refactoring should be conducted at the class, component, or module level. Distinctive positive and negative adjectives (e.g., big, clear, 43 transparent, poor, effective, and coupled) suggest different reasons for refactoring such as god class, ineffective approach, and high coupling. • [Cornerstone #1813] 在满足一定条件下，SDK 做成通用的，可以自动发现硬件型号，并自适应运行，这样可以提高代码的维护性及通用性。 [Translation] In some conditions, SDK should be universal and more adaptive to different versions of hardware. By this way, the maintainability and universality of code can be improved. • [Cornerstone #5729] 目前审计产品中的协议审计功能，需要重新设计，包括接口，页面等等。 [Translation] Currently, the protocol audit function of the audit product needs to be redesigned including the interface and page. • [Cornerstone #5732] 目前协议审计功能与 flowdate 共用一套 topic 接口，导致控制上比较麻烦，考虑优化这部分设计。 [Translation] Currently, the protocol audit function and flowdate share the same topic interface, resulting in the control is cumbersome. We should optimize this part of design. Likewise, some verbs in Chinese such as “提高” (improve), “优化”(optimize) and “重新设计”(redesign) implicate that code refactoring should be performed while the words like “维护性” (maintainability), “通用性” (universality) and “麻烦” (cumbersome) suggest different reasons for refactoring in Cornerstone data set. ➢ Code Debt • [Eclipse Che #8637] But we still have many places where jdt code is used. So after closing next issues we should make some code cleanuping again: • [Eclipse Che #6140] Remove unsupported legacy code of ant support. It was discontinued before Eclipse Che 4.x Now I've just remove unused source code. • [Eclipse Che #5221] Clean-up deprecated, not used and obsolete code: 44 • [Cornerstone #4678] 在 all-in-one 的版本里面需要把 led 指示的 code 从 DPI 移植回 MW [Translation] We need to transplant the code for led from DPI to MW in all-in-one version. • [Cornerstone #9070]整合 suricata 3.0 代码进入 southwest-engine 的 src； [Translation] Merge the code of suricata 3.0 into southwest-engine Removing or cleaning up unused, deprecated or obsolete code is the most common expression pattern of code debt. Besides, the verbs like “移植” (transplant), “整合” (merge) and so on are also frequently used to express the existence of code debt. ➢ Architecture Debt • [Eclipse Che #4677] We need to organise source code by separate modules according to their logical structure • [Eclipse Che #2453] - Improves factory components structure: separates the logic between service, manager and DAO • [Eclipse Che #1790] Today to store information we using in memory storages which is backed as json files during application suspend-startup. This approach is not scalable enough. To make it more scalable, fault tolerant and concurrent we decided to move all our persistent storages to JPA based implementation. Potentially we will replace our in-memory implementation with JPA based. As JPA implementation we will take EclipseLink and h2 database. • [Cornerstone #369] 改进 DPI Box 中 agent 的架构设计 [Translation] Improve the design of agent architecture in DPI Box • [Cornerstone #384] DPI engine 实时包转发和业务处理是第一任务，所以 DPI 的架构设计上是越简单、直接越好。引入太多的东西，使得架构过于复杂，同时给系统稳定性带来挑战。万一在部署过程中遇到一个深层次的问题，将非常难于定位。 [Translation] Real-time packet transfer and transaction processing is the prime task of DPI engine, so the simpler and more direct DPI architecture is, the better it is. Introducing too much stuff makes the architecture too complicated and brings the challenge in system stability. It is hard to locate the deep-seated problem encountered in the process of deployment. 45 • [Cornerstone #9067] 整个过程包括 southwest-engine 基础框架升级、modbus 的重构和southwest-engine 架构的初步优化； [Translation] The whole process includes the upgrade of southwest-engine framework, the refactoring of modbus, and the initial optimization of southwest-engine architecture. Architecture debt related issues can be indicated by the concurrence of some verbs (e.g., organize, improve, separate, replace, etc.) and nouns (e.g., modules, structure, architecture, etc.). Meanwhile, some adjectives (e.g., scalable, complicated, logical, etc.) signify why the architecture should be improved. ➢ Usability Debt • [Eclipse Che #6330] The scrolled line usually appears in the bottom of the editor that isn't a convenient. • [Eclipse Che #5128] The current Eclipse Che version is providing a Git plugin with many of features but the overall experience is tedious for users. Simple actions are needing a lot of interactions and the user flow have been made complex. The current git integration is also lacking of feedbacks to show to the user the changes that have been made. The goal of this epic is to rethink the Git experience in order to improve the developer efficiency and provide integrated features (in the editor and in the explorer). • [Eclipse Che #4678] It is very cumbersome to have to scroll to the exact branch that I need ( I'd rather use the git command line to do it ). This can be fixed if there was a filter input box on the top of the branches sections so that I can fuzzy find the branch that I am looking for. • [Eclipse Che #4359] In the past, we have unified the flows to create and edit workspaces, but it added more usability challenges to the end-user than the benefits. We also introduced the multi-machines workspaces which also complexified the usability of the Dashboard when it's time to configure a workspace. The goals of this epic is to improve the user experience around creating workspaces from Dashboard and managing them. We will also rethink the loading experience and the transitions between the Dashboard and the browser IDE. • [Cornerstone #1275] 建议数字和文字采用左对齐格式，并给数字留够长度，看起来更46 美观些，也更符合中国人习惯，看着更舒服 [Translation] Suggest adopting left-alignment format for numbers and words and leaving enough space for numbers, which makes it look prettier, more comfortable and also conform to Chinese habits more. • [Cornerstone #1799] 当系统消息过多时，通知列表显示太长，影响体验，建议缩短至显示 10条+查看所有功能 [Translation] The notification list is too long, affecting the experience when there are too many system messages. Suggest displaying only 10 messages by default and checking the function of checking all messages. • [Cornerstone #7925] 目前入侵检测页面 Tab 的 style 和其他页面颜色不一样，需要统一 [Translation] Currently, the style of tabs in the page of intrusion detection is different from that of other pages. The styles need to be unified. Usability debt mainly relates to bad user experience or bad UI design such as cumbersome operation, ugly UI design, and non-uniform of UI style, which can be indicated by a set of key phrases like UI, user experience, style, pretty, ugly, convenient, inconvenient, tedious, comfortable, etc. ➢ Requirement Debt • [Eclipse Che #6330] It works fine for `openshift` recipes but it doesn't work while converting `dockerimage` recipes to `openshift` recipes. To solve this problem it is needed to add an ability to match machine from configuration to OpenShift container without the composite machine name. It's proposed to rework OpenShift container to machine configuration in the following way: • [Cornerstone #2108] 目前暂先把“USR”指示灯隐藏，与卡轨式智能保护终端风格保持一致，等研发同事有时间了再开发这个需求，目前研发同事先做需求优先级高的。 [Translation] Currently, hide the USR indicator light temporarily, and keep the style consistent with that of smart protection terminal. The developers implement the requirements of high priority first and implement this requirement later when they have time. 47 The above examples are two typical instances of requirement debt, where requirements are not implemented for all cases or it is deferred to implement a requirement due to some reason such as time constraint. ➢ Test Debt • [Eclipse Che #7829] The test has many unnecessary steps and illogical operations for instance: we use 2 projects for checking pull feature. We perform Push, add to index, remove operation in first project from IDE. In second project we perform pull and check previews operations. This add unstable parts in the test and add many extra - steps. Also this make hard to understand what the test do. • [Eclipse Che #6332] Existing selenium tests for checking Junit and TestNg are deprecated we should adapt existing tests and write new for the covering current state of the plugin. • [Cornerstone #9502] 修改适配之前的 unit_test 框架，使之适应新的 thrift 通信，整理输出格式。 [Translation] Modify the framework of unit test to adapt to the new thrift communication. Clean up the output format. Test debt related issues generally comprise of two cases. One is the problem of test methods or classes themselves such as poor-quality code. The other is that test methods or classes should be refactored to adapt to the change of code base. ➢ Combination of Multiple Debts • [Eclipse Che #7829] * Apply kohsuke github library for removing redundant actions with UI (for example: we should not perform GitPull from the CHE IDE for checking Push operation) * Remove duplicated steps from the tests. We had a lot of checks that made duplicate Push actions from UI. (For example change html file and push, change jsp file and push). On this moment we change the file, add new file and check that pushing operation to remote repo works properly. 48 * Add method to the TestGitHubRepository class for reading content * Split the PushingChangesTest to 2 independent separated tests for checking pushing and pushing with force feature * Remove extra steps like hard reset checking. It will be moved to separate test class Many issues involve several different types of technical debt. The above example is an issue related to usability debt and test debt. Specifically, the words “removing redundant actions with UI” indicate an instance of usability debt while the other underlined words such as “remove duplicated steps from the tests” are indicators of test debt. Through manual analysis on both issue data sets, we found that there are many similarities in communication patterns of technical debt in issue trackers as discussed above. For instance, the verbs such as improve, simplify, transplant, remove, merge, refactor are widely used to communicate that code refactoring should be conducted to pay off various types of technical debt in Eclipse Che issue data set while the counterparts of these verbs (e.g., “提高”, “简化”, “移植”, “移除”, “整合”, and “重构”) in Chinese are also important indicators of technical debt in Cornerstone issue data set. 6.3.2 Features Selected for Identifying Technical Debt The number of original features for Eclipse Che and Cornerstone issue data set without bigrams and trigrams is 3209 and 5772 respectively. We firstly conducted feature selection using Chi-squared statistic technique on both data sets and evaluated how efficient these different features are in identifying technical debt issues using the F1-score metric of the SVM classifier trained based on the selected features. 49 Figure 6.6 The F1-score of the SVM Classifier against the Number of Features Using Chi-squared Statistic Technique to Select Features The mean F1-score of the SVM classifier trained using different numbers of features selected by this method in the 10-fold cross validation for both data sets are shown in Figure 6.6. F1-score reaches the highest value (0.61) at the number of features is 601 for Eclipse Che while the same value is attained when 1221 features are selected for Cornerstone. The shadow in Figure 6.6 represents the standard deviation of F1-score. 50 Figure 6.7 The F1-score of the SVM Classifier against the Penalty Parameter C Using L1 Regularization Technique to Select Features Afterward, we performed feature selection using L1 regularization technique and evaluated how important the features selected by tuning the value of the penalty parameter C are in detecting technical debt issues by F1-score. Specifically, we initialized the value of C as 0.01 and then increased it with the step size of 0.01 until it was equal to 3.0. Figure 6.7 shows the mean F1-score of the SVM classifier in the 10-fold cross validation against the value of the penalty parameter C for both data sets. For Eclipse Che, the classifier retains 563 features and reaches the best F1-score (0.64) at C = 0.43; for Cornerstone, the best F1-score (0.71) is achieved training the classifier on 480 features selected when the value of C is tuned as 1.70. 51 Figure 6.8 The F1-score of the SVM Classifier against the Penalty Parameter C Using L1 Regularization Technique to Select Features Including Bigrams and Trigrams You may have noticed that not only did L1 regularization select fewer features than Chi-squared Statistic, but also the classifier trained on the features selected by L1 regularization has better F1-score for both data sets. As a result, we adopted L1 regularization for feature selection. Furthermore, we would like to probe into whether the performance of the classifier can be improved in terms of F1-score by including bigram and trigram features in our model. Figure 6.8 shows the mean F1-score of the SVM classifier in the 10-fold cross validation against the value of the penalty parameter C for both data sets with bigram and trigram features included. The F1-score indeed improves further as you may have observed. The number of features with bigrams and trigrams is 52952 and 61425 for Eclipse Che and Cornerstone originally, shrinks to 1350 and 52 528 when the value of C is 2.66 and 2.37 respectively, where the classifier trained based on their respective selected features achieves the best F1-score (0.74 and 0.76). Table 6.1 The Number of Features of Each Part of Speech Part of Speech # of Features in Eclipse Che # of Features in Cornerstone Verb 41 36 Noun 32 50 Adjective 35 25 Adverb 4 0 Bigram 26 28 Trigram 2 6 Total 140 145 To construct a more generalizable model, we further filtered out the features that are related to domain knowledge and that are far from the concept of technical debt at the semantic level manually based on our expertise on technical debt. Finally, we retained 140 and 145 features for the two issue data sets respectively. The features for the two issue data sets are comprised of four parts of speech of words (i.e., verb, noun, adjective, and adverb), bigrams and trigrams. No articles, pronouns, prepositions are in both feature sets as they have been removed as stop words in the phase of text preprocessing. Verbs, nouns, and adjectives are the three most common feature types. The details about the number of features of each part of speech are shown in Table 6.1. All features for Eclipse Che are in English while most features for Cornerstone are in Chinese except five English words (i.e., optimize, refactor, UI, risk, and code) shown in Table 6.2 and Table 6.3. The features for Eclipse Che and Cornerstone issue data set share some similarities in terms of semantic meaning. Table 6.4 lists some example words expressing similar semantic meaning in the two features sets. 53 Table 6.2 The Features for Eclipse Che Part of Speech Features Verb clean, fix, resolve, add, change, modify, update, upgrade, remove, move, split, migrate, transplant, extract, decouple, reduce, improve, deprecate, rework, rewrite, revise, reuse, refactor, redesign, reimplement, reorganize, restructure, replace, restyle, rename, separate, stabilize, simplify, refine, adapt, optimize, extend, unify, reproduce, fail, conflict Noun design, comment, method, class, interface, function, component, module, architecture, structure, code, codebase, legacy, debt, shortcut, workaround, solution, approach, typo, bug, implementation, framework, extensibility, experience, user, case, UI, layout, font, choice, lifecycle, future Adjective unused, redundant, unnecessary, duplicated, deprecated, obsolete, outdated, complicated, complex, temporary, hard, unexpected, illegal, illogical, pool, inconsistent, simple, stable, easy, clear, readable, better, usable, reusable, compatible, universal, extensible, convenient, handy, straight, dependent, independent, friendly, comfortable, current Adverb now, previously, currently, effectively Bigram technical debt, code cleanup, clean up, cleanup unused, unused resources, unnecessary steps, UI improvement, hard coded, temporary solution, duplicate code, duplicate button, redundant file, improve readability, remove deprecated, remove duplicated, remove redundant, remove unnecessary, remove unused, remove useless, replace current, replace implementation, replace native, separate class, separate module, structural refactoring, refactor method Trigram refactoring and decomposition, remove duplicated 54 dependency Table 6.3 The Features for Cornerstone Part of Speech Features Verb 修改，修正，修复，更改，变更，整改，改动，改成，改为，取代，替换，改进，改善，优化，简化，删除，取消，完善，提高，增强，重构，移植，整理，整合，合并，归并，解耦，拆分，调整，适应，扩展，维护，限制，影响，optimize，refactor Noun 目前，当前，现在，现有，前期，过去，原有，将来，今后，实际，现实，功能，需求，设计，理论，机制，算法，逻辑，架构，结构，代码，性能，效率，用户，客户，体验，操作，习惯，风格，样式，格式，困难，并发，可用性，易用性，可读性，易读性，可扩展性，可维护性，安全性，兼容性，稳定性，通用性，实时性，局限性，有效性，一致性，UI，risk，code Adjective 充分，方便，灵活，简单，直观，合理，晦涩，难懂，复杂，重复，不好，不正确，不正常，不一致，不合理，不方便，不清晰，不准确，不直观，不美观，不协调，不流畅，不符合，不友好，不容易 Bigram 数据结构，代码优化，接口简化，代码修改，优化方案，功能模块，功能设计，架构设计，现有设计，用户体验，界面显示，设计不合理，统一整理，移植 MW，机制改进，需要完善，代码移植，代码整理，框架修改，UI 修改，统一风格，保持一致，并不需要，调整布局，代码合并，易用性修改，功能未完善，不方便操作 55 Trigram 功能需要完善，统一页面风格，协议识别优化，完善单元测试，代码需要修改，架构初步优化 Table 6.4 Example Words Expressing Similar Semantic Meaning Eclipse Che Cornerstone change, modify 修改，更改，变更，整改，改动，改成，改为 fix 修正，修复 replace 取代，替换 improve 改进，改善，完善，提高，增强 optimize 优化 simplify 简化 refactor 重构 transplant 移植 clean, clean up 整理 merge 整合，合并，归并 decouple 解耦 split 拆分 adapt 适应 now, currently 目前，当前，现在 previously 前期，过去 future 将来，今后 design 设计 code 代码 function 功能 user 用户 experience 体验 convenient 方便 simple 简单 complicated 复杂 hard 不容易 56 poor 不好 6.3.3 The Semantic Dimensions of Technical Debt Issue Description In last subsection, we present the features for both issue data sets by parts of speech. From the semantic perspective, technical debt items are generally communicated from four dimensions (time accumulation, rework action, quality attributes, and problem scope) listed below. ➢ Time Accumulation [Eclipse Che] now, previously, currently, current, temporary, future [Cornerstone] 目前，当前，现在，现有，前期，过去，原有，将来，今后 ➢ Rework Action [Eclipse Che] clean, fix, resolve, add, change, modify, update, upgrade, remove, move, split, migrate, transplant, extract, decouple, reduce, improve, deprecate, rework, rewrite, revise, reuse, refactor, reorganize, replace, restyle, rename, separate, stabilize, simplify, refine, adapt, optimize, unify [Cornerstone] 修改，修正，修复，更改，变更，整改，改动，改成，改为，取代，替换，改进，改善，优化，简化，删除，取消，完善，提高，增强，重构，移植，整理，整合，合并，归并，解耦，拆分，调整，适应，扩展 ➢ Quality Attributes [Eclipse Che] unused, redundant, duplicated, obsolete, outdated, complicated, complex, illegal, illogical, pool, consistent, simple, stable, readable, usable, compatible, universal, convenient, handy, straight, dependent, independent, friendly [Cornerstone] 可用性，易用性，可读性，易读性，可扩展性，可维护性，安全性，兼容性，稳定性，通用性，实时性，局限性，有效性，一致性，方便，灵活，简单，直观，合理，晦涩，难懂，复杂，重复，不好，不正确，不正常，不一致，不合理，不方便，不清晰，不准确，不直观，不美观，不协调，不流畅，不符合，不友好，不容易 57 ➢ Problem Scope [Eclipse Che] design, comment, method, class, interface, function, component, module, architecture, structure, code, framework, experience, UI [Cornerstone] 功能，需求，设计，机制，算法，逻辑，架构，结构，代码，性能，效率，体验，UI The following issue description is a typical example showing how these four types of features indicate technical debt. In this example, “In the past” and “added more usability challenges” indicate time accumulation and affected quality attributes respectively. Furthermore, “improve the user experience” signify that this is a user experience problem and what rework action should be taken. However, these four dimensions of indicators do not always concur. For instance, the indicators of time accumulation do not occur as frequently as indicators of other dimensions or can be deduced from the context or other key words such as temporary approach and workaround. In addition, some key words are indicators of multiple dimensions. For example, scalable, as an indicator of quality attributes, may also suggest that the issue is an architecture-level problem. • [Eclipse Che #4359] In the past, we have unified the flows to create and edit workspaces, but it added more usability challenges to the end-user than the benefits. We also introduced the multi-machines workspaces which also complexified the usability of the Dashboard when it's time to configure a workspace. The goals of this epic is to improve the user experience around creating workspaces from Dashboard and managing them. We will also rethink the loading experience and the transitions between the Dashboard and the browser IDE. 6.4 Automatic Classification Results The performance of six classifiers trained using three classification algorithms (i.e., BNB, SVM, and RF) combined with two feature extraction methods (i.e., Boolean vector and TFIDF vector) is evaluated by the precision, recall, and F1-score for Eclipse Che and Cornerstone issue data set 58 respectively. BNB classifier has the best F1-score while SVM classifier performs best in terms of recall and RF classifier attains the highest precision shown in Table 6.5. The performance of six classifiers is also evaluated using the ROC plot on both data sets respectively. As we can see from Figure 6.9 and Figure 6.10, all six classifiers have better performance on Cornerstone than on Eclipse Che. On the other hand, BNB classifier has the best performance on both data sets in terms of AUC. Table 6.5 The Performance of Six Classifiers Classifier Eclipse Che Cornerstone Precision Recall F1-score Precision Recall F1-score BNB+Boolean 0.66 0.60 0.62 0.62 0.72 0.66 BNB+TFIDF 0.66 0.60 0.62 0.62 0.72 0.66 SVM+Boolean 0.47 0.80 0.59 0.53 0.87 0.65 SVM+TFIDF 0.41 0.82 0.54 0.42 0.86 0.56 RF+Boolean 0.71 0.48 0.56 0.74 0.38 0.51 RF+TFIDF 0.73 0.47 0.57 0.76 0.43 0.55 Since our study is the first one that applies NLP and machine learning techniques to identifying technical debt through issue trackers, there is no benchmark for us to compare the performance of our model with it. To have a more intuitive perception of how effective our model is, we use randomly labeling as a simple baseline to compare the F1-score of our model with that of this baseline. We suppose that each issue is labeled as technical debt with the same probability as not technical debt. The precision of randomly labeling can be calculated by dividing the number of technical debt issues by the total number of issues for each project. Then the precision of this approach for Eclipse Che is 9218678≈ 0.106 while that of this approach for Cornerstone is 3318149≈0.041. The recall of randomly labeling is 0.5 for both projects as there is 50 percent chance that the issues are labeled as technical debt. Thus, the F1-score of randomly labeling for Eclipse Che can be calculated as 2 ×0.106×0.50.106+0.5≈ 0.17 while that of randomly labeling for Cornerstone can be calculated as 2 ×0.041×0.50.041+0.5≈ 0.08. We can see that the best F1-score achieved by our classifiers 59 improves significantly compared with randomly labeling (3.65 times for Eclipse Che; 8.25 times for Cornerstone). Figure 6.9 The ROC of Six Classifiers for Eclipse Che 60 Figure 6.10 The ROC of Six Classifiers for Cornerstone 61 Chapter 7 Discussion This chapter answers the research questions we proposed in Chapter 1 and discusses the threats to validity in our study in detail. 7.1 Answering Research Questions • (RQ1) How are different kinds of technical debt issues distributed? Our study demonstrates that design debt is the most frequent technical debt type. Code debt and usability debt are also common technical debt types in both issue data sets. The differences in the distribution of technical debt between these two issue data sets mainly come from requirement debt and test debt. Requirement debt is very rare in Eclipse issue data set but has highly extensive instances in Cornerstone issue data set. On the contrary, test debt is frequent in Eclipse Che issue data set, but uncommon in Cornerstone issue data set. This can be ascribed to different nature of these two projects to a large extent. Since Cornerstone is a commercial software project, development tasks are oriented towards business requirements from customers. It is not uncommon in the development of commercial software projects that developers misunderstand or do not fully grasp the requirements of customers and therefore fails to implement some function that completely satisfies the specific requirement. Besides, time constraint is a big concern for commercial software projects. Software development teams may have to implement the requirements partially or in a compromising manner to provide their products and services for customers as early as possible. However, as Eclipse Che is an open source software project, requirements are not collected from customers, but are decided by developers’ ideas or concept of the product. Moreover, there is no delivery deadline before which the product should be released. Development progress is not controllable and predictable as the development of the product is the gratuitous and voluntary behavior of developers. Consequently, it makes sense that why there are so many requirement debt issues in Cornerstone but few in Eclipse Che. 62 For the difference in the number of test debt issues between Eclipse Che and Cornerstone issue data set, a possible reason may be test driven development have been adopted for the development of Eclipse Che. Every time when a method or class was refactored, the corresponding test method or class should adapt to the change of the method or class. • (RQ2) Do technical debt issues share some distinguishing characteristics compared to bugs or defects? The results of data analysis suggest that technical debt issues had no distinguishing characteristics in terms of open time and priority compared with bugs or defects and also different types of technical debt showed no consistent pattern in these two aspects as described in Section 6.2. The analysis results reject our hypothesis that technical debt issues may have lower priority and remain in the system longer than bugs or defects. Intuitively, bugs or defects are less tolerable than technical debt and thus need to be processed preferentially since bugs directly exert negative effects on the use of software and even cause the software system fails. Unfortunately, the analysis results do not accord with our preconceived notions of the characteristics of technical debt in open time and priority. However, different findings may be made when more issue data sets are analyzed in the future. • (RQ3) How do software practitioners communicate technical debt issues in issue trackers? ▪ (RQ3.1) Do software practitioners use technical debt as an issue type? The findings from our study show that none of the two issue data sets (i.e., Eclipse Che and Cornerstone) used technical debt as an issue type. All technical debt issues we identified are distributed among different issue types the software development team defined in issue trackers. ▪ (RQ3.2) Do software practitioners communicate technical debt explicitly or implicitly? 63 Although it is impossible to retrieve a technical debt issue directly according to its issue type, we can still acquire the evidence of whether the issue is a technical debt instance or not from the description of the issue. For Eclipse Che, both explicit indicators (e.g., debt and technical debt) and implicit indicators (e.g., workaround and decouple) were used to describe technical debt issues. However, no explicit indicators such as “债务” and “技术债务” appear in the whole Cornerstone issue data set. All technical debt issues were identified through implicit indicators in the description of issues for Cornerstone. The difference in the expression mode of technical debt between Eclipse and Cornerstone issue data set derive from software developers’ cognition about technical debt. We learned that many developers of Cornerstone project had little knowledge about technical debt through our communication with them. Even some developers had never heard of technical debt or “技术债务” (the corresponding Chinese term) although they were aware that technical debt indeed existed and they had ever incurred technical debt in the process of development after we explained to them what technical debt is like. ▪ (RQ3.3) Are there text patterns that may indicate technical debt items? We have demonstrated in Section 6.3 that a set of key phrases indicating technical debt have been extracted from both issue data sets respectively. These key phrases not only include ontological words in software engineering such as algorithm, interface, and codebase, but also involve common words like change, complicated and currently. These words comprise the English and Chinese vocabulary of technical debt respectively. In general, both vocabularies cover four dimensions of technical debt: time accumulation, rework action, quality attributes and problem scope. Time accumulation indicators like previously, now and future suggest that a technical debt issue has stayed or will stay in the software system for a period of time. Rework action refers to the specific action taken to pay off a technical debt issue, such as decoupling two modules, refactoring a method and simplifying a model. The words like usability, scalability, readability may indicate which quality attributes are jeopardized by a technical debt issue while problem scope signifies at which level (e.g., class, module, 64 requirement, and architecture) the health of the software is affected by the technical debt. A technical debt issue can be identified via these key phrases, especially when multiple words of different dimensions coexist in the description of issues. Furthermore, some key phrases extracted from these two issue data sets share the same semantic meaning although the issues are recorded in different languages. This suggests the similarities in the description of technical debt between English and Chinese. • (RQ4) Can we really identify non-code-level technical debt through issue trackers? We have presented that we can truly identify non-code-level technical debt such as architecture debt and requirement debt. These kinds of technical debt are not possible to be identified from source code using static source code analysis tools. Specifically, architecture debt issues are mainly those inconsiderate technological choices as a result of hindsight. We also found requirement debt issues that are incurred by implementing the functions that do not fully or accurately meet the requirements for various reasons. • (RQ5) Is it possible to apply NLP and machine learning techniques to identifying technical debt items through issue trackers? In Section 6.4, we have demonstrated the results of automatic identification of technical debt issues implemented based on different machine learning algorithms and feature extraction approaches. The results show that the performance of our model is much better than that of random labeling. However, the performance of our model needs to be improved further. We have confidence that better performance can be achieved when more sophisticated machine learning and NLP techniques (e.g., LSTM and word embedding) are employed on a wide range of issue-tracking data sets. 7.2 Threats to Validity In this study, we investigated the distribution, characteristics, and text patterns of technical debt in issue trackers from two software projects (i.e., Eclipse Che and Cornerstone). We also 65 explored an approach to identifying technical debt issues using NLP and machine learning techniques and evaluated the performance of our approach in these two cases. However, our study may not be absolutely accurate and sound since analysis results, evaluation approaches and even conclusions may be affected by validity threats. There are mainly two validity threats impacting the soundness of our study: internal validity and external validity, which are discussed in the following subsections thoroughly. 7.2.1 Internal Validity Internal validity refers to the extent to which experimental results or causal conclusions based on a study are warranted [45]. A threat to internal validity in our study comes from manual data tagging. A researcher may more or less have some subjectivity in the understanding of issue description and make some mistakes when tagging the issues. To counteract the bias in issue tagging, we have two researchers proficient in both English and Chinese review and tag the issues independently. For the discrepancy in the label of an issue, we exchanged our respective understanding of issue description, elaborated the grounds for tagging the issue as technical debt or not, and finally reached consensus by choosing the label that makes more sense. Furthermore, we also asked an expert in technical debt to validate our manual classification results. 7.2.2 External Validity External validity refers to the extent to which the findings of a study can be generalized to other situations [46]. In order to mitigate the threats to external validity, we selected two issue data sets in which issues are recorded in different languages (i.e., English and Chinese) from different types of software projects (i.e., commercial and open source). Both issue data sets include more than 8000 samples. 66 However, these two issue data sets may not be representative. That is to say, we cannot guarantee that the same results will be obtained when our approach is applied to other software projects. In particular, different phrases may be used to indicate technical debt in the issue data sets from other projects since every developer has his own language style in expressing technical debt. 67 Chapter 8 Conclusion The goal of this empirical multiple-case study is to investigate how technical debt is communicated in issue trackers and whether the description of technical debt issues conforms to some text patterns and also to explore an approach to identifying technical debt through issue trackers automatically. The findings of our study show that none of these two issue data sets uses technical debt as an issue type and most technical debt issues identified are indicated by implicit indicators except for the infrequent occurrence of the term technical debt in Eclipse Che issue data set. Text patterns in the description of technical debt issues indeed exist and also share some similarities in semantic meaning for both issue data sets. Furthermore, a prototype classification model was developed based on these text patterns to identify technical debt automatically using machine learning techniques. However, there are several limitations in this study; correspondingly, future work should be conducted to address them. Firstly, the number of issue data sets we analyzed is rather limited. This undermines the generalizability of our findings from this study. More issue data sets from software projects in different domains will be analyzed to gain more generalizable findings. Secondly, the prototype classification model we developed is a set of binary classifiers that only distinguish technical debt from other issues. A multiple classifier will be developed to identify different types of technical debt such as design debt, architecture debt, and usability debt. Finally, the performance of our classifier needs to be improved further. In the future, we will apply more sophisticated NLP and machine learning techniques to the automatic identification of technical debt through issue trackers. 68 Bibliography [1] Ke Dai and Philippe Kruchten. Detecting technical debt through issue trackers. In Proceedings of the 5th International Workshop on Quantitative Approaches to Software Quality co-located with 24th Asia-Pacific Software Engineering Conference (APSEC 2017), Nanjing, Jiangsu, China, December 4, 2017, pages 59-65, 2017. [2] Paris Avgeriou, Philippe Kruchten, Ipek Ozkaya, and Carolyn B. Seaman. Managing technical debt in software engineering (dagstuhl seminar 16162). Dagstuhl Reports, 6(4):110-138, 2016. [3] Ward Cunningham. The wycash portfolio management system. OOPS Messenger, 4(2):29-30, 1993. [4] Philippe Kruchten, Robert L. Nord, and Ipek Ozkaya. Technical debt: From metaphor to theory and practice. IEEE Software, 29(6):18-21,2012. [5] Steve McConnell. Managing technical debt. The Construx Whitepaper. https://www.construx.com/developer-resources/whitepaper-managing-technical-debt. [6] Martin Fowler. Technical debt quadrant. Martin Fowler’s Blog. https://martinfowler.com/bliki/TechnicalDebtQuadrant.html. [7] Martin Fowler. Refactoring - Improving the Design of Existing Code. Addison Wesley object technology series. Addison-Wesley, 1999. [8] Radu Marinescu. Detection strategies: Metrics-based rules for detecting design flaws. In 20th International Conference on Software Maintenance (ICSM 2004), 11-17 September 2004, Chicago, IL, USA, pages 350-359, 2004. 69 [9] Matthew James Munro. Product metrics for automatic identification of "bad smell" design problems in java source-code. In 11th IEEE International Symposium on Software Metrics (METRICS 2005), 19-22 September 2005, Como Italy, page 15, 2005. [10] Steffen M. Olbrich, Daniela Cruzes, Victor R. Basili, and Nico Zazworka. The evolution and impact of code smells: A case study of two open source systems. In Proceedings of the Third International Symposium on Empirical Software Engineering and Measurement, ESEM 2009, October 15-16, 2009, Lake Buena Vista, Florida, USA, pages 390-400, 2009. [11] Sunny Wong, Yuanfang Cai, Miryung Kim, and Michael Dalton. Detecting software modularity violations. In Proceedings of the 33rd International Conference on Software Engineering, ICSE 2011, Waikiki, Honolulu, HI, USA, May 21-28, 2011, pages 411-420, 2011. [12] Everton da S. Maldonado, Emad Shihab, and Nikolaos Tsantalis. Using natural language processing to automatically detect self-admitted technical debt. IEEE Trans. Software Eng., 43(11):1044-1062, 2017. [13] John Brondum and Liming Zhu. Visualising architectural dependencies. In Proceedings of the Third International Workshop on Managing Technical Debt, MTD 2012, Zurich, Switzerland, June 5, 2012, pages 7-14, 2012. [14] Zengyang Li, Peng Liang, Paris Avgeriou, Nicolas Guelfi, and Apostolos Ampatzoglou. An empirical investigation of modularity metrics for indicating architectural technical debt. In QoSA'14, Proceedings of the 10th International ACM SIGSOFT Conference on Quality of Software Architectures (part of CompArch 2014), Marcq-en-Baroeul, Lille, France, June 30-July 04, 2014, pages 119-128, 2014. [15] Zengyang Li, Peng Liang, and Paris Avgeriou. Architectural technical debt identification based on architecture decisions and change scenarios. In 12th Working IEEE/IFIP Conference on Software Architecture, WICSA 2015, Montreal, QC, Canada, May 4-8, 2015, pages 65-74, 2015. 70 [16] Stephany Bellomo, Robert L. Nord, Ipek Ozkaya, and Mary Popeck. Got technical debt?: surfacing elusive technical debt in issue trackers. In Proceedings of the 13th International Conference on Mining Software Repositories, MSR 2016, Austin, TX, USA, May 14-22, 2016, pages 327-338, 2016. [17] Giuliano Antoniol, Kamel Ayari, Massimiliano Di Penta, Foutse Khomh, and Yann-Gaël Guéhéneuc. Is it a bug or an enhancement?: a text-based approach to classify change requests. In Proceedings of the 2008 conference of the Centre for Advanced Studies on Collaborative Research, October 27-30, 2008, Richmond Hill, Ontario, Canada, page 23, 2008. [18] Per Runeson, Magnus Alexandersson, and Oskar Nyholm. Detection of duplicate defect reports using natural language processing. In 29th International Conference on Software Engineering (ICSE 2007), Minneapolis, MN, USA, May 20-26, 2007, pages 499-510, 2007. [19] Xiaoyin Wang, Lu Zhang, Tao Xie, John Anvik, and Jiasu Sun. An approach to detecting duplicate bug reports using natural language and execution information. In 30th International Conference on Software Engineering (ICSE 2008), Leipzig, Germany, May 10-18, 2008, pages 461-470, 2008. [20] Nicholas Jalbert and Westley Weimer. Automated duplicate detection for bug tracking systems. In The 38th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2008, June 24-27, 2008, Anchorage, Alaska, USA, Proceedings, pages 52-61, 2008. [21] Chengnian Sun, David Lo, Xiaoyin Wang, Jing Jiang, and Siau-Cheng Khoo. A discriminative model approach for accurate duplicate bug report retrieval. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1, ICSE 2010, Cape Town, South Africa, 1-8 May 2010, pages 45-54, 2010. 71 [22] Ashish Sureka and Pankaj Jalote. Detecting duplicate bug report using character n-gram-based features. In 17th Asia Pacific Software Engineering Conference, APSEC 2010, Sydney, Australia, November 30 - December 3, 2010, pages 366-374, 2010. [23] Constantine Aaron Cois and Rick Kazman. Natural language processing to quantify security effort in the software development lifecycle. In The 27th International Conference on Software Engineering and Knowledge Engineering, SEKE 2015, Wyndham Pittsburgh University Center, Pittsburgh, PA, USA, July 6-8, 2015, pages 716-721, 2015. [24] Nicolli S. R. Alves, Thiago Souto Mendes, Manoel Gomes de Mendonça Neto, Rodrigo O. Spínola, Forrest Shull, and Carolyn B. Seaman. Identification and management of technical debt: A systematic mapping study. Information and Software Technology, 70:100-121, 2016. [25] Wikipedia contributors. Text segmentation. Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=Text_segmentation&oldid=846307005, 2018 [26] J Sun. “jieba” Chinese word segmentation tool. https://github.com/fxsjy/jieba. [27] Wikipedia contributors. Stemming. Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=Stemming&oldid=849076995, 2018. [28] Wikipedia contributors. Stop words. Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=Stop_words&oldid=865611548, 2018. [29] C. J. van Rijsbergen. Information Retrieval. Butterworth, 1979. [30] Christopher J. Fox. A stop list for general text. SIGIR Forum, 24(1-2):19-35, 1990. [31] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake VanderPlas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and 72 Edouard Duchesnay. Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12:2825-2830, 2011. [32] Wikipedia contributors. Bag-of-words model. Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=Bag-of-words_model&oldid=863868725, 2018. [33] Gerard Salton, A. Wong, and Chung-Shu Yang. A vector space model for automatic indexing. Commun. ACM, 18(11):613-620, 1975. [34] Jure Leskovec, Anand Rajaraman, and Jeffrey D. Ullman. Mining of Massive Datasets, 2nd Ed. Cambridge University Press, 2014. [35] Manoranjan Dash and Huan Liu. Feature selection for classification. Intell. Data Anal., 1(1-4):131-156, 1997. [36] Wikipedia contributors. Chi-squared test. Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=Chi-squared_test&oldid=863272322, 2018. [37] Karl Pearson. X. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 50(302):157-175, 1900. [38] Wikipedia contributors. Regularization (mathematics). Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=Regularization_(mathematics)&oldid=861757490, 2018. [39] Harry Zhang. The optimality of naive bayes. In Proceedings of the Seventeenth International Florida Artificial Intelligence Research Society Conference, Miami Beach, Florida, USA, pages 562-567, 2004. 73 [40] Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine Learning, 20(3):273-297, 1995. [41] Tin Kam Ho. Random decision forests. In Third International Conference on Document Analysis and Recognition, ICDAR 1995, August 14 - 15, 1995, Montreal, Canada. Volume I, pages 278-282, 1995. [42] Wikipedia contributors. Cross-validation (statistics). Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=Cross-validation_(statistics)&oldid=863233097, 2018. [43] Wikipedia contributors. Precision and recall. Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=Precision_and_recall&oldid=853202943, 2018. [44] Wikipedia contributors. Receiver operating characteristic. Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=Receiver_operating_characteristic&oldid=864407292, 2018. [45] Wikipedia contributors. Internal validity. Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=Internal_validity&oldid=863479569, 2018. [46] Wikipedia contributors. External validity. Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=External_validity&oldid=864210353, 2018. 74 Appendices Appendix A The Screenshot of Issue Spreadsheet of Eclipse Che 75 Appendix B The Screenshot of Issue Spreadsheet of Cornerstone 76 Appendix C The Translation of Key Phrases for Cornerstone Chinese Phrases Translation Chinese Phrases Translation 修改 change 归并 merge 修正 correct 解耦 decouple 修复 fix 拆分 split 更改 change 调整 adjust 整改 change 适应 adapt 改动 change 扩展 extend 改成 change 维护 maintain 改为 change 限制 limit 取代 replace 影响 affect 替换 replace 目前 currently 改进 improve 当前 currently 改善 improve 现在 now 优化 optimize 现有 current 简化 simplify 前期 previously 删除 delete 过去 in the past 取消 cancel 原有 original 完善 perfect 将来 future 提高 improve 今后 future 增强 enhance 实际 actual 重构 refactor 现实 reality 移植 transplant 功能 function 整理 clean up 需求 requirement 整合 clean up and merge 设计 design 合并 merge 理论 theory 77 机制 mechanism 通用性 universality 算法 algorithm 实时性 real time 逻辑 logic 局限性 limitation 架构 architecture 有效性 effectiveness 结构 structure 一致性 consistency 代码 code 充分 sufficient 性能 performance 方便 convenient 效率 efficiency 灵活 flexible 用户 user 简单 simple 客户 client 直观 intuitionistic 体验 experience 合理 reasonable 操作 operation 晦涩 difficult to understand 习惯 habit 难懂 difficult to understand 风格 style 复杂 complicated 样式 style 重复 duplicate 格式 format 不好 not good 困难 difficulty 不正确 incorrect 并发 concurrency 不正常 abnormal 可用性 usability 不一致 inconsistent 易用性 usability 不合理 unreasonable 可读性 readability 不方便 inconvenient 易读性 readability 不清晰 unclear 可扩展性 extensibility 不准确 inaccurate 可维护性 maintainability 不直观 not intuitionistic 安全性 security 不美观 not pretty 兼容性 compatibility 不协调 incongruous 78 稳定性 stability 不流畅 not smooth 不符合 do not conform to 代码移植 code transplantation 不友好 unfriendly 代码整理 code cleanup 不容易 uneasy 框架修改 framework modification 数据结构 data structure UI修改 UI modification 代码优化 code optimization 统一风格 unify the style 接口简化 interface simplification 保持一致 keep consistent 代码修改 code modification 并不需要 do not need 优化方案 optimize the strategy 调整布局 adjust the layout 功能模块 functional module 代码合并 code merge 功能设计 function design 易用性修改 usability modification 架构设计 architecture design 功能未完善 The function is imperfect 现有设计 current design 不方便操作 inconvenient to operate 用户体验 user experience 功能需要完善 The function needs optimization 界面显示 UI display 统一页面风格 unify the page style 设计不合理 unreasonable design 协议识别优化 protocol identification optimization 统一整理 unify the cleanup 完善单元测试 improve unit test 移植 MW transplant the MW 代码需要修改 Code needs modification 机制改进 mechanism improvement 架构初步优化 initial architecture optimization 需要完善 need perfecting
Thesis/Dissertation
eng
Electrical and Computer Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Identifying technical debt through issue trackers
Text