Science, Irving K. Barber Faculty of (Okanagan)
Computer Science, Mathematics, Physics and Statistics, Department of (Okanagan)
DSpace
UBCO
Hadi, Mohammad A
Master of Science - MSc
University of British Columbia
Software Engineering (SE) related natural language texts, such as app reviews and crowdsourced Questions-Answers (Q&A) play pivotal roles for software engineers and developers to gain knowledge regarding different stages of software life-cycle, such as application development, deployment, and maintenance. Successful classification and clustering of these texts can help developers and engineers quickly understand and process the bulk of information in the most effective way. Therefore, this thesis focuses on the efficient classification and clustering of SE-related texts using state-of-the-art neural language models and adaptive topic modeling techniques, respectively. An extensive empirical study is performed to understand the strength, effectiveness, and competence of Pre-trained Transformer based neural language Models (PTM) for the app review classifications task, which also identified the best-performing PTMs. Two of the best-performing PTM models have also been pre-trained from scratch on domain-specific data to yield better classification performance. The largest domain-specific app review dataset has been scraped from Google Play Store for this pretraining purpose. For SE texts clustering purposes, a new online adaptive Topic Model, Adaptive Online Bi-term Topic Model (AOBTM) has been proposed that can efficiently identify topics from corpora sliced over different time and version slices. This topic model leverages and adapts the statistical data inferred in the preceding slices to infer latent topics from the latest slice. The approach yields good results for the short and noisy SE-related natural language texts.
On the Understanding of SoftwareEngineering Related Texts via theTransfer of Prior KnowledgebyMohammad A HadiB.Sc. Hons., North South University, 2018A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF SCIENCEinTHE COLLEGE OF GRADUATE STUDIES(Computer Science)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)February 2023© Mohammad A Hadi, 2023The following individuals certify that they have read, and recommendto the College of Graduate Studies for acceptance, a thesis entitled:On the Understanding of Software Engineering Related Textsvia the Transfer of Prior Knowledgesubmitted byMohammad A Hadi in partial fulfilment of the requirementsof the degree of Master of Science.Examining Committee:Dr. Fatemeh H FardIrving K. Barber Faculty of ScienceSupervisorDr. Yong GaoIrving K. Barber Faculty of ScienceSupervisory Committee MemberDr. Zheng LiuIrving K. Barber Faculty of ScienceSupervisory Committee MemberDr. Anas ChaabanFaculty of Applied ScienceExternal ExamineriiAbstractSoftware Engineering (SE) related natural language texts, such as appreviews and crowdsourced Questions-Answers (Q&A) play pivotal roles forsoftware engineers and developers to gain knowledge regarding differentstages of software life-cycle, such as application development, deployment,and maintenance. Successful classification and clustering of these texts canhelp developers and engineers quickly understand and process the bulk ofinformation in the most effective way. Therefore, this thesis focuses onthe efficient classification and clustering of SE-related texts using state-of-the-art neural language models and adaptive topic modeling techniques,respectively. An extensive empirical study is performed to understand thestrength, effectiveness, and competence of Pre-trained Transformer basedneural language Models (PTM) for the app review classifications task, whichalso identified the best-performing PTMs. Two of the best-performing PTMmodels have also been pre-trained from scratch on domain-specific data toyield better classification performance. The largest domain-specific app re-view dataset has been scraped from Google Play Store for this pretrainingpurpose. For SE texts clustering purposes, a new online adaptive TopicModel, Adaptive Online Bi-term Topic Model (AOBTM) has been proposedthat can efficiently identify topics from corpora sliced over different time andversion slices. This topic model leverages and adapts the statistical data in-ferred in the preceding slices to infer latent topics from the latest slice.The approach yields good results for the short and noisy SE-related naturallanguage texts.iiiLay SummaryThe exploration of Software Engineering related natural language textshas many advantages; however, there are many challenges due to its shortlength, noisy nature, dependence on software versions, and various classi-fication and clustering purposes intended by developers. A large portionof Software Engineering-related natural language texts originate from mo-bile app reviews and questions answered on crowd-sourced platforms, butno study has been conducted to leverage pre-trained knowledge to classifyand cluster these texts more efficiently. In order to fill the gap in the cur-rent studies, an empirical study was conducted to assess the role of thepre-trained models for app issue classification. In the next part, the focuswas on the review changes based on the released versions of the apps, whichled to the development of a new topic modeling algorithm, Adaptive OnlineBiterm Topic Modeling (AOBTM).ivPrefaceThis thesis, titled “On the Understanding of Software Engineering Re-lated Texts via the Transfer of Prior Knowledge” presents an empirical studyon the pre-trained language models for effective issue classification, alongwith an online topic model for identifying emerging and trending topics fromany large corpus of Software texts. The work done in this thesis has beenwritten and developed in collaboration with my supervisor Dr. FatemehHendijani Fard.The empirical study presented in this thesis in Chapter 4 is currentlyunder review at the peer-reviewed journal ‘Springer’s Empirical SoftwareEngineering with submission ID ‘EMSE-D-22-00080’. The ideas of all thementioned papers originated from me and were directed by my supervisor.I have written all the scripts and conducted all experiments of these studies(except the mentioned ones). Also, during my Master’s, I got the opportu-nity to work on multiple papers with my supervisor and colleagues, someof which are currently under review or some are works in progress. Thesepapers can be classified into related and unrelated to my thesis which islisted as follows:- M. A. Hadi and F. H. Fard, “AOBTM: Adaptive Online Biterm TopicModeling for Version Sensitive Short-texts Analysis,” 2020 IEEE In-ternational Conference on Software Maintenance and Evolution (IC-SME), 2020, pp. 593-604, doi: 10.1109/ICSME46990.2020.00062. (Re-lated, Published [HF20a])- Hadi, M.A., & Fard, F.H. (2021). “Evaluating Pre-Trained Models forUser Feedback Analysis in Software Engineering: A Study on Classifi-cation of App-Reviews.” ArXiv, abs/2104.05861. (Related, Submittedand under review [HF21])- Luong, K.G., Hadi, M.A., Thung, F., Fard, F., & Lo, D. (2021). “FA-COS: Finding API Relevant Contents on Stack Overflow with Seman-tic and Syntactic Analysis.” ArXiv, abs/2111.07238. (Unrelated, Pub-lished [LHT+21])v- Hadi, M.A., Yusuf, I.N., Thung, F., Luong, K.G., Jiang, L., Fard,F.H., & Lo, D. (2022). “On the Effectiveness of Pre-trained Modelsfor API Learning.” ArXiv, abs/2204.03498. (Unrelated, Published[HYT+22])viTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . viiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . xiiDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiChapter 1: Introduction . . . . . . . . . . . . . . . . . . . . . . . 11.1 Overview and Motivation . . . . . . . . . . . . . . . . . . . . 11.2 Research Gaps . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 Research Objective . . . . . . . . . . . . . . . . . . . . . . . . 31.4 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.5 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . 51.6 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . 6Chapter 2: Related Work . . . . . . . . . . . . . . . . . . . . . . 72.1 Related Work to the Software Engineering Text Classification 72.1.1 App Review Classification in Software Engineering . . 72.1.2 Pre-trained Models in Text Classification . . . . . . . 92.2 Related Work to the Software Engineering Topic Modeling . . 102.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11Chapter 3: Background Knowledge . . . . . . . . . . . . . . . . 12viiTABLE OF CONTENTS3.1 Deep Neural Network with Word Embedding . . . . . . . . . 123.2 Overview of a Transformer Architecture . . . . . . . . . . . . 143.3 Pre-Trained Transformer-based Models . . . . . . . . . . . . . 163.4 Overview of Topic Modeling . . . . . . . . . . . . . . . . . . . 173.5 Biterm Topic Model (BTM) . . . . . . . . . . . . . . . . . . . 183.5.1 Comparison of BTM with Other Topic Models . . . . 183.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19Chapter 4: Assisting Developers to Better Classify User Feed-back with Pre-Trained Transformer-based Models 204.1 Motivation and Objective . . . . . . . . . . . . . . . . . . . . 224.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . 234.2.1 Research Questions . . . . . . . . . . . . . . . . . . . . 244.2.2 Research Overview . . . . . . . . . . . . . . . . . . . . 254.2.3 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . 264.2.4 Prior Approaches . . . . . . . . . . . . . . . . . . . . . 304.2.5 Pre-Trained Models (PTM) . . . . . . . . . . . . . . . 334.2.6 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . 364.3 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . 384.3.1 Prior Approaches’ Implementation . . . . . . . . . . . 384.3.2 Model Training Setup . . . . . . . . . . . . . . . . . . 394.3.3 Approaches to Answer the Research Questions . . . . 414.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464.4.1 RQ1: Accuracy and Efficiency of PTMs Compared toPrior approaches . . . . . . . . . . . . . . . . . . . . . 464.4.2 RQ2: Domain-Specific PTMs vs. General PTMs . . . 494.4.3 RQ3: Experimenting PTMs in Different Settings . . . 504.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55Chapter 5: Adaptive Online Bi-term Topic Model . . . . . . . 575.1 Motivation and Objective . . . . . . . . . . . . . . . . . . . . 605.1.1 Topic Modeling for Conventional Text Documents . . 605.1.2 Topic Modeling for Short Text Documents . . . . . . . 605.1.3 Online Topic Modeling for Short Texts . . . . . . . . . 615.1.4 Adaptively Online Topic Modeling for Short Texts . . 625.1.5 Objective . . . . . . . . . . . . . . . . . . . . . . . . . 625.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . 635.2.1 Applied Biterm Extraction Technique . . . . . . . . . 635.2.2 Topic Model Description . . . . . . . . . . . . . . . . . 645.2.3 AOBTM Complexity and Comparison with Baselines . 67viiiTABLE OF CONTENTS5.2.4 Algorithms To Find Optimal Number Of Topics AndVersion-Window . . . . . . . . . . . . . . . . . . . . . 695.3 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . 745.3.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . 745.3.2 Baselines . . . . . . . . . . . . . . . . . . . . . . . . . 755.3.3 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . 755.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 775.4.1 RQ1: Comparison Results with Different Methods . . 785.4.2 RQ2: Effect of Different Parameter Settings, Docu-ment Lengths, and Preprocessing Approaches . . . . . 805.4.3 RQ3: Quality of Discovered Topics Using ParametersDetermined by Proposed Algorithms . . . . . . . . . . 835.4.4 Real-World Application . . . . . . . . . . . . . . . . . 845.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86Chapter 6: Discussion . . . . . . . . . . . . . . . . . . . . . . . . 876.1 Discussion on the Software Engineering Text Classification . . 876.1.1 Implications For Developers and Practitioners . . . . . 876.1.2 Implications For Researchers . . . . . . . . . . . . . . 906.2 Discussion on Software Engineering Topic Modeling . . . . . 916.2.1 Performance of AOBTM . . . . . . . . . . . . . . . . . 916.2.2 Guidelines for the Users . . . . . . . . . . . . . . . . . 926.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92Chapter 7: Conclusion . . . . . . . . . . . . . . . . . . . . . . . . 937.1 Threats to Validity . . . . . . . . . . . . . . . . . . . . . . . . 947.1.1 Study on Assisting Developers to Better Classify UserFeedback . . . . . . . . . . . . . . . . . . . . . . . . . 947.1.2 Study on Adaptive Online Bi-term Topic Model . . . 967.2 Future Works . . . . . . . . . . . . . . . . . . . . . . . . . . . 977.2.1 Study on Assisting Developers to Better Classify UserFeedback . . . . . . . . . . . . . . . . . . . . . . . . . 977.2.2 Study on Adaptive Online Bi-term Topic Model . . . 98Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125Appendix A: Tables . . . . . . . . . . . . . . . . . . . . . . . . . . 126ixList of TablesTable 4.1 Description for Merged Dataset (D7) . . . . . . . . . . 29Table 4.2 Details of PTMs Used in the Study . . . . . . . . . . . 36Table 5.1 Time Complexities and the Number of In-Memory Vari-ables in Different Topic Models . . . . . . . . . . . . . 68Table 5.2 Subject apps from different app-stores . . . . . . . . . 75Table 5.3 Comparison result of different methods . . . . . . . . . 79Table 5.4 Five most contributing terms in a topic from Tweets2020 83Table 5.5 Time Complexities for Proposed Parallel Algorithms . 83Table 5.6 Five most contributing terms from two sample topics . 85Table 5.7 PMI-Scores & Time Cost evaluation for Zoom app-reviews . . . . . . . . . . . . . . . . . . . . . . . . . . . 85Table A.1 RQ1 results for Prior approaches. The best scores foreach dataset are shown in bold. . . . . . . . . . . . . . 126Table A.2 RQ1 results for PTMs. The best scores for each datasetare shown in bold. . . . . . . . . . . . . . . . . . . . . 127Table A.3 RQ2 results for C-PTM-X1 (PTM-X1 = ALBERT).The best scores for each dataset are bold. . . . . . . . 128Table A.4 RQ2 results for C-PTM-X2 (PTM-X2 = RoBERTa).The best scores are bold. . . . . . . . . . . . . . . . . . 129Table A.5 RQ3 results for Binary Classifications (Part i) . . . . . 130Table A.6 RQ3 results for Binary Classifications (Part ii) . . . . 131Table A.7 RQ3 results for Zero-Shot Classification . . . . . . . . 132Table A.8 RQ3 results for Multi-Task (Category) Classification . 133Table A.9 RQ3 results for Multi-Task (Sentiment) Classification . 133Table A.10 RQ3 results for Multi-Resource Classification (Trainedon App Reviews and Tested on Twitter) . . . . . . . . 134Table A.11 RQ3 results for Multi-Resource Classification (Trainedon Twitter and Tested on App Reviews) . . . . . . . . 134xList of FiguresFigure 3.1 Neural Network Architecture (CNN) with Word Em-bedding . . . . . . . . . . . . . . . . . . . . . . . . . . 13Figure 3.2 High Level Look of Transformer Model architecture[Ala18] . . . . . . . . . . . . . . . . . . . . . . . . . . 14Figure 3.3 Building Modules of Encoder and Decoder [Ala18] . . 15Figure 4.1 Overview of Research Steps for RQ1 . . . . . . . . . . 26Figure 4.2 Overview of Research Steps for RQ2 (left) and RQ3(right) . . . . . . . . . . . . . . . . . . . . . . . . . . 26Figure 4.3 Creating Up-sampled dataset for PTM pre-trainingtasks . . . . . . . . . . . . . . . . . . . . . . . . . . . 42Figure 4.4 PTM and Prior Performance (micro-F1 score) . . . . 47Figure 4.5 PTM and Prior Prediction Time (seconds) . . . . . . 48Figure 4.6 Performance of Custom PTMs (micro F1 score) . . . 49Figure 5.1 Overview of the AOBTM Framework . . . . . . . . . 62Figure 5.2 Overview of AOBTM . . . . . . . . . . . . . . . . . . 64Figure 5.3 Two Phases of Algorithm 2, , determining the optimaltopic number . . . . . . . . . . . . . . . . . . . . . . . 73Figure 5.4 PMI-Scores for the varying number of considered ver-sions or time-slices. . . . . . . . . . . . . . . . . . . . 81Figure 5.5 PMI-Scores for the varying number of topics. . . . . . 81Figure 5.6 PMI-Scores for varying document lengths. . . . . . . 82Figure 5.7 PMI-Scores generated by setting the parameters de-termined by Algorithms 2 and 3 . . . . . . . . . . . . 84xiAcknowledgementsI want to express my appreciation and gratitude to my supervisor, Dr.Fatemeh H. Fard. She was always the most incredible support in my aca-demic life and a true inspiration in my personal life. Over the years, I havelearned a lot from her, especially how to stay persistent and vigilant towardthe end goal. Her support, knowledge, mentorship, and advice assisted mein finishing my research. I feel privileged to be one of her MS students andwill always value her guidance.I am grateful to Dr. Yong Gao and Dr. Zheng Liu for providing supportand helpful feedback on my work during my annual meetings. I would alsolike to thank the members of my defense committee for their valuable time.I want to thank Dr. David Lo, who supervised me during my four monthsof internship at Singapore Management University, provided insightful sug-gestions about my work, and helped me conceptually understand some cru-cial parts of my study designs.I finally would like to thank my friends who supported me abroad to geton my feet and my family in my country, who have endured many miseriesdue to my absence. They have encouraged me despite their tribulations andhelped me continue studying. Without their support and patience, I wouldnot have overcome my depression and finished my work.xiiDedicationÑ ê  P' á Ò ê  P è é Ê Ò   In the name of Allah, the Most Gracious, the Most Merciful.For my mother, aunts, grandma, and wife - my wonder women!I am eternally grateful for their love, kindness, care, prayers, support,guidance, patience, and faith in me!ZÉ Ë È èX ¬  à@ ñ Ë@ ð Õ º J  Ë @ É K Z é » @ ð PB ð Õæ  Ë Ê ê » á Ó ðAnd one of His signs is the creation of the heavens and the earth, and thediversity of your languages and colours. Surely in this are signs for thoseof sound knowledge. [Al-Qura´n 30:22]xiiiChapter 1IntroductionIn this chapter, the overview and motivation of this thesis will be dis-cussed along with the gaps in the current literature review, the thesis ob-jective, author’s contribution, and the research questions that are discussedelaborately in the following chapters.1.1 Overview and MotivationSoftware Engineering (SE) related natural language texts are great re-sources for software engineers for learning about software, collecting software-related feedback, and analyzing them to identify or update requirements ondaily basis. One of the most popular software engineering-related naturallanguage texts is app reviews as these reviews of mobile apps on App Storesand on social media are valuable resources for app developers. Classifyingapp reviews have proved to be useful for many areas of software engineering(e.g., requirement engineering, testing, etc.). Existing approaches rely onthe manual curating of a labeled dataset to classify app reviews automati-cally. In practice, new datasets must be labeled as the classification purposechanges (e.g., identifying bugs versus usability issues or sentiment), limitingthe power of developed models to classify new classes or issues. Employingmodels that can produce the same or better result using less labeled data caneliminate the problem by reducing this manual effort. Recent Pre-trainedTransformer based Models (PTM) are trained on large natural language cor-pora in an unsupervised manner to retain contextual knowledge about thelanguage and have found success in solving similar Natural Language Pro-cessing (NLP) problems. However, the applicability of PTMs has not yetbeen explored for issue classification from app reviews. This study aims toapply PTMs to understand their capabilities and limitations for classifyingapp reviews.Another issue that this study addresses, is identify the topics in the appreviews. Although the classification of mobile app reviews has shown itsimportant role in software maintenance and the evolution of mobile apps, itdoes not help much if the latent topics or classes are not known to the devel-11.2. RESEARCH GAPSopers and engineers. Mobile app developers have to check the app reviewsfrequently to clarify the issues experienced by users and to capture the newlatent issues that are introduced due to a recent app update. As the appreviews have a dynamic nature and their discussed topics change over time,the changes in the topics among collected reviews for different versions ofan app can reveal important issues about the app update. Topic modelingalgorithms can solve this problem rather elegantly for long texts. However,app reviews are short and it is challenging to unveil their latent topics overtime using the conventional topic modeling approaches. Conventional topicmodels suffer from the sparsity of word co-occurrence patterns while infer-ring topics for short texts. Furthermore, these algorithms cannot capturetopics over numerous consecutive time-slices. Online topic modeling algo-rithms speed up the inference of topic models for the texts collected in thelatest time-slice by saving a fraction of data from the preceding time-slice.But these algorithms do not analyze the statistical data of all the precedingtime-slices, which can confer contributions to the topic distribution of thecurrent time-slice. Thus, this study also proposes a novel adaptive onlinetopic model to find the latent topics from the short app review dataset.1.2 Research GapsThe research gaps that this thesis aims to resolve are outlined here.These gaps or deficits were identified by an extensive literature review.− The applicability of PTMs has not yet been explored by any studyfor issue classification from app reviews. It would be interesting toexamine PTMs’ performance and limitations in the field of SE relatedNLP text, such as app reviews classification.− The performance of PTMs has not yet been examined when they arepre-trained on the domain-specific datasets, in this case, app reviews.This study will aim to comprehend how to improve PTMs’ perfor-mance by pre-training them with focused and enriched data.− PTM performance has not yet been analyzed when they operate indifferent settings, such as zero-shot classification, multi-resource, andmulti-task settings. These settings can shed further lights into theapplicability and generalizability of these PTMs.− There is no adaptive online topic modeling algorithm that works wellwith short texts. As the app reviews are generated frequently and in21.3. RESEARCH OBJECTIVElarge volume, this type of algorithm can help developers to understandthe latent topics quickly and concisely.1.3 Research ObjectiveClassification methods presently used for app review classification re-quire manual curating of labeled datasets, which must be re-labeled as clas-sification purposes change, limiting the ability of the models to classify newissues or classes. The manual labor associated with labeling data can bereduced using models that produce the same results with less labeled data.Using extensive natural language corpora, recent PTMs have been found tobe successful in solving similar NLP challenges. Although, it has not yetbeen explored whether PTMs can be used for issue classification from appreviews.The first part of the study aims to explore the advantages of PTMs forapp review classification tasks by comparing them with existing models; itwill also examine the transferability of PTMs by applying them in multiplesettings. Multiple datasets from the literature will be selected, which willcontain manually labeled app reviews from Google Play Store, Apple AppStore, and Twitter data. This thesis will empirically study the performanceand time efficiency of PTMs compared to Prior approaches. In addition, itwill evaluate the performance of the PTMs when app reviews are incorpo-rated into their pre-training (i.e., domain-specific PTMs). Different studiesmust be set up to evaluate PTMs in multiple settings: binary vs. multi-class classification, zero-shot classification (when new labels are introducedto the model), multi-task setting, and classification of reviews from differentresources. In all cases, Micro and Macro Precision, Recall, and F1 scoreswill be used and the time required for training and prediction with themodels will also be reported. The supposition is that the results will showthat PTMs can classify the app issue with higher scores. However, in themulti-resource setting, the Ensemble method and Deep learning with wordembedding could score higher than PTMs. When PTMs are pre-trainedwith app-specific reviews (i.e. Custom PTMs), the models may achieve thehighest scores among all in all settings, and the prediction times may alsobe reduced; with the highest scores belonging to customized PTMs that arepre-trained with a larger number of app reviews.The second part of the study will focus on the latent topics in app re-views that are challenging to uncover over time because they are short andnoisy texts. Due to sparse word co-occurrence patterns, it is tough to infer31.4. CONTRIBUTIONStopics from short texts using conventional topic models. In addition, thesealgorithms cannot capture topics across multiple consecutive time-slices. Bysaving data from the preceding time-slice, online topic modeling algorithmscan solve the problem and infer topic models more quickly for the texts col-lected in the latest time-slice. However, none of these algorithms analyze thestatistical data from all preceding time-slices, which can contribute to thetopic distribution in the current slice. Hence, this study will propose Adap-tive Online Biterm Topic Model (AOBTM) to model topics in short textsadaptively. AOBTM will alleviate the sparsity problem in short-texts andconsiders the statistical-data for an optimal number of preceding time-slices.It will also incorporate two parallel algorithms to automatically determinethe optimal number of topics and the best number of preceding versions thatshould be considered in the topic inference phase. Automatic evaluation ofcollections of app reviews and real-world short text datasets will be usedto confirm that AOBTM can find more coherent topics and outperform thestate-of-the-art baselines.1.4 ContributionsThe contributions of this thesis are as follows:− The largest Google App Review Dataset to date has been collectedto train Pre-trained language models for various app review analysis-related purposes.− As the first study exploring the applicability of PTMs for automaticapp review classification tasks compared to the existing tools, it canhelp developers and researchers build classification tools quickly andefficiently by adopting pre-trained models that are suitable for theiruse cases.− An extensive comparison between four PTMs and four existing tools/approaches on six different app review datasets with different sizesand labels will help the researchers navigate this avenue of app reviewanalysis in different settings with more certainty and confidence.− This is the first study to explore the performance of general versusdomain-specific pre-trained PTMs for app review classification.− This thesis includes the first empirical study to examine the accuracyand efficiency of PTMs in four different settings: binary vs. multi-class classification, zero-shot setting, multi-task setting, and setting41.5. RESEARCH QUESTIONSin which training data is from one resource (e.g., App Store) and themodel is tested on data from another platform (e.g., Twitter).− A novel method called AOBTM for version-sensitive content analysisfor short texts. This method adaptively combines the topic distribu-tions of a selected number of prior versions to generate topic distribu-tions of the current version.− Two parallel algorithms have been proposed to determine two param-eters of AOBTM; the first algorithm can identify an optimal numberof topics to be derived in the latest version, and the second algorithmcan identify the optimal number of preceding versions to be taken intoconsideration for adaptive aggregation of statistical data.− To encourage replicability of the research results, all scripts, codes,and datasets are made available to the community 1 2.1.5 Research QuestionsThe research questions that are answered in this thesis are listed below.The first three research questions are related to Chapter 4 and the last threeresearch questions are discussed in Chapter 5.RQ1 How accurate and efficient are the Pre-Trained Models (PTM) for appreview classification compared to the existing tools? By answeringthis research question, we will understand if the PTMs can be appliedfor SE text classification or not. The PTMs’ performance will betested on six datasets collected from the literature and the results willbe reported using micro and macro Precision, Recall, and F1-score.(Discussed elaborately in Chapter 4)RQ2 How does the performance of the PTMs change when they are pre-trained on an app-review dataset? By answering this research ques-tion, we will understand the capabilities of the PTMs. The PTMswill be enhanced in their pre-training stage using a large domain spe-cific dataset to observe their performance improvement. (Discussedelaborately in Chapter 4)1https://github.com/fardfh-lab/emse-hadi-august-20222https://github.com/Mohammad-Abdul-Hadi/AOBTM-Adaptive-Online-Biterm-Topic-Modeling51.6. THESIS OUTLINERQ3 How do the PTMs perform in the binary, zero-shot, multi-task, andmulti-resource settings? By answering this research question, we willunderstand the limitations of the PTMs. All the PTMs, includingthe enhanced (with domain specific datasets) PTMs will be testedwithin different settings to understand the limitations and capabilities.(Discussed elaborately in Chapter 4)RQ4 Can the Adaptive Online Biterm Topic Model (AOBTM) achieve bet-ter performance compared to other topic modeling methods? Theproposed model is tested against the state-of-the-art topic models tounderstand its performance for short and noisy texts, such as appreviews. (Discussed elaborately in Chapter 5)RQ5 How do different parameter settings, document lengths, and pre pro-cessing approaches impact the performance of AOBTM? This researchquestion will shed light on how to tune and apply the model in differentsettings. (Discussed elaborately in Chapter 5)RQ6 Using the parameters set by the proposed parallel algorithms, howdiscriminative and coherent are the topics discovered by different topicmodeling methods? The results of this research question will show howeffective the proposed algorithms are for the use of the adaptive onlinetopic model. (Discussed elaborately in Chapter 5)1.6 Thesis OutlineThe rest of the thesis is organized as follows: the related studies fromthe literature are included in Chapter 2. In Chapter 3, the backgroundknowledge are incorporated, which are relevant to the following chapters.Chapter 4 discusses the Empirical Study done on the Pre-trained LanguageModels for the classification of app reviews, which supplies the most domi-nant portion for software engineering-related texts. Chapter 5 discusses theproposed topic modeling algorithm, AOBTM. The outcomes and limitationsare discussed in Chapter 6. Finally, the thesis is concluded in Chapter 7.Appendix - A includes 11 long and content-rich tables related to Chapter 4.6Chapter 2Related WorkThis chapter discusses the related work that were studied during theextensive literature review. These works paved the way for this thesis andinspired many aspects of this study. For the sake of comprehensibility andhigh cohesion, this chapter is divided into two sections. These sectionsdiscuss the studies that are strictly related to the SE text-classification partand the SE topic modeling part of this thesis, respectively.2.1 Related Work to the Software EngineeringText ClassificationThe literature review for the SE text classification focused on two branches:one concentrate on what was done before (the conventional approaches) andthe other on what this study aim to achieve (PTM approaches). As our studyproposed PTM based SE text classification approaches and compared themwith the conventional methods, it is important to discuss both branches.Hence, this section is further divided into two subsections that discusses thestate-of-the-art traditional non-PTM approaches and the PTM models fortext classification.2.1.1 App Review Classification in Software EngineeringResearchers have utilized three main approaches to accomplish the taskof classifying SE text: topic models, generic machine learning algorithmsand their ensembles, and deep learning approach. The studies related tothese approaches are provided below.Topic Modeling Based Text Classification ApproachesAccording to Silva et al. [SGG21], topic modeling has been applied tosoftware engineering research, including app review classification. Amongthe most frequent topic modeling techniques used to categorize app reviews,the researchers found that Latent Dirichlet Allocation (LDA) [BNJ03] and72.1. RELATED WORK TO THE SOFTWARE ENGINEERING TEXT CLASSIFICATIONLDA-based techniques are the most common. AR-Miner is one of the initialworks for mining app reviews proposed by Chen et al. [CLH+14], whichuses topic modeling to group the informative reviews and investigates cat-egories of users’ discussions. Nayebi et al. [NCR18] examined how Twitterapp reviews can contribute to mobile app development and applied topicmodeling and crowdsourcing. Adaptive Online LDA was developed by Gaoet al. to classify app reviews based on users’ feedback on various versions ofmobile apps [GZLK18]. Consequently, Hadi and Fard developed the Adap-tive Online Biterm Topic Model (AOBTM) [HF20a] to model topics of appreviews in different categories adaptively, therefore, alleviating the sparsityproblems in short-texts, which considered the statistical data for multiplepreceding time-slices.The topic modeling approach is used in other works [GZLK18, GM14,YGZ+21, WS+21, GZLK18] to categorize app reviews for detecting emerg-ing issues for developers to update their apps, identifying fine-grained appfeatures in the reviews to extract users’ sentiments, prioritizing importantuser reviews for developers, and performing aspect-based sentiment analysisof app reviews.Machine Learning Based Text Classification ApproachesEarlier research has focused on using machine learning approaches forapp review filtering. Chen et al. [CLH+14] classified non-informative re-views by training a classifier and categorizing them into informative andnon-informative groups. Using a regression model, Fu et al. [FLL+13] filterout rating-inconsistent reviews that differ in sentiment from their ratings. In[GK15], Gu et al. identified software features and distinguish between fea-tures that are evaluated and those that are requested, using Max-Entropy[NLM99]. There are other studies that used machine learning techniquesfor app issue classification: [SSP18] and [RMA+20] use lexical features andmulti-text features to use in conjunction with different machine learningmodels for app review classification tasks. Maalej and Nabil [MN15] empiri-cally evaluated different feature extraction techniques to work with machinelearning models to automatically classify bug reports and feature requests.Martin et al. [MSJ+16] and Sarro et al. [SASH+15] have also exploredthe breadth of research done on App Store analysis (i.e., feature life cycle,dormant feature identification) for software engineering.In [GEHB15, TDG20, MJJM19, FLLY21, GOG+19], researchers haveused ensemble machine learning techniques to classify app reviews into cat-egories relevant to software evolution, to identify more efficiently and earlier82.1. RELATED WORK TO THE SOFTWARE ENGINEERING TEXT CLASSIFICATIONterms in reviews that could be classified into specific topics, and to filter outuseful feedback from users, such as feature requests, bug descriptions, andrequirement analysis.Deep Learning Based ApproachesUsing machine learning, [SHM19] demonstrated that they could stillachieve comparable results with deep learning when analyzing user feedbackin English and Italian into problem reports, inquiries, and irrelevant. Aslamet al. proposed an approach for app reviews classification using non-textualinformation to classify app reviews and exploiting deep learning technique[ARXS20]. In [MIG+21], a BERT-based sequence classifier was developedand validated to achieve a state-of-the-art average classification accuracy(87%) for feedback analysis. From a large volume of online product reviews,Qiao et al. [QWAF20] proposed to apply a domain-oriented approach todeep learning to discover the most critical users’ needs, such as app prod-uct new features and bug reports. Henao et al. [HFS+21] investigated theprospect of transfer learning for the classification of app reviews and foundthat monolingual BERT models outperform existing baseline methods in theclassification of English App Reviews.Other studies [HHC+19, HSM21, ASGM18, WWWX20, HF20b, ZZ19]also applied deep learning techniques to different extents, such as detectingpromotion attacks, matching bug reports, detecting adversarial spam, an-alyzing energy-related reviews, and requirement evolution predictions fromapp reviews.2.1.2 Pre-trained Models in Text ClassificationThere are two generations of pre-trained models: static and contextualword embeddings [QSX+20]. The Global Vectors for Word Representa-tion (Glove) [PSM14] and word2vec [Chu17] are examples of first-generationstatic word embeddings. Software Engineering (SE) research has used wordembeddings for various tasks, including code retrieval [VNNP+17], detect-ing incoherent comments [CCCS19], specifying SE-relevant tweets [SPS+20],and program comprehension with graphs [ABK17]. These kinds of embed-dings are context-independent, where each word has only one embeddingand cannot change with different contexts. These embeddings only take thesyntax of the words in a sentence into account. Therefore, non-contextualembeddings cannot capture semantic information and model polysemouswords [QSX+20].92.2. RELATED WORK TO THE SOFTWARE ENGINEERING TOPIC MODELINGThe second-generation pre-trained models learn context-sensitive wordrepresentations and can be tailored to perform downstream tasks effectively.LSTM [Gra12]-based Universal Language Model Fine-tuning (ULMFiT)[HR18] and Embeddings from Language Models (ELMo) [SWWP+21] havealso been used for different SE-related tasks, such as identifying ambigu-ous software requirements [SSRW21] and sentiment analysis for softwareengineering [RJ19]. Transformer architectures [VSP+17] have been usedto build several pre-trained Transformer-based models [DCLT18, LCG+19,LOG+19, YDY+19] that have achieved state-of-the-art performance for avariety of downstream tasks, such as sentiment classification of SE-relatedtexts [ZXT+20] and identifying error-prone software [SYTP85].Other studies [RZLL16, RCM19, YQZ+18, ZYZ20, JZC+19, THW+20]have used PTMs for solving Question-Answering problems. PTMs have alsobeen extensively used the sentiment analysis task [BW19, SHQ19, XLSY19,RSOE19, KRP21, LBZL19, WZZ+19, PABP17]. Previous studies [LRS+18,HP19, EBA19] developed Named Entity Recognition systems with the helpof PTMs. Others [CJN19, IS19, ZWZ19] studied how PTMs can improvemachine translation tasks. Other works include using PTMs for text sum-marization [LL19, ZLC+20, JJZS19] and adverserial defense and attacksidentification [WFK+19, SHY+20, LMG+20, ZCG+19, LCH+20].Though the studies that use PTMs or classify app reviews are numerous,there is no previous work that investigates their capability for app issueclassification similar to the work conducted in this thesis.2.2 Related Work to the Software EngineeringTopic ModelingFor the topic modeling for short texts, such as Probabilistic LatentSemantic Analysis (PLSA), Latent Dirichlet Allocation (LDA), and theirvariants suffer from the lack of enough word co-occurrences. To boost theperformance of topic models, researchers had utilized external knowledgeto produce supplementary essential word co-occurrences across short texts[JLZ+11, PNH08, MSBX13]. The problem rests in that auxiliary informa-tion can be too scarce or too expensive (or both) for deployment.In the short text topic modeling regime, Yin at al. introduced the DMM-based topic modeling method in [YW14], where it is presumed that eachshort text is sampled from only one latent topic. But this proved to betoo simple and too strong of an assumption for any reasonable short texttopic model [ZJW+11]. In self-aggregation based methods, short texts are102.3. SUMMARYmerged into long pseudo-documents before topic inference to help developrich word co-occurrence information. Researchers have used this type ofmethod in [QKGP15, ZWZ+16], where they have presumed that each shorttext is sampled from a long concatenated pseudo-document (unobserved incurrent text collection). This presumption enables inferring latent topicsfrom long pseudo-documents. But the concatenation yielded suboptimalresults in [LWZ+16, BPP+17] as merging short texts into long pseudo-documents using word embeddings cannot alleviate the loss of auxiliaryinformation or metadata. Global word co-occurrences based methods (i.e.,[YGLC13, ZZX16]) try to use the rich global word co-occurrence patterns forinferring latent topics, where the adequacy of these co-occurrences alleviatesthe sparsity problem of short texts. [YGLC13] posits that the two wordsin a biterm share the same topic drawn from a mixture of topics over thewhole corpus. This topic modeling algorithm is comparatively more robustand suitable for all the mentioned characteristics of short texts [LDW+17].To solve the problems with streaming short texts and unordered topicgeneration, researchers proposed online models such as OLDA [HBB10] andOnline BTM [YGLC13]. In essence, online algorithms fit conventional topicmodels (i.e., BTM, LDA, respectively) over the data in a time-slice t anduse the inferred statistical data to adjust Dirichlet hyperparameters for thenext time-slice.Gao et al. [GZLK18] introduced the Adaptively Online LDA (AOLDA)by factoring in all the preceding time-slices’ contributions, instead of justthe preceding one. Here, they have shown that the comparison among thetopic distributions for more than two consecutive time-slices/versions canlead to more coherent and distinguishable topic learning. But this specificmethod uses LDA as their underlying topic model, which suffers from severaldiscussed issues when it comes to short texts [YGLC13].2.3 SummaryThis chapter covered the studies related to the conventional approachesfor SE text classification adopted by the researchers, the applications ofPTMs for regular text classification, and different types of traditional topicmodels that deal with long text corpora. This thesis adopted the best prac-tices from the PTM related studies and compared their performance withthe state-of-the-art approaches for SE text classification discussed in thischapter. It also proposed an adaptive online topic model for short texts bycarefully examining topic model-related work discussed here.11Chapter 3Background KnowledgeIn this chapter, background knowledge is included that will help thereaders of the thesis to better understand the following chapters and asso-ciated approaches. Chapter 4 discusses Pre-trained Language models, suchas BERT, ALBERT, RoBERTa, and XLNet. These models are based onTransformer architecture. The first two sections of this chapter include ahigh-level overview of this transformer architecture and the concept of apre-trained model. Chapter 5 discusses topic modeling for Software Engi-neering. The last two sections discuss the concept of topic modeling andthe Biterm Topic model, which is modified to design the proposed topicmodeling approach in Chapter 5.For brevity, I have refrained from providing a detailed description of allthese approaches, but instead, a very high-level overview has been presentedto accommodate the researchers who were not previously familiar with thisfield. In order to gain a more granular understanding of the models, readersshould consult the cited papers. Readers who are already familiar with thefollowing topics that will be covered in brief can skip to the next chapter.− Deep Neural Network with Non-Contextual Word Embedding Layer(Discussed in Chapter 4)− Overview of a Transformer Architecture (Discussed in Chapter 4)− Pre-Trained Transformer based Models (Discussed in Chapter 4)− Overview of Topic Modeling (Discussed in Chapter 5)− Biterm Topic Model (BTM) (Discussed in Chapter 5)3.1 Deep Neural Network with Word EmbeddingTraditional machine learning models depend on a data representationrelying upon hand-crafted features, chosen by users or domain experts,based on their usefulness for the classification problem at hand. On the123.1. DEEP NEURAL NETWORK WITH WORD EMBEDDINGFigure 3.1: Neural Network Architecture (CNN) with Word Embeddingother hand, neural networks (used in deep learning approaches) learn high-level feature representations automatically by using raw textual data asinput. Researchers have previously applied Convolutional Neural Networks(CNN) successfully to natural language processing problems such as textclassification [SHM19]. Usually, deep learning approaches require a largeamount of training data to outperform traditional machine learning ap-proaches [MN15]. If the dataset is smaller for conventional deep learningapproaches, pre-trained word embeddings can be leveraged to yield betterperformance.Figure 3.1 shows a simple architecture of a neural network which wasused by Stanik et al. [SHM19] to identify user comments. This architecturewill serve as one of the approaches for the empirical study in Chapter 4.The details of this architecture are provided below.The input layer for this architecture requires its’ textual inputs to be offixed size; so, the length of the longest input in the dataset is assigned as thefixed size of input for the specified layer. To make the other inputs reach therequired length for the input layer, we have to pad them. Right after theinput layer, the network consists of an embedding layer that contains thepre-trained word embeddings, where the non-contextual embedding matrixis used as the weights of this embedding layer and set the trainable parameterfor this layer to be false so that the weights of the embedding layer staysfrozen during training. After this embedding layer, the network consists of a1D convolution layer, a 1D global max pooling layer, and a dense layer witha concluding output layer. ReLU (Rectified Linear Unit) has been used asthe activation function for the convolutional layer and sigmoid function was133.2. OVERVIEW OF A TRANSFORMER ARCHITECTUREused for the dense output layer.3.2 Overview of a Transformer ArchitectureFigure 3.2: High Level Look of Transformer Model architecture [Ala18]The transformer contains a set of encoders (for processing the textualinput) and a set of decoders (for producing accurate output that varies fromtask to task) [VSP+17]. In Figure 3.2, the encoders are on the left-hand side,whereas the decoders are on the right-hand side. Both of these componentscontain modules of similar architecture that enable them to be repeatedlyplaced on top of one another any number of times. On a high level, theencoder maps a text input (embedded into an n-dimensional space) sequenceinto a continuous abstract representation by optimizing on different tasks.This abstract representation holds all the learned information. The decoderthen analyzes the learned abstract information about the input sequence togenerate a single output, step by step [VSP+17].The encoders inside the transformer are all identical in structure andhave separate weight matrices associated with them. These matrices arerandomly initialized and trained with the model. The encoders inside thetransformer generally do not share weight matrices. In the original trans-former architecture, authors stacked six encoders and six decoders, but more143.2. OVERVIEW OF A TRANSFORMER ARCHITECTUREencoders or decoders can always be added.Both encoder and decoder modules comprised of Multi-Head Attention(MHA) and a Feed Forward Neural Network (FFNN) layers. The individualterms in an input sequence are assigned their own query, key, and value vec-tors in each encoder. A vector corresponding to the encoder’s representationof a particular word is generated by multiplying the encoder’s representa-tion with these weight matrices generated during training. As soon as asequence of words is presented to the attention mechanism, it analyzes thequery vector attributed to each word in the sequence against each mappedkey. This approach enables the model to understand how the consideredword correlates with the other words in the input sequence. To retain focuson the words relevant to the query, the values are scaled based on atten-tion weights. As a result, the word under consideration receives an atten-tion output. Multi-head attention modules run several of these attentionmechanisms simultaneously. In order to obtain the expected dimension, theindependent attention outputs are concatenated and linearly transformed.Having multiple heads of attention allows the model to pay attention todifferent parts of a sequence differently.First, the encoders’ inputs pass through the attention layer. Then, theattention layer’s output vectors are supplied to the adjacent FFNN layer.The attention and feed-forward network of the same structures are inde-pendently applied to each encoder. Additional to the attention layer andthe FFNN, The decoder has an in-between layer known as Encoder-DecoderAttention Layer, which helps concentrate on different related componentsof the input sequence. Fig. 3.3 provides a high-level look at the buildingblocks of encoder and decoder units.Figure 3.3: Building Modules of Encoder and Decoder [Ala18]153.3. PRE-TRAINED TRANSFORMER-BASED MODELS3.3 Pre-Trained Transformer-based ModelsPre-trained models are machine-learning models developed, trained, andmade available by third parties. Instead of building a model from scratch,researchers or developers can use a pre-trained model as a starting point.As opposed to traditional statistical NLP, which trains task-specific mod-els on labeled datasets, this pe-trained approach trains one large model ona shared, fundamental task and then adapts (fine-tunes) it to a variety oftasks. In almost all cases, the pre-training task involves language model-ing. Learning representations that benefit a range of NLP tasks is possibleby leveraging a massive quantity of unlabelled data through various pre-training tasks, including classical next-word prediction, uncorrelated spans,and fill-in-the-blanks in the input sequence, among many others [QSX+20].Vaswani et al. introduced the Transformer architecture in late 2018 thatcan be used for pre-training language models, accelerating the shift fromtraditional recurrent language models to modern pre-trained Transformerarchitecture [VSP+17]. A variety of NLP tasks benefitted from using multi-ple layers of multi-head self-attention rather than expensive recurrent com-putation. Due to this, nearly all popular language models, which differ inseveral important ways, are now based on the Transformer architecture.According to Min et al., there are three classes of pre-trained languagemodels: autoregressive language models, masked language models, and en-coder - decoder models [MRS+21]. Because encoder-decoder models areprimarily used to translate text and rarely used to categorize it, they areirrelevant to this thesis. Therefore, this thesis focuses on the first two classes.An auto-regressive language model predicts the next word xi based onprevious words x1, x2, ..., and xi−1. The training objective is to maxi-mize the log-likelihood∑i log(P (xi|x1, x2, ..., xi−1);wT ). Here, wT repre-sents the model weights or parameters. In contrast to autoregressive mod-els, masked language models (MLMs) predict a word based on all otherterms in a sequence. During training, words are randomly masked usingthe token [MASK], or replaced by pre-defined specific tokens, ensuring thatthe model retains bidirectional information in making predictions. Duringtraining, the original tokens are recovered at the masked positions by max-imizing the log-likelihood:∑imilog(P (xi|x1, ..., xi−1, xi+1, ..., xn);wT ), inwhich mi ∈ {0, 1} indicates whether xi is masked or not, and wT are theencoder weights [MRS+21].Many models are derived from Transformer architecture, including BERT[DCLT18], RoBERTa [LOG+19], ALBERT [LCG+19], and XLNet [YDY+19].RoBERTa improves BERT’s pre-training, and ALBERT makes BERT faster163.4. OVERVIEW OF TOPIC MODELINGto train by sharing encoder weight. XLNet better handles long-distance de-pendencies by incorporating an auto-regressive pre-training approach. Thesemodels use different datasets for their pre-training while working with vary-ing sets of hyper-parameters, such as the number of epochs, batch sizes, droprates, learning rates, etc. The details of the adopted models and the reasonsbehind choosing them for Software Engineering related text classificationare provided in chapter 4.3.4 Overview of Topic ModelingStatistical models are used in natural language processing to clusterabstract/latent topics in a corpus or collections of documents, referred to astopic modeling. The topic modeling approach is often used in data miningto discover hidden semantic structures. Typically, topic models are basedon the presumption that either the whole document belongs to a particulartopic or that any two terms in the document (referred to as biterm) canbelong to a topic. Intuitively, given that a document or a biterm belongsto a particular topic, one would expect specific words to appear more orless frequently in the corpus. Despite the presumption, both types of topicmodels can split the document into multiple topics in varying proportions. Intopic modeling, words with similar meanings are grouped into clusters. Ananalysis of a set of documents based on statistics of words allows discovering,based on the balance of topics within each document, what the topics maybe. A topic model delivers this intuition in a mathematical framework.Topic modeling approaches can be divided into two main categories: theprobabilistic model and the non-probabilistic topic model [JZY+19]. A non-probabilistic approach is a matrix factorization algebraic process based onthe Bag of Words (BOW) method, which ignores the order of terms in thecorpus and only counts terms in the document. A probabilistic model, onthe other hand, was developed to improve an algebraic model by adding asense of probability using generative methods. The probabilistic models canbe further classified based on Supervised and unsupervised approaches totopic modeling. Early probabilistic models were all unsupervised, but later,researchers started incorporating supervised techniques for model learning.In a very limited domain of application, researchers also explored theseprobabilistic models in a semi-supervised manner.173.5. BITERM TOPIC MODEL (BTM)3.5 Biterm Topic Model (BTM)The Biterm Topic Model considers the whole corpus as a mixture of top-ics, where each biterm is drawn from a specific topic [YGLC13]. Assumingα and β are the Dirichlet priors, the specific generative process of the corpusin BTM can be described as follows:1. For each topic z,(a) Draw a topic-specific word distribution ϕz ∼ Dir(β).2. Draw a topic distribution θ ∼ Dir(α) for the whole collection.3. For each biterm b in the biterm set B,(a) draw a topic assignment z ∼Multi(θ).(b) draw two words: wi, wj ∼Multi(ϕz)The joint probability of a biterm (wi, wj) is: P (wi, wj) =∑z θzϕi|zϕj|z,and the joint probability of all the biterm collection =∏(i,j) P (wi, wj).3.5.1 Comparison of BTM with Other Topic ModelsBTM directly models the word co-occurrence pattern. BTM is comparedhere with Latent Dirichlet Allocation (LDA) and a mixture of unigrams tocomprehend its distinct nature.LDA generates a document by drawing a document-level topic distribu-tion θd, then iteratively sampling a topic assignment z for each word w inthe document. As a result of its excessive reliance on local observations,LDA suffers from the sparsity problem when documents are short, whichhinders the learning of topics.Mixtures of unigrams, unlike LDA, rely on corpus-level topic distribu-tions to determine topic assignments for documents. It overcomes the spar-sity problem in topic inference by leveraging the information of the wholecorpus, thereby improving topic inference.Neither of these models can model fine-grained topics in documents sinceboth assume that all words in a document are sampled from the same topic,but documents may contain multiple topics. BTM overcomes the data spar-sity problem by drawing topic assignments from the corpus-level topic distri-bution and also surmounts the disadvantage of the mixture of unigrams by183.6. SUMMARYbreaking documents into biterms. Since the topic assignments of differentbiterms in a document are independent, BTM keeps the correlation betweenwords and can capture more topic variations in a document.3.6 SummaryThis thesis has applied a multitude of models, approaches, methods,and techniques - either to design a new approach or to evaluate the perfor-mance of the proposed approach. For example, the Deep Neural Networkwith non-contextual Word Embedding Vectors is one of the prior approachesthat is used as a baseline to evaluate the PTMs’ performance for SE textclassification in Chapter 4; on the other hand, the transformer architecturediscussed here serves as the building block for PTMs, which in turn is di-rectly used and modified to better classify SE texts in Chapter 4. Moreover,the BTM topic modeling technique discussed in this chapter is used to de-velop the adaptive online topic model discussed in Chapter 5. Please notethat this chapter only includes concepts that are too extensive and intricateto discuss in the respective chapters. The other models, approaches, meth-ods, and techniques that are not discussed in this chapter (e.g., AR-Miner,SUR-Miner) are relatively simpler and uninvolved; so, they are succinctlydiscussed in the respective chapters.19Chapter 4Assisting Developers toBetter Classify UserFeedback with Pre-TrainedTransformer-based ModelsMobile application (app) marketplaces, such as Apple App Store andGoogle Play enable the users to rate and review apps [HF20b]. Users expresstheir usage experience by writing reviews from different perspectives, suchas the app’s quality, performance, and functionality. The reviews providea distinct way for the application developers to acquire customer feedback[LL17]. The developers monitor the reviews regularly to address users’ ma-jor concerns and resolve the reported issues in the forthcoming app update.If the app is not regularly updated addressing the issues, it will gradu-ally lose its popularity [LZC+15, IM10]. Therefore, app reviews have beenstudied extensively [KK18] and are shown to be resourceful for require-ment engineering [LL17], release planning [CSPG17], software maintenance[AHNS21], change-file localization [ZSC+20], and testing [GCP+18]. Differ-ent studies also focused on analyzing the reviews to filter non-informativereviews [CLH+14], identify bugs [MN15], classify usability and quality con-cerns [LL17], and find security and privacy issues [BWB20].Other than App Stores, mobile app users also express their opinions onsocial media such as Twitter, which regularly reveals new information forthe developers [SHM19]. Nonetheless, the manual extraction of informativefeedback from App Stores or social media is difficult, as there are many unre-lated and noisy user comments. Therefore, researchers have developed tech-niques, including rule-based, machine learning, or deep learning approaches,for the automatic extraction of useful information from user reviews to helpapp developers [CLH+14, GK15, GEHB15, SHM19]. There are many studiesthat classify the reviews for different purposes, which range from identifyingproblems, user inquiries, feature requests, and aspect evaluations (e.g., fea-20ture strengths, weaknesses, and performance) to sieving usability, portabil-ity, and reliability [SHM19, KK18, LL17, CSPG17, MN15, BWB20]. Thesestudies provide app developers with a variety of serviceable information thatfacilitates making informed decisions for planning app updates. This extrainformation is crucial for app developers as mobile applications and theirupdates get released quite frequently over a short period to meet the mar-ket requirements; so, it often proves challenging to identify and prune allthe software defects and bugs during the testing phase [JMK13].The topics of interest in app review classification are numerous and dif-ferent studies leverage various supervised machine learning techniques toextract useful software engineering information from app reviews [KK18].The main requirement of supervised techniques is the availability of labeleddatasets, which needs to be done manually by field-experts [SHM19]. Al-though app review analysis is shown to be useful for software developers inpractice, this manual effort is expensive and seems to be a barrier in manystudies [KK18]. Recent works try to alleviate the problem by incorporatingsemi-supervised learning [DHR17] or active learning [DPAM18]. However,with the emergence of new labels of interest, e.g., privacy or security issues[BWB20], developers must label new datasets, which renders the previouslycurated datasets obsolete. In addition, the distribution of the data collectedfrom different platforms (i.e., App Stores or social media) varies. Therefore,even though the same labels are to be classified from two platforms, a modelthat has been trained on one (e.g., Google Play) needs to be retrained toextract the same labels from another (e.g., Twitter) to yield similar perfor-mance [SHM19]. Whenever the classification purpose (i.e., labels of interest)or platform changes, the models require retraining on newly labeled datasetssince they do not retain the natural language properties learned from previ-ous training, including the linguistic features and the context of the trainedlabels [MN15]. Training from scratch prevents the extensibility of the de-veloped models for classifying new issues in practice.A more recently established and widely accepted practice in Natural Lan-guage Processing (NLP) is using Pre-Trained Language Models (PTM) andthen transferring its learned knowledge to various downstream NLP tasks,such as sentiment analysis, question answering, or classification [QSX+20].In NLP, PTMs (such as BERT) are large language models that are trainedon large natural language corpora using a deep neural network in an unsu-pervised manner [ZXT+20]. These models are then fine-tuned for variousdownstream tasks using limited labeled datasets. As PTMs are trainedon large general domain corpora, they learn contextual linguistic informa-tion and eliminate the need to train downstream task models from scratch214.1. MOTIVATION AND OBJECTIVE[WTM+20].Pre-trained models or PTMs reduce the amount of effort (i.e., new modeldevelopment time per task) to build models for each task separately, andthey reduce the amount of required labeled dataset [LOG+19]. Conse-quently, PTMs are used to transfer the learned knowledge to a new domainor a new task, and in settings where a model has not seen any exampleof the required task or interested label during training (known as zero-shotlearning) [YHR19]. Although PTMs are used extensively and led to manyadvances in NLP, their applicability to software engineering is barely known.Only a few studies exist that explore the use of PTMs for sentiment anal-ysis in software engineering [ZXT+20, BKPVS20], or for tasks related toprogramming languages such as comment generation [SDFS20]. But, towhat extent PTMs can be applied for app review classification is unknown.4.1 Motivation and ObjectiveThe PTMs that are trained on domain-specific data, show significant im-provements over general purpose PTMs such as BERT (i.e. PTMs that aretrained on general purpose data) for NLP tasks in fields such as science orlaw [BLC19, CFM+20]. These domain-specific PTMs leverage unsupervisedtraining on a large domain-specific corpus to compensate for the lack ofhigh-quality, large-scale labeled data in the specified domains. The problemof interest, the classification of app reviews, aims to extract useful informa-tion for software engineers/app developers, which is domain-specific infor-mation. In addition, app reviews are short text, and many reviews are noisyand unrelated, which can introduce more complications [HF20b]. To whatextent the PTMs can be useful in the context of app review classificationand whether domain-specific PTMs trained on app reviews can improve theresults is unknown. Previously, the deep learning approaches used Convolu-tional Neural Networks (CNN) to implement the transfer of static linguisticknowledge from non-contextual word embedding, which has shown to havecomparable results with traditional machine learning approaches for classi-fication of requirements related app reviews [SHM19]. In contrast, anotherstudy using CNN shows improvements over previous approaches for app re-view classification [ARXS20]. Though these studies do not use PTMs, theyuse a deep learning approach which sheds doubt on whether PTMs can beuseful.Therefore, this thesis aims to explore the benefits of PTMs compared tothe existing approaches for app review analysis, specifically the app review224.2. METHODOLOGYclassification tasks. Here, the app review classification is defined as the taskof extracting useful information from users’ feedback which can be relatedto app requirements, release planning, and software maintenance. The ex-tracted information may help identify different aspects of the application,such as feature requests, aspect evaluations (e.g., feature strength, featureshortcoming, application performance), usability, portability, reliability, en-ergy consumption, problem reports, privacy, and security inquiries aboutthe application. My goal is to investigate the accuracy and time efficiencyof PTMs for the app issue classification task over different selected datasetsfrom literature with various labels and multiple tasks (i.e., issue classifica-tion and sentiment analysis of app reviews). Therefore, experiments will beconducted in different settings: by fine-tuning the PTMs on different sizes ofthe labeled dataset for downstream tasks, exploring the same PTMs for mul-tiple tasks in app review analysis, and finally, comparing the performance ofPTMs when they are trained on non-domain-specific dataset versus PTMstrained on the domain-specific dataset. These experiments will provide base-lines on the applicability of PTMs for app review analysis, including the costof using them (in terms of required time for predictions) and their capabilityto reduce the manual effort required to label large datasets.This chapter contributes the following to the thesis:− A study exploring the applicability of PTMs for automatic app issueclassification tasks compared to the existing tools.− An extensive comparison between four PTMs and four existing toolsor approaches on six different app review datasets with different sizesand labels.− A study to explore the performance of general versus domain-specificpre-trained PTMs for app review classification.− An empirical study to examine the accuracy and efficiency of PTMs infour different settings: binary vs. multi-class classification, zero-shotsetting, multi-task, and multi-resource setting.4.2 MethodologyIn this section, three research questions are discussed along with theoverview of the research design. The details about selected datasets are alsoprovided with the discussion of the Prior approaches and chosen Pre-Trained234.2. METHODOLOGYModels. The evaluation metrics used to evaluate the models have also beendiscussed a the end of this section.4.2.1 Research QuestionsThis chapter explores the following three research questions that answerthe three most important aspects of this empirical study:RQ1: How accurate and efficient are the PTMs in the classifi-cation of app reviews compared to the existing tools?This research question explores how the current PTMs perform com-pared to the existing tools, including their required time for training andprediction. The existing tools are based on curated rules, feature-engineeredmachine learning algorithms, and deep learning models. This research ques-tion answers whether PTMs are effective for SE text classification or not.Findings: From the results, it is determined that different PTMs outper-form Prior approaches by ∼3% to ∼15% on all datasets. Additionally, thebest-performing PTMs require slightly more time for prediction, althoughthe difference in prediction time is negligible (+0.04s to +2.87s).RQ2: How does the performance of the PTMs change whenthey are pre-trained on the app-review dataset, instead of a genericdataset (e.g. Wiki documents, book corpus)?The current PTMs are trained using general text scraped from the web.In some studies, domain-specific models are trained e.g. LEGAL-BERT[CFM+20] and medical BERT [WTM+20] which have been shown to im-prove the performance compared to the non-domain-specific models. Theunderlying intention of RQ2 is to explore the performance of a domain-specific pre-trained model when trained on app reviews. The questiontests the limits of PTMs, where the results confirm that the PTMs thatreceive domain-specific pre-training can yield better results than out-of-the-box PTMs.Findings: The PTMs trained from scratch on domain-specific data (i.e.adding app-reviews to the model) performed better than off-the-shelf models(+0.8% to +2.2% micro-F1 score). The Custom PTMs trained from scratchdo not fluctuate much in prediction times with respect to the readily avail-able PTMs for app issue classification. In addition, the results show thatincorporating a greater number of app reviews in the pre-training can helpPTMs produce up to 15.2% better micro-F1 scores.244.2. METHODOLOGYRQ3: How do the PTMs perform in the following settings?(RQ3-1) Binary vs multi-class setting,(RQ3-2) Zero-shot classification,(RQ3-3) Multi-task setting (i.e. different app-review analysis tasks),(RQ3-4) Classification of user reviews collected from different re-sources (i.e., Twitter, App Store).The answers to this research question will help us understand the ap-plicability of PTMs and their generalizability through comparison of theirperformance in different settings: (i) when the classification is on two or moreclasses; (ii) when a labeled dataset is not available; (iii) the task changes;and (iv) different resources are used. In part (iv), the transferability of themodels are explored as the distributions of data on various platforms aredifferent [RP17].Findings: Both Prior and PTM models yield better performance for bi-nary classification tasks than for multi-class settings. PTMs are the bestchoice for zero-shot classification settings. Custom PTMs improve the re-sults of their non-domain-specific models. For multi-task and multi-resourcesettings, the Custom LARGE PTMs (i.e., PMTs pre-trained with 10 mil-lion app reviews) performed better than readily available PTMs. RoBERTa-based Custom PTMs have the best scores among all models. They also havelower prediction times in the multi-resource setting than Prior approaches.4.2.2 Research OverviewFigures 4.1 and 4.2 show the overall process of the study. First, 6 widelyused datasets are selected from the literature for app review classification.These datasets are later combined to build another merged - dataset, whichhas multiple labels and is highly imbalanced (D7). In total, four approachesare chosen from the literature that are collectively referred to as Prior ap-proaches. Four pre-trained language models are selected to evaluate theircapabilities regarding the research questions in this thesis. All the Prior ap-proaches and the PTMs are evaluated for these datasets. From the PTMs,the best performing ones are selected to train Custom PTMs (CPTM) andassess them in RQ2 for all seven datasets. In RQ3, the performance of thePTMs and Custom PTMs are evaluated in four different settings. The de-tails of the datasets and selected approaches and PTMs are explained in thenext sub-sections.254.2. METHODOLOGYFigure 4.1: Overview of Research Steps for RQ1Figure 4.2: Overview of Research Steps for RQ2 (left) and RQ3 (right)4.2.3 DatasetsApp reviews are analyzed automatically for various purposes and reviewsare classified with different labels such as user concerns, feature requests,feature modifications, bug reports, and usability analysis [HF20a]. In theexperiments, the performance of PTMs are examined for app issue classifi-cation. Therefore, a diverse set of datasets are selected from the literature264.2. METHODOLOGYas described below. These datasets have various sizes, and different labels,and are collected from different platforms (i.e., Google Play, Apple AppStore, Twitter), and therefore, can represent to what extent the PTMs canbe useful for app issue classification.Dataset 1 (D1) This dataset is procured by Gu and Kim [GK15] andcontains reviews for 17 popular Android apps from Google Play in 16 mostpopular categories, such as games and social. The authors have manuallylabeled 2,000 reviews for each app (34,000 in total) in five classes accordingto their predefined rules: Aspect Evaluation (5,937), Praise (8,112), FeatureRequest (2,323), Bug Report (2,338), and Others (15,290).Dataset 2 (D2) This dataset contains 6,406 app reviews from GooglePlay and 10,364 tweets (sampled from 5 million tweets written in English)which are labeled manually into three classes: Problem Report, Inquiry,and Irrelevant [SHM19]. They included Problem Report (1,437), Inquiry(1,100), and Irrelevant (3,869) records from Google Play. The number ofrecords in each of these categories from Twitter data is 2,933, 1,405, and6,026, respectively.Dataset 3 (D3) This dataset is provided by Lu and Liang. [LL17].Researchers selected two popular Apps, one from Apple App Store (iBooksin the books category) and one from Google Play (WhatsApp in the com-munication category). They sampled the raw reviews collected from eachplatform and manually classified 2,000 reviews for each app (4,000 in total).The reviews are classified into six categories: Usability (432), Reliability(587), Portability (119), Performance (121), Feature Request (558), andOthers (2,183).Dataset 4 (D4) This dataset was procured by Maalej and Nabil [MN15]and contains 2,000 manually labeled reviews from random apps selected fromtop apps in different categories (1,000 from Apple App Store and 1,000 fromGoogle Play), where half of the apps are paid and half of them are freeapps. The authors ensured that there are 200 reviews for each of the 1,2, 3, 4, and 5 stars in each of the 1000-review datasets. In addition, theyprovide 2,400 more reviews that are manually labeled. These reviews arefor selected 3 random Android apps and 3 iOS apps from the top 100 apps(400 reviews for each app). This dataset contains 4,400 labeled reviews intotal with four categories: Bug Report (378), Feature Request (299), UserExperience (737), and Rating (2721).Dataset 5 (D5) This dataset is published by Guo et al. [GS20] andcontains 1,500 app reviews, which are manually labeled as User Action (428),App Problem (399), and Neither (673). The reviews are selected from over5.8 million records for 151 apps from Apple App Store.274.2. METHODOLOGYDataset 6 (D6) This dataset was procured by Guzman et al. [GEHB15];it contains reviews of 3 apps from the Apple App Store and 4 apps fromGoogle Play. The apps are popular and from diverse categories. They sam-pled 260 reviews per app and five annotators labeled the records, summingto 1,820 reviews that are manually labeled. This dataset includes seven cat-egories, namely: Bug Report (990), Feature Strength (644), Feature Short-coming (1281), User request (404), Praise (1703), Complaint (277), andUsage Scenario (593)3.Reasons behind choosing these Datasets (D1 to D6): This study consid-ered these datasets for their diverse characteristics from three perspectives,other than having different categories. First, the considered datasets wereconstructed from various repositories: Apple App Store, Google Play, andTwitter. Second, the size of the datasets varies and they have a differentnumber of records for each label. For example, Dataset 2 is around fourtimes larger than Dataset 5. Also, Dataset 2 contains approximately 2,000reviews per label, whereas Dataset 1 has around 400 reviews per label. Thesize differences can provide insights into the ability of PTMs for differentsizes (total training set and the available data for each label). Third, ex-cept Dataset 1 and Dataset 5 which are balanced datasets, the other fourdatasets are imbalanced. These differences ensure that this study exploresthe capability of PTMs for different sizes, and platforms, and in more real-istic settings.Merged Dataset (D7)This dataset will be compiled by merging all 6 datasets, D1–D6. ThisMerged Dataset has 55,933 app reviews with 16 labels. As some of the la-bels used in the datasets are the same or have similar definitions, they aregrouped together under a single label. For grouping the labels, the defi-nitions of the classes have been consulted from the studies that publishedthe datasets D1–D6. This dataset would be closer to practical applications,where a lot of irrelevant data exists and multiple classes of informative re-views are of interest to be extracted from this highly imbalanced dataset.The labels in the Merged Dataset, the grouped labels from D1–D6, and thenumber of app reviews per group are provided in Table 4.1.3Note that adding numbers in all categories will exceed the total number because somereviews belong to multiple groups. This study followed the steps in [GEHB15] to calculatethe evaluation metrics for this dataset.284.2. METHODOLOGYTable 4.1: Description for Merged Dataset (D7)Labels inMergedDatasetGrouped Labels [LabelName: Dataset Initial (#of Reviews)]# AppReviewsin TotalPerformance Performance: D3 (121) 121Portability Portability: D3 (119) 119Usability Usability: D3 (432) 432Reliability Reliability: D3 (587) 587Usage Sce-narioUsage Scenario: D6 (593) 593FeatureStrengthFeature Strength: D6 (644) 644User Experi-enceUser Experience: D4 (737) 737FeatureShortcomingFeature Shortcoming: D6(1,281)1,281InquiryInquiry: D2 (1,100), User Ac-tion: D5 (428)1,528ProblemProblem Report: D2 (1,437),App Problem: D5 (399), Com-plaint: D6 (277)2,113Rating Rating: D4 (2,721) 2,721Bug ReportBug Report: D1 (2,338), BugReport: D4 (378), Bug Report:D6 (990)3,706Feature Re-questFeature Request: D1 (2,323),Feature Request: D3 (558), Fea-ture Request: D4 (299), User re-quest: D6 (404)3,584Aspect Evalu-ationAspect Evaluation: D1 (5,937) 5,937PraisePraise: D1 (8,112), Praise: D6(1703)9,815IrrelevantOthers: D1 (15,290), Irrelevant:D2 (3,869), Others: D3 (2,183),Neither: D5 (673)22,015Total 55,933294.2. METHODOLOGY4.2.4 Prior ApproachesIn this section, the four most widely-used approaches for app reviewanalysis and the reasons for choosing them in the experiments are discussed.An extensive literature review was conducted and a broad range of analysesof app review surveys were reviewed to select these Prior approaches.AR-MinerChen et al. proposed App Review Miner (AR-Miner) [CLH+14] whichextracts valuable information from user reviews. Provided a collection ofuser reviews, AR-Miner first applies a pre-trained classifier that separatesnon-informative reviews. Then, AR-Miner applies Latent Dirichlet Alloca-tion (LDA) over the informative reviews to chunk them into different groupsfor prioritizing them by an efficient ranking model proposed by the au-thors. AR-Miner has been widely used to mine app issues from user reviews[PDSG+15, VBR+16, DSPA+16, BLVBC+14, PLVB+15, MLVBC+16]. There-fore, it is considered as a prior approach in the experiments.Reason behind choosing AR-Miner: AR-Miner is an unsupervised ap-proach and its performance could be evaluated against other models. Thesetechniques were mainly chosen to evaluate the PTMs against unsupervised,supervised, ensemble methods using classical machine learning approaches,and finally deep learning technique that is enriched by contextual embed-ding. AR-Miner is the first framework to accommodate application devel-opers in mining informative topics from a large volume of app reviews. Thiswas the first effective attempt to leverage an unsupervised Topic Modelingtechnique to extract edifying issues from app reviews. Developers embracedAR-Miner for its ability to filter out non-informative reviews and displaythe key user feedbacks in an intuitive, concise manner. The effectiveness ofAR-Miner was validated by conducting a comprehensive set of experimentson user reviews of four Android apps. Researchers compared the AR-Minerresults against real app developers’ decisions in different studies [PDSG+15,VBR+16, DSPA+16, BLVBC+14, PLVB+15, MLVBC+16]. They analyzedthe advantages of AR-Miner over the manual inspection and other techniquesused in a traditional channel [FHJ+14, JSABM17, MLVBCP15, SASH+15].Based on the empirical results, AR-Miner has proven to be effective andefficient in extracting informative reviews.SUR-MinerGu et al. [GK15] proposed Software User Review Miner (SUR-Miner),304.2. METHODOLOGYwhich is a framework that summarizes users’ sentiments, opinions, and emo-tions toward different aspects of an application. SUR-Miner parses aspect-opinion pairs from reviews by considering their structures. For this pur-pose, it uses pre-defined sentence templates/patterns4. Then, for each re-view, it combines the sentiment of the sentences with their aspect-opinionpairs. These are used in the final step to summarize the software aspects.SUR-Miner generates reliable summaries and achieved a high F1 score foraspect-opinion identification, sentiment analysis, and app review classifica-tion compared to the previous works [YNTT11, GM14] by using a simplebut effective machine learning algorithm, Max-Entropy . It is adopted inmany studies [ZXW19, PSC+17, PLVB+18, DSPA+17, BG17] and thereforeit is chosen as a prior approach.Reason behind choosing SUR-Miner: SUR-Miner is the first frameworkthat fully took advantage of user reviews’ monotonic structure and seman-tics; it also defined sentence patterns to extract aspect opinion pairs fromapp-review. Researchers carefully selected five distinct text features, specif-ically tailored for app reviews, and leveraged these features to train a su-pervised machine learning classifier. Based on the empirical evaluation, thismachine learning approach was chosen as the terminal classes categorized bythe SUR-Miner model was shown to be significantly more accurate and pre-cise than other methods, demonstrating its effectiveness [YNTT11, GM14].Ensemble MethodsGuzman et al. systematically evaluated different sets of ensemble meth-ods and identified one for classifying user reviews [GEHB15]. They se-lected machine learning techniques that were used for text classification,and compared the performance of each algorithm individually, for app re-view classification. They also studied the performance of these models whentheir results are combined using ensemble methods. Ensemble methods arewell-known machine learning techniques as they can enhance the predic-tion performance of single classifiers, maintaining their strengths and re-ducing their vulnerabilities. In this work, Logistic Regression, Naive Bayes,Support Vector Machines, and Neural Network Classifiers were grouped tovote for the final prediction. This ensemble method outperformed the in-dividual algorithms with statistical significance. It was tested on a largedataset of seven diverse apps, where it shows that in all cases, this en-semble method either outperformed or matched its best baseline. Also,this method influenced further research in the field of app review analysis4https://guxd.github.io/srminer/appendix.html314.2. METHODOLOGY[MSJ+16, GAS16, CSPG17, GIG17, AJM17, SBR+17] and therefore is usedin the thesis.Reason behind choosing Ensemble Methods: Different studies have at-tempted to identify the best machine learning approach for app issue classi-fication without spending too much time perfecting custom-engineered fea-tures for different issues [MKNS16, WWL+19, LL17, WZL+18, BG17]. Guz-man et al. [GEHB15] first leveraged different machine learning algorithmsand combined their predictions to circumvent the problem of various ma-chine learning models being used for the classification of issues from appreviews. Their study showed that recall (an evaluation metric) improvedsignificantly after combining the predictions of Logistic Regression and Neu-ral Networks. Software developers and software evolution experts widelyadopted this approach to analyze user reviews and prioritize their tasks[SFSA17, ASSB+19, PTT+19, DLPS19].Deep Learning Model leveraging Non-contextual Word Embed-dingStanik et al. used a Deep Convolutional Neural Network (CNN) [SHM19]for classifying app reviews. Their model contains an embedding layer andits weights are initialized with a word embedding model, e.g. word2vec orFastText. This CNN architecture outperforms the shallow Neural Networksclassifying app reviews in the determined groups, by a small margin. Thismethod uses non-contextual word embedding within a deep neural networkand performed a transfer of knowledge representation. This approach is oneof the first studies leveraging transfer learning and deep learning and is usedin other studies for issue classification task [SSP21, Sta20, ARXS20], andtherefore, it is chosen as a prior approach in the thesis.Reason behind choosing this Deep Learning Approach: This study isa recent work with the purpose of understanding and evaluating the ex-tent to which deep learning models could be used to categorize user feed-back into different predefined categories [SHM19]. Based on this study,researchers determined that the domain experts’ knowledge incorporationwith the traditional machine learning model could achieve comparable re-sults to those of the deep learning approach due to the substantial perfor-mance improvements provided by using simple yet powerful features in thetraditional machine learning techniques. Nevertheless, the developers andresearchers widely adopted this deep learning approach as the improvementof pre-trained word embeddings bring considerable performance gain to theclassifier in other domains [YMO18, RSB+19, RZZJ16].324.2. METHODOLOGYIt is worth noting that the number of studies for app review classificationis numerous. These four Prior approaches are chosen as they are well-citedand widely used in many other studies. More importantly, they use differentmachine learning and deep learning approaches which can be representativeof multiple techniques used in other studies. Additionally, these models areopen-sourced and are publicly available. The best of these four approachesin terms of the performance metrics defined in Section 5.3.3 will be chosenas the baseline.Two more tools had also been considered to be Incorporated in thisstudy that was later eliminated from the thesis: AR-Doc and CLAP. AR-doc uses a parser based on an NLP Classifier, which uses a set of 500 sentencestructures manually evaluated for four types of classes: Information Giving,Information Seeking, Feature Request, and Problem Discovery. To use AR-Doc, different sets of sentence structures needed to be built for each datasetand their respective labels, which is infeasible. In addition, no publicly avail-able implementation was found for CLAP and therefore it was not possibleto use CLAP5.4.2.5 Pre-Trained Models (PTM)In this section, the choice of the four PTMs is briefly discussed. Allthese models have Transformer deep learning architecture which is based onattention mechanism [YDY+19]. As Transformer is currently the main ar-chitecture for PTMs [LOG+19], the following Transformer PTMs are chosenfor the study: BERT, XLNet, RoBERTa, and ALBERT. The same PTMsare also used for sentiment classification in software engineering [ZXT+20].BERTDevlin et al. [DCLT18] designed Bidirectional Encoder Representationsfrom Transformers (BERT) to learn contextual word representations fromunlabeled texts. Contextual word embeddings designate a word’s representa-tion based on its context by capturing applications of words across differentcontexts. BERT employed a bidirectional encoder to learn the words’ con-textual representations by optimizing for Masked Language Model (MLM)and Next Sentence Prediction (NSP) tasks. For MLM, 15% of all the tokensare replaced with a masked token (i.e., [MASK]) beforehand, and the modelis trained to predict the masked words, based on the context provided bythe non-masked words. For NSP, the model takes sentence pairs as input5The authors of CLAP were contacted but they did not respond to the artifact request.334.2. METHODOLOGYfor learning to predict whether a pair match is correct or wrong. Duringtraining, 50% of the inputs are true consequent pairs, while the other 50%are randomized non-consequent sentence pairs. Devlin et al. trained twoversions: small-sized BERTBASE and big-sized BERTLARGE. BERTBASE isa smaller model with 12 layers and 110 million parameters. BERTLARGE has24 layers and 340 million parameters. BERTLARGE is more computationallyexpensive and consumes more memory compared to BERTBASE. Thus, inthis work, BERTBASE is being utilized.Reason behind choosing BERT: BERT advanced the state-of-the-art for11 NLP tasks. It achieved an absolute improvement of 7.6% over the pre-vious best score on General Language Understanding Evaluation (GLUE)benchmark6; it also achieved 93.2% accuracy on SQuAD 1.17, outperforminghuman performance. As the tasks in these datasets require natural languageunderstanding and include classification tasks, BERT is selected.XLNetXLNet [YDY+19] uses Auto-Regressive language modeling and Auto-Encoding. The model is “generalized” because it captures the bi-directionalcontext using a mechanism called Permutation Language Modeling (PLM).XLNet integrates auto-regressive models and bi-directional context model-ing, yet overcoming the disadvantages of BERT. PLM is the idea of capturingbidirectional context on all permutations of terms present in an input se-quence. XLNet discards the one-directional linear modeling to maximize thelog-likelihood over all permutations of the sequence terms. Each position isexpected to learn utilizing contextual information from the entire sequence,thereby capturing the bidirectional context. No [MASK] is needed, andinput data need not be corrupted.In addition, XLNET addresses capturing the dependency between maskedpositions, which is neglected by BERT. Consider the sentence “New Delhi isa city.” Then, input to BERT to be “[MASK] [MASK] is a city,” and the ob-jective of BERT would be predicting “New” given “is a city” and predicting“Delhi” given “is a city”. In this objective, there is no dependency betweenlearning “New” and “Delhi.” So, BERT can result in a prediction like “NewGotham is a city.” If it is assumed that the current permutation is [is, a,city, New, Delhi], BERT would predict the tokens 4 and 5 independent ofeach other. Whereas, XLNet, predicts in the order of the sequence. i.e.,6https://gluebenchmark.com/7Stanford Question Answering Dataset: https://rajpurkar.github.io/SQuAD-explorer/explore/1.1/dev/344.2. METHODOLOGYfirst predicts token 4 and then predicts token 5: it computes the likelihoodof “New” given “is a city” plus the likelihood of “Delhi” given “New, is acity”.Reason behind choosing XLNet: XLnet outperformed BERT on manyNLP tasks; for 8 different tasks XLNet beat BERT by a substantial margin.This model achieved the best results for 18 NLP tasks, including sentimentclassification and natural language inference. As XLNET outperforms othermodels on text classification tasks, it is chosen as one of the PTMs in thethesis.RoBERTaRobustly optimized BERT approach (RoBERTa) outperformed all thestate-of-the-art benchmarks upon release [LOG+19]. Liu et al. modifiedBERT’s pre-training steps which yield substantially better performance onall the classification tasks. RoBERTa increased the amount of mini-batchsizes, data, and training time to train the model. RoBERTa is also trained ona dataset that includes longer sequences than before. The masking patternin RoBERTa was also modified to be generated spontaneously.Reason behind choosing RoBERTa: RoBERTa outperforms BERT onnine different NLP tasks on the GLUE benchmark; it also equals or exceedsXLNet’s model in four out of nine individual tasks. Based on these results,RoBERTa can present a reasonable choice for PTM in the thesis.ALBERTA Lite BERT (ALBERT) [LCG+19] applies three parameter reductiontechniques: Factorized embedding parameterization, Cross-layer parametersharing, and Inter-sentence coherence loss. In the first one, researchers sep-arated the hidden layers’ size from the input embeddings’ size (previously ofthe same sizes). They projected one-hot vectors to embedding and the hid-den space with lower dimensions; it increased the hidden layer size withoutsignificantly increasing the vocabulary embeddings’ parameter size. Sec-ond, all the parameters across all layers are shared. The big-scale ALBERTmodel has substantially fewer parameters than BERTLARGE. Finally, theNSP task is swapped with Sentence-Order Prediction (SOP) loss which helpsALBERT perform better.Reason behind choosing ALBERT: ALBERT uses parameter reductiontechniques which resulted in having 18 times fewer parameters than BERT.Its’ training time is 1.7 faster and has negligible inferior performance thanthe original BERTLARGE model. The much larger ALBERT architecture,354.2. METHODOLOGYwhich contains fewer parameters than BERTLARGE, achieved higher F1scores on the SQuAD 2.0 and the GLUE benchmarks; it also achieved highaccuracy on the ReAding Comprehension from Examinations benchmark(RACE benchmark). Therefore, ALBERT is chosen as one of the PTMs inthe experiments to evaluate its performance and inference time for app is-sue classification, because ALBERT is shown to be faster than other PTMswithout sacrificing much performance.For BERT, XLNET, RoBERTa and ALBERT, their base versions areused. Table 4.2 includes the names of the models used from the HuggingfaceTransformers library [WDS+19] and default configurations.Table 4.2: Details of PTMs Used in the StudyArchitecture Used Model Parameters Layers Hidden HeadsBERT bert-base-cased 110M 12 768 12RoBERTa roberta-base 125M 12 768 12XLNet xlnet-base-cased 110M 12 768 12ALBERT albert-base-v1 11M 12 768 124.2.6 Evaluation MetricsIn this section, the evaluation metrics are described. For all the classi-fications, three metrics are used: Precision (P), Recall (R), F1 score (F1),and their micro and macro average values. To answer the time efficiency ofthe models, the training time for the Prior approaches and the fine-tuningtime for PTMs are examined. The prediction times and changes in time(increase or decrease of time compared to a baseline) are also reported asevaluation metrics.Precision (P): Precision can be calculated by dividing the number ofrecords their labels are correctly predicted by the total number of predictedobservations in that class: P = TPTP+FP . Here, TP refers to the number ofrecords whose label is correctly predicted, and FP refers to the number ofrecords falsely predicted to belong to this class. In multi-class classification,for each group A, all the observations that belong to other labels and arefalsely predicted as group A are added to compute the FP.Recall (R): For each group A, Recall can be calculated by dividingthe number of accurately predicted observations in A by the number of allobservations available in the corresponding class: R = TPTP+FN Here, FN is364.2. METHODOLOGYthe number of observations in class A which are falsely predicted as otherlabels.F1 Score (F1): F1 score is the harmonic mean of Precision and Recall:F1 =2 · (P ·R)P +R(4.1)As it is multi-class classification for all the datasets and all the datasetsare imbalanced, micro-average and macro-average metrics are used as fol-lows. Here, the micro-average calculates the contribution of all records in allclasses (therefore the contribution of the class with the predominant numberof records is taken into account), whereas the macro-average is the averageof the values for each class (therefore, each class contributes equally in thefinal value). The micro- and macro-averaged precision (P) are computed as:Pmicro =∑mj=1 TPj∑mj=1 TPj +∑mj=1 FPj(4.2)Pmacro =∑mj=1 Pjm(4.3)TPj and FPj are the numbers of true positive and false positive predic-tions for the j-th class, respectively. Pj is the precision for class j and mis the number of classes. Similarly, the micro- and macro-average of Recall(R) and F1 score will be calculated and denoted as Rmicro, Rmacro, F1micro,and F1macro.As a stratified 5-fold cross-validation is being used, for each dataset D,averages of these metrics obtained from each of the k classifiers are reported:F1avgmicro = 1/kk∑n=1F1micro(i) (4.4)F1avgmacro = 1/kk∑n=1F1macro(i) (4.5)The F1micro(i) and F1macro(i) refer to the test performance of classifierc(i) on held out test set D(i). Similarly, the Pavgmicro, Pavgmacro, Ravgmicro,and Ravgmacro will be computed.Following previous works [ZXT+20], if a model has higher values for bothF1avgmicro and F1avgmacro, it is considered as to be better than other models.Time: To compare the time efficiency, the training time for Prior ap-proaches and fine-tuning time for PTMs, and the prediction time of all the374.3. EXPERIMENTAL SETUPapproaches are reported. Prediction time is the time the model requiresto process the test data and predict labels. The time will be measured inseconds and will be reported for each dataset. A model is considered to bemore time efficient than another if the prediction time is less; as training amodel or fine-tuning a PTM is a one-time process and need not be repeated.Time Change in Percentage: All four Prior approaches are employedon all datasets D1–D7 to determine the best approach as the baseline ap-proach. In addition to the Time mentioned above, the increase and decreasein percentage for the studied PTMs with respect to the considered baselinefor the measured Time are reported. The reason for reporting this changein percentage is that time duration depends on hardware configuration, andpercentage change provides a better understanding of the change in time forthe readers.4.3 Experimental SetupThe implementations of both Prior and PTM approaches for the men-tioned datasets are individual and intricate. The details about these imple-mentations are discussed below in the separate sub-sections.4.3.1 Prior Approaches’ ImplementationFrom the extensive literature review, four approaches have been selectedthat are frequently used by developers and adopted by researchers to classifySE texts. The details about these approaches are provided below.AR-Miner uses LDA [BNJ03] topic modeling algorithm to categorizetext document corpus into a predefined number of different related andcoherent topics. For app review classification, the labels of the considereddatasets are hidden and the app reviews are presented altogether as a textcorpus, where each review represented different documents. If a dataset hasn classes, n number of topics are extracted from the app review corpus;so, the documents will be separated into n number of segments. Afterthe extraction of topics, the associated labels for each review are examinedfor assigning each document cluster (topic) to a relevant class/label. Therelevance was measured by the prevalence of a class or label in a topic afterextraction. Suppose, the highest number of app reviews belong to label/classx in a topic (document cluster), it is assumed that AR-Miner has labeledall documents in that cluster with class x, where the app reviews that truly384.3. EXPERIMENTAL SETUPbelonged to class/label x are regarded as the True Positives and the rest ofthe reviews are regarded as False Positive.SUR-Miner used Max-Entropy model to classify app reviews. Thisclassifier model considers all probability distributions empirically consistentwith the training data and chooses the distribution with the highest entropy.Empirically consistent probability distribution with training data is one inwhich an estimated frequency of occurrence of a class and a feature value isequal to the actual frequency in the training data. Following the literature[GK15], Generalized Iterative Scaling (GIS) have been implemented as thescaling method and performed Early Stopping to avoid over-fitting and tostop the iterative solver from taking a long time to optimize over a largenumber of feature weights. As mentioned in [GK15], 2-4 Grams (CharacterN-Gram) have also been adopted for the classification task.Ensemble Approach adopted an ensemble of Logistic Regression clas-sifier and shallow Neural Network classifier and applied the majority votingscheme to combine the output of the classification approaches, by followingthe literature [GEHB15]. TF-IDF vector representations of app reviews havebeen concatenated, along with the number of words in the review, the num-ber of characters in the review, the number of lowercase characters, numberof uppercase characters to use as features for the ensemble approach. Adamoptimizer and binary cross entropy have been used as the loss function inthe shallow neural network.Deep Learning approach withWord Embeddings followed [SHM19].For the neural network’s input layer, text inputs must have a fixed size. Thisinput size has been fixed at 400 words, a measure which is found to be suit-able for all the app reviews in all of the datasets reviewed for this study, andthere was no app reviews identified that exceeded 400 words. In addition tothe input layer, the network consists of an embedding layer (pre-filled witha fast-text model), a 1D convolution layer, a 1D global max-pooling layer, adense layer, and a final output layer with a softmax activation. Tanh acti-vation was used for the previous layers. The embedding layer weights werefrozen during training, leaving the trainable parameters.4.3.2 Model Training SetupAll the Prior approaches are trained and all the PTMs are fine-tuned oneach of the datasets separately. In the following, general information abouttraining the models using stratified k-fold cross-validation is provided, along394.3. EXPERIMENTAL SETUPwith how PTMs will be trained and a classifier is added on top of each model.The detailed process for each research question is provided in section 4.3.To avoid introducing bias to the results due to probable differences in thedistribution of the training and test splits, following [SHM19], k-fold crossvalidation was used on each dataset Di separately, where Di represents oneof the seven datasets in the study. k-fold cross-validation was employed as amore rigorous approach than splitting the dataset into train and test set once[ZXT+20]. Also, this method is used previously for app review classificationin the software engineering domain [NGL18]. In the k-fold cross validation,a dataset D is split into k equal size disjoint parts/folds D(1), ..., D(k). Thenk number of classifiers c(i) is built, each time using k − 1 number of splitsas training and one part D(i) as a test set. This way, each split of the datacan be used as a test set once. As a result, we end up with k differenttest set performances. As the datasets in the study are imbalanced, thealternate option of cross-validation called stratified k-fold cross-validationwas implemented [FS10, HM13]. In this method, the only difference is thatthe distribution of the examples from each class in the original dataset ispreserved in each fold D(i). Stratified k-fold cross-validation is commonlyused in machine learning practices as it reduces the experimental variance.Therefore, when comparing different methods, it becomes easier to identifythe best one [FS10]. k = 5 is considered as this value has been empiricallyshown to have such test error rate estimates, which do not suffer from highbias or high variance [JWHT13]. The value of k = 5 is also used in previousapp review classification studies [SHM19]. For the evaluation, metrics ex-plained in section 4.2.6 are used. It is worth mentioning that the stratified5-fold cross-validation is applied on each of the datasets D1–D7 separately.For example, D1 is split into 5 folds D1(1), ..., D1(5) and 5 classifiers are builtfor D1. Then, the evaluation metrics are computed for D1. This processis continued for each of the datasets D2–D7 and report the performance oneach of them separately.Note that the number of labels in the datasets D1 −D6 ranges between[3,7] and all the considered Prior approaches were designed for multi-classclassification. Therefore, they can be retrained and evaluated using thestratified 5-fold cross-validation. Among the Prior approaches, only AR-Miner requires some adaptation. AR-Miner has three phases: classificationof non-informative reviews, topic modeling of informative reviews, and rank-ing of the groups according to their relevance to developers’ applications. Asall datasets D1–D7 are labeled, The second step is directly utilized to groupall the reviews (from both training and test set) in a dataset into the cor-responding number of labels available. Following the literature [CLH+14],404.3. EXPERIMENTAL SETUPthe training set reviews in a resulting group are counted and the label withthe majority count is used to annotate the group. This annotation indicatesthe class that the reviews in the group belong to. Then, the group’s test setreviews are examined to compute the evaluation metrics.For evaluating the PTMs, for each dataset, I used the training set tofine-tune the PTMs and then evaluate them on the test set. I obtained allfour PTMs that are pre-trained on general domain corpora from HuggingFace library8. To build a classifier, a feed-forward dense layer and Soft-maxactivation function will be added on top of each model. I follow [ZXT+20]and [DCLT18] to set the hyper-parameter values. In the work, I set thebatch size to 16 and Adam learning rate to 2e− 5. The models are trainedfor 4 epochs and AdamW optimizer is used for all models.I execute all the experiments on a Linux machine with Intel 2.21 GHzCPU and 16GBmemory. For training PTMs from scratch, I use 2×NVIDIATesla V100 32GB to enhance the parallelization performance.4.3.3 Approaches to Answer the Research QuestionsThe research questions discussed in the previous section (methodology)cover a vast area and thus different evolved approaches need to be adoptedto answer these research questions. The following steps will be taken toanswer each research question.Approach for Research Question 1For RQ1, I identify the accuracy and efficiency of PTMs compared toPrior approaches. For this RQ, all Prior approaches and PTMs will betrained and tested on all of the datasets separately, as explained in section4.3.2. I only consider the reviews (not tweets) from Dataset 2 for RQ1. Foreach dataset and approach, all the evaluation metrics described in section5.3.3 will be reported. I highlight the best model for each dataset withthe highest F1avgmicro and F1avgmacro. The lowest training/fine-tuning andprediction times will also be highlighted for each dataset, along with thetime change in percentages. I denote the best performing PTM amongothers as PTM-X to pursue experiments in other research questions. If noPTM achieves the best results in terms of both micro- and macro-F1 score,the PTM that achieves the highest F1avgmicro-score (denoted as PTM-XM1)and the PTM with the highest F1avgmacro-score (denoted as PTM-XM2) willbe selected to be studied in RQ2.8https://huggingface.co/414.3. EXPERIMENTAL SETUPApproach for Research Question 2RQ2 is interested to evaluate the performance of PTMs when they arepre-trained on domain-specific corpus rather than non-domain-specific cor-pora. In RQ2, I use PTM-X (or PTM-X1 and PTM-X2) from RQ1 and allthe seven datasets. I can either pre-train PTM-X using just the domain-specific app reviews or combine them with the general domain documentsto pre-train PTM-X from scratch. In the first approach, PTM-X with thesame architecture will be trained from scratch on app review sentences. Butpre-training PTM-X on a small number of domain-specific documents has itsdisadvantages; the model may overfit the dataset and it can result in perfor-mance degradation in downstream tasks. By pre-training PTMs using bothgeneral and domain-specific datasets, I can avoid this problem. I use theapproach of Wada et al. [WTM+20], where I pre-train the PTM-X simultane-ously with both general and domain-specific documents. Following the liter-ature [WTM+20], I increased the frequency of pre-training domain-specificdocuments proportionately during the optimization on the Masked Lan-guage Modeling (MLM) task, which is the task the ALBERT and RoBERTamodels are trained on.To accomplish this, I must increase the frequency of pre-training forMLM over app reviews. The objective can be achieved by improving thevocabulary representation of app reviews. Therefore, I use simultaneouspre-training after the up-sampling introduced in [WTM+20]. I create pre-training instances from a set of corpora with different sizes to pre-traina PTM, as described in Figure 4.3. Upsampling refers to the copying andpasting the app reviews portion of the corpus until the predetermined ratio ofapp reviews and general pre-training data is achieved; also, I have uniformlymixed the book documents, wiki documents, and app reviews consecutivelyto form the pre-training corpus. With this up-sampling technique, moreinstances from the small app review corpus are used for MLM.Figure 4.3: Creating Up-sampled dataset for PTM pre-training tasks424.3. EXPERIMENTAL SETUPI pre-train three different PTM-Xs, where I respectively integrate 2.8 mil-lion, 5.6 million, and 10 million app reviews that I have collected from GooglePlay, with the general domain documents from Wiki-texts9. The collecteddataset has app reviews for more than 2000 apps from different categories.The dataset includes app name, app category, review, rating, reply text,and date. This will give us three versions of a domain-specific pre-trainedmodel which I refer to as Customized PTM-X (CPTM-X). I denote the threesizes of the CPTM-X as CPTM-Xbase, CPTM-Xmedium, and CPTM-Xlarge, respec-tive to the number of the app review sentences integrated for pre-training.Finally, I fine-tune all the CPTM-Xs on the seven datasets according to thesteps and evaluation metrics explained in sections 4.3.2 and 5.3.3. The bestCPTM-X will be used in RQ3.Approach for Research Question 3I have used both the readily available PTMs and the CPTM-X in differ-ent settings for evaluating RQ3. I am considering four different settings todetermine the PTMs’ capacity regarding app review analysis.Binary vs. Multi-Class SettingPrevious studies have shown that the pre-trained Transformer basedmodels deviate from their original performance when the classification taskinvolves multiple classes instead of binary classes [ARTL19, CYZ+19]. Stud-ies on app review classification also show that binary classification yieldsbetter results [MKNS16]. This research question is, therefore, interested inevaluating the performance of the PTMs in both binary and multi-class set-tings, especially, investigating the performance of CPTM-X. I investigate theaccuracy and time efficiency of PTMs for binary classification compared tothe original multi-class settings. I convert all the seven datasets into binary-class datasets by randomly choosing one class (i.e. label) for each datasetand switching all the other labels in that dataset into “Others”. Then, I usethese modified datasets to regenerate the PTM and CPTM results for thebinary classification task on each dataset. The four PTMs and CPTM-X willbe used to record the evaluation metrics. All other steps will remain similarto RQ1.Zero-Shot Classification SettingThe Zero-Shot Learning (ZSL) that I investigate in the study is the9The Wiki-texts dataset is open sourced at https://dumps.wikimedia.org/.434.3. EXPERIMENTAL SETUPrecent approach in which I evaluate the models on fully-unseen labels. Inthis setting, I assume that the system is never trained on any labeled datafor the task I am interested in, which is different from some ZSL settingsin which labels are partially seen by the model. The setting I choose hereis more realistic and can provide insights about the potential benefits ofPTMs in practice, where the developers might want to use the same modelto derive new unseen aspects of the dataset. Using PTMs for this kind ofzero-shot classification requires different training.For ZSL, I explore the method proposed by Yin et al. [YHR19] in whichclassification is considered a Natural Language Inference (NLI) task. Thisapproach determines the compatibility of two distinct sequences by em-bedding both sequences and labels into the same space. In NLI, a pairof sentences are considered: “premise” and “hypothesis”; and the task isto predict whether the “hypothesis” is an entailment of “premise” or not(contradiction). I follow the steps in [YHR19] to prepare the datasets andmodels and set up the study for ZSL. I use the reviews as “premise” and can-didate labels as “hypothesis”. For Example, for the app review “Please adda back button in the armory page.” and its label “Feature Request”, I usethe review as a premise and the label as hypothesis. So, the NLI model canpredict whether the hypothesis is an entailment of the premise or not. Thisprediction will be compared to the actual labels for the reviews to calculatethe evaluation metrics. If the NLI model correctly predicts the hypothesis asan entailment of the premise, it is recorded as a True Positive. On the otherhand, if the NLI model predicts the hypothesis as contradiction or neutralto the premise, it is recorded as a False Negative. If the NLI model predictsthe hypothesis as contradiction or neutral, where the hypothesis is actuallyan entailment, I refer to this as False Positive (FP); else if the NLI modelpredicts the hypothesis as contradiction or neutral, where the hypothesis isactually contradiction or neutral, it is regarded as True Negative (TN).I fine-tune CPTM-X on NLI dataset10 and test it on the seven datasetsto determine how well PTMs can classify issues from app-reviews withoutbeing trained on them. Note that Multi-NLI data does not contain appreviews. In addition, I use RoBERTa-large-nli11, which is a readilyavailable fine-tuned model for zero-shot classification 12. This will provideinsights about using a custom PTM trained on app reviews and the PTMtrained on general-purpose corpora in a zero-shot setting.10https://cims.nyu.edu/ sbowman/multinli/11https://huggingface.co/joeddav/xlm-roberta-large-xnli12I only use the readily available PTMs for this study and avoid fine-tuning other PTMson Multi-NLI data in this setting, due to high computational expenses.444.3. EXPERIMENTAL SETUPMulti-Task SettingResearch has confirmed that when a new classification task (e.g. senti-ment classification) is introduced, a new model should be trained and themodels for another classification task (e.g. app issue classifiers) cannot beapplied for this new task [SHM19]. In addition, the same model or classifi-cation technique can have different performances on various analysis tasks.For example, Hemmatian and Sohrabi found Naive Bayes (a probabilisticclassifier) to be performing better than Decision Tree (a non-probabilisticclassifier) for sentiment classification tasks [HS19]; on the other hand, Maalejet al. reported that using Bag-of-Word technique and Decision Tree outper-formed Naive Bayes for app issue classification task [MKNS16]. Therefore,I am interested in evaluating the performance of PTMs in both app issueclassification and sentiment analysis task settings. In sentiment analysis,the task is to classify the sentiment of a given review into positive, neu-tral, or negative polarities [ZXT+20], which is different from the app issueclassification. In this setting, I evaluate the PTMs and CPTM-X on the newtask of sentiment classification for app reviews. I fine-tune these models onthe sentiment classification dataset and test them for sentiment classifica-tion. This setup will help evaluate whether the models trained for app issueclassification can be applied to another task, thus providing insights intothe multi-task ability of the PTMs. For the sentiment classification, I useanother dataset13 that consists of 37,185 app reviews with three sentimentpolarity labels: positive (13,758), neutral (9,993), and negative (13,434).I have gone beyond sentiment classification and implemented the PTMsfor category classification as well. The same Kaggle datasets also containcategory information for 20,955 app reviews. The number of user reviews inthis categories are: Family (18.9%), Games (50.3%), Tools (12.5%), Medical(10.7%), and Business (7.6%). As it is an imbalanced dataset, I use stratifiedK-fold cross-validation for maintaining uniformity with other experiments.The experimental settings and evaluation metrics I use are similar to RQ1.The results will determine how these models perform for different classifica-tion tasks in the same domain (app reviews).Multi-Resources SettingThe distribution of the data from multiple resources varies and modelsbuilt for App Stores are not suitable to classify user feedback from anotherresource such as Twitter [GAS17]. In this setting, I study the accuracy13https://www.kaggle.com/lava18/google-play-store-apps454.4. RESULTSand time efficiency of the PTMs for the classification of data collected frommultiple resources. For this experiment, Dataset 2 will be used as it containsApp Store reviews and Twitter data with the same labels. First, I fine-tune the PTM-X and CPTM-X on App Store reviews of Dataset 2 and reportthe evaluation metrics (training steps are similar to RQ1) when tested onTwitter data. In addition, I fine-tune the PTM-X and CPTM-X on Twitterpart of Dataset 2 and report the results when tested on App Store reviews.This way, I gain insights into the PTMs’ capacity to classify app reviewscollected from different resources on which the PTMs were not fine-tuned.The evaluation metrics used here are explained in section 5.3.3. To have acomparison, I also test the Prior approaches for this setting. From the Priorapproaches, AR-Miner does not require training. For the other approaches,I train them on the App Store reviews of Dataset 2 and test them on Tweetsin Dataset 2.4.4 ResultsIn this section, I present the results of the experiments for each of theresearch questions. Note that I have computed all the scores (micro- andmacro-scores of P, R, F1) and the training and prediction times for all of themodels for each dataset. But due to a large number of experiments, I putall the associated tables in the appendix. In the following, I only discuss theF1- scores in addition to the prediction time. The full results are shown inTables A.1 and A.2 for RQ1, Tables A.3 and A.4 for RQ2, Tables A.5 andA.6 for RQ3.1, Table A.7 for RQ3.2, Table A.8 for RQ3.3, and Table A.10for RQ3.4 in the appendices.4.4.1 RQ1: Accuracy and Efficiency of PTMs Compared toPrior approachesFig 4.4 shows the micro-F1 scores for Prior approaches and PTMs, wherethe performance of all models are shown for each dataset. Among the Priorapproaches, DL+WE is the best performing model, followed by Ensembleapproach. Only for D5, which has the lowest number of app reviews, theEnsemble method has better scores than DL+WE. I conjecture that this lownumber of reviews favored the Ensemble model to enhance the predictionperformance of participating classifiers. For the cumulative dataset (Dataset7 ), DL+WE improves the results of Ensemble Method, SUR-Miner and AR-Miner by ∼8%, ∼20%, and ∼40%, respectively. So, I have selected DL+WEas the baseline from Prior approaches.464.4. RESULTSFigure 4.4: PTM and Prior Performance (micro-F1 score)Among the PTMs, on most datasets ALBERT and RoBERTa achievethe highest F1-micro and F1-macro scores. The F1-micro results of thesePTMs are higher than the best Prior Approach and other PTMs for alldatasets, except that BERT outperforms other models for D3. The En-semble and DL+WE approaches can match the performance of some worseperforming PTMs in the respective Dataset, such as DL+WE outperformedBERT on Dataset 2. The other Prior approaches (i.e., AR-Miner and SUR-Miner) have significantly lower scores than the PTMs for all Datasets. Theperformance of XLNET varies compared to Prior approaches. For somedatasets such as D1, D4, and D5, it has comparable or lower micro-F1 com-pared to Ensemble and DL+WE ; but for D7 it has higher scores. RoBERTaperforms best on all metrics for Datasets 4 and 6, and yields best micro-average scores for D5. These three datasets have the least number of appreviews. On the other hand, ALBERT performs best on Datasets D1 andD2, the two datasets with the highest number of app reviews. For the cu-mulative dataset (Dataset 7), RoBERTa outperforms the other PTMs onmicro-scores, XLNET has the highest macro-scores, and the other PTMshave relatively close results.For D3, the established order of PTM’s performance persists (RoBERTa>ALBERT >XLNet), except for the performance yielded by BERT. BERToutperformed the other three PTMs by a small margin for this dataset(∼1.5% to ∼4.5%). D3 has 4,000 records with six labels, for which the‘Other’ class has more than 2,000 records. In other words, the models learnabout 5 classes from a small portion of reviews. When I further analyzed theconfusion matrices of ALBERT and RoBERTa for D3, I find that the mainreason for dropping their performance in this dataset is that they predictthe records of the ’Other’ class wrong, but their prediction for the other fiveclasses are mostly correct. So, the False Negative predictions are higher and474.4. RESULTSthis is also confirmed by having higher precision than recall for these modelsin D3. This is in contrast to D1. Dataset 1 has a large number of records(15,290) labeled as ’Other’ out of 34,000 records. The rest of this dataset islabeled by 4 other classes. Therefore, in D1, the number of available reviewsper label is much higher than the number of reviews for each label in D3,and the two PTMs perform better as they learn the features for each class.That being said, still, ALBERT and RoBERTa have F1-micro scores of over85% and F1-macro of over 83% for D1 and D3. The score of most modelsfor D6 is their lowest. This is also due to the specific characteristics of thisdataset, where it has the lowest number of records per label.As both RoBERTa and ALBERT have the highest or second-highestscores in the maximum number of datasets, I choose these two PTM modelsas the best-performing ones to pursue with the next research questions.Here, these two PTMs will be the PTM-X models and are chosen for trainingfrom scratch to develop CPTM-X (Custom PTMs). I refer to these CustomPTMs as C-ALBERT and C-RoBERTa.Figure 4.5: PTM and Prior Prediction Time (seconds)Fig 4.5 illustrates the prediction time for Prior and PTM approaches inseconds. AR-Miner uses topic modeling to predict the labels/classes; so, theprediction time is not reported for AR-Miner. From Fig 4.5, I can see thatboth Prior and PTM approaches follow a similar trend, and their predictiontimes are related to the size of the dataset, i.e., the larger the size of thedataset, the longer the prediction time is.Among the Prior approaches, the previously selected baseline DL+WEyielded the least prediction time on 3 datasets (D1, D4, and D7), whereasSUR-Miner method’s prediction time is the least for rest of the datasets484.4. RESULTS(D2, D3, D5, and D6). Among the PTMs, ALBERT consistently yields theleast prediction time for all the datasets. On the other hand, the predictiontime for XLNet is the highest for all datasets.In the last column of Table A.2 (Time-Diff (%)), I have included thepercentage change of the prediction times of PTMs with respect to the pre-diction time of the Prior approach that has the lowest time. For example,for D1, the Time-Diff (%) for PTMs are calculated based on the predictiontime of DL+WE ; and for D2 it is calculated based on the prediction timeof SUR-Miner. For all datasets except D7, the prediction time of PTMscompared to the best Prior Approach increases by 0.49% to ∼500%. ForD7, BERT and ALBERT predicted the classes ∼ 3% and ∼ 22%, respec-tively, faster than the fastest Prior approach (DL+WE ). For D7, RoBERTaincreases the time by 0.15% and XLNET increases the time by ∼89%.Findings of RQ1PTMs are generally superior to Prior approaches when predictingclasses with higher scores across all datasets. The two candidates forthe best PTMs I choose are ALBERT and RoBERTa, as they invari-ably perform well across all dataset sizes. Based on the time taken forprediction, DL+WE took the shortest amount of time among all Priorand PTM approaches, and ALBERT has taken the shortest amountof time among the PTM approaches.4.4.2 RQ2: Domain-Specific PTMs vs. General PTMsFigure 4.6: Performance of Custom PTMs (micro F1 score)494.4. RESULTSFig 4.6 shows the performance of C-ALBERT and C-RoBERTa afterthey are pre-trained on domain-specific app-reviews along with general do-main corpora. In CPTM-X-Base, I have incorporated 2.8 million app re-views during pre-training. For CPTM-X-Medium and CPTM-X-Large, I haveincreased the number of app reviews during pre-training to 5.6 million and 10million, respectively. From the result, I observe that all the Custom PTMsperformed better in all datasets than their corresponding out-of-the-boxPTMs. I also find that increasing the number of app reviews in pre-trainingcorpora helps boost the performance of Custom PTMs.ALBERT has lower F1-micro scores for datasets D3 and D4. Afterpre-training with app reviews, I notice that all C-ALBERT models (i.e.,C-ALBERT-BASE, C-ALBERT-MEDIUM, and C-ALBERT-LARGE) havesignificantly better predictions for both of these datasets (∼ 5% to ∼ 13%improved performance). Although this is not the case forD6, all C-ALBERTmodels still improve their performances. I relate this to the low number ofreviews per class in D6. The Custom PTMs produce better predictions thantheir readily available counterpart. C-RoBERTa models also have betterperformance, yielding up to 15.2% increase in performance over RoBERTa.As seen in Fig 4.6, C-RoBERTa improves RoBERTa’s F1-micro scores fordatasets D1–D4, where RoBERTa has scores of below 95. There is none tolittle fluctuation in prediction time for all the Custom PTMs.Findings of RQ2Incorporating domain-specific data (i.e., app reviews) during pre-training of the PTMs improves their performance for app reviewclassification, and slightly improves the prediction time of CPTMscompared to PTMs. The performance of the models benefits fromincorporating more domain-specific data in the pre-training.4.4.3 RQ3: Experimenting PTMs in Different SettingsThis research question deals with different settings, which are extensivein their own merit. The results of different settings are discussed below indifferent subsections.RQ3-1: Binary vs. Multi-class SettingI report the binary classification results in Tables A.5 and A.6 for RQ3-1.The multi-class classification results of the PTMs and CPTMs are previously504.4. RESULTSreported in Table A.1 to Table A.4. The micro- and macro- scores werecalculated to accommodate the existence of multi-class in the datasets, whichis no longer the case after I converted each multi-class datasets into binaryclass datasets.The randomly selected classes for D1 to D7 are Bug-Report, Inquiry,Reliability, Feature Requests, App Problem, Bug-Report, and Aspect Eval-uation, respectively. All the other classes in these datasets are labeled asOther or Irrelevant.From the results in Table A.5 and Table A.6 we observe that both Priorand PTM approaches have performed better in binary classification com-pared to multi-class classifications. The improvement for the Prior ap-proaches ranges from 0 (for AR-Miner) to ∼3.2%. On the other hand, theimprovement for the PTM approaches ranges from 1 to ∼7.7%. All thePTM approaches yielded better performance for binary classification thanthey did for multi-class classifications and we observe that readily-availablePTMs’ performance boost is similar to the performance boost produced bythe Custom PTMs.RQ3-2: Zero-Shot SettingI report Precision, Recall, and F1 score for zero-shot classification resultsin Table A.7. As in this setting, I am not fine-tuning the model on a portionof the considered Datasets D1 to D7 (rather, I am fine-tuning the PTMs ona separate NLI dataset), I no longer require cross-validation over multiplefolds.In this setting, I anticipate lower performance because the PTM modelsdo not have any exposure to any instances of the datasets to fine-tune on.In this experiment, I use AR-Miner as the baseline to compare the resultsof this setting for PTMs and CPTMs. AR-Miner is chosen as it is a topicmodeling based approach that does not require seeing the label beforehand.The other models from Prior approaches are not considered here as all ofthem are supervised machine learning or deep learning techniques that re-quire training with the labels beforehand. The C-RoBERTa-LARGE-NLIand C-ALBERT-LARGE-NLI are the CPTMs that are pre-trained on 10million app reviews (See RQ2). Overall, in D7, RoBERTa-LARGE-MNLIyielded best performance (71%). We observe that RoBERTa-based mod-els achieve higher scores compared to ALBERT-based PTMs. Results ofAR-Miner vary between 36% and 44% F1 score for the datasets, and AR-Miner has closer results to ALBERT-NLI. For D5, ALBERT-NLI yieldeda ∼37% F1 score, which is ∼5% worse than the performance of AR-Miner.514.4. RESULTSFor Datasets D1 to D7, the PTMs yield on an average 45.875%, 44.425%,43.125%, 43.225%, 46.45%, 38.8%, 42.725% less F1 score than their counter-parts in RQ1. All PTM approaches outperformed AR-Miner, the only priorapproach capable of zero-shot classification, except in D5 for ALBERT-NLI.The Custom PTMs in all cases improve the results by approximately 10 F1scores in all datasets. Similar to previous RQs, the ALBERT-based modelhas a lower prediction time.RQ3-3: Multi-task SettingIn Tables A.8 and A.9, I report the micro and macro Precision, Recall,and F1 score and the training and prediction times for two classificationtasks in the app review domain: category classification and sentiment clas-sification.Among all the PTMs, C-ALBERT-LARGE yielded the best result forcategory classification tasks with micro-average F1 scores of ∼91% and C-RoBERTa-LARGE yielded the best result for sentiment classification taskswith micro-average F1 scores of ∼93%, respectively.BERT, ALBERT, RoBERTa, and XLNET performed 2.4%, 1.8%, 7.5%,and 4.3% worse for category classification, respectively; they performed1.1%, 0.1%, 5.5%, and 0.9% worse for Sentiment classification task, respec-tively with respect to their averaged micro-F1 scores over 7 datasets, in TableA.2. C-ALBERT-BASE, MEDIUM, and LARGE performed 4%, 2.7%, and2.4% worse for category classification, respectively, compared to their aver-aged micro-F1 scores over 7 datasets, in Table A.2; they performed 2.8%,3.4%, and 2.3% worse for the sentiment classification task, respectively withrespect to their F1 scores averaged over 7 datasets, in Table A.3. On theother hand, C-RoBERTa-BASE, MEDIUM, and LARGE performed 4.1%,5.9%, and 6.2% worse for category classification, respectively; they per-formed 1.5%, 2.3%, and 3.3% worse for the sentiment classification task,respectively with respect to their F1 scores averaged over 7 datasets, in Ta-ble A.4. For both classification tasks, we observe that the custom PTMsachieve higher scores compared to the PTMs, showing that although thetask is different (e.g., sentiment analysis vs. app review classification), theystill benefit from pre-training the models on the same domain (app reviews).RQ3-4: Multi-resource SettingDataset D2 contains labeled app reviews from two different sources: AppStore and Twitter. In the previous results (Table A.1 to A.8), I have not524.4. RESULTSincorporated app reviews collected from Twitter. Here, I use the reviewsfrom both App Store and Twitter from D2.Training on App Store App-reviews and Testing on TwitterIn the first part of this setting, I fine-tuned all the available approaches(except for AR-Miner) on app reviews from App Store and tested them onapp reviews collected from a different resource, i.e., Twitter. The results areavailable in Table A.10. Being a topic modeling approach, AR-Miner doesnot require any supervised training.C-RoBERTa-LARGE achieves the best score among all the consideredapproaches for this setting, in terms of F1 score (0.86). From the previ-ous approaches, DL+WE approach scored higher than the readily avail-able PTMs, and all Custom PTMs, except C-RoBERTa-LARGE. It alsobeats the C-ALBERT-LARGE by 1% F1 score. The Ensemble method sur-passed only readily available PTMs and closely matched the performanceof C-ALBERT-MEDIUM. AR-Miner and SUR-Miner have the lowest scoresamong the Priors, and, I do not observe a big reduction in their perfor-mance with respect to single-resource setting, in terms of F1 scores. Theseapproaches have scored (∼5%) and (∼2%) less in multi-resource settingsthan their single-source counterparts.Compared to the setting where PTMs are evaluated only on app re-views from App Store, we observe ∼13% to ∼31% performance reductionfor micro-F1 scores. This performance reduction is less for Custom-PTMs.Among the Custom ALBERT models, the C-ALBERT-LARGE model pro-duced the best micro-F1 score of 0.80 (∼25% reduction) and C-ALBERT-BASE model produced the least micro-F1 score of 0.71 (∼21% reduction).The C-RoBERTa group has less reduction compared to C-ALBERT mod-els, where the C-RoBERTa-LARGE produced the best micro-F1 score of0.86 (∼15% reduction) and C-RoBERTa-BASE model produced the leastmicro-F1 score of 0.77 (∼15% reduction) among all C-RoBERTa models.Training on Twitter and Testing on App Store App-reviewsIn the second part of this setting, I trained/fine-tuned all the availableapproaches (except for AR-Miner) on Twitter and tested them on app re-views collected from a different resource, App Store. The results are avail-able in Table A.11.C-RoBERTa-LARGE achieves the best score among all the consideredapproaches for this setting, with an F1 score of 0.92. From the previousapproaches, the DL+WE approach scored higher than the readily available534.4. RESULTSPTMs in this setting. Here, it could not surpass the C-ALBERT-LARGEbut beat the C-ALBERT-MEDIUM by 1% F1 score. The Ensemble methodoutperformed only readily available PTMs and C-ALBERT-BASE from theCustom PTMs. AR-Miner and SUR-Miner have the lowest scores amongthe Priors, which have scored (∼10%) less and (∼2%) more in multi-resourcesetting than their single-source counterparts, respectively. SUR-Miner beatthe BERT in this setting by ∼4%.Compared to the setting where PTMs are evaluated only on app reviewsfrom App Store, we observe ∼9% to ∼30% performance reduction for micro-F1 scores. This performance reduction is less for Custom-PTMs. Amongthe Custom ALBERT models, the C-ALBERT-LARGE model producedthe best micro-F1 score of 0.87 (∼8% reduction), and C-ALBERT-BASEmodel produced the least micro-F1 score of 0.74 (∼18% reduction). The C-RoBERTa group has less reduction compared to C-ALBERT models, wherethe C-RoBERTa-LARGE produced the best micro-F1 score of 0.92 (∼5%reduction) and C-RoBERTa-BASE model produced the least micro-F1 scoreof 0.78 (∼14% reduction) among all C-RoBERTa models. The smaller per-formance reduction for the models in this part, compared to when modelsare trained on App Store data and tested on Twitter data, could be relatedto the size of the training data. The dataset D2 has a higher number oflabeled tweets (∼ 10K) than the app reviews from App Store (∼ 6.5K).The prediction times of all Custom PTMs are lower than Prior ap-proaches in this setting.544.5. SUMMARYFindings of RQ31. Both Prior and PTMmodels yield better performance for binaryclassification tasks than they do for multi-class setting.2. PTMs are the best choice for zero-shot classification settingand Custom PTMs, though having less score than the largeRoBERTa-MNLI model, improve the results of their non-domain specific models by approximately 10 F1 scores.3. For multi-task and multi-resource settings, the readily availablePTMs and Custom PTMs showed the same trend as demon-strated in RQ1 and RQ2, that is: Readily Available PTM <Custom-BASE < Custom-MEDIUM < Custom-LARGE.4. RoBERTa-based Custom PTMs have the best scores among allmodels. They also have lower prediction times in the multi-resource setting compared to Prior approaches.4.5 SummaryThis study aimed to investigate how PTM can improve the app reviewclassification performance over traditional prior approaches. Each approachwas applied to seven app review datasets with varying sizes, app categories,and label distributions. Our analysis showed that PTMs are generally moreeffective at predicting high scores than prior approaches across all datasets.We enhanced two of the best-performing PTMs by pre-training them ondomain-specific app review datasets to varying degrees and evaluating theirperformance across the seven datasets to better understand the implicationsof domain-specific pre-training. Our findings show that PTMs have betterperformance for app review classification when domain-specific data is uti-lized during pre-training; greater domain-specific data incorporation duringpre-training translates into better performance. Additionally, the inclusionof domain-specific data into these PTMs also narrowly improves predictiontimes.Moreover, we have modified the experiment settings in order to fullyexplore the applicability of the PTMs, whether pre-trained on the domain-specific dataset or not. These experiment settings include binary, zero-shot,multi-task, and multi-resource classification settings. In our experiments,we found that Prior and PTM models performed better as binary classi-554.5. SUMMARYfiers than as multi-classifiers. Custom PTMs are recommended for zero-shot classification settings since they have improved the results of non-domain-specific models by approximately 10 F1. Across multi-task andmulti-resource settings, readily available PTMs and Custom PTMs demon-strated the same trend, where the readily available PTMs perform less thanCustom-BASE PTMs, Custom-BASE PTMs perform less than Custom-MEDIUM PTMs, and Custom-MEDIUM PTMs perform less than Custom-LARGE PTMs. Readily available PTMs and PTMs with domain-specificpre-training displayed the same trend across multi-task and multi-resourcesettings, where readily available PTMs performed worse than PTMs withthe least domain-specific pre-training (CPTM-BASE), which in turn per-formed worse than the PTMs with the moderate domain-specific pre-training(CPTM-MEDIUM). Most of the top-performing PTMs across all settings in-corporated the largest amount of domain-specific data during pre-training(CPTM-LARGE).56Chapter 5Adaptive Online Bi-termTopic ModelAs discussed the previous chapters, mobile app reviews form a mainfeedback channel for the app developers [GZL+18] to evaluate their productsand improve application maintenance and evolution tasks [GK15]. The appdevelopers require to analyze app reviews in order to gain insights aboutthe current state of their apps from users’ perspectives. Several studies havebeen proposed to analyze the app reviews, including extracting informativetext [CLH+14], summarizing user reviews [MSJ+16], identifying the bugsor feature requests [PLVB+15, PM13, MN15], prioritizing feature inclusions[CW13], and extracting insights about the apps [GZL+18]. Although the(popular) apps are updated frequently [GZD+19], the app review analysisstudies mostly consider the app reviews static [GZLK18, HBH18]. However,app reviews have a dynamic nature and their discussed topics change overtime. If the update-centric analysis is neglected, it misses the point thatfeedback are written on a certain update [HBH18]. The change in the topicsextracted from reviews for different app versions can reveal important issuesabout the app [GZLK18, HBH18].A recent example of the importance of discussed topics over time is theZoom Cloud Meeting, a popular app for video conferencing. Zoom receivedmassive one-star ratings (the lowest rating) in Google Play Store during theCOVID-19 outbreak in March 2020. Most of the issues were related to users’concerns about data-privacy and security-malpractices. These issues wereso severe that in a letter to Zoom, the New York attorney general’s officeexpressed concerns and addressed security flaws [HS20]. These issues wereextensively discussed in the user reviews and could have been flushed outmonths ago through proper inspection of the user reviews and topic changesfrom user feedbacks on Google Play Store.App reviews are short texts that are time/version sensitive as these textsare generated constantly and are collected regularly for consecutive appversions [GZLK18, GZD+19]. The underlying latent topics derived from appreviews can benefit the developers extensively [NZZ19]. However, extracting57relevant topics from app reviews is challenging due to their dynamic natureand lack of rich context in short texts.The most popular topic modeling methods for discovering the underly-ing topics from text-corpus are LDA [BNJ03] and PLSA [Hof17]. But thesetopic models do not perform well with text-corpus containing short-texts asdocuments [YGLC13]. These algorithms consider individual short texts asseparate documents and model each of these documents as a mixture of top-ics, where each topic is considered as a probability distribution over words.The models then utilize various statistical techniques to determine the topiccomponents and mixture coefficients of each document by implicitly captur-ing the document-level word co-occurrence patterns [BGB08, WM06]. Whiledealing with the typical lengthy documents, the mentioned algorithms couldrely on larger word counts to know how the words are related. However, thenatural sparseness of the word co-occurrence patterns in each short doc-ument makes these models suffer from the data-sparsity problem [HD10].Moreover, short texts lack the richness of context, making it more difficultfor these topic models to identify the senses of ambiguous words in shortdocuments. Biterm Topic Model (BTM) alleviates these problems by learn-ing topics over short texts and explicitly modeling the generation of bitermsin the whole corpus to enhance topic learning [YGLC13].As mentioned, app reviews are usually collected in batches of consecu-tive time-slices [GZLK18]. Each time a new batch of text-data arrives, thesetopic models (e.g. LDA, BTM) require retraining to discover latent topicdistributions from the new dataset, which is prohibitively time and memoryconsuming. The popular way to alleviate the scalability problem is to de-velop online algorithms such as Online LDA (OLDA) [HBB10] and OnlineBTM (OBTM) [YGLC13]. These online algorithms store a small fraction ofdata on the fly in order to accommodate the dataset of the upcoming time-slice. When a new batch of text-data arrives, online algorithms model thetopics of texts either by using the statistics of samples collected in the im-mediately preceding version/time-slice14 [HWL18] or by naively aggregatingstatistics of all the preceding time-slices [YGLC13]. However, these onlinealgorithms do not take different versions’ varying contributions into the ac-count. Statistics of the textual data collected over different time periodsor different versions may have a non-negligible difference in similarity withthat of the latest time-slice; and thus, can contribute differently to the latestversion[WM06, GZLK18]. Adaptive versions of online algorithms, such asAdaptively Online LDA (AOLDA), can be used to address the problem of14slice, time-slice, and version are semantically equal in this thesis.58the varying contribution of different versions [GZLK18]. But, the underlyingmodel in AOLDA is LDA, which again makes it suffer from the mentioneddata-sparsity problem while working with short-texts.In this thesis, I propose a new online topic model for short texts whichtakes preceding versions’ varying contribution into account by adaptivelyintegrating prior knowledge in the form of topic distributions derived inprevious slices. I refer to this novel model as the Adaptive Online BitermTopic Model (AOBTM). AOBTM inherits the characteristics of BTM todeal with the data sparsity issue. It is an online algorithm that can scale forthe increasing volume of the dataset that is generated frequently. AOBTMalso endows the statistics of the preceding versions with different contri-butions to the topic distributions of the current version of the dataset toleverage the prior knowledge. Also, I have employed a preprocessing tech-nique that is useful for yielding better top contributing key-terms to helpthe manual investigation of the inferred topics. The contributions of thischapter to the overall thesis are enlisted below:1. A novel topic modeling method called AOBTM for version sensitivecontent analysis for short texts that adaptively combines the priorknowledge in the form of derived topic distributions of the selectednumber of prior versions to generate topic distributions of the currentversion.2. Two parallel algorithms to identify an optimal number of topics to bederived in the first version, and to identify the optimal number of pre-ceding versions to be taken into consideration for adaptive aggregationof statistical data.3. The open sourced scripts, codes, and dataset to replicate the results.15.I have conducted experiments on app review datasets and Twitter datasetswith a large number of records to evaluate the performance of AOBTMcompared to five baseline algorithms. Also, I integrated AOBTM into thestate-of-the-art online app-review analysis framework called IDEA for com-parison [GZLK18]. The results show that topics captured by AOBTM aremore coherent compared to the topics extracted by baseline methods.The rest of the chapter is organized as follows: Section 5.1 and 5.2describe the motivation and the model design methodology. Section 5.3 is15https://github.com/Mohammad-Abdul-Hadi/AOBTM-Adaptive-Online-Biterm-Topic-Modeling595.1. MOTIVATION AND OBJECTIVEdedicated to the experimental setup, which is followed by the results anddiscussion in Section 5.4.5.1 Motivation and ObjectiveIn the following four subsections, conventional topic models have beendiscussed along with their deficits that make them incompetent to work wellon the SE texts. The last subsection entails our objective to propose a topicmodel effective for SE texts, which is both adaptive and online.5.1.1 Topic Modeling for Conventional Text DocumentsTopic modeling algorithms such as PLSA and LDA are widely embracedfor identifying latent semantic structures from text corpus without requir-ing any prior annotations of the documents [JZY+19]. These algorithmsobserve each document as a mixture of topics while a distribution over thevocabulary terms characterizes each topic. Statistical techniques such asVariational methods and Gibbs sampling are then applied to infer the latenttopic distributions of given documents and the word distributions of inferredtopics [Ble12]. Although these algorithms and their variants contributedlargely in modeling text collections such as blogs, research papers, and newsarticles [HBB10, XX13, XYX15], these topic models endure considerableperformance deterioration while handling short texts [LTMC14, YGLC13].Directly applying these models on short texts suffer from severe data spar-sity problem [HD10] as the frequency of words in the short texts play lessdiscriminative role, which makes it hard to infer words correlation fromshort documents [HD10]. The limited contexts in the short text also makeit challenging to identify the sense of ambiguous words.5.1.2 Topic Modeling for Short Text DocumentsResearchers have proposed numerous topic modeling algorithms for shorttexts by trying to solve one or two of the following inherent characteristics ofthe short texts: i) lack of enough word co-occurrence information, probabilityof most individual short texts being generated by singular topic, ii) inabilityto fully capture semantically co-related but rarely co-occurring words (due tolack of statistical information of words among texts), and iii) the probabilitythat a single-topic assumption is too strong for some short texts [JZY+19].In [JZY+19], Qiang et al. divided the short text topic modeling algorithmsinto three major categories: Dirichlet multinomial mixture (DMM) based605.1. MOTIVATION AND OBJECTIVEmethods, Global word co-occurrences based methods, and Self-aggregationbased methods. Brief introductions to the related work can be found insection 2.2.5.1.3 Online Topic Modeling for Short TextsA particular issue with the traditional topic modeling algorithms is thatthey can not scale with the expanding dataset. Whenever a new batchof data arrives, these topic models (i.e., LDA, PLSA) need to train fromscratch. Moreover, these conventional topic models can not guarantee con-sistency in the sequence of topics if independent training is performed ondifferent batches of the same corpus [BNJ03, YGLC13]. This inconsistencyoccurs because, before each of the independent training, the prior topic dis-tribution is set to a default value, a Dirichlet parameter; so, the fixed numberof topics (assume, K) can be generated in any sequence. [YGLC13]. Forexample, when a corpus is trained, using BTM or LDA, with K numberof topics, each independent training of BTM generates K number of topicdistributions, where it can not be ascertained that each time one trains thecorpus, a topic no. K = k (here, k = [1, ...,K]) will always correspond to aspecific topic (i.e., “UI component”).Researchers proposed online models (i.e., OBTM, OLDA) to circumventthe problems with streaming datasets. Here, it can be assumed that thedocuments would be generated in streams and can be collected from differenttime-slices or versions, where the documents are exchangeable in a time-slice.For example, Online BTM (OBTM) accommodates and deals with batchesof short-text documents divided into different time-slices or versions. Let’sassume that OBTM has already got the topic distribution for (t − 1)-thtime-slice. When a new batch (t-th time-slice) arrives, OBTM utilizes thetopic distribution of (t − 1)-th time-slice to set the prior topic distributionof t-th time-slice. It, in turn, ensures that after the training of the t-thtime-slice, the k-th topic in the t-th time-slice is closely related to the k-thtopic generated in the (t − 1)-th time-slice. If a completely new topic isintroduced in the latest batch of documents (t-th time-slice), the new topicwill merge into one (or more) existing topic(s) generated in the (t − 1)-thtime-slice that has (/have) strong correlation(s) with the introduced topic.[YGLC13]615.1. MOTIVATION AND OBJECTIVE5.1.4 Adaptively Online Topic Modeling for Short TextsThe problem with online topic modeling algorithms is that they do notconsider or compare the varying consequential correlation among all thepreceding time-slices or versions of the short texts while inferring topics forthe latest time-slice. For example, in OBTM, this limitation transpires asthe topic model generates the topic distribution of a time-slice by making itdirectly dependent on the topic distribution of preceding time-slices. If newtopics are being introduced in each time-slice, the k-th topic in the latesttime-slice would be significantly different than that of the first time-slice.I can not reliably compare the distribution of the K = k-th topic betweentwo non-consecutive time-slices as OBTM does not impose or investigatethe varying correlation between them.5.1.5 ObjectiveIn the proposed method, I aim to alleviate the mentioned problem byleveraging prior knowledge through the adaptive integration of the topicdistributions of all the preceding time-slices with their respective weights ascontributions, for generating the prior distribution of the latest, t-th time-slice. This way, I can warrant both coherence of specific topics and consis-tency in the topics’ sequence in all the available versions. The adaptivenessenables us to compare topic distributions of any two different time-slicesreliably.Figure 5.1: Overview of the AOBTM FrameworkIn Figure 5.1, I show an overview of the framework. Here, version taggedshort-texts are processed and fed into the AOBTM algorithm to find better625.2. METHODOLOGYtopic distribution for the latest version by leveraging preceding versions’statistical data as prior knowledge. The details of each part are discussedin Sections 5.2, 5.2.4.In the following section, I discuss the details of the Adaptive OnlineBiterm Topic Model (AOBTM). This method introduces Adaptiveness togive the online algorithm, OBTM, a version or time-slice sensitivity so thatthe prior topic distribution of the latest time-slice takes varying contribu-tions (topic distribution-wise) of the preceding time-slices into account. Af-ter setting the prior, I train the model to find out the final topic distribution.The details of the proposed method are described below.5.2 MethodologyThis section addresses the biterm extraction technique, the topic modeldescription, and its complexity analysis. It also includes the definition of twoproposed parallel algorithms that help the topic model infer better qualitytopics by determining two key hyper-parameters.5.2.1 Applied Biterm Extraction TechniqueI have adopted the definition of Biterm from [YGLC13], where it de-notes an unordered term-pair co-occurring in a small, fixed-size windowover a term sequence. The fixed-sized window is referred to as short-context. The optimal size of short-context varies from dataset to datasetand can be considered as an important parameter setting. In a given short-context, an unordered pair of any two distinct terms can form a biterm.For example, a short context with size=3, generates the following biterms:(w1, w2, w3)⇒ {(w1, w2), (w2, w3), (w1, w3)}In previous works of topic labeling (i.e., [LNKB10, MSZ07]), inferredtopics have been labeled with the term(s), which has(/have) a more signif-icant contribution to the respective topics. Here, term denotes any non-redundant word in the document, which cannot be found in Natural Lan-guage Toolkit’s (NLTK) stop-word list. But Gao et al. [GZD+19], showedthat the most contributing singular term or their combination could not ad-equately represent the respective topic. So, instead of using singular terms,I use meaningful phrases to label the topics, where a Phrase refers to twofrequently co-occurring words. To ensure the comprehensibility of the ex-tracted phrases, I use a Pointwise Mutual Information (PMI)- based phraseextraction method [NLGB10], where the higher frequency of two words’ co-occurrence warrants the generation of a more meaningful phrase. For the635.2. METHODOLOGYmodel, I have empirically set the frequency threshold to 24. After identi-fying the phrases, I convert them into single terms using ‘ ’ (i.e., w1 w2),to train them along with other terms using the algorithm. During bitermsextraction, words constructing the phrases are also considered as term whenthey appear outside identified phrases. I train the phrases to capture theirunderlying semantics, which, in turn, would help us to label the topics withthe most relevant phrases. I further demonstrate this modification’s impactin the experiment section.5.2.2 Topic Model DescriptionTo alleviate the data-sparsity problem faced by AOLDA and to cap-ture more coherent, comprehensible, and discriminative topics, I propose anadaptive online topic modeling method, AOBTM, which improves OBTMby adaptively combining the topic distributions in preceding versions. Thedetails of the proposed AOBTM method are described in figure 5.2.Figure 5.2: Overview of AOBTMAfter the preprocessing, the short texts are separated into different time-slices or versions, and input into AOBTM sequentially. AOBTM treats theshort texts-set from each time-slice as a separate corpus. I denote the wholecorpus as R = R1, R2, ..., Rt, where t indicates the t-th time-slice. Followingthe literature, I denote the prior distributions over corpus-topic as α andthe prior distributions over topic-words as β; both α and β are definedinitially. The topic-word distributions determine the topic’s distributionover all the non-redundant terms (including the phrases) that appear in thecorpus. The number of the topics is specified as K. For the k-th topic, ϕtk645.2. METHODOLOGYis the probability distribution vector over all the input terms in the t-thtime-slice. I introduce a new parameter- win (window size), which definesthe number of preceding versions to be considered for inferring the topicdistributions of the current version. The overview of the AOBTM model isdepicted in Figure 5.2. The red rectangle highlights the adaptive integrationof the topics of the win preceding versions for generating the prior β in thet-th version. Different from OBTM (and similar to AOLDA), as Figure5.2 shown, I adaptively integrate the topic distributions of the previouswin versions, denoted as ϕt−1, ϕt−2, ..., ϕt−i, ..., ϕt−win, for generating theprior, βt for the t-th version. The adaptive integration sums up the topicdistributions of different versions with different weights, γt,i:βtk =∑wini=1 γt,ik ϕt−ik + ntw|k (5.1)Here, i denotes the i-th preceding version (1 ≤ i ≤ w). ntw|k denotes thenumber of times word, w is assigned to topic k in time-slice, t. The weightγt,ik is determined by considering the similarity of the k-th topic between the(t-i)th version and the (t-1)th version, which is calculated by the followingsoftmax function:γt,ik =exp(ϕt−ik ·βt−1k )∑winj=1 exp(ϕt−jk ·βt−1k )(5.2)655.2. METHODOLOGYAlgorithm 1: Adaptive Online BTMInput : K,win, α, β,B(1), ....,B(T)Output: {ϕ(t), θ(t)}Tt=11 Set α(1) = (α, ...., α) and {β(1)k = (β, ...., β)}Kk=12 for t← 1 to T do3 Randomly assign topics to biterms in B(t);4 for iter ← 1 to Niter do5 foreach biterm bi = (wi,1, wi,2) ∈ B(t) do6 Draw topic k from Eq. 5.3;7 Update n(t)k , nwi,1(t)|k , and nwi,2(t)|k ;8 end9 Set α(t+1) by Eq.5.4;10 Set {β(t+1)k }Kk=1 by Eq.5.1;11 end12 Compute ϕ(t) by ϕk,w =nw|k+β,n·|k+Wβ; (refer to [YGLC13]);13 Compute θ(t) by θk =nk+α,NB+Kα; (refer to [YGLC13]);14 endIn Equation 5.2, [ϕt−ik · βt−1k ] represents Einstein Summation and com-putes the similarity between the topic distribution, ϕt−ik and the prior ofthe (t-1)th version, βt−ik . This adaptive aggregation allows the topics of thepreceding versions to endow different contributions to the topic distributionsof the current version. The steps are shown in Algorithm 1. In Algorithm1, B denotes the biterms collection. Here, NB is the number of biterms andbi denotes a biterm with two terms: wi,1 and wi,2 in i-th biterm. I use Was the total number of words in the vocabulary, and θ(t) as a K-dimensionalmultinomial distribution which denotes the corpus-topic distribution for atime-slice. Here, nk, nk|d, and nw|k denote number of words in topic k, num-ber of words in document d assigned to topic k, and number of times wordw is assigned to topic k, respectively.In Algorithm 1, the topics are drawn from Eq. 5.3 and prior distribution,α for the latest time-slice is calculated using Eq.5.4:P(zi = k|z(t)−i , B(t), α(t), {β(t)k }Kk=1)∝ (n(t)−i,k + α(t)k )(n−i,wi|k + β(t)k,wi)(n−i,wj |k + β(t)k,wj)[∑Ww=1(n−i,w|k + β(t)k,w)]2(5.3)665.2. METHODOLOGYα(t+1)k = α(t)k + n(t)k (5.4)where, z ∈ [1,K] refers to the topic indicator variable and P(z) refers to theprevalence of topics in the corpus. I use symmetric Dirichlet distributionsas the initial priors by setting α1 = (α, ..., α) and β1k = (β, ..., β). Given αtand [βtk]Kk=1, I iteratively draw topic assignments for each biterm bi ∈ Bt,according to the conditional distribution stated in Eq. 5.3. Once iterationsare completed, I obtain the counts ntk and ntw|k. I adjust the hyperparametersαt and [βtk]Kk=1 for time-slice (t+1) by setting α(t+1)k and βt+1k using Eq. 5.4and Eq. 5.1, respectively. The derivation of Eq. 5.3 and Eq. 5.4 can befound in [YGLC13].5.2.3 AOBTM Complexity and Comparison with BaselinesIn this section, I discuss the details of the running time and memoryrequirement for AOBTM and compare it with different batch, online, andadaptive online algorithms. I have listed the time-complexity and the num-ber of in-memory variables for different topic models in Table 5.1.In the following discussion, l¯ refers to the average document length, andND refers to the number of documents in the corpus, respectively. I canassume that all the documents in the short-text corpus have almost thesame length [YGLC13, HWL18]. It is reasonable to infer NB (number ofbiterms in the corpus) using this assumption as I am applying NB onlyfor the topic models, which are devised for short texts (i.e., BTM, OBTM,and AOBTM). According to the assumption, each document with length l¯,would produce l¯(l¯ − 1)/2 biterms; so, I have the equivalence of NB as:≈ ND · l¯ · (l¯ − 1)/2Furthermore, in Table 5.1, win denotes the user-defined window-size (thenumber of preceding versions to consider) in the adaptive online algorithms,W denotes the total number of terms, and v refers to the number of availabletime-slices.Time Complexity. The most time-consuming part in these topicmodels is the component calculating the conditional probability of topicassignments, which requires O(K) time. While LDA draws a topic foreach word occurrence, BTM draws a topic for each biterm. So, the over-all time-complexity for LDA and BTM turn out as O(NiterKND l¯) andO(NiterKNB), respectively [BNJ03, YGLC13]. From the previous assumption-based calculation of NB, I can further expand the time-complexity for BTM:O(NiterKND l¯(l¯ − 1)/2), which is approximately (l¯ − 1)/2 times the time-675.2. METHODOLOGYTable 5.1: Time Complexities and the Number of In-Memory Variables inDifferent Topic ModelsMethods Time Complexities # of Variables in MemoryLDA O(NiterKND l¯) NDK +WK +ND l¯BTM O(NiterKNB) K +WK +NBOLDA O(NiterK|N (t)D l¯(t)|) NDK +WK + |N (t)D l¯(t)|OBTM O(NiterK|N (t)B |) K +WK + |N (t)B |AOLDA O(NiterK|N (t)D l¯(t)|+ vKW ) NDK + vWK + |N (t)D l¯(t)|AOBTM O(NiterK|N (t)B |+ vKW ) K + vWK + |N (t)B |complexity of LDA. As BTM works with short texts where value of l¯ isconsiderably small, the runtime of BTM can still be compared to that ofLDA [HWL18].The online algorithms, such as OLDA and OBTM, deal with documentsand short texts, respectively, present in the latest time-slice. In Table 5.1, Ihave used superscript t to denote the latest time-slice or version. But, theadaptively online algorithms (i.e., AOLDA, AOBTM) compare and deter-mine contributions of the previous v number of topic-word distributions fordifferent time-slices, which require an additional O(vKW ) time.Number of Variables Stored in Memory. LDA maintains the fol-lowing counts as the cached memory: the number of words in a documentd assigned to topic k, nk|d (=NDK), and the number of times word w as-signed to topic k, nw|k (=WK). LDA also stores the topic assignment foreach word occurrence (=ND l¯) [Hei05]. On the other hand, BTM stores thefollowing variables: the number of topics, nk (=K), the number of timesword w assigned to topic k, nw|k (=WK), and the topic assignment for eachbiterm (=NB) [YGLC13].Unlike the batch algorithms, online topic models do not require runningover all documents (in case of OLDA), or all biterms (in case of OBTM)observed up to the latest time-slice. Instead, OLDA only iteratively runsover the words present in the current time-slice documents, whereas OBTMonly iterates over the biterm set in the latest time-slice. These online al-gorithms require almost constant memory cost to update the models, sincethe number of documents, their average length, and the number of bitermsare often stable [HWL18, HBB10].685.2. METHODOLOGYIn the adaptively online algorithms, topic-word assignments for differentversions are compared, weighted, and combined to set the prior topic-worddistribution of the latest time-slice. Therefore, the counts, nw|k (=WK)for all the preceding time-slices, need to be stored as cache-memory. Aswin ∈ [1, ..., v], I consider vWK as the counts stored in memory.From table 5.1, I observe that AOBTM’s time complexity is higher, butit is comparable to other algorithms while dealing with a fewer number ofshort texts. In practice, the number of texts is bound to decline as they areseparated into different versions or time-slices. On the other hand, AOBTMhas to store some additional variables to accommodate adaptiveness, yetincur less memory cost than the other Adaptive Online algorithm, AOLDA.5.2.4 Algorithms To Find Optimal Number Of Topics AndVersion-WindowTwo parameters in the adaptive online topic modeling method, playkey-roles in the quality of the topics discovered: (i) the number of topics toderive, (ii) and the number of preceding versions to consider for adaptiveintegration. In previous studies, the values of these crucial parameters wereset via informed guess established from the manual examinations performedover the dataset [GZLK18]. I propose algorithms to determine the values ofthese parameters automatically. Before developing algorithms to find opti-mal values for these parameters, I need to determine a suitable evaluationmetric for measuring the quality of discovered topics.Perplexity (or, marginal likelihood) evaluated on a held-out test set havebeen utilized in many studies to assess the effectiveness and efficiency of agenerated topic model [BNJ03, GWRZ07, GSBT05]. But, the minimizedperplexity as a metric is not suitable for the approach for the followingreasons. First, the mentioned studies focused on LDA-based topic modelswhere the likelihood of word occurrences in documents is optimized, whereas,in the approach, the likelihood of biterm occurrences in the latest time-sliceis optimized. Second, it was argued in [CBGG+09], that topic models withbetter held-out likelihood might infer less semantically meaningful topics,which deviates from the underlying expectations of topic models (e.g., betterinterpretability and coherence of the derived topics).For this purpose, I can use Coherence Score or PMI-Score. CoherenceScore is a metric used for measuring the quality of the discovered topicsautomatically [MWT+11]. It depicts that a topic is more coherent if themost probable words in that topic co-occur more frequently in the corpus.On the other hand, PMI-Score measures the coherence of a topic based on695.2. METHODOLOGYpoint-wise mutual information using large scale text datasets from externalsources, such as Wikipedia [NLGB10]. This idea resonates with the under-lying assumption of the approach, which maintains that words co-occurringmore frequently in an external dataset, should be more likely to belong to thesame topic. Since the external dataset is model-independent, the generatedPMI-Score would fluctuate consistently for distinct topic models with differ-ent parameter values [YGLC13]. Therefore, I exploit PMI-Score to evaluatethe discovered topic quality, which measures the pairwise association amongT most contributing words in a discovered topic, k:PMI-Score(k) = 1T (T−1)∑1≤i<j≤T logP (wi,wj)P (wi)P (wi)(5.5)Here, P (wi), P (wj), and P (wi, wj) are the probabilities of word wi,wj , and co-occurring word-pair (wi, wj), respectively. The probabilities areestimated empirically from a fixed external dataset. Following the literature[YGLC13, HWL18], I computed the PMI-Scores using 5.4 million EnglishWikipedia articles as the external dataset. I have used an open sourceweb-scraper API 16 to scrape the articles with an average length of 362.7words. To determine the overall PMI-Score for the topic model, I take theaverage of all PMI-Scores produced by distinct topics: PMI-Score(terminal)= 1K∑k PMI-Score(k).Algorithm to Determine Appropriate Topic NumberAn inadequate number of topics could render the topic model too coarseto identify distinct and particular topics. Conversely, an inordinate num-ber of topics could deliver a model that is too involved, making subjectivevalidation and interpretation difficult.16https://github.com/martin-majlis/Wikipedia-API705.2. METHODOLOGYAlgorithm 2: Optimal Number of TopicsInput : InputArr, iter, span, datasetOutput: optTopicNum1 Set maxPMI ← 0.0, optV al← InputArr[0];2 #pragma omp parallel for reduction(max:maxPMI )3 for i← 0 to InputArr.size()− 1 do4 //Each thread works on 1 element of InputArr5 PMI sum ← 0.0;6 for j ← 0 to iter − 1 do7 Set threadId← omp get thread num();8 Set topicNum← InputArr[threadId];9 Set K in Algorithm 1 with topicNum;10 {ϕ(t), θ(t)}Tt=1 ← Run AOBTM (Algorithm 1);11 PMI[K] ← new array of double;12 for t← 0 to K − 1 do13 PMI[t] ← by Eq. 5.5;14 end15 PMI score ← 1K∑k PMI-Score[k];16 PMI sum + = PMI score;17 end18 PMI final ← PMI sum / iter;19 if (PMI final > maxPMI) then20 optVal ← topicNum; maxPMI ← PMI final;21 end22 #pragma omp parallel for reduction(max:topicNum)23 for i← optV al − ⌈span/2⌉ to optV al + ⌈span/2⌉ do24 Set threadId← omp get thread num();25 Set tmp← threadId+ optV al − ⌈span/2⌉ ;26 if (tmp == optV al) then27 break;28 else29 repeat lines 4 to 6;30 Set topicNum← tmp;31 repeat lines 7 to 21;32 end33 optTopicNum← topicNum;To estimate the most appropriate number of topics for the topic model-ing approach, I propose a 2-step parallel algorithm. For the parallelization,I have employed OpenMP, a set of compiler directives and an API for theprogram (written in C++) that provides support for multi-platform, mul-tiprocessing programming in shared-memory environments. OpenMP en-abled us to write the algorithm so that the multithreading directives are715.2. METHODOLOGYskipped (or replaced with regular arguments) in the machines that do nothave OpenMP installed. The designed algorithm to determine the optimalnumber for topics inference is provided in Algorithm 2.The first step of the parallel algorithm takes an array of integers, In-putArr. This array stores candidate number of topics, such as [n1,n2,...,nt],where n is an integer and t is the array-size. If core refers to the numberof CPU-cores available, it is advisable to limit t within [2, (cores− 1)]. Thislimit warrants that only one core would be assigned for each element in thearray. Each core, in turn, builds iter number of AOBTM models and cal-culates respective PMI-Scores. If I independently train AOBTM multipletimes, on the same dataset with the same number of topics, I end up withslightly different PMI-Scores with each independent training. So, I designedthe algorithm in such a way so that, for one candidate number of topics,each core builds iter number of models and generates separate PMI-Scores.I stabilize the metric for the corresponding candidate number by taking anaverage of the distinct PMI-Scores. I find the optimal candidate number(optV al) through reduction (OpenMP operator) after all the threads finishtheir execution.In the second step of the algorithm, I finetune near the optimal candidatenumber (optV al) to determine the final value of the optimal topic number,optTopicNum. The user may specify span with an integer, which definesthe breadth of grid-search around optV al, observed in the algorithm’s firststep. Without any user specification, span is automatically set as core− 1.For each integer in the range of span around optV al, I repeat the procedureof the first step to determine the optimal value and set it as the appropriatenumber of topics.Fig. 5.3 illustrate the first and second phase of Algorithm 2, respectively.Each element of the InputArr is handled by one CPU-core. Circle-maxrepresent the reduction operator. In essence, the first phase determines theappropriate number for topics inference (optV al) by evaluating the elementsof the InputArr; the second phase determines the optimal topic number byevaluating integers around the optV al. The algorithm ensures that in eachphase, one core evaluates only one integer topic number.Algorithm Determining Number of Versions to ConsiderEarlier, I discussed how preceding versions or time-slices incur differentcontributions to the topic distributions of the latest time-slice. In AOBTM,I form the prior topic distributions for t-th time-slice by taking weightedcontributions of the previous win number of time-slices into account. Users725.2. METHODOLOGYFigure 5.3: Two Phases of Algorithm 2, , determining the optimal topicnumbercan define the parameter win ∈ [1, v − 1] in Algorithm 1, where v denotesthe available time-slices. To make an educated decision about the parameterwin, I can analyze the change in PMI Scores for different values of win.I build (v − 1) number of AOBTM models for the latest time-slice withdifferent values of win and calculate the PMI-Scores. In the parallel block, Iuse OpenMP reduction clause to find the maximized PMI-score. If OpenMPis not enabled in a machine, I store the scores in an array, where the score’sindex corresponds to the number of considered versions. Then, with onescan of the array, I determine the cutoff point where the PMI-score droppedand did not rise again.I propose Algorithm 3 to determine the appropriate number of precedingversions to consider automatically.Algorithm 3: Finding Optimal Number of Versions to ConsiderInput : dataset, v, iterOutput: optV erNum1 Set maxPMI ← 0.0, optV erNum← 1;2 #pragma omp parallel for reduction(max:maxPMI )3 for i← 1 to v − 1 do4 Set threadId← omp get thread num();5 Set win← threadId+ 1;6 /* Each thread runs AOBTM iter time by taking win number ofpreceding versions into account, calculate the PMI-Scores andsaves them in PMI final; (repeating lines 5 to 18 of Algo. 2);*/7 if (PMI final > maxPMI) then8 optVerNum ← w; maxPMI ← PMI final;9 end735.3. EXPERIMENTAL SETUP5.3 Experimental SetupIn this section, the experimental setup to evaluate the performance ofAOBTM in identifying consistent and distinctive latent topics from corporacomprising of short text documents is discussed. I explain the datasets,baselines, and evaluation metrics to compare the results of different topicmodeling algorithms.5.3.1 DatasetsTo show the effectiveness of the approach, in addition to using app re-views, I use a large dataset of Twitter microblogs. Tweets are consideredas short text and evaluation on this dataset can show the applicability ofAOBTM on short text analysis. The details of the datasets are as follows:− App Reviews from Apple Store and Google Play. I use the datasetprovided by Gao et al. [GZLK18], which is previously studied to eval-uate AOLDA for extracting topics from app reviews. The datasetincludes reviews that are related to a number of versions of the col-lected apps. The subject apps are distributed in different categoriesand platforms; this choice ensures the generalization of the approach.I enriched the provided datasets by adding the user-reviews collectedfrom the latest versions of the subject apps. However, one of the appsin this dataset, “Clean Master,” was discontinued, and I could notacquire app-changelogs. Another app in the provided dataset, namely“eBay,” had pulled enormous app-reviews from the app-stores. As thechangelogs are critical to the evaluation metrics, I have decided todiscard these two apps from the evaluation. I double-checked the pro-vided app-reviews and changelogs in the dataset from the play storesand discarded the ones that could not be found. Table 5.2 summarizesthe specifications of the app reviews datasets.− Tweets2020 is a collection of approximately 200,000 tweets scrapedfrom Twitter between January 1st and May 20th, 2020, where eachmonth is considered as a time-slice. For the collection of the tweets,I have used an open-sourced twitter-scraper 17. I used 300 top trend-ing topics over the region of North America to collect the tweets withtimestamp. Besides the content, each tweet includes user id, times-tamp, number of retweets, and likes.17https://pypi.org/project/twitter-scraper/745.3. EXPERIMENTAL SETUPThe user reviews and tweets collections contain many noisy words, suchas repetitive words, casual words, misspelled words, and non-informativewords (e.g., “normally”). I have performed common text preprocessing tech-niques including removing meaningless words, lowercasing, lemmatization,digit and name replacement following [MGLJ16]. I apply the preprocessingtechnique in for lemmatization and replace all digits with “<digit>.” I alsoremoved duplicate records and documents with a single word.Table 5.2: Subject apps from different app-storesApp Name Category Platform #Reviews #VersionsNOAA Radar Weather App Store 10,112 16Youtube Multimedia App Store 44,531 35Viber Communication Google Play 19,327 9Swiftkey Productivity Google Play 23,121 175.3.2 BaselinesI select LDA [BNJ03], BTM [YGLC13], OLDA [HBB10], OBTM [HWL18],and AOLDA [GZLK18] as the baseline methods to evaluate the performanceof AOBTM. The details of the baseline algorithms are explained in Section5.1. All the experiments were carried out on a Linux machine with Intel2.21 GHz CPU and 16G memory. Following the literature [GZLK18], I haveused all algorithms implemented by Gibbs sampling in C++ 18.5.3.3 Evaluation MetricsGood topic models deliver coherent [MSZ07] and discriminative topics,which cover unique and comprehensive aspects of the corpus [GZLK18].So, I utilized PMI-Score as a measure of coherence [YGLC13] and Discrete-ness Score (Dis Score) to measure the discriminative property of the derivedtopics, which is inspired from the semantic similarity mapping in[GZLK18].Higher values of PMI Score and Dis Score suggest the discovery of morecoherent and discriminative topics. I also presented time-cost (seconds) periteration (Time Cost in Table 5.3) as the third performance metric. I pickedthe top 10 terms from each generated topic to calculate the PMI-Scores, asexplained in section 5.2.4. For calculating Dis Score, I use Jensen Shannon18Code of BTM: http://code.google.com/p/btm/755.3. EXPERIMENTAL SETUP(JS) Divergence DJS [JS20], to estimate the difference between two topicdistributions (ϕ). The equations are provided below:Dis Score =∑Kk=1(∑Kj=1,j ̸=kDJS(ϕtk||ϕtj)K)/K; (5.6)DJS(ϕtk||ϕtj) = 12DKL(ϕtk||M) + 12DKL(ϕtj ||M); (5.7)DKL(P ||Q) =∑i P (i)logP (i)Q(i) ; M =12(ϕtk + ϕtj); (5.8)Eq. 5.7 elaborates DJS of Eq. 5.6. Eq. 5.8 defines the DKL (Kullback-Leibler Divergence) and M from Eq. 5.7. In Eq. 5.6, the innter term,∑Kj=1,j ̸=kDJS(ϕtk||ϕtj)/K measures the difference of a single topic distribu-tion’s average with the rest of the topic distributions. In Eq. 5.8, P (i) (orQ(i)) is the i-th item in P (or Q).To provide a better comparison, I adopted three more performance met-rics used in [GZLK18] to evaluate the performance of AOLDA. These metricsare PrecisionE, RecallL, and Fhybrid. Further details about these metrics arediscussed in [GZLK18]. For app reviews and Twitter data, I have taken app-changelogs and popular hashtags, respectively as the ground-truths. Here,PrecisionE measures the accuracy in detecting emerging topics in the lat-est time-slice, t [GZLK18]. RecallL evaluates whether the prioritized topics(including both emerging and non-emerging) reflect the changes mentionedin the change-logs or hashtags. Higher Fhybrid suggests that the change-logsand hashtags are more explicitly covered by detected topics. The higherscore in Fhybrid also signifies that the prioritized issues reflect more of thechange-logs and hashtags contents [GZLK18].Batch algorithms (OLDA, OBTM, AOLDA, AOBTM) capture topic evo-lutions for texts divided into different versions (app-reviews) or time-slices(Twitter). Based on the topic evolutions captured by the batch algorithms(OLDA, OBTM, AOLDA, AOBTM), I identify the anomaly topics thatpresent noticeable differences from those of the previous versions. I haveused Anomaly Discovery technique proposed by Gao et al. [GZLK18] toidentify the anomaly topics, which I regard as emerging topics. Then, Ihave taken the app-changelogs and popular hashtags as the ground-truthfor app-reviews distributed over versions and tweets over different time-slices, respectively. I manually extracted key-terms from these change-logsand hashtags to determine if these terms are covered in the emerging topics.I have used the word2vec model [MSC+13] to find the cosine similarities765.4. RESULTSbetween each key term and the emerging topics based on the models. Iconsider the key term as covered if the similarity is greater than 0.6 [II08].PrecisionE =I(E ∩G)I(Y ),RecallL =I(L ∩G)I(G)Fhybrid = 2× PrecisionE × RecallLPrecisionE +RecallLIn the equations, E, G, and L represents sets of detected emerging top-ics, the key terms from the changelogs and hashtags, and prioritized topics(covering both emerging, non-emerging topics), respectively, where I(·) isused to denote the number of the topics in ·. Here, PrecisionE measuresthe accuracy in detecting emerging topics. RecallL evaluates whether theprioritized topics (including both emerging and non-emerging) reflect thechanges mentioned in the change-logs or hashtags. The final metric, Fhybridmeasures the balance between PrecisionE and RecallL.5.4 ResultsThis chapter explores the following three research questions that answerthe three most important aspects of the proposed topic model. The focusof this section is to answer and discuss the following research questions. Areal-world application of the proposed model is also provided at the end ofthe section.RQ1: Can AOBTM achieve better performance compared toother topic modeling methods?In this research question, AOBTM is compared with other topic mod-eling approaches, including the non-adaptive and offline models using theevaluation metrics detailed in the previous section. This research questiondeals with the research objective of proposing a novel adaptive and onlinetopic modeling approach and comparing the model to understand how wellthe proposed approach works for SE texts.RQ2: How do different parameter settings, document lengths,and preprocessing approaches impact the performance of AOBTM?This research question aims to help the researchers understand the ef-fect of different hyper-parameter tuning, and how well the proposed model775.4. RESULTSperforms for different document lengths. It also put focuses on how differentpreprocessing approaches can help the model yield better performance. Asour objective was defined as to propose a topic model that works well forshort and noisy texts, such as SE texts, this research question plays a vi-tal role in determining the parameter settings and pre-processing techniquethat help the proposed topic model extract topics with best quality fromdocuments with short lengths.RQ3: Using the parameters set by the parallel algorithms, howdiscriminative and coherent are the topics discovered by differenttopic modeling methods?This research question deals with the two most important aspects of theproposed topic model: how many topics to infer and how many previoustime/version slices to consider. The literature and related studies have alsoput a lot of emphasis on these two parameters as they have the potential togreatly influence the quality of inferred topics. This research question alsoserves our objective to deliver a topic model that can extract discriminative,coherent, and relevant topics.5.4.1 RQ1: Comparison Results with Different MethodsTable 5.3 presents the evaluation results, where, PE , RL, Fh refer toPrecisionE, RecallL, and Fhybrid, respectively.During the evaluation for RQ1, I set the parameters as w = 3 andK = 10 for the adaptive online algorithms for the sake of uniformity. Ihave initialized α = 0.05 and β = 0.01 for LDA based methods as they haveachieved the best performance with these values for short texts in [YGLC13].I have set α = 50/K and β = 0.01 for BTM based algorithms [YGLC13].From Table 5.3, we observe that AOBTM delivers the highest PMI-Scores with every dataset by alleviating the data sparsity problem and con-sidering the varying contributions of different time-slices or versions. So, thetopics discovered by AOBTM are more coherent and comprehensible. Fordiscriminative topic learning, AOBTM performs better than other meth-ods, except for the Tweets2020 dataset. A large amount of short texts pertime-slice in the Tweets2020 dataset helps AOLDA to learn better documentlevel word correlations and infer more discriminative topics; still, AOLDAdid not generate higher PMI-Score than AOBTM for Tweets2020. Fromthe result, it is apparent that AOBTM exceeds the benchmark methods in-cluding its online version, OBTM, as well as AOLDA. Although AOBTM785.4. RESULTSTable 5.3: Comparison result of different methodsApp-Name Methods PMI Dis Time PE RL Fh(#Avg. Texts) Scores Scores CostTweets2020(~39,803)LDA 2.03 ± 0.04 0.68 43.27 NA NA NABTM 2.04 ± 0.02 0.61 46.8 NA NA NAOLDA 1.90 ± 0.03 0.79 17.11 0.581 0.612 0.592OBTM 1.92 ± 0.03 0.79 21.91 0.574 0.605 0.588AOLDA 2.09 ± 0.03 0.89 56.7 0.603 0.677 0.657AOBTM 2.13 ± 0.04 0.82 63.87 0.608 0.684 0.662NOAARadar(~632)LDA 1.34 ± 0.03 0.57 18.8 NA NA NABTM 1.36 ± 0.03 0.63 22.31 NA NA NAOLDA 1.38 ± 0.03 0.68 8.25 0.461 0.519 0.47OBTM 1.41 ± 0.03 0.71 14.6 0.576 0.495 0.548AOLDA 1.45 ± 0.04 0.75 23.44 0.568 0.492 0.533AOBTM 1.48 ± 0.04 0.78 20.54 0.578 0.508 0.556Youtube(~1,272)LDA 1.56 ± 0.04 0.64 20.74 NA NA NABTM 1.62 ± 0.04 0.73 25.36 NA NA NAOLDA 1.53 ± 0.03 0.76 11.51 0.439 0.455 0.448OBTM 1.55 ± 0.04 0.74 15.33 0.483 0.464 0.463AOLDA 1.61 ± 0.03 0.8 30.92 0.598 0.474 0.529AOBTM 1.66 ± 0.03 0.82 31.08 0.615 0.482 0.538Viber(~2,147)LDA 1.65 ± 0.03 0.68 29.06 NA NA NABTM 1.72 ± 0.02 0.75 33.4 NA NA NAOLDA 1.58 ± 0.03 0.8 14.47 0.458 0.395 0.419OBTM 1.62 ± 0.02 0.82 20.32 0.417 0.308 0.365AOLDA 1.76 ± 0.04 0.81 34.64 0.465 0.409 0.428AOBTM 1.89 ± 0.03 0.85 37.55 0.572 0.411 0.508Swiftkey(~1,360)LDA 1.53 ± 0.02 0.67 23.18 NA NA NABTM 1.61 ± 0.02 0.68 28.85 NA NA NAOLDA 1.42 ± 0.04 0.74 16.06 0.209 0.551 0.291OBTM 1.49 ± 0.04 0.73 21.55 0.313 0.52 0.392AOLDA 1.67 ± 0.03 0.76 29.48 0.526 0.631 0.584AOBTM 1.71 ± 0.02 0.83 31.26 0.542 0.658 0.591795.4. RESULTSmarginally improved the performance of AOLDA, the performance improve-ment of AOLDA over OLDA is approximately the same as that of AOBTMover OBTM. AOBTM also generated the highest scores for every datasetfor PrecisionE, RecallL, and Fhybrid, which indicates that the topic modelcan select emerging topics more precisely.I acknowledge that AOBTM is more time-expensive than all the otherbaselines, but the runtime is comparable to adaptive online methods whenthe dataset is small. From Table 5.3, we observe that the difference ofruntime between AOLDA and AOBTM is trivial; AOBTM even outperformsAOLDA in runtime for NOAA Radar dataset, which has the lowest numberof average short texts per version.5.4.2 RQ2: Effect of Different Parameter Settings,Document Lengths, and Preprocessing ApproachesIn this subsection, impacts of three crucial factors are discussed on theproposed topic model. These factors affect the performance of the topicmodels differently; and so, these factors merit separate protracted discus-sions.Effect of Different Parameter SettingsIn Fig. 5.4 and Fig. 5.5, I have compared the method with AOLDA, asthe number of preceding versions to consider is unique to adaptive onlinealgorithms. In Fig. 5.4, I consider distinct uniform topic-number for eachdataset and calculate PMI-Scores for the different numbers of previous ver-sions. The topic numbers for the dataset are calculated by Algorithm 2. InFig. 5.5, I consider fixed window size (number of versions) to calculate thePMI-Scores for the varying numbers of topics.I observe that AOBTM in general generates the highest PMI-Scores, andthe trendlines of both methods are analogous. In Fig. 5.4, the declines inthe performance of AOBTM (i.e., win=25 in the YouTube dataset) can tran-spire for the following reasons: the emergence of an unrelated novel topic inthe recently considered versions (i.e., from 20th to 25th versions in YouTubedataset), content drifting, and higher occurrence of meaningless texts in thenewly included versions [HWL18]. In Fig. 5.5, in all cases, the methods pro-duced an increasing number of PMI-Scores until the tipping point generatesthe highest score. Once the methods reach their peak, a further increasein the number of topics generates incoherent coinciding topics inducing thereduction in PMI-Scores.805.4. RESULTSFigure 5.4: PMI-Scores for the varying number of considered versions ortime-slices.Figure 5.5: PMI-Scores for the varying number of topics.815.4. RESULTSEffect of Document LengthIn Fig. 5.6, I have presented AOBTM’s performance using PMI-Scoreswith respect to varying document lengths. I have considered the aver-age document length for each dataset to evaluate the considered methods.Tweets2020, NOAA Radar, Youtube, Viber, and Swiftkey have average docu-ment lengths of 68.3, 8.5, 13.6, 9.4, and 6.2, respectively. AOBTM performsbetter than other methods for all datasets. It is worth noting that, as ex-pected, LDA based methods performed well with large document lengthsand bigger corpus, mostly because of the dataset’s content richness andabundance of document-level word co-occurrences.Figure 5.6: PMI-Scores for varying document lengths.Effect of PreprocessingIn section 5.2, I have mentioned a preprocessing technique with PhraseExtraction that can deliver more comprehensible top contributing terms ineach discovered topic. I implement AOBTM+ with the phrase extractionpreprocessing technique. In Table 5.4, I explicate how this technique inAOBTM+ generates better topic words than AOBTM with no phrase ex-traction. I selected a topic discovered from the Twitter dataset, which isrelated to Racism. I can see that extracting phrases during preprocessingand training them with the rest of the terms help distinguish key terms fortopic representation, which is captured only by AOBTM+.825.4. RESULTSTable 5.4: Five most contributing terms in a topic from Tweets2020Methods Key-TermsAOLDA hate, race, black, white, stopAOBTM black, hate, white, race, crimeAOBTM+ stop racism, black, stop hate, police brutality, white supremacy5.4.3 RQ3: Quality of Discovered Topics Using ParametersDetermined by Proposed AlgorithmsIn [ASVMNM10, CXL+09, DSB14, GS04], researchers have explored dif-ferent ways to finetune the topic models’ parameters. Their basic approachis to train various topic models (with different parameter settings) overseveral iterations to select one with the best performance. All the proposedprocedures are computationally expensive, especially when executed sequen-tially. Moreover, all the existing libraries and packages that implement thementioned procedures use LDA based models [Pon12]. So, I proposed twoparallel algorithms as described in Section 5.2.4 to determine 2 importantparameters in the approach automatically: i) the number of topics to dis-cover (K) and ii) the number of preceding versions/time-slices to consider(win). In Table 5.5, time-complexities are provided for both algorithms. Itis worth noting that the proposed algorithms run sequentially if the envi-ronment is not set up to perform parallelism.Table 5.5: Time Complexities for Proposed Parallel AlgorithmsTime-ComplexityAlgorithm 2 O([iter + span][NiterK|N (t)B |+ vKW ])Algorithm 3 O([iter][NiterK|N (t)B |+ vKW ])I have employed the proposed algorithms to determine the best valuesfor the parameters K and win for each dataset. For Tweets2020, Youtube,Viber, NOAA, and Swiftkey, Algorithm 2 determined the correspondingnumbers of topics to be derived as 31, 22, 18, 13, and 11, respectively,whereas Algorithm 3 determined the best win value to be considered as 5,34, 9, 16, and 11, respectively. After setting the best detected parametersfor the online and adaptively online algorithms, I have calculated the PMI-Scores for each dataset and present the results in Fig. 5.7. The plot showsthat AOBTM outperforms all the other online algorithms for all datasets.835.4. RESULTSCompared to OBTM, AOLDA and OLDA perform better for datasets thatcontain longer documents. Furthermore, OBTM outperforms both of theLDA based methods when it comes to the limited datasets containing shorttexts.Figure 5.7: PMI-Scores generated by setting the parameters determined byAlgorithms 2 and 35.4.4 Real-World ApplicationPrompt detection of emerging topics from user reviews is crucial forapp developers as these topics reveal users’ requirements, preferences, andcomplaints [GZLK18]. Developers can proactively identify user-complaintsand take quick actions to improve user experience through efficient anal-ysis of the app-reviews. Timely and precisely identifying emerging issueshelps developers to fix bugs, refine existing features, and add essential func-tions in the subsequent update of the application. For this purpose, Gaoet al. developed a framework named IDEA to detect emerging issues fromthe app-reviews of popular applications [GZLK18]. IDEA collects user re-views after the publication of different versions of the app and implementAOLDA to get the topic-word distributions for the app reviews collectedafter the publication of the latest version. I have modified the open-sourceframework IDEA and incorporated AOBTM instead of AOLDA to generatethe topic-word distribution. The rest of the framework’s components, suchas preprocessing, emerging topic identification, and topic interpretation, re-main the same. The modified version is denoted as OPRA (Online AppReview Analysis).Inspired by the zoom case study as explained in Section 1, I have col-845.4. RESULTSTable 5.6: Five most contributing terms from two sample topicsTopics IDEA OPRATopic1password zoombombmeeting passwordabuse securityattack policypolicy disturbTopic2message group chatstatus messagechannel notificationchat transferlink linkTable 5.7: PMI-Scores & Time Cost evaluation for Zoom app-reviewsFrameworks PMI Score Time Cost PrecisionE RecallL FhybridIDEA 2.08 12.52 0.572 0.608 0.586OPRA 2.35 16.8 0.593 0.619 0.608lected around 15,000 app reviews for Zoom Cloud Meetings from GooglePlay. The average review length for this dataset is 7.8. These reviews weregenerated after the publication of the latest 5 versions of the app. For emerg-ing topic detection, I set the parameters as win = 3 and K = 10 for fairevaluation. I also changed the initial values of α and β to 0.1 and 0.01, re-spectively, as these values yielded best performance for IDEA (implementingAOLDA) in [GZLK18].In Table 5.6, I have reported the top 5 most contributing words fromtwo topics generated by IDEA and OPRA: the first topic is closely relatedto app-security, and the second topic is closely related to messaging featureof the app. In Table 5.7, I have reported the corresponding PMI-Scores andTime-Cost for both frameworks. I observe that applying AOBTM in theframework slightly increases the time-cost, but generates more comprehen-sive and coherent topics.855.5. SUMMARY5.5 SummaryAn Adaptive Online Biterm Topic Model (AOBTM) is proposed in thischapter to model topics adaptively in short texts. In short texts, AOBTMalleviates the sparsity problem by considering the statistical data of previ-ous time slices. Furthermore, AOBTM uses a version-sensitive strategy toaddress the problems associated with conventional topic models. In conjunc-tion with AOBTM, the preprocessing technique allows us to capture uniqueterms within an extracted topic. AOBTM provides the highest PMI-Scoreswith every dataset since it eliminates the data sparsity problem and con-siders the varying contributions of various time slices and versions. So, thetopics discovered by AOBTM are more cohesive and discernible. For differ-ential topic learning, AOBTM is proven more effective as it surpasses theperformance of the other benchmark methods and their respective onlineequivalent. Additionally, we propose parallel algorithms for automaticallydetermining how many topics should be derived and how many previousversions to consider during topic inference. Our experiments revealed thatintegrating AOBTM with these two algorithms could find more coherenttopics and outperform the state-of-the-art baselines for app reviews andshort text datasets.86Chapter 6DiscussionBased on the results discussed in Chapters 4 and 5, this chapter confersthe implications and outcomes that may benefit different user-segments dif-ferently19. As this thesis has two main focuses: SE text classification and SEtopics modeling, we discuss their significance and connotations in differentsubsections. This section discusses the results to suggest implications fordifferent user segments. Depending on the respective fields, different usersutilize different parts of the results differently. For example, time constraintsmight be more crucial for app developers than it is for researchers; on theother hand, a small boost in the model’s performance can be deemed asmore important for the researchers than the engineers or practitioners. Thischapter tries to cater individual needs of these user segments and serve themwith focused and concrete suggestions to build robust and effective SE textclassification pipelines.6.1 Discussion on the Software Engineering TextClassificationThe motivation of our study was to help two distinct user segments toclassify SE texts in a more efficient and effective way: the app developersand the SE researchers. Thus, the implications of this study are also fur-ther divided into two categories for ease of comprehension: implications fordevelopers or engineers and implications for researchers.6.1.1 Implications For Developers and PractitionersIn the following subsections, the implications for the developers and prac-titioners are discussed on a case-by-case basis. Here, the use cases for thePrior and PTM approaches are discussed elaborately. The informative sug-gestions about the PTM and Prior approach usage scenarios and trade-offs19Here, users can be developers, practitioners, or researchers who want to use the studiedmodels to classify issues related to mobile apps from user feedbacks.876.1. DISCUSSION ON THE SOFTWARE ENGINEERING TEXT CLASSIFICATIONfor different settings are provided in the following subsections that can po-tentially help the readers make educated decisions based on their respectivescenarios. This section also includes a comparative analysis between CustomPTMs and out-of-the-box PTMs, which will guide developers to understandwhen to pre-train the PTMs extensively with domain specific datasets.Prior approachesDeep learning with word embedding can be used for app issue classifica-tion when one does not want to use PTMs. From the Prior approaches, DeepLearning Technique leveraging Word Embedding (DL+WE) generated thebest results for predicting the correct classes of app reviews (RQ1), followedclosely by the Ensemble Method. DL+WE also took the least amount oftime to predict the classes, whereas SUR-Miner took the second least amountof time. However, if no labels are available to train the mentioned models,AR-Miner should be used, which utilizes topic modeling to separate theclasses of app reviews.Prior approaches vs. PTMsIf the prediction time does not matter much but high performance is re-quired, PTMs should be used to classify app issues. From the results of RQ1,we observed that all PTMs outperformed the Prior approaches (except for 1out of 28 cases) to predict classes accurately. RoBERTa and ALBERT scoredthe highest F1 scores in most of the datasets among D1 to D7. Comparedwith the fastest prior approach, ALBERT has the lowest, and XLNET hasthe highest increase in prediction time. Consequently, it is worth applyingPTMs for app-review classification if the developer, researcher, or softwareengineer can spare marginally additional time to achieve higher predictionscores.PTMs can be used for all settings when higher performance is required,except for the multi-resource setting, where the deep learning model with wordembedding has better scores. For the multi-class classification (RQ1), PTMsperform significantly better than the Prior approaches for most datasets.For Dataset D2, the difference in the scores among the two groups is notmuch, but when I consider D7, the results of PTMs are almost 10 F1 scorehigher than the best Prior Approach. The prediction times also vary, but,some PTMs can predict the labels at approximately the same time as thebest Prior approaches (Ensemble and DL+WE). For binary classification,the best Prior approaches achieve scores of below 90% and mostly in the low886.1. DISCUSSION ON THE SOFTWARE ENGINEERING TEXT CLASSIFICATION80% F1 score. But, the F1 scores of PTMs are mainly above 90% and theCustom PTMs have even higher scores. However, for multi-resource setting(RQ3-4), the Ensemble and DL+WE have better F1 scores than all PTMs,but not higher than CPTMs. This result is interesting, showing that thelearned knowledge about the app review domain for issue classification isimportant in this setting, no matter if the model is trained on App Storedata and tested on Twitter data or vice versa. This is confirmed by theobtained results of the C-RoBERTa-LARGE, which have the highest scoresamong all models in this setting.Binary vs. Multi-classWhen possible, the classification task should be turned into a binary clas-sification to achieve higher scores. All PTMs and Prior approaches per-formed better in binary classification than in multi-class classification. FromTables A.5 and A.6, we observe that converting or trimming a multi-classdataset to a binary class dataset helps all the prior and PTM approachesto yield better performance. Prior approaches are improved by 0 to ∼3%,whereas the gain for the PTMs ranges from 1 to ∼7%. For binary classifica-tion, all PTM approaches performed better than they did for multi-class clas-sifications, and the performance boost produced by readily-available PTMsis proportionate to the performance boost produced by Custom PTMs.Zero-shotIn a zero-shot setting, the Roberta-based Custom PTMs or RoBERTa-LARGE-MNLI can be used to classify app issues. All PTMs can be imple-mented in a Zero-Shot setting and yield adequate performance. The onlyPrior approach that can be used directly in this setting is AR-Miner whichis based on topic modeling. All the readily available and Custom PTMsare fine-tuned on the NLI dataset for the Inference classification task. Dueto the fact that these PTM models had not been exposed to any instancesof the datasets in the study in the zero-shot setting, they are expected toperform worse than their reported performance in Tables A.2, A.3, and A.4.The Custom-PTMs pre-trained with domain-specific knowledge could notoutperform the readily available RoBERTa-LARGE-MNLI. This can be re-lated to the dataset used for its training, which is a much larger datasetthan the NLI datasets I used to train the PTMs. Although the CPTMscannot beat the RoBERTa-LARGE-MNLI model, it is notable that theseCPTMs are smaller models and the RoBERTa-LARGE-MNLI model is a896.1. DISCUSSION ON THE SOFTWARE ENGINEERING TEXT CLASSIFICATIONvery large language model; but still, the CPTMs I studied achieve significantimprovements compared to their general domain PTMs in this setting.Custom PTMs vs. PTMsCustom PTMs are the best models among all that should be used toachieve better prediction scores with lower prediction times. CustomizedPTMs that are pre-trained with the app review data has the best perfor-mance in all settings: binary classification, multi-class classification, multi-task, and multi-resource setting. Even for zero-shot setting, they performbetter than all the other models (except for RoBERTa-LARGE-MNLI whichis trained on a much larger NLI data). The benefit of using Custom PTMs,specifically RoBERTa-based ones, compared to PTMs, is more obvious forthe multi-task and multi-resource setting in RQ3-3 and RQ3-4. Interest-ingly, the PTMs do not perform well in the multi-task and multi-resourcesetting, while CPTMs have the highest scores. The PTMs in these two set-tings perform lower than the Ensemble and DL+WE models. But, the bestCPTMs achieve scores of above 90% for multi-task and 86% and 92% F1scores for multi-resource. Other than achieving the best score, CPTMs havethe lowest prediction time among all models for multi-class classification,multi-task, and multi-resource settings. They also achieve the lowest pre-diction time for some of the datasets in binary classification. The results forRQ3-4 confirm that using over-the-shelf PTMs might not be the best optionwhen classifying issues for app reviews in all settings. PTMs can still havegood performance in binary and multi-class classification. But for harderscenarios (i.e., zero-shot, multi-task, and multi-resource), pre-training themwith domain-specific data increases their performance and reduces their pre-diction time. Moreover, the more app review data is used in the pre-trainingof CPTMs, the better their performances are.6.1.2 Implications For ResearchersThe two most important parts of this study for the researchers are dis-cussed below. Further exploration of the Custom PTM building approach,Multi-class, and zero-shot classification approaches can help the researchersto improve them and illuminate new insights into app review classification.Training Custom PTMsResearch for incorporating app review data in the pre-training of the cur-rent PTMs is needed. The inclusion of app reviews in the pre-training corpus906.2. DISCUSSION ON SOFTWARE ENGINEERING TOPIC MODELINGallows PTMs to classify user feedback with higher F1 scores. I have pre-trained three models of both ALBERT and RoBERTa with three differentsizes of corpora. This experiment found that more domain-specific infor-mation during the pre-training helps the newly trained PTMs yield betterpredictions of app reviews. C-ALBERT and C-RoBERTa models can im-prove the ALBERT and RoBERTa scores for up to ∼13% and ∼14%, respec-tively. Here, I followed the work of Wada et al. [WTM+20] to incorporateapp review data for pre-training of the PTMs. However, there are mul-tiple research on training the PTMs with domain-specific data [CFM+20,HP19]. Other approaches can be investigated for domain-specific modelsthat achieve higher scores in the settings of RQ3.Multi-class Classification and Zero-shot SettingsResearch needed to build models that perform better in the multi-classand zero-shot settings. The results showed that the studied models per-form better in a balanced binary classification setting, rather than an im-balanced multi-class setting. This suggests that more research is needed todevelop models that perform better when multiple labels are available andthe dataset is imbalanced. Moreover, researchers can work on a methodto increase the performance of the models when classifying with new labelsthat the model has not seen are required.6.2 Discussion on Software Engineering TopicModelingThe performance of the adaptive online topic model and general guide-lines for the users are discussed in this section. It will help the users toadopt the correct metrics, ground truths, and external datasets while mea-suring the performance of the model applied to any text corpus; otherwise,the significance of the derived latent topics may be misrepresented or mis-interpreted.6.2.1 Performance of AOBTMBy alleviating the data sparsity issue and considering the different con-tributions of different time-slices or versions, AOBTM produces the highestPMI-Scores with every dataset. AOBTM discovers topics that are coher-ent, comprehensible, and discriminative. The result shows that AOBTMexceeds the benchmark methods, including its online version, OBTM, and916.3. SUMMARYAOLDA. The highest scores in PrecisionE, RecallL, and Fhybrid for everydataset indicate that the topic model can select emerging topics more pre-cisely and reflect more information in the prioritized topics. While AOBTMis more time-intensive than all the other baselines, its runtime is comparableto adaptive online methods when the dataset is small.6.2.2 Guidelines for the UsersAfter generating deterministic scores to evaluate the results against thebenchmarks, the outcomes of the considered topic models were manuallyreviewed to confirm that AOBTM generates more coherent keywords forextracted topics. Users are advised to use formal human evaluations withintermediate to high-level experts to assess the derived topics, as we havefound that a closer look at the topics may disclose a lot of latent informationto the evaluators. Although our experiments have confirmed that AOBTMworks well with both time and version-sliced datasets, users of the algorithmmust take precautions while building those slices, as highly unbalanced slicesmay result in performance degradation. While measuring the extensibility ofthe topic model, we compared the online algorithms for detecting prioritizedemerging topics. We used app changelogs as ground truths for the appreviews, but different ground truths must be adopted for short text modelingfrom different domains. A model’s performance will surely suffer if theground truths are not cohesive with each slice.6.3 SummaryIn essence, the chapter provides informed suggestions to different usersegments based on respective usage scenarios. For app issue classificationwith Prior approaches, DL+WE is suggested for its accuracy and speed,while AR-Miner is recommended when no training labels are available.PTMs can be used to classify app issues with high performance when predic-tion time is not a concern, except for multi-resource settings. The chapteralso suggests that turning the classification task into a binary classificationleads to higher scores. The RoBERTa-based Custom PTMs are more suit-able for use in a Zero-Shot setting and Custom PTMs pre-trained with appreview data performs the best among all models with the lowest predictiontime, and are recommended for classifying app issues in all settings overover-the-shelf PTMs. For app review topic modeling, AOBTM discovers co-herent, comprehensible, and discriminative topics while having comparableruntime to adaptive online methods for small datasets.92Chapter 7ConclusionThis thesis conducted an extensive exploratory study comparing appissue classification tools and pre-trained Transformer-based models in var-ious settings. A novel adaptive topic modeling algorithm, AOBTM is alsoproposed, which can discover coherent and discriminative topics from shorttexts.In the first part of the thesis, experiments are conducted on six avail-able datasets and a highly imbalanced dataset, a combination of the sixdatasets. Domain-specific PTMs were trained using different sizes of appreview data collected from Google Play, and these Customized PTMs werealso employed on all the datasets. The results confirm that PTMs are achiev-ing higher scores in both binary and multi-class classification compared toPrior approaches, but the readily available PTMs are not always the bestmodels to be used in all scenarios. Instead, CPTMs have the highest scoresand can perform better than other models in all settings. Moreover, in-corporating app-specific data in pre-training PTMs reduces the predictiontime. The findings of the study are surmised as follows: (i) PTMs aregenerally superior to Prior approaches when predicting classes with higherscores across all datasets; (ii) The performance of the Pre-Trained Modelsbenefits from incorporating more domain-specific data in the pre-trainingphase; (iii) Both Prior and PTM models yield better performance for bi-nary classification tasks than they do for multi-class settings. PTMs are thebest choice for zero-shot classification settings; (iv) For multi-task and multi-resource settings, the readily available PTMs and Custom PTMs showed theperformance order as: Readily Available PTM < Custom-BASE < Custom-MEDIUM < Custom-LARGE; (v) RoBERTa-based Custom PTMs have thebest scores among all models. They also have lower prediction times in themulti-resource setting compared to Prior approaches.In the second part of the thesis, AOBTM addresses the problems withconventional topic models by adopting a version-sensitive strategy. Alongwith AOBTM, a preprocessing technique was used that enables capturingdistinguishable terms in an extracted topic. Moreover, two parallel algo-rithms are also implemented to automatically determine the value of my937.1. THREATS TO VALIDITYmodel’s two most important parameters. The results of several experimentson app reviews and Twitter datasets confirm the performance of AOBTMcompared to the state-of-the-art algorithms.In the following two sections, the threats to the validity of these twostudies (along with how these issues were tackled) and future works areoutlined elaborately.7.1 Threats to ValidityThe threats to validity of this thesis are divided into two subsectionsbased on their respective study: the first subsection focuses on the threatsto classification of SE texts using PTMs, and the second study focuses onthe adaptive online topic modeling for SE texts. These subsections alsoinclude how these threats were neutralized.7.1.1 Study on Assisting Developers to Better ClassifyUser FeedbackThis subsection is further divided into three parts to outline the threatsto validity in a more comprehensive manner. As this is a comprehensiveempirical study, the overall threats have been categorized as: internal, con-struct, and external threats to validity and delineated with their respectivecircumventions.Internal ValidityA possible internal validity can be related to the obtained results. Tomitigate this threat, I use stratified k-fold cross-validation to avoid the biasthat might be introduced to the results by the test set. In addition, to mit-igate threats to the validity of the results, hyper-parameter values are keptthe same for all PTMs and Deep learning models in all the fine-tuning steps.Also, I prevent overfitting by using early stopping and a higher dropout rate.I also keep the values of alpha and beta for LDA as reported in AR-Miner.I run all the experiments on a single machine, and I report the machineconfiguration to enforce the reproducibility of the results. Furthermore, Iconsider the same metrics to compare the PTMs with Prior approaches.Along with the generated raw duration, I also provide the change of timein percentage as compared to the baseline. The adopted stricter micro andmacro-metrics diminish the ambiguity while providing comparisons amongthe considered approaches’ associated results.947.1. THREATS TO VALIDITYConstruct ValidityThe selection of the Prior approaches and PTMs can pose a validitythreat to the study. I identified the four most common approaches as priorsby examining the highly practiced methods, tools, and techniques employedby researchers and application developers. The considered approaches in-clude different Machine Learning algorithms, Ensemble methods, and DeepLearning Approaches for app issue classification tasks and are selected af-ter conducting a literature review. The PTMs were adopted by followingprevious study [ZXT+20] that conducted an empirical study on PTMs per-formance for sentiment analysis in Software Engineering. Additionally, Ipre-trained transformer-based models by incorporating app-review-relateddatasets with the previously used generic dataset to better understand thepotential of domain-specific PTMs (CPTM-X) for app issue classification. An-other threat to the study can be related to dataset D7. This dataset is amerge of other datasets and has labels borrowed from them. Although Iconsulted the definitions of the labels from their publishers, there might bea chance that the samples in one group of a dataset are closely related tothe samples from another label of another dataset, thus, affecting the re-sults. I mitigate this threat by combining labels from different datasets thathave similar names. I also used stratified k-fold cross-validation to alleviatepotential threats.External ValidityIn this work, I empirically study the ability of the PTMs in issue clas-sification of app reviews. The app review classification mainly focuses onextracting useful information in the context of software engineering; whichcan be used by app developers for requirements engineering, release plan-ning, and other tasks as mentioned in the thesis. The study is thereforelimited to app review classification for software engineers and I do not studyPTMs for other purposes such as finding issues from business perspectives[Tan19], product reviews [ZGC+17], and applications such as intention min-ing [HXLM18]. I also do not study the summarization of relevant issuesfrom app reviews. However, I conduct the study on six different datasets.In addition, by merging them, I run experiments on D7 which has multipleclasses covering different aspects of apps useful for app developers. Anotherthreat to the generalizability of the results lies in the different tasks I con-sidered. I mitigate this threat by studying the sentiment classification andcategory classification of app reviews in the multitask setting. Although957.1. THREATS TO VALIDITYPTMs might be useful for other tasks, I do not study them in this work. Toensure the generalizability of the study in the specified scope, I have incor-porated datasets that include app reviews from diverse app categories andfrom two platforms i.e., Google Play and Apple App Store. The datasetshave various sizes and different labels. In addition, I experiment with issueclassification from another platform, i.e. Twitter, which has been shownto require different models than App Stores due to the differences of theplatforms, noise data, and user feedback.7.1.2 Study on Adaptive Online Bi-term Topic ModelThe threats to the validity of this study can be traced to four distinctaspects: human evaluation, dataset curation, ground truth selection, andmemory-time tradeoff. All these aspects are described below in separatesub-sections.Human EvaluationIn the experiments, I have only used deterministic scores to evaluatethe results against the benchmarks. My supervisor and I have manuallyreviewed the outcomes of the considered topic models. I have consultedwith fellow researchers about the model outcome and have considered theopinions of industry developers which confirmed that AOBTM generatesmore coherent key-words for extracted topics. But I did not perform anyformal human evaluation to assess how well the model performs in practicecompared to others.DatasetsThe proposed topic model derives topics from the text corpus distributedover different versions or time-slices. Evaluating the models using onlyversion-tagged app-reviews from mobile applications that have multiple pub-lished versions in the app-store would not give us a precise idea of how thisalgorithm works with time-slices. I have evaluated the approach using a fewmobile applications, which might affect the generality of the model. To mit-igate the problem, I have incorporated a new dataset with around 200,000timeline-tagged tweets scraped by considering 300 top trending topics fromCanada and the USA. Furthermore, I carefully selected apps so that I coulddemonstrate our topic model’s performance for apps that have a small orlarge number of reviews per version (~623-2,147)967.2. FUTURE WORKSGround TruthTo measure the extensibility of the topic model, I wanted to know howit scales to other online algorithms for prioritizing topics and detectingemerging ones. For selecting ground-truth, I have used key-terms fromapp-changelogs for app-reviews, as Gao et al. did in [GZLK18]. In or-der to calculate precision, recall, and F-score, app-changelogs are used asground truth, similar to [GZLK18]. However, I did not have any changelogsor tweet-summary to take as ground-truth for Tweets2020. I have man-ually selected top-trending hashtags over different time-slices to mine thekey-terms. Other approaches for evaluation should be studied.Memory and Time CostI acknowledge that the model cannot compare to the benchmarks whenit comes to memory and time-cost. Still, I am currently endeavoring toincorporate word-cooccurrence pattern algorithm to make the topic modelfaster while using significantly fewer resources.7.2 Future WorksThe studies discussed in this thesis have the potential for further im-provements and extensions that can expose new avenues of research anddevelopment. In the following subsections, future works about these studiesare discussed that contain valuable hints of new research directions or ideasthat can help developers, engineers, and researchers alike.7.2.1 Study on Assisting Developers to Better ClassifyUser FeedbackI conducted an extensive exploratory study comparing app issue classifi-cation tools and pre-trained Transformer-based models in various settings. Iconducted the experiments on six available datasets and a highly imbalanceddataset, which is a combination of the six datasets. Domain-specific PTMswere trained using different sizes of app review data I collected from GooglePlay and these customized PTMs were also studied here. The results confirmthat PTMs are achieving higher scores in binary and multi-class classifica-tion compared to Prior approaches, but the over-the-shelf PTMs are notalways the best models to be used in all scenarios. Instead, CPTMs havethe highest scores and are able to perform better than other models in all977.2. FUTURE WORKSsettings. Moreover, incorporating app-specific data in the pre-training ofPTMs reduces the prediction time. One of the future directions of this re-search is assessing domain-specific PTMs in other areas of app reviews andexploring ways to increase performance in a zero-shot setting.7.2.2 Study on Adaptive Online Bi-term Topic ModelIn this chapter, I proposed a novel adaptive topic modeling algorithm,AOBTM, which is able to discover coherent and discriminative topics fromshort texts. AOBTM addresses the problems with conventional topic mod-els by adopting a version-sensitive strategy. Along with AOBTM, I usea preprocessing technique that enables capturing distinguishable terms inan extracted topic. Moreover, I implemented two parallel algorithms todetermine the value of the two most important parameters of the model au-tomatically. The results of several experiments on app reviews and Twitterdatasets confirm the performance of AOBTM compared to the state-of-the-art algorithms.The underlying BTM method can be improved using short text expan-sion and concept drifting detection and integrated with a topic visualizationtool specifically designed for app reviews. For the parallel algorithms, theGPU cores and shared memory cache can be leveraged to make the programrun faster. Word-cooccurrence pattern algorithms can also be incorporatedto make the topic model faster while using significantly fewer resources.98Bibliography[ABK17] Miltiadis Allamanis, Marc Brockschmidt, and MahmoudKhademi. Learning to represent programs with graphs.arXiv preprint arXiv:1711.00740, 2017. → pages 9[AHNS21] Assem Al-Hawari, Hassan Najadat, and Raed Shatnawi.Classification of application reviews into software mainte-nance tasks using data mining techniques. Software QualityJournal, 29(3):667–703, 2021. → pages 20[AJM17] Mohamed Ali, Mona Erfani Joorabchi, and Ali Mesbah.Same app, different app stores: A comparative study. In 2017IEEE/ACM 4th International Conference on Mobile Soft-ware Engineering and Systems (MOBILESoft), pages 79–90.IEEE, 2017. → pages 32[Ala18] Jay Alammar. The illustrated transformer, jay alammar,2018. → pages xi, 14, 15[ARTL19] Ashutosh Adhikari, Achyudh Ram, Raphael Tang, andJimmy Lin. Docbert: Bert for document classification. arXivpreprint arXiv:1904.08398, 2019. → pages 43[ARXS20] Naila Aslam, Waheed Yousuf Ramay, Kewen Xia, andNadeem Sarwar. Convolutional neural network based clas-sification of app reviews. IEEE Access, 8:185619–185628,2020. → pages 9, 22, 32[ASGM18] Rahul Aralikatte, Giriprasad Sridhara, Neelamadhav Gan-tayat, and Senthil Mani. Fault in your stars: an analysis ofandroid app reviews. In Proceedings of the acm india jointinternational conference on data science and management ofdata, pages 57–66, 2018. → pages 9[ASSB+19] Afnan A Al-Subaihin, Federica Sarro, Sue Black, LiciaCapra, and Mark Harman. App store effects on software99Bibliographyengineering practices. IEEE Transactions on Software Engi-neering, 47(2):300–319, 2019. → pages 32[ASVMNM10] R. Arun, V. Suresh, C. E. Veni Madhavan, and M. N.Narasimha Murthy. On finding the natural number oftopics with latent dirichlet allocation: Some observations.In Proceedings of the 14th Pacific-Asia Conference on Ad-vances in Knowledge Discovery and Data Mining - VolumePart I, PAKDD’10, page 391–402, Berlin, Heidelberg, 2010.Springer-Verlag. → pages 83[BG17] Elsa Bakiu and Emitza Guzman. Which feature is unusable?detecting usability and user experience issues from user re-views. In 2017 IEEE 25th international requirements engi-neering conference workshops (REW), pages 182–187. IEEE,2017. → pages 31, 32[BGB08] Jordan Boyd-Graber and David Blei. Syntactic topic models.In Proceedings of the 21st International Conference on Neu-ral Information Processing Systems, NIPS’08, page 185–192,Red Hook, NY, USA, 2008. Curran Associates Inc. → pages58[BKPVS20] Eeshita Biswas, Mehmet Efruz Karabulut, Lori Pollock, andK Vijay-Shanker. Achieving reliable sentiment analysis inthe software engineering domain using bert. In 2020 IEEEInternational Conference on Software Maintenance and Evo-lution (ICSME), pages 162–173. IEEE, 2020. → pages 22[BLC19] Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: A pre-trained language model for scientific text. arXiv preprintarXiv:1903.10676, 2019. → pages 22[Ble12] David M. Blei. Probabilistic topic models. Commun. ACM,55(4):77–84, April 2012. → pages 60[BLVBC+14] Gabriele Bavota, Mario Linares-Vasquez, Carlos EduardoBernal-Cardenas, Massimiliano Di Penta, Rocco Oliveto,and Denys Poshyvanyk. The impact of api change-and fault-proneness on the user ratings of android apps. IEEE Trans-actions on Software Engineering, 41(4):384–407, 2014. →pages 30100Bibliography[BNJ03] David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latentdirichlet allocation. J. Mach. Learn. Res., 3(null):993–1022,March 2003. → pages 7, 38, 58, 61, 67, 69, 75[BPP+17] Paulo Bicalho, Marcelo Pita, Gabriel Pedrosa, Anisio Lac-erda, and Gisele L. Pappa. A general framework to expandshort text for topic modeling. Inf. Sci., 393(C):66–81, jul2017. → pages 11[BW19] Enkhbold Bataa and Joshua Wu. An investigation of transferlearning-based sentiment analysis in japanese. arXiv preprintarXiv:1905.09642, 2019. → pages 10[BWB20] Andrew R Besmer, Jason Watson, and M Shane Banks. In-vestigating user perceptions of mobile app privacy: An anal-ysis of user-submitted app reviews. International Journalof Information Security and Privacy (IJISP), 14(4):74–91,2020. → pages 20, 21[CBGG+09] Jonathan Chang, Jordan Boyd-Graber, Sean Gerrish, ChongWang, and David M. Blei. Reading tea leaves: How humansinterpret topic models. In Proceedings of the 22nd Inter-national Conference on Neural Information Processing Sys-tems, NIPS’09, page 288–296, Red Hook, NY, USA, 2009.Curran Associates Inc. → pages 69[CCCS19] Alfonso Cimasa, Anna Corazza, Carmen Coviello, andGiuseppe Scanniello. Word embeddings for comment coher-ence. In 2019 45th Euromicro Conference on Software Engi-neering and Advanced Applications (SEAA), pages 244–251.IEEE, 2019. → pages 9[CFM+20] Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasio-tis, Nikolaos Aletras, and Ion Androutsopoulos. Legal-bert:The muppets straight out of law school. arXiv preprintarXiv:2010.02559, 2020. → pages 22, 24, 91[Chu17] Kenneth Ward Church. Word2vec. Natural Language Engi-neering, 23(1):155–162, 2017. → pages 9[CJN19] Stephane Clinchant, Kweon Woo Jung, and VassilinaNikoulina. On the use of bert for neural machine transla-tion. arXiv preprint arXiv:1909.12744, 2019. → pages 10101Bibliography[CLH+14] Ning Chen, Jialiu Lin, Steven CH Hoi, Xiaokui Xiao, andBoshen Zhang. Ar-miner: mining informative reviews fordevelopers from mobile app marketplace. In Proceedings ofthe 36th international conference on software engineering,pages 767–778, 2014. → pages 8, 20, 30, 40, 57[CSPG17] Adelina Ciurumelea, Andreas Schaufelbu¨hl, SebastianoPanichella, and Harald C. Gall. Analyzing reviews and codeof mobile apps for better release planning. In 2017 IEEE24th International Conference on Software Analysis, Evolu-tion and Reengineering (SANER), pages 91–102, 2017. →pages 20, 21, 32[CW13] Laura Carren˜o and Kristina Winbladh. Analysis of user com-ments: An approach for software requirements evolution.In International Conference on Software Engineering, pages582–591, 05 2013. → pages 57[CXL+09] Juan Cao, Tian Xia, Jintao Li, Yongdong Zhang, and ShengTang. A density-based method for adaptive lda model se-lection. Neurocomput., 72(7–9):1775–1781, March 2009. →pages 83[CYZ+19] Wei-Cheng Chang, Hsiang-Fu Yu, Kai Zhong, Yiming Yang,and Inderjit Dhillon. X-bert: extreme multi-label text clas-sification with using bidirectional encoder representationsfrom transformers. arXiv preprint arXiv:1905.02331, 2019.→ pages 43[DCLT18] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and KristinaToutanova. Bert: Pre-training of deep bidirectionaltransformers for language understanding. arXiv preprintarXiv:1810.04805, 2018. → pages 10, 16, 33, 41[DHR17] Roger Deocadez, Rachel Harrison, and Daniel Rodriguez.Preliminary study on applying semi-supervised learning toapp store analysis. In Proceedings of the 21st InternationalConference on Evaluation and Assessment in Software En-gineering, pages 320–323, 2017. → pages 21[DLPS19] Jacek Dabrowski, Emmanuel Letier, Anna Perini, and An-gelo Susi. Finding and analyzing app reviews related to spe-cific features: A research preview. In International Working102BibliographyConference on Requirements Engineering: Foundation forSoftware Quality, pages 183–189. Springer, 2019. → pages32[DPAM18] Venkatesh T Dhinakaran, Raseshwari Pulle, Nirav Ajmeri,and Pradeep K Murukannaiah. App review analysis via ac-tive learning: reducing supervision effort without compro-mising classification accuracy. In 2018 IEEE 26th interna-tional requirements engineering conference (RE), pages 170–181. IEEE, 2018. → pages 21[DSB14] Romain Deveaud, Eric SanJuan, and Patrice Bellot. Accu-rate and effective latent concept modeling for ad hoc infor-mation retrieval. Document nume´rique, 17(1):61–84, 2014.→ pages 83[DSPA+16] Andrea Di Sorbo, Sebastiano Panichella, Carol V Alexandru,Junji Shimagaki, Corrado A Visaggio, Gerardo Canfora, andHarald C Gall. What would users change in my app? sum-marizing app reviews for recommending software changes. InProceedings of the 2016 24th ACM SIGSOFT InternationalSymposium on Foundations of Software Engineering, pages499–510, 2016. → pages 30[DSPA+17] Andrea Di Sorbo, Sebastiano Panichella, Carol V Alexan-dru, Corrado A Visaggio, and Gerardo Canfora. Surf: sum-marizer of user reviews feedback. In 2017 IEEE/ACM 39thInternational Conference on Software Engineering Compan-ion (ICSE-C), pages 55–58. IEEE, 2017. → pages 31[EBA19] Sergey Edunov, Alexei Baevski, and Michael Auli. Pre-trained language model representations for language genera-tion. arXiv preprint arXiv:1903.09722, 2019. → pages 10[FHJ+14] Anthony Finkelstein, Mark Harman, Yue Jia, William Mar-tin, Federica Sarro, and Yuanyuan Zhang. App store anal-ysis: Mining app stores for relationships between customer,business and technical characteristics. RN, 14(10):24, 2014.→ pages 30[FLL+13] Bin Fu, Jialiu Lin, Lei Li, Christos Faloutsos, Jason Hong,and Norman Sadeh. Why people hate your app: Making103Bibliographysense of user feedback in a mobile app store. In Proceedings ofthe 19th ACM SIGKDD international conference on Knowl-edge discovery and data mining, pages 1276–1284, 2013. →pages 8[FLLY21] Liming Fu, Peng Liang, Xueying Li, and Chen Yang. A ma-chine learning based ensemble method for automatic multi-class classification of decisions. In Evaluation and Assess-ment in Software Engineering, pages 40–49. arxiv, 2021. →pages 8[FS10] George Forman and Martin Scholz. Apples-to-applesin cross-validation studies: pitfalls in classifier perfor-mance measurement. Acm Sigkdd Explorations Newsletter,12(1):49–57, 2010. → pages 40[GAS16] Emitza Guzman, Rana Alkadhi, and Norbert Seyff. A needlein a haystack: What do twitter users say about software?In 2016 IEEE 24th International Requirements EngineeringConference (RE), pages 96–105. IEEE, 2016. → pages 32[GAS17] Emitza Guzman, Rana Alkadhi, and Norbert Seyff. An ex-ploratory study of twitter messages about software applica-tions. Requirements Engineering, 22(3):387–412, 2017. →pages 45[GCP+18] Giovanni Grano, Adelina Ciurumelea, Sebastiano Panichella,Fabio Palomba, and Harald C Gall. Exploring the integra-tion of user feedback in automated testing of android applica-tions. In 2018 IEEE 25Th international conference on soft-ware analysis, evolution and reengineering (SANER), pages72–83. IEEE, 2018. → pages 20[GEHB15] Emitza Guzman, Muhammad El-Haliby, and Bernd Bruegge.Ensemble methods for app review classification: An ap-proach for software evolution (n). In 2015 30th IEEE/ACMInternational Conference on Automated Software Engineer-ing (ASE), pages 771–776. IEEE, 2015. → pages 8, 20, 28,31, 32, 39[GIG17] Emitza Guzman, Mohamed Ibrahim, and Martin Glinz. Alittle bird told me: Mining tweets for requirements and soft-ware evolution. In IEEE 25th International Requirements104BibliographyEngineering Conference (RE), pages 11–20, Lisbon, Portu-gal, 2017. IEEE. → pages 32[GK15] Xiaodong Gu and Sunghun Kim. “what parts of your appsare loved by users?”. In Proceedings of the 30th IEEE/ACMInternational Conference on Automated Software Engineer-ing, ASE ’15, page 760–770. IEEE Press, 2015. → pages 8,20, 27, 30, 39, 57[GM14] Emitza Guzman and Walid Maalej. How do users like thisfeature? a fine grained sentiment analysis of app reviews.In 2014 IEEE 22nd international requirements engineeringconference (RE), pages 153–162. Ieee, 2014. → pages 8, 31[GOG+19] Bin Guo, Yi Ouyang, Tong Guo, Longbing Cao, and ZhiwenYu. Enhancing mobile app user understanding and market-ing with heterogeneous crowdsourced data: a review. IEEEAccess, 7:68557–68571, 2019. → pages 8[Gra12] Alex Graves. Long short-term memory. Supervised sequencelabelling with recurrent neural networks, pages 37–45, 2012.→ pages 10[GS04] Thomas L Griffiths and Mark Steyvers. Finding scientifictopics. Proceedings of the National academy of Sciences,101(suppl 1):5228–5235, 2004. → pages 83[GS20] Hui Guo and Munindar P Singh. Caspar: extracting and syn-thesizing user stories of problems from app reviews. In 2020IEEE/ACM 42nd International Conference on Software En-gineering (ICSE), pages 628–640. IEEE, 2020. → pages 27[GSBT05] Thomas L Griffiths, Mark Steyvers, David M Blei, andJoshua B Tenenbaum. Integrating topics and syntax. InAdvances in neural information processing systems, pages537–544, 2005. → pages 69[GWRZ07] Amit Gruber, Yair Weiss, and Michal Rosen-Zvi. Hiddentopic markov models. In Artificial intelligence and statistics,pages 163–170, 2007. → pages 69[GZD+19] Cuiyun Gao, Wujie Zheng, Yuetang Deng, David Lo, JichuanZeng, Michael R. Lyu, and Irwin King. Emerging app issue105Bibliographyidentification from user feedback: Experience on wechat. InProceedings of the 41st International Conference on SoftwareEngineering: Software Engineering in Practice, ICSE-SEIP’19, page 279–288. IEEE Press, 2019. → pages 57, 63[GZL+18] Cuiyun Gao, Jichuan Zeng, David Lo, Chin-Yew Lin,Michael R Lyu, and Irwin King. Infar: Insight extractionfrom app reviews. In Proceedings of the 2018 26th ACMJoint Meeting on European Software Engineering Conferenceand Symposium on the Foundations of Software Engineering,pages 904–907, 2018. → pages 57[GZLK18] Cuiyun Gao, Jichuan Zeng, Michael R. Lyu, and Irwin King.Online app review analysis for identifying emerging issues. InProceedings of the 40th International Conference on SoftwareEngineering, ICSE ’18, page 48–58, New York, NY, USA,2018. Association for Computing Machinery. → pages 8, 11,57, 58, 59, 69, 74, 75, 76, 84, 85, 97[HBB10] Matthew D. Hoffman, David M. Blei, and Francis Bach. On-line learning for latent dirichlet allocation. In Proceedingsof the 23rd International Conference on Neural InformationProcessing Systems - Volume 1, NIPS’10, page 856–864, RedHook, NY, USA, 2010. Curran Associates Inc. → pages 11,58, 60, 68, 75[HBH18] Safwat Hassan, Cor-Paul Bezemer, and Ahmed E Hassan.Studying bad updates of top free-to-download apps in thegoogle play store. IEEE Transactions on Software Engineer-ing, 2018. → pages 57[HD10] Liangjie Hong and Brian D. Davison. Empirical study oftopic modeling in twitter. In Proceedings of the First Work-shop on Social Media Analytics, SOMA ’10, page 80–88, NewYork, NY, USA, 2010. Association for Computing Machin-ery. → pages 58, 60[Hei05] Gregor Heinrich. Parameter estimation for text analysis.Technical report, Technical report, 2005. → pages 68[HF20a] M. Hadi and F. H. Fard. AOBTM: Adaptive Online BitermTopic Modeling for Version Sensitive Short-texts Analysis.106BibliographyIn International Conference on Software Maintenance andEvolution (ICSME), pages 593–604, USA, 2020. IEEE Com-puter Society. → pages v, 8, 26[HF20b] Mohammad Abdul Hadi and Fatemeh Hendijani Fard. Re-viewviz: assisting developers perform empirical study on en-ergy consumption related reviews for mobile applications. InProceedings of the IEEE/ACM 7th International Conferenceon Mobile Software Engineering and Systems, pages 27–30,2020. → pages 9, 20, 22[HF21] Mohammad Abdul Hadi and Fatemeh H. Fard. Evaluatingpre-trained models for user feedback analysis in software en-gineering: A study on classification of app-reviews. CoRR,abs/2104.05861, 2021. → pages v[HFS+21] Pablo Restrepo Henao, Jannik Fischbach, Dominik Spies, Ju-lian Frattini, and Andreas Vogelsang. Transfer learning formining feature requests and bug reports from tweets and appstore reviews. In 2021 IEEE 29th International RequirementsEngineering Conference Workshops (REW), pages 80–86.IEEE, 2021. → pages 9[HHC+19] Daojing He, Kai Hong, Yao Cheng, Zongli Tang, and MohsenGuizani. Detecting promotion attacks in the app marketusing neural networks. IEEE Wireless Communications,26(4):110–116, 2019. → pages 9[HM13] Haibo He and Yunqian Ma. Imbalanced Learning: Founda-tions, Algorithms, and Applications. Wiley-IEEE Press, 1stedition, 2013. → pages 40[Hof17] Thomas Hofmann. Probabilistic latent semantic indexing.SIGIR Forum, 51(2):211–218, August 2017. → pages 58[HP19] Kai Hakala and Sampo Pyysalo. Biomedical named entityrecognition with multilingual bert. In Proceedings of The5th Workshop on BioNLP Open Shared Tasks, pages 56–61,2019. → pages 10, 91[HR18] Jeremy Howard and Sebastian Ruder. Universal languagemodel fine-tuning for text classification. arXiv preprintarXiv:1801.06146, 2018. → pages 10107Bibliography[HS19] Fatemeh Hemmatian and Mohammad Karim Sohrabi. Asurvey on classification techniques for opinion mining andsentiment analysis. Artificial intelligence review, 52(3):1495–1545, 2019. → pages 45[HS20] Danny Hakim and Natasha Singer. New york attorney gen-eral looks into zoom’s privacy practices, March 2020. →pages 57[HSM21] Marlo Haering, Christoph Stanik, and Walid Maalej. Auto-matically matching bug reports with related app reviews. In2021 IEEE/ACM 43rd International Conference on SoftwareEngineering (ICSE), pages 970–981. IEEE, 2021. → pages 9[HWL18] Xuegang Hu, Haiyan Wang, and Peipei Li. Online bitermtopic model based short text stream classification usingshort text expansion and concept drifting detection. Pat-tern Recognition Letters, 116, 10 2018. → pages 58, 67, 68,70, 75, 80[HXLM18] Qiao Huang, Xin Xia, David Lo, and Gail C Murphy. Au-tomating intention mining. IEEE Transactions on SoftwareEngineering, 46(10):1098–1119, 2018. → pages 95[HYT+22] Mohammad Abdul Hadi, Imam Nur Bani Yusuf, FerdianThung, Kien Gia Luong, Lingxiao Jiang, Fatemeh Hendi-jani Fard, and David Lo. On the effectiveness of pretrainedmodels for api learning. ArXiv, abs/2204.03498, 2022. →pages vi[II08] Aminul Islam and Diana Inkpen. Semantic text similarity us-ing corpus-based word similarity and string similarity. ACMTrans. Knowl. Discov. Data, 2(2), July 2008. → pages 77[IM10] Dr. MD Rashedul Islam and Tridib Mazumder. Mobile ap-plication and its global impact. International Journal of En-gineering & Technology, 10:72–78, 01 2010. → pages 20[IS19] Kenji Imamura and Eiichiro Sumita. Recycling a pre-trainedbert encoder for neural machine translation. In Proceedingsof the 3rd Workshop on Neural Generation and Translation,pages 23–31, 2019. → pages 10108Bibliography[JJZS19] Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits.Is bert really robust? natural language attack on text clas-sification and entailment. arXiv preprint arXiv:1907.11932,2, 2019. → pages 10[JLZ+11] Ou Jin, Nathan N. Liu, Kai Zhao, Yong Yu, and Qiang Yang.Transferring topical knowledge from auxiliary long texts forshort text clustering. In Proceedings of the 20th ACM Inter-national Conference on Information and Knowledge Man-agement, CIKM ’11, page 775–784, New York, NY, USA,2011. Association for Computing Machinery. → pages 10[JMK13] Mona Erfani Joorabchi, Ali Mesbah, and Philippe Kruchten.Real challenges in mobile app development. In 2013ACM/IEEE International Symposium on Empirical SoftwareEngineering and Measurement, pages 15–24. IEEE, 2013. →pages 21[JS20] Johan Jensen and Claude Shannon. Jensen–shannon diver-gence, April 2020. → pages 76[JSABM17] Timo Johann, Christoph Stanik, Alireza M. Alizadeh B.,and Walid Maalej. Safe: A simple approach for feature ex-traction from app descriptions and app reviews. In 2017IEEE 25th International Requirements Engineering Confer-ence (RE), pages 21–30, 2017. → pages 30[JWHT13] Gareth James, Daniela Witten, Trevor Hastie, and RobertTibshirani. An introduction to statistical learning, volume112. Springer, 2013. → pages 40[JZC+19] Ying Ju, Fubang Zhao, Shijie Chen, Bowen Zheng, XuefengYang, and Yunfeng Liu. Technical report on conversationalquestion answering. arXiv preprint arXiv:1909.10772, 2019.→ pages 10[JZY+19] Qiang Jipeng, Qian Zhenyu, Li Yun, Yuan Yunhao, andWu Xindong. Short text topic modeling techniques, applica-tions, and performance: A survey, 2019. → pages 17, 60[KK18] Anureet Kaur and Kulwant Kaur. Systematic literature re-view of mobile application development and testing effort109Bibliographyestimation. Journal of King Saud University-Computer andInformation Sciences, 2018. → pages 20, 21[KRP21] Akbar Karimi, Leonardo Rossi, and Andrea Prati. Adversar-ial training for aspect-based sentiment analysis with bert. In2020 25th International Conference on Pattern Recognition(ICPR), pages 8797–8803. IEEE, 2021. → pages 10[LBZL19] Xin Li, Lidong Bing, Wenxuan Zhang, and Wai Lam. Ex-ploiting bert for end-to-end aspect-based sentiment analysis.arXiv preprint arXiv:1910.00883, 2019. → pages 10[LCG+19] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, KevinGimpel, Piyush Sharma, and Radu Soricut. Albert: A litebert for self-supervised learning of language representations.arXiv preprint arXiv:1909.11942, 2019. → pages 10, 16, 35[LCH+20] Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen,Yu Wang, Hoifung Poon, and Jianfeng Gao. Adversarialtraining for large neural language models. arXiv preprintarXiv:2004.08994, 2020. → pages 10[LDW+17] Chenliang Li, Yu Duan, Haoran Wang, Zhiqian Zhang, AixinSun, and Zongyang Ma. Enhancing topic modeling for shorttexts with auxiliary word embeddings. ACM Trans. Inf.Syst., 36(2), August 2017. → pages 11[LHT+21] Kien Luong, Mohammad Hadi, Ferdian Thung, FatemehFard, and David Lo. FACOS: finding API relevant con-tents on stack overflow with semantic and syntactic analysis.CoRR, abs/2111.07238, 2021. → pages v[LL17] Mengmeng Lu and Peng Liang. Automatic classification ofnon-functional requirements from augmented app user re-views. In Proceedings of the 21st International Conference onEvaluation and Assessment in Software Engineering, pages344–353, 2017. → pages 20, 21, 27, 32[LL19] Yang Liu and Mirella Lapata. Text summarization with pre-trained encoders. arXiv preprint arXiv:1908.08345, 2019. →pages 10110Bibliography[LMG+20] Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, andXipeng Qiu. Bert-attack: Adversarial attack against bertusing bert. arXiv preprint arXiv:2004.09984, 2020. → pages10[LNKB10] Jey Han Lau, David Newman, Sarvnaz Karimi, and Timo-thy Baldwin. Best topic word selection for topic labelling.In Proceedings of the 23rd International Conference on Com-putational Linguistics: Posters, COLING ’10, page 605–613,USA, 2010. Association for Computational Linguistics. →pages 63[LOG+19] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, MandarJoshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettle-moyer, and Veselin Stoyanov. Roberta: A robustly optimizedbert pretraining approach. arXiv preprint arXiv:1907.11692,2019. → pages 10, 16, 22, 33, 35[LRS+18] Liyuan Liu, Xiang Ren, Jingbo Shang, Jian Peng, and Ji-awei Han. Efficient contextualized representation: Lan-guage model pruning for sequence labeling. arXiv preprintarXiv:1804.07827, 2018. → pages 10[LTMC14] Tianyi Lin, Wentao Tian, Qiaozhu Mei, and Hong Cheng.The dual-sparse topic model: Mining focused topics and fo-cused terms in short text. In Proceedings of the 23rd Inter-national Conference on World Wide Web, WWW ’14, page539–550, New York, NY, USA, 2014. Association for Com-puting Machinery. → pages 60[LWZ+16] Chenliang Li, Haoran Wang, Zhiqian Zhang, Aixin Sun, andZongyang Ma. Topic modeling for short texts with auxiliaryword embeddings. In Proceedings of the 39th InternationalACM SIGIR Conference on Research and Development inInformation Retrieval, SIGIR ’16, page 165–174, New York,NY, USA, 2016. Association for Computing Machinery. →pages 11[LZC+15] Wei Liu, Ge Zhang, Jun Chen, Yuze Zou, andWenchao Ding.A measurement-based study on application popularity in an-droid and ios app stores. In Proceedings of the 2015 Work-shop on Mobile Big Data, Mobidata ’15, page 13–18, New111BibliographyYork, NY, USA, 2015. Association for Computing Machin-ery. → pages 20[MGLJ16] Y. Man, C. Gao, M. R. Lyu, and J. Jiang. Experience report:Understanding cross-platform app issues from user reviews.In 2016 IEEE 27th International Symposium on SoftwareReliability Engineering (ISSRE), pages 138–149, 2016. →pages 75[MIG+21] Rohan Reddy Mekala, Asif Irfan, Eduard C Groen, AdamPorter, and Mikael Lindvall. Classifying user requirementsfrom online feedback in small dataset environments usingdeep learning. In 2021 IEEE 29th International Require-ments Engineering Conference (RE), pages 139–149. IEEE,2021. → pages 9[MJJM19] Montassar Ben Messaoud, Ilyes Jenhani, Nermine Ben Je-maa, and Mohamed Wiem Mkaouer. A multi-label activelearning approach for mobile app user review classification.In International Conference on Knowledge Science, Engi-neering and Management, pages 805–816. Springer, 2019. →pages 8[MKNS16] Walid Maalej, Zijad Kurtanovic´, Hadeer Nabil, andChristoph Stanik. On the automatic classification of appreviews. Requirements Engineering, 21(3):311–331, 2016. →pages 32, 43, 45[MLVBC+16] Kevin Moran, Mario Linares-Va´squez, Carlos Bernal-Ca´rdenas, Christopher Vendome, and Denys Poshyvanyk.Automatically discovering, reporting and reproducing an-droid application crashes. In 2016 IEEE international con-ference on software testing, verification and validation (icst),pages 33–44. IEEE, 2016. → pages 30[MLVBCP15] Kevin Moran, Mario Linares-Va´squez, Carlos Bernal-Ca´rdenas, and Denys Poshyvanyk. Auto-completing bug re-ports for android applications. In Proceedings of the 201510th Joint Meeting on Foundations of Software Engineering,ESEC/FSE 2015, page 673–686, New York, NY, USA, 2015.Association for Computing Machinery. → pages 30112Bibliography[MN15] Walid Maalej and Hadeer Nabil. Bug report, feature request,or simply praise? on automatically classifying app reviews.In 2015 IEEE 23rd international requirements engineeringconference (RE), pages 116–125. IEEE, 2015. → pages 8,13, 20, 21, 27, 57[MRS+21] Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran BenVeyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre,Ilana Heinz, and Dan Roth. Recent advances in natural lan-guage processing via large pre-trained language models: Asurvey. arXiv preprint arXiv:2111.01243, 2021. → pages 16[MSBX13] Rishabh Mehrotra, Scott Sanner, Wray Buntine, and Lex-ing Xie. Improving lda topic models for microblogs viatweet pooling and automatic labeling. In Proceedings ofthe 36th International ACM SIGIR Conference on Researchand Development in Information Retrieval, SIGIR ’13, page889–892, New York, NY, USA, 2013. Association for Com-puting Machinery. → pages 10[MSC+13] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado,and Jeffrey Dean. Distributed representations of words andphrases and their compositionality. In Proceedings of the 26thInternational Conference on Neural Information ProcessingSystems - Volume 2, NIPS’13, page 3111–3119, Red Hook,NY, USA, 2013. Curran Associates Inc. → pages 76[MSJ+16] William Martin, Federica Sarro, Yue Jia, Yuanyuan Zhang,and Mark Harman. A survey of app store analysis for soft-ware engineering. IEEE transactions on software engineer-ing, 43(9):817–847, 2016. → pages 8, 32, 57[MSZ07] Qiaozhu Mei, Xuehua Shen, and ChengXiang Zhai. Auto-matic labeling of multinomial topic models. In Proceedings ofthe 13th ACM SIGKDD International Conference on Knowl-edge Discovery and Data Mining, KDD ’07, page 490–499,New York, NY, USA, 2007. Association for Computing Ma-chinery. → pages 63, 75[MWT+11] David Mimno, Hanna M. Wallach, Edmund Talley, MiriamLeenders, and Andrew McCallum. Optimizing semantic co-herence in topic models. In Proceedings of the Conference on113BibliographyEmpirical Methods in Natural Language Processing, EMNLP’11, page 262–272, USA, 2011. Association for Computa-tional Linguistics. → pages 69[NCR18] Maleknaz Nayebi, Henry Cho, and Guenther Ruhe. Appstore mining is not enough for app improvement. EmpiricalSoftware Engineering, 23(5):2764–2794, Oct 2018. → pages8[NGL18] Nicole Novielli, Daniela Girardi, and Filippo Lanubile. Abenchmark study on sentiment analysis for software engi-neering research. In 2018 IEEE/ACM 15th InternationalConference on Mining Software Repositories (MSR), pages364–375. IEEE, 2018. → pages 40[NLGB10] David Newman, Jey Han Lau, Karl Grieser, and TimothyBaldwin. Automatic evaluation of topic coherence. In Hu-man Language Technologies: The 2010 Annual Conferenceof the North American Chapter of the Association for Com-putational Linguistics, HLT ’10, page 100–108, USA, 2010.Association for Computational Linguistics. → pages 63, 70[NLM99] Kamal Nigam, John Lafferty, and Andrew McCallum. Usingmaximum entropy for text classification. In IJCAI-99 work-shop on machine learning for information filtering, volume 1,pages 61–67. Stockholom, Sweden, 1999. → pages 8[NZZ19] Ehsan Noei, Feng Zhang, and Ying Zou. Too Many User-Reviews, What Should App Developers Look at First? IEEETransactions on Software Engineering, 2019. → pages 57[PABP17] Matthew E Peters, Waleed Ammar, Chandra Bhagavat-ula, and Russell Power. Semi-supervised sequence tag-ging with bidirectional language models. arXiv preprintarXiv:1705.00108, 2017. → pages 10[PDSG+15] Sebastiano Panichella, Andrea Di Sorbo, Emitza Guzman,Corrado A Visaggio, Gerardo Canfora, and Harald C Gall.How can i improve my app? classifying user reviews for soft-ware maintenance and evolution. In 2015 IEEE internationalconference on software maintenance and evolution (ICSME),pages 281–290. IEEE, 2015. → pages 30114Bibliography[PLVB+15] Fabio Palomba, Mario Linares-Vasquez, Gabriele Bavota,Rocco Oliveto, Massimiliano Di Penta, Denys Poshyvanyk,and Andrea De Lucia. User reviews matter! tracking crowd-sourced reviews to support evolution of successful apps.In 2015 IEEE international conference on software mainte-nance and evolution (ICSME), pages 291–300. IEEE, 2015.→ pages 30, 57[PLVB+18] Fabio Palomba, Mario Linares-Va´squez, Gabriele Bavota,Rocco Oliveto, Massimiliano Di Penta, Denys Poshyvanyk,and Andrea De Lucia. Crowdsourcing user reviews to sup-port the evolution of mobile apps. Journal of Systems andSoftware, 137:143–162, 2018. → pages 31[PM13] Dennis Pagano and Walid Maalej. User feedback in the app-store: An empirical study. In 2013 21st IEEE internationalrequirements engineering conference (RE), pages 125–134.IEEE, 2013. → pages 57[PNH08] Xuan-Hieu Phan, Le-Minh Nguyen, and Susumu Horiguchi.Learning to classify short and sparse text & web with hiddentopics from large-scale data collections. In Proceedings of the17th International Conference on World Wide Web, WWW’08, page 91–100, New York, NY, USA, 2008. Association forComputing Machinery. → pages 10[Pon12] Martin Ponweiser. Latent dirichlet allocation in r. WU Vi-enna University Journal, 2012. → pages 83[PSC+17] Fabio Palomba, Pasquale Salza, Adelina Ciurumelea, Se-bastiano Panichella, Harald Gall, Filomena Ferrucci, andAndrea De Lucia. Recommending and localizing changerequests for mobile apps based on user reviews. In 2017IEEE/ACM 39th International Conference on Software En-gineering (ICSE), pages 106–117. IEEE, 2017. → pages 31[PSM14] Jeffrey Pennington, Richard Socher, and Christopher DManning. Glove: Global vectors for word representation.In Proceedings of the 2014 conference on empirical methodsin natural language processing (EMNLP), pages 1532–1543,2014. → pages 9115Bibliography[PTT+19] Gede Artha Azriadi Prana, Christoph Treude, FerdianThung, Thushari Atapattu, and David Lo. Categorizing thecontent of github readme files. Empirical Software Engineer-ing, 24(3):1296–1327, 2019. → pages 32[QKGP15] Xiaojun Quan, Chunyu Kit, Yong Ge, and Sinno Jialin Pan.Short and sparse text topic modeling via self-aggregation.In Proceedings of the 24th International Conference on Ar-tificial Intelligence, IJCAI’15, page 2270–2276. AAAI Press,2015. → pages 11[QSX+20] Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, NingDai, and Xuanjing Huang. Pre-trained models for naturallanguage processing: A survey. Science China TechnologicalSciences, 63(10):1872–1897, 2020. → pages 9, 16, 21[QWAF20] Zhilei Qiao, Alan Wang, Alan Abrahams, and Weiguo Fan.Deep learning-based user feedback classification in mobileapp reviews. Association for Information Systems, 2020. →pages 9[RCM19] Siva Reddy, Danqi Chen, and Christopher D Manning. Coqa:A conversational question answering challenge. Transactionsof the Association for Computational Linguistics, 7:249–266,2019. → pages 10[RJ19] Romain Robbes and Andrea Janes. Leveraging small soft-ware engineering data sets with pre-trained neural networks.In 2019 IEEE/ACM 41st International Conference on Soft-ware Engineering: New Ideas and Emerging Results (ICSE-NIER), pages 29–32. IEEE, 2019. → pages 10[RMA+20] Furqan Rustam, Arif Mehmood, Muhammad Ahmad,Saleem Ullah, Dost Muhammad Khan, and Gyu Sang Choi.Classification of shopify app user reviews using novel multitext features. IEEE Access, 8:30234–30244, 2020. → pages8[RP17] Sebastian Ruder and Barbara Plank. Learning to selectdata for transfer learning with bayesian optimization. arXivpreprint arXiv:1707.05246, 2017. → pages 25116Bibliography[RSB+19] Nils Reimers, Benjamin Schiller, Tilman Beck, JohannesDaxenberger, Christian Stab, and Iryna Gurevych. Clas-sification and clustering of arguments with contextualizedword embeddings. arXiv preprint arXiv:1906.09821, 2019.→ pages 32[RSOE19] Alexander Rietzler, Sebastian Stabinger, Paul Opitz, andStefan Engl. Adapt or get left behind: Domain adaptationthrough bert language model finetuning for aspect-targetsentiment classification. arXiv preprint arXiv:1908.11860,2019. → pages 10[RZLL16] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, andPercy Liang. Squad: 100,000+ questions for machine com-prehension of text. arXiv preprint arXiv:1606.05250, 2016.→ pages 10[RZZJ16] Yafeng Ren, Yue Zhang, Meishan Zhang, and DonghongJi. Improving twitter sentiment classification using topic-enriched multi-prototype word embeddings. In ThirtiethAAAI conference on artificial intelligence, 2016. → pages32[SASH+15] Federica Sarro, Afnan A. Al-Subaihin, Mark Harman, YueJia, William Martin, and Yuanyuan Zhang. Feature lifecy-cles as they spread, migrate, remain, and die in app stores.In 2015 IEEE 23rd International Requirements EngineeringConference (RE), pages 76–85, 2015. → pages 8, 30[SBR+17] Simone Scalabrino, Gabriele Bavota, Barbara Russo, Massi-miliano Di Penta, and Rocco Oliveto. Listening to the crowdfor the release planning of mobile apps. IEEE Transactionson Software Engineering, 45(1):68–86, 2017. → pages 32[SDFS20] Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and NeelSundaresan. Intellicode compose: Code generation usingtransformer. In Proceedings of the 28th ACM Joint Meet-ing on European Software Engineering Conference and Sym-posium on the Foundations of Software Engineering, pages1433–1443, 2020. → pages 22[SFSA17] Melanie Stade, Farnaz Fotrousi, Norbert Seyff, and OliverAlbrecht. Feedback gathering from an industrial point of117Bibliographyview. In 2017 IEEE 25th International Requirements Engi-neering Conference (RE), pages 71–79. IEEE, 2017. → pages32[SGG21] Camila Costa Silva, Matthias Galster, and Fabian Gilson.Topic modeling in software engineering research. EmpiricalSoftware Engineering, 26(6):1–62, 2021. → pages 7[SHM19] Christoph Stanik, Marlo Haering, and Walid Maalej. Clas-sifying multilingual user feedback using traditional ma-chine learning and deep learning. In 2019 IEEE 27th in-ternational requirements engineering conference workshops(REW), pages 220–226. IEEE, 2019. → pages 9, 13, 20, 21,22, 27, 32, 39, 40, 45[SHQ19] Chi Sun, Luyao Huang, and Xipeng Qiu. Utilizing bert foraspect-based sentiment analysis via constructing auxiliarysentence. arXiv preprint arXiv:1903.09588, 2019. → pages10[SHY+20] Lichao Sun, Kazuma Hashimoto, Wenpeng Yin, Akari Asai,Jia Li, Philip Yu, and Caiming Xiong. Adv-bert: Bert isnot robust on misspellings! generating nature adversarialsamples on bert. arXiv preprint arXiv:2003.04985, 2020. →pages 10[SPS+20] Agus Sulistya, Gede Artha Azriadi Prana, Abhishek Sharma,David Lo, and Christoph Treude. Sieve: Helping developerssift wheat from chaff via cross-platform analysis. EmpiricalSoftware Engineering, 25(1):996–1030, 2020. → pages 9[SSP18] Faiz Ali Shah, Kairit Sirts, and Dietmar Pfahl. Simple appreview classification with only lexical features. In ICSOFT,pages 146–153, 2018. → pages 8[SSP21] Rahul Kumar Singh, Manoj Kumar Sachan, and RB Patel.360 degree view of cross-domain opinion classification: a sur-vey. Artificial Intelligence Review, 54(2):1385–1506, 2021. →pages 32[SSRW21] Ishan Mani Subedi, Maninder Singh, Vijayalakshmi Ra-masamy, and Gursimran Singh Walia. Application of back-translation: a transfer learning approach to identify ambigu-118Bibliographyous software requirements. In Proceedings of the 2021 ACMSoutheast Conference, pages 130–137, 2021. → pages 10[Sta20] Christoph Stanik. Requirements intelligence: On the anal-ysis of user feedback. PhD thesis, Staats-und Univer-sita¨tsbibliothek Hamburg Carl von Ossietzky, 2020. → pages32[SWWP+21] Justyna Sarzynska-Wawer, Aleksander Wawer, Aleksan-dra Pawlak, Julia Szymanowska, Izabela Stefaniak, MichalJarkiewicz, and Lukasz Okruszek. Detecting formal thoughtdisorder by deep contextualized word representations. Psy-chiatry Research, 304:114135, 2021. → pages 10[SYTP85] Vincent Yun Shen, Tze-jie Yu, Stephen M. Thebaut, andLorri R. Paulsen. Identifying error-prone software—an em-pirical study. IEEE Transactions on Software Engineering,1(4):317–324, 1985. → pages 10[Tan19] Ailie KY Tang. A systematic literature review and analy-sis on mobile apps in m-commerce: Implications for futureresearch. Electronic Commerce Research and Applications,37:100885, 2019. → pages 95[TDG20] Ioannis Triantafyllou, Ioannis C Drivas, and Georgios Gi-annakopoulos. How to utilize my app reviews? a novel top-ics extraction machine learning schema for strategic businesspurposes. Entropy, 22(11):1310, 2020. → pages 8[THW+20] Ming Tu, Kevin Huang, Guangtao Wang, Jing Huang, Xi-aodong He, and Bowen Zhou. Select, answer and explain:Interpretable multi-hop reading comprehension over multi-ple documents. In Proceedings of the AAAI Conference onArtificial Intelligence, volume 34, pages 9073–9080, 2020. →pages 10[VBR+16] Lorenzo Villarroel, Gabriele Bavota, Barbara Russo, RoccoOliveto, and Massimiliano Di Penta. Release planning ofmobile apps based on user reviews. In 2016 IEEE/ACM 38thInternational Conference on Software Engineering (ICSE),pages 14–24. IEEE, 2016. → pages 30119Bibliography[VNNP+17] Thanh Van Nguyen, Anh Tuan Nguyen, Hung Dang Phan,Trong Duc Nguyen, and Tien N Nguyen. Combiningword2vec with revised vector space model for better coderetrieval. In 2017 IEEE/ACM 39th International Confer-ence on Software Engineering Companion (ICSE-C), pages183–185. IEEE, 2017. → pages 9[VSP+17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko-reit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and IlliaPolosukhin. Attention is all you need. Advances in neuralinformation processing systems, 30, 2017. → pages 10, 14,16[WDS+19] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chau-mond, Clement Delangue, Anthony Moi, Pierric Cistac, TimRault, Re´mi Louf, Morgan Funtowicz, et al. Huggingface’stransformers: State-of-the-art natural language processing.arXiv preprint arXiv:1910.03771, 2019. → pages 36[WFK+19] Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, andSameer Singh. Universal adversarial triggers for attackingand analyzing nlp. arXiv preprint arXiv:1908.07125, 2019.→ pages 10[WM06] Xuerui Wang and Andrew McCallum. Topics over time: Anon-markov continuous-time model of topical trends. In Pro-ceedings of the 12th ACM SIGKDD International Conferenceon Knowledge Discovery and Data Mining, KDD ’06, page424–433, New York, NY, USA, 2006. Association for Com-puting Machinery. → pages 58[WS+21] Janu Akrama Wardhana, Yuliant Sibaroni, et al. Aspectlevel sentiment analysis on zoom cloud meetings app reviewusing lda. Jurnal RESTI (Rekayasa Sistem Dan TeknologiInformasi), 5(4):631–638, 2021. → pages 8[WTM+20] Shoya Wada, Toshihiro Takeda, Shiro Manabe, Shozo Kon-ishi, Jun Kamohara, and Yasushi Matsumura. A pre-trainingtechnique to localize medical bert and enhance biobert.arXiv preprint arXiv:2005.07202, 2020. → pages 22, 24, 42,91120Bibliography[WWL+19] Chong Wang, Tao Wang, Peng Liang, Maya Daneva, andMarten van Sinderen. Augmenting app review with appchangelogs: An approach for app review classification. InSEKE, pages 398–512, 2019. → pages 32[WWWX20] Jianyu Wang, Rui Wen, Chunming Wu, and Jian Xiong.Analyzing and detecting adversarial spam on a large-scaleonline app review system. In Companion Proceedings of theWeb Conference 2020, pages 409–417, 2020. → pages 9[WZL+18] Chong Wang, Fan Zhang, Peng Liang, Maya Daneva, andMarten van Sinderen. Can app changelogs improve require-ments classification from app reviews? an exploratory study.In Proceedings of the 12th ACM/IEEE International Sympo-sium on Empirical Software Engineering and Measurement,pages 1–4, 2018. → pages 32[WZZ+19] Xing Wu, Tao Zhang, Liangjun Zang, Jizhong Han, andSonglin Hu. ” mask and infill”: Applying maskedlanguage model to sentiment transfer. arXiv preprintarXiv:1908.08039, 2019. → pages 10[XLSY19] Hu Xu, Bing Liu, Lei Shu, and Philip S Yu. Bert post-training for review reading comprehension and aspect-basedsentiment analysis. arXiv preprint arXiv:1904.02232, 2019.→ pages 10[XX13] Pengtao Xie and Eric P. Xing. Integrating document cluster-ing and topic modeling. In Proceedings of the Twenty-NinthConference on Uncertainty in Artificial Intelligence, UAI’13,page 694–703, Arlington, Virginia, USA, 2013. AUAI Press.→ pages 60[XYX15] Pengtao Xie, Diyi Yang, and Eric Xing. Incorporating wordcorrelation knowledge into topic modeling. In Proceedingsof the 2015 Conference of the North American Chapter ofthe Association for Computational Linguistics: Human Lan-guage Technologies, pages 725–734, Denver, Colorado, May–June 2015. Association for Computational Linguistics. →pages 60[YDY+19] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell,Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized121Bibliographyautoregressive pretraining for language understanding. Ad-vances in neural information processing systems, 32, 2019.→ pages 10, 16, 33, 34[YGLC13] Xiaohui Yan, Jiafeng Guo, Yanyan Lan, and Xueqi Cheng.A Biterm Topic Model for Short Texts. In Proceedings of the22nd International Conference on World Wide Web, WWW’13, page 1445–1456, USA, 2013. ACM. → pages 11, 18, 58,60, 61, 63, 66, 67, 68, 70, 75, 78[YGZ+21] Tianyi Yang, Cuiyun Gao, Jingya Zang, David Lo, andMichael Lyu. Tour: Dynamic topic and sentiment analysisof user reviews for assisting app release. In Companion Pro-ceedings of the Web Conference 2021, pages 708–712, 2021.→ pages 8[YHR19] Wenpeng Yin, Jamaal Hay, and Dan Roth. Benchmarkingzero-shot text classification: Datasets, evaluation and entail-ment approach. arXiv preprint arXiv:1909.00161, 2019. →pages 22, 44[YMO18] Xiao Yang, Craig Macdonald, and Iadh Ounis. Using wordembeddings in twitter election classification. InformationRetrieval Journal, 21(2):183–207, 2018. → pages 32[YNTT11] Koji Yatani, Michael Novati, Andrew Trusty, and Khai NTruong. Review spotlight: a user interface for summarizinguser-generated reviews using adjective-noun word pairs. InProceedings of the SIGCHI conference on human factors incomputing systems, pages 1541–1550, 2011. → pages 31[YQZ+18] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio,William W Cohen, Ruslan Salakhutdinov, and Christo-pher D Manning. Hotpotqa: A dataset for diverse, ex-plainable multi-hop question answering. arXiv preprintarXiv:1809.09600, 2018. → pages 10[YW14] Jianhua Yin and Jianyong Wang. A dirichlet multinomialmixture model-based approach for short text clustering. InProceedings of the 20th ACM SIGKDD International Confer-ence on Knowledge Discovery and Data Mining, KDD ’14,page 233–242, New York, NY, USA, 2014. Association forComputing Machinery. → pages 10122Bibliography[ZCG+19] Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein,and Jingjing Liu. Freelb: Enhanced adversarial train-ing for natural language understanding. arXiv preprintarXiv:1909.11764, 2019. → pages 10[ZGC+17] Wei Zhao, Ziyu Guan, Long Chen, Xiaofei He, Deng Cai,Beidou Wang, and Quan Wang. Weakly-supervised deep em-bedding for product review sentiment analysis. IEEE Trans-actions on Knowledge and Data Engineering, 30(1):185–197,2017. → pages 95[ZJW+11] Wayne Xin Zhao, Jing Jiang, Jianshu Weng, Jing He, Ee-Peng Lim, Hongfei Yan, and Xiaoming Li. Comparing twitterand traditional media using topic models. In Proceedings ofthe 33rd European Conference on Advances in InformationRetrieval, ECIR’11, page 338–349, Berlin, Heidelberg, 2011.Springer-Verlag. → pages 10[ZLC+20] Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang,Xipeng Qiu, and Xuanjing Huang. Extractive summarizationas text matching. arXiv preprint arXiv:2004.08795, 2020. →pages 10[ZSC+20] Yu Zhou, Yanqi Su, Taolue Chen, Zhiqiu Huang, Harald CGall, and Sebastiano Panichella. User review-based changefile localization for mobile applications. IEEE Transactionson Software Engineering, 2020. → pages 20[ZWZ+16] Yuan Zuo, Junjie Wu, Hui Zhang, Hao Lin, Fei Wang, Ke Xu,and Hui Xiong. Topic modeling of short texts: A pseudo-document view. In Proceedings of the 22nd ACM SIGKDDInternational Conference on Knowledge Discovery and DataMining, KDD ’16, page 2105–2114, New York, NY, USA,2016. Association for Computing Machinery. → pages 11[ZWZ19] Xingxing Zhang, Furu Wei, and Ming Zhou. Hibert:Document level pre-training of hierarchical bidirectionaltransformers for document summarization. arXiv preprintarXiv:1905.06566, 2019. → pages 10[ZXT+20] Ting Zhang, Bowen Xu, Ferdian Thung, Stefanus Agus Hary-ono, David Lo, and Lingxiao Jiang. Sentiment analysis for123Bibliographysoftware engineering: How far can pre-trained transformermodels go? In 2020 IEEE International Conference onSoftware Maintenance and Evolution (ICSME), pages 70–80.IEEE, 2020. → pages 10, 21, 22, 33, 37, 40, 41, 45, 95[ZXW19] Yabing Zhao, Xun Xu, and Mingshu Wang. Predicting over-all customer satisfaction: Big data evidence from hotel onlinetextual reviews. International Journal of Hospitality Man-agement, 76:111–121, 2019. → pages 31[ZYZ20] Zhuosheng Zhang, Junjie Yang, and Hai Zhao. Retrospectivereader for machine reading comprehension. arXiv preprintarXiv:2001.09694, 1:1–9, 2020. → pages 10[ZZ19] Lingling Zhao and Anping Zhao. Sentiment analysis basedrequirement evolution prediction. Future Internet, 11(2):52,2019. → pages 9[ZZX16] Yuan Zuo, Jichang Zhao, and Ke Xu. Word network topicmodel: A simple but general solution for short and imbal-anced texts. Knowl. Inf. Syst., 48(2):379–398, August 2016.→ pages 11124Appendix125Appendix ATablesTable A.1: RQ1 results for Prior approaches. The best scores for eachdataset are shown in bold.Micro-avg Scores Macro-avg scoresTrainingTime (s)PredictionTime (s)P R F1 P R F1Dataset-1AR-MINER 0.36 0.36 0.36 No micro or macro average 261.71 NASUR-MINER 0.57 0.56 0.56 0.53 0.47 0.5 105.31 21.06Ensemble 0.71 0.7 0.7 0.7 0.6 0.65 115.84 23.17DL+WE 0.84 0.71 0.77 0.83 0.67 0.74 99.62 14.23Dataset-2AR-MINER 0.49 0.32 0.39 No micro or macro average 81.17 NASUR-MINER 0.6 0.55 0.57 0.65 0.51 0.57 27.18 3.88Ensemble 0.8 0.69 0.74 0.7 0.66 0.68 29.9 4.98DL+WE 0.93 0.84 0.88 0.94 0.84 0.89 26.31 5.26Dataset-3AR-MINER 0.44 0.36 0.4 No micro or macro average 74.62 NASUR-MINER 0.62 0.63 0.62 0.59 0.53 0.56 19.33 3.22Ensemble 0.77 0.75 0.76 0.69 0.71 0.7 21.46 4.29DL+WE 0.86 0.84 0.85 0.77 0.72 0.74 19.74 3.95Dataset-4AR-MINER 0.45 0.33 0.38 No micro or macro average 71.96 NASUR-MINER 0.63 0.58 0.6 0.55 0.53 0.54 11.74 2.35Ensemble 0.84 0.74 0.79 0.82 0.7 0.76 13.5 2.7DL+WE 0.87 0.76 0.81 0.85 0.73 0.79 12.29 2.05Dataset-5AR-MINER 0.53 0.35 0.42 No micro or macro average 57.02 NASUR-MINER 0.62 0.54 0.58 0.57 0.56 0.56 9.11 1.3Ensemble 0.86 0.76 0.81 0.84 0.81 0.82 10.93 2.19DL+WE 0.84 0.75 0.79 0.83 0.81 0.82 9.73 1.62Dataset-6AR-MINER 0.47 0.41 0.44 No micro or macro average 71.91 NASUR-MINER 0.58 0.52 0.55 0.6 0.57 0.58 10.07 1.44Ensemble 0.75 0.73 0.74 0.85 0.81 0.83 11.98 1.71DL+WE 0.77 0.76 0.76 0.86 0.82 0.84 10.3 1.47Dataset-7 Merged DatasetAR-MINER 0.49 0.34 0.4 No micro or macro average 401.57 NASUR-MINER 0.66 0.55 0.6 0.64 0.6 0.62 156.54 31.31Ensemble 0.76 0.7 0.73 0.79 0.76 0.77 176.89 29.48DL+WE 0.84 0.78 0.81 0.82 0.74 0.78 155.66 25.94126Appendix A. TablesTable A.2: RQ1 results for PTMs. The best scores for each dataset areshown in bold.Micro-avg Scores Macro-avg scoresTrainingTime (s)PredictionTime (s)Time-diff (%)P R F1 P R F1Dataset-1BERT 0.79 0.74 0.764 0.76 0.69 0.723 345.8 19.21 34.34ALBERT 0.93 0.91 0.92 0.89 0.84 0.864 286.5 15.92 11.33RoBERTa 0.89 0.82 0.854 0.85 0.82 0.835 357.35 19.85 38.81XLNET 0.76 0.68 0.718 0.72 0.68 0.699 514.92 36.78 157.2Dataset-2BERT 0.77 0.69 0.728 0.86 0.83 0.845 185.14 10.29 165.21ALBERT 0.96 0.88 0.918 0.94 0.89 0.914 119.93 6.66 71.65RoBERTa 0.9 0.85 0.874 0.79 0.74 0.764 202.44 11.25 189.95XLNET 0.85 0.77 0.808 0.87 0.78 0.823 345.57 24.68 536.08Dataset-3BERT 0.93 0.91 0.92 0.9 0.82 0.858 156.9 6.82 111.8ALBERT 0.91 0.87 0.89 0.89 0.79 0.837 105.53 4.59 42.55RoBERTa 0.93 0.88 0.904 0.89 0.81 0.848 175.29 7.62 136.65XLNET 0.88 0.86 0.87 0.79 0.73 0.759 307.66 17.09 430.75Dataset-4BERT 0.84 0.78 0.809 0.79 0.72 0.753 75.69 2.91 41.95ALBERT 0.85 0.81 0.83 0.72 0.65 0.683 53.66 2.06 0.49RoBERTa 0.93 0.9 0.915 0.87 0.81 0.839 87.61 3.37 64.39XLNET 0.78 0.75 0.765 0.8 0.76 0.779 149.36 7.11 246.83Dataset-5BERT 0.88 0.81 0.844 0.8 0.77 0.785 74.52 2.57 97.69ALBERT 0.93 0.89 0.91 0.93 0.88 0.904 38.63 1.33 2.31RoBERTa 0.94 0.89 0.914 0.87 0.83 0.85 80.69 2.78 113.85XLNET 0.87 0.79 0.828 0.924 0.85 0.904 127.46 4.9 276.92Dataset-6BERT 0.79 0.74 0.764 0.84 0.82 0.83 74.83 2.77 92.36ALBERT 0.8 0.77 0.785 0.93 0.86 0.894 48.74 1.81 25.69RoBERTa 0.82 0.77 0.794 0.93 0.91 0.92 81.2 3.01 109.03XLNET 0.73 0.66 0.693 0.93 0.85 0.888 143.15 6.22 331.94Dataset-7 Merged DatasetBERT 0.91 0.84 0.874 0.88 0.8 0.844 574.32 24.97 -3.74ALBERT 0.95 0.85 0.897 0.85 0.8 0.824 460.19 20.01 -22.86RoBERTa 0.97 0.91 0.939 0.87 0.82 0.838 597.65 25.98 0.15XLNET 0.92 0.86 0.889 0.89 0.84 0.864 836.96 49.23 89.78127Appendix A. TablesTable A.3: RQ2 results for C-PTM-X1 (PTM-X1 = ALBERT). The bestscores for each dataset are bold.Micro-avg Scores Macro-avg scoresTrainingTime (s)PredictionTime (s)Time-Diff (%)P R F1 P R F1Dataset-1ALBERT (PTM-X1) 0.93 0.91 0.92 0.89 0.84 0.864 286.5 15.92 11.33C-ALBERT-BASE 0.94 0.916 0.928 0.902 0.844 0.872 295.1 15.6 9.09C-ALBERT-MEDIUM 0.95 0.926 0.938 0.908 0.865 0.886 280.77 15.44 7.97C-ALBERT-LARGE 0.977 0.945 0.961 0.92 0.87 0.894 295.1 15.6 9.09Dataset-2ALBERT (PTM-X1) 0.96 0.88 0.918 0.94 0.89 0.914 119.93 6.66 71.65C-ALBERT-BASE 0.972 0.888 0.928 0.947 0.894 0.92 117.53 6.79 75C-ALBERT-MEDIUM 0.987 0.906 0.945 0.958 0.903 0.93 123.53 6.86 76.8C-ALBERT-LARGE 0.991 0.91 0.955 0.978 0.933 0.955 121.13 6.86 76.8Dataset-3ALBERT (PTM-X1) 0.91 0.87 0.886 0.8 0.75 0.774 105.53 4.59 42.55C-ALBERT-BASE 0.938 0.922 0.93 0.91 0.829 0.868 107.64 4.64 44.1C-ALBERT-MEDIUM 0.949 0.926 0.937 0.914 0.841 0.876 106.59 4.5 39.75C-ALBERT-LARGE 0.965 0.945 0.955 0.941 0.845 0.89 104.47 4.64 44.1Dataset-4ALBERT (PTM-X1) 0.85 0.81 0.83 0.72 0.65 0.683 53.66 2.06 0.49C-ALBERT-BASE 0.94 0.907 0.923 0.875 0.821 0.847 54.2 2.02 -1.46C-ALBERT-MEDIUM 0.945 0.92 0.932 0.886 0.832 0.858 53.12 2.04 -0.49C-ALBERT-LARGE 0.976 0.945 0.96 0.9 0.841 0.87 52.59 2.08 1.46Dataset-5ALBERT (PTM-X1) 0.93 0.89 0.91 0.93 0.88 0.904 38.63 1.33 2.31C-ALBERT-BASE 0.952 0.899 0.925 0.942 0.892 0.916 39.4 1.3 0C-ALBERT-MEDIUM 0.965 0.917 0.94 0.957 0.898 0.927 37.86 1.3 0C-ALBERT-LARGE 0.984 0.922 0.952 0.967 0.908 0.937 38.24 1.34 3.08Dataset-6ALBERT (PTM-X1) 0.8 0.77 0.785 0.93 0.86 0.894 48.74 1.81 25.69C-ALBERT-BASE 0.831 0.775 0.802 0.938 0.921 0.929 49.71 1.83 27.08C-ALBERT-MEDIUM 0.842 0.791 0.816 0.95 0.924 0.937 49.71 1.77 22.92C-ALBERT-LARGE 0.86 0.799 0.828 0.963 0.943 0.953 49.71 1.83 27.08Dataset-7 Merged DatasetALBERT (PTM-X1) 0.95 0.85 0.897 0.85 0.8 0.824 460.19 20.01 -22.86C-ALBERT-BASE 0.98 0.92 0.949 0.903 0.848 0.875 464.79 20.21 -22.09C-ALBERT-MEDIUM 0.993 0.933 0.962 0.908 0.86 0.883 450.99 20.41 -21.32C-ALBERT-LARGE 0.998 0.942 0.979 0.935 0.866 0.899 469.39 19.81 -23.63128Appendix A. TablesTable A.4: RQ2 results for C-PTM-X2 (PTM-X2 = RoBERTa). The bestscores are bold.Micro-avg Scores Macro-avg scoresTrainingTime (s)PredictionTime (s)TimeDiff (%)P R F1 P R F1Dataset-1RoBERTa (PTM-X2) 0.89 0.82 0.854 0.85 0.82 0.835 357.35 19.85 38.81C-RoBERTa-BASE 0.949 0.925 0.937 0.903 0.85 0.876 360.92 19.65 37.41C-RoBERTa-MEDIUM 0.958 0.959 0.958 0.933 0.869 0.9 360.92 20.25 41.61C-RoBERTa-LARGE 0.991 0.982 0.996 0.97 0.911 0.94 364.5 19.45 36.01Dataset-2RoBERTa (PTM-X2) 0.9 0.85 0.874 0.79 0.74 0.764 202.44 11.25 189.95C-RoBERTa-BASE 0.985 0.895 0.938 0.949 0.902 0.925 204.46 11.36 192.78C-RoBERTa-MEDIUM 0.962 0.914 0.937 0.989 0.938 0.963 200.42 11.03 184.28C-RoBERTa-LARGE 0.985 0.956 0.97 0.997 0.946 0.971 204.46 11.36 192.78Dataset-3RoBERTa (PTM-X2) 0.93 0.88 0.904 0.89 0.81 0.848 175.29 7.62 136.65C-RoBERTa-BASE 0.953 0.931 0.942 0.925 0.845 0.883 173.54 7.7 139.13C-RoBERTa-MEDIUM 0.978 0.942 0.96 0.927 0.859 0.892 177.04 7.47 131.99C-RoBERTa-LARGE 0.995 0.986 0.99 0.965 0.884 0.923 177.04 7.47 131.99Dataset-4RoBERTa (PTM-X2) 0.93 0.9 0.915 0.87 0.81 0.839 87.61 3.37 64.39C-RoBERTa-BASE 0.943 0.912 0.927 0.879 0.82 0.848 88.49 3.44 67.8C-RoBERTa-MEDIUM 0.966 0.94 0.953 0.897 0.841 0.868 89.36 3.4 65.85C-RoBERTa-LARGE 0.986 0.962 0.974 0.947 0.877 0.911 89.36 3.4 65.85Dataset-5RoBERTa (PTM-X2) 0.94 0.89 0.914 0.87 0.83 0.85 80.69 2.78 113.85C-RoBERTa-BASE 0.95 0.917 0.933 0.948 0.901 0.924 79.08 2.84 118.46C-RoBERTa-MEDIUM 0.982 0.917 0.948 0.974 0.917 0.945 81.5 2.75 111.54C-RoBERTa-LARGE 0.997 0.949 0.972 0.974 0.947 0.96 79.08 2.81 116.15Dataset-6RoBERTa (PTM-X2) 0.82 0.77 0.794 0.93 0.91 0.92 81.2 3.01 109.03C-RoBERTa-BASE 0.833 0.78 0.806 0.954 0.925 0.939 82.01 3.07 113.19C-RoBERTa-MEDIUM 0.869 0.802 0.834 0.961 0.945 0.953 79.58 2.95 104.86C-RoBERTa-LARGE 0.874 0.837 0.855 0.987 0.97 0.978 82.82 2.98 106.94Dataset-7 Merged DatasetRoBERTa (PTM-X2) 0.97 0.91 0.939 0.87 0.82 0.838 597.65 25.98 0.15C-RoBERTa-BASE 0.98 0.921 0.95 0.903 0.856 0.879 585.7 25.46 -1.85C-RoBERTa-MEDIUM 0.983 0.938 0.96 0.926 0.889 0.907 603.63 25.72 -0.85C-RoBERTa-LARGE 0.995 0.975 0.984 0.949 0.895 0.921 591.67 26.5 2.16129Appendix A. TablesTable A.5: RQ3 results for Binary Classifications (Part i)Datasets & Approaches P R F1 Training Time (s) Prediction Time (s)Dataset-1 Randomly Selected Class: Bug ReportAR-MINER 0.37 0.36 0.36 260.92 NASUR-MINER 0.58 0.56 0.57 105.63 21.08Ensemble 0.73 0.72 0.72 115.61 23.05DL+WE 0.87 0.73 0.79 99.72 14.3BERT 0.81 0.75 0.78 344.42 19.29ALBERT 0.95 0.92 0.93 287.93 15.95RoBERTa 0.91 0.83 0.87 358.78 19.87XLNET 0.78 0.69 0.73 514.41 36.74C-ALBERT-BASE 0.95 0.95 0.95 293.92 15.66C-ALBERT-MEDIUM 0.96 0.95 0.95 281.89 15.52C-ALBERT-LARGE 0.97 0.98 0.97 295.99 15.62C-RoBERTa-BASE 0.98 0.95 0.96 361.64 19.55C-RoBERTa-MEDIUM 0.98 0.97 0.97 359.84 20.17C-RoBERTa-LARGE 1 0.99 0.99 366.32 19.37Dataset-2 Randomly Selected Class: InquiryAR-MINER 0.5 0.32 0.39 80.93 NASUR-MINER 0.6 0.56 0.58 27.04 3.88Ensemble 0.81 0.7 0.75 29.78 4.99DL+WE 0.94 0.86 0.9 26.34 5.29BERT 0.8 0.7 0.75 184.58 10.32ALBERT 0.98 0.91 0.94 120.29 6.69RoBERTa 0.93 0.86 0.89 203.45 11.21XLNET 0.87 0.78 0.82 345.92 24.66C-ALBERT-BASE 0.91 0.94 0.92 117.77 6.8C-ALBERT-MEDIUM 0.92 0.95 0.93 124.15 6.87C-ALBERT-LARGE 0.92 0.98 0.95 120.65 6.87C-RoBERTa-BASE 1 0.92 0.96 203.85 11.39C-RoBERTa-MEDIUM 0.98 0.93 0.95 200.02 11.07C-RoBERTa-LARGE 1 0.99 0.99 204.66 11.33Dataset-3 Randomly Selected Class: ReliabilityAR-MINER 0.44 0.36 0.4 74.77 NASUR-MINER 0.63 0.65 0.64 19.43 3.21Ensemble 0.79 0.77 0.78 21.42 4.29DL+WE 0.87 0.86 0.86 19.76 3.97BERT 0.93 0.93 0.93 157.53 6.83ALBERT 0.93 0.88 0.9 105.95 4.59RoBERTa 0.95 0.89 0.92 175.64 7.66XLNET 0.89 0.89 0.89 307.04 17.16C-ALBERT-BASE 0.94 0.95 0.92 107.1 4.65C-ALBERT-MEDIUM 0.93 0.94 0.95 107.02 4.49C-ALBERT-LARGE 0.96 0.96 0.96 104.68 4.63C-RoBERTa-BASE 0.98 0.97 0.96 173.37 7.72C-RoBERTa-MEDIUM 1 0.97 0.98 177.93 7.5C-RoBERTa-LARGE 1 1 1.0 177.22 7.46Dataset-4 Randomly Selected Class: Feature RequestAR-MINER 0.46 0.34 0.39 71.67 NASUR-MINER 0.64 0.59 0.61 11.76 2.35Ensemble 0.85 0.75 0.8 13.54 2.7DL+WE 0.88 0.78 0.83 12.23 2.05BERT 0.86 0.8 0.83 75.84 2.9ALBERT 0.86 0.83 0.84 53.93 2.06RoBERTa 0.96 0.93 0.94 87.87 3.36XLNET 0.79 0.78 0.78 148.91 7.14C-ALBERT-BASE 0.94 0.94 0.9 54.15 2.02C-ALBERT-MEDIUM 0.94 0.94 0.91 53.33 2.03C-ALBERT-LARGE 0.96 0.97 0.91 52.38 2.07C-RoBERTa-BASE 0.97 0.95 0.94 88.31 3.42C-RoBERTa-MEDIUM 1 0.97 0.98 89.27 3.39C-RoBERTa-LARGE 1 0.99 0.99 89 3.42130Appendix A. TablesTable A.6: RQ3 results for Binary Classifications (Part ii)Datasets & Approaches P R F1 Training Time (s) Prediction Time (s)Dataset-5 Randomly Selected Class: App ProblemAR-MINER 0.54 0.35 0.42 56.96 NASUR-MINER 0.63 0.56 0.59 9.08 1.3Ensemble 0.88 0.77 0.82 10.94 2.18DL+WE 0.85 0.76 0.8 9.72 1.62BERT 0.91 0.82 0.86 74.45 2.58ALBERT 0.95 0.91 0.93 38.82 1.32RoBERTa 0.97 0.92 0.94 80.53 2.78XLNET 0.9 0.81 0.85 127.08 4.92C-ALBERT-BASE 0.91 0.94 0.92 39.6 1.29C-ALBERT-MEDIUM 0.93 0.97 0.95 37.82 1.3C-ALBERT-LARGE 0.94 0.98 0.96 38.35 1.35C-RoBERTa-BASE 0.98 0.92 0.95 79.48 2.83C-RoBERTa-MEDIUM 1 0.93 0.96 81.34 2.74C-RoBERTa-LARGE 1 0.97 0.98 79.48 2.82Dataset-6 Randomly Selected Class: Bug ReportAR-MINER 0.48 0.42 0.45 71.98 NASUR-MINER 0.59 0.53 0.56 10.02 1.44Ensemble 0.76 0.75 0.75 11.92 1.72DL+WE 0.78 0.77 0.77 10.32 1.47BERT 0.81 0.75 0.78 75.13 2.76ALBERT 0.82 0.79 0.8 48.79 1.82RoBERTa 0.84 0.78 0.81 81.52 3.01XLNET 0.75 0.67 0.71 142.72 6.24C-ALBERT-BASE 0.79 0.81 0.8 49.96 1.83C-ALBERT-MEDIUM 0.81 0.84 0.82 49.91 1.78C-ALBERT-LARGE 0.82 0.86 0.84 49.51 1.82C-RoBERTa-BASE 0.85 0.79 0.82 82.17 3.08C-RoBERTa-MEDIUM 0.88 0.81 0.84 79.66 2.94C-RoBERTa-LARGE 0.89 0.84 0.86 82.65 2.99Dataset-7 Merged Dataset; Randomly Selected Class: Aspect EvaluationAR-MINER 0.51 0.35 0.42 402.77 NASUR-MINER 0.68 0.56 0.61 157.01 31.25Ensemble 0.78 0.71 0.74 177.6 29.45DL+WE 0.86 0.79 0.82 156.13 26.04BERT 0.93 0.86 0.89 573.75 25.04ALBERT 0.97 0.87 0.92 462.49 20.05RoBERTa 1.0 0.93 0.96 600.64 25.88XLNET 0.94 0.88 0.91 834.45 48.98C-ALBERT-BASE 0.94 0.98 0.96 463.4 20.11C-ALBERT-MEDIUM 0.96 0.99 0.97 450.09 20.51C-ALBERT-LARGE 0.96 1.0 0.98 467.98 19.79C-RoBERTa-BASE 0.99 0.93 0.96 584.53 25.33C-RoBERTa-MEDIUM 0.99 0.95 0.97 605.44 25.8C-RoBERTa-LARGE 1.0 0.98 0.99 592.26 26.47131Appendix A. TablesTable A.7: RQ3 results for Zero-Shot ClassificationDataset # P R F1 Prediction Time (s)Dataset-1AR-MINER 0.36 0.36 0.36 N.A.ALBERT-NLI 0.47 0.42 0.44 15.35RoBERTa-NLI 0.5 0.46 0.48 20.35C-ALBERT-LARGE-NLI 0.5 0.47 0.48 16.7C-RoBERTa-LARGE-NLI 0.59 0.53 0.56 20.66RoBERTa-LARGE-MNLI 0.64 0.6 0.62 19.1Dataset-2AR-MINER 0.49 0.32 0.39 N.A.ALBERT-NLI 0.46 0.42 0.44 6.98RoBERTa-NLI 0.5 0.47 0.48 10.85C-ALBERT-LARGE-NLI 0.51 0.48 0.49 6.47C-RoBERTa-LARGE-NLI 0.56 0.5 0.53 11.64RoBERTa-LARGE-MNLI 0.64 0.58 0.61 10.8Dataset-3AR-MINER 0.44 0.36 0.4 N.A.ALBERT-NLI 0.43 0.4 0.41 4.37RoBERTa-NLI 0.51 0.46 0.48 7.28C-ALBERT-LARGE-NLI 0.55 0.51 0.53 4.72C-RoBERTa-LARGE-NLI 0.61 0.57 0.59 7.95RoBERTa-LARGE-MNLI 0.71 0.65 0.68 7.78Dataset-4AR-MINER 0.45 0.33 0.38 N.A.ALBERT-NLI 0.42 0.38 0.4 2.00RoBERTa-NLI 0.51 0.47 0.49 3.22C-ALBERT-LARGE-NLI 0.47 0.42 0.44 2.12C-RoBERTa-LARGE-NLI 0.65 0.59 0.62 3.51RoBERTa-LARGE-MNLI 0.69 0.64 0.66 3.21Dataset-5AR-MINER 0.53 0.35 0.42 N.A.ALBERT-NLI 0.39 0.35 0.37 1.37RoBERTa-NLI 0.53 0.49 0.51 2.91C-ALBERT-LARGE-NLI 0.47 0.44 0.45 1.37C-RoBERTa-LARGE-NLI 0.59 0.54 0.56 2.72RoBERTa-LARGE-MNLI 0.74 0.67 0.70 2.9Dataset-6AR-MINER 0.47 0.41 0.44 N.A.ALBERT-NLI 0.4 0.36 0.38 1.73RoBERTa-NLI 0.43 0.4 0.41 2.95C-ALBERT-LARGE-NLI 0.43 0.41 0.42 1.88C-RoBERTa-LARGE-NLI 0.52 0.48 0.5 3.13RoBERTa-LARGE-MNLI 0.64 0.58 0.61 2.87Dataset-7 Merged DatasetAR-MINER 0.49 0.34 0.4 N.A.ALBERT-NLI 0.47 0.44 0.45 19.11RoBERTa-NLI 0.58 0.53 0.55 24.94C-ALBERT-LARGE-NLI 0.48 0.44 0.46 19.39C-RoBERTa-LARGE-NLI 0.65 0.61 0.63 24.84RoBERTa-LARGE-MNLI 0.74 0.68 0.71 27.28132Appendix A. TablesTable A.8: RQ3 results for Multi-Task (Category) ClassificationMicro-Scores Macro-Scores TrainingTime (s)PredictionTime (s)P R F1 P R F1Category Labeled DataBERT 0.813 0.759 0.791 0.752 0.714 0.732 287.16 9.988ALBERT 0.883 0.865 0.86 0.842 0.785 0.823 230.095 12.006RoBERTa 0.844 0.782 0.81 0.736 0.693 0.698 239.06 10.392XLNET 0.798 0.717 0.753 0.822 0.721 0.779 418.48 19.692C-ALBERT-BASE 0.888 0.85 0.872 0.853 0.785 0.807 139.437 12.126C-ALBERT-MEDIUM 0.897 0.864 0.897 0.865 0.809 0.837 180.396 6.123C-ALBERT-LARGE 0.933 0.89 0.917 0.859 0.804 0.831 281.634 5.943C-RoBERTa-BASE 0.945 0.837 0.878 0.887 0.858 0.86 292.85 10.184C-RoBERTa-MEDIUM 0.894 0.86 0.877 0.942 0.894 0.904 301.815 12.86C-RoBERTa-LARGE 0.932 0.90 0.901 0.931 0.879 0.927 295.835 10.6Table A.9: RQ3 results for Multi-Task (Sentiment) ClassificationMicro-Scores Macro-Scores TrainingTime (s)PredictionTime (s)P R F1 P R F1Sentiment Labeled DataBERT 0.852 0.769 0.804 0.759 0.748 0.739 689.184 32.461ALBERT 0.886 0.889 0.877 0.848 0.806 0.822 552.228 26.013RoBERTa 0.862 0.805 0.83 0.75 0.711 0.74 836.71 31.176XLNET 0.808 0.737 0.787 0.849 0.754 0.794 1088.048 63.999C-ALBERT-BASE 0.908 0.883 0.884 0.868 0.796 0.838 604.227 26.273C-ALBERT-MEDIUM 0.913 0.878 0.89 0.879 0.829 0.845 541.188 26.533C-ALBERT-LARGE 0.93 0.911 0.918 0.892 0.823 0.858 610.207 23.772C-RoBERTa-BASE 0.958 0.861 0.904 0.913 0.858 0.903 644.27 33.098C-RoBERTa-MEDIUM 0.934 0.864 0.913 0.955 0.905 0.929 784.719 28.292C-RoBERTa-LARGE 0.964 0.934 0.93 0.972 0.907 0.935 828.338 31.800133Appendix A. TablesTable A.10: RQ3 results for Multi-Resource Classification (Trained on AppReviews and Tested on Twitter)Datasets & Approaches P R F1 Prediction Time (s)Dataset-2AR-Miner 0.42 0.28 0.34 N.A.SUR-Miner 0.61 0.56 0.58 39.22Ensemble 0.78 0.72 0.75 38.79DL+WE 0.85 0.78 0.81 27.79BERT 0.57 0.50 0.53 17.49ALBERT 0.69 0.65 0.67 9.99RoBERTa 0.67 0.60 0.63 18.0XLNET 0.64 0.56 0.60 44.42C-ALBERT-BASE 0.75 0.67 0.71 11.54C-ALBERT-MEDIUM 0.79 0.75 0.77 10.29C-ALBERT-LARGE 0.84 0.76 0.80 12.35C-RoBERTa-BASE 0.80 0.74 0.77 20.45C-RoBERTa-MEDIUM 0.82 0.77 0.79 19.85C-RoBERTa-LARGE 0.87 0.86 0.86 20.45Table A.11: RQ3 results for Multi-Resource Classification (Trained on Twit-ter and Tested on App Reviews)Datasets & Approaches P R F1 Prediction Time (s)Dataset-2AR-Miner 0.36 0.24 0.29 N.ASUR-Miner 0.66 0.59 0.62 25.1Ensemble 0.83 0.76 0.79 23.66DL+WE 0.89 0.83 0.86 16.4BERT 0.62 0.55 0.58 9.62ALBERT 0.73 0.72 0.72 6.29RoBERTa 0.70 0.64 0.67 11.7XLNET 0.69 0.62 0.65 28.43C-ALBERT-BASE 0.79 0.70 0.74 6.81C-ALBERT-MEDIUM 0.85 0.83 0.84 5.87C-ALBERT-LARGE 0.93 0.82 0.87 6.79C-RoBERTa-BASE 0.82 0.75 0.78 13.09C-RoBERTa-MEDIUM 0.89 0.82 0.85 11.12C-RoBERTa-LARGE 0.92 0.92 0.92 11.45134
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Fard, Fatemeh Hendijani
On the understanding of software engineering related texts via the transfer of prior knowledge
Text