Science, Irving K. Barber Faculty of (Okanagan)
Computer Science, Mathematics, Physics and Statistics, Department of (Okanagan)
DSpace
UBCO
Sharma, Rishab
Master of Science - MSc
University of British Columbia
Code comment generation is the task of generating a high-level natural language description for a given code snippet. Comments help software developers maintain programs; however, comments are mostly missing or are outdated. Many studies develop models to generate comments automatically, mainly using deep neural networks. A missing point in the current research is capturing each character's information and the syntactic differences of tokens. Moreover, the contextual meaning of code tokens is generally overlooked. In this thesis, we present LAnguage Model and Named Entity Recognition Code comment generator (LAMNER-Code). A character-level language model is used to learn the semantic representation and a Named Entity Recognition model is trained for learning the code entities. These representations are used in a Neural Machine Translation architecture to produce comments. We evaluate the generated comments using our model and other baselines against ground truth on a Java dataset with four standard metrics, BLEU, ROGUE-L, METEOR, and CIDEr, which are improved by 3.26, 5.27, 1.25, and 0.1 points, respectively. The existing techniques and our proposed work are complementary to each other. Experiments on abstracted code further demonstrate the value of the LAMNER-Code embeddings. The human evaluation confirms the quality of LAMNER-Code comments compared to baselines and the reference comments. Also, the new decoder sampling strategy presented in this works can better recall identifiers during comments generation. Despite the improvement in the performance, we see performance from Transformer based models is comparable to this work. Therefore, we conduct an additional exploratory study to understand source code comprehension by the Transformer based models. The findings from this study reveal some similarities with the natural languages and also present differences in the attention of the Transformer-based language model for source code. Finally, we use the findings from this study to develop a new embedding for classification task based on identifiers which further improves the performance for code clone detection over the vanilla techniques which uses the 'CLS' token.
Source Code Representation forComment Generation and ProgramComprehensionbyRishab SharmaB.Tech., Guru Nanak Dev University, 2018A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF SCIENCEinTHE COLLEGE OF GRADUATE STUDIES(Computer Science)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)December 2021© Rishab Sharma, 2021The following individuals certify that they have read, and recommendto the College of Graduate Studies for acceptance, a thesis/dissertation en-titled:Source Code Representation for Comment Generation andProgram Comprehensionsubmitted by Rishab Sharma in partial fulfilment of the requirements ofthe degree of Master of ScienceDr. Fatemeh Hendijani Fard, Irving K. Barber Faculty of ScienceSupervisorDr. Mohamed Shehata, Irving K. Barber Faculty of ScienceSupervisory Committee MemberDr. Chen Feng, School of EngineeringSupervisory Committee MemberDr. Anas Chaaban, School of EngineeringUniversity ExamineriiAbstractCode comment generation is the task of generating a high-level naturallanguage description for a given code snippet. Comments help software de-velopers maintain programs; however, comments are mostly missing or areoutdated. Many studies develop models to generate comments automati-cally, mainly using deep neural networks. A missing point in the current re-search is capturing each character’s information and the syntactic differencesof tokens. Moreover, the contextual meaning of code tokens is generallyoverlooked. In this thesis, we present LAnguage Model and Named EntityRecognition Code comment generator (LAMNER-Code). A character-levellanguage model is used to learn the semantic representation and a NamedEntity Recognition model is trained for learning the code entities. These rep-resentations are used in a Neural Machine Translation architecture to pro-duce comments. We evaluate the generated comments using our model andother baselines against ground truth on a Java dataset with four standardmetrics, BLEU, ROGUE-L, METEOR, and CIDEr, which are improvedby 3.26, 5.27, 1.25, and 0.1 points, respectively. The existing techniquesand our proposed work are complementary to each other. Experiments onabstracted code further demonstrate the value of the LAMNER-Code em-beddings. The human evaluation confirms the quality of LAMNER-Codecomments compared to baselines and the reference comments. Also, thenew decoder sampling strategy presented in this works can better recallidentifiers during comments generation. Despite the improvement in theperformance, we see performance from Transformer based models is compa-rable to this work. Therefore, we conduct an additional exploratory studyto understand source code comprehension by the Transformer based mod-els. The findings from this study reveal some similarities with the naturallanguages and also present differences in the attention of the Transformer-based language model for source code. Finally, we use the findings from thisstudy to develop a new embedding for classification task based on identifierswhich further improves the performance for code clone detection over thevanilla techniques which uses the ’CLS’ token.iiiLay SummaryWriting comments for code is an important software engineering practiceoften overlooked by developers. Comments can be useful during the main-tenance of the project as it helps the developers easily understand whatcode does, without reading the complete code. Therefore, automatic codecomment generation can be helpful. Previous works demonstrate the im-portance of incorporating the syntax of code for comment generation usingAbstract Syntax Trees (AST). This thesis presents an alternative techniqueto learn the syntax of code that is complementary to AST. Moreover, wedevise a new decoder sampling strategy that can better recall identifiers.Additionally, we make changes to the encoder architecture of the neuralmachine translation model to incorporate the dynamic generation of em-bedding for tokens, which leads to better performance on the normalizeddata. Finally, we present a study to understand the comprehension abilitiesof a pre-trained language model based on Transformers.ivPrefaceThis thesis, titled “Source Code Representation for Comment Generationand Program Comprehension” presents an automatic code comments gener-ation framework based on character based language model and named entityrecognition model. A brief study is also conducted to understand BERT’sbehavior for source code. The work done in this thesis has been written anddeveloped in collaboration with my supervisor Dr. Fatemeh Hendijani Fard.Ramin Shahbazi has assisted in running the experiments for two baselineAST-Attend-GRU and ELMO, generation of Abstract syntax tree and waspart of discussion analysis mentioned in Chapter 7. The LAMNER-Codeframework presented in this thesis, Chapters 4 untill 7 is currently under re-view at peer-reviewed journal ’ACM Transactions on Software Engineeringand Methodology’ with submission ID ’TOSEM-2021-0055.R1’. Chapter 8on the exploratory study of the BERT model for source code and its com-parison with natural language will be submitted to peer reviewed ’IEEETransactions on Software Engineering’. As the timeline for this study goesbeyond the defense time, Mohamad Khajezade will apply similar studies onRoBERTa model and will be a co-author of this paper. The ideas of bothpapers are originated by myself and directed by my supervisor. I have writ-ten all the scripts and conducted all experiments of these studies (except thementioned ones). Also, during my Master’s I got the opportunity to workon multiple papers with my supervisor and colleagues, which are currentlyunder review or are work in progress These papers can be classified intorelated and unrelated to my thesis which are listed as follows− “LAMNER-Code: Code Comment Generation Using Character Lan-guage Model and Named Entity Recognition” (Related, Under Re-view)− “Attention analysis of Pre-trained Models for Source Code” (Related,Work in progress)− “API2Com: On the Improvement of Automatically Generated CodeComments Using API Documentations” (Unrelated, Published)vPreface− “A Semantic-Based Framework for Analyzing App Users’ Feedback”(Unrelated, Published)− “Developers’ Discussions on PyTorch Discussion Forum Over DifferentReleases” (Unrelated, Work in Progress)− “Retrieval Enhanced Ensemble Model Framework For Rumor Detec-tion On Micro-blogging Platforms” (Unrelated, Published)− “Self-Admitted Technical Debt in R” (Unrelated, Submitted)viTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . viiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . xivChapter 1: Introduction . . . . . . . . . . . . . . . . . . . . . . . 11.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Research Motivation and Objectives . . . . . . . . . . . . . . 21.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.4 Proposed Approach . . . . . . . . . . . . . . . . . . . . . . . . 31.5 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . 4Chapter 2: Literature Review . . . . . . . . . . . . . . . . . . . 52.1 Code Comment Generation Techniques . . . . . . . . . . . . . 52.1.1 Information Retrieval based comment generation . . . 52.1.2 Deep Learning Based Comment generation models . . 72.2 Techniques to Learn Source Code/Token Embeddings . . . . 112.3 Differences of This Work with Current Studies . . . . . . . . 13Chapter 3: Background . . . . . . . . . . . . . . . . . . . . . . . 153.1 Recurrent Neural Network . . . . . . . . . . . . . . . . . . . . 153.1.1 Long Short Term Memory . . . . . . . . . . . . . . . . 16viiTABLE OF CONTENTS3.1.2 Gated Recurrent Unit . . . . . . . . . . . . . . . . . . 173.2 Language Model . . . . . . . . . . . . . . . . . . . . . . . . . 183.3 Named Entity Recognition . . . . . . . . . . . . . . . . . . . . 203.4 Neural Machine Translation . . . . . . . . . . . . . . . . . . . 20Chapter 4: Proposed Approach . . . . . . . . . . . . . . . . . . 224.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224.2 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254.3 Learning Semantics of Code Sequences using Character-LevelLanguage Model . . . . . . . . . . . . . . . . . . . . . . . . . 254.4 Learning Syntactic Structure of Code Sequences . . . . . . . 274.5 Semantic-Syntax Model . . . . . . . . . . . . . . . . . . . . . 29Chapter 5: Experiments . . . . . . . . . . . . . . . . . . . . . . . 325.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325.2 Model Training . . . . . . . . . . . . . . . . . . . . . . . . . . 335.2.1 Language Models . . . . . . . . . . . . . . . . . . . . . 335.2.2 Named Entity Recognition Model . . . . . . . . . . . . 335.2.3 Semantic-Syntax Model . . . . . . . . . . . . . . . . . 385.3 Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385.4 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . . . 40Chapter 6: Results . . . . . . . . . . . . . . . . . . . . . . . . . . 426.1 RQ1: LAMNER-Code Performance . . . . . . . . . . . . . . . 436.2 RQ2: Contribution of Semantic vs. Syntax Embedding inLAMNER-Code . . . . . . . . . . . . . . . . . . . . . . . . . . 466.3 RQ3: Effect of Code and Comment Length Increase on Per-formance of Models . . . . . . . . . . . . . . . . . . . . . . . 476.4 RQ4: Effect of Language Models and NER Embedding . . . . 486.5 RQ5: Evaluation of Models When Tested on Abstracted Code 526.6 RQ6: Effect of Combining AST and IR Techniques and TheUse of Transformer Architecture . . . . . . . . . . . . . . . . 56Chapter 7: Discussion . . . . . . . . . . . . . . . . . . . . . . . . 607.1 Inspecting Higher BLEU Score Comments . . . . . . . . . . . 607.2 Inspecting Low BLEU Score Comments . . . . . . . . . . . . 627.3 Human Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 657.4 Threats to validity . . . . . . . . . . . . . . . . . . . . . . . . 67viiiTABLE OF CONTENTSChapter 8: Exploratory Study for Attention Comprehensionin BERT . . . . . . . . . . . . . . . . . . . . . . . . . 698.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 698.2 Transformer and BERT Architecture . . . . . . . . . . . . . . 708.2.1 Transformer . . . . . . . . . . . . . . . . . . . . . . . . 708.3 BERT: Bidirectional Encoder Representations From Trans-formers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 738.4 BERT Attention for Source Code . . . . . . . . . . . . . . . . 748.4.1 Experimental Setup . . . . . . . . . . . . . . . . . . . 748.4.2 Generating BERT Attention Maps . . . . . . . . . . . 768.5 RQ1: BERT Attention Over Different Source Code Tokens . 778.6 RQ2: Layers That Learn the Semantic Information of SourceCode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 808.6.1 Code Clone Detection . . . . . . . . . . . . . . . . . . 828.7 Implications of Findings from RQ1 . . . . . . . . . . . . . . . 83Chapter 9: Conclusion and Future Work . . . . . . . . . . . . . 87Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107Appendix A: Code Samples . . . . . . . . . . . . . . . . . . . . . . 108ixList of TablesTable 5.1 Dataset details for code comment generation . . . . . 33Table 5.2 Results of Named Entity Recognition model . . . . . . 37Table 6.1 Evaluation of various baselines and our model. Thebest results are bold and the second top results areshown with underline. . . . . . . . . . . . . . . . . . . 45Table 6.2 Evaluation of language models and NER embedding.The best results are bold and the second top scores areshown with underline. For simplicity of comparison,the RENCOS, LAM-Code and LAMNER-Code resultsfrom Table 6.1 are repeated here. . . . . . . . . . . . . 51Table 6.3 Evaluation of various baselines and our model on theabstracted code. The highest scores are shown in boldand the second highest scores are underlined. . . . . . 55Table 6.4 Results of fused models. For simplicity of compari-son, the LAMNER-Code results from Table 6.1 arerepeated here. B stands for BLEU. The best scoresare shown in bold. . . . . . . . . . . . . . . . . . . . . 58Table 7.1 Functionality distribution of the 100 sampled Java meth-ods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61Table 7.2 Number of identifiers in comments of each model com-pared to the Reference . . . . . . . . . . . . . . . . . . 63Table 7.3 Classification of samples with low BLEU score . . . . . 64Table 7.4 Count of low frequency tokens in the test data . . . . 65Table 7.5 Results of human evaluation . . . . . . . . . . . . . . . 66Table 8.1 Dataset details for pre-training BERT . . . . . . . . . 75Table 8.2 Accuracy of pre-trained BERT-Base on Valid dataset 76Table 8.3 Dataset details for Code Clone Detection . . . . . . . 83Table 8.4 Using Identifier embedding for code clone detection . . 84xLIST OF TABLESTable 8.5 Using Identifier embedding in Codebert for code clonedetection . . . . . . . . . . . . . . . . . . . . . . . . . . 85xiList of FiguresFigure 3.1 Elementary Neural Language Model architecture . . . 19Figure 3.2 Encoder Decoder NMT model . . . . . . . . . . . . . 21Figure 4.1 Processing code sequence vs. incorporating externalknowledge for code comment generation . . . . . . . 22Figure 4.2 Initializing the NMT model with pre-trained LAMand NER embedding in our work . . . . . . . . . . . 24Figure 4.3 Change of the meaning of sql token with changingcontext . . . . . . . . . . . . . . . . . . . . . . . . . . 24Figure 4.4 Overview of LAMNER-Code Framework . . . . . . . 25Figure 4.5 Bidirectional language model architecture . . . . . . . 27Figure 4.6 Extracting semantic code embedding . . . . . . . . . 27Figure 5.1 Clustering of syntactic tokens based on NER embed-dings . . . . . . . . . . . . . . . . . . . . . . . . . . . 36Figure 6.1 BLEU and ROUGE-L scores compared to the lengthof the input code applied on test data . . . . . . . . . 48Figure 6.2 BLEU and ROUGE-L scores compared to the lengthof the generated comment applied on test data . . . . 48Figure 6.3 Code Abstraction . . . . . . . . . . . . . . . . . . . . 52Figure 7.1 Samples with higher BLEU-1 score . . . . . . . . . . 63Figure 7.2 Samples with BLEU-1 score equal to zero . . . . . . . 65Figure 8.1 Converting input to query, key, value vectors . . . . . 71Figure 8.2 Converting input to query, key, value vectors . . . . . 72Figure 8.3 Attention on Special Tokens . . . . . . . . . . . . . . 77Figure 8.4 Attention on Identifier and Separators . . . . . . . . . 78Figure 8.5 Attention on Other Token Types . . . . . . . . . . . . 79Figure 8.6 Attention on Self tokens . . . . . . . . . . . . . . . . . 80xiiLIST OF FIGURESFigure 8.7 Jenson-Shannon Divergence Between Different Atten-tion Heads . . . . . . . . . . . . . . . . . . . . . . . . 81Figure 8.8 Code Clone with CLS token . . . . . . . . . . . . . . 82Figure 8.9 Attention on Special Tokens . . . . . . . . . . . . . . 85Figure A.1 Samples with better BLEU-1 score for LAMNER-Code as compared to baselines . . . . . . . . . . . . . 109Figure A.2 Samples with zero BLEU-1 score . . . . . . . . . . . . 110xiiiAcknowledgementsI want to express gratitude to my supervisor Dr. Fatemeh HendijaniFard, for providing me the opportunity to work under her supervision. Iam grateful to her for all the hard work she has put into teaching me thefundamentals of conducting research. I am thankful for her guidance onworking on research problems, providing the freedom to think and work onnew ideas, and then conducting detailed hours-long sessions on reviewingthe writings done for the research papers. I would always value not onlyher research skills but also as a person which has helped me learn on howto be a better person. This thesis would not have been possible without herguidance. I would also like to thank Dr. Apurva Narayan for his guidanceduring the COSC-519B project and my committee members, Dr. MohamedShehata and Dr. Chen Feng, for regularly reviewing my work.I owe a huge thanks to my friends and colleagues Pradeep Kumar Ma-hato, Mohammad Abdul Hadi, Nusrat Suzana Momo, Ayazhan Akhmet,Tenzin Jinpa, Biswapriyo Chakrabarty , Pinku Deb Nath, Ramin Shahbazi,and Mohamad Khajezade for helping me during my master’s study journey.A noteworthy thanks is due to Pradeep for providing all the support andsharing his experiences with me which made me learn valuable lessons.Finally, I can not thank enough to my family for believing in me andtheir everlasting support throughout my life.xivChapter 1Introduction1.1 OverviewA software system can be defined as a set of instructions that dictatesthe computer to execute the specified instructions to accomplish the targettask [ABBS14]. Programming languages (Python, C++, Java) form thefundamental building blocks of a software system. Natural languages (e.g.,English, French) allow humans with a common interface to interact with eachother. Likewise, programming languages provide humans with the commoninterface to communicate with computers, which is interpretable by bothhumans and machines. In recent years the software has gradually becomean integral part of our lifestyle. Today, we rely on software for many triv-ial and non-trivial tasks (e.g., healthcare and entertainment). Despite thesoftware brings simplicity to our lives, developing robust and high-qualitysoftware systems is complex. Therefore, a direction of the software engi-neering research focuses on developing tools that helps the developers buildcomplicated software and support them by providing automatic tools such astools to generate code [CTJ+21, WLX+19, LGR+21], detect bugs [AJFB21,PS18], repair programs [WTM+19, DUSK17, AJFB21, WSS17, CM18], gen-erate code comments [HLX+18a, WLL+20, HLX+20, HAMM10, LHWM20],and predict variable names [SBH+18, CYF21, SRTR17]. The availability ofa large amount of source code on software repositories, especially on GitHuband the latest developments in high-performance computing and ease ofavailability of graphical processing units (GPU) have expedited using deeplearning techniques for some of these works. These works have achieved thestate of the art results and advanced the field.However, once the software is developed, it is equally important to main-tain the software system [BSM04]. Maintaining a software system includesfixing bugs, adding new feature requests, optimizing/updating the currentsoftware system code-base, etc. [BR00]. This process also requires read-ing and understanding code by the developers. Understanding and readingcode requires a lot of mental effort as it is not a natural practice for humans[CSD18]. Software developers spend around 59% of their time on program11.2. Research Motivation and Objectivesunderstanding activities. A well-commented code aids in easier code com-prehension [Ten88, WDS81] and assists in the efficient maintenance of thesoftware projects. However, only a small percentage of methods in softwareprojects are commented [LZ18]. Moreover, code comments are not updatedregularly as the software evolves. The software evolution and inconsistencyof code and comments brings additional difficulty for the developers. There-fore, automatic techniques to generate code comments are proposed as asolution and can be useful to developers [LHWM20]. These models receivea piece of code (i.e., code snippet) as input and generate a description aboutthe functionality of code in natural language.1.2 Research Motivation and ObjectivesCurrent literature for code comment generation considers it as a NeuralMachine Translation (NMT) problem [LHWM20]. Here, the source lan-guage is any programming language, and the target language is an Englishdescription of the code. A straightforward approach could be to use theexisting Natural Language Processing (NLP) techniques. Although usingsuch techniques on code has shown good results, there are dissimilarities be-tween programming language and natural language [CSD18]. Programminglanguages are repetitive, mainly due to certain syntax, and can have infinitevocabulary based on the developers’ naming choices for variables/functions[CSD18]. Therefore, the NMT models used to translate natural languagesrequire specific techniques to handle the differences for programming lan-guages.Current research often leverages Abstract Syntax Trees (AST) to cap-ture the syntax of the programming language [HLX+18a, HLX+20, LJM19,WZY+18, ABLY19]. AST provides the structural representation of codein token sequences, which is obtained using a traversal technique to flattenit. However, as AST only represents the sequence of tokens, it first needsto be mapped to a vector representation to be used in the NMT model.Therefore, the tokens are generally randomly initialized in the NMT modelsand do not have any semantic and syntax knowledge of the code tokens.Thus, to enrich the learning of the NMT model by incorporating the se-mantic and syntax of the given code snippet, current works rely on externalsources like API knowledge [HLX+18b], Unified Modeling Language dia-grams (UML) [WDS+20], data flow graphs [GGH+21], and retrieved similarsamples [ZWZ+20, WLL+20, LCX+21].Studies in natural language processing have shown the importance of21.3. Contributionslearning semantic and syntactic information through different embeddingtechniques [QSF+18]. Employing these embeddings inside the NMT modelhas shown improvement for machine translation. Another aspect of codetokens which current literature misses to address is the use of context of thesource code. Based on these results and gap in the literature, this thesispresents following contributions as mentioned below in section1.3.1.3 ContributionsThis thesis has the following contributions:− Develop a syntax embedding for source code, as an alternative andcomplementary technique to AST, which learns the syntactic informa-tion of code.− Incorporating the character information of source code tokens whichare crucial to learn the context of source code tokens.− Develop a decoder sampling strategy, which can correctly include iden-tifiers in the generated comments.− Develop a code comment generation model, which leverages the dy-namic contextual embeddings of the source code rather than staticembeddings.− Study the effect of the proposed embeddings.− Study the effect of different components of the proposed framework.− Study the attention mechanism of Transformer models for source code.− Present a new embedding for pre-trained BERT model for classificationtasks based on identifiers which shows improvement over the vanillatechniques.1.4 Proposed ApproachIn this thesis, we present a novel syntax embedding for source code,an alternative and complementary technique to AST to learn the syntac-tic information of code. We combine this syntactic information with se-mantic embeddings in our model called LAMNER-Code: LAnguage Model31.5. Thesis Outlineand Named Entity Recognition Code comment generator. Quantitative andqualitative studies are conducted to evaluate the performance of the pro-posed framework against the existing baselines. We also conduct a qualita-tive study to manually evaluate (human evaluation) the generated commentsby LAMNER-Code. To further understand the contribution of each part ofthe framework, ablation studies are designed. Finally, we study fused mod-els where we leverage our embeddings on existing techniques in a model;and evaluate the models on a normalized dataset (abstracted code). Addi-tionally, we develop a new sampling strategy for the decoder of the NMTmodel, which can recall identifiers in the comments better than the currentdecoding strategies.We also compare the results of our study with a newer architecture inNLP, the Transformer [VSP+17]. Our results show that the Transformer canproduce a comparable result with models that are based on Recurrent Neu-ral Networks (RNN) and use other information sources. The literature in-sinuates similar results [ACRC20, WZZX20, GGH+21, FGT+20, GRL+21].However, the success of the Transformer based models for programminglanguages is still unclear. Therefore, in this thesis, we also conduct anexploratory study to understand the reasons for the success of Transformerarchitecture. For this purpose, we use a pre-trained language model (BERT)trained on code and explore the model’s learning for different code entitiesand tokens. Findings from this study reveal some new information that canbe useful in designing models that represent source code.1.5 Thesis OutlineThis thesis is organized as follows:We start with reviewing the current literature for code comment gen-eration and code embeddings in Chapter 2. Then we present the requiredbackground for the thesis in Chapter 3 The motivation and proposed ap-proach of our work is explained in Chapter 4, followed by the experimentalsetup in Chapter 5. Results are presented in Chapter 6. Chapter 7 is dedi-cated to discussions followed by human evaluations in Section 7.3. We pointout the threats to validity in Section 7.4. In Chapter 8 we provide a briefstudy on BERT’s attention on source code. Finally, we conclude the thesisin Chapter 9.4Chapter 2Literature ReviewThis section provides a summary of the related works and existing litera-ture available for code comment generation. The literature review presentedin this thesis has referred to other systematic surveys [SSWY19, ZP19] andliving literature [ABDS18] to ensure it is complete and consistent, but is notlimited to them. As these surveys [SSWY19, ZP19] mostly refer to studiespresented before 2019, we include more recent works by referring to mainvenues in Software Engineering, Natural Language Processing, and ArtificialIntelligence.2.1 Code Comment Generation TechniquesCode comment generation is defined as the task of generating naturallanguage (English) description for a given code snippet [SSWY19]. Thegenerated comment explains the functionality of the given code. Softwareengineering researchers have proposed many techniques to improve auto-matic code comment generation. Mainly, the literature for code commentgeneration has been working to generate comments for methods. However,some works apply the techniques for classes and complete software projects[BHM21, HAMM10]. Primarily the techniques for code comment genera-tion can be divided into two categories, (i) Information Retrieval, (ii) DeepLearning-based works.2.1.1 Information Retrieval based comment generationInformation retrieval (IR) based works leverage techniques like Vec-tor Space Modeling (VSM) [SWY75] and Latent Semantic Indexing (LSI)[DDF+90]. These techniques allow the IR-based systems to identify theimportant terms from the code, which can generate the comments for thegiven code snippets. Rather than considering code as a bag of words anduniformly sampling the tokens from the code, specific weights are assignedto each term. One of the common techniques to assign weights is the TermFrequency-Inverse Document Frequency (TF-IDF) for VSM. TF-IDF tends52.1. Code Comment Generation Techniquesto give higher importance to code tokens with higher frequency in a codesnippet and does not occur quite frequently in the whole corpus of code.This helps identify influential code tokens contributing to better commentgeneration. Similar to VSM, LSI also uses an occurrence matrix, whereeach row represents a term from the code, and each column represents acode snippet. However, it further applies Singular Value Decomposition(SVD) to a smaller document occurrence matrix, corresponding to the high-est Eigenvalues in the original matrix. The smaller matrix is then used tofind relevant topics/terms for a given code snippet.The VSM and LSI techniques are used by Haiduc et al. in [HAMM10] togenerate term-based summaries for methods and classes. For a given set ofcode documents, first, a corpus is generated by splitting the identifier namesbased on camelcase, snake case of the identifiers. Then LSI and VSM areapplied to assign weights for each term in the corpus. For weighting tokens,authors experimented with TF-IDF, log and binary-entropy. Finally, top Kterms (K = 5 or 10) are selected and presented as summaries. The authorsfurther combined their work with the lead terms technique, which presentslead terms of the method/class and the terms selected by VSM. Further,Haiduc et al. [HAM10] combined the text retrieval technique with struc-tural information, which is the role of each term method name, class name,etc. A topic modeling approach was followed by Eddy et al. [ERKC13] todraw a comparison between their work and the method used in [HAMM10].A code completion technique was built [MAC13] by combining a topic mod-eling approach called Latent Dirichlet Allocation (LDA) [BNJ03], to gen-erate the relevant topics from the code. Further, the code documents areconsidered a code and text tokens relationship, and the n-gram languagemodel is used. Panichella et al. [PADP+12] leveraged conversations amongdevelopers from bug reports and email chains to generate the code descrip-tions. The motivation behind this work was, the doubts/questions aboutthe project would have started conversations. So these conversations couldbe leveraged to generate summaries. The authors first collected the dataset(bug reports and emails) for two projects, Eclipse and Lucene. The authorstraced and filtered the paragraphs for the methods. Finally, the textualsimilarity between the sections and source code is calculated by finding thecosine similarity between the VSM weights. The paragraph with the high-est score is presented as a summary. Another paradigm that is relevantfor retrieval-based techniques is similar code should have similar comments.Therefore this motivated the researchers to identify code clones of the codesnippet and use the known words to generate comments for the code infocus. Wong et al. [WYT13] leveraged the code and description mapping62.1. Code Comment Generation Techniquesavailable on the stack overflow platform. Further descriptions for the codeare refined, and later code clones are detected for the know code sample, andtheir documentation is used to generate the comments. These techniquesare mainly used prior to the deep learning approaches.2.1.2 Deep Learning Based Comment generation modelsDeep learning forms the basis for recent development in code commentgeneration. IR-based techniques were primarily used before 2016 [SSWY19].However, the readily available data and compute enabled the use of deeplearning systems for comment generation. Researchers that use deep learn-ing for comment generation consider it as a Neural Machine Translation(NMT) problem. Therefore, Recurrent Neural Networks (RNN)- and Con-volutional Neural Networks (CNN)- based encoder-decoder frameworks havebeen primarily used to generate comments, similar to other NMT systemsin NLP. An attention mechanism is used with the NMT models to identifykey tokens from code snippets and generate coherent code comments.In 2016, Iyer et al. [IKCZ16] presented the first neural model for codecomment generation for C# language. In their work, they used Long ShortTerm Memory networks (LSTM), a variant of RNN to learn the full codecontext. Further, an attention mechanism is used to select important codetokens to generate the comments using the output from the encoder. Intheir work, they initialized code tokens using one-hot vector encoding. Sub-sequently, Zheng et al. [ZZLW17] presented a code attention technique anda larger benchmark dataset C2CGit. The dataset is collected from GitHubto test the different models for the Java programming language. Their workuses a three-layer framework based on the Gated Recurrent Unit (GRU), avariant of RNN. Identifier ordering is done for statements containing ‘for’and ‘if’ statements and specific token encodings are learned as an extra em-bedding matrix for token type “operators.” This helps the model adjust itsvanilla attention according to the important token types.Both techniques discussed above used lexical or token knowledge of code.However, code can be represented with richer representation, Abstract Syn-tax Trees (AST), program dependency graphs, data flow graphs, etc. Priortechniques have extensively leveraged abstract syntax trees. AST is a treerepresentation of code where non-terminal nodes of the tree contain thesyntactic constructs like ‘BlockStmt’, ‘ReturnStmt’, ‘NameExpr’, etc. andterminal nodes are the code tokens. AST has the syntactic elements of codeand is used to represent code with its enriched syntax information. Hu etal. [HLX+18a] presented a model named DeepCom, which employed the72.1. Code Comment Generation Techniquesrepresentation from AST in the encoder-decoder model to generate com-ments. They presented a new technique, Structure-Based Traversal (SBT),to flatten the AST. SBT is a modified version of depth-first traversal of thetree, which, when employed in the NMT model, generates accurate com-ments. An intuitive technique to use AST to generate embedding for eachnon-terminal node of AST could be to utilize recursive neural networks onthe AST of the given code snippet. When used on tree representation as abottom-up model, a recursive neural network uses the leaf nodes of a tree togenerate the embedding for their intermediary parent node (syntactic type)by using different pooling strategies (sum, average). The embedding forleaf nodes is initialized randomly at the start and learned during the modeltraining. The intermediary parent nodes further act as child nodes for theroot of the tree so that a similar bottom-up pooling can generate the em-bedding for the complete code. A similar technique is used in [LZ18], whichleverages recursive neural network in a bottom-up approach to generate arepresentation for source code in the encoder. The decoder further uses it tocreate comments. This technique experiments with both sum and averagepooling and name their model as Code-RNN. Training an NMT model onlywith AST representation and lexical representation in a single encoder maynot be ideal. Therefore, [LJM19] introduced a dual encoder model. Oneof the encoders takes the lexical representation of source code to learns thesemantic representation of code, and the other encoder takes AST as theinput. Both representations are then combined to feed into the decoder. Asimilar technique is used inside the Hybrid Deepcom [HLX+20] model, whichtakes lexical and syntactic information to generate the summaries. Contraryto other AST-based works discussed, Alon et al. [ABLY19] leveraged bagof paths from the AST to generate comments. Rather than flattening ASTfrom end to end in a single representation, this work considered all possiblepaths between each terminal node of the tree. For each path, its terminalnodes are appended at the start and end of the path. Then attention mech-anism is then applied to select the more relevant paths and then selectedpaths to generate comments.Hu et al. [HLX+18b] made efforts to include another important aspectof source code, which is Application Programming Interface (API). As theyare frequently used in source code, and developers often refer to their in-vocations while programming [HLX+18b], it makes sense to leverage theirknowledge inside NMT models. The authors first trained an NMT model,which only takes APIs in the code as input in the encoder and uses thecontext representation of APIs to generate comments. Later, another NMTmodel is trained using the complete code; however, it also leverages the82.1. Code Comment Generation Techniquespreviously trained NMT model using API knowledge.Other works [WLX+19, YPS19] consider the task of generating com-ments as complementary to other software engineering tasks. These workstry to leverage the correlation between two different tasks to generate thecomments. For example, Wei et al. [WLX+19] presents the task of commentgeneration as a dual learning task with code generation. The motivation be-hind this work is that input to comment generation model is the output ofthe code generation model and vice versa. Therefore, they present a single,dual learning framework for both tasks. This framework learns a commonvector space modeling space between code and comment generation. Thelearned space/ constraint is used to generate the comments for code andvice versa. In [YPS19] the modeled relationship between the code annota-tion and code retrieval using a Reinforcement Learning (RL) framework isused to generate the retrieve similar code samples. A reinforcement learning-based technique is also used in [WZY+18]. The primary motive for usingan RL model is to reduce the exposure bias issue generated by the maxi-mum likelihood cost function of the encoder-decoder model because of thenon-availability of ground truth in the inference phase of the test sample.This model also uses the hybrid representation of code which includes lexical(code tokens) and structural (AST). The problem of exposure bias occurs atthe decoder. A reward function is introduced, which rewards the model forcorrect token predicted and penalized when an incorrect token is predictedduring the training. Therefore from maximum likelihood, now the objectiveor cost function also accounts for maximizing reward function, which helpsthe model handle the exposure bias in the decoder.Another upcoming and interesting technique for code comment genera-tion is combining the classical information retrieval approaches with contem-porary deep learning-based works. The primary motivation for these workshas been similar code should have similar comments [WLL+20]. Contraryto previous IR-based works, these works are generative rather than simplyretrieving text from the sample and providing comments for the developers.In [ZWZ+20], initially, a vanilla NMT model based on LSTM is trained us-ing the training corpus. Two similar code snippets, one based on semanticsimilarity and another based on syntactic similarity, are retrieved duringthe testing time. For the semantic retrieval, the authors used the trainedencoder of the NMT model to generate vector representation for code. Thecosine similarity function is used to find the most similar sample from thetrain set for the given code snippet. In the case of syntactic retrieval, the au-thors first converted the code to its AST representation. A lexical indexingand searching tool is used to find the most similar sample by comparing the92.1. Code Comment Generation TechniquesAST’s. Then finally, during the decoding stage, the predictions from thesemantically similar code snippet, syntactic code, and test code are com-bined using the weighted sum of prediction based on the similarity scores ofretrieved samples to make the final prediction. A similar technique is usedinside [WLL+20]; however, rather than retrieving two similar code sam-ples, this work only retrieved a single similar code sample based on lexicalsimilarity using the BM25 searching approach. Different from [ZWZ+20],this work had four separate inputs, test code sample, similar code sample,similar code summary, and AST representation of test code, which is fed in-side four different individual encoders. In another work [LCX+21], retrievalof a similar code sample was used, however contrary to previous work, itused Graph Neural Network (GNN) rather than RNN based neural net-work. GNN suffers from acquiring global relationships among different codeconstructs; therefore, the authors use code property graphs to handle thissituation. [LHWM20] also employed GNN to generate code summaries. Itleveraged convolutional GNN along with code sample and AST for commentgeneration. Other than using AST’s in GNN based models, [WDS+20] com-bined a Unified Modeling Language (UML’s) using a multi-relational GNN.[WSD+21] improves this work by incorporating contextual semantic knowl-edge of class names.Other than software engineering works, recent developments in natu-ral language processing are guiding the way for neural comment generationor NMT in general by developing new architectures for translation systems.As shown in [ACRC20, FGT+20, ACRC21, GGH+21] employing new Trans-former based architecture [DCLT19] for code comment generation can pro-duce better or equivalent performance when compared with the models thatleverage specialize techniques like AST and retrieval with RNN based mod-els. Further, including other code properties like data flow can improve theresults of Transformer [GGH+21]. These results represent the importanceof the newly developed architecture for NLP systems. [FGT+20, ACRC21]shows that, pretraining Transformer networks as language models on hugecorpus and fine-tuning the encoder on a smaller dataset can improve theperformance of the vanilla Transformer model by a significant margin.Most of the literature discussed above has leveraged a single function ora class to generate comments for a single code entity (function and class).However, literature has started acknowledging that information for a singlefunction could be available within other functions/classes available withinthe whole file. Therefore, researchers have started looking outside of a singlefunction definition. Focus is turning towards the entire file in which thefunction is defined. One of the first works which leveraged the file context102.2. Techniques to Learn Source Code/Token Embeddingswas presented by Haque et al. in [HLWM20]. This work leveraged thecomplete file context to generate comments for a single function. This workused two different encoders to create summaries. The first encoder takesa code sample, and all the functions in the context file as input, and theother encoder takes AST as input. Each function available in the contextfile is then passed through a single RNN unit combined with the sourcefunction and AST using an attention mechanism. Another work [BHM21]further extends this idea to the whole project, which generates project-levelencoding for source code for code comment generation. In this work, theauthors used the first K tokens from each subroutine in the project andpassed them through an RNN unit to generate its vector representation. Allthe subroutines in separate files are combined by performing attention totheir representations. Similarly, attention is performed on the selected fileembedding to generate the project-level encoding.2.2 Techniques to Learn Source Code/TokenEmbeddingsLearning to represent tokens/words in a text into a vector representa-tion/embedding has contributed to the widespread success of NLP. Deeplearning-based systems do not understand text in its raw form. Deep learn-ing for NLP requires the words/tokens must first be mapped to its vectorrepresentation. Earlier works in NLP considered documents a bag of wordsand used token frequencies to represent words in techniques like Count vec-torizer and TF-IDF. However, the bag of word techniques could learn thesemantic meaning of tokens. Initial efforts of word2vec [MSC+13], whichleveraged the distributional hypothesis, arguing that the meaning of its wordis represented by the company it keeps to encode semantic meaning in wordembeddings. Vector operation on the generated embeddings led to mean-ingful insights about the language. [PSM14, BGJM17] made similar effortsto encode the semantic meaning of tokens in the embeddings.Motivated by these works, software engineering also recognized the needto develop specific embedding techniques for source code. Moreover, gen-erating embedding forms the fundamental of deep learning; therefore, thegenerated embeddings are task agnostic and can be used with different soft-ware engineering tasks.Nguyen et al. [NNN16] specifically trained continuous bag of word vari-ant of word2vec model to learn two separate embedding spaces for API inJava and C# programming. Later, the learned embeddings are then used112.2. Techniques to Learn Source Code/Token Embeddingsfor mapping API in the Java programming language to C#. This mappingallows the model to identify the same API during the code migration to acompletely new codebase. A transition matrix is learned as a part of modeltraining to generate the corresponding API in C# to enable the mapping.Other works also leverage the word2vec model [WTM+19, CM18]. Whiteet al. [WTM+19] uses the skip-gram model of word2vec to initialize theencoder of the recursive neural network for the program repair. Using pre-trained embeddings rather than random initialization provides the model agood starting weight to build model. Chen et al. [CM18] uses word2vecto first maps different code parts to their vector representation. The vec-tor representation is then used to find the relevant or correct code partsby comparing different code constructs using the cosine similarity. Themost similar constructs are then employed for the program repair. Otherembedding techniques from natural language programming like FastText[BGJM17] is used in [ES19] which provides pre-trained embedding in sixdifferent programming languages. In this works, the FastText model from[BGJM17] is trained specifically on source code and made publicly availableto the researchers. Allamanis et al. [ABBS15] learn a local and global sen-sitive encoding for code using the code tokens and features in code. Whenused for method name prediction, the generated embedding generates plau-sible method names that do not even exist in the train set of the data used.Wang et al. [WSS17] consider a different code construct which is a mix-ture of symbolic and concrete execution traces. The generated embeddingis used to predict common errors in the submissions made by students to aprogramming course. The primary motive to use dynamic system traces togenerate embedding is based on the fact that lexical-based systems do notconsider the natural program flow. When a program is executed at run time,different modules interact with each other. Dynamic system traces can learnthis relationship between programming constructs from the system traces.It would be impossible to understand the relationship if we consider codeas lexical tokens unless the written code is sequential and techniques forobject-oriented programming are not used. Kanade et al. [KMBS20] trainedthe BERT [DCLT19] language model to generate the contextual embeddingsand showed their effectiveness on variable misuse classification, variable mis-use localization and repair. Other technique which uses language model is[KS20], which is used for bug prediction. This work extends and applies theELMO (Embedding from Language model) [PNI+18] model from NLP tothe source code. This model leverages the character based information ofthe source code to learn its representation.Techniques discussed above, considered code as lexical tokens. Other122.3. Differences of This Work with Current Studiesworks consider the structural aspect of code. Therefore these works usedAST and control flow graphs to generate the code embedding for sourcecode. For example, [DUSK17], first flattens the AST using the dep firsttraversal of the tree. The flattened AST is passed through a bi-LSTM unitwhich sequentially takes each node as input. The final representation fromthe LSTM units is concatenated and used to provide input into another en-coder which fine-tunes for the program repair task. Another work [AZLY19]parses all the possible paths between all the leaf nodes of the tree. Thegenerated paths are then used to create method names for a given functionin Java. DeFreez et al. [DTR18] use control flow graphs to generate em-bedding, which can find synonym functions in a Linux kernel written in Cprogramming language. The approach first uses a programmer encoder togenerate the labeled push-down system based on CFG. Random walks overthe CFG are made o collect the relationships, and finally, a neural modelis trained to generate the embedding for the functions. Another imple-mentation to generate code embedding comes from using a recursive neuralnetwork with AST. A recursive neural network uses the aggregated repre-sentation of the leaf nodes to represent the parent nodes. The authors usedthe average aggregation of child nodes to represent the parent. Similarly,the aggregation of all the non-terminal nodes of the AST into the root nodecan describe complete program representation. When used in LSTM basedneural model, it gives very good results for code clone detections. Graphneural networks are another crucial architecture to generate the programembedding. It can naturally leverage the natural structure of source codeto learn the interaction and relationship among different nodes of the code[ABK17]. The representation from all the nodes can then be combined usinga pooLing/aggregation strategy to represent a program.2.3 Differences of This Work with CurrentStudiesAlthough some works leverage the class or project context for commentgeneration, the main direction of the current studies is generating commentsat the function/method level. The new large datasets that are recently pub-lished as benchmarks are all at the method level [HWG+19, LGR+21, LM19,HLX+18b, HLX+18a]. Therefore, we generate comments at the methodlevel.Contrary to most previous works, this thesis presents a syntactic embed-ding for code comment generation rather than using external knowledge to132.3. Differences of This Work with Current Studieslearn the syntax of code. The presented syntactic embedding is complemen-tary to previous works and does not use any external knowledge like AST,UML, or similar exemplar samples. Moreover, the presented embedding iscontext-sensitive and encapsulates some behavior of AST implicitly. There-fore, embedding for each code token can be generated dynamically accordingto context, which is not possible with existing techniques.The importance of identifiers has been recognized in the literature forgenerating descriptive comments [IKCZ16, HBWM21]. However, the pre-vious works miss to address the issue and use the vanilla decoder. A newdecoding sampling strategy is developed in this thesis that addresses thisissue and better captures the identifiers, unlike previous works.Finally, there is no literature available for attention analysis of pre-trained models for source code. Abundant studies have explored pre-trainedmodels for natural languages. However, the knowledge from NLP may notbe directly applicable to programming language due to the differences inthe two domains [CSD18]. Therefore, this thesis presents the first study toexplore the attention mechanism of the Transformer model on source code.14Chapter 3BackgroundIn this chapter, the background information required to understand thework done in this thesis is provided.3.1 Recurrent Neural NetworkRecurrent Neural Networks (RNN), is a type of neural network whichutilizes the sequential information of data to generate its outputs [AA19].RNN learns the probabilistic distrbution of the sequence, which is utilizedto predict the next token of the sequence [CvMG+14]. RNN’s have beenextensively used in tasks like language modeling and neural machine trans-lation [CvMG+14], [BCB15]. For all these tasks, sequential ordering of thedata is important to generate coherent outputs.To train an RNN, over a sequence (w1, w2...wn), where wn representsnth token in sequence of length n, previous hidden state ht−1 and currenttoken xt is fed into the networks to generate the current output yt and thenext hidden state ht. As shown in equation 3.1 and 3.2, xt is the vectorrepresentation of token t in the sequence and Whh, Wxh, and Wyh, representthe sets of weights shared between pair of hidden states, input and hiddenstate, and hidden state and output, respectively. Function F , is generally anon linear [CvMG+14] and b represents the bias. These weights are commonacross each time step t. This way, RNN can leverage previous sequentialknowledge stored in these weight matrices to generate the current output.ht = F (Whhht−1 +Wxhxt + b) (3.1)yt = Wyhht (3.2)Theoretically, the final hidden state of the last token in sequence of theRNN should contain the complete context of the sequence. However, as thesequence length of the input increases, RNN cannot capture the context dueto the effect of vanishing or exploding gradients [PMB13]. The vanishinggradient problem causes the gradients of the model to become vanishingly153.1. Recurrent Neural Networksmall during the backpropagation in the neural network, which prevents themodel from changing its weights [PMB13]. This eventually leads to thenominal learning of the model. Whereas in the exploding gradient problem,the gradients during backpropagation become so large that the learningmade by the neural network is unstable [PMB13]. Long Short Term Memory(LSTM) [HS97] is introduced as a solution to this problem. Another wayto handle the exploding gradient problem is by gradient clipping [PMB13].In this technique, the gradients are not allowed to grow beyond a particularthreshold value; if the gradients of the model pass a threshold, they areclipped and set to the maximum gradients. This thesis uses LSTM, one ofthe variants of RNN which is explained in the following section.3.1.1 Long Short Term MemoryLong Short Term Memory (LSTM) is designed to address the problemsof RNN as described in 3.1. To counter the problem of long term memory,Hochreiter et al. [HS97] introduce gates in LSTM [YSHZ19]. LSTM unitsuse three gates to calculate the output yt and hidden state ht, namely inputgate, output gate and forget gate [HS97]. These gates will learn which partof the sequence data is important to keep or forget to pass to the next LSTMunits. LSTM also introduces the cell state ct, an extra memory to carry thecontext of the sequence. Forget gate Gforget modulates what part of theprevious cell state will be kept:Gforget = F (Wxfxt +Whfht−1) (3.3)where xt is the input at state t, Wi are the weights and ht−1 is theprevious hidden state.Input gate Ginput decides whether to change or pass the value of ct−1 tothe next LSTM state, where ct−1 denotes the cell state:Ginput = F (Wxixt +Whiht−1) (3.4)The output gate Goutput helps in determining the current hidden statevalue at current LSTM state:Goutput = F (Wxoxt +Whoht−1) (3.5)Wi in all equations denote the weights. The three gate values are usedto produce the LSTM hidden state ht and cell state ct.ccandidate = F (Whc(ht−1) +Wxcxt) (3.6)163.1. Recurrent Neural NetworkGinput and Gforget decide which cell state value to choose.ct = Gforget.ct−1 +Ginput.ccandidate (3.7)Where ccandidate is the cell state which should have been passed to thenext state, if Ginput gate was not introduced.Finally, the current cell state ct and Goutput is used to compute thecurrent hidden state value ht.ht = Goutput.F (ct) (3.8)3.1.2 Gated Recurrent UnitGated Recurrent Unit (GRU) is a more recent generation of RNNs pro-posed by Cho et al.[CvMBB14] . GRU only uses the hidden state (cell stateis not used in GRU) to transfer information. GRU introduces two addi-tional gates namely (1). Update gate and (2). Reset gate. As the problemof vanishing gradients occur because of the frequent change in the weightsof the RNN unit, GRU introduced a technique to decide whether to updatethe weight of the previous hidden state ht−1 at current GRU unit or passit as it is to the next GRU unit. This decision is made by the update gateGupdate.Gupdate = F (Wxuxt +Whuht−1) (3.9)Whereas reset gate Greset is used to modulate the contribution of theprevious hidden state ht−1 to the current hidden state htGreset = F (Wxrxt +Whrht−1) (3.10)In equation 3.9 and 3.10 (Wxu), (Whu), (Wxr), (Whr) are the weightsshared by all the GRU units.Therefore the hidden state ht at current time step is represented byequation(3.12).hcandidate = F (Whh(ht−1.Greset) +Wxhxt) (3.11)In equation (3.11) hcandidate is the candidate value of the current hiddenstate which should have been used if Gupdate was not introduced.173.2. Language Modelht = (1−Gupdate).ht−1 +Gupdate.hcandidate (3.12)3.2 Language ModelLanguage modeling is the task of predicting the next word or character,depending on the granularity level of the tokens we are working with duringtraining of the language models. Statistical language models or pre-neurallanguage models, performed this task by learning the probability distribu-tion of sequences of tokens. Mathematically, they are language models aredefined as a probabilistic function on the sequence of text as shown in equa-tion 3.13.P (w1w2..wt) = P (w1).P (w2|w1).P (w3|w2w1)..P (wt|wt−1wt−2...w1) (3.13)Where (w1w2...wt) represents the complete sequence or sentence andw1, w2, ..., wt are individual tokens in the sentence with t tokens. Granularityfor language model can either be the words or characters.One such type of language models are known as N-gram language model.Here N represents the number of N-grams to consider, for calculating thematching sequences. These models calculates the probability of occurrenceof the next token by counting the number of occurrences of next token wiwith previous sequence of wi−1, wi−2, ...., w1 tokens in the whole corpus asshown in equation 3.14.P(wiwi−1, wi−2, ...., w1)=Count (wi, wi−1, wi−2, ...., w1)Count (wi−1, wi−2, ...., w1)(3.14)However, such a count-based approach suffers from the problem of spar-sity, curse of dimensionality, and finding the optimal n (number of previoustokens to take into account while calculating the probability). Also, as then grows, the issue of sparsity worsens(despite smoothing), because it is im-possible to find all possible sequences of (wi, wi−1, wi−2, ...., w1) in a singlecorpus.The neural language model introduced by Bengio et al. [BDVJ03] hasreplaced statistical count based language models. It solves the sparsity prob-lem possessed by the earlier models by representing words as vectors. Re-search has shown that Neural Language Model outperforms count based183.2. Language Modellanguage models [NSR18]. Recurrent neural network form the fundamentalarchitecture of the neural language model. Figure 3.1 illustrates an ele-mentary language model architecture. The input to each RNN unit is theembedding of the current token. Embedding could either be randomly ini-tialized or is some learned vector. In the model, we ignore the output fromeach intermediate RNN unit. However, the output of the last RNN unithelps in decoding the next token in the code sequence. Softmax function isapplied on the output of the last RNN unit.The softmax is applied on the whole vocabulary V of the language model.The vocabulary V of a language model is defined as the set of word tokenswhich are used during the training of the language model. These are gener-ally most frequently occurring words in the whole corpus.Softmax(ti) =expti∑vj=1 exptj(3.15)In equation 3.15, v is the total number of words in the vocabulary V .Softmax function maps each token in the vocabulary V to a probability.The token with the highest probability in the vocabulary is chosen as theoutput and included as the next token in the sequence.Figure 3.1: Elementary Neural Language Model architecture193.3. Named Entity Recognition3.3 Named Entity RecognitionNamed Entity Recognition (NER) is the task of recognizing and tag-ging entities to their corresponding types. This is also known as a sequencelabeling task, as it tags sequence with appropriate entity types. In the do-main of coding these entities can either be objects, methods, classes, condi-tional statements, loops, data-types etc. Most traditional high-performancesequence labeling models are linear statistical models, including HiddenMarkov Models (HMM) which rely heavily on hand-crafted features andtask-specific resources. These hand-crafted features are difficult to developand changes with the change of domain.Currently, neural network architecture are employed for the sequencetagging task. Neural networks does not rely on any hand crafted featuresand task specific resources. They use an lstm architecture and decodinglayer to make the predictions of code tokens to code entities.We employ abilstm-crf sequence tagging model introduced by [MH16].This model consistsof bidirectional LSTM and a CRF decoding layer.3.4 Neural Machine TranslationNeural Machine Translation (NMT) is the task of converting a sequenceof text in a source language (e.g. English) to a sequence of text in thetarget language (e.g., French), which is performed using neural networks[LHWM20]. Therefore, neural machine translation models are also referredto as sequence to sequence models. Sequence to Sequence model introducedby Bahdanau et al. [BCB15] is an encoder-decoder framework that jointlylearns and aligns the translation. RNN units form the fundamental structureof encoder and decoder, as shown in Figure 3.2. It is important to note thatany variant of RNN such as LSTM, GRU can be used in this architecture.To understand NMT better, let us consider input sequence St.St = s1s2 ......sn (3.16)Where n is the total number of tokens in a sequence St.The first token s1 of the input sequence St is first mapped to its vectorrepresentation xt. This vector representation xt and previous hidden stateht−1 of the token s1 serves as input into the first encoder RNN unit. Initiallyht−1 is randomly initialized and learned as the model trains. Similarly xtcould also be randomly initialised or could be a learned embedding from[BGJM17]. A single RNN unit processes the two input vectors input token203.4. Neural Machine TranslationFigure 3.2: Encoder Decoder NMT modelembedding and previous hidden state; and generates two new vectors, thenext hidden state ht and output yt.In the encoder RNN units, the output vector yt generated at each timestep t is ignored. The generated hidden state ht serves as input to the nextRNN unit’s hidden state. RNN unit inside encoders is used to capture themaximum amount of context of the input sentence. This process is followedfor each input token. The final hidden state of the encoder from the lastinput sequence token sn is fed into attention-based decoder RNN. This finalencoder hidden state is also known as context vector as it contains completecontextual information of the input sentence.RNN’s are vulnerable to lose information as the length of the sentenceincreases. Modern NMT models use attention mechanisms in the decoder toaccess input directly from the encoder[BCB15]. Inside decoder, the currenthidden state ht of the decoder calculates attention score by performing thedot product of the current hidden state of the decoder and each encoder’shidden state (h1, h2...hn ). The softmax function is applied to the generatedattention weights, which gives the probability distribution for each token.A high attention score of a word means higher importance of a word atthe current time step t. Hence, attention further helps the neural machinetranslation model to attend towards more important tokens in the input togenerate more meaningful summaries.Each decoder RNN unit generates an output vector yt, which is sampledto the token in vocabulary by applying Softmax function on yt. The NMTmodel is trained end to end. The sequence of generated tokens from thedecoder forms the output of the NMT model.21Chapter 4Proposed ApproachThe details of the proposed approach are detailed in this chapter.4.1 Motivation((a)) Random initialization ((b)) Random initialization withexternal knowledgeFigure 4.1: Processing code sequence vs. incorporating external knowledgefor code comment generationThe tokens for a given code snippet are processed sequentially insideRNN-based NMT models. Therefore, before each token is processed by theneural model, it must be mapped to its corresponding vector representa-tion. Generally, the embedding for code comment generation models areinitialized randomly. Let us consider the token ‘public’ as shown in Figure4.1(a). When this sample is processed inside the NMT model, the modeldoes not have any semantic and syntax knowledge of the token. Thus, toenrich the learning of the NMT model by incorporating the semantic andsyntax of the given code snippet, current works rely on external sourceslike API knowledge [HLX+18b], AST [HLX+18a, LJM19], Unified ModelingLanguage diagrams (UML) [WDS+20], data flow graphs [GRL+21], and re-trieved similar samples [ZWZ+20] as shown in Figure 4.1(b). The authorsfor these works extracted external entities using tools like srcML [CDM13].224.1. MotivationFor example, in order to incorporate AST with the source code, it first needsto be extracted using tools like SrcML [CDM13]. Once AST is extracted,various tree traversal techniques like structure based traversal (SBT) as pro-posed in [HLX+18a] are used to flatten the tree representation, which is thenfed into a separate encoder as input to the model. Similarly external API’s,UML diagrams and similar samples can be combined with the existing codesamples to enrich the semantic and structural knowledge of the source code.Studies in natural language processing have shown the importance oflearning semantic and syntactic information through different embeddingtechniques and employing these embeddings inside the NMT model hasshown improvement for machine translation [QSF+18]. Based on these re-sults, we present a new neural embedding for source code tokens in thiswork. Our work employs a character-based language model and NER modelto learn the code’s general semantic and syntactic structure. Charactersplay an important role in programming languages. For example, Java codingconventions recommend using UpperCamelCase for class names and lower-CamelCase for parameter names. Consider the token ImageSprite, whichrepresents a class name, whereas image sprite can define a variable name.Here, a minor change in the upper case or lower case of the letter ‘i’ maychange the token type (depending on the context). These differences canbe captured using character-level language modeling. Current works in thecode comment generation model have not leveraged the character-based in-formation to learn semantics.Similarly, learning the syntax of code is crucial in code comment gen-eration. As shown in Figure 4.1(a), it is hard for the NMT model to drawsyntactic types between the tokens. To supplement the NMT model withthis information, we apply NER model to detect the code entities of eachtoken, such as ‘Modifier’ for ‘public’. As shown in Figure 4.2(a), we transferthis learned knowledge through the pre-trained embeddings. This allowsthe NMT model to draw some differences between the tokens before train-ing. NER models have never been used to generate syntactic embeddingsof code tokens. We combine this information with embeddings learned fromthe character-based language model and use this embedding (LAM + NER)to initialize tokens’ vectors of the NMT model. Now, if we compare ourembeddings with other works as shown in Figure 4.2(b), we can draw aclear difference: our work presents a syntactic representation at the embed-ding/vector level. In contrast, AST provides a structure-enriched lexicalsequence of tokens for code snippets. As our approach focuses on learningthe semantic and syntax at the embedding level, it can be combined withother works as demonstrated in Figure 4.2(b).234.1. Motivation((a)) Initializing tokens with pre-trained embedding((b)) Integration of our work withother approachesFigure 4.2: Initializing the NMT model with pre-trained LAM and NERembedding in our workAnother important aspect, which current literature misses to address,is the use of dynamic contextual embedding for code comment generation.Like natural languages, each code token can change its meaning and syn-tactic role with its context. For example, Figure 4.3, shows two methodsthat are related to accessing and processing data stored in a SQL database,and the sql token in each one has a distinct meaning. The sql in the leftmethod is given as a String in the parameters. This method returns a Pre-paredStatement (“an object that represents a precompiled SQL statement”1) for the given sql. Whereas the second method on the right retrievesthe Boolean value of the designated column. In this method, the token sqlis a Java API 2 as it is printed as java.sql. In both methods, the sametoken is present with a different meaning, context, and type. The dynamicgeneration of the embedding according to the token context can be usefulto make the differentiation.Figure 4.3: Change of the meaning of sql token with changing context1https://docs.oracle.com/javase/7/docs/api/java/sql/PreparedStatement.html2https://docs.oracle.com/javase/8/docs/api/java/sql/package-summary.html244.2. Overview4.2 OverviewFigure 4.4 illustrates the high-level architecture of our proposed ap-proach. First, a bidirectional character-based language model and an NERmodel are trained separately on a code corpus. The trained models are thenused to generate the pre-trained embeddings that we employ in the NMT ar-chitecture. These embeddings are the vector representation of code tokens.For each token, one vector is learned using each of the language and NERmodels, which are concatenated together to make a Semantic-Syntax Em-bedding. Like existing studies, [LHWM20, ZWZ+20, HLX+18a, HLX+18b]we use RNN based encoder-decoder architecture for the code comment gen-eration. This architecture provides a single end-to-end pipeline to generatecomments. Moreover researchers consider the problem of code comment gen-eration similar to the problem of Neural Machine Translation [LHWM20],therefore it is a preferred architecture for our use case. The NMT frame-work consists of a Semantic-Syntax encoder and one decoder. The input tothe encoder is the Semantic-Syntax Embedding. The encoder leverages thisembedding to learn the semantic and syntactical context of the given codesequence. The decoder uses the attention mechanism to decode the inputcode snippet to code comments. The details are discussed in the followingsections.Figure 4.4: Overview of LAMNER-Code Framework4.3 Learning Semantics of Code Sequences usingCharacter-Level Language ModelSemantic representation of code is critical to generate code comments[KBL19]. Recent works on natural languages show that language modelsthat are trained on characters intrinsically learn the semantic informationbetter, compared to the models that are trained on complete words [SMH11,254.3. Learning Semantics of Code Sequences using Character-Level Language ModelGra13, KJSR16]. Therefore, in contrast to previous works [FGT+20, KMBS20],we learn the semantic information of a code by leveraging a bidirectionalcharacter-level language model.Akbik et al. [ABV18] show that their proposed character-level contextualembedding is helpful in several classification tasks, including NER in naturallanguages. It can capture specific features of the text, such as differentiatingbetween lower and upper-case letters. These points are especially importantin source code representation, where each character plays an important rolein defining the semantic and syntactic structure of code.Language Model ArchitectureWe use the approach of Akbik et al. [ABV18] to train the languagemodel. Figure 4.5 illustrates the architecture of the language model, whichemploys a single layer bidirectional LSTM (Bi-LSTM). Bi-LSTM capturesthe left and right context of a character in the sequence [HXY15] usinga forward and a backward language model as shown in Figure 4.5. Theinput to each LSTM unit is an embedding of a character which is initializedrandomly. Each LSTM unit processes the embedding to generate the outputand the next hidden state for the character. The output of the last LSTMunit is used to select the next output character with maximum likelihood.In Figure 4.5, the model predicts the character l for the given sequence ofpublic Boo, which makes last letter of public Bool sequence.Extracting Semantic Embedding of Tokens from Language ModelThe language model explained so far, learns the character embeddings.However, we require to extract the embedding of each code token –we referto this encoding as semantic embedding. We use the trained character-levellanguage model and for each code snippet, we follow the technique proposedby Akbik et al.[ABV18] to extract the semantic embedding of each token.In this approach, the intermediate hidden states of the language model areused as the embedding for each token. Each hidden state of each charactermaintains the contextual information of its previous characters. Therefore,in the left to right model (forward model), the last character of a tokenencapsulates the information of all characters of the token based on its leftcontext. Similarly, in the right to left model (backward model), the firstcharacter of the token contains the information of the whole token fromits right context. This process is shown in Figure 4.6 which demonstratesthe extraction process of the embedding for the getBoolean2, shown as264.4. Learning Syntactic Structure of Code SequencesFigure 4.5: Bidirectional language model architectureE-getBoolean2, in the sequence public Boolean getBoolean2 (. Thisembedding is generated by concatenating the output hidden state after thelast character in the word in the forward language model, and the outputhidden state before the word’s first character from the backward languagemodel. This way, we have the information encoded from both left and rightcontexts.4.4 Learning Syntactic Structure of CodeSequencesEncoding the syntactical structure of the code has proven to producegood results as shown in previous works [HLX+18a, EHT16, AZLY19, ABLY19].Figure 4.6: Extracting semantic code embedding274.4. Learning Syntactic Structure of Code SequencesIn this section, we present our novel technique to encode the syntactical con-text of code. First, we train an NER model and then extract the learnedrepresentations to use as pre-trained embeddings for the NMT model. Weemploy a bidirectional LSTM sequence tagging model with a conditional ran-dom field decoding layer [HXY15] to map each input code token to its syn-tactic entity type. For example, using the NER model for the given code se-quence public Boolean getBoolean2 (String columnLabel) {...}, thetoken public is labeled as modifier, Boolean as return type, and getBoolean2as function. Note that the NER modeling used here is a supervised taskand requires a labelled dataset. We use the code entities extracted from theJavalang parser and use it to make a labelled dataset for training the NERmodel. The details of this step are explained in our experiments in Section5.2.2.To train the NER model, semantic representation from the trained char-acter language model of the code tokens are used as input to the LSTMunits, as proposed by [ABV18]. This semantic representation is generatedusing a language model with the same architecture as explained previouslyin section 4.3. As shown in [ABV18], embedding learned from the charac-ter based language model has better performance than the classical wordembedding FastText [BGJM17] and Glove [PSM14] when used to train theNER model. Therefore, we use character based embedding from the lan-guage model over other embedding techniques [PSM14, BGJM17] to trainthe NER model.After the NER model is trained, it is used to extract the syntactic embed-ding for each code token in the new, unseen sequences. Inside NER modelduring the decoding phase, CRF uses the knowledge from LSTM outputstates to predict the correct entity type. This means that the informationpassed from the Bi-LSTM to the CRF layer helps decode the correct syn-tactic type. Hence, the output state of Bi-LSTM units in the trained NERmodel encodes each token’s syntactic structure in a given code sequence. Itis important to note that we do not use the predicted entity types of the NERin the NMT framework. Rather, the extracted syntactical representation ofthe code tokens from the NER model is used. Moreover, as seen in [ABV18]the performance of NER model trained using character based model with-out CRF decoding layer is still better than other works. This part is crucialfor our work because we leverage the intermediate representations from theNER model before the CRF layer is applied on it.Note that the scope of this work is not to detect these code entities.We believe that parsers can detect the syntactic entities better. In com-parison, our work presents a new technique to generate context-sensitive284.5. Semantic-Syntax Modelsyntactic encoding of code tokens, which is not available through parsers.Interestingly, the NER model can detect parser labelled code entities withsufficient accuracy, which further corroborates that the model is learningcorrect embedding for each syntactic token type.4.5 Semantic-Syntax ModelThe pre-trained embeddings from the language model and NER modelare concatenated to represent the semantic-syntax embedding for tokens ina given code snippet, which will be used by the Semantic-Syntax encoder.Therefore, the encoder takes the semantic-syntactic structure of the codetokens as input and models them together to output a semantic-syntaxencoding of the code sequence. The attentive decoder uses this encodingand is trained to generate the code comments in natural language. Thedetails of the encoder and decoder architecture are in the following.Semantic-Syntax EncoderThe Semantic-Syntax encoder processes the input using a single layerof bidirectional GRU. We use GRU as it provides faster processing [She20],however still preserving the information for long sequences [CGCB14]. Wedenote Embedt as the input semantic-syntax embedding of token t, whichhas been learned from character language model and NER model. The pre-trained embedding is passed through a dropout layer: Et = Dropout(Embedt).The Dropout layer randomly drops the part of input embedding, disablingthe model to see the input vector completely. This eventually stops themodel to overfit on training data [SHK+14]. The obtained embedding Et isthen passed through the bidirectional GRU layer. Each GRU state processesthe input sequence embedding Et for the current token in the code sequenceand generates two different outputs: output state outt and hidden state ht.outt, ht = GRU(Embedt) (4.1)We ignore the output state and only use the hidden state. As the com-plete code sequence is processed, the final hidden state of the last token inthe sequence contains the sequential information of the complete sequence(i.e., including its previous tokens). As bidirectional GRU is used, both finalhidden states htleft , htright from left and right GRU are concatenated.hconcat = Concat(htleft , htright) (4.2)294.5. Semantic-Syntax ModelThe concatenated hidden state hconcat is fed into a fully connected linearlayer.yfc = hconcat ∗Wt + b (4.3)Where Wt and b are weight matrix and bias values, respectively. Theoutput yfc of the fully connected layer is passed through the tanh layer.The output of the tanh layer forms the final output hfinal from the encoder,which contains the semantic-syntax information of the code sequence.hfinal = tanh(yfc) (4.4)DecoderThe decoder uses the Semantic-Syntax encoder’s output and final hiddenstate of each input token to pay attention to the input sequence’s importanttokens. Attention mechanism prevents the decoder from generating naturallanguage comments based on a single final context vector. Incorporatingattention is important because theoretically RNN based neural networkscan carry longer contexts, however practically RNN units can only carry theinformation from the recent context of the language [CvMBB14]. Therefore,attention mechanism calculates weights for each input token hidden stateand pays more attention to tokens with larger attention values during thedecoding. Hence, leveraging the knowledge from longer context of sourcecode.As shown in equation (4.5) the context vector ci is a weighted sum ofall the input hidden states of each token. The weight is represented using αand is calculated using equation (4.6). Here eij is the energy with respectto previous hidden state i and is calculated as a function of previous hiddenstate si−1 and hidden state hj as shown in equation (4.7) [BCB15].ci =t∑j=1αijhj (4.5)αij =exp(eij)∑tj=1 exp(eik)(4.6)eij = f(si−1, hj) (4.7)304.5. Semantic-Syntax ModelOur model implements Bahdanau’s attention mechanism [BCB15]. Thedecoder only uses a unidirectional GRU. In addition, it receives the se-mantic embedding of the comments obtained from training a character-levellanguage model, with the same architecture described in Section 4.3. The de-coder continues predicting the the next token until it reaches the maximumsequence length or end of sentence token 〈eos〉. In this work, we experi-ment with two different techniques in the decoder that we denote as GreedyDecoder and Beam Search Action-Word Decoder. Both the decoders have acommon working process as explained here, and the difference is in the searchtechnique used to generate text. At each time step, the Greedy Decoder pre-dicts and keeps only one token with maximum probability score which willbe chosen as the generated output token. However, when beam search isused, at each time step, k tokens are predicted –k determines the width ofbeam search and can vary based on the computational support available torun the models. Therefore, different paths with various sequences are gener-ated. Finally, the sequence with the highest average probability score of itstokens is selected as the generated comment [BCB15]. When k = 1, beamsearch is similar to greedy search. We use beam width k = 3 in our model.In addition, we apply the ”action words” in our decoder using beam search.In a recent work, Haque et al. [HBWM21] demonstrate the importance of”action words”, referred as ’verbs’ in their work, to generate the code com-ments. It is shown that ”action words” are key to generate overall bettercomments. Therefore, to incorporate this idea, we use the Beam SearchAction-Word Decoder. Here, we select the samples with the highest proba-bility that end with an action word and has length equal to the average ofthe top 50 comments with maximum probability. This average length is usedbecause beam-search tends to select shorter sequences [YHM18]. The effec-tiveness of the techniques used in the Beam Search Action-Word Decoder isdemonstrated in our human evaluation study (see Section 7.3).31Chapter 5ExperimentsIn this section, the dataset preparation, training details, and baselinesto evaluate the effectiveness of the proposed approach are discussed.5.1 DatasetWe use the Java dataset collected by Husain et al. [HWG+19]. Thisdataset is used in other code comment generation works [WDS+20, FGT+20]and includes method and comment pairs gathered from the popular open-source non-fork repositories on GitHub [HWG+19]. The dataset is splitin train, validation, and test sets. We use the same splits in our study.It is important to note that we use the same splits as provided by theauthors of the dataset Husain et al .[HWG+19], to maintain consistencywith other works which used the dataset. The statistics for the dataset isprovided in Table 5.1, under the ‘#Records in Original Dataset’ column.For our experiments, we set the maximum number of code tokens to 256and comment tokens to 64, similar to Feng et al.’s work [FGT+20], wherethe length of code and comment is truncated to 256 and 64, respectively.However, a truncated code cannot generate a correct abstract syntax tree.Therefore, for fair comparison with AST based approaches, we further filterthe dataset to include only samples that have a maximum of 256 code tokensand 64 comment tokens. In addition, Code2Seq which is one of our baselinemodels [ABLY19], is not able to handle all of the records in this dataset. Ithappens because either the method does not contain any paths or cannotbe parsed by the Code2Seq parser 3. A similar issue has been reported in arecent paper by Mahmud et .al. [MFA+21]. Therefore, some more recordsfrom Husain et al. [HWG+19] dataset are dropped to have a consistentdataset for evaluation among all baselines. The number of the records afterremoving these samples that we use in our experiments are shown in Table5.1, in the ‘#Records Used in Our Experiments’ column.3We have contacted the authors of Code2Seq and these reasons are mentioned basedon consultation with Code2Seq authors.325.2. Model TrainingTable 5.1: Dataset details for code comment generationSplit # Records in Original Dataset # Records Used in Our ExperimentsTrain 164,923 137,112Validation 5,183 4,333Test 10,955 9,0165.2 Model Training5.2.1 Language ModelsIn our approach, we train two separate language models: i) a languagemodel to learn the semantic representation of code tokens, and ii) a lan-guage model to encode the natural language comments. The underlyingarchitecture for both models is the same as described in section 4.3. We usethe FlairNLP library to train both language models [ABB+19]. The firstlanguage model is trained on the code corpus of the train set, without usingtheir comments. Before training, we tokenize the code and remove inlinecomments using the Javalang library as previously applied in [ZWZ+20].Similar to previous works [KBR+20], the tokenized code is then further pre-processed to replace all the strings and numbers in the code with the specialtokens STR and NUM. The replacement with special tokens helps reduce thevocabulary and promote generalization of the language model [KBR+20].The second language model is trained on comments only without perform-ing any pre-processing. This language model is later used in the decoder. Toprevent the over-fitting, we use the dropout technique and set the dropoutof the language model to 0.1.5.2.2 Named Entity Recognition ModelTo train the NER model, we require a labelled dataset, where each codetoken is linked to its corresponding syntactic type. We use the Javalang li-brary to obtain a labelled dataset using the code part of the train set. How-ever, we ended up using only 50% of the train set of the original dataset dueto the computational restrictions (GPU runs out of memory). The Javalanglibrary provides a lexer and parser for the Java programming language,which provides syntactic type of each code token e.g., identifier, operator,separator, and keyword. Javalang parser is used previously in [BPB+20]and [LLZJ20] and is able to accurately label the code tokens with theirassociated types. Javalang parser labels the code tokens with an abstractgranularity. However, we require a more detailed information about code335.2. Model Trainingtokens so the NER model learns the finer nuances of the tokens. Therefore,we further modify the labels to draw the differences between the type ofidentifiers. We use a rule-based approach to divide the identifier type intofour different subclasses: class, function, object, and return types. More-over, we divide the separators into three categories: body-start-delimiter,body-end-delimiter, and end-of-line (EOL) character. The other separatorsare included in the pre-defined separator class identified by Javalang. Othertypes are self-contained and do not need a finer grained type. The modifieddataset has 18 different categories as listed in Table 5.2.We train the NER model on the train set and evaluate its performanceon the test set. The code entity types inferred by the Javalang parser areconsidered as the correct labels and the code entities are predicted usingNER model. To reduce the over-fitting threat, we used dropout layer withdropout probability 0.5, so that the model does not see the complete in-formation to over-fit on it. For evaluation of the NER model, we use themetrics applied for the evaluation of NER models, Precision, Recall, andF1 score as described below [ABB+19]. It is worth mentioning that thePrecision, Recall, and F1 score are only used to evaluate the NER model.Other metrics required for evaluation of the comment generation models aredescribed in Section 5.4.Precision =TruePositiveTruePositive+ FalsePositive(5.1)Recall =TruePositiveTruePositive+ FalseNegative(5.2)F1 = 2 ∗ Precision ∗RecallPrecision+Recall(5.3)Here, TruePositive is the number of samples that are correctly labeledby the NER model to belong to a group. FalsePositive are samples are themodel predicted to belong to a group incorrectly, and FalseNegative are theones that NER model incorrectly labelled to have another entity type. Theresults of the NER model evaluation are presented in Table 5.2. The lastcolumn of the table shows the number of tokens in the test set for each typethat are identified by the Javalang parser. The scores show that the NERmodel can correctly recognize the code entities. As mentioned previously,the goal of this work is not to predict more accurate labels for code tokensas the scope of our work is not to improve the language parsers. Based onthe obtained results, the embeddings learned through the NER model canbe used for representing the syntactical information of each code token.345.2. Model TrainingTo test the ability of the NER model, we test this model on a datasetof abstracted code. This evaluates the robustness of the NER model forrandom token names. The same dataset used for testing the NER modelis abstracted, where the identifier names and types are replaced by moregeneralized names (e.g. ”identifier” and ”type”). We follow the approachof previous studies for code abstraction [TWB+19], and add an id to eachgeneral token (e.g. ”identifier id”), which is used to capture the differencesof various identifier names (and similarly different types). The results ofthis study are shown in Table 5.2 under the column ”Abstracted”. A dropis seen in the precision, recall and F1 scores in this setting. But, the NERmodel receives an average score of above 89% in all metrics. This shows thatthe NER model can still predict the labels for many code entities correctly,even when the identifier names are abstracted. For some categories, such as’string’ the NER scores are not high, and for some other such as ’function’,the predictions does not change much. The decrease in the performance insome categories can be attributed to not using the Java coding conventionfor the identifiers. This can affect the results of the NER prediction, as themodel is learned using embedding from character based language model.Therefore, a change in the case of characters makes the model lose someinformation.Another interesting property of embeddings learned from the NER modelis revealed in Figure 5.1. This figure represents the T-Distributed Stochas-tic Neighbor Embedding (T-SNE) [vdMH08] plot of the syntactic embed-dings obtained from the NER model. We observe that similar to path-based embeddings, the vectors learned through the NER model form clusters[WRP19] around similar identifier types. Here, each cluster contains tokensthat belong to the same code entity group. This can visually show thatthe obtained embeddings from NER can be reliably used for representingsyntactic type of the tokens.355.2. Model TrainingFigure 5.1: Clustering of syntactic tokens based on NER embeddings365.2. Model TrainingTable5.2:ResultsofNamedEntityRecognitionmodelEntityPrecisionRecallF1NumberofTokensOriginalAbstractedOriginalAbstractedOriginalAbstractedmodifier1.00001.00001.00000.99971.00000.99998995class0.98090.67650.98330.43620.53040.556120920function0.98320.97290.97540.97820.97930.975527408object0.98440.77750.99450.98990.98940.870957317return-type1.00000.90981.00000.99831.00000.95204809separator0.99990.99831.00000.99861.00000.9985115159body-start-delimiter1.00000.99931.00000.99891.00000.999112130body-end-delimiter1.00000.99951.00000.99971.00000.999612128eol1.00000.99830.99970.99860.99980.998422760number0.98860.96250.99400.98770.99130.97503666string0.96080.72660.84900.37510.90140.49483840data-type0.99970.99500.99970.30800.99970.470411473boolean0.99220.99051.00000.99790.99610.99421151operator0.99990.99660.99990.99770.99990.997228147loop0.99920.99821.00000.99910.99960.99871331keyword0.99990.99940.99990.98960.99990.994513865conditionalSt.1.00000.99870.99980.99870.99990.99875951annotation1.00001.00001.00001.00001.00001.0000595Average0.99380.94440.98860.89170.96590.9040351645375.3. Baselines5.2.3 Semantic-Syntax ModelTo train the NMT model, we use the dataset as described in section5.1. For the input code, we apply pre-processing steps similar to [LZ18]and [ZWZ+20]: method names and other code tokens are split based onCamelCase. Variable names are split on ’ ’ character, and all the numer-ical and string values are replaced with NUM and STR tokens respectively.This pre-processing reduces the vocabulary size of source code from 346,804unique tokens to 29,268 tokens, while natural language comments have theirseparate vocabulary . Semantic-Syntax model is implemented with modifi-cations to Pytorch 4. The hidden size of the Semantic-Syntax model is setto 512 and it is trained for 100 epochs or until the learning rate decays to1e− 7. Initial learning rate is set to 0.1 and batch size is 16. Both encoderand decoder use a dropout layer with keep probability of 0.5. In our ex-periments, if the valid loss does not improve after 7 consecutive epochs, thelearning rate is decayed by a factor of 0.1. All experiments are conductedon a NVIDIA Tesla V100 GPU with 32 GB memory.5.3 BaselinesOther ApproachesWe compare the performance of our approach with the following mod-els. These models are used as a baseline in many of the recent commentgeneration studies in Software Engineering [WLL+20, HLWM20, ZWZ+20,LCX+21]. The availability of the models is another important factor tochoose these baselines.Seq2Seq uses a single encoder-decoder with attention mechanism [SVL14].The encoder takes a sequence of code tokens as input, and the decoder gen-erates comments as output by performing attention on the code tokens.Hybrid-DeepCom uses Structure Based Traversal (SBT) of AST rep-resentation to represent code [HLX+20]. SBT representation is used insidean encoder-decoder model to generate comments.Code2Seq uses a specific technique to represent code snippets fromAST. It considers all pairwise paths between leaf nodes of the tree, repre-senting them as sequences of the terminal and non-terminal nodes. Thispath is concatenated with the tokens of each leaf node and serves as inputto the encoder-decoder model [ABLY19]. Code2Seq is a recent model that4https://pytorch.org385.3. Baselinesuses the specified techniques to leverage the code’s syntactic structure fromAST.AST-Attend-GRU takes code and Structure Based Traversal (SBT)representation of AST as the input to two separate encoders. The inputinside two encoders is processed and combined to generate the final output[LJM19].TL-CodeSum uses a double encoder architecture to generate code com-ments [HLX+18b]. It applies a transfer learning method to utilize APIknowledge, which is learned as a separate task. This knowledge is trans-ferred to the model to produce comments.RENCOS [ZWZ+20] combines the information retrieval techniques withNMT based models. It takes input test sequence and two syntactically andsemantically similar code snippets as the input. The conditional probabilityfrom the three inputs is then fused to generate the final output.For all of these works, we use the official implementation available ontheir GitHub repositories. For the Seq2Seq model, its official implementationfrom OpenNMT 5 [KKD+17] is used.Variations of Our ModelWe consider four variations of our model as described below.LAM-Code. In this model, we only use the semantic embeddings ob-tained from the character-level language model and feed it to the NMTmodel.NER-Code. In this version, we only use the syntactic embedding fromthe NER model for training the NMT modelLAMNER-CodeStatic. This model uses the same NMT model shownin Figure 4.4. However, in this model, the Semantic-Syntax embeddingsare kept static and are not trained after the first epoch with other modelparameters. This model shows how the embedding would have performedwithout training them further.LAMNER-Code. This model uses the full concatenated semantic andsyntax encoding in the Semantic-Syntax encoder directly, as shown in Fig-ure 4.4. . The results of this model evaluate the effectiveness of combiningthe two embeddings for code comment generation. Note that in LAMNER-Code, the model is initialized with the Semantic-Syntax encoding and thevectors are tuned further during training. This model uses the Greedy De-coder.5https://opennmt.net/395.4. Evaluation MetricsLAMNER-CodeBeam-Act. This model is same as LAMNER-Code;however it applies the Beam Search Action-Word Decoder and the beamwidth is set to 3.5.4 Evaluation MetricsComment generation models are evaluated similar to the NMT mod-els, where the generated comments by the model are compared againstthe comments that exist in the dataset (i.e., ground truth). The qual-ity of generated comments is evaluated against ground truth using differ-ent metrics [PRWZ02, VZP15, Lin04, BL05]. Similar to existing works[WZY+18, ZWZ+20], we evaluate the performance of our model and base-lines using BLEU [PRWZ02], ROUGE-L [Lin04], METEOR [BL05], andCIDER scores [VZP15]. These metrics compute the scores for generatedcomments based on the ground truth (i.e., Reference comments). We usethe implementations from Sharma et al. [SASZ17] to calculate these metrics,which is used in previous work [CXH+21]. The BLEU score measures then-gram (n ∈ [1, 4]) geometric precision pn between generated summary (C)and ground truth (G) [PRWZ02]. Brevity penalty, BP is used to penalizethe shorter candidate sentences with length cl than the reference translationswith length rl [PRWZ02]. BLEU score is calculated as follows.BLEU = BP.exp4∑N=11Nlog pn, BP =1 if cl > rle1−rlcl , if cl 6 rl (5.4)ROUGE-L is a recall-based score and calculates the number of tokensfrom the References that appears in the generated text [Lin04]. ROUGE-Lfinds the F-score of the longest common sub-sequence (LCS) between thetwo sentences X and Y with length m and n, respectively [Lin04]. It isdenoted as FLCS below.RLCS =LCS(X,Y )m,PLCS =LCS(X,Y )n, FLCS =(1 + β2)RLCSPLCSRLCS + β2PLCS(5.5)β =PLCSRLCS(5.6)405.4. Evaluation MetricsMETEOR calculates the score using an alignment approach, in which aunigram in a sentence is mapped to zero or one unigram in another sentencein a phased manner [BL05]. Here, P and R are the precision and recallvalues and α, β and γ have default values of 0.9, 3.0 and 0.5, respectively.METEOR = (1− γ.fragβ). P.Rα.P + (1− α).R (5.7)CIDEr metric rates the correctness of the comments [ZWZ+20, WDS+20].It performs Term Frequency-Inverse Document Frequency (TF-IDF) weight-ing for each token and uses cosine similarity between the Reference and can-didate sentences [VZP15]. It is calculated with the following formula whereN = 4 and gn(ci) is the TF-IDF weighted vector. Term Frequency-InverseDocument Frequency (TF-IDF) is used to find important terms/tokens basedon their frequency and their frequency within documents. It is calculatedas a product of count of a term in the document which is then normalizedby the inverse of count of a term occurring in various documents. As shownin equation (5.8), the left termhk(sij)∑ωΩhl(sij)represents the term frequencyin the reference sentence sij . The remaining term is the inverse documentfrequency of the the given n-gram. If a term is rare and occurs only infew documents the inverse document frequency gets high value which giveshigher weight to the term. Moreover, vice versa is also true, if a term occursin all documents, TF-IDF would give it a lower weight for its calculation asit occurs in all the documents and therefore the term is not contributing tothe learning of the model.gn(sij) =hk(sij)∑ωΩhl(sij)log|I|∑IpImin(1,∑qhkspq)(5.8)CIDErn(Ci, Si) =1m∑jgn(ci).gn(sij)||gn(ci)|| ||gn(ci)|| (5.9)CIDEr(Ci, Si) =N∑11NCIDErn(Ci, Si) (5.10)All scores range between 0 - 1 except CIDEr, which uses positive realvalue to quantify the quality. For all the metrics higher numbers are con-sidered as better scores. For better comparison, BLEU and other scores areusually reported as the percentage score [ZWZ+20, LCX+21].41Chapter 6ResultsIn this section, we present the results to answer the following researchquestions:− RQ1: How does our proposed model perform compared to other base-lines? In this research question, we use the BLEU-n (n ∈ [1, 4]),ROUGE-L, METEOR, and CIDEr scores to evaluate the performanceof LAMNER-Code compared to baselines.− RQ2: Which parts of our model contribute more to the performance?Here, we compare the the results of LAM-Code and NER-Code withthe full model (both LAMNER-Code and LAMNER-CodeBeam-Act).We also investigate the effect of keeping the embeddings static on themodel’s performance.− RQ3: How does our model perform as code and comment length in-creases? An important aspect of NMT models is their ability to workwell as the length of input or output sequence increases. In this re-search question, we study how LAMNER-Code behaves as the codeand comment length increases compared to baselines.− RQ4: What is the effect of different language models and NER modelin our approach for comment generation? We experiment with twoother language models and then use one of them instead of the LAMembedding in our model, LAMNER-Code. We also incorporated ourembeddings in RENCOS and report the results for all evaluation met-rics.− RQ5: How well different models perform/generalize, when they aretested on abstracted code, where the identifier names and types arereplaced with more general tokens? For this research question, weexperiment testing RENCOS, two language models studied in RQ4and the variations of our model on abstracted code.426.1. RQ1: LAMNER-Code Performance− RQ6: Can integrating existing techniques or replacing the neural net-work architecture enhance the performance of our model? In this re-search question, we investigate the effect of adding an informationretrieval technique and combining AST to our model on the resultsobtained by evaluation metrics, as well as adding two models based onthe Transformer architecture that is considered as the state of the artneural architecture in natural language processing.6.1 RQ1: LAMNER-Code PerformanceTable 6.1 summarizes the results of the baselines and all variations ofour proposed approach. BLEU score, METEOR, and ROUGE-L have valuesbetween 0 and 1 and are represented as % scores. CIDEr is shown with realnumbers. Higher scores are considered to have better results. Effort hasbeen made to train the baselines with the best hyper-parameters.As shown in Table 6.1, the highest scores except BLEU-4 belongs toLAMNER-Code or its variation LAM-Code (with slight differences). REN-COS achieves the highest BLEU-4 which is 0.14 above LAM-Code. In al-most all scores, all variations of our model have the highest scores. ForBLEU-3, LAMNER-CodeStatic and NER-Code are 0.44 and 0.01 below REN-COS scores. The CIDEr score of LAMNER-CodeStatic is approximately thesame as RENCOS (0.02 score lower). Among the baselines, RENCOS hasthe best scores for all metrics. Our best results is obtained using the fullmodel, LAMNER-Code or LAMNER-CodeBeam-Act for BLEU-1, BLEU-2,and BLEU-3 which improves the RENCOS score by 3.26, 1.23, and 0.26,respectively. The best result for other metrics is obtained by the seman-tic part of our model, LAM-Code. In this case, ROUGE-L, METEOR, andCIDER scores of RENCOS are improved by 5.27, 1.25, and 0.1, respectively.The increase in the performance of our model is attributed to the training ofsemantic embedding with bidirectional character-level language model thatlearns the left and right contextual information.Among the AST-based models, the AST-Attend-GRU performs the best,and its performance is comparable to TL-CodeSum; although TL-CodeSumdoes not use structural information. The lowest results belong to Code2Seqwhich has surprisingly low scores compared to the scores reported in theirpaper. We ran Code2Seq a couple of times and double checked our code 6.6To double check, we applied Code2Seq on another Java dataset [HLX+18b], for whichit obtains good results. Therefore, we think the low scores of Code2Seq on the datasetreported here is not related to a problem in running the model.436.1. RQ1: LAMNER-Code PerformanceWe also contacted the authors of Code2Seq to ensure about correctness ofour results. The primary reason for low results of Code2Seq could be thedataset used in our work. It is a well curated dataset to remove trainingbiases [HWG+19] and contains more complex methods.Similarly, the scores for RENCOS is lower than what they reported,which can be related to having less similar code snippets available in ourdataset. The average similarity score between code snippets in this dataset is25.19% whereas the original dataset on which RENCOS used for training has41.83% similar code snippets 7. Therefore, as the similarity score betweenthe code samples decreases, the predictions by RENCOS become closer tothe predictions of seq2seq model [ZWZ+20].Summary of RQ1 results: LAMNER-Code or its variation LAM-Code achieve the best score among all baselines.7We computed the similarity score using the PyLucene API (https://lucene.apache.org/pylucene/) that is used in RENCOS.446.1. RQ1: LAMNER-Code PerformanceTable6.1:Evaluationofvariousbaselinesandourmodel.Thebestresultsareboldandthesecondtopresultsareshownwithunderline.ModelBLEU-1(%)BLEU-2(%)BLEU-3(%)BLEU-4(%)ROUGE-L(%)METEOR(%)CIDErSeq2Seq19.2910.297.323.8619.287.650.44Hybrid-DeepCom17.258.904.832.7221.069.50.57Code2Seq7.702.270.760.029.203.500.11AST-Attend-GRU22.029.975.213.2023.069.450.38TL-CodeSum21.6510.105.693.7422.108.360.34RENCOS22.8212.297.545.1527.4610.550.65LAMNER-Code Static23.0812.477.14.3831.1510.890.63NER-Code23.7213.117.534.6432.2111.580.72LAM-Code24.0713.427.795.0132.7311.800.75LAMNER-Code24.4513.527.804.8532.5211.770.73LAMNER-Code Beam-Act26.0812.536.753.8828.7011.240.59456.2. RQ2: Contribution of Semantic vs. Syntax Embedding in LAMNER-Code6.2 RQ2: Contribution of Semantic vs. SyntaxEmbedding in LAMNER-CodeThe results of variations of our model are reported in Table 6.1. LAM-Code has higher scores than NER-Code for all the evaluation metrics. Thisconfirms the importance of the semantic information that is captured bythis model. LAM-Code and LAMNER-Code have close scores for BLEU-3,ROUGE-L, METEOR, and CIDER, with LAM-Code having slightly lowerBLEU-1/2/3 scores. These results show that capturing the syntactic in-formation through the NER model is still important and can improve theresults, although there is a slight change in the scores. Albeit the perfor-mance of NER-Code is not better than LAM-Code, this model has higherscores than all the baselines (except for BLEU-4 compared to RENCOS).The reason is that the embedding from the NER model is able to cap-ture meaningful information. The NER model is trained using a context-sensitive representation of the code tokens, which is then tuned to capturethe syntactic knowledge of the code. This result is another confirmationabout the importance of learning the semantic information from code se-quences. Furthermore, it is worth mentioning that comparing the resultsof LAMNER-CodeStatic with the baselines shows the contributions of theembeddings in our model. The semantic and syntactic embeddings learnedthrough the language model and NER model are contributing in the modelperformance. Even when we use the embeddings as static vectors withoutfurther training, i.e. the LAMNER-CodeStatic, the results are improved. Al-though LAMNER-CodeStatic has less scores than the full model, for almostall metrics it has higher scores than the best baseline, RENCOS. Interest-ingly, LAMNER-CodeBeam-Act has the highest BLEU-1 score, but except forROUGE-L and METEOR, the other scores are not confirming the improve-ment of the results. In contrast to automatic scores, the human evaluationsshow that this model performs the best result (details are explained in Sec-tion 7.3). This can be due to the fact that the automatic metrics are based onthe common number of tokens between the generated and Reference texts,not relying on the semantic similarity (i.e. meaning) of two texts [CCS19].Among the LAM-Code and NER-Code, the results of LAM-Code are closeto our full model, LAMNER-Code. This behavior aligns with the results ofthe previous works that combine semantic and syntax information of codefor comment generation; the model that is only based on semantic learningor code tokens has the most contribution to the performance of the modeland its results are close to the full model which combines both information466.3. RQ3: Effect of Code and Comment Length Increase on Performance of Models[LJM19, ZWZ+20, ABLY19].Summary of RQ2 results: The semantic embedding (i.e., LAM) hasmore contribution to the model’s performance. LAM embedding and NERembedding both learn meaningful information about code and their combina-tion improves the results of using each embedding.6.3 RQ3: Effect of Code and Comment LengthIncrease on Performance of ModelsCode length and comment length are important factors studied for theperformance of neural machine translation [WZY+18]. To understand thiseffect, we plot the diagrams representing ROUGE-L and BLEU-1 scoresof the sentences versus code or comment length. The length identifies thenumber of tokens in the code/comment. Figure 6.1 shows two plots ofthe performance of LAMNER-Code, RENCOS, AST-Attend-GRU and TL-CodeSum for different lengths of code snippets. The left plot uses the BLEU-1 score and the right plot represents the ROUGE-L score. The value at eachcode length x is the average score of all code snippets with lengths between xand the previous length y shown on the plot. For example, the value shownas the score of code length 50, is the average score of all code snippetswith lengths between 26 and 50, inclusive. The same approach is used forplotting the comment length in Figure 6.2. All the models have the highestscores when the code sequence has 50 tokens. Like other models, the scoreof our model decreases as the code length increases; but, LAMNER-Codehas approximately a steady behavior as the code length increases. For allcode lengths, LAMNER-Code scores are above 20.Figure 6.2 represents the BLEU-1 and ROUGE-L scores of the four mod-els as the comment length increases from 5 to 30. The scores for commentlength 5 in the plots show the average scores for comments with length lessthan or equal to 5. Figure 6.2 shows that after length 10, as the lengthof the generated comments by TL-CodeSum, AST-Attend-GRU and REN-COS increases, their performance decreases. In contrast, LAMNER-Code’sBLEU-1 score has a steady increase as the number of tokens in its generatedcomments increases. Similar behavior is also seen in the ROUGE-L score ofthe LAMNER-Code, with a small drop in between. The better performancefor longer comments is only seen for our model, which is important as thehuman should read the comments and the quality should be maintainedwhen comment length increases. This result is attributed to pre-trainedlanguage model used in LAMNER-Code. Examples of the produced results476.4. RQ4: Effect of Language Models and NER EmbeddingFigure 6.1: BLEU and ROUGE-L scores compared to the length of the inputcode applied on test dataFigure 6.2: BLEU and ROUGE-L scores compared to the length of thegenerated comment applied on test dataare discussed in the Section 7.Summary of RQ3 results: For all code lengths, LAMNER-Code hasthe highest BLEU score and follows a similar trend as other baselines. How-ever, in contrast to other models, with the increase of comment length,LAMNER-Code’s BLEU score does not decrease.6.4 RQ4: Effect of Language Models and NEREmbeddingIn order to analyze the effect of the language model in LAMNER-Code,we explore ELMO [PNI+18] embedding and FastText [BGJM17] embed-ding and replace them with the character level language model that is usedin LAMNER-Code. ELMO is a character-based embedding that leveragesthe Neural Network-based architecture to learn the vector representation ofthe characters. FastText is a word embedding model that learns the sub-486.4. RQ4: Effect of Language Models and NER Embeddingword information. FastText has shown its effectiveness for semantic tasksin software engineering [WRP19] and ELMO can provide another perspec-tive to encode characters other than the LSTM based language model weused. Therefore, we choose these two embeddings for this research question.For this study, we train ELMO and FastText embeddings from scratch onthe same data that we used to train LAM embedding. For FastText, weset the minimum token size to one, so FastText can learn all possible sub-words information. We then employ the trained embeddings in the samearchitecture as used in LAM-Code to generate code comments, by replacingthe LAM embeddings with ELMO or FastText. The results of these newmodels are shown in Table 6.2 as ELMO and FastText. We compare thesemodels with the LAM-Code model. For all evaluation metrics, LAM-Codeachieves the best score. It improves the ELMO’s BLEU scores by 3.35%–13.35%, ROUGE-L and METEOR by less than 2% and CIDEr by 5.63%.LAM-Code improves the FastText’s BLEU1-3, ROUGE-L, and METEORslightly, BLEU-4 by 4.59%, and CIDEr by 10.29%. We attribute the betterresults of LAM-Code to the fact that this character based language modelhas shown to implicitly learn the semantic as well as syntactic knowledge ofa language [ABV18]. Hence, LAM embedding learns some syntactic infor-mation along with the semantic information of the code. However, FastTextembedding misses to learn such information [WRP19].Both ELMO and FastText models perform better than RENCOS in allscores, except BLEU-4 (and ELMO’s BLEU-3 is also lower than RENCOS)8.FastText performs better than ELMO in almost all scores. This is be-cause FastText learns the sub-word (n-gram) embedding of code tokens. Forexample, FastText learns the representation of identifier getMax using theinformation of get and Max and this sub-word learning helps FastText togenerate better results. It also helps FastText to generalize across lexicallysimilar tokens [WRP19].Based on the obtained results, we use FastText to analyze the usefulnessof NER embedding for comment generation. We train a new model withthe same architecture as LAMNER-Code and replace the LAM embeddingwith FastText embedding. In this model, the NER embedding is concate-nated to FastText and is fed in the model for generating comments. Theresults of this experiment are shown as FastTextNER-Code in Table 6.2.Interestingly, the results are improved compared to the scores obtained by8This result is interesting, although we made sure that all of the models are trainedand tested correctly.496.4. RQ4: Effect of Language Models and NER EmbeddingLAMNER-Code. This improvement is 0.33%–6.8% for BLEU1-4, 2.55% forROUGE-L, 2.8% for METEOR, and 8.22% for CIDEr. Among all mod-els, FastTextNER-Code achieves the best score in all metrics. As discussedpreviously, LAM embedding learns the semantic and syntactic information.Therefore, when concatenating the language model with NER embedding,we do not see much improvement as it already has learned some syntacti-cal information. Whereas, the NER embedding enriches FastText with thesyntactic information, as FastText lacks learning the syntactical information[WRP19] and combining NER adds new information to the model.As RENCOS is the best model among the baselines, we run anotherexperiment to evaluate the effect of our embeddings in RENCOS. We incor-porate the embeddings from character level language model and the NERmodel into the RENCOS architecture by initializing each token in the vo-cabulary of RENCOS with LAM and LAMNER embeddings. These modelsare denoted as RENCOS+LAM and RENCOS+LAMNER in Table 6.2. In-corporating LAMNER embedding has more effect compared to only LAMembedding, but the results of RENCOS are marginally increased for BLEU-1,2, ROUGE-L, METEOR, and CIDEr scores (from 0.33% to 1.62%). Thiscan be attributed to the retrieval part of the RENCOS as it provides extrasemantic and syntactic context to the model, thus, reducing the effect of theused embedding in the model.506.4. RQ4: Effect of Language Models and NER EmbeddingTable6.2:EvaluationoflanguagemodelsandNERembedding.Thebestresultsareboldandthesecondtopscoresareshownwithunderline.Forsimplicityofcomparison,theRENCOS,LAM-CodeandLAMNER-CoderesultsfromTable6.1arerepeatedhere.ModelBLEU-1(%)BLEU-2(%)BLEU-3(%)BLEU-4(%)ROUGE-L(%)METEOR(%)CIDErRENCOS22.8212.297.545.1527.4610.550.65RENCOS+LAM23.0512.387.544.9527.4410.600.66RENCOS+LAMNER23.1912.457.444.9427.5510.720.66ELMO23.2912.677.254.4232.2411.610.71FastText24.0413.237.694.7932.1811.760.68LAM-Code24.0713.427.795.0132.7311.800.75LAMNER-Code24.4513.527.804.8532.5211.770.73FastTextNER-Code24.5313.758.165.1833.3512.100.79516.5. RQ5: Evaluation of Models When Tested on Abstracted CodeFigure 6.3: Code AbstractionSummary of RQ4 results: Both LAM and NER embeddings are help-ful in comment generation. Language models are effective in creating com-ments, and the character level language model (LAM) achieves the highestscore among all. When NER embedding is combined with FastText, it canboost its scores by 2%–16% for all metrics.6.5 RQ5: Evaluation of Models When Tested onAbstracted CodeAll variations of our model and other baseline models presented in previ-ous research questions are trained on code snippets that include the identifiernames. Identifiers are defined by the developers, which means developers canuse different names for the tokens of the same source code without changingits functionality. Ideally, code comment generation models should handlesuch scenarios independent of the identifier names and demonstrate robustperformance against such cases. Therefore, in this research question, weevaluate the performance of the comment generation models on abstractedcode. For this experiment, we define a ’challenge dataset’, in which the codesnippets in the test set are abstracted. For the code abstraction, we followthe approach used in [TWB+19]. We replace each identifier name and itstype in the code snippet with a more generalized name like ’Identifier.’ and’Type’. An id is appended to each identifier and type depending on theirsequential occurrence. Figure. 6.3 presents an example of a code and itsabstraction. The code abstraction tests the ability of the models to gen-erate comments independently of identifier names and types by only usingthe generalized names. This dataset is only used during the testing of themodel, therefore, the new identifier and type names are not seen by themodels during the training.The results are shown in Table 6.3 and the highest scores are bold in thetable. We see a substantial decrease in the performance of all the commentgeneration models when tested on abstracted code. ELMO’s performancedrops significantly and is close to zero. For the NER-Code we also seevery low score in all metrics. Among the language models, LAM-Code526.5. RQ5: Evaluation of Models When Tested on Abstracted Codeand FastText are performing better and are comparable. The results ofRENCOS is close to FastText and LAM-Code. In case of RENCOS, themodel retrieves the most similar semantic and syntactic code samples duringthe decoding, which allows the model to see non-abstracted code tokens fromthe retrieved samples, introducing an advantage for RENCOS. Finally, thescores of LAMNER-Code and FastTextNER-Code are higher than othermodels. For these models, the NER embedding can boost their resultscompared to LAM-Code and FastText.The obtained results are expected, as most of the tokens in the abstractedcode are not available within the vocabulary of the models. Therefore, for allunseen tokens, i.e., the abstracted identifier names and types, the models usethe static representation of the unknown token (‘UNK’) at the embeddinglayer for the out of vocabulary words. This is also the case for LAM-Code,LAMNER-Code and FastTextNER-Code, however, these models use thepre-trained semantic embeddings of the tokens that are fine tuned duringmodel training, which help the models generate better results compared toothers when tested on abstracted code. The low scores of the NER-Code isbecause, as seen from previous experiments, syntax embedding individuallydoes not generate best results as it lacks the semantic learning and thesemantic learning has shown to be the most contributing factor for commentgeneration [LJM19, ZWZ+20, ABLY19].Based on this result, we provide another experiment with the LAM-Code and LAMNER-Code. In this research, rather than using the embed-ding lookup table for the tokens in the vocabulary, we generate embeddingfor each token at run time. These models are shown as LAM-Code-DYNand LAMNER-Code-DYN in Table 6.3, in which the contextual embed-ding of each token is dynamically generated using the language model andLanguage-NER model, respectively. This behavior of the LAM-Code-DYNand LAMNER-Code-DYN removes the dependency of the encoder on itsembedding look up table which makes the model use the static embeddingof the ‘UNK’ token. Therefore, each new ’UNK’ token is initialized to itscontextual representation, contrary to using the embedding of the ‘UNK’ to-ken in pre-trained embedding. The LAMNER-Code-DYN provides the bestperformance among all other models, and the LAM-Code-DYN achieves thesecond best scores (underlined in the table). This is one of the main advan-tages of using contextual embedding as it generalizes better compared toother baselines and even compared to models that use pre-trained embed-ding (i.e., LAM-Code, LAMNER-Code, and FastTextNER-Code). Similarexperiment cannot be applied on FastText because it would generate thesame embedding for the same token which does not change with the context536.5. RQ5: Evaluation of Models When Tested on Abstracted Codeof the given code.546.5. RQ5: Evaluation of Models When Tested on Abstracted CodeTable6.3:Evaluationofvariousbaselinesandourmodelontheabstractedcode.Thehighestscoresareshowninboldandthesecondhighestscoresareunderlined.ModelBLEU-1(%)BLEU-2(%)BLEU-3(%)BLEU-4(%)ROUGE-L(%)METEOR(%)CIDErRENCOS7.901.380.280.0710.982.660.02ELMO0.200.020.000.001.140.710.006FastText8.711.310.180.0017.753.200.01NER-Code1.820.060.000.004.361.210.01LAM-Code7.981.140.140.0011.202.760.02FastTextNER-Code12.662.050.440.1216.043.800.02LAMNER-Code9.061.910.400.0017.903.380.02LAM-Code-DYN12.493.831.070.3318.384.310.06LAMNER-Code-DYN11.833.861.470.7418.134.340.08556.6. RQ6: Effect of Combining AST and IR Techniques and The Use of Transformer ArchitectureSummary of RQ5 results: There is a huge performance decrease whenmodels are tested on abstracted code, which is attributed to the unknowntoken in the generated embeddings. The models which incorporate proposedembedding have better results. Moreover, dynamically generating contextualembedding of each token at run time using character level language modeland NER model alleviates this problem a bit and these models achieve thehighest scores in this setting.6.6 RQ6: Effect of Combining AST and IRTechniques and The Use of TransformerArchitectureIn this research question, first we investigate models in which we takeadvantage of the components of the best baseline models, namely AST andinformation retrieval techniques. Thus, we develop two models, where onecombines LAMNER-Code with AST and the other uses a retrieval tech-nique similar to RENCOS. The results of these models are shown in Table6.4 as LAMNER-Code+AST and LAMNER-Code+Retrieval. To incorpo-rate AST with LAMNER-Code in the former model, we introduce a dualencoder model similar to the AST-Attend-GRU model [LJM19]. We useAST as the input in one of the encoders and the source code as input tothe second encoder. The former learns the syntactical structure of codeand the latter learns the semantic information of code [HLX+20]. We com-bine both representations from the two encoders using a weighted atten-tion mechanism as described in Hybrid-Deepcom [HLX+20]. This attentionmechanism projects the outputs from both of encoders into a common vec-tor space and attend to tokens from both encoders. When AST is added toLAMNER-Code model, the precision-based metrics, i.e. BLEU scores areincreased by 1.85%–3.59%, and the other metrics, METEOR, ROUGE-L,and CIDEr are decreased by 1.19%–6.85%. The increase in the perfor-mance can be related to the exclusive encoder assigned in the model tolearn the syntactic structure of the code from ASTs. We also build a sim-ilar model to combine the FastTextNER-Code model with AST which isshown as FastTextNER-Code+AST. Adding AST to this model slightly de-creases the performance of FastTextNER-Code in most metrics and slightlyimproves the others (BLEU-1 and BLEU-2 scores). This results can be re-lated to the fact that the pre-trained embedding is successfully able to learnthe semantic and syntax information. Therefore, adding similar extra in-formation through AST (as a sequence of tokens and their types) keeps the566.6. RQ6: Effect of Combining AST and IR Techniques and The Use of Transformer Architectureperformance more or less same. Similar behavior is seen in previous workswhen adding AST improves the performance slightly [LJM19].For the other model, LAMNER-Code+Retrieval, similar to the approachused in RENCOS [ZWZ+20], we retrieve two similar semantic and syntacticsamples for each code during testing, with the difference that we use LAMand NER embeddings respectively to find the similar samples. Then, theretrieved samples are fused to generate a probability distribution of the out-put token, leveraging the retrieved samples. The performance of LAMNER-Code for all scores is decreased when a retrieval technique is used. Thiscan be due to finding the similar samples using the embeddings, instead ofincorporating the lexical similarity, as used in RENCOS, which can be amore effective retrieval technique.The final experiment in this part is related to the neural network archi-tecture. As explained previously, we use GRU architecture in our model.Although this architecture is used in other comment generation studies[LJM19], a more powerful architecture known as Transformer is consid-ered as the state of the art for many tasks in natural language process-ing field [VSP+17]. Therefore, we explore the use of our embeddings withthe Transformer. In this model, we incorporate the pre-trained embeddingsof LAMNER-Code to initialize the vocabulary of the Transformer basedmodel. We use Transofrmer with six stacked encoders and decoders and setthe embedding dimension to 512 to make it a fair comparison with otherbaselines. This model is shown as Transformer+LAMNER in Table 6.1. Wealso train a Transformer model, without using our embeddings and presentits results as the Transformer in the table. Combining LAMNER embeddinginto the Transformer degrades its results. This is aligned with the results ofa previous study where the authors incorporate structural information to theTransformer-based code summarization model and report a decrease in itsperformance [ACRC20]. They hypothesize that the Transformer architec-ture implicitly learns the code structure and including structural informationreduces the effectiveness of learned information. We hypothesize that samereason could be possible for the degradation of performance using semanticencoding as well, as the Transformer learns such contextual information.However, learning the structure or semantics of code is important for com-ment generation for which using semantic and syntax embedding may notbe the ideal technique to incorporate such information in Transformers. In-corporating other techniques like data flow graphs may be a better approachto learn this knowledge [GRL+21].576.6. RQ6: Effect of Combining AST and IR Techniques and The Use of Transformer ArchitectureTable6.4:Resultsoffusedmodels.Forsimplicityofcomparison,theLAMNER-CoderesultsfromTable6.1arerepeatedhere.BstandsforBLEU.Thebestscoresareshowninbold.ModelB-1(%)B-2(%)B-3(%)B-4(%)ROUGE-L(%)METEOR(%)CIDErTransformer25.2513.237.414.5729.6110.800.57Transformer+LAMNER24.0112.286.563.7829.1310.160.48LAMNER-Code+Retrieval20.349.584.952.8826.488.150.36LAMNER-Code24.4513.527.804.8532.5211.770.73LAMNER-Code+AST25.3213.777.985.0031.8211.630.68FastTextNER-Code24.5313.758.165.1833.3512.100.79FastTextNER-Code+AST25.0213.858.155.1133.1212.100.76586.6. RQ6: Effect of Combining AST and IR Techniques and The Use of Transformer ArchitectureSummary of RQ6 results: Our proposed embedding, LAMNER, canbe complemented by AST-based code representation. The LAMNER-Codehas better performance than the Transformer, but incorporating this embed-ding to Transformer is not ideal to learn the syntactical information.59Chapter 7DiscussionAlthough LAMNER-Code has better results than the baselines, it stillremains unclear in what scenarios LAMNER-Code can generate good sum-maries. Manual investigation similar to [HLX+20] is conducted to learn suchscenarios for the generated comments. Therefore, in this section, we ex-plore samples from the test data generated by LAMNER-Code, LAMNER-CodeBeam-Act, RENCOS, AST-Attend-GRU, and TL-CodeSum to answerthe following questions.− Where does LAMNER-Code perform better than the top baselines?− Where does LAMNER-Code fail to generate good comments?7.1 Inspecting Higher BLEU Score CommentsWe randomly sample 100 methods for which LAMNER-Code has at least25 or more BLEU points than the baselines. 25 is used as the minimumthreshold here because LAMNER-Code’s average BLEU-1 score is close to25. We then investigate the performance of our model based on function-ality of the methods. To categorize the code functionalities, we follow anapproach similar to the one used in [HLX+20]. Two graduate students,with 4 and 6 years experience in Java development, independently label thefunctionality for each of the 100 methods. We measured the inter-rater reli-ability using Cohen’s Kappa coefficient [McH12]. This metric measures thelevel of agreement between two raters when rating categorical values. TheCohen’s Kappa test result is a number between −1 and +1, where −1 showsthe highest disagreement rate and +1 represents that raters are in 100%agreement. In software engineering field, the rate of 0.79 or higher is consid-ered as a high agreement rate [LHX+20, HSX+18]. We obtained the CohenKappa value of +0.82 which shows the two raters have high agreement intheir defined categories. Finally, the labeling results are discussed betweenboth raters to reach to a common decision about all categories.The categories from this manual investigation are task implementation,object creation, update/set value, connection operation and string operation.607.1. Inspecting Higher BLEU Score CommentsThe task implementation category includes code samples that accomplishsome task such as finding the maximum row count. Code samples whichupdate and set the values for objects are listed in the update/set value andcode samples that create new objects are labeled as object Creation. Con-nection operation group includes samples that are related to setup, opening,closing, and deleting connections between client and server. Finally, stringoperation category includes samples that perform operations such as splitand concatenation of strings. The number of samples in each category areshown in Table 7.1. The distribution in Table 7.1 shows that LAMNER-Code generates better comments primarily in the task implementation, up-date/set value and object creation categories with 31%, 24% and 21% ofcode samples, respectively. The lowest percentage is 9 and belongs to stringoperation. It is worth mentioning that these results are only based on thesample data used for manual evaluation and may not reflect better perfor-mance on a certain category in the whole dataset. The distribution of thedata might differ in the sample and in the whole dataset. However, findingthe distribution of the whole dataset is not feasible as it requires a lot ofhuman effort to annotate the data for each category correctly. Therefore,we followed the techniques and approach of the previous studies for thisdiscussion [HLX+20]. Figure 7.1 represents a code sample and its gener-ated comments by LAMNER-Code and other baselines. This is a samplewhere LAMNER-Code has at least 25 higher BLEU score compared to thebest baselines. The Example ID in the figure shows the ID of the recordin the test set. From the sample in Figure 7.1, we observe that LAMNER-Code is able to generate important code tokens like buffer and offset inthe method. More examples are included in the appendix. We investigatethe ability of the models in capturing the identifiers in their comments asfollows.Table 7.1: Functionality distribution of the 100 sampled Java methodsFunctionality Number of samplesTask Implementation 31Object Creation 21Update/Set Value 24Connection Operation 15String Operation 9Generating Identifiers. Identifiers are names used for identification617.2. Inspecting Low BLEU Score Commentsin Java such as method names and variable names. Including, identifiersfor comment generation can be beneficial as describing the functionality ofcode depends on the identifiers used inside code [HLX+18b]. In our dataset,separators and identifiers contribute 46.00% and 37.87% of total tokens re-spectively. However, separators such as [ {, }, (, ), ”,”, and ”;” ], if used inthe comments, do not reveal much information about the functionality ofcode. Whereas, including identifiers inside the comments can reveal a lot ofinformation about what the method does [IKCZ16]. In this section, we studythe ability of the baselines, LAMNER-Code and LAMNER-CodeBeam-Act tocorrectly refer to the identifiers in their generated comments. To evaluatethis, for each code sample in the test set, we collect all the unique identifiersusing the Javalang parser. The collected identifiers are then checked for theiroccurrence in the corresponding comments of LAMNER-Code, LAMNER-CodeBeam-Act, RENCOS, AST-Attend-GRU, TL-CodeSum, and Referencesentences. As NMT models can generate repetitive tokens, for each modelwe count the occurrence of an identifier in each sample once, even if it hasappeared multiple times in that sample. For example if the identifier ”max-imum” appears twice in the comment finds the maximum and returns themaximum, we count the occurrence of ”maximum” only once. We com-pare these numbers with the number of occurrences of the identifiers in theReference sentences.The results are shown in Table 7.2. LAMNER-CodeBeam-Act and LAMNER-Code are able to produce 76.33% and 73.19% of the identifiers in the gener-ated comments, compared to 62.42%, 66.98% and 71.98% of the identifierscaptured by AST-Attend-GRU, RENCOS and TL-CodeSum respectively.This is 1323, 889 and 413 more identifiers than AST-Attend-GRU, REN-COS and TL-CodeSum for LAMNER-CodeBeam-Act. Interestingly, althoughthe NMT results for RENCOS are better than TL-CodeSum, TL-CodeSumcan better refer to identifiers in its comments. We relate this to the usage ofAPI knowledge in TL-CodeSum. On the other hand, LAMNER-Code andLAMNER-CodeBeam-Act can correctly predict identifiers without using anyexternal knowledge source.7.2 Inspecting Low BLEU Score CommentsTo investigate the low BLEU score comments, we follow the same ap-proach used in [HLX+20]. We randomly sample 100 records from the testset which have BLEU scores less than 20%. These samples are then dividedinto three categories, i) Good, ii) Related and iii) Not Related. The Good627.2. Inspecting Low BLEU Score CommentsFigure 7.1: Samples with higher BLEU-1 scoreTable 7.2: Number of identifiers in comments of each model compared tothe ReferenceModels Number of identifiers in commentsHuman Written References 9510LAMNER-CodeBeam-Act 7259LAMNER-Code 6960TL-CodeSum 6846RENCOS 6370AST-Attend-GRU 5936category contains samples that are accurate and may have low BLEU scorebecause of using synonyms or related words rather than repeating the orig-inal tokens used in the Reference. Related group contains samples that arerelated to the method definition, but the comments are not exactly accurate.Finally, Not Related is the group of sample codes for which LAMNER-Codeis not able to generate meaningful comments. For this categorization, sim-ilar to the previous section, the comments are labelled independently bytwo graduate students. The Cohen’s Kappa score is 0.81 and the studentsdiscussed the disagreements until they reach a consensus. The percentagesof comments in each group are shown in Table 7.3. The results show thatdespite having low BLEU scores, LAMNER-Code is able to generate accu-rate comments for 37% of the samples, and 39% of its generated commentsare related to the functionality of code. LAMNER-Code only fails in pro-ducing meaningful comments for 24% of the samples. Below, we discuss ourobservations for comments with BLEU score equal to zero.Comments with BLEU Score 0. In our study, we found exampleswith zero BLEU score, where the human written comments are either un-637.2. Inspecting Low BLEU Score CommentsTable 7.3: Classification of samples with low BLEU scoreCategory Number of samplesGood 37Related 39Not Related 24related or not updated with the project update. An example is shown inFigure 7.2, where the BLEU scores for all models are zero. The referencecomments of this sample aims to describe methods’ functionality; but thecomment is outdated and is not a correct description of what the methoddoes. In contrast, comments generated automatically by LAMNER-Code,LAMNER-CodeBeam-Act, and the baselines are more logical and related tothe code. This example emphasize the need for automatic code commentgeneration techniques as part of the project maintenance [NPLN20]. Moreexamples are included in the appendix.Analysing Low frequency word: Similar to RENCOS[ZWZ+20] westudy how well LAMNER-Code can generate the rare words present withinthe train split of the dataset during the comment generation for the testsplit. To investigate this behavior, we follow the same technique followed inRENCOS[ZWZ+20] and draw a comparison between LAMNER-Code, REN-COS, and TL-CodeSum. Like RENCOS, we first collect the valid words pre-dicted by the baselines that overlap with the test’s true reference predictions.A word is counted as a valid word if the word was there in the ground truthand the prediction. Then we find all the low frequency words in the train-ing set frequency = (1, 2, 5, 10, 20, 30, 50, 100). Then each low-frequencytraining set word is checked for its frequency in the test predictions validwords. The results of the experiment are shown in Table7.4. RENCOS,which is a specifically devised technique to overcome the problem of lowfrequency, performs best out of all three models, illustrating the importanceof retrieving similar code samples for RENCOS. LAMNER-Code performspoorly for very low-frequency words (frequency = 1, 2, 5, 10, 20). However,as the frequency of the words increases, it starts to perform better than TL-CodeSum. We expected such behavior because essentially, LAMNER-Codedoes not leverage any other extra knowledge from similar code snippets andneither from transferred API knowledge.647.3. Human EvaluationTable 7.4: Count of low frequency tokens in the test dataFrequency 1 2 5 10 20 30 50 100RENCOS 24 29 54 98 180 284 504 1062LAMNER-Code 0 0 1 5 38 103 278 728TL-CodeSum 9 18 31 42 73 101 186 429Figure 7.2: Samples with BLEU-1 score equal to zero7.3 Human EvaluationBLEU [PRWZ02], ROUGE-L [Lin04] and METEOR [BL05] are exten-sively used in machine translation to draw a quantitative comparison be-tween the NMT models. The models with more overlapping tokens betweenthe References and predictions receive higher scores. However, the textscan have the same meaning without using common tokens. This makes itdifficult to comprehend the effectiveness of different models [SDP20]. Re-search in both software engineering [SGL+20] and natural language process-ing [SDP20] fields has raised these issues when automatic translation metricsare used. Therefore, following previous works [HLX+20, IKCZ16, ZWZ+20],we conduct a human evaluation survey to compare the generated commentsby LAMNER-Code and the best performing baseline, RENCOS. The hu-man evaluation in this section, is to assess the quality of the sample com-ments based on three criteria of correctness, naturalness, and informative-ness, which cannot be shown using the automatic evaluation metrics.For human evaluation study, we randomly select 100 samples from thetest set. For each sample, we collect the Java code, reference comment, andthe comments generated by LAMNER-Code, LAMNER-CodeBeam-Act, andRENCOS models. Then, we divide the 100 samples into four equal size657.3. Human Evaluationgroups. We invite eight volunteers with previous one to ten years of experi-ence in Java development for the evaluation. Each group of the data contains25 records and is rated by two volunteers using a survey system 9. The sur-vey contains three criteria for each of the code samples. To reduce bias, thename of the models and Reference are not shown to evaluators and they onlysee four lines of text without knowing the origins. The volunteers are askedto rate the comments in a range of [1, 5] stars , where one star represents theworst and five stars is the best score, based on the following criteria: i) Cor-rectness, ii) Naturalness/Fluency, and iii) Informativeness. Similar ratingsystem is used in previous works [IKCZ16, LCX+21, WZS+20]. In Cor-rectness, we ask whether each comment describes the functionality of thecode correctly. Naturalness/Fluency is the criterion where we care aboutthe fluency of the comment considering only the English aspect of it: beinggrammatically correct and be fluent in terms of human readability. Infor-mativeness is the last criterion where we ask whether the comment correctlymentions the required content from the code (e.g. capturing necessary iden-tifier) to describe its functionality, ignoring the fluency of the text. Thevolunteers are shown an example with explanations for all criteria beforethey start rating.Table 7.5: Results of human evaluationModel Correctness Naturalness InformativenessGround-Truth 3.61 4.37 3.79LAMNER-Code 3.22 4.00 3.22LAMNER-CodeBeam-Act 3.74 4.56 3.84RENCOS 2.77 3.85 2.94Table 7.5 shows the average scores of the models as rated by the volun-teers on three metrics. LAMNER-Code predictions are rated better thanRENCOS on all three metrics, which show the effectiveness of the em-beddings from language model and the NER model. Although LAMNER-CodeBeam-Act showed to have the best results only for BLUE-1 score, surpris-ingly, the evaluators rated the LAMNER-CodeBeam-Act results better thanthe ground truth summaries (i.e. References). It can be related to the pre-diction of action words in the generated comments, confirming importanceof action words as suggested by Haque et al. [HBWM21]. We believe this isrelated to the fact that action words can help producing texts that are more9https://it.ubc.ca/services/teaching-learning-tools/survey-tool667.4. Threats to validityreadable and increase the comprehension of the text, though this shouldbe verified in a separate research. We relate the discrepancy between theautomatic and human evaluation results of LAMNER-CodeBeam-Act to thefact that quantitative metrics might not suffice evaluating the performanceof a model [SDP20].7.4 Threats to validityInternal Validity: Internal threats in our work relates to the errors inbuilding our own models and replication of the baselines, as these are the in-ternal factors that might have impacted the results. We have double checkedthe implementation of our model for its correctness and open sourced theimplementations for easier replication of the results. For all the baselines,we used the official code provided with their papers. During training thebaselines, if we encountered any errors or had any questions, we directlyasked the authors by raising GitHub issues.An important difficulty of training the models is preparing the datasets inthe required format. Various baselines required separate pre-processing suchas generating AST, using third part libraries like py-lucene 10, and extractingAPI knowledge. For these steps, we ensured about the correctness of therequired input by following the instructions provided for the baseline modelsor the libraries. This ensures that correct input is used for each baseline. Inaddition, we made our best efforts to choose the best hyper-parameters forall the models.Another threat to validity could be the dataset we built for the NERmodel and the dataset refining did for the training of the code commentgeneration models. The dataset refining lead to change of count distributionof the train, valid and test set, which is different than the original splitpercentage. Hence,there is a small threat for the model training. However,the same split and samples are used during model training of the baselinesas well, therefore the evaluation and results has no threat to validity forits correctness. We used the Javalang parser to label the dataset. Thisparser reliably tags the Java methods and have been used in several softwareengineering studies [ZWZ+20, LLZJ20]. Further, we labeled the identifiersinto four sub-categories using a rule based script. As the major type ofthese sub-categories, i.e. identifier, is tagged by Javalang parser, even if therule-based approach is not 100% accurate, the sub-categories are still partof the identifiers; therefore, the threat is alleviated.10https://lucene.apache.org/pylucene/677.4. Threats to validityExternal Validity: In our study, threats to external validity whichrelates to the generalizability of the results [HLX+20], refers to the choice ofdataset. We used an external and extensively used dataset to test our work.The authors of this dataset have removed biases by eliminating versions ofduplicate codes [HLX+20]. We only applied our model on Java programminglanguage. Although we hypothesize that the embeddings used in our workis beneficial to other programming languages, more studies are required.Construct Validity: In our work, the automatic evaluation metricsmight affect the validity of the results. We used four different automatic ma-chine translation metrics, to reduce the bias of a particular metric. Thesemetrics are frequently used in natural language processing and in relatedand similar studies in software engineering domain [FGT+20, ABLY19,ZWZ+20, WZY+18, HLX+20, WLL+20].Conclusion Validity: The conclusion validity refers to the researcherbias or bias in the statistical analysis that can lead to weak results [ABA+19].The nature of this study does not depend on the researchers’ bias. The con-clusion threat can be related to the reliability of the measurements obtainedby the automatic metrics. To alleviate this problem, we used different eval-uation metrics (BLEU1-4, ROUGE-L, METEOR, and CIDEr) and appliedthe same implementation of these scores to evaluate all of models. Althoughusing the automatic metrics to compare the results cannot quantify the qual-ity of the generated comments, these are the only metrics used in naturallanguage processing and software engineering for neural machine translationmodels. Therefore, we also conducted human studies to compare the resultsof different models from developers’ perspective. A potential threat couldbe related to the conclusions obtained by the human study. To reduce thisthreat, we set up our experiments in such a way that the name of the mod-els and the Reference comment are not shown to the evaluators, and eachrecord is rated by two evaluators.68Chapter 8Exploratory Study forAttention Comprehension inBERT8.1 IntroductionIntroduction of pre-trained language models (PTM) [DCLT19] based onTransformer networks [VSP+17] are widely used in the field of Natural Lan-guage Processing (NLP) since 2018. These language models are trained ona large corpus of data and have proved to capture the general understandingof any language [PRL+19]. These models have an exceptional performanceto recall their learning done during the pre-training stage of the models[PRL+19]. Later, PTM can be used to fine-tune any given task. More re-cently, the software engineering community has employed language modelingextensively on tasks like code retrieval, program repair, code documentation,bug detection [KS20, FGT+20, KMBS20, GGH+21]. Interestingly, the re-sults obtained by fine-tuning the language models outperform the previousstate-of-the-art models. Under the hood, these language models fundamen-tally use Transformer architecture in their design. The importance of thisarchitecture can be derived from the fact that employing it without anyother technique can generate state of the art results in various tasks suchas code summarization, code retrieval, code clone detection, program re-pair [WZZX20, ACRC20, ACRC21, FGT+20, GRL+21]. Moreover, as wesaw in Chapter 6 , the results obtained by the transformer model withoutusing any external knowledge, for code comment generation was very closeto the model proposed in this thesis. However, little is known about howTransformer models the source code. Currently, the knowledge from NLPstudies is directly applied to reason the success of the PTM’s. However,both programming languages and natural languages have similarities anddissimilarities between them [CSD18]. Programming languages are repet-itive and are structural as compared to other natural languages [CSD18].698.2. Transformer and BERT ArchitectureTherefore, directly applying NLP knowledge may not be ideal to assume forsimilar behavior for source code. Hence, it becomes important to study themodeling of the code in a Transformer architecture using language models.Hence, the reasons mentioned above motivate us to explore the PTM forsource code.This study evaluates the BERT model, trained on source code corpusas a classical NLP pipeline. An NLP pipeline analyses the language basedon its linguistic features, which can contain semantic and structural prop-erties [MCH+20, TDP19]. Currently, the literature for understanding thepre-trained models also considers them models based on distributional hy-pothesis. The distributional hypothesis assumes words that occur in thesame context should have the same meaning [RG65]. Therefore, the co-occurrence of tokens is an important factor for studies based on distribu-tional hypothesis [SJH+21]. However, as the distributional hypothesis doesnot consider semantic and syntactic learning as two separate tasks, it wouldnot be ideal to use it to understand the knowledge learned by PTM’s forsource code. The primary reason is that source code is meant to have asemantic and syntactic structure in it. Also, programming languages arehighly structured. Therefore, initially in our study, we explore the attentionbehavior of the BERT model on different token types. Then, in a separateresearch question, we want to understand what layers in the BERT modellearn in terms of semantic and syntactic knowledge.8.2 Transformer and BERT ArchitectureIn this section, we explain the necessary background about the Trans-former architecture and BERT model.8.2.1 TransformerTransformer forms the fundamental units of the BERT [DCLT19] whichsupport and introduced the idea of parallelization. Transformers are stackedstructures of encoders and decoders. BERT solely uses the Transformer’sencoder stack because only encoders allow the bidirectional nature in theTransformer. Each encoder unit contains two layers (i) Multi-Head SelfAttention layer (ii) Feed Forward layer.Multi-head Attention Layer:Transformer encoders uses self-attention[VSP+17] technique exclusively. It helps the encoder attend to other wordsin the input sentence as it encodes a specific word. Effectively self-attention708.2. Transformer and BERT Architectureis a way of computing the representation of the input sequence by relatingevery input sequence element with each other.Figure 8.1: Converting input to query, key, value vectorsFigure 8.1 and figure 8.2 comprehensively explains self-attention. Figure8.1 and figure 8.2 uses two token sentence “Love NLP” and performs self-attention on it. Each input token is first converted into its vector embedding.Each encoder layer consists of three sets of weights matrices, namely queryW(q), key W(k), and value W(v), which are in the beginning randomlyinitialized. The vector multiplication between input embedding x(i) andquery W(q), key W(k), and value W(v) matrices generates unique queryq(i), key k(i), and value v(i) vectors, respectively. The attention scores areobtained by performing dot product between each vector q(i) and vector k(i)as depicted in figure 8.2. The attention score obtained using the dot productis then divided by the square root of the dimension of the input embedding.The softmax function is applied to the attention scores to obtain attentiondistribution. Each softmax score is multiplied with the value vector v(i) togenerate weighted vectors added to get the final vector z(i). Similarly, other718.2. Transformer and BERT ArchitectureFigure 8.2: Converting input to query, key, value vectors728.3. BERT: Bidirectional Encoder Representations From Transformersvectors are also generated for all the tokens. This same task is repeatedmultiple times to form multi-headed attention. Each attention head learnsa different vector space which is then combined.Adding Positional Encoding: In Transformer, a complete sequenceof text is input into a multi-head attention layer at once, which helps themodel promote parallelization. However, we also want our model to main-tain and learn the syntactic correctness of the language. Languages aresequential, and changing the order of words can lead to either a completelydifferent meaning or a disgruntled sentence, which does not have any congru-ous meaning. So positional encodings (vectors) are added to input embed-ding to achieve this task. The word embedding added with the positionalencodings is fed into the multi-head attention layer. There are two waysusing which these positional encodings can be added to the model. Eitherby using a cyclic, sinusoidal function or static vectors are added to the inputembedding.Residual connection and Add Normalize: The output of multi-head attention is added to residual connection from x(i) (embedding layer),and layer normalization is performed on the added vector. Normalizationgenerally helps in the faster training and stabilization of hidden states.Feed forward neural network layer: The output from the add nor-malize layer is then fed into the feed-forward neural network, which has acommon architecture for all the tokens. However, this step is done in par-allel on each token. The output is then added and normalized. This finallycompletes a single feed-forward propagation in one encoder unit. The out-put from this encoder unit is fed into the next unit, and the same processingis done on it.8.3 BERT: Bidirectional EncoderRepresentations From TransformersBERT is a pre-trained language representation model trained on thetask of language modeling. BERT’s core motivation is to build a generalbi-directional model, which can be fine-tuned to various downstream NLPtasks.BERT-Pretraining: BERT is pre-trained on two unsupervised tasks(i) Masked Language Modelling (ii) Next Sentence Prediction. BooksCor-pus (800M words) and English Wikipedia (2,500M words) are used to trainBERT. Masked Language Modeling: Unlike neural language models,BERT is pre-trained using Masked Language Modeling. In Masked Lan-738.4. BERT Attention for Source Codeguage Modeling, 15% of the tokens are randomly masked. Masking of thetokes follows a set pattern in which a token is replaced with (1) the [MASK]token 80% of the time (2) a random token 10% of the time (3) 10% ofthe time token remains unchanged. The intuition behind keeping the tokenunchanged is that we would not encounter the” MASK” token during fine-tuning the model. So to empower the model for successful fine-tuning, this isa pivotal step. Masked language modeling restricts the model from knowingits left and right context at the same time. Next Sentence Prediction:BERT is also pre-trained on a binary next sentence prediction task. Thistask focuses on training a language model that can discern whether thegiven sentence is the next sentence of the current sentence. This is done byrandomly replacing the actual next sentence 50% of the time with randomsentences.8.4 BERT Attention for Source CodeThe Transformer-based model primarily uses the multi-headed atten-tion to model languages. An attention distribution for a given sentence canreveal how much importance the Transformer model gives to a particulartoken. Higher attention values mean a higher concentration of the modelfor learning the representation of a given sentence. Therefore, knowing theimportant tokens within the source code is important for extracting mean-ingful information from the PTM’s. To learn this attention information, weplot the attention maps for different kinds of token types similar to a studyreleased for the English language by Clark et al. [CKLM19].8.4.1 Experimental SetupIn our study, we use BERT-Base cased language model for a direct com-parison against the natural language, English. There are two variants ofBERT available BERT-Large and BERT-Base. BERT-Base model contains12 stacked encoders, with 12 attention heads in each layer. Moreover, wemaintain the casing of the source code, which means we do not lower casethe source code tokens. This is done so that the BERT model can learnthe Java coding conventions for the identifiers, which promotes the use ofdifferent casing styles. We could not find an open-source BERT-Base modeltrained for the Java programming language. Therefore, we trained a BERT-Base cased model from scratch to conduct this study. All these decisionsare consistent with the experimental setting of the NLP study by Clark etal. [CKLM19]. For pre-training the model, we used the benchmark dataset748.4. BERT Attention for Source Codereleased by Husain et al. [HWG+19]. This dataset contains a large corpusfor six different programming languages mined from open source repositoriesavailable on GitHub. However, we only confine our work to Java program-ming language for this study as the work in NLP is also trained on a singlenatural language. Hence, adding a multi-lingual dimension in our studywould not allow us to compare to NLP results directly. The statistics of thedifferent methods available in train, test, and valid set are shown in Table8.1.Table 8.1: Dataset details for pre-training BERTSplit # Records in the DatasetTrain 164,923Validation 5,183Test 10,955For training the BERT-Base cased model, we used the open-source imple-mentation of BERT that is available by its authors, Devlin et al. [DCLT19].Before training the language model, we remove any inline comments writ-ten in the source code files. This is done so that we can purely comparethe performance of the BERT that is only trained on source code. Also, tobring uniformity in the tokenization of the source code, we use an externaltokenizer, ’javalang’ 11 to tokenize the code tokens. Moreover, to make thedataset eligible for the next sentence prediction task, we keep the separationof sentences of source code in the different lines as mentioned on BERT’sofficial implementation repository 12. Also, a similar setting is used by Fanget al. [LLZJ20] for next sentence prediction for source code.The tokenized data that does not include natural language commentsand the mentioned pre-processing steps are then used in the BERT modelfor its pre-training. The performance of the pre-trained BERT model on thevalidation dataset for the two pre-training tasks ’Masked Language Model-ing’(MLM) and ’Next Sentence Prediction’(NSP) is shown in Table 8.2. Theresults mentioned for next sentence prediction in Table 8.2 is close to whatis reported by the authors of the BERT model (97% - 98% accuracy forNSP). The accuracy achieved by the BERT for MLM is not reported in theoriginal paper. Therefore, we are not able to make any direct comparisonwith it. However, we still think 87.44% is a decent accuracy for predictingthe masked tokens, as the task involves recalling the exact token name.11https://github.com/c2nes/javalang12https://github.com/google-research/bert758.4. BERT Attention for Source CodeTable 8.2: Accuracy of pre-trained BERT-Base on Valid datasetPre-Training Task AccuracyMasked Language Modeling 0.8744Next Sentence Prediction 0.94758.4.2 Generating BERT Attention MapsOnce the BERT model is pre-trained on the Java dataset, we use themodel’s trained weights to extract the attention distribution. For the at-tention extraction, we select samples from the test set of the dataset. Thestudy in NLP randomly selected samples from Wikipedia. Therefore, wedo not use the train data for understanding the attention mechanism. Asthe BERT model uses sentence piece tokenization of text, it can lead to thesub-tokenization of a token into multiple tokens depending on the vocabu-lary. As we are interested in different types of tokens in the source code,this sub-tokenization cannot be used. Hence, we only select samples fromthe test data for which the length before and after the tokenization remainsthe same. This allows us to study the role of identifiers better than caseswhen the sub-tokenization of code tokens is permitted. Around 2,000 sam-ples from the test set remain after the filtering. The results discussed beloware the plots that are generated for these filtered samples. The input pre-sented to the BERT model during the attention extraction is in the ’[CLS]Sentence1 [SEP] Sentence2 [SEP]’ format.In this thesis, we would study two different research questions for an-alyzing BERT’s attention behavior and comprehension abilities for sourcecode.− RQ1: What is the behavior of BERT attention over different sourcecode tokens? In this research question, we study how the BERT’sattention for different source code elements (identifier, separators, etc.)changes at different layers.− RQ2: Which layers learn the semantic information of source code? Inthis research question, we explore the semantic abilities of BERT atdifferent layers and investigate which layers can be useful for semanticrepresentation.768.5. RQ1: BERT Attention Over Different Source Code Tokens8.5 RQ1: BERT Attention Over Different SourceCode TokensThis research question would study how the attention distribution changesfor various tokens available in the source code.Before studying the attention behavior of source code constructs, we lookat the attention behavior over the special tokens introduced by the BERT’spre-processing. Two special tokens are introduced to feed input to the BERTmodel, namely ’[CLS]’ and ’SEP’ tokens. Both of these tokens have showntheir importance for different tasks in NLP. The [CLS] token’s representationis assumed to contain the complete information about the sequence, and[SEP] token acts as a delimiter between two sentences. To visualize theattention distribution for these tokens, we follow the methodology mentionedin [CKLM19]. The BERT architecture we use has 12 layers, and each layercontains 12 attention heads. Therefore, for each layer and each head inthe layer, we aggregate the attention score that the model has on [CLS]and [SEP] token put separately by all of the different tokens. Then, wenormalize the aggregated attention for [CLS] and [SEP] with their totalnumber of occurrences. The results are shown in Figure 8.3. The multiplescatter on the y axis of each layer shows the values of different attentionheads at each layer.Figure 8.3: Attention on Special TokensAs shown in Figure 8.3, both of these special tokens have very low atten-tion values. This result is contrary to what is mentioned in the NLP study[CKLM19]. In NLP, the average attention on special tokens ranges between0 to 0.6 over different layers. However, for source code, we find that the at-tention on special tokens is not dominant and ranges only between 0 - 0.10.For the [SEP] token, the average attention increases between layers 4 - 6,and then there is a consistent decline until layer 10, after which it becomesuniform for the remaining layers. Overall, the [SEP] token consistently gets778.5. RQ1: BERT Attention Over Different Source Code Tokensmore attention over [CLS] tokens over all layers. Similar behavior is seen forthe attention of [SEP] over [CLS] for different attention heads. This result isdifferent from the English language. As shown by Clark et al. in [CKLM19],initial layers until Layer-4 attend more towards [CLS] over the [SEP] token,which is not the case as seen in Figure 8.3. We see a gradual decrease for the[CLS] token until Layer-5 and then a gradual increase until Layer-9. Thisoverall behavior of source code for lower attention values could be attributedto the fact that source code is more structural and repetitive than Englishlanguage. Therefore, the same token is seen again and again which makesthe dependency over the ’CLS’ token lower, hence, resulting in lower valuesfor these special tokens.Figure 8.4: Attention on Identifier and SeparatorsSource code contains multiple syntactic constructs like identifiers, sepa-rators, and data types. These different language constructs are not availablein natural languages. Therefore, it is important to study these syntactictypes to learn about the comprehension abilities of the BERT model. Wedraw an attention distribution plot similar to the plot for special tokens,[SEP] and [CLS]. In this plot, we collect the attention distribution for eachof the syntactic type. In order to tag each token to a syntactic type, we usejavalang parser, and extract all the syntactic types of source code. Thesesyntactic types include identifiers, separators (‘,’, ‘;’, ‘{’, ‘}’, etc.), opera-tors, data-types, keywords, and access-modifiers. Note that the [SEP] tokenin Figure 8.3 is different from ‘SEPS’ in Figure 8.4. The former is a specialtoken introduced by the BERT to act as a delimiter between two sentences,and the latter is for the syntactic types (i.e., ‘,’, ‘;’, ‘{’, ‘}’, etc.) availablewithin source code. The resultant plots are shown in Figure 8.4, whichshows that identifiers and separators present in source code get the mostamount of attention. Other special tokens, including operator, datatype,keywords, and access-modifiers are not shown in this plot as their atten-tion scores are very small and therefore are shown separately in Figure 8.5.788.5. RQ1: BERT Attention Over Different Source Code TokensTherefore, both of these types of tokens are important for the comprehen-sion abilities of the BERT model. Literature on program comprehension hasacknowledged the importance of identifiers for easier program understanding[SBH+18, FMN+20, AND+21, NAD+20]. Interestingly, BERT also modelsmost of its attention over identifiers inherently. Until Layer-11, BERT at-tends most towards identifiers with attention values ranging between 35%- 55%. Unlike other works like [HLX+18b, HAMM10], Transformer modellearns identifiers representation and importance automatically without anyexternal knowledge which accounts for its better performance as a vanillaentitiy compared to previous works. At the final layer, the separators tend toget the most amount of attention, which helps the model learn the structuralinformation of source code automatically. A possible reason for such behav-ior could be that both identifiers and separators form the most frequentlyoccurring tokens (greater than 50% in the dataset used) in the source code.Also, the same token with same syntactic repeats at different positions withthe same code sample due to the structure of the source code [CSD18].Figure 8.5: Attention on Other Token TypesFigure 8.5 shows the BERT’s attention towards other syntactic types:operators, data types, key words and access modifiers. The operators, keywords and access modifiers get the lower amount of attention while modelingsource code. Data type gets the least attention, which is surprising becauseJava is a statically typed language, and knowing data types is very importantfor the correct compilation of source code. Moreover, all of these syntactictypes are related more towards the structural part of the source code. Basedon these results, using BERT to generate source code may not be ideal as itcan lead to code that throws compile-time errors.We also generate similar plots for the attention of the model towardsthe current token. It is important to know whether BERT for source codeleverages the longer or shorter scope of tokens for attention. To understandthis, we plot an attention distribution to see how much attention BERT puts798.6. RQ2: Layers That Learn the Semantic Information of Source CodeFigure 8.6: Attention on Self tokenson the token itself, which is shown in Figure 8.6. As shown in this plot, littleattention is put on the token itself except in the 11-th Layer, which tells usthat BERT is looking at other tokens to understand source code modeling.This is consistent with what we see for natural language English, and it isthe primary objective of multi-headed attention to learn the vector spacesbetween different tokens.The final plot is shown in Figure 8.7, where we see the difference betweenthe information that multiple heads learn in each layer. To understand thisinformation, the Jensen-Shannon divergence between different heads avail-able in the BERT model is calculated. The more the distance betweentwo heads, the more disparate information is learned by them. In total,BERT contains 144 attention heads. Therefore, we have a 144*144 matrixto represent such information, as shown in Figure 8.7. We see that simi-lar to NLP, redundancy in the attention distribution insinuates that not allattention heads learn unique information. Therefore there is scope for ap-plying model distillation strategies that can help reduce the model size andmaintain equivalent performance. The middle layers retain diverse informa-tion when compared to the lower and higher layers. Contrary to NLP, theJensen-Shannon values are tighter, possibly due to the repetition of sourcecode tokens and code syntax.8.6 RQ2: Layers That Learn the SemanticInformation of Source CodeIn this section we run a ‘probing classifier’ to know which layers learnsemantic information of source code. For this probing, we choose the taskof code clone detection.808.6. RQ2: Layers That Learn the Semantic Information of Source CodeFigure 8.7: Jenson-Shannon Divergence Between Different Attention Heads818.6. RQ2: Layers That Learn the Semantic Information of Source Code8.6.1 Code Clone DetectionFigure 8.8: Code Clone with CLS tokenCode clones are the type of code samples that are identical to each other[BYM+98, Kri01]. These code samples may have structural or semanticequivalences [SIK+14]. Generally, the task of code clone detection requiresfinding identical code samples based on the different aspects of source code[SIK+14]. If the code samples have the same functionality, they are taggedas semantic clones, and otherwise, they are classified as non-clones of eachother. For the code clone detection, we use the pipeline made available byLu et al. [LGR+21], with few changes. The original pipeline was built forROBERTa, and it was changed to accommodate the BERT model. Thepipeline uses the BERT model to initialize the encoder, and then a linearhead is used over the pre-trained model for the classification into clones andnonclones. The classification model is trained end to end, so, similar toworks in NLP [TXC+19, TDP19], we freeze the weights of the BERT modelbecause we want to learn about the model’s capabilities for the target taskwithout further training. Therefore, the weights of the encoder are not fine-tuned with the task, and we can only extract the semantic capabilities ofthe pre-trained model. The linear head over the BERT encoder is still fine-828.7. Implications of Findings from RQ1tuned for the task of code clone detection. The linear head is not originallypart of the BERT model and is only introduced as a projection for therepresentation of BERT. The linear head uses the hidden representationfrom BERT to detect the code clones. The fine-tuning is done for threeonly epochs as it was done in the original BERT paper [DCLT19]. For thefine-tuning, we use the Big-Clone-Bench dataset [SIK+14]. The statistics ofthe dataset are shown in Table 8.3.Table 8.3: Dataset details for Code Clone DetectionSplit # Records in the DatasetTrain 901,028Validation 415,416Test 415,416To understand the semantic comprehension abilities of each layer, weextract the output hidden representation of [CLS] token starting from Layer-1 until Layer-12. The extracted hidden state is then used by the linear headover the BERT encoder. The fine-tuning results are shown in Figure 8.8.The code clone detection is a classification task; therefore, its performanceis evaluated using F1 score, precision, and recall. As seen in the figure,the lower layers have worse performance than the upper-level layers. Thismeans that upper layers are better at representing the semantic relationshipof the source code. Therefore, semantic tasks should use upper-level layersto represent source code to achieve the best performance. For the middlelayers (4-10), we see the performance remains uniform.8.7 Implications of Findings from RQ1Based on the attention distribution of the BERT model trained on JAVAsource code, we find that most of the attention is on the identifiers comparedto other syntactic types. Therefore, this insinuates that identifiers are im-portant entities and should encapsulate better token representation for thedownstream tasks than the CLS tokens. Accordingly, we experimented withusing the embedding generated from the identifiers rather than the CLStoken to confirm this finding. To incorporate such representation, we do aweighted sum of all the identifiers present within the code. Each identifierembedding is normalized with its attention value and then added to formthe weighted attention embedding of the identifiers. Finally, we extract theweighted embedding of the identifier from each layer and use it with theclassification head.838.7. Implications of Findings from RQ1Table 8.4: Using Identifier embedding for code clone detectionLayer Precision Recall F1Layer1 0.8136 (1090%) 0.9013 (80%) 0.8483 (605%)Layer2 0.8243 (49%) 0.9150 (62%) 0.8603 (185%)Layer3 0.8264 (33%) 0.9173 (28%) 0.8625 (38%)Layer4 0.8333 (34%) 0.9170 (25%) 0.8674 (40%)Layer5 0.8331 (32%) 0.9211 (24%) 0.8686 (37%)Layer6 0.8307 (30%) 0.9211 (22%) 0.8668 (23%)Layer7 0.8259 (29%) 0.9201 (23%) 0.8629 (33%)Layer8 0.8266 (31%) 0.9198 (24%) 0.8634 (35%)Layer9 0.8243 (31%) 0.9182 (24%) 0.8613 (35%)Layer10 0.8293 (30%) 0.9182 (22%) 0.8649 (34%)Layer11 0.8207 (15%) 0.9201 (11%) 0.8591 (15%)Layer12 0.8365 (11%) 0.9240 (7%) 0.8719 (10%)As seen in Table 8.4, we see an increase in performance for each metricat each layer. However, the increase in performance is prominent in theinitial layers. We see a performance improvement of 1090%, 80% and 605%respectively for precision, recall, and f1 score, whereas this performanceincrease reduced to 11%, 7%, and 10% respectively for the twelfth layer.As seen above, the attention put on CLS token is very low comparedto other NLP works. To further extend the hypothesis generated out ofthis study, we perform the same experiments on the Roberta-based Code-BERT model. The Roberta-based CodeBERT model is trained on six differ-ent programming languages on the Codesearchnet dataset [HWG+19] andis pre-trained on only masked language modeling objective. It uses theRoberta-base [LOG+19] model architecture which is a 12 layer transformerbased encoder architecture. Moreover, this model pre-trains both on nat-ural language and as well as code [FGT+20] The results are shown in Fig.8.9. In this case, we see that the attention put on CLS equivalent token< s > is higher than the BERT model trained in this work. As there is no[SEP] token for Roberta-based models, it is not shown in the figure. Despitesuch behavior, we experimented with analyzing the role of identifiers in thisscenario.Therefore, a similar experiment is also conducted on variations of opensource model CodeBERT [FGT+20]. We use the CodeBERT model availableon the HuggingFace model’s website and use the same code made availableby the authors. There is an extra pre-processing step that we introduce be-848.7. Implications of Findings from RQ1Figure 8.9: Attention on Special Tokensfore using the CodeBERT model. Similar to the pre-processing of the BERTmodel trained in this work, we first tokenize the code using the javalang to-kenizer. Moreover, within the CodeBERT tokenizer, each token has ”G” asa suffix, which is removed (although this step did not change to tokeniza-tion of CodeBERT) to identify the syntactic types. The tokenized code isused inside code clone detection. This tokenization is necessary to extractand separate the tokens into their respective syntactic types. We repeatthis experiment only for the pooled layer of CodeBERT as it was the bestperforming model. The results for this experiment is shown in Table 8.5,which shows 14.61%, 13.08% and 21.05% improvement for precision, recall,and f1-score. Similar to the previous result, using identifiers in source coderepresentation is useful. Even for the Roberta-based CodeBERT model.This result is unusual because although the attention on < s > tokenis high, the scores achieved by the identifier embedding are better than theCLS equivalent token.Table 8.5: Using Identifier embedding in Codebert for code clone detectionLayer Precision Recall F1Codebert-Layer-Pool(IDF) 0.7263 (14.61%) 0.8631 (13.08%) 0.7619 (21.05%)Codebert-Layer-Pool(CLS) 0.6337 0.7632 0.6294Codebert-MLM-Layer-Pool(IDF) 0.6887 (35.01%) 0.9319 (11.35%) 0.7921 (24.95%)Codebert-MLM-Layer-Pool(CLS) 0.5101 0.8369 0.6339Summary of Results. The experiments from our attention analysisof the BERT model for source code demonstrate similarities and dissimi-larities with the studies done in NLP. The major difference was found tobe in the attention distribution of special tokens, namely [CLS] and [SEP]token of the BERT, which gets very low attention compared to the worksin NLP. Also, BERT attends more on identifiers and separators when com-pared to other token types of source code. A possible reasoning could be basedon the fact that both the identifiers and separators are the most frequently858.7. Implications of Findings from RQ1occurring tokens in source code.. This result is similar to what we see inother program comprehension studies. Moreover, similar to NLP, differentattention heads within BERT tend to learn identical information; however,the difference is scarce in source code. We also see that BERT attends tomodel the source code more towards other tokens than the tokens themselves.Overall Transformer based models are able to learn the semantic and syn-tactic information better than existing techniques without using any externalknowledge, which accounts for the performance gain of the model. Finally,based on our findings, we use the identifiers embedding inside the classifica-tion head for code clone detection, which improves the performance of CLStoken representation over all the layers.86Chapter 9Conclusion and Future WorkThis thesis presents a novel model called LAMNER-Code for code com-ment generation. The model can be combined with other techniques, suchas using AST in the model. For the first time, we introduce the use ofthe character-level language model and NER model to encode the seman-tic and syntactical structure of code. In both models, the context of thecharacters is captured using their left and right information. The evalu-ation on BLEU, ROUGE-L, METEOR, and CIDEr metrics confirms ourapproach outperforms other recent models. We relate this result to thepre-trained embeddings we used and their ability to extract the semanticand syntactical representation of unseen code sequences. This result is alsosupported through human evaluation. Several experiments have shown theimportance of both embeddings for comment generation and their robust-ness when tested on abstracted code. The comments of LAMNER-Codecontain identifiers and include relevant tokens even if they are not usedin the given code directly. The new decoding strategy proposed here fur-ther improves the recall of the identifiers. LAMNER-Code comments alsopreserve the fluency and grammatical correctness of the English text.Moreover, we present the first attention analysis study for BERT onsource code. Findings from this study reveal similarities and dissimilari-ties with the natural language. Our findings reveals, identifiers embeddingserves as a better representaion than the ’CLS’ token when used withinthe classification head for code clone detection. Therefore, this study canserve as fundamental work for understanding nuances of pre-trained Trans-former based language models for source code. Future work is to applyLAMNER-Code on other programming languages and different tasks suchas bug prediction. Another direction is expanding the techniques used forLAMNER-Code-DYN and conducting research to build open vocabularymodels that can solve the problem of out of vocabulary words in code com-ment generation and reduce the model size. However, model size can changedepending on the size of vocabulary and embeddings. The embedding ma-trix in the NMT model is a N*M matrix, where N is vocabulary size, and Mis the embedding size. Open vocabulary can remove these learnable param-87Chapter 9. Conclusion and Future Worketers of the embedding matrix of the models. In LAMNER-Code, we saw46% of the model size is occupied by this embedding matrix, which can bereduced with open vocabulary model. In addition, the exploratory study onBERT can be extended to investigate syntactical capabilities of the PTM’s.Also, from this study, we hope to provide explainability for the success ofthe PTM’s. Moreover, as future work, it would be interesting to see how theattention behavior changes with the change of different pre-training tasksusing models like ROBERTa.88Bibliography[AA19] Afshine Amidi and Shervine Amidi. Recurrent neural networkscheatsheet star, Jan 2019. → pages 15[ABA+19] Apostolos Ampatzoglou, Stamatia Bibi, Paris Avgeriou, Mar-ijn Verbeek, and Alexander Chatzigeorgiou. Identifying, cate-gorizing and mitigating threats to validity in software engineer-ing secondary studies. Information and Software Technology,106:201–230, 2019. → pages 68[ABB+19] Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul,Stefan Schweter, and Roland Vollgraf. FLAIR: An easy-to-useframework for state-of-the-art NLP. In Proceedings of the 2019Conference of the North American Chapter of the Associationfor Computational Linguistics (Demonstrations), pages 54–59,Minneapolis, Minnesota, June 2019. Association for Computa-tional Linguistics. → pages 33, 34[ABBS14] Miltiadis Allamanis, Earl T Barr, Christian Bird, and CharlesSutton. Learning natural coding conventions. In Proceedingsof the 22nd ACM SIGSOFT International Symposium onFoundations of Software Engineering, pages 281–293, 2014. →pages 1[ABBS15] Miltiadis Allamanis, Earl T. Barr, Christian Bird, andCharles Sutton. Suggesting accurate method and class names.ESEC/FSE 2015, page 38–49, New York, NY, USA, 2015. As-sociation for Computing Machinery. → pages 12[ABDS18] Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, andCharles Sutton. A survey of machine learning for big code andnaturalness. ACM Computing Surveys (CSUR), 51(4):1–37,2018. → pages 5[ABK17] Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud89BibliographyKhademi. Learning to represent programs with graphs. CoRR,abs/1711.00740, 2017. → pages 13[ABLY19] Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav.code2seq: Generating sequences from structured representa-tions of code. In 7th International Conference on LearningRepresentations, ICLR 2019, New Orleans, LA, USA, May 6-9,2019. OpenReview.net, 2019. → pages 2, 8, 27, 32, 38, 47, 53,68[ABV18] Alan Akbik, Duncan Blythe, and Roland Vollgraf. Contextualstring embeddings for sequence labeling. In Proceedings of the27th International Conference on Computational Linguistics,pages 1638–1649, Santa Fe, New Mexico, USA, August 2018.Association for Computational Linguistics. → pages 26, 28, 49[ACRC20] Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, andKai-Wei Chang. A transformer-based approach for source codesummarization. In ACL, pages 4998–5007, 2020. → pages 4,10, 57, 69[ACRC21] Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, andKai-Wei Chang. Unified pre-training for program understand-ing and generation. CoRR, abs/2103.06333, 2021. → pages 10,69[AJFB21] Miltiadis Allamanis, Henry Jackson-Flux, and MarcBrockschmidt. Self-supervised bug detection and repair.arXiv preprint arXiv:2105.12787, 2021. → pages 1[AND+21] Reem S. Alsuhaibani, Christian D. Newman, Michael J. Decker,Michael L. Collard, and Jonathan I. Maletic. On the nam-ing of methods: A survey of professional developers. In2021 IEEE/ACM 43rd International Conference on SoftwareEngineering (ICSE), pages 587–599, 2021. → pages 79[AZLY19] Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav.Code2vec: Learning distributed representations of code. Proc.ACM Program. Lang., 3(POPL), January 2019. → pages 13,27[BCB15] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.Neural machine translation by jointly learning to align and90Bibliographytranslate. In Yoshua Bengio and Yann LeCun, editors, 3rdInternational Conference on Learning Representations, ICLR2015, San Diego, CA, USA, May 7-9, 2015, Conference TrackProceedings, 2015. → pages 15, 20, 21, 30, 31[BDVJ03] Yoshua Bengio, Re´jean Ducharme, Pascal Vincent, and Chris-tian Janvin. A neural probabilistic language model. The journalof machine learning research, 3:1137–1155, 2003. → pages 18[BGJM17] Piotr Bojanowski, Edouard Grave, Armand Joulin, and TomasMikolov. Enriching word vectors with subword information.Transactions of the Association for Computational Linguistics,5:135–146, 2017. → pages 11, 12, 20, 28, 48[BHM21] Aakash Bansal, Sakib Haque, and Collin McMillan. Project-level encoding for neural source code summarization of subrou-tines. CoRR, abs/2103.11599, 2021. → pages 5, 11[BL05] Satanjeev Banerjee and Alon Lavie. METEOR: An auto-matic metric for MT evaluation with improved correlationwith human judgments. In Proceedings of the ACL Workshopon Intrinsic and Extrinsic Evaluation Measures for MachineTranslation and/or Summarization, pages 65–72, Ann Arbor,Michigan, June 2005. Association for Computational Linguis-tics. → pages 40, 41, 65[BNJ03] David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latentdirichlet allocation. J. Mach. Learn. Res., 3(null):993–1022,March 2003. → pages 6[BPB+20] Luca Buratti, Saurabh Pujar, Mihaela Bornea, Scott McCar-ley, Yunhui Zheng, Gaetano Rossiello, Alessandro Morari, JimLaredo, Veronika Thost, Yufan Zhuang, and Giacomo Domeni-coni. Exploring Software Naturalness through Neural LanguageModels. arXiv e-prints, page arXiv:2006.12641, June 2020. →pages 33[BR00] Keith H. Bennett and Va´clav T. Rajlich. Software maintenanceand evolution: A roadmap. In Proceedings of the Conferenceon The Future of Software Engineering, ICSE ’00, page 73–87,New York, NY, USA, 2000. Association for Computing Ma-chinery. → pages 191Bibliography[BSM04] Pankaj Bhatt, Gautam Shroff, and Arun K. Misra. Dynamics ofsoftware maintenance. SIGSOFT Softw. Eng. Notes, 29(5):1–5,September 2004. → pages 1[BYM+98] I.D. Baxter, A. Yahin, L. Moura, M. Sant’Anna, and L. Bier.Clone detection using abstract syntax trees. In Proceedings.International Conference on Software Maintenance (Cat. No.98CB36272), pages 368–377, 1998. → pages 82[CCS19] Elizabeth Clark, Asli Celikyilmaz, and Noah A. Smith. Sen-tence mover’s similarity: Automatic evaluation for multi-sentence texts. In Proceedings of the 57th Annual Meetingof the Association for Computational Linguistics, pages 2748–2760, Florence, Italy, July 2019. Association for ComputationalLinguistics. → pages 46[CDM13] Michael Collard, Michael Decker, and Jonathan Maletic. srcml:An infrastructure for the exploration, analysis, and manipula-tion of source code: A tool demonstration. pages 516–519, 092013. → pages 22, 23[CGCB14] Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, andYoshua Bengio. Empirical evaluation of gated recurrent neuralnetworks on sequence modeling. In NIPS 2014 Workshop onDeep Learning, December 2014, 2014. → pages 29[CKLM19] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christo-pher D. Manning. What does BERT look at? an analysis ofbert’s attention. CoRR, abs/1906.04341, 2019. → pages 74,77, 78[CM18] Zimin Chen and Martin Monperrus. The remarkable roleof similarity in redundancy-based program repair. CoRR,abs/1811.05703, 2018. → pages 1, 12[CSD18] Casey Casalnuovo, Kenji Sagae, and Premkumar T. Devanbu.Studying the difference between natural and programming lan-guage corpora. CoRR, abs/1806.02437, 2018. → pages 1, 2,14, 69, 79[CTJ+21] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Hen-rique Ponde, Jared Kaplan, Harri Edwards, Yura Burda,Nicholas Joseph, Greg Brockman, et al. Evaluating92Bibliographylarge language models trained on code. arXiv preprintarXiv:2107.03374, 2021. → pages 1[CvMBB14] KyungHyun Cho, Bart van Merrienboer, Dzmitry Bahdanau,and Yoshua Bengio. On the properties of neural machine trans-lation: Encoder-decoder approaches. CoRR, abs/1409.1259,2014. → pages 17, 30[CvMG+14] Kyunghyun Cho, Bart van Merrie¨nboer, Caglar Gulcehre,Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, andYoshua Bengio. Learning phrase representations using RNNencoder–decoder for statistical machine translation. InProceedings of the 2014 Conference on Empirical Methodsin Natural Language Processing (EMNLP), pages 1724–1734,Doha, Qatar, October 2014. Association for ComputationalLinguistics. → pages 15[CXH+21] Qiuyuan Chen, Xin Xia, Han Hu, David Lo, and ShanpingLi. Why my code summarization model does not work: Codecomment improvement with category prediction. ACM Trans.Softw. Eng. Methodol., 30(2), February 2021. → pages 40[CYF21] Roee Cates, Nadav Yunik, and Dror G Feitelson. Does codestructure affect comprehension? on using and naming inter-mediate variables. arXiv preprint arXiv:2103.11008, 2021. →pages 1[DCLT19] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and KristinaToutanova. BERT: Pre-training of deep bidirectional trans-formers for language understanding. In Proceedings of the 2019Conference of the North American Chapter of the Associationfor Computational Linguistics: Human Language Technologies,Volume 1 (Long and Short Papers), pages 4171–4186, Min-neapolis, Minnesota, June 2019. Association for ComputationalLinguistics. → pages 10, 12, 69, 70, 75, 83[DDF+90] Scott Deerwester, Susan T Dumais, George W Furnas,Thomas K Landauer, and Richard Harshman. Indexing bylatent semantic analysis. Journal of the American society forinformation science, 41(6):391–407, 1990. → pages 593Bibliography[DTR18] Daniel DeFreez, Aditya V. Thakur, and Cindy Rubio-Gonza´lez.Path-based function embedding and its application to specifi-cation mining. CoRR, abs/1802.07779, 2018. → pages 13[DUSK17] Jacob Devlin, Jonathan Uesato, Rishabh Singh, and PushmeetKohli. Semantic code repair using neuro-symbolic transforma-tion networks. CoRR, abs/1710.11054, 2017. → pages 1, 13[EHT16] Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka.Tree-to-sequence attentional neural machine translation. InProceedings of the 54th Annual Meeting of the Associationfor Computational Linguistics (Volume 1: Long Papers), pages823–833, Berlin, Germany, August 2016. Association for Com-putational Linguistics. → pages 27[ERKC13] Brian P. Eddy, Jeffrey A. Robinson, Nicholas A. Kraft, andJeffrey C. Carver. Evaluating source code summarization tech-niques: Replication and expansion. In 2013 21st InternationalConference on Program Comprehension (ICPC), pages 13–22,2013. → pages 6[ES19] Vasiliki Efstathiou and Diomidis Spinellis. Semantic sourcecode models using identifier embeddings. In Proceedingsof the 16th International Conference on Mining SoftwareRepositories, MSR ’19, page 29–33. IEEE Press, 2019. → pages12[FGT+20] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, XiaochengFeng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, DaxinJiang, and Ming Zhou. CodeBERT: A pre-trained modelfor programming and natural languages. In Findings of theAssociation for Computational Linguistics: EMNLP 2020,pages 1536–1547, Online, November 2020. Association for Com-putational Linguistics. → pages 4, 10, 26, 32, 68, 69, 84[FMN+20] Dror Feitelson, Ayelet Mizrahi, Nofar Noy, Aviad Ben Shabat,Or Eliyahu, and Roy Sheffer. How developers choose names.IEEE Transactions on Software Engineering, pages 1–1, 2020.→ pages 79[GGH+21] Shuzheng Gao, Cuiyun Gao, Yulan He, Jichuan Zeng, Lun YiuNie, and Xin Xia. Code structure guided transformer for source94Bibliographycode summarization. CoRR, abs/2104.09340, 2021. → pages2, 4, 10, 69[Gra13] Alex Graves. Generating Sequences With Recurrent NeuralNetworks. arXiv e-prints, page arXiv:1308.0850, August 2013.→ pages 26[GRL+21] Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang,Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy,Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement,Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, andMing Zhou. Graphcodebert: Pre-training code representationswith data flow, 2021. → pages 4, 22, 57, 69[HAM10] Sonia Haiduc, Jairo Aponte, and Andrian Marcus. Supportingprogram comprehension with source code summarization. InProceedings of the 32nd ACM/IEEE International Conferenceon Software Engineering - Volume 2, ICSE ’10, page 223–226,New York, NY, USA, 2010. Association for Computing Ma-chinery. → pages 6[HAMM10] Sonia Haiduc, Jairo Aponte, Laura Moreno, and Andrian Mar-cus. On the use of automated text summarization techniquesfor summarizing source code. WCRE ’10, page 35–44, USA,2010. IEEE Computer Society. → pages 1, 5, 6, 79[HBWM21] Sakib Haque, Aakash Bansal, Lingfei Wu, and Collin McMillan.Action word prediction for neural source code summarization.arXiv preprint arXiv:2101.02742, 2021. → pages 14, 31, 66[HLWM20] Sakib Haque, Alexander LeClair, Lingfei Wu, and CollinMcMillan. Improved automatic summarization of subroutinesvia attention to file context. In Proceedings of the 17thInternational Conference on Mining Software Repositories,MSR ’20, page 300–310, New York, NY, USA, 2020. Associ-ation for Computing Machinery. → pages 11, 38[HLX+18a] Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. Deep codecomment generation. In Proceedings of the 26th Conferenceon Program Comprehension, ICPC ’18, page 200–210, NewYork, NY, USA, 2018. Association for Computing Machinery.→ pages 1, 2, 7, 13, 22, 23, 25, 2795Bibliography[HLX+18b] Xing Hu, Ge Li, Xin Xia, David Lo, Shuai Lu, and ZhiJin. Summarizing source code with transferred api knowl-edge. In Proceedings of the Twenty-Seventh International JointConference on Artificial Intelligence, IJCAI-18, pages 2269–2275. International Joint Conferences on Artificial IntelligenceOrganization, 7 2018. → pages 2, 8, 13, 22, 25, 39, 43, 62, 79[HLX+20] Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. Deep codecomment generation with hybrid lexical and syntactical in-formation. Empirical Software Engineering, 25:2179–2217, 052020. → pages 1, 2, 8, 38, 56, 60, 61, 62, 65, 68[HS97] Sepp Hochreiter and Ju¨rgen Schmidhuber. Long short-termmemory. Neural Comput., 9(8):1735–1780, November 1997. →pages 16[HSX+18] Qiao Huang, Emad Shihab, Xin Xia, David Lo, and ShanpingLi. Identifying self-admitted technical debt in open sourceprojects using text mining. Empirical Software Engineering,23(1):418–451, 2018. → pages 60[HWG+19] Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allama-nis, and Marc Brockschmidt. CodeSearchNet Challenge: Eval-uating the State of Semantic Code Search. arXiv e-prints, pagearXiv:1909.09436, September 2019. → pages 13, 32, 44, 75, 84[HXY15] Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional LSTM-CRF models for sequence tagging. CoRR, abs/1508.01991,2015. → pages 26, 28[IKCZ16] Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and LukeZettlemoyer. Summarizing source code using a neural atten-tion model. In Proceedings of the 54th Annual Meeting of theAssociation for Computational Linguistics (Volume 1: LongPapers), pages 2073–2083, Berlin, Germany, August 2016. As-sociation for Computational Linguistics. → pages 7, 14, 62, 65,66[KBL19] H. J. Kang, T. F. Bissyande´, and D. Lo. Assessing the generaliz-ability of code2vec token embeddings. In 2019 34th IEEE/ACMInternational Conference on Automated Software Engineering(ASE), pages 1–12, 2019. → pages 2596Bibliography[KBR+20] Rafael-Michael Karampatsis, Hlib Babii, Romain Robbes,Charles Sutton, and Andrea Janes. Big code != big vocab-ulary: Open-vocabulary models for source code. ICSE ’20,page 1073–1085, New York, NY, USA, 2020. Association forComputing Machinery. → pages 33[KJSR16] Yoon Kim, Yacine Jernite, David Sontag, and Alexander M.Rush. Character-aware neural language models. In Proceedingsof the Thirtieth AAAI Conference on Artificial Intelligence,AAAI’16, page 2741–2749. AAAI Press, 2016. → pages 26[KKD+17] Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senel-lart, and Alexander M. Rush. OpenNMT: Open-SourceToolkit for Neural Machine Translation. arXiv e-prints, pagearXiv:1701.02810, January 2017. → pages 39[KMBS20] Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, andKensen Shi. Learning and evaluating contextual embeddingof source code. In Hal Daume´ III and Aarti Singh, editors,Proceedings of the 37th International Conference on MachineLearning, volume 119 of Proceedings of Machine LearningResearch, pages 5110–5121, Virtual, 13–18 Jul 2020. PMLR.→ pages 12, 26, 69[Kri01] J. Krinke. Identifying similar code with program dependencegraphs. In Proceedings Eighth Working Conference on ReverseEngineering, pages 301–309, 2001. → pages 82[KS20] Rafael-Michael Karampatsis and Charles Sutton. SCELMo:Source Code Embeddings from Language Models. arXive-prints, page arXiv:2004.13214, April 2020. → pages 12, 69[LCX+21] Shangqing Liu, Yu Chen, Xiaofei Xie, Jing Kai Siow, and YangLiu. Retrieval-augmented generation for code summarizationvia hybrid GNN. In International Conference on LearningRepresentations, 2021. → pages 2, 10, 38, 41, 66[LGR+21] Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svy-atkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain,Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou,Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, NanDuan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and97BibliographyShujie Liu. Codexglue: A machine learning benchmark datasetfor code understanding and generation. CoRR, abs/2102.04664,2021. → pages 1, 13, 82[LHWM20] Alexander LeClair, Sakib Haque, Lingfei Wu, and CollinMcMillan. Improved code summarization via a graph neuralnetwork. CoRR, abs/2004.02843, 2020. → pages 1, 2, 10, 20,25[LHX+20] Jiakun Liu, Qiao Huang, Xin Xia, Emad Shihab, David Lo,and Shanping Li. Is using deep learning frameworks free?characterizing technical debt in deep learning frameworks. InProceedings of the ACM/IEEE 42nd International Conferenceon Software Engineering: Software Engineering in Society,pages 1–10, 2020. → pages 60[Lin04] Chin-Yew Lin. ROUGE: A package for automatic evaluation ofsummaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for ComputationalLinguistics. → pages 40, 65[LJM19] Alexander LeClair, Siyuan Jiang, and Collin McMillan. A neu-ral model for generating natural language summaries of pro-gram subroutines. ICSE ’19, page 795–806. IEEE Press, 2019.→ pages 2, 8, 22, 39, 47, 53, 56, 57[LLZJ20] Fang Liu, Ge Li, Yunfei Zhao, and Zhi Jin. Multi-task learn-ing based pre-trained language model for code completion. In2020 35th IEEE/ACM International Conference on AutomatedSoftware Engineering (ASE), pages 473–485, 2020. → pages 33,67, 75[LM19] Alexander LeClair and Collin McMillan. Recommendationsfor datasets for source code summarization. In Proceedingsof the 2019 Conference of the North American Chapter of theAssociation for Computational Linguistics: Human LanguageTechnologies, Volume 1 (Long and Short Papers), pages 3931–3937, Minneapolis, Minnesota, June 2019. Association for Com-putational Linguistics. → pages 13[LOG+19] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, MandarJoshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer,98Bibliographyand Veselin Stoyanov. Roberta: A robustly optimized bert pre-training approach. arXiv preprint arXiv:1907.11692, 2019. →pages 84[LZ18] Yuding Liang and Kenny Q. Zhu. Automatic generation of textdescriptive comments for code blocks. CoRR, abs/1808.06880,2018. → pages 2, 8, 38[MAC13] Dana Movshovitz-Attias and William W. Cohen. Naturallanguage models for predicting programming comments. InProceedings of the 51st Annual Meeting of the Association forComputational Linguistics (Volume 2: Short Papers), pages35–40, Sofia, Bulgaria, August 2013. Association for Computa-tional Linguistics. → pages 6[McH12] Mary L McHugh. Interrater reliability: the kappa statistic.Biochemia medica: Biochemia medica, 22(3):276–282, 2012. →pages 60[MCH+20] Christopher D Manning, Kevin Clark, John Hewitt, Ur-vashi Khandelwal, and Omer Levy. Emergent linguis-tic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences,117(48):30046–30054, 2020. → pages 70[MFA+21] Junayed Mahmud, Fahim Faisal, Raihan Islam Arnob, Anto-nios Anastasopoulos, and Kevin Moran. Code to commenttranslation: A comparative study on model effectiveness er-rors, 2021. → pages 32[MH16] Xuezhe Ma and Eduard Hovy. End-to-end sequence labelingvia bi-directional LSTM-CNNs-CRF. In Proceedings of the54th Annual Meeting of the Association for ComputationalLinguistics (Volume 1: Long Papers), pages 1064–1074, Berlin,Germany, August 2016. Association for Computational Lin-guistics. → pages 20[MSC+13] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, andJeff Dean. Distributed representations of words and phrasesand their compositionality. Advances in neural informationprocessing systems, 26:3111–3119, 2013. → pages 1199Bibliography[NAD+20] Christian D. Newman, Reem S. AlSuhaibani, Michael J.Decker, Anthony Peruma, Dishant Kaushik, Mohamed WiemMkaouer, and Emily Hill. On the generation, structure, and se-mantics of grammar patterns in source code identifiers. Journalof Systems and Software, 170:110740, 2020. → pages 79[NNN16] Trong Duc Nguyen, Anh Tuan Nguyen, and Tien N. Nguyen.Mapping api elements for code migration with vector represen-tations. In Proceedings of the 38th International Conferenceon Software Engineering Companion, ICSE ’16, page 756–758,New York, NY, USA, 2016. Association for Computing Ma-chinery. → pages 11[NPLN20] Son Nguyen, Hung Phan, Trinh Le, and Tien N. Nguyen. Sug-gesting natural method names to check name consistencies. InProceedings of the ACM/IEEE 42nd International Conferenceon Software Engineering, ICSE ’20, page 1372–1384, New York,NY, USA, 2020. Association for Computing Machinery. →pages 64[NSR18] Hatem M Noaman, Shahenda S Sarhan, and Mohsen AARashwan. Enhancing recurrent neural network-based languagemodels by word tokenization. Human-centric Computing andInformation Sciences, 8(1):1–13, 2018. → pages 19[PADP+12] Sebastiano Panichella, Jairo Aponte, Massimiliano Di Penta,Andrian Marcus, and Gerardo Canfora. Mining source code de-scriptions from developer communications. In 2012 20th IEEEInternational Conference on Program Comprehension (ICPC),pages 63–72, 2012. → pages 6[PMB13] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio.On the difficulty of training recurrent neural networks.In Proceedings of the 30th International Conference onInternational Conference on Machine Learning - Volume 28,ICML’13, page III–1310–III–1318. JMLR.org, 2013. → pages15, 16[PNI+18] Matthew E. Peters, Mark Neumann, Mohit Iyyer, MattGardner, Christopher Clark, Kenton Lee, and Luke Zettle-moyer. Deep contextualized word representations. CoRR,abs/1802.05365, 2018. → pages 12, 48100Bibliography[PRL+19] Fabio Petroni, Tim Rockta¨schel, Patrick Lewis, Anton Bakhtin,Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. Lan-guage models as knowledge bases?, 2019. → pages 69[PRWZ02] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-JingZhu. Bleu: a method for automatic evaluation of machinetranslation. In Proceedings of the 40th Annual Meeting ofthe Association for Computational Linguistics, pages 311–318,Philadelphia, Pennsylvania, USA, July 2002. Association forComputational Linguistics. → pages 40, 65[PS18] Michael Pradel and Koushik Sen. Deepbugs: A learning ap-proach to name-based bug detection. Proc. ACM Program.Lang., 2(OOPSLA), October 2018. → pages 1[PSM14] Jeffrey Pennington, Richard Socher, and Christopher Man-ning. GloVe: Global vectors for word representation. InProceedings of the 2014 Conference on Empirical Methodsin Natural Language Processing (EMNLP), pages 1532–1543,Doha, Qatar, October 2014. Association for ComputationalLinguistics. → pages 11, 28[QSF+18] Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padman-abhan, and Graham Neubig. When and why are pre-trainedword embeddings useful for neural machine translation? InProceedings of the 2018 Conference of the North AmericanChapter of the Association for Computational Linguistics:Human Language Technologies, Volume 2 (Short Papers),pages 529–535, New Orleans, Louisiana, June 2018. Associa-tion for Computational Linguistics. → pages 3, 23[RG65] Herbert Rubenstein and John B. Goodenough. Contextual cor-relates of synonymy. Commun. ACM, 8(10):627–633, October1965. → pages 70[SASZ17] Shikhar Sharma, Layla E. Asri, Hannes Schulz, and JeremieZumer. Relevance of Unsupervised Metrics in Task-OrientedDialogue for Evaluating Natural Language Generation. arXive-prints, page arXiv:1706.09799, June 2017. → pages 40[SBH+18] Andrea Schankin, Annika Berger, Daniel V. Holt, Johannes C.Hofmeister, Till Riedel, and Michael Beigl. Descriptive com-pound identifier names improve source code comprehension. In101Bibliography2018 IEEE/ACM 26th International Conference on ProgramComprehension (ICPC), pages 31–3109, 2018. → pages 1, 79[SDP20] Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT:Learning robust metrics for text generation. In Proceedings ofthe 58th Annual Meeting of the Association for ComputationalLinguistics, pages 7881–7892, Online, July 2020. Associationfor Computational Linguistics. → pages 65, 67[SGL+20] Sean Stapleton, Yashmeet Gambhir, Alexander LeClair,Zachary Eberhart, Westley Weimer, Kevin Leach, andYu Huang. A human study of comprehension and code summa-rization. In Proceedings of the 28th International Conferenceon Program Comprehension, ICPC ’20, page 2–13, New York,NY, USA, 2020. Association for Computing Machinery. →pages 65[She20] Alex Sherstinsky. Fundamentals of recurrent neural network(rnn) and long short-term memory (lstm) network. Physica D:Nonlinear Phenomena, 404:132306, 2020. → pages 29[SHK+14] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, IlyaSutskever, and Ruslan Salakhutdinov. Dropout: A simple wayto prevent neural networks from overfitting. Journal of MachineLearning Research, 15(56):1929–1958, 2014. → pages 29[SIK+14] Jeffrey Svajlenko, Judith F. Islam, Iman Keivanloo, Chan-chal K. Roy, and Mohammad Mamun Mia. Towards a bigdata curated benchmark of inter-project code clones. In 2014IEEE International Conference on Software Maintenance andEvolution, pages 476–480, 2014. → pages 82, 83[SJH+21] Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Ad-ina Williams, and Douwe Kiela. Masked language modeling andthe distributional hypothesis: Order word matters pre-trainingfor little. CoRR, abs/2104.06644, 2021. → pages 70[SMH11] Ilya Sutskever, James Martens, and Geoffrey Hinton. Gen-erating text with recurrent neural networks. ICML’11, page1017–1024, Madison, WI, USA, 2011. Omnipress. → pages 25102Bibliography[SRTR17] Giuseppe Scanniello, Michele Risi, Porfirio Tramontana, andSimone Romano. Fixing faults in c and java source code: Ab-breviated vs. full-word identifier names. ACM Transactions onSoftware Engineering and Methodology (TOSEM), 26(2):1–43,2017. → pages 1[SSWY19] Xiaotao Song, Hailong Sun, Xu Wang, and Jiafei Yan. A surveyof automatic generation of source code comments: Algorithmsand techniques. CoRR, abs/1907.10863, 2019. → pages 5, 7[SVL14] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequenceto sequence learning with neural networks. NIPS’14, page3104–3112, Cambridge, MA, USA, 2014. MIT Press. → pages38[SWY75] Gerard Salton, Anita Wong, and Chung-Shu Yang. A vectorspace model for automatic indexing. Communications of theACM, 18(11):613–620, 1975. → pages 5[TDP19] Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscov-ers the classical nlp pipeline. arXiv preprint arXiv:1905.05950,2019. → pages 70, 82[Ten88] Ted Tenny. Program readability: Procedures versus comments.IEEE Trans. Software Eng., 14:1271–1279, 1988. → pages 2[TWB+19] Michele Tufano, Cody Watson, Gabriele Bavota, Massimil-iano Di Penta, Martin White, and Denys Poshyvanyk. An em-pirical study on learning bug-fixing patches in the wild via neu-ral machine translation. ACM Trans. Softw. Eng. Methodol.,28(4):19:1–19:29, 2019. → pages 35, 52[TXC+19] Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Po-liak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme,Sam Bowman, Dipanjan Das, and Ellie Pavlick. What do youlearn from context? probing for sentence structure in contex-tualized word representations. In International Conference onLearning Representations, 2019. → pages 82[vdMH08] Laurens van der Maaten and Geoffrey Hinton. Visualizing datausing t-sne. Journal of Machine Learning Research, 9(86):2579–2605, 2008. → pages 35103Bibliography[VSP+17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit,Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polo-sukhin. Attention is all you need. CoRR, abs/1706.03762, 2017.→ pages 4, 57, 69, 70[VZP15] Ramakrishna Vedantam, C. Lawrence Zitnick, and DeviParikh. Cider: Consensus-based image description evaluation.In 2015 IEEE Conference on Computer Vision and PatternRecognition (CVPR), pages 4566–4575, 2015. → pages 40, 41[WDS81] S. N. Woodfield, H. E. Dunsmore, and V. Y. Shen. The effect ofmodularization and comments on program comprehension. InProceedings of the 5th International Conference on SoftwareEngineering, ICSE ’81, page 215–223. IEEE Press, 1981. →pages 2[WDS+20] Yanlin Wang, Lun Du, Ensheng Shi, Yuxuan Hu, Shi Han, andDongmei Zhang. Cocogum: Contextual code summarizationwith multi-relational gnn on umls. Technical Report MSR-TR-2020-16, Microsoft, May 2020. → pages 2, 10, 22, 32, 41[WLL+20] Bolin Wei, Yongmin Li, Ge Li, Xin Xia, and Zhi Jin. Re-trieve and Refine: Exemplar-based Neural Comment Genera-tion. arXiv e-prints, page arXiv:2010.04459, October 2020. →pages 1, 2, 9, 10, 38, 68[WLX+19] Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu, and Zhi Jin. Code gen-eration as a dual task of code summarization. In H. Wal-lach, H. Larochelle, A. Beygelzimer, F. d'Alche´-Buc, E. Fox,and R. Garnett, editors, Advances in Neural InformationProcessing Systems 32, pages 6563–6573. Curran Associates,Inc., 2019. → pages 1, 9[WRP19] Yaza Wainakh, Moiz Rauf, and Michael Pradel. Evaluating se-mantic representations of source code. CoRR, abs/1910.05177,2019. → pages 35, 49, 50[WSD+21] Yanlin Wang, Ensheng Shi, Lun Du, Xiaodi Yang, Yuxuan Hu,Shi Han, Hongyu Zhang, and Dongmei Zhang. Cocosum: Con-textual code summarization with multi-relational graph neuralnetwork, 2021. → pages 10104Bibliography[WSS17] Ke Wang, Rishabh Singh, and Zhendong Su. Dynamicneural program embedding for program repair. CoRR,abs/1711.07163, 2017. → pages 1, 12[WTM+19] Martin White, Michele Tufano, Mat´ıas Mart´ınez, Martin Mon-perrus, and Denys Poshyvanyk. Sorting and transforming pro-gram repair ingredients via deep learning code similarities. In2019 IEEE 26th International Conference on Software Analysis,Evolution and Reengineering (SANER), pages 479–490, 2019.→ pages 1, 12[WYT13] Edmund Wong, Jinqiu Yang, and Lin Tan. Autocomment:Mining question and answer sites for automatic comment gen-eration. 2013 28th IEEE/ACM International Conference onAutomated Software Engineering (ASE), pages 562–567, 2013.→ pages 6[WZS+20] Wenhua Wang, Yuqun Zhang, Yulei Sui, Yao Wan, ZhouZhao, Jian Wu, Philip Yu, and Guandong Xu. Reinforcement-learning-guided source code summarization via hierarchical at-tention. IEEE Transactions on Software Engineering, pages1–1, 2020. → pages 66[WZY+18] Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, HaochaoYing, Jian Wu, and Philip S. Yu. Improving automaticsource code summarization via deep reinforcement learning. InProceedings of the 33rd ACM/IEEE International Conferenceon Automated Software Engineering, ASE 2018, page 397–407,New York, NY, USA, 2018. Association for Computing Ma-chinery. → pages 2, 9, 40, 47, 68[WZZX20] Wenhua Wang, Yuqun Zhang, Zhengran Zeng, and Guan-dong Xu. Trans3ˆ: A transformer-based framework for unify-ing code summarization and code search. arXiv e-prints, pagearXiv:2003.03238, March 2020. → pages 4, 69[YHM18] Yilin Yang, Liang Huang, and Mingbo Ma. Breaking thebeam search curse: A study of (re-) scoring methods and stop-ping criteria for neural machine translation. arXiv preprintarXiv:1808.09582, 2018. → pages 31[YPS19] Ziyu Yao, Jayavardhan Reddy Peddamail, and Huan Sun. Coa-cor: Code annotation for code retrieval with reinforcement105Bibliographylearning. In Ling Liu, Ryen W. White, Amin Mantrach, Fab-rizio Silvestri, Julian J. McAuley, Ricardo Baeza-Yates, andLeila Zia, editors, The World Wide Web Conference, WWW2019, San Francisco, CA, USA, May 13-17, 2019, pages 2203–2214. ACM, 2019. → pages 9[YSHZ19] Yong Yu, Xiaosheng Si, Changhua Hu, and Jianxun Zhang. Areview of recurrent neural networks: Lstm cells and networkarchitectures. Neural computation, 31(7):1235–1270, 2019. →pages 16[ZP19] Yuxiang Zhu and Minxue Pan. Automatic code summariza-tion: A systematic literature review. arXiv e-prints, pagearXiv:1909.04352, September 2019. → pages 5[ZWZ+20] Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, andXudong Liu. Retrieval-based neural source code sum-marization. In Proceedings of the ACM/IEEE 42ndInternational Conference on Software Engineering, ICSE ’20,page 1385–1397, New York, NY, USA, 2020. Association forComputing Machinery. → pages 2, 9, 10, 22, 25, 33, 38, 39, 40,41, 44, 47, 53, 57, 64, 65, 67, 68[ZZLW17] Wenhao Zheng, Hong-Yu Zhou, Ming Li, and Jianxin Wu. Codeattention: Translating code to comments by exploiting domainfeatures. CoRR, abs/1709.07642, 2017. → pages 7106Appendix107Appendix ACode SamplesIn this section we provide some of the example summaries generated byLAMNER-Code. These samples are continuation of dicsussion in section7. Figure A.1 represents code samples and the generated comments byLAMNER-Code and other baselines. Important identifiers are sucessfullygenerated by LAMNER-Code. For example ’page’ and ’url’ in the firstexample (ID: 620), ’regular expression’ in the second sample (ID: 7213), idin the comments of the third method (ID: 1221).Samples where all the baselines including LAMNER-Code generate rel-evant summaries despite zero BLEU scores are shown in Figure A.2. Thefirst two examples with IDs 6690 and 1247 have human written commentswhich are for internal use; these texts are not intended to define the func-tionality of the methods. The Reference comments of examples with IDs7712 are outdated, but the generated summaries highlights the importanceof automatic techniques.108Appendix A. Code SamplesFigure A.1: Samples with better BLEU-1 score for LAMNER-Code as com-pared to baselines109Appendix A. Code SamplesFigure A.2: Samples with zero BLEU-1 score110
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-ShareAlike 4.0 International
Graduate
Fard, Fatemeh Hendijani
Source code representation for comment generation and program comprehension
Text